Sat Jan 25 14:24:42 UTC 2025  I: starting to build golang-github-hashicorp-raft/trixie/armhf on jenkins on '2025-01-25 14:24'
Sat Jan 25 14:24:42 UTC 2025  I: The jenkins build log is/was available at https://jenkins.debian.net/userContent/reproducible/debian/build_service/armhf_14/9529/console.log
Sat Jan 25 14:24:42 UTC 2025  I: Downloading source for trixie/golang-github-hashicorp-raft=1.5.0-1
--2025-01-25 14:24:42--  http://deb.debian.org/debian/pool/main/g/golang-github-hashicorp-raft/golang-github-hashicorp-raft_1.5.0-1.dsc
Connecting to 46.16.76.132:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2480 (2.4K) [text/prs.lines.tag]
Saving to: ‘golang-github-hashicorp-raft_1.5.0-1.dsc’

     0K ..                                                    100%  361M=0s

2025-01-25 14:24:42 (361 MB/s) - ‘golang-github-hashicorp-raft_1.5.0-1.dsc’ saved [2480/2480]

Sat Jan 25 14:24:42 UTC 2025  I: golang-github-hashicorp-raft_1.5.0-1.dsc
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 3.0 (quilt)
Source: golang-github-hashicorp-raft
Binary: golang-github-hashicorp-raft-dev
Architecture: all
Version: 1.5.0-1
Maintainer: Debian Go Packaging Team <team+pkg-go@tracker.debian.org>
Uploaders: Tim Potter <tpot@hpe.com>, Dmitry Smirnov <onlyjob@debian.org>,
Homepage: https://github.com/hashicorp/raft
Standards-Version: 4.6.2
Vcs-Browser: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft
Vcs-Git: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git
Testsuite: autopkgtest-pkg-go
Build-Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
Package-List:
 golang-github-hashicorp-raft-dev deb golang optional arch=all
Checksums-Sha1:
 cea1c93d584c6f8dfbf53aef3438ab18ec7e7222 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 015fac87182a135b38ce31d1eb99f68ca251ae23 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Checksums-Sha256:
 f4dc9aabc7784a1b9b5da36b85a3797ac37911f2a6dc54a6fd6cb60a6fd643e9 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 0d95ee03abc0b447cfb92e400d914ecee5113fd181a4ea1d51fa5b6c37288658 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Files:
 7ebc703c8f7e2d7fc02568216f2e3b0a 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 1f0ee65ab76bca0e05a75b8f9ad32d8d 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Go-Import-Path: github.com/hashicorp/raft

-----BEGIN PGP SIGNATURE-----

iQJKBAEBCgA0FiEEMD9oek78sa58GjWjtwAXP7uAWikFAmTCiiIWHGRzd2FyYnJp
Y2tAZGViaWFuLm9yZwAKCRC3ABc/u4BaKYS6D/wN81mTx6pt43be/9H7Ovy6z+5p
nv0+kgrOLtH5y4KUjNLZJuPljAp+uoELmotxp1wqkoLSYs1AwCZ11X4NVs8YoLXj
sLd+Ub+QcWb0Gejqjw7Nk9VZqqXjwfFJW54n8NC2Zy8D6OelCl47sDPmiu03v3uz
oRu4dp2DW/uKkC0vb4RczT81GBmJ19Tu2ZdDvtk7kwnvF48cqDhXMhUTQAEjoyVU
Q8kfY13blza+uXZk6xhWyYsc+wpTh6RHZIWsbpgejKmEYnZCoCdaR0UqOoau33Bs
T1OXVNM0P6vrmeXyoiEy66Q2XyHeei+HYRUgED16Ujg0qxPxSEVE5timSwcXHa/u
xb5ZcKQwqEoJISjWTFDG8gww8MBPOKNGW5BKRdQSdtDhNEsnmn58jl1lsRs5L6Xg
+DrufjU6KKqqYyOBS+H+s68KXleg2t/qP+K8GZjJl/eX3+Tw5xOdt3ZWDMWxB2qF
gS+ASmxYV0Uu/pd0MLWXjAFcllnl93MzPVkF14A8eYrghujLjN3+oUOD0RQzxpAb
LHc5b0iPnN4/YQtJX3fZi2y6c62BCZ5Slb5sB0bsgNPbk1T3qQrtMPWJXUfr32ys
VOow6eNA+VDxNxHZ3aSUiVC0lcUm6dJDYqcfk/PgDdBR/E7iigkTi3O/2rgW0Y7Z
14FQMVvbK31veGXJjA==
=okTI
-----END PGP SIGNATURE-----
Sat Jan 25 14:24:42 UTC 2025  I: Checking whether the package is not for us
Sat Jan 25 14:24:42 UTC 2025  I: Starting 1st build on remote node virt32z-armhf-rb.debian.net.
Sat Jan 25 14:24:42 UTC 2025  I: Preparing to do remote build '1' on virt32z-armhf-rb.debian.net.
Sat Jan 25 14:30:21 UTC 2025  I: Deleting $TMPDIR on virt32z-armhf-rb.debian.net.
I: pbuilder: network access will be disabled during build
I: Current time: Sat Jan 25 02:24:48 -12 2025
I: pbuilder-time-stamp: 1737815088
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/trixie-reproducible-base.tgz]
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: policy-rc.d already exists
I: Copying source file
I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc]
I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz]
I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz]
I: Extracting source
dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: unsupported subcommand
dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying 0001-Skip-unreliable-tests.patch
I: Not using root during the build.
I: Installing the build-deps
I: user script /srv/workspace/pbuilder/22464/tmp/hooks/D02_print_environment starting
I: set
  BUILDDIR='/build/reproducible-path'
  BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other'
  BUILDUSERNAME='pbuilder1'
  BUILD_ARCH='armhf'
  DEBIAN_FRONTEND='noninteractive'
  DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=3 '
  DISTRIBUTION='trixie'
  HOME='/root'
  HOST_ARCH='armhf'
  IFS=' 	
  '
  INVOCATION_ID='c8b9d59d771c4c03a0992e1db6f7a075'
  LANG='C'
  LANGUAGE='en_US:en'
  LC_ALL='C'
  MAIL='/var/mail/root'
  OPTIND='1'
  PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games'
  PBCURRENTCOMMANDLINEOPERATION='build'
  PBUILDER_OPERATION='build'
  PBUILDER_PKGDATADIR='/usr/share/pbuilder'
  PBUILDER_PKGLIBDIR='/usr/lib/pbuilder'
  PBUILDER_SYSCONFDIR='/etc'
  PPID='22464'
  PS1='# '
  PS2='> '
  PS4='+ '
  PWD='/'
  SHELL='/bin/bash'
  SHLVL='2'
  SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.qVeudEaw/pbuilderrc_B4Jn --distribution trixie --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/trixie-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.qVeudEaw/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc'
  SUDO_GID='110'
  SUDO_UID='103'
  SUDO_USER='jenkins'
  TERM='unknown'
  TZ='/usr/share/zoneinfo/Etc/GMT+12'
  USER='root'
  _='/usr/bin/systemd-run'
  http_proxy='http://10.0.0.15:3142/'
I: uname -a
  Linux virt32z 6.1.0-30-armmp-lpae #1 SMP Debian 6.1.124-1 (2025-01-12) armv7l GNU/Linux
I: ls -l /bin
  lrwxrwxrwx 1 root root 7 Nov 22 14:40 /bin -> usr/bin
I: user script /srv/workspace/pbuilder/22464/tmp/hooks/D02_print_environment finished
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: armhf
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 19569 files and directories currently installed.)
Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ...
Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however:
  Package debhelper-compat is not installed.
 pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however:
  Package dh-sequence-golang is not installed.
 pbuilder-satisfydepends-dummy depends on golang-any; however:
  Package golang-any is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however:
  Package golang-github-armon-go-metrics-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however:
  Package golang-github-hashicorp-go-hclog-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however:
  Package golang-github-hashicorp-go-msgpack-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however:
  Package golang-github-stretchr-testify-dev is not installed.

Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
Building tag database...
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
The following NEW packages will be installed:
  autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.23-go{a} golang-1.23-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libunistring5{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} 
The following packages are RECOMMENDED but will NOT be installed:
  curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 
0 packages upgraded, 106 newly installed, 0 to remove and 0 not upgraded.
Need to get 123 MB of archives. After unpacking 681 MB will be used.
Writing extended state information...
Get: 1 http://deb.debian.org/debian trixie/main armhf sensible-utils all 0.0.24 [24.8 kB]
Get: 2 http://deb.debian.org/debian trixie/main armhf libmagic-mgc armhf 1:5.45-3+b1 [314 kB]
Get: 3 http://deb.debian.org/debian trixie/main armhf libmagic1t64 armhf 1:5.45-3+b1 [98.5 kB]
Get: 4 http://deb.debian.org/debian trixie/main armhf file armhf 1:5.45-3+b1 [42.3 kB]
Get: 5 http://deb.debian.org/debian trixie/main armhf gettext-base armhf 0.22.5-4 [196 kB]
Get: 6 http://deb.debian.org/debian trixie/main armhf libuchardet0 armhf 0.0.8-1+b2 [65.6 kB]
Get: 7 http://deb.debian.org/debian trixie/main armhf groff-base armhf 1.23.0-7 [1095 kB]
Get: 8 http://deb.debian.org/debian trixie/main armhf bsdextrautils armhf 2.40.4-1 [84.6 kB]
Get: 9 http://deb.debian.org/debian trixie/main armhf libpipeline1 armhf 1.5.8-1 [35.0 kB]
Get: 10 http://deb.debian.org/debian trixie/main armhf man-db armhf 2.13.0-1 [1382 kB]
Get: 11 http://deb.debian.org/debian trixie/main armhf m4 armhf 1.4.19-5 [272 kB]
Get: 12 http://deb.debian.org/debian trixie/main armhf autoconf all 2.72-3 [493 kB]
Get: 13 http://deb.debian.org/debian trixie/main armhf autotools-dev all 20220109.1 [51.6 kB]
Get: 14 http://deb.debian.org/debian trixie/main armhf automake all 1:1.16.5-1.3 [823 kB]
Get: 15 http://deb.debian.org/debian trixie/main armhf autopoint all 0.22.5-4 [723 kB]
Get: 16 http://deb.debian.org/debian trixie/main armhf libdebhelper-perl all 13.23 [90.6 kB]
Get: 17 http://deb.debian.org/debian trixie/main armhf libtool all 2.5.4-2 [539 kB]
Get: 18 http://deb.debian.org/debian trixie/main armhf dh-autoreconf all 20 [17.1 kB]
Get: 19 http://deb.debian.org/debian trixie/main armhf libarchive-zip-perl all 1.68-1 [104 kB]
Get: 20 http://deb.debian.org/debian trixie/main armhf libfile-stripnondeterminism-perl all 1.14.0-1 [19.5 kB]
Get: 21 http://deb.debian.org/debian trixie/main armhf dh-strip-nondeterminism all 1.14.0-1 [8448 B]
Get: 22 http://deb.debian.org/debian trixie/main armhf libelf1t64 armhf 0.192-4 [184 kB]
Get: 23 http://deb.debian.org/debian trixie/main armhf dwz armhf 0.15-1+b2 [106 kB]
Get: 24 http://deb.debian.org/debian trixie/main armhf libunistring5 armhf 1.3-1 [444 kB]
Get: 25 http://deb.debian.org/debian trixie/main armhf libicu72 armhf 72.1-6 [9086 kB]
Get: 26 http://deb.debian.org/debian trixie/main armhf libxml2 armhf 2.12.7+dfsg+really2.9.14-0.2+b1 [605 kB]
Get: 27 http://deb.debian.org/debian trixie/main armhf gettext armhf 0.22.5-4 [1489 kB]
Get: 28 http://deb.debian.org/debian trixie/main armhf intltool-debian all 0.35.0+20060710.6 [22.9 kB]
Get: 29 http://deb.debian.org/debian trixie/main armhf po-debconf all 1.0.21+nmu1 [248 kB]
Get: 30 http://deb.debian.org/debian trixie/main armhf debhelper all 13.23 [919 kB]
Get: 31 http://deb.debian.org/debian trixie/main armhf dh-golang all 1.62 [24.7 kB]
Get: 32 http://deb.debian.org/debian trixie/main armhf golang-1.23-src all 1.23.5-1 [19.1 MB]
Get: 33 http://deb.debian.org/debian trixie/main armhf golang-1.23-go armhf 1.23.5-1 [23.6 MB]
Get: 34 http://deb.debian.org/debian trixie/main armhf golang-src all 2:1.23~2 [5088 B]
Get: 35 http://deb.debian.org/debian trixie/main armhf golang-go armhf 2:1.23~2 [44.3 kB]
Get: 36 http://deb.debian.org/debian trixie/main armhf golang-any armhf 2:1.23~2 [5172 B]
Get: 37 http://deb.debian.org/debian trixie/main armhf golang-github-alecthomas-units-dev all 0.0~git20240626.61d1e34-1 [9032 B]
Get: 38 http://deb.debian.org/debian trixie/main armhf golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB]
Get: 39 http://deb.debian.org/debian trixie/main armhf golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB]
Get: 40 http://deb.debian.org/debian trixie/main armhf golang-golang-x-sys-dev all 0.22.0-1 [431 kB]
Get: 41 http://deb.debian.org/debian trixie/main armhf golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B]
Get: 42 http://deb.debian.org/debian trixie/main armhf golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB]
Get: 43 http://deb.debian.org/debian trixie/main armhf golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB]
Get: 44 http://deb.debian.org/debian trixie/main armhf golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB]
Get: 45 http://deb.debian.org/debian trixie/main armhf golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB]
Get: 46 http://deb.debian.org/debian trixie/main armhf golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB]
Get: 47 http://deb.debian.org/debian trixie/main armhf golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB]
Get: 48 http://deb.debian.org/debian trixie/main armhf golang-github-stretchr-testify-dev all 1.9.0-1 [74.7 kB]
Get: 49 http://deb.debian.org/debian trixie/main armhf golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB]
Get: 50 http://deb.debian.org/debian trixie/main armhf golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB]
Get: 51 http://deb.debian.org/debian trixie/main armhf golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB]
Get: 52 http://deb.debian.org/debian trixie/main armhf golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B]
Get: 53 http://deb.debian.org/debian trixie/main armhf golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB]
Get: 54 http://deb.debian.org/debian trixie/main armhf golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB]
Get: 55 http://deb.debian.org/debian trixie/main armhf golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB]
Get: 56 http://deb.debian.org/debian trixie/main armhf golang-google-protobuf-dev all 1.33.0-1 [625 kB]
Get: 57 http://deb.debian.org/debian trixie/main armhf golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB]
Get: 58 http://deb.debian.org/debian trixie/main armhf golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B]
Get: 59 http://deb.debian.org/debian trixie/main armhf golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB]
Get: 60 http://deb.debian.org/debian trixie/main armhf golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB]
Get: 61 http://deb.debian.org/debian trixie/main armhf golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB]
Get: 62 http://deb.debian.org/debian trixie/main armhf golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB]
Get: 63 http://deb.debian.org/debian trixie/main armhf golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB]
Get: 64 http://deb.debian.org/debian trixie/main armhf golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B]
Get: 65 http://deb.debian.org/debian trixie/main armhf golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB]
Get: 66 http://deb.debian.org/debian trixie/main armhf golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB]
Get: 67 http://deb.debian.org/debian trixie/main armhf golang-github-klauspost-compress-dev all 1.17.11+ds1-1 [33.3 MB]
Get: 68 http://deb.debian.org/debian trixie/main armhf golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB]
Get: 69 http://deb.debian.org/debian trixie/main armhf golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB]
Get: 70 http://deb.debian.org/debian trixie/main armhf golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB]
Get: 71 http://deb.debian.org/debian trixie/main armhf golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB]
Get: 72 http://deb.debian.org/debian trixie/main armhf golang-golang-x-term-dev all 0.22.0-1 [14.8 kB]
Get: 73 http://deb.debian.org/debian trixie/main armhf golang-golang-x-text-dev all 0.16.0-1 [4003 kB]
Get: 74 http://deb.debian.org/debian trixie/main armhf golang-golang-x-crypto-dev all 1:0.25.0-1 [1682 kB]
Get: 75 http://deb.debian.org/debian trixie/main armhf golang-golang-x-net-dev all 1:0.27.0-1 [898 kB]
Get: 76 http://deb.debian.org/debian trixie/main armhf golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB]
Get: 77 http://deb.debian.org/debian trixie/main armhf golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB]
Get: 78 http://deb.debian.org/debian trixie/main armhf golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB]
Get: 79 http://deb.debian.org/debian trixie/main armhf golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB]
Get: 80 http://deb.debian.org/debian trixie/main armhf golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B]
Get: 81 http://deb.debian.org/debian trixie/main armhf golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B]
Get: 82 http://deb.debian.org/debian trixie/main armhf golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB]
Get: 83 http://deb.debian.org/debian trixie/main armhf golang-golang-x-oauth2-dev all 0.15.0-1 [38.0 kB]
Get: 84 http://deb.debian.org/debian trixie/main armhf golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB]
Get: 85 http://deb.debian.org/debian trixie/main armhf golang-github-prometheus-common-dev all 0.61.0-1 [120 kB]
Get: 86 http://deb.debian.org/debian trixie/main armhf golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB]
Get: 87 http://deb.debian.org/debian trixie/main armhf golang-github-prometheus-procfs-dev all 0.14.0-1 [183 kB]
Get: 88 http://deb.debian.org/debian trixie/main armhf golang-github-prometheus-client-golang-dev all 1.20.5-1 [188 kB]
Get: 89 http://deb.debian.org/debian trixie/main armhf golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB]
Get: 90 http://deb.debian.org/debian trixie/main armhf golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB]
Get: 91 http://deb.debian.org/debian trixie/main armhf golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B]
Get: 92 http://deb.debian.org/debian trixie/main armhf golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB]
Get: 93 http://deb.debian.org/debian trixie/main armhf golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B]
Get: 94 http://deb.debian.org/debian trixie/main armhf libsasl2-modules-db armhf 2.1.28+dfsg1-8+b1 [18.6 kB]
Get: 95 http://deb.debian.org/debian trixie/main armhf libsasl2-2 armhf 2.1.28+dfsg1-8+b1 [50.6 kB]
Get: 96 http://deb.debian.org/debian trixie/main armhf libssl-dev armhf 3.4.0-2 [2431 kB]
Get: 97 http://deb.debian.org/debian trixie/main armhf libsasl2-dev armhf 2.1.28+dfsg1-8+b1 [219 kB]
Get: 98 http://deb.debian.org/debian trixie/main armhf golang-github-kr-text-dev all 0.2.0-2 [10.2 kB]
Get: 99 http://deb.debian.org/debian trixie/main armhf golang-golang-x-mod-dev all 0.19.0-1 [104 kB]
Get: 100 http://deb.debian.org/debian trixie/main armhf golang-github-yuin-goldmark-dev all 1.7.4-1 [113 kB]
Get: 101 http://deb.debian.org/debian trixie/main armhf golang-golang-x-tools-dev all 1:0.23.0+ds-1 [1593 kB]
Get: 102 http://deb.debian.org/debian trixie/main armhf golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB]
Get: 103 http://deb.debian.org/debian trixie/main armhf golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB]
Get: 104 http://deb.debian.org/debian trixie/main armhf golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB]
Get: 105 http://deb.debian.org/debian trixie/main armhf golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB]
Get: 106 http://deb.debian.org/debian trixie/main armhf golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB]
Fetched 123 MB in 11s (11.0 MB/s)
Preconfiguring packages ...
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 19569 files and directories currently installed.)
Preparing to unpack .../000-sensible-utils_0.0.24_all.deb ...
Unpacking sensible-utils (0.0.24) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3+b1_armhf.deb ...
Unpacking libmagic-mgc (1:5.45-3+b1) ...
Selecting previously unselected package libmagic1t64:armhf.
Preparing to unpack .../002-libmagic1t64_1%3a5.45-3+b1_armhf.deb ...
Unpacking libmagic1t64:armhf (1:5.45-3+b1) ...
Selecting previously unselected package file.
Preparing to unpack .../003-file_1%3a5.45-3+b1_armhf.deb ...
Unpacking file (1:5.45-3+b1) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../004-gettext-base_0.22.5-4_armhf.deb ...
Unpacking gettext-base (0.22.5-4) ...
Selecting previously unselected package libuchardet0:armhf.
Preparing to unpack .../005-libuchardet0_0.0.8-1+b2_armhf.deb ...
Unpacking libuchardet0:armhf (0.0.8-1+b2) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../006-groff-base_1.23.0-7_armhf.deb ...
Unpacking groff-base (1.23.0-7) ...
Selecting previously unselected package bsdextrautils.
Preparing to unpack .../007-bsdextrautils_2.40.4-1_armhf.deb ...
Unpacking bsdextrautils (2.40.4-1) ...
Selecting previously unselected package libpipeline1:armhf.
Preparing to unpack .../008-libpipeline1_1.5.8-1_armhf.deb ...
Unpacking libpipeline1:armhf (1.5.8-1) ...
Selecting previously unselected package man-db.
Preparing to unpack .../009-man-db_2.13.0-1_armhf.deb ...
Unpacking man-db (2.13.0-1) ...
Selecting previously unselected package m4.
Preparing to unpack .../010-m4_1.4.19-5_armhf.deb ...
Unpacking m4 (1.4.19-5) ...
Selecting previously unselected package autoconf.
Preparing to unpack .../011-autoconf_2.72-3_all.deb ...
Unpacking autoconf (2.72-3) ...
Selecting previously unselected package autotools-dev.
Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ...
Unpacking autotools-dev (20220109.1) ...
Selecting previously unselected package automake.
Preparing to unpack .../013-automake_1%3a1.16.5-1.3_all.deb ...
Unpacking automake (1:1.16.5-1.3) ...
Selecting previously unselected package autopoint.
Preparing to unpack .../014-autopoint_0.22.5-4_all.deb ...
Unpacking autopoint (0.22.5-4) ...
Selecting previously unselected package libdebhelper-perl.
Preparing to unpack .../015-libdebhelper-perl_13.23_all.deb ...
Unpacking libdebhelper-perl (13.23) ...
Selecting previously unselected package libtool.
Preparing to unpack .../016-libtool_2.5.4-2_all.deb ...
Unpacking libtool (2.5.4-2) ...
Selecting previously unselected package dh-autoreconf.
Preparing to unpack .../017-dh-autoreconf_20_all.deb ...
Unpacking dh-autoreconf (20) ...
Selecting previously unselected package libarchive-zip-perl.
Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ...
Unpacking libarchive-zip-perl (1.68-1) ...
Selecting previously unselected package libfile-stripnondeterminism-perl.
Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.0-1_all.deb ...
Unpacking libfile-stripnondeterminism-perl (1.14.0-1) ...
Selecting previously unselected package dh-strip-nondeterminism.
Preparing to unpack .../020-dh-strip-nondeterminism_1.14.0-1_all.deb ...
Unpacking dh-strip-nondeterminism (1.14.0-1) ...
Selecting previously unselected package libelf1t64:armhf.
Preparing to unpack .../021-libelf1t64_0.192-4_armhf.deb ...
Unpacking libelf1t64:armhf (0.192-4) ...
Selecting previously unselected package dwz.
Preparing to unpack .../022-dwz_0.15-1+b2_armhf.deb ...
Unpacking dwz (0.15-1+b2) ...
Selecting previously unselected package libunistring5:armhf.
Preparing to unpack .../023-libunistring5_1.3-1_armhf.deb ...
Unpacking libunistring5:armhf (1.3-1) ...
Selecting previously unselected package libicu72:armhf.
Preparing to unpack .../024-libicu72_72.1-6_armhf.deb ...
Unpacking libicu72:armhf (72.1-6) ...
Selecting previously unselected package libxml2:armhf.
Preparing to unpack .../025-libxml2_2.12.7+dfsg+really2.9.14-0.2+b1_armhf.deb ...
Unpacking libxml2:armhf (2.12.7+dfsg+really2.9.14-0.2+b1) ...
Selecting previously unselected package gettext.
Preparing to unpack .../026-gettext_0.22.5-4_armhf.deb ...
Unpacking gettext (0.22.5-4) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../027-intltool-debian_0.35.0+20060710.6_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.6) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../028-po-debconf_1.0.21+nmu1_all.deb ...
Unpacking po-debconf (1.0.21+nmu1) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../029-debhelper_13.23_all.deb ...
Unpacking debhelper (13.23) ...
Selecting previously unselected package dh-golang.
Preparing to unpack .../030-dh-golang_1.62_all.deb ...
Unpacking dh-golang (1.62) ...
Selecting previously unselected package golang-1.23-src.
Preparing to unpack .../031-golang-1.23-src_1.23.5-1_all.deb ...
Unpacking golang-1.23-src (1.23.5-1) ...
Selecting previously unselected package golang-1.23-go.
Preparing to unpack .../032-golang-1.23-go_1.23.5-1_armhf.deb ...
Unpacking golang-1.23-go (1.23.5-1) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../033-golang-src_2%3a1.23~2_all.deb ...
Unpacking golang-src (2:1.23~2) ...
Selecting previously unselected package golang-go:armhf.
Preparing to unpack .../034-golang-go_2%3a1.23~2_armhf.deb ...
Unpacking golang-go:armhf (2:1.23~2) ...
Selecting previously unselected package golang-any:armhf.
Preparing to unpack .../035-golang-any_2%3a1.23~2_armhf.deb ...
Unpacking golang-any:armhf (2:1.23~2) ...
Selecting previously unselected package golang-github-alecthomas-units-dev.
Preparing to unpack .../036-golang-github-alecthomas-units-dev_0.0~git20240626.61d1e34-1_all.deb ...
Unpacking golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ...
Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev.
Preparing to unpack .../037-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ...
Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev.
Preparing to unpack .../038-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ...
Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Selecting previously unselected package golang-golang-x-sys-dev.
Preparing to unpack .../039-golang-golang-x-sys-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-sys-dev (0.22.0-1) ...
Selecting previously unselected package golang-github-mattn-go-isatty-dev.
Preparing to unpack .../040-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ...
Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Selecting previously unselected package golang-github-mattn-go-colorable-dev.
Preparing to unpack .../041-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ...
Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Selecting previously unselected package golang-github-fatih-color-dev.
Preparing to unpack .../042-golang-github-fatih-color-dev_1.17.0-1_all.deb ...
Unpacking golang-github-fatih-color-dev (1.17.0-1) ...
Selecting previously unselected package golang-github-davecgh-go-spew-dev.
Preparing to unpack .../043-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ...
Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Selecting previously unselected package golang-github-pmezard-go-difflib-dev.
Preparing to unpack .../044-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ...
Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Selecting previously unselected package golang-github-stretchr-objx-dev.
Preparing to unpack .../045-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ...
Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ...
Selecting previously unselected package golang-gopkg-yaml.v3-dev.
Preparing to unpack .../046-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ...
Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Selecting previously unselected package golang-github-stretchr-testify-dev.
Preparing to unpack .../047-golang-github-stretchr-testify-dev_1.9.0-1_all.deb ...
Unpacking golang-github-stretchr-testify-dev (1.9.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-hclog-dev.
Preparing to unpack .../048-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ...
Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev.
Preparing to unpack .../049-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ...
Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Selecting previously unselected package golang-github-pkg-errors-dev.
Preparing to unpack .../050-golang-github-pkg-errors-dev_0.9.1-3_all.deb ...
Unpacking golang-github-pkg-errors-dev (0.9.1-3) ...
Selecting previously unselected package golang-github-tv42-httpunix-dev.
Preparing to unpack .../051-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ...
Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev.
Preparing to unpack .../052-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ...
Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Selecting previously unselected package golang-github-datadog-datadog-go-dev.
Preparing to unpack .../053-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ...
Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Selecting previously unselected package golang-github-google-go-cmp-dev.
Preparing to unpack .../054-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ...
Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ...
Selecting previously unselected package golang-google-protobuf-dev.
Preparing to unpack .../055-golang-google-protobuf-dev_1.33.0-1_all.deb ...
Unpacking golang-google-protobuf-dev (1.33.0-1) ...
Selecting previously unselected package golang-github-golang-protobuf-1-5-dev.
Preparing to unpack .../056-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ...
Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Selecting previously unselected package golang-github-hashicorp-go-uuid-dev.
Preparing to unpack .../057-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ...
Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Selecting previously unselected package golang-github-hashicorp-golang-lru-dev.
Preparing to unpack .../058-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ...
Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev.
Preparing to unpack .../059-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ...
Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Selecting previously unselected package golang-github-pascaldekloe-goe-dev.
Preparing to unpack .../060-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ...
Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Selecting previously unselected package golang-github-beorn7-perks-dev.
Preparing to unpack .../061-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ...
Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ...
Selecting previously unselected package golang-github-cespare-xxhash-dev.
Preparing to unpack .../062-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ...
Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ...
Selecting previously unselected package golang-github-modern-go-concurrent-dev.
Preparing to unpack .../063-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ...
Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Selecting previously unselected package golang-github-modern-go-reflect2-dev.
Preparing to unpack .../064-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ...
Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-json-iterator-go-dev.
Preparing to unpack .../065-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ...
Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ...
Selecting previously unselected package golang-github-klauspost-compress-dev.
Preparing to unpack .../066-golang-github-klauspost-compress-dev_1.17.11+ds1-1_all.deb ...
Unpacking golang-github-klauspost-compress-dev (1.17.11+ds1-1) ...
Selecting previously unselected package golang-github-kylelemons-godebug-dev.
Preparing to unpack .../067-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ...
Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Selecting previously unselected package golang-github-prometheus-client-model-dev.
Preparing to unpack .../068-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ...
Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ...
Selecting previously unselected package golang-gopkg-yaml.v2-dev.
Preparing to unpack .../069-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ...
Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Selecting previously unselected package golang-github-jmespath-go-jmespath-dev.
Preparing to unpack .../070-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ...
Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Selecting previously unselected package golang-golang-x-term-dev.
Preparing to unpack .../071-golang-golang-x-term-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-term-dev (0.22.0-1) ...
Selecting previously unselected package golang-golang-x-text-dev.
Preparing to unpack .../072-golang-golang-x-text-dev_0.16.0-1_all.deb ...
Unpacking golang-golang-x-text-dev (0.16.0-1) ...
Selecting previously unselected package golang-golang-x-crypto-dev.
Preparing to unpack .../073-golang-golang-x-crypto-dev_1%3a0.25.0-1_all.deb ...
Unpacking golang-golang-x-crypto-dev (1:0.25.0-1) ...
Selecting previously unselected package golang-golang-x-net-dev.
Preparing to unpack .../074-golang-golang-x-net-dev_1%3a0.27.0-1_all.deb ...
Unpacking golang-golang-x-net-dev (1:0.27.0-1) ...
Selecting previously unselected package golang-github-aws-aws-sdk-go-dev.
Preparing to unpack .../075-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ...
Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Selecting previously unselected package golang-github-go-logfmt-logfmt-dev.
Preparing to unpack .../076-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ...
Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Selecting previously unselected package golang-github-go-kit-log-dev.
Preparing to unpack .../077-golang-github-go-kit-log-dev_0.2.1-1_all.deb ...
Unpacking golang-github-go-kit-log-dev (0.2.1-1) ...
Selecting previously unselected package golang-github-julienschmidt-httprouter-dev.
Preparing to unpack .../078-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ...
Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Selecting previously unselected package golang-github-munnerz-goautoneg-dev.
Preparing to unpack .../079-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ...
Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Selecting previously unselected package golang-github-jpillora-backoff-dev.
Preparing to unpack .../080-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ...
Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev.
Preparing to unpack .../081-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ...
Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Selecting previously unselected package golang-golang-x-oauth2-dev.
Preparing to unpack .../082-golang-golang-x-oauth2-dev_0.15.0-1_all.deb ...
Unpacking golang-golang-x-oauth2-dev (0.15.0-1) ...
Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev.
Preparing to unpack .../083-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ...
Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Selecting previously unselected package golang-github-prometheus-common-dev.
Preparing to unpack .../084-golang-github-prometheus-common-dev_0.61.0-1_all.deb ...
Unpacking golang-github-prometheus-common-dev (0.61.0-1) ...
Selecting previously unselected package golang-golang-x-sync-dev.
Preparing to unpack .../085-golang-golang-x-sync-dev_0.7.0-1_all.deb ...
Unpacking golang-golang-x-sync-dev (0.7.0-1) ...
Selecting previously unselected package golang-github-prometheus-procfs-dev.
Preparing to unpack .../086-golang-github-prometheus-procfs-dev_0.14.0-1_all.deb ...
Unpacking golang-github-prometheus-procfs-dev (0.14.0-1) ...
Selecting previously unselected package golang-github-prometheus-client-golang-dev.
Preparing to unpack .../087-golang-github-prometheus-client-golang-dev_1.20.5-1_all.deb ...
Unpacking golang-github-prometheus-client-golang-dev (1.20.5-1) ...
Selecting previously unselected package golang-github-armon-go-metrics-dev.
Preparing to unpack .../088-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ...
Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ...
Selecting previously unselected package golang-github-creack-pty-dev.
Preparing to unpack .../089-golang-github-creack-pty-dev_1.1.21-1_all.deb ...
Unpacking golang-github-creack-pty-dev (1.1.21-1) ...
Selecting previously unselected package golang-github-vmihailenco-tagparser-dev.
Preparing to unpack .../090-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ...
Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev.
Preparing to unpack .../091-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ...
Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Selecting previously unselected package golang-gopkg-tomb.v2-dev.
Preparing to unpack .../092-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ...
Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Selecting previously unselected package libsasl2-modules-db:armhf.
Preparing to unpack .../093-libsasl2-modules-db_2.1.28+dfsg1-8+b1_armhf.deb ...
Unpacking libsasl2-modules-db:armhf (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package libsasl2-2:armhf.
Preparing to unpack .../094-libsasl2-2_2.1.28+dfsg1-8+b1_armhf.deb ...
Unpacking libsasl2-2:armhf (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package libssl-dev:armhf.
Preparing to unpack .../095-libssl-dev_3.4.0-2_armhf.deb ...
Unpacking libssl-dev:armhf (3.4.0-2) ...
Selecting previously unselected package libsasl2-dev.
Preparing to unpack .../096-libsasl2-dev_2.1.28+dfsg1-8+b1_armhf.deb ...
Unpacking libsasl2-dev (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package golang-github-kr-text-dev.
Preparing to unpack .../097-golang-github-kr-text-dev_0.2.0-2_all.deb ...
Unpacking golang-github-kr-text-dev (0.2.0-2) ...
Selecting previously unselected package golang-golang-x-mod-dev.
Preparing to unpack .../098-golang-golang-x-mod-dev_0.19.0-1_all.deb ...
Unpacking golang-golang-x-mod-dev (0.19.0-1) ...
Selecting previously unselected package golang-github-yuin-goldmark-dev.
Preparing to unpack .../099-golang-github-yuin-goldmark-dev_1.7.4-1_all.deb ...
Unpacking golang-github-yuin-goldmark-dev (1.7.4-1) ...
Selecting previously unselected package golang-golang-x-tools-dev.
Preparing to unpack .../100-golang-golang-x-tools-dev_1%3a0.23.0+ds-1_all.deb ...
Unpacking golang-golang-x-tools-dev (1:0.23.0+ds-1) ...
Selecting previously unselected package golang-github-rogpeppe-go-internal-dev.
Preparing to unpack .../101-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ...
Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Selecting previously unselected package golang-github-kr-pretty-dev.
Preparing to unpack .../102-golang-github-kr-pretty-dev_0.3.1-1_all.deb ...
Unpacking golang-github-kr-pretty-dev (0.3.1-1) ...
Selecting previously unselected package golang-gopkg-check.v1-dev.
Preparing to unpack .../103-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ...
Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Selecting previously unselected package golang-gopkg-mgo.v2-dev.
Preparing to unpack .../104-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ...
Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev.
Preparing to unpack .../105-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ...
Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up libpipeline1:armhf (1.5.8-1) ...
Setting up golang-github-google-go-cmp-dev (0.6.0-1) ...
Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Setting up libicu72:armhf (72.1-6) ...
Setting up bsdextrautils (2.40.4-1) ...
Setting up golang-github-creack-pty-dev (1.1.21-1) ...
Setting up golang-github-stretchr-objx-dev (0.5.2-1) ...
Setting up libmagic-mgc (1:5.45-3+b1) ...
Setting up golang-github-pkg-errors-dev (0.9.1-3) ...
Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Setting up libarchive-zip-perl (1.68-1) ...
Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Setting up libdebhelper-perl (13.23) ...
Setting up golang-golang-x-sys-dev (0.22.0-1) ...
Setting up libmagic1t64:armhf (1:5.45-3+b1) ...
Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Setting up gettext-base (0.22.5-4) ...
Setting up m4 (1.4.19-5) ...
Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Setting up golang-golang-x-term-dev (0.22.0-1) ...
Setting up file (1:5.45-3+b1) ...
Setting up golang-github-beorn7-perks-dev (1.0.1-1) ...
Setting up libelf1t64:armhf (0.192-4) ...
Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Setting up libsasl2-modules-db:armhf (2.1.28+dfsg1-8+b1) ...
Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ...
Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Setting up autotools-dev (20220109.1) ...
Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Setting up golang-google-protobuf-dev (1.33.0-1) ...
Setting up golang-github-kr-text-dev (0.2.0-2) ...
Setting up golang-github-stretchr-testify-dev (1.9.0-1) ...
Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Setting up golang-github-yuin-goldmark-dev (1.7.4-1) ...
Setting up libunistring5:armhf (1.3-1) ...
Setting up libssl-dev:armhf (3.4.0-2) ...
Setting up golang-golang-x-sync-dev (0.7.0-1) ...
Setting up golang-golang-x-mod-dev (0.19.0-1) ...
Setting up golang-1.23-src (1.23.5-1) ...
Setting up autopoint (0.22.5-4) ...
Setting up golang-github-klauspost-compress-dev (1.17.11+ds1-1) ...
Setting up libsasl2-2:armhf (2.1.28+dfsg1-8+b1) ...
Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Setting up autoconf (2.72-3) ...
Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Setting up dwz (0.15-1+b2) ...
Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Setting up sensible-utils (0.0.24) ...
Setting up golang-golang-x-text-dev (0.16.0-1) ...
Setting up libuchardet0:armhf (0.0.8-1+b2) ...
Setting up golang-github-prometheus-procfs-dev (0.14.0-1) ...
Setting up golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ...
Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Setting up golang-golang-x-crypto-dev (1:0.25.0-1) ...
Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ...
Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Setting up golang-golang-x-oauth2-dev (0.15.0-1) ...
Setting up golang-src (2:1.23~2) ...
Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Setting up libxml2:armhf (2.12.7+dfsg+really2.9.14-0.2+b1) ...
Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Setting up automake (1:1.16.5-1.3) ...
update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode
Setting up libfile-stripnondeterminism-perl (1.14.0-1) ...
Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Setting up golang-github-go-kit-log-dev (0.2.1-1) ...
Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Setting up gettext (0.22.5-4) ...
Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Setting up libtool (2.5.4-2) ...
Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Setting up golang-github-json-iterator-go-dev (1.1.12-2) ...
Setting up intltool-debian (0.35.0+20060710.6) ...
Setting up dh-autoreconf (20) ...
Setting up golang-1.23-go (1.23.5-1) ...
Setting up libsasl2-dev (2.1.28+dfsg1-8+b1) ...
Setting up golang-golang-x-net-dev (1:0.27.0-1) ...
Setting up dh-strip-nondeterminism (1.14.0-1) ...
Setting up groff-base (1.23.0-7) ...
Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Setting up golang-github-fatih-color-dev (1.17.0-1) ...
Setting up golang-go:armhf (2:1.23~2) ...
Setting up po-debconf (1.0.21+nmu1) ...
Setting up golang-golang-x-tools-dev (1:0.23.0+ds-1) ...
Setting up man-db (2.13.0-1) ...
Not building database; man-db/auto-update is not 'true'.
Setting up golang-any:armhf (2:1.23~2) ...
Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Setting up golang-github-kr-pretty-dev (0.3.1-1) ...
Setting up debhelper (13.23) ...
Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Setting up dh-golang (1.62) ...
Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Setting up golang-github-prometheus-common-dev (0.61.0-1) ...
Setting up golang-github-prometheus-client-golang-dev (1.20.5-1) ...
Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ...
Processing triggers for libc-bin (2.40-5) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Building tag database...
 -> Finished parsing the build-deps
I: Building the package
I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S  > ../golang-github-hashicorp-raft_1.5.0-1_source.changes
dpkg-buildpackage: info: source package golang-github-hashicorp-raft
dpkg-buildpackage: info: source version 1.5.0-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Daniel Swarbrick <dswarbrick@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture armhf
 debian/rules clean
dh clean --buildsystem=golang
   dh_auto_clean -O--buildsystem=golang
   dh_autoreconf_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules binary
dh binary --buildsystem=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-arm-linux-gnueabihf && go install -trimpath -v -p 3 github.com/hashicorp/raft github.com/hashicorp/raft/bench
internal/unsafeheader
internal/goarch
internal/cpu
internal/byteorder
internal/abi
internal/chacha8rand
internal/bytealg
internal/coverage/rtcov
internal/godebugs
internal/goexperiment
internal/goos
internal/profilerecord
internal/runtime/atomic
internal/runtime/syscall
internal/stringslite
runtime/internal/math
internal/runtime/exithook
runtime/internal/sys
internal/race
sync/atomic
unicode
runtime
unicode/utf8
container/list
internal/asan
internal/itoa
internal/msan
math/bits
math
crypto/internal/alias
crypto/internal/boring/sig
cmp
encoding
unicode/utf16
log/internal
internal/nettrace
vendor/golang.org/x/crypto/cryptobyte/asn1
vendor/golang.org/x/crypto/internal/alias
internal/reflectlite
crypto/subtle
sync
iter
slices
internal/weak
maps
internal/bisect
internal/testlog
internal/singleflight
errors
internal/godebug
io
internal/oserror
syscall
bytes
runtime/cgo
strings
time
bufio
hash
strconv
context
crypto
crypto/cipher
crypto/internal/randutil
internal/syscall/unix
reflect
crypto/internal/boring
path
io/fs
internal/filepathlite
internal/poll
internal/syscall/execenv
math/rand
os
github.com/hashicorp/golang-lru/simplelru
sort
github.com/hashicorp/go-immutable-radix
vendor/golang.org/x/net/dns/dnsmessage
math/rand/v2
internal/fmtsort
fmt
encoding/binary
internal/concurrent
unique
encoding/base64
hash/crc32
crypto/aes
net/netip
math/big
encoding/json
log
net
compress/flate
compress/gzip
crypto/rand
crypto/des
crypto/internal/edwards25519/field
crypto/internal/nistec/fiat
crypto/internal/bigmod
crypto/internal/boring/bbig
crypto/sha512
encoding/asn1
vendor/golang.org/x/crypto/cryptobyte
crypto/internal/edwards25519
crypto/ed25519
crypto/internal/nistec
crypto/hmac
vendor/golang.org/x/crypto/chacha20
vendor/golang.org/x/crypto/internal/poly1305
vendor/golang.org/x/crypto/chacha20poly1305
vendor/golang.org/x/crypto/hkdf
vendor/golang.org/x/sys/cpu
crypto/ecdh
crypto/elliptic
crypto/internal/hpke
vendor/golang.org/x/crypto/sha3
crypto/ecdsa
crypto/internal/mlkem768
crypto/md5
crypto/rc4
crypto/rsa
crypto/sha1
crypto/sha256
crypto/dsa
encoding/hex
encoding/pem
crypto/x509/pkix
net/url
path/filepath
vendor/golang.org/x/text/transform
vendor/golang.org/x/text/unicode/bidi
vendor/golang.org/x/text/unicode/norm
vendor/golang.org/x/text/secure/bidirule
vendor/golang.org/x/net/http2/hpack
crypto/x509
net/textproto
mime
vendor/golang.org/x/net/idna
mime/quotedprintable
mime/multipart
vendor/golang.org/x/net/http/httpguts
crypto/tls
vendor/golang.org/x/net/http/httpproxy
net/http/internal
net/http/internal/ascii
os/signal
golang.org/x/sys/unix
io/ioutil
regexp/syntax
regexp
internal/saferio
encoding/gob
github.com/mattn/go-isatty
github.com/mattn/go-colorable
github.com/fatih/color
net/http/httptrace
github.com/hashicorp/go-hclog
net/http
go/token
html
text/template/parse
hash/crc64
flag
internal/sysinfo
runtime/debug
text/template
runtime/trace
testing
html/template
net/rpc
github.com/armon/go-metrics
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/raft
github.com/hashicorp/raft/bench
   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
dh_auto_test
	cd obj-arm-linux-gnueabihf && go test -vet=off -v -p 3 github.com/hashicorp/raft github.com/hashicorp/raft/bench
?   	github.com/hashicorp/raft/bench	[no test files]
=== RUN   TestCommitment_setVoters
--- PASS: TestCommitment_setVoters (0.00s)
=== RUN   TestCommitment_match_max
--- PASS: TestCommitment_match_max (0.00s)
=== RUN   TestCommitment_match_nonVoting
--- PASS: TestCommitment_match_nonVoting (0.00s)
=== RUN   TestCommitment_recalculate
--- PASS: TestCommitment_recalculate (0.00s)
=== RUN   TestCommitment_recalculate_startIndex
--- PASS: TestCommitment_recalculate_startIndex (0.00s)
=== RUN   TestCommitment_noVoterSanity
--- PASS: TestCommitment_noVoterSanity (0.00s)
=== RUN   TestCommitment_singleVoter
--- PASS: TestCommitment_singleVoter (0.00s)
=== RUN   TestConfiguration_Configuration_Clone
--- PASS: TestConfiguration_Configuration_Clone (0.00s)
=== RUN   TestConfiguration_configurations_Clone
--- PASS: TestConfiguration_configurations_Clone (0.00s)
=== RUN   TestConfiguration_hasVote
--- PASS: TestConfiguration_hasVote (0.00s)
=== RUN   TestConfiguration_checkConfiguration
--- PASS: TestConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_nextConfiguration_table
--- PASS: TestConfiguration_nextConfiguration_table (0.00s)
=== RUN   TestConfiguration_nextConfiguration_prevIndex
--- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
=== RUN   TestConfiguration_nextConfiguration_checkConfiguration
--- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_encodeDecodePeers
--- PASS: TestConfiguration_encodeDecodePeers (0.00s)
=== RUN   TestConfiguration_encodeDecodeConfiguration
--- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
=== RUN   TestDiscardSnapshotStoreImpl
--- PASS: TestDiscardSnapshotStoreImpl (0.00s)
=== RUN   TestDiscardSnapshotSinkImpl
--- PASS: TestDiscardSnapshotSinkImpl (0.00s)
=== RUN   TestFileSnapshotStoreImpl
--- PASS: TestFileSnapshotStoreImpl (0.00s)
=== RUN   TestFileSnapshotSinkImpl
--- PASS: TestFileSnapshotSinkImpl (0.00s)
=== RUN   TestFileSS_CreateSnapshotMissingParentDir
2025-01-25T14:28:10.153Z [INFO]  creating new snapshot: path=/tmp/raft986191241/raft904987765/snapshots/3-10-1737815290153.tmp
--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.01s)
=== RUN   TestFileSS_CreateSnapshot
2025-01-25T14:28:10.162Z [INFO]  creating new snapshot: path=/tmp/raft1671639067/snapshots/3-10-1737815290162.tmp
2025-01-25T14:28:10.167Z [WARN]  found temporary snapshot: name=3-10-1737815290162.tmp
--- PASS: TestFileSS_CreateSnapshot (0.02s)
=== RUN   TestFileSS_CancelSnapshot
2025-01-25T14:28:10.179Z [INFO]  creating new snapshot: path=/tmp/raft3437335716/snapshots/3-10-1737815290179.tmp
--- PASS: TestFileSS_CancelSnapshot (0.01s)
=== RUN   TestFileSS_Retention
2025-01-25T14:28:10.188Z [INFO]  creating new snapshot: path=/tmp/raft3908741870/snapshots/3-10-1737815290188.tmp
2025-01-25T14:28:10.203Z [INFO]  creating new snapshot: path=/tmp/raft3908741870/snapshots/3-11-1737815290203.tmp
2025-01-25T14:28:10.218Z [INFO]  creating new snapshot: path=/tmp/raft3908741870/snapshots/3-12-1737815290218.tmp
2025-01-25T14:28:10.232Z [INFO]  reaping snapshot: path=/tmp/raft3908741870/snapshots/3-10-1737815290188
2025-01-25T14:28:10.232Z [INFO]  creating new snapshot: path=/tmp/raft3908741870/snapshots/3-13-1737815290232.tmp
2025-01-25T14:28:10.248Z [INFO]  reaping snapshot: path=/tmp/raft3908741870/snapshots/3-11-1737815290203
2025-01-25T14:28:10.248Z [INFO]  creating new snapshot: path=/tmp/raft3908741870/snapshots/3-14-1737815290248.tmp
2025-01-25T14:28:10.262Z [INFO]  reaping snapshot: path=/tmp/raft3908741870/snapshots/3-12-1737815290218
--- PASS: TestFileSS_Retention (0.08s)
=== RUN   TestFileSS_BadPerm
--- PASS: TestFileSS_BadPerm (0.00s)
=== RUN   TestFileSS_MissingParentDir
--- PASS: TestFileSS_MissingParentDir (0.00s)
=== RUN   TestFileSS_Ordering
2025-01-25T14:28:10.265Z [INFO]  creating new snapshot: path=/tmp/raft2509995759/snapshots/5-130350-1737815290265.tmp
2025-01-25T14:28:10.281Z [INFO]  creating new snapshot: path=/tmp/raft2509995759/snapshots/36-204917-1737815290281.tmp
--- PASS: TestFileSS_Ordering (0.03s)
=== RUN   TestDeferFutureSuccess
--- PASS: TestDeferFutureSuccess (0.00s)
=== RUN   TestDeferFutureError
--- PASS: TestDeferFutureError (0.00s)
=== RUN   TestDeferFutureConcurrent
--- PASS: TestDeferFutureConcurrent (0.00s)
=== RUN   TestInmemSnapshotStoreImpl
--- PASS: TestInmemSnapshotStoreImpl (0.00s)
=== RUN   TestInmemSnapshotSinkImpl
--- PASS: TestInmemSnapshotSinkImpl (0.00s)
=== RUN   TestInmemSS_CreateSnapshot
--- PASS: TestInmemSS_CreateSnapshot (0.00s)
=== RUN   TestInmemSS_OpenSnapshotTwice
--- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
=== RUN   TestInmemTransportImpl
--- PASS: TestInmemTransportImpl (0.00s)
=== RUN   TestInmemTransportWriteTimeout
--- PASS: TestInmemTransportWriteTimeout (0.01s)
=== RUN   TestRaft_Integ
--- SKIP: TestRaft_Integ (0.00s)
=== RUN   TestRaft_RestartFollower_LongInitialHeartbeat
--- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s)
=== RUN   TestLogCache
--- PASS: TestLogCache (0.00s)
=== RUN   TestLogCacheWithBackendStoreError
--- PASS: TestLogCacheWithBackendStoreError (0.00s)
=== RUN   TestOldestLog
=== RUN   TestOldestLog/empty_logs
=== RUN   TestOldestLog/simple_case
--- PASS: TestOldestLog (0.00s)
    --- PASS: TestOldestLog/empty_logs (0.00s)
    --- PASS: TestOldestLog/simple_case (0.00s)
=== RUN   TestEmitsLogStoreMetrics
--- PASS: TestEmitsLogStoreMetrics (0.01s)
=== RUN   TestNetworkTransport_CloseStreams
--- PASS: TestNetworkTransport_CloseStreams (0.01s)
=== RUN   TestNetworkTransport_StartStop
--- PASS: TestNetworkTransport_StartStop (0.00s)
=== RUN   TestNetworkTransport_Heartbeat_FastPath
--- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s)
=== RUN   TestNetworkTransport_AppendEntries
--- PASS: TestNetworkTransport_AppendEntries (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline
--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.01s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_CloseStreams
--- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.03s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0
2025-01-25T14:28:10.420Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2
2025-01-25T14:28:10.474Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10
--- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s)
=== RUN   TestNetworkTransport_RequestVote
2025-01-25T14:28:10.526Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_RequestVote (0.00s)
=== RUN   TestNetworkTransport_InstallSnapshot
--- PASS: TestNetworkTransport_InstallSnapshot (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode
--- PASS: TestNetworkTransport_EncodeDecode (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode_AddressProvider
--- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
=== RUN   TestNetworkTransport_PooledConn
--- PASS: TestNetworkTransport_PooledConn (0.01s)
=== RUN   TestNetworkTransport_ListenBackoff
    net_transport_test.go:839: countingWriter: 2025-01-25T14:28:10.543Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-25T14:28:10.549Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-25T14:28:10.559Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-25T14:28:10.580Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-25T14:28:10.621Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-25T14:28:10.702Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-25T14:28:10.862Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-25T14:28:11.183Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-25T14:28:11.823Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-25T14:28:12.826Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-25T14:28:13.826Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
--- PASS: TestNetworkTransport_ListenBackoff (4.00s)
=== RUN   TestPeersJSON_BadConfiguration
--- PASS: TestPeersJSON_BadConfiguration (0.00s)
=== RUN   TestPeersJSON_ReadPeersJSON
--- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
=== RUN   TestPeersJSON_ReadConfigJSON
--- PASS: TestPeersJSON_ReadConfigJSON (0.00s)
=== RUN   TestRaft_StartStop
2025-01-25T14:28:14.546Z [INFO]  server-122b83cd-22b5-4287-9bde-04a0b50e2f6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-122b83cd-22b5-4287-9bde-04a0b50e2f6a Address:122b83cd-22b5-4287-9bde-04a0b50e2f6a}]"
--- PASS: TestRaft_StartStop (0.00s)
=== RUN   TestRaft_AfterShutdown
2025-01-25T14:28:14.547Z [INFO]  server-2c8c72a6-0931-e149-8e4b-641d0f405666: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c8c72a6-0931-e149-8e4b-641d0f405666 Address:2c8c72a6-0931-e149-8e4b-641d0f405666}]"
--- PASS: TestRaft_AfterShutdown (0.00s)
=== RUN   TestRaft_LiveBootstrap
2025-01-25T14:28:14.549Z [INFO]  server-f9c860e8-09cd-2778-edef-754deda31728: initial configuration: index=0 servers=[]
2025-01-25T14:28:14.549Z [INFO]  server-3caa1664-b24f-f6c0-aa9c-895ee1f3c378: initial configuration: index=0 servers=[]
2025-01-25T14:28:14.549Z [INFO]  server-16ab997b-d653-d66c-71ce-cacf38299658: initial configuration: index=0 servers=[]
2025-01-25T14:28:14.549Z [INFO]  server-f9c860e8-09cd-2778-edef-754deda31728: entering follower state: follower="Node at f9c860e8-09cd-2778-edef-754deda31728 [Follower]" leader-address= leader-id=
2025-01-25T14:28:14.549Z [INFO]  server-3caa1664-b24f-f6c0-aa9c-895ee1f3c378: entering follower state: follower="Node at 3caa1664-b24f-f6c0-aa9c-895ee1f3c378 [Follower]" leader-address= leader-id=
2025-01-25T14:28:14.549Z [INFO]  server-16ab997b-d653-d66c-71ce-cacf38299658: entering follower state: follower="Node at 16ab997b-d653-d66c-71ce-cacf38299658 [Follower]" leader-address= leader-id=
2025-01-25T14:28:14.549Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:14.610Z [WARN]  server-f9c860e8-09cd-2778-edef-754deda31728: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:14.610Z [INFO]  server-f9c860e8-09cd-2778-edef-754deda31728: entering candidate state: node="Node at f9c860e8-09cd-2778-edef-754deda31728 [Candidate]" term=2
2025-01-25T14:28:14.610Z [INFO]  server-f9c860e8-09cd-2778-edef-754deda31728: election won: term=2 tally=2
2025-01-25T14:28:14.610Z [INFO]  server-f9c860e8-09cd-2778-edef-754deda31728: entering leader state: leader="Node at f9c860e8-09cd-2778-edef-754deda31728 [Leader]"
2025-01-25T14:28:14.610Z [INFO]  server-f9c860e8-09cd-2778-edef-754deda31728: added peer, starting replication: peer=server-3caa1664-b24f-f6c0-aa9c-895ee1f3c378
2025-01-25T14:28:14.610Z [INFO]  server-f9c860e8-09cd-2778-edef-754deda31728: added peer, starting replication: peer=server-16ab997b-d653-d66c-71ce-cacf38299658
2025-01-25T14:28:14.611Z [WARN]  server-16ab997b-d653-d66c-71ce-cacf38299658: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-01-25T14:28:14.611Z [WARN]  server-f9c860e8-09cd-2778-edef-754deda31728: appendEntries rejected, sending older logs: peer="{Voter server-16ab997b-d653-d66c-71ce-cacf38299658 16ab997b-d653-d66c-71ce-cacf38299658}" next=1
2025-01-25T14:28:14.611Z [INFO]  server-f9c860e8-09cd-2778-edef-754deda31728: pipelining replication: peer="{Voter server-16ab997b-d653-d66c-71ce-cacf38299658 16ab997b-d653-d66c-71ce-cacf38299658}"
2025-01-25T14:28:14.611Z [WARN]  server-3caa1664-b24f-f6c0-aa9c-895ee1f3c378: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-01-25T14:28:14.611Z [WARN]  server-f9c860e8-09cd-2778-edef-754deda31728: appendEntries rejected, sending older logs: peer="{Voter server-3caa1664-b24f-f6c0-aa9c-895ee1f3c378 3caa1664-b24f-f6c0-aa9c-895ee1f3c378}" next=1
2025-01-25T14:28:14.611Z [INFO]  server-f9c860e8-09cd-2778-edef-754deda31728: pipelining replication: peer="{Voter server-3caa1664-b24f-f6c0-aa9c-895ee1f3c378 3caa1664-b24f-f6c0-aa9c-895ee1f3c378}"
2025-01-25T14:28:14.717Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:14.611404565 +0000 UTC m=+4.465729547 (2 nodes), 61.441109ms from start of poll, 62.200002ms from cluster start. Timeout at 2025-01-25 14:28:14.716414405 +0000 UTC m=+4.570739347, 105.0098ms after stability
2025-01-25T14:28:14.717Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:14.823Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:14.717741138 +0000 UTC m=+4.572066080 (1 nodes), 1.84µs from start of poll, 168.536535ms from cluster start. Timeout at 2025-01-25 14:28:14.822773617 +0000 UTC m=+4.677098559, 105.032479ms after stability
2025-01-25T14:28:14.835Z [INFO]  server-f9c860e8-09cd-2778-edef-754deda31728: aborting pipeline replication: peer="{Voter server-16ab997b-d653-d66c-71ce-cacf38299658 16ab997b-d653-d66c-71ce-cacf38299658}"
2025-01-25T14:28:14.835Z [INFO]  server-f9c860e8-09cd-2778-edef-754deda31728: aborting pipeline replication: peer="{Voter server-3caa1664-b24f-f6c0-aa9c-895ee1f3c378 3caa1664-b24f-f6c0-aa9c-895ee1f3c378}"
--- PASS: TestRaft_LiveBootstrap (0.29s)
=== RUN   TestRaft_LiveBootstrap_From_NonVoter
2025-01-25T14:28:14.837Z [INFO]  server-3b47beb7-0922-d507-d8f3-c0117edab7ef: initial configuration: index=0 servers=[]
2025-01-25T14:28:14.837Z [INFO]  server-4e3ad652-d8cf-a433-489f-c332805ebb18: initial configuration: index=0 servers=[]
2025-01-25T14:28:14.837Z [INFO]  server-3b47beb7-0922-d507-d8f3-c0117edab7ef: entering follower state: follower="Node at 3b47beb7-0922-d507-d8f3-c0117edab7ef [Follower]" leader-address= leader-id=
2025-01-25T14:28:14.837Z [INFO]  server-4e3ad652-d8cf-a433-489f-c332805ebb18: entering follower state: follower="Node at 4e3ad652-d8cf-a433-489f-c332805ebb18 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s)
=== RUN   TestRaft_RecoverCluster_NoState
2025-01-25T14:28:14.839Z [INFO]  server-12290731-d721-ab25-5ce7-121a65f5de79: initial configuration: index=0 servers=[]
--- PASS: TestRaft_RecoverCluster_NoState (0.00s)
=== RUN   TestRaft_RecoverCluster
=== RUN   TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs
2025-01-25T14:28:14.841Z [INFO]  server-72e4e832-f6c8-eefc-918e-38def62c0f95: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72e4e832-f6c8-eefc-918e-38def62c0f95 Address:72e4e832-f6c8-eefc-918e-38def62c0f95} {Suffrage:Voter ID:server-36934e59-7602-1f6e-1619-13735feb8156 Address:36934e59-7602-1f6e-1619-13735feb8156} {Suffrage:Voter ID:server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924 Address:9b6bcc83-9a5a-f8f8-1dcd-2472348e6924}]"
2025-01-25T14:28:14.841Z [INFO]  server-36934e59-7602-1f6e-1619-13735feb8156: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72e4e832-f6c8-eefc-918e-38def62c0f95 Address:72e4e832-f6c8-eefc-918e-38def62c0f95} {Suffrage:Voter ID:server-36934e59-7602-1f6e-1619-13735feb8156 Address:36934e59-7602-1f6e-1619-13735feb8156} {Suffrage:Voter ID:server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924 Address:9b6bcc83-9a5a-f8f8-1dcd-2472348e6924}]"
2025-01-25T14:28:14.841Z [INFO]  server-72e4e832-f6c8-eefc-918e-38def62c0f95: entering follower state: follower="Node at 72e4e832-f6c8-eefc-918e-38def62c0f95 [Follower]" leader-address= leader-id=
2025-01-25T14:28:14.841Z [INFO]  server-36934e59-7602-1f6e-1619-13735feb8156: entering follower state: follower="Node at 36934e59-7602-1f6e-1619-13735feb8156 [Follower]" leader-address= leader-id=
2025-01-25T14:28:14.841Z [INFO]  server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72e4e832-f6c8-eefc-918e-38def62c0f95 Address:72e4e832-f6c8-eefc-918e-38def62c0f95} {Suffrage:Voter ID:server-36934e59-7602-1f6e-1619-13735feb8156 Address:36934e59-7602-1f6e-1619-13735feb8156} {Suffrage:Voter ID:server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924 Address:9b6bcc83-9a5a-f8f8-1dcd-2472348e6924}]"
2025-01-25T14:28:14.841Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:14.841Z [INFO]  server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924: entering follower state: follower="Node at 9b6bcc83-9a5a-f8f8-1dcd-2472348e6924 [Follower]" leader-address= leader-id=
2025-01-25T14:28:14.899Z [WARN]  server-72e4e832-f6c8-eefc-918e-38def62c0f95: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:14.899Z [INFO]  server-72e4e832-f6c8-eefc-918e-38def62c0f95: entering candidate state: node="Node at 72e4e832-f6c8-eefc-918e-38def62c0f95 [Candidate]" term=2
2025-01-25T14:28:14.900Z [INFO]  server-72e4e832-f6c8-eefc-918e-38def62c0f95: election won: term=2 tally=2
2025-01-25T14:28:14.900Z [INFO]  server-72e4e832-f6c8-eefc-918e-38def62c0f95: entering leader state: leader="Node at 72e4e832-f6c8-eefc-918e-38def62c0f95 [Leader]"
2025-01-25T14:28:14.900Z [INFO]  server-72e4e832-f6c8-eefc-918e-38def62c0f95: added peer, starting replication: peer=server-36934e59-7602-1f6e-1619-13735feb8156
2025-01-25T14:28:14.900Z [INFO]  server-72e4e832-f6c8-eefc-918e-38def62c0f95: added peer, starting replication: peer=server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924
2025-01-25T14:28:14.900Z [INFO]  server-72e4e832-f6c8-eefc-918e-38def62c0f95: pipelining replication: peer="{Voter server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924 9b6bcc83-9a5a-f8f8-1dcd-2472348e6924}"
2025-01-25T14:28:14.900Z [INFO]  server-72e4e832-f6c8-eefc-918e-38def62c0f95: pipelining replication: peer="{Voter server-36934e59-7602-1f6e-1619-13735feb8156 36934e59-7602-1f6e-1619-13735feb8156}"
2025-01-25T14:28:15.006Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:14.900309729 +0000 UTC m=+4.754634711 (1 nodes), 58.626531ms from start of poll, 59.495939ms from cluster start. Timeout at 2025-01-25 14:28:15.005314449 +0000 UTC m=+4.859639431, 105.00472ms after stability
2025-01-25T14:28:15.006Z [INFO]  server-72e4e832-f6c8-eefc-918e-38def62c0f95: aborting pipeline replication: peer="{Voter server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924 9b6bcc83-9a5a-f8f8-1dcd-2472348e6924}"
2025-01-25T14:28:15.006Z [INFO]  server-72e4e832-f6c8-eefc-918e-38def62c0f95: aborting pipeline replication: peer="{Voter server-36934e59-7602-1f6e-1619-13735feb8156 36934e59-7602-1f6e-1619-13735feb8156}"
2025-01-25T14:28:15.006Z [INFO]  creating new snapshot: path=/tmp/raft911086418/snapshots/2-2-1737815295006.tmp
2025-01-25T14:28:15.008Z [INFO]  server-72e4e832-f6c8-eefc-918e-38def62c0f95: starting restore from snapshot: id=2-2-1737815295006 last-index=2 last-term=2 size-in-bytes=1
2025-01-25T14:28:15.008Z [INFO]  server-72e4e832-f6c8-eefc-918e-38def62c0f95: snapshot restore progress: id=2-2-1737815295006 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-01-25T14:28:15.008Z [INFO]  server-72e4e832-f6c8-eefc-918e-38def62c0f95: restored from snapshot: id=2-2-1737815295006 last-index=2 last-term=2 size-in-bytes=1
2025-01-25T14:28:15.008Z [INFO]  server-72e4e832-f6c8-eefc-918e-38def62c0f95: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72e4e832-f6c8-eefc-918e-38def62c0f95 Address:72e4e832-f6c8-eefc-918e-38def62c0f95} {Suffrage:Voter ID:server-36934e59-7602-1f6e-1619-13735feb8156 Address:36934e59-7602-1f6e-1619-13735feb8156} {Suffrage:Voter ID:server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924 Address:9b6bcc83-9a5a-f8f8-1dcd-2472348e6924}]"
2025-01-25T14:28:15.009Z [INFO]  creating new snapshot: path=/tmp/raft1592049412/snapshots/2-2-1737815295009.tmp
2025-01-25T14:28:15.009Z [INFO]  server-72e4e832-f6c8-eefc-918e-38def62c0f95: entering follower state: follower="Node at 72e4e832-f6c8-eefc-918e-38def62c0f95 [Follower]" leader-address= leader-id=
2025-01-25T14:28:15.010Z [INFO]  server-36934e59-7602-1f6e-1619-13735feb8156: starting restore from snapshot: id=2-2-1737815295009 last-index=2 last-term=2 size-in-bytes=1
2025-01-25T14:28:15.010Z [INFO]  server-36934e59-7602-1f6e-1619-13735feb8156: snapshot restore progress: id=2-2-1737815295009 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-01-25T14:28:15.011Z [INFO]  server-36934e59-7602-1f6e-1619-13735feb8156: restored from snapshot: id=2-2-1737815295009 last-index=2 last-term=2 size-in-bytes=1
2025-01-25T14:28:15.011Z [INFO]  server-36934e59-7602-1f6e-1619-13735feb8156: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72e4e832-f6c8-eefc-918e-38def62c0f95 Address:72e4e832-f6c8-eefc-918e-38def62c0f95} {Suffrage:Voter ID:server-36934e59-7602-1f6e-1619-13735feb8156 Address:36934e59-7602-1f6e-1619-13735feb8156} {Suffrage:Voter ID:server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924 Address:9b6bcc83-9a5a-f8f8-1dcd-2472348e6924}]"
2025-01-25T14:28:15.011Z [INFO]  creating new snapshot: path=/tmp/raft539846833/snapshots/2-2-1737815295011.tmp
2025-01-25T14:28:15.011Z [INFO]  server-36934e59-7602-1f6e-1619-13735feb8156: entering follower state: follower="Node at 36934e59-7602-1f6e-1619-13735feb8156 [Follower]" leader-address= leader-id=
2025-01-25T14:28:15.012Z [INFO]  server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924: starting restore from snapshot: id=2-2-1737815295011 last-index=2 last-term=2 size-in-bytes=1
2025-01-25T14:28:15.013Z [INFO]  server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924: snapshot restore progress: id=2-2-1737815295011 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-01-25T14:28:15.013Z [INFO]  server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924: restored from snapshot: id=2-2-1737815295011 last-index=2 last-term=2 size-in-bytes=1
2025-01-25T14:28:15.013Z [INFO]  server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72e4e832-f6c8-eefc-918e-38def62c0f95 Address:72e4e832-f6c8-eefc-918e-38def62c0f95} {Suffrage:Voter ID:server-36934e59-7602-1f6e-1619-13735feb8156 Address:36934e59-7602-1f6e-1619-13735feb8156} {Suffrage:Voter ID:server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924 Address:9b6bcc83-9a5a-f8f8-1dcd-2472348e6924}]"
2025-01-25T14:28:15.013Z [INFO]  server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924: entering follower state: follower="Node at 9b6bcc83-9a5a-f8f8-1dcd-2472348e6924 [Follower]" leader-address= leader-id=
2025-01-25T14:28:15.072Z [WARN]  server-36934e59-7602-1f6e-1619-13735feb8156: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:15.072Z [INFO]  server-36934e59-7602-1f6e-1619-13735feb8156: entering candidate state: node="Node at 36934e59-7602-1f6e-1619-13735feb8156 [Candidate]" term=3
2025-01-25T14:28:15.072Z [INFO]  server-36934e59-7602-1f6e-1619-13735feb8156: election won: term=3 tally=2
2025-01-25T14:28:15.072Z [INFO]  server-36934e59-7602-1f6e-1619-13735feb8156: entering leader state: leader="Node at 36934e59-7602-1f6e-1619-13735feb8156 [Leader]"
2025-01-25T14:28:15.072Z [INFO]  server-36934e59-7602-1f6e-1619-13735feb8156: added peer, starting replication: peer=server-72e4e832-f6c8-eefc-918e-38def62c0f95
2025-01-25T14:28:15.072Z [INFO]  server-36934e59-7602-1f6e-1619-13735feb8156: added peer, starting replication: peer=server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924
2025-01-25T14:28:15.072Z [INFO]  server-36934e59-7602-1f6e-1619-13735feb8156: pipelining replication: peer="{Voter server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924 9b6bcc83-9a5a-f8f8-1dcd-2472348e6924}"
2025-01-25T14:28:15.073Z [INFO]  server-36934e59-7602-1f6e-1619-13735feb8156: pipelining replication: peer="{Voter server-72e4e832-f6c8-eefc-918e-38def62c0f95 72e4e832-f6c8-eefc-918e-38def62c0f95}"
2025-01-25T14:28:15.328Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:15.433Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:15.328399604 +0000 UTC m=+5.182724586 (1 nodes), 48.318µs from start of poll, 487.585814ms from cluster start. Timeout at 2025-01-25 14:28:15.433401204 +0000 UTC m=+5.287726186, 105.0016ms after stability
2025-01-25T14:28:15.433Z [INFO]  server-36934e59-7602-1f6e-1619-13735feb8156: aborting pipeline replication: peer="{Voter server-9b6bcc83-9a5a-f8f8-1dcd-2472348e6924 9b6bcc83-9a5a-f8f8-1dcd-2472348e6924}"
2025-01-25T14:28:15.433Z [INFO]  server-36934e59-7602-1f6e-1619-13735feb8156: aborting pipeline replication: peer="{Voter server-72e4e832-f6c8-eefc-918e-38def62c0f95 72e4e832-f6c8-eefc-918e-38def62c0f95}"
=== RUN   TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs
2025-01-25T14:28:15.437Z [INFO]  server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3 Address:117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3} {Suffrage:Voter ID:server-1b894bae-bd68-2120-0a73-49314a23d461 Address:1b894bae-bd68-2120-0a73-49314a23d461} {Suffrage:Voter ID:server-17c065a1-4cd4-c1ee-1880-5d926ce79e57 Address:17c065a1-4cd4-c1ee-1880-5d926ce79e57}]"
2025-01-25T14:28:15.438Z [INFO]  server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3: entering follower state: follower="Node at 117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3 [Follower]" leader-address= leader-id=
2025-01-25T14:28:15.438Z [INFO]  server-1b894bae-bd68-2120-0a73-49314a23d461: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3 Address:117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3} {Suffrage:Voter ID:server-1b894bae-bd68-2120-0a73-49314a23d461 Address:1b894bae-bd68-2120-0a73-49314a23d461} {Suffrage:Voter ID:server-17c065a1-4cd4-c1ee-1880-5d926ce79e57 Address:17c065a1-4cd4-c1ee-1880-5d926ce79e57}]"
2025-01-25T14:28:15.438Z [INFO]  server-1b894bae-bd68-2120-0a73-49314a23d461: entering follower state: follower="Node at 1b894bae-bd68-2120-0a73-49314a23d461 [Follower]" leader-address= leader-id=
2025-01-25T14:28:15.438Z [INFO]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3 Address:117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3} {Suffrage:Voter ID:server-1b894bae-bd68-2120-0a73-49314a23d461 Address:1b894bae-bd68-2120-0a73-49314a23d461} {Suffrage:Voter ID:server-17c065a1-4cd4-c1ee-1880-5d926ce79e57 Address:17c065a1-4cd4-c1ee-1880-5d926ce79e57}]"
2025-01-25T14:28:15.438Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:15.438Z [INFO]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: entering follower state: follower="Node at 17c065a1-4cd4-c1ee-1880-5d926ce79e57 [Follower]" leader-address= leader-id=
2025-01-25T14:28:15.503Z [WARN]  server-1b894bae-bd68-2120-0a73-49314a23d461: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:15.503Z [INFO]  server-1b894bae-bd68-2120-0a73-49314a23d461: entering candidate state: node="Node at 1b894bae-bd68-2120-0a73-49314a23d461 [Candidate]" term=2
2025-01-25T14:28:15.503Z [INFO]  server-1b894bae-bd68-2120-0a73-49314a23d461: election won: term=2 tally=2
2025-01-25T14:28:15.503Z [INFO]  server-1b894bae-bd68-2120-0a73-49314a23d461: entering leader state: leader="Node at 1b894bae-bd68-2120-0a73-49314a23d461 [Leader]"
2025-01-25T14:28:15.504Z [INFO]  server-1b894bae-bd68-2120-0a73-49314a23d461: added peer, starting replication: peer=server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3
2025-01-25T14:28:15.504Z [INFO]  server-1b894bae-bd68-2120-0a73-49314a23d461: added peer, starting replication: peer=server-17c065a1-4cd4-c1ee-1880-5d926ce79e57
2025-01-25T14:28:15.504Z [WARN]  server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:15.504Z [INFO]  server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3: entering candidate state: node="Node at 117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3 [Candidate]" term=2
2025-01-25T14:28:15.504Z [INFO]  server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3: duplicate requestVote for same term: term=2
2025-01-25T14:28:15.504Z [WARN]  server-1b894bae-bd68-2120-0a73-49314a23d461: rejecting vote request since we have a leader: from=117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3 leader=1b894bae-bd68-2120-0a73-49314a23d461 leader-id=server-1b894bae-bd68-2120-0a73-49314a23d461
2025-01-25T14:28:15.504Z [INFO]  server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3: entering follower state: follower="Node at 117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3 [Follower]" leader-address=1b894bae-bd68-2120-0a73-49314a23d461 leader-id=server-1b894bae-bd68-2120-0a73-49314a23d461
2025-01-25T14:28:15.504Z [INFO]  server-1b894bae-bd68-2120-0a73-49314a23d461: pipelining replication: peer="{Voter server-17c065a1-4cd4-c1ee-1880-5d926ce79e57 17c065a1-4cd4-c1ee-1880-5d926ce79e57}"
2025-01-25T14:28:15.504Z [WARN]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: rejecting vote request since we have a leader: from=117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3 leader=1b894bae-bd68-2120-0a73-49314a23d461 leader-id=server-1b894bae-bd68-2120-0a73-49314a23d461
2025-01-25T14:28:15.504Z [INFO]  server-1b894bae-bd68-2120-0a73-49314a23d461: pipelining replication: peer="{Voter server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3 117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3}"
2025-01-25T14:28:15.610Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:15.504386912 +0000 UTC m=+5.358711894 (1 nodes), 66.018625ms from start of poll, 66.662242ms from cluster start. Timeout at 2025-01-25 14:28:15.609402152 +0000 UTC m=+5.463727134, 105.01524ms after stability
2025-01-25T14:28:15.611Z [INFO]  server-1b894bae-bd68-2120-0a73-49314a23d461: aborting pipeline replication: peer="{Voter server-17c065a1-4cd4-c1ee-1880-5d926ce79e57 17c065a1-4cd4-c1ee-1880-5d926ce79e57}"
2025-01-25T14:28:15.611Z [INFO]  server-1b894bae-bd68-2120-0a73-49314a23d461: aborting pipeline replication: peer="{Voter server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3 117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3}"
2025-01-25T14:28:15.611Z [INFO]  creating new snapshot: path=/tmp/raft1685918046/snapshots/2-6-1737815295611.tmp
2025-01-25T14:28:15.613Z [INFO]  server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3: starting restore from snapshot: id=2-6-1737815295611 last-index=6 last-term=2 size-in-bytes=25
2025-01-25T14:28:15.613Z [INFO]  server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3: snapshot restore progress: id=2-6-1737815295611 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-01-25T14:28:15.613Z [INFO]  server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3: restored from snapshot: id=2-6-1737815295611 last-index=6 last-term=2 size-in-bytes=25
2025-01-25T14:28:15.613Z [INFO]  server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3 Address:117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3} {Suffrage:Voter ID:server-1b894bae-bd68-2120-0a73-49314a23d461 Address:1b894bae-bd68-2120-0a73-49314a23d461} {Suffrage:Voter ID:server-17c065a1-4cd4-c1ee-1880-5d926ce79e57 Address:17c065a1-4cd4-c1ee-1880-5d926ce79e57}]"
2025-01-25T14:28:15.614Z [INFO]  server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3: entering follower state: follower="Node at 117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3 [Follower]" leader-address= leader-id=
2025-01-25T14:28:15.614Z [INFO]  creating new snapshot: path=/tmp/raft1303537295/snapshots/2-6-1737815295614.tmp
2025-01-25T14:28:15.615Z [INFO]  server-1b894bae-bd68-2120-0a73-49314a23d461: starting restore from snapshot: id=2-6-1737815295614 last-index=6 last-term=2 size-in-bytes=25
2025-01-25T14:28:15.615Z [INFO]  server-1b894bae-bd68-2120-0a73-49314a23d461: snapshot restore progress: id=2-6-1737815295614 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-01-25T14:28:15.615Z [INFO]  server-1b894bae-bd68-2120-0a73-49314a23d461: restored from snapshot: id=2-6-1737815295614 last-index=6 last-term=2 size-in-bytes=25
2025-01-25T14:28:15.615Z [INFO]  server-1b894bae-bd68-2120-0a73-49314a23d461: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3 Address:117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3} {Suffrage:Voter ID:server-1b894bae-bd68-2120-0a73-49314a23d461 Address:1b894bae-bd68-2120-0a73-49314a23d461} {Suffrage:Voter ID:server-17c065a1-4cd4-c1ee-1880-5d926ce79e57 Address:17c065a1-4cd4-c1ee-1880-5d926ce79e57}]"
2025-01-25T14:28:15.615Z [INFO]  server-1b894bae-bd68-2120-0a73-49314a23d461: entering follower state: follower="Node at 1b894bae-bd68-2120-0a73-49314a23d461 [Follower]" leader-address= leader-id=
2025-01-25T14:28:15.615Z [INFO]  creating new snapshot: path=/tmp/raft2329099858/snapshots/2-6-1737815295615.tmp
2025-01-25T14:28:15.616Z [INFO]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: starting restore from snapshot: id=2-6-1737815295615 last-index=6 last-term=2 size-in-bytes=25
2025-01-25T14:28:15.617Z [INFO]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: snapshot restore progress: id=2-6-1737815295615 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-01-25T14:28:15.617Z [INFO]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: restored from snapshot: id=2-6-1737815295615 last-index=6 last-term=2 size-in-bytes=25
2025-01-25T14:28:15.617Z [INFO]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3 Address:117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3} {Suffrage:Voter ID:server-1b894bae-bd68-2120-0a73-49314a23d461 Address:1b894bae-bd68-2120-0a73-49314a23d461} {Suffrage:Voter ID:server-17c065a1-4cd4-c1ee-1880-5d926ce79e57 Address:17c065a1-4cd4-c1ee-1880-5d926ce79e57}]"
2025-01-25T14:28:15.617Z [INFO]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: entering follower state: follower="Node at 17c065a1-4cd4-c1ee-1880-5d926ce79e57 [Follower]" leader-address= leader-id=
2025-01-25T14:28:15.699Z [WARN]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:15.699Z [INFO]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: entering candidate state: node="Node at 17c065a1-4cd4-c1ee-1880-5d926ce79e57 [Candidate]" term=3
2025-01-25T14:28:15.699Z [INFO]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: election won: term=3 tally=2
2025-01-25T14:28:15.699Z [INFO]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: entering leader state: leader="Node at 17c065a1-4cd4-c1ee-1880-5d926ce79e57 [Leader]"
2025-01-25T14:28:15.699Z [INFO]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: added peer, starting replication: peer=server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3
2025-01-25T14:28:15.699Z [INFO]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: added peer, starting replication: peer=server-1b894bae-bd68-2120-0a73-49314a23d461
2025-01-25T14:28:15.699Z [INFO]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: pipelining replication: peer="{Voter server-1b894bae-bd68-2120-0a73-49314a23d461 1b894bae-bd68-2120-0a73-49314a23d461}"
2025-01-25T14:28:15.699Z [INFO]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: pipelining replication: peer="{Voter server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3 117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3}"
2025-01-25T14:28:15.932Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:16.038Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:15.933077446 +0000 UTC m=+5.787402388 (1 nodes), 69.758µs from start of poll, 495.352736ms from cluster start. Timeout at 2025-01-25 14:28:16.038105525 +0000 UTC m=+5.892430507, 105.028119ms after stability
2025-01-25T14:28:16.038Z [INFO]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: aborting pipeline replication: peer="{Voter server-1b894bae-bd68-2120-0a73-49314a23d461 1b894bae-bd68-2120-0a73-49314a23d461}"
2025-01-25T14:28:16.038Z [INFO]  server-17c065a1-4cd4-c1ee-1880-5d926ce79e57: aborting pipeline replication: peer="{Voter server-117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3 117a8b7d-8b5c-1c6d-285f-1f6f74a0bbb3}"
=== RUN   TestRaft_RecoverCluster/snapshot,_with_trailing_logs
2025-01-25T14:28:16.041Z [INFO]  server-da29ad0a-06b9-b157-b461-4e63f2707990: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da29ad0a-06b9-b157-b461-4e63f2707990 Address:da29ad0a-06b9-b157-b461-4e63f2707990} {Suffrage:Voter ID:server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30 Address:24cd48bf-6e20-3d67-dc4a-e767e1e7dd30} {Suffrage:Voter ID:server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4 Address:1480d5ab-4df9-17e1-303b-8c4a9ce584a4}]"
2025-01-25T14:28:16.042Z [INFO]  server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da29ad0a-06b9-b157-b461-4e63f2707990 Address:da29ad0a-06b9-b157-b461-4e63f2707990} {Suffrage:Voter ID:server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30 Address:24cd48bf-6e20-3d67-dc4a-e767e1e7dd30} {Suffrage:Voter ID:server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4 Address:1480d5ab-4df9-17e1-303b-8c4a9ce584a4}]"
2025-01-25T14:28:16.042Z [INFO]  server-da29ad0a-06b9-b157-b461-4e63f2707990: entering follower state: follower="Node at da29ad0a-06b9-b157-b461-4e63f2707990 [Follower]" leader-address= leader-id=
2025-01-25T14:28:16.042Z [INFO]  server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30: entering follower state: follower="Node at 24cd48bf-6e20-3d67-dc4a-e767e1e7dd30 [Follower]" leader-address= leader-id=
2025-01-25T14:28:16.042Z [INFO]  server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da29ad0a-06b9-b157-b461-4e63f2707990 Address:da29ad0a-06b9-b157-b461-4e63f2707990} {Suffrage:Voter ID:server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30 Address:24cd48bf-6e20-3d67-dc4a-e767e1e7dd30} {Suffrage:Voter ID:server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4 Address:1480d5ab-4df9-17e1-303b-8c4a9ce584a4}]"
2025-01-25T14:28:16.042Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:16.042Z [INFO]  server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4: entering follower state: follower="Node at 1480d5ab-4df9-17e1-303b-8c4a9ce584a4 [Follower]" leader-address= leader-id=
2025-01-25T14:28:16.107Z [WARN]  server-da29ad0a-06b9-b157-b461-4e63f2707990: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:16.107Z [INFO]  server-da29ad0a-06b9-b157-b461-4e63f2707990: entering candidate state: node="Node at da29ad0a-06b9-b157-b461-4e63f2707990 [Candidate]" term=2
2025-01-25T14:28:16.107Z [INFO]  server-da29ad0a-06b9-b157-b461-4e63f2707990: election won: term=2 tally=2
2025-01-25T14:28:16.107Z [INFO]  server-da29ad0a-06b9-b157-b461-4e63f2707990: entering leader state: leader="Node at da29ad0a-06b9-b157-b461-4e63f2707990 [Leader]"
2025-01-25T14:28:16.107Z [INFO]  server-da29ad0a-06b9-b157-b461-4e63f2707990: added peer, starting replication: peer=server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30
2025-01-25T14:28:16.107Z [INFO]  server-da29ad0a-06b9-b157-b461-4e63f2707990: added peer, starting replication: peer=server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4
2025-01-25T14:28:16.107Z [INFO]  server-da29ad0a-06b9-b157-b461-4e63f2707990: pipelining replication: peer="{Voter server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30 24cd48bf-6e20-3d67-dc4a-e767e1e7dd30}"
2025-01-25T14:28:16.107Z [INFO]  server-da29ad0a-06b9-b157-b461-4e63f2707990: pipelining replication: peer="{Voter server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4 1480d5ab-4df9-17e1-303b-8c4a9ce584a4}"
2025-01-25T14:28:16.213Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:16.107409573 +0000 UTC m=+5.961734515 (1 nodes), 65.141536ms from start of poll, 65.783073ms from cluster start. Timeout at 2025-01-25 14:28:16.212411652 +0000 UTC m=+6.066736594, 105.002079ms after stability
2025-01-25T14:28:16.219Z [INFO]  server-da29ad0a-06b9-b157-b461-4e63f2707990: aborting pipeline replication: peer="{Voter server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30 24cd48bf-6e20-3d67-dc4a-e767e1e7dd30}"
2025-01-25T14:28:16.219Z [INFO]  server-da29ad0a-06b9-b157-b461-4e63f2707990: aborting pipeline replication: peer="{Voter server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4 1480d5ab-4df9-17e1-303b-8c4a9ce584a4}"
2025-01-25T14:28:16.219Z [INFO]  creating new snapshot: path=/tmp/raft807531108/snapshots/2-27-1737815296219.tmp
2025-01-25T14:28:16.221Z [INFO]  server-da29ad0a-06b9-b157-b461-4e63f2707990: starting restore from snapshot: id=2-27-1737815296219 last-index=27 last-term=2 size-in-bytes=168
2025-01-25T14:28:16.221Z [INFO]  server-da29ad0a-06b9-b157-b461-4e63f2707990: snapshot restore progress: id=2-27-1737815296219 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-01-25T14:28:16.221Z [INFO]  server-da29ad0a-06b9-b157-b461-4e63f2707990: restored from snapshot: id=2-27-1737815296219 last-index=27 last-term=2 size-in-bytes=168
2025-01-25T14:28:16.221Z [INFO]  server-da29ad0a-06b9-b157-b461-4e63f2707990: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da29ad0a-06b9-b157-b461-4e63f2707990 Address:da29ad0a-06b9-b157-b461-4e63f2707990} {Suffrage:Voter ID:server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30 Address:24cd48bf-6e20-3d67-dc4a-e767e1e7dd30} {Suffrage:Voter ID:server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4 Address:1480d5ab-4df9-17e1-303b-8c4a9ce584a4}]"
2025-01-25T14:28:16.221Z [INFO]  creating new snapshot: path=/tmp/raft3444269178/snapshots/2-27-1737815296221.tmp
2025-01-25T14:28:16.221Z [INFO]  server-da29ad0a-06b9-b157-b461-4e63f2707990: entering follower state: follower="Node at da29ad0a-06b9-b157-b461-4e63f2707990 [Follower]" leader-address= leader-id=
2025-01-25T14:28:16.222Z [INFO]  server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30: starting restore from snapshot: id=2-27-1737815296221 last-index=27 last-term=2 size-in-bytes=168
2025-01-25T14:28:16.223Z [INFO]  server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30: snapshot restore progress: id=2-27-1737815296221 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-01-25T14:28:16.223Z [INFO]  server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30: restored from snapshot: id=2-27-1737815296221 last-index=27 last-term=2 size-in-bytes=168
2025-01-25T14:28:16.223Z [INFO]  server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da29ad0a-06b9-b157-b461-4e63f2707990 Address:da29ad0a-06b9-b157-b461-4e63f2707990} {Suffrage:Voter ID:server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30 Address:24cd48bf-6e20-3d67-dc4a-e767e1e7dd30} {Suffrage:Voter ID:server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4 Address:1480d5ab-4df9-17e1-303b-8c4a9ce584a4}]"
2025-01-25T14:28:16.223Z [INFO]  server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30: entering follower state: follower="Node at 24cd48bf-6e20-3d67-dc4a-e767e1e7dd30 [Follower]" leader-address= leader-id=
2025-01-25T14:28:16.223Z [INFO]  creating new snapshot: path=/tmp/raft4161443190/snapshots/2-27-1737815296223.tmp
2025-01-25T14:28:16.224Z [INFO]  server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4: starting restore from snapshot: id=2-27-1737815296223 last-index=27 last-term=2 size-in-bytes=168
2025-01-25T14:28:16.224Z [INFO]  server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4: snapshot restore progress: id=2-27-1737815296223 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-01-25T14:28:16.224Z [INFO]  server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4: restored from snapshot: id=2-27-1737815296223 last-index=27 last-term=2 size-in-bytes=168
2025-01-25T14:28:16.224Z [INFO]  server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da29ad0a-06b9-b157-b461-4e63f2707990 Address:da29ad0a-06b9-b157-b461-4e63f2707990} {Suffrage:Voter ID:server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30 Address:24cd48bf-6e20-3d67-dc4a-e767e1e7dd30} {Suffrage:Voter ID:server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4 Address:1480d5ab-4df9-17e1-303b-8c4a9ce584a4}]"
2025-01-25T14:28:16.224Z [INFO]  server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4: entering follower state: follower="Node at 1480d5ab-4df9-17e1-303b-8c4a9ce584a4 [Follower]" leader-address= leader-id=
2025-01-25T14:28:16.295Z [WARN]  server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:16.295Z [INFO]  server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30: entering candidate state: node="Node at 24cd48bf-6e20-3d67-dc4a-e767e1e7dd30 [Candidate]" term=3
2025-01-25T14:28:16.295Z [INFO]  server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30: election won: term=3 tally=2
2025-01-25T14:28:16.295Z [INFO]  server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30: entering leader state: leader="Node at 24cd48bf-6e20-3d67-dc4a-e767e1e7dd30 [Leader]"
2025-01-25T14:28:16.295Z [INFO]  server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30: added peer, starting replication: peer=server-da29ad0a-06b9-b157-b461-4e63f2707990
2025-01-25T14:28:16.295Z [INFO]  server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30: added peer, starting replication: peer=server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4
2025-01-25T14:28:16.295Z [INFO]  server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30: pipelining replication: peer="{Voter server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4 1480d5ab-4df9-17e1-303b-8c4a9ce584a4}"
2025-01-25T14:28:16.295Z [INFO]  server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30: pipelining replication: peer="{Voter server-da29ad0a-06b9-b157-b461-4e63f2707990 da29ad0a-06b9-b157-b461-4e63f2707990}"
2025-01-25T14:28:16.540Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:16.646Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:16.54044623 +0000 UTC m=+6.394771212 (1 nodes), 59.918µs from start of poll, 498.81977ms from cluster start. Timeout at 2025-01-25 14:28:16.645489149 +0000 UTC m=+6.499814091, 105.042879ms after stability
2025-01-25T14:28:16.646Z [INFO]  server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30: aborting pipeline replication: peer="{Voter server-1480d5ab-4df9-17e1-303b-8c4a9ce584a4 1480d5ab-4df9-17e1-303b-8c4a9ce584a4}"
2025-01-25T14:28:16.646Z [INFO]  server-24cd48bf-6e20-3d67-dc4a-e767e1e7dd30: aborting pipeline replication: peer="{Voter server-da29ad0a-06b9-b157-b461-4e63f2707990 da29ad0a-06b9-b157-b461-4e63f2707990}"
--- PASS: TestRaft_RecoverCluster (1.81s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.60s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.60s)
    --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.61s)
=== RUN   TestRaft_HasExistingState
    raft_test.go:258: Skipping unreliable test
--- SKIP: TestRaft_HasExistingState (0.00s)
=== RUN   TestRaft_SingleNode
2025-01-25T14:28:16.649Z [INFO]  server-a10f318f-3368-47b9-ecfd-e399dbb3480f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a10f318f-3368-47b9-ecfd-e399dbb3480f Address:a10f318f-3368-47b9-ecfd-e399dbb3480f}]"
2025-01-25T14:28:16.649Z [INFO]  server-a10f318f-3368-47b9-ecfd-e399dbb3480f: entering follower state: follower="Node at a10f318f-3368-47b9-ecfd-e399dbb3480f [Follower]" leader-address= leader-id=
2025-01-25T14:28:16.737Z [WARN]  server-a10f318f-3368-47b9-ecfd-e399dbb3480f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:16.737Z [INFO]  server-a10f318f-3368-47b9-ecfd-e399dbb3480f: entering candidate state: node="Node at a10f318f-3368-47b9-ecfd-e399dbb3480f [Candidate]" term=2
2025-01-25T14:28:16.737Z [INFO]  server-a10f318f-3368-47b9-ecfd-e399dbb3480f: election won: term=2 tally=1
2025-01-25T14:28:16.737Z [INFO]  server-a10f318f-3368-47b9-ecfd-e399dbb3480f: entering leader state: leader="Node at a10f318f-3368-47b9-ecfd-e399dbb3480f [Leader]"
--- PASS: TestRaft_SingleNode (0.09s)
=== RUN   TestRaft_TripleNode
2025-01-25T14:28:16.740Z [INFO]  server-c57e67b9-345e-2411-7165-ca2cbc39ed65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c57e67b9-345e-2411-7165-ca2cbc39ed65 Address:c57e67b9-345e-2411-7165-ca2cbc39ed65} {Suffrage:Voter ID:server-66814d4c-919b-4586-6941-30a133915fd8 Address:66814d4c-919b-4586-6941-30a133915fd8} {Suffrage:Voter ID:server-54f9a64c-2f91-e6ab-d9a7-d8732d9a7868 Address:54f9a64c-2f91-e6ab-d9a7-d8732d9a7868}]"
2025-01-25T14:28:16.740Z [INFO]  server-66814d4c-919b-4586-6941-30a133915fd8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c57e67b9-345e-2411-7165-ca2cbc39ed65 Address:c57e67b9-345e-2411-7165-ca2cbc39ed65} {Suffrage:Voter ID:server-66814d4c-919b-4586-6941-30a133915fd8 Address:66814d4c-919b-4586-6941-30a133915fd8} {Suffrage:Voter ID:server-54f9a64c-2f91-e6ab-d9a7-d8732d9a7868 Address:54f9a64c-2f91-e6ab-d9a7-d8732d9a7868}]"
2025-01-25T14:28:16.740Z [INFO]  server-c57e67b9-345e-2411-7165-ca2cbc39ed65: entering follower state: follower="Node at c57e67b9-345e-2411-7165-ca2cbc39ed65 [Follower]" leader-address= leader-id=
2025-01-25T14:28:16.740Z [INFO]  server-66814d4c-919b-4586-6941-30a133915fd8: entering follower state: follower="Node at 66814d4c-919b-4586-6941-30a133915fd8 [Follower]" leader-address= leader-id=
2025-01-25T14:28:16.740Z [INFO]  server-54f9a64c-2f91-e6ab-d9a7-d8732d9a7868: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c57e67b9-345e-2411-7165-ca2cbc39ed65 Address:c57e67b9-345e-2411-7165-ca2cbc39ed65} {Suffrage:Voter ID:server-66814d4c-919b-4586-6941-30a133915fd8 Address:66814d4c-919b-4586-6941-30a133915fd8} {Suffrage:Voter ID:server-54f9a64c-2f91-e6ab-d9a7-d8732d9a7868 Address:54f9a64c-2f91-e6ab-d9a7-d8732d9a7868}]"
2025-01-25T14:28:16.740Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:16.740Z [INFO]  server-54f9a64c-2f91-e6ab-d9a7-d8732d9a7868: entering follower state: follower="Node at 54f9a64c-2f91-e6ab-d9a7-d8732d9a7868 [Follower]" leader-address= leader-id=
2025-01-25T14:28:16.807Z [WARN]  server-54f9a64c-2f91-e6ab-d9a7-d8732d9a7868: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:16.807Z [INFO]  server-54f9a64c-2f91-e6ab-d9a7-d8732d9a7868: entering candidate state: node="Node at 54f9a64c-2f91-e6ab-d9a7-d8732d9a7868 [Candidate]" term=2
2025-01-25T14:28:16.807Z [INFO]  server-54f9a64c-2f91-e6ab-d9a7-d8732d9a7868: election won: term=2 tally=2
2025-01-25T14:28:16.807Z [INFO]  server-54f9a64c-2f91-e6ab-d9a7-d8732d9a7868: entering leader state: leader="Node at 54f9a64c-2f91-e6ab-d9a7-d8732d9a7868 [Leader]"
2025-01-25T14:28:16.807Z [INFO]  server-54f9a64c-2f91-e6ab-d9a7-d8732d9a7868: added peer, starting replication: peer=server-c57e67b9-345e-2411-7165-ca2cbc39ed65
2025-01-25T14:28:16.807Z [INFO]  server-54f9a64c-2f91-e6ab-d9a7-d8732d9a7868: added peer, starting replication: peer=server-66814d4c-919b-4586-6941-30a133915fd8
2025-01-25T14:28:16.807Z [INFO]  server-54f9a64c-2f91-e6ab-d9a7-d8732d9a7868: pipelining replication: peer="{Voter server-66814d4c-919b-4586-6941-30a133915fd8 66814d4c-919b-4586-6941-30a133915fd8}"
2025-01-25T14:28:16.808Z [INFO]  server-54f9a64c-2f91-e6ab-d9a7-d8732d9a7868: pipelining replication: peer="{Voter server-c57e67b9-345e-2411-7165-ca2cbc39ed65 c57e67b9-345e-2411-7165-ca2cbc39ed65}"
2025-01-25T14:28:16.913Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:16.808130398 +0000 UTC m=+6.662455340 (2 nodes), 67.592808ms from start of poll, 68.245225ms from cluster start. Timeout at 2025-01-25 14:28:16.913141438 +0000 UTC m=+6.767466420, 105.01108ms after stability
2025-01-25T14:28:16.913Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:17.019Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:16.913979909 +0000 UTC m=+6.768304851 (1 nodes), 2.32µs from start of poll, 174.094736ms from cluster start. Timeout at 2025-01-25 14:28:17.018991349 +0000 UTC m=+6.873316331, 105.01148ms after stability
2025-01-25T14:28:17.030Z [INFO]  server-54f9a64c-2f91-e6ab-d9a7-d8732d9a7868: aborting pipeline replication: peer="{Voter server-66814d4c-919b-4586-6941-30a133915fd8 66814d4c-919b-4586-6941-30a133915fd8}"
2025-01-25T14:28:17.030Z [INFO]  server-54f9a64c-2f91-e6ab-d9a7-d8732d9a7868: aborting pipeline replication: peer="{Voter server-c57e67b9-345e-2411-7165-ca2cbc39ed65 c57e67b9-345e-2411-7165-ca2cbc39ed65}"
--- PASS: TestRaft_TripleNode (0.29s)
=== RUN   TestRaft_LeaderFail
2025-01-25T14:28:17.032Z [INFO]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 Address:4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041} {Suffrage:Voter ID:server-8967c740-002b-c919-8b54-d6f78ad1c486 Address:8967c740-002b-c919-8b54-d6f78ad1c486} {Suffrage:Voter ID:server-e4cec951-a9ef-47d9-077c-6c2670d776c0 Address:e4cec951-a9ef-47d9-077c-6c2670d776c0}]"
2025-01-25T14:28:17.033Z [INFO]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: entering follower state: follower="Node at 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 [Follower]" leader-address= leader-id=
2025-01-25T14:28:17.033Z [INFO]  server-8967c740-002b-c919-8b54-d6f78ad1c486: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 Address:4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041} {Suffrage:Voter ID:server-8967c740-002b-c919-8b54-d6f78ad1c486 Address:8967c740-002b-c919-8b54-d6f78ad1c486} {Suffrage:Voter ID:server-e4cec951-a9ef-47d9-077c-6c2670d776c0 Address:e4cec951-a9ef-47d9-077c-6c2670d776c0}]"
2025-01-25T14:28:17.033Z [INFO]  server-8967c740-002b-c919-8b54-d6f78ad1c486: entering follower state: follower="Node at 8967c740-002b-c919-8b54-d6f78ad1c486 [Follower]" leader-address= leader-id=
2025-01-25T14:28:17.033Z [INFO]  server-e4cec951-a9ef-47d9-077c-6c2670d776c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 Address:4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041} {Suffrage:Voter ID:server-8967c740-002b-c919-8b54-d6f78ad1c486 Address:8967c740-002b-c919-8b54-d6f78ad1c486} {Suffrage:Voter ID:server-e4cec951-a9ef-47d9-077c-6c2670d776c0 Address:e4cec951-a9ef-47d9-077c-6c2670d776c0}]"
2025-01-25T14:28:17.033Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:17.033Z [INFO]  server-e4cec951-a9ef-47d9-077c-6c2670d776c0: entering follower state: follower="Node at e4cec951-a9ef-47d9-077c-6c2670d776c0 [Follower]" leader-address= leader-id=
2025-01-25T14:28:17.085Z [WARN]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:17.085Z [INFO]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: entering candidate state: node="Node at 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 [Candidate]" term=2
2025-01-25T14:28:17.085Z [INFO]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: election won: term=2 tally=2
2025-01-25T14:28:17.085Z [INFO]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: entering leader state: leader="Node at 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 [Leader]"
2025-01-25T14:28:17.085Z [INFO]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: added peer, starting replication: peer=server-8967c740-002b-c919-8b54-d6f78ad1c486
2025-01-25T14:28:17.085Z [INFO]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: added peer, starting replication: peer=server-e4cec951-a9ef-47d9-077c-6c2670d776c0
2025-01-25T14:28:17.085Z [INFO]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: pipelining replication: peer="{Voter server-e4cec951-a9ef-47d9-077c-6c2670d776c0 e4cec951-a9ef-47d9-077c-6c2670d776c0}"
2025-01-25T14:28:17.086Z [INFO]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: pipelining replication: peer="{Voter server-8967c740-002b-c919-8b54-d6f78ad1c486 8967c740-002b-c919-8b54-d6f78ad1c486}"
2025-01-25T14:28:17.191Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:17.086133834 +0000 UTC m=+6.940458816 (2 nodes), 52.814179ms from start of poll, 53.436317ms from cluster start. Timeout at 2025-01-25 14:28:17.191143314 +0000 UTC m=+7.045468296, 105.00948ms after stability
2025-01-25T14:28:17.191Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:17.297Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:17.192015864 +0000 UTC m=+7.046340846 (1 nodes), 2.36µs from start of poll, 159.318347ms from cluster start. Timeout at 2025-01-25 14:28:17.297047703 +0000 UTC m=+7.151372725, 105.031879ms after stability
    raft_test.go:378: [INFO] Disconnecting Node at 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 [Leader]
2025-01-25T14:28:17.312Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to heartbeat to: peer=e4cec951-a9ef-47d9-077c-6c2670d776c0 backoff time=10ms error="failed to connect to peer: e4cec951-a9ef-47d9-077c-6c2670d776c0"
2025-01-25T14:28:17.312Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:17.312Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to pipeline appendEntries: peer="{Voter server-8967c740-002b-c919-8b54-d6f78ad1c486 8967c740-002b-c919-8b54-d6f78ad1c486}" error="append pipeline closed"
2025-01-25T14:28:17.313Z [INFO]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: aborting pipeline replication: peer="{Voter server-8967c740-002b-c919-8b54-d6f78ad1c486 8967c740-002b-c919-8b54-d6f78ad1c486}"
2025-01-25T14:28:17.315Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to pipeline appendEntries: peer="{Voter server-e4cec951-a9ef-47d9-077c-6c2670d776c0 e4cec951-a9ef-47d9-077c-6c2670d776c0}" error="append pipeline closed"
2025-01-25T14:28:17.315Z [INFO]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: aborting pipeline replication: peer="{Voter server-e4cec951-a9ef-47d9-077c-6c2670d776c0 e4cec951-a9ef-47d9-077c-6c2670d776c0}"
2025-01-25T14:28:17.315Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to heartbeat to: peer=8967c740-002b-c919-8b54-d6f78ad1c486 backoff time=10ms error="failed to connect to peer: 8967c740-002b-c919-8b54-d6f78ad1c486"
2025-01-25T14:28:17.321Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to appendEntries to: peer="{Voter server-8967c740-002b-c919-8b54-d6f78ad1c486 8967c740-002b-c919-8b54-d6f78ad1c486}" error="failed to connect to peer: 8967c740-002b-c919-8b54-d6f78ad1c486"
2025-01-25T14:28:17.326Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to appendEntries to: peer="{Voter server-e4cec951-a9ef-47d9-077c-6c2670d776c0 e4cec951-a9ef-47d9-077c-6c2670d776c0}" error="failed to connect to peer: e4cec951-a9ef-47d9-077c-6c2670d776c0"
2025-01-25T14:28:17.333Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to heartbeat to: peer=e4cec951-a9ef-47d9-077c-6c2670d776c0 backoff time=10ms error="failed to connect to peer: e4cec951-a9ef-47d9-077c-6c2670d776c0"
2025-01-25T14:28:17.335Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to heartbeat to: peer=8967c740-002b-c919-8b54-d6f78ad1c486 backoff time=10ms error="failed to connect to peer: 8967c740-002b-c919-8b54-d6f78ad1c486"
2025-01-25T14:28:17.339Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to appendEntries to: peer="{Voter server-8967c740-002b-c919-8b54-d6f78ad1c486 8967c740-002b-c919-8b54-d6f78ad1c486}" error="failed to connect to peer: 8967c740-002b-c919-8b54-d6f78ad1c486"
2025-01-25T14:28:17.344Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to appendEntries to: peer="{Voter server-e4cec951-a9ef-47d9-077c-6c2670d776c0 e4cec951-a9ef-47d9-077c-6c2670d776c0}" error="failed to connect to peer: e4cec951-a9ef-47d9-077c-6c2670d776c0"
2025-01-25T14:28:17.351Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to heartbeat to: peer=e4cec951-a9ef-47d9-077c-6c2670d776c0 backoff time=10ms error="failed to connect to peer: e4cec951-a9ef-47d9-077c-6c2670d776c0"
2025-01-25T14:28:17.354Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to heartbeat to: peer=8967c740-002b-c919-8b54-d6f78ad1c486 backoff time=10ms error="failed to connect to peer: 8967c740-002b-c919-8b54-d6f78ad1c486"
2025-01-25T14:28:17.356Z [WARN]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to contact: server-id=server-8967c740-002b-c919-8b54-d6f78ad1c486 time=50.108197ms
2025-01-25T14:28:17.357Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to appendEntries to: peer="{Voter server-8967c740-002b-c919-8b54-d6f78ad1c486 8967c740-002b-c919-8b54-d6f78ad1c486}" error="failed to connect to peer: 8967c740-002b-c919-8b54-d6f78ad1c486"
2025-01-25T14:28:17.363Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to appendEntries to: peer="{Voter server-e4cec951-a9ef-47d9-077c-6c2670d776c0 e4cec951-a9ef-47d9-077c-6c2670d776c0}" error="failed to connect to peer: e4cec951-a9ef-47d9-077c-6c2670d776c0"
2025-01-25T14:28:17.367Z [WARN]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to contact: server-id=server-8967c740-002b-c919-8b54-d6f78ad1c486 time=61.278435ms
2025-01-25T14:28:17.367Z [WARN]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to contact: server-id=server-e4cec951-a9ef-47d9-077c-6c2670d776c0 time=60.084677ms
2025-01-25T14:28:17.367Z [WARN]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to contact quorum of nodes, stepping down
2025-01-25T14:28:17.367Z [INFO]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: entering follower state: follower="Node at 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 [Follower]" leader-address= leader-id=
2025-01-25T14:28:17.373Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to heartbeat to: peer=8967c740-002b-c919-8b54-d6f78ad1c486 backoff time=20ms error="failed to connect to peer: 8967c740-002b-c919-8b54-d6f78ad1c486"
2025-01-25T14:28:17.381Z [WARN]  server-8967c740-002b-c919-8b54-d6f78ad1c486: heartbeat timeout reached, starting election: last-leader-addr=4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 last-leader-id=server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041
2025-01-25T14:28:17.382Z [INFO]  server-8967c740-002b-c919-8b54-d6f78ad1c486: entering candidate state: node="Node at 8967c740-002b-c919-8b54-d6f78ad1c486 [Candidate]" term=3
2025-01-25T14:28:17.382Z [WARN]  server-e4cec951-a9ef-47d9-077c-6c2670d776c0: rejecting vote request since we have a leader: from=8967c740-002b-c919-8b54-d6f78ad1c486 leader=4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 leader-id=server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041
2025-01-25T14:28:17.382Z [ERROR] server-8967c740-002b-c919-8b54-d6f78ad1c486: failed to make requestVote RPC: target="{Voter server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041}" error="failed to connect to peer: 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041" term=3
2025-01-25T14:28:17.388Z [WARN]  server-e4cec951-a9ef-47d9-077c-6c2670d776c0: heartbeat timeout reached, starting election: last-leader-addr=4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 last-leader-id=server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041
2025-01-25T14:28:17.388Z [INFO]  server-e4cec951-a9ef-47d9-077c-6c2670d776c0: entering candidate state: node="Node at e4cec951-a9ef-47d9-077c-6c2670d776c0 [Candidate]" term=3
2025-01-25T14:28:17.388Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to appendEntries to: peer="{Voter server-8967c740-002b-c919-8b54-d6f78ad1c486 8967c740-002b-c919-8b54-d6f78ad1c486}" error="failed to connect to peer: 8967c740-002b-c919-8b54-d6f78ad1c486"
2025-01-25T14:28:17.388Z [INFO]  server-8967c740-002b-c919-8b54-d6f78ad1c486: duplicate requestVote for same term: term=3
2025-01-25T14:28:17.388Z [ERROR] server-e4cec951-a9ef-47d9-077c-6c2670d776c0: failed to make requestVote RPC: target="{Voter server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041}" error="failed to connect to peer: 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041" term=3
2025-01-25T14:28:17.447Z [WARN]  server-8967c740-002b-c919-8b54-d6f78ad1c486: Election timeout reached, restarting election
2025-01-25T14:28:17.447Z [INFO]  server-8967c740-002b-c919-8b54-d6f78ad1c486: entering candidate state: node="Node at 8967c740-002b-c919-8b54-d6f78ad1c486 [Candidate]" term=4
2025-01-25T14:28:17.447Z [INFO]  server-e4cec951-a9ef-47d9-077c-6c2670d776c0: entering follower state: follower="Node at e4cec951-a9ef-47d9-077c-6c2670d776c0 [Follower]" leader-address= leader-id=
2025-01-25T14:28:17.447Z [INFO]  server-8967c740-002b-c919-8b54-d6f78ad1c486: election won: term=4 tally=2
2025-01-25T14:28:17.447Z [INFO]  server-8967c740-002b-c919-8b54-d6f78ad1c486: entering leader state: leader="Node at 8967c740-002b-c919-8b54-d6f78ad1c486 [Leader]"
2025-01-25T14:28:17.447Z [INFO]  server-8967c740-002b-c919-8b54-d6f78ad1c486: added peer, starting replication: peer=server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041
2025-01-25T14:28:17.447Z [INFO]  server-8967c740-002b-c919-8b54-d6f78ad1c486: added peer, starting replication: peer=server-e4cec951-a9ef-47d9-077c-6c2670d776c0
2025-01-25T14:28:17.447Z [ERROR] server-8967c740-002b-c919-8b54-d6f78ad1c486: failed to make requestVote RPC: target="{Voter server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041}" error="failed to connect to peer: 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041" term=4
2025-01-25T14:28:17.448Z [ERROR] server-8967c740-002b-c919-8b54-d6f78ad1c486: failed to appendEntries to: peer="{Voter server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041}" error="failed to connect to peer: 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041"
2025-01-25T14:28:17.448Z [INFO]  server-8967c740-002b-c919-8b54-d6f78ad1c486: pipelining replication: peer="{Voter server-e4cec951-a9ef-47d9-077c-6c2670d776c0 e4cec951-a9ef-47d9-077c-6c2670d776c0}"
2025-01-25T14:28:17.455Z [ERROR] server-8967c740-002b-c919-8b54-d6f78ad1c486: failed to heartbeat to: peer=4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 backoff time=10ms error="failed to connect to peer: 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041"
2025-01-25T14:28:17.460Z [WARN]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:17.460Z [INFO]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: entering candidate state: node="Node at 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 [Candidate]" term=3
2025-01-25T14:28:17.460Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to make requestVote RPC: target="{Voter server-e4cec951-a9ef-47d9-077c-6c2670d776c0 e4cec951-a9ef-47d9-077c-6c2670d776c0}" error="failed to connect to peer: e4cec951-a9ef-47d9-077c-6c2670d776c0" term=3
2025-01-25T14:28:17.460Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to make requestVote RPC: target="{Voter server-8967c740-002b-c919-8b54-d6f78ad1c486 8967c740-002b-c919-8b54-d6f78ad1c486}" error="failed to connect to peer: 8967c740-002b-c919-8b54-d6f78ad1c486" term=3
2025-01-25T14:28:17.467Z [ERROR] server-8967c740-002b-c919-8b54-d6f78ad1c486: failed to appendEntries to: peer="{Voter server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041}" error="failed to connect to peer: 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041"
2025-01-25T14:28:17.474Z [ERROR] server-8967c740-002b-c919-8b54-d6f78ad1c486: failed to heartbeat to: peer=4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 backoff time=10ms error="failed to connect to peer: 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041"
2025-01-25T14:28:17.483Z [ERROR] server-8967c740-002b-c919-8b54-d6f78ad1c486: failed to appendEntries to: peer="{Voter server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041}" error="failed to connect to peer: 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041"
2025-01-25T14:28:17.492Z [ERROR] server-8967c740-002b-c919-8b54-d6f78ad1c486: failed to heartbeat to: peer=4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 backoff time=10ms error="failed to connect to peer: 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041"
2025-01-25T14:28:17.498Z [WARN]  server-8967c740-002b-c919-8b54-d6f78ad1c486: failed to contact: server-id=server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 time=50.31803ms
2025-01-25T14:28:17.509Z [ERROR] server-8967c740-002b-c919-8b54-d6f78ad1c486: failed to heartbeat to: peer=4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 backoff time=20ms error="failed to connect to peer: 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041"
2025-01-25T14:28:17.510Z [ERROR] server-8967c740-002b-c919-8b54-d6f78ad1c486: failed to appendEntries to: peer="{Voter server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041}" error="failed to connect to peer: 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041"
2025-01-25T14:28:17.536Z [ERROR] server-8967c740-002b-c919-8b54-d6f78ad1c486: failed to heartbeat to: peer=4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 backoff time=25ms error="failed to connect to peer: 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041"
2025-01-25T14:28:17.546Z [WARN]  server-8967c740-002b-c919-8b54-d6f78ad1c486: failed to contact: server-id=server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 time=98.11543ms
2025-01-25T14:28:17.557Z [ERROR] server-8967c740-002b-c919-8b54-d6f78ad1c486: failed to appendEntries to: peer="{Voter server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041}" error="failed to connect to peer: 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041"
2025-01-25T14:28:17.560Z [WARN]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: Election timeout reached, restarting election
2025-01-25T14:28:17.560Z [INFO]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: entering candidate state: node="Node at 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 [Candidate]" term=4
2025-01-25T14:28:17.560Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to make requestVote RPC: target="{Voter server-e4cec951-a9ef-47d9-077c-6c2670d776c0 e4cec951-a9ef-47d9-077c-6c2670d776c0}" error="failed to connect to peer: e4cec951-a9ef-47d9-077c-6c2670d776c0" term=4
2025-01-25T14:28:17.560Z [ERROR] server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: failed to make requestVote RPC: target="{Voter server-8967c740-002b-c919-8b54-d6f78ad1c486 8967c740-002b-c919-8b54-d6f78ad1c486}" error="failed to connect to peer: 8967c740-002b-c919-8b54-d6f78ad1c486" term=4
2025-01-25T14:28:17.565Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:17.460462724 +0000 UTC m=+7.314787706 (1 nodes), 147.580689ms from start of poll, 427.765207ms from cluster start. Timeout at 2025-01-25 14:28:17.565472003 +0000 UTC m=+7.419797065, 105.009359ms after stability
    raft_test.go:413: [INFO] Reconnecting Node at 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 [Candidate]
2025-01-25T14:28:17.568Z [INFO]  server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041: entering follower state: follower="Node at 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 [Follower]" leader-address=8967c740-002b-c919-8b54-d6f78ad1c486 leader-id=server-8967c740-002b-c919-8b54-d6f78ad1c486
2025-01-25T14:28:17.646Z [INFO]  server-8967c740-002b-c919-8b54-d6f78ad1c486: pipelining replication: peer="{Voter server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041}"
2025-01-25T14:28:17.649Z [INFO]  server-8967c740-002b-c919-8b54-d6f78ad1c486: aborting pipeline replication: peer="{Voter server-e4cec951-a9ef-47d9-077c-6c2670d776c0 e4cec951-a9ef-47d9-077c-6c2670d776c0}"
2025-01-25T14:28:17.649Z [INFO]  server-8967c740-002b-c919-8b54-d6f78ad1c486: aborting pipeline replication: peer="{Voter server-4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041 4bf52fbd-cc7e-d3c3-1e19-0dfd3cb65041}"
--- PASS: TestRaft_LeaderFail (0.62s)
=== RUN   TestRaft_BehindFollower
2025-01-25T14:28:17.651Z [INFO]  server-e5108ba7-450f-e118-c1b0-c48038aef9e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e5108ba7-450f-e118-c1b0-c48038aef9e4 Address:e5108ba7-450f-e118-c1b0-c48038aef9e4} {Suffrage:Voter ID:server-5734fc47-f636-8665-780c-90a00ec2398a Address:5734fc47-f636-8665-780c-90a00ec2398a} {Suffrage:Voter ID:server-404920e1-e475-3b28-01bd-ec1e4c8213ac Address:404920e1-e475-3b28-01bd-ec1e4c8213ac}]"
2025-01-25T14:28:17.651Z [INFO]  server-e5108ba7-450f-e118-c1b0-c48038aef9e4: entering follower state: follower="Node at e5108ba7-450f-e118-c1b0-c48038aef9e4 [Follower]" leader-address= leader-id=
2025-01-25T14:28:17.652Z [INFO]  server-5734fc47-f636-8665-780c-90a00ec2398a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e5108ba7-450f-e118-c1b0-c48038aef9e4 Address:e5108ba7-450f-e118-c1b0-c48038aef9e4} {Suffrage:Voter ID:server-5734fc47-f636-8665-780c-90a00ec2398a Address:5734fc47-f636-8665-780c-90a00ec2398a} {Suffrage:Voter ID:server-404920e1-e475-3b28-01bd-ec1e4c8213ac Address:404920e1-e475-3b28-01bd-ec1e4c8213ac}]"
2025-01-25T14:28:17.652Z [INFO]  server-5734fc47-f636-8665-780c-90a00ec2398a: entering follower state: follower="Node at 5734fc47-f636-8665-780c-90a00ec2398a [Follower]" leader-address= leader-id=
2025-01-25T14:28:17.652Z [INFO]  server-404920e1-e475-3b28-01bd-ec1e4c8213ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e5108ba7-450f-e118-c1b0-c48038aef9e4 Address:e5108ba7-450f-e118-c1b0-c48038aef9e4} {Suffrage:Voter ID:server-5734fc47-f636-8665-780c-90a00ec2398a Address:5734fc47-f636-8665-780c-90a00ec2398a} {Suffrage:Voter ID:server-404920e1-e475-3b28-01bd-ec1e4c8213ac Address:404920e1-e475-3b28-01bd-ec1e4c8213ac}]"
2025-01-25T14:28:17.652Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:17.652Z [INFO]  server-404920e1-e475-3b28-01bd-ec1e4c8213ac: entering follower state: follower="Node at 404920e1-e475-3b28-01bd-ec1e4c8213ac [Follower]" leader-address= leader-id=
2025-01-25T14:28:17.747Z [WARN]  server-e5108ba7-450f-e118-c1b0-c48038aef9e4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:17.747Z [WARN]  server-404920e1-e475-3b28-01bd-ec1e4c8213ac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:17.747Z [INFO]  server-404920e1-e475-3b28-01bd-ec1e4c8213ac: entering candidate state: node="Node at 404920e1-e475-3b28-01bd-ec1e4c8213ac [Candidate]" term=2
2025-01-25T14:28:17.747Z [WARN]  server-5734fc47-f636-8665-780c-90a00ec2398a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:17.747Z [INFO]  server-5734fc47-f636-8665-780c-90a00ec2398a: entering candidate state: node="Node at 5734fc47-f636-8665-780c-90a00ec2398a [Candidate]" term=2
2025-01-25T14:28:17.747Z [INFO]  server-e5108ba7-450f-e118-c1b0-c48038aef9e4: entering candidate state: node="Node at e5108ba7-450f-e118-c1b0-c48038aef9e4 [Candidate]" term=2
2025-01-25T14:28:17.747Z [INFO]  server-5734fc47-f636-8665-780c-90a00ec2398a: duplicate requestVote for same term: term=2
2025-01-25T14:28:17.747Z [INFO]  server-404920e1-e475-3b28-01bd-ec1e4c8213ac: duplicate requestVote for same term: term=2
2025-01-25T14:28:17.747Z [INFO]  server-e5108ba7-450f-e118-c1b0-c48038aef9e4: duplicate requestVote for same term: term=2
2025-01-25T14:28:17.747Z [INFO]  server-404920e1-e475-3b28-01bd-ec1e4c8213ac: duplicate requestVote for same term: term=2
2025-01-25T14:28:17.747Z [INFO]  server-e5108ba7-450f-e118-c1b0-c48038aef9e4: duplicate requestVote for same term: term=2
2025-01-25T14:28:17.747Z [INFO]  server-5734fc47-f636-8665-780c-90a00ec2398a: duplicate requestVote for same term: term=2
2025-01-25T14:28:17.824Z [WARN]  server-5734fc47-f636-8665-780c-90a00ec2398a: Election timeout reached, restarting election
2025-01-25T14:28:17.824Z [INFO]  server-5734fc47-f636-8665-780c-90a00ec2398a: entering candidate state: node="Node at 5734fc47-f636-8665-780c-90a00ec2398a [Candidate]" term=3
2025-01-25T14:28:17.824Z [INFO]  server-404920e1-e475-3b28-01bd-ec1e4c8213ac: entering follower state: follower="Node at 404920e1-e475-3b28-01bd-ec1e4c8213ac [Follower]" leader-address= leader-id=
2025-01-25T14:28:17.824Z [INFO]  server-5734fc47-f636-8665-780c-90a00ec2398a: election won: term=3 tally=2
2025-01-25T14:28:17.824Z [INFO]  server-5734fc47-f636-8665-780c-90a00ec2398a: entering leader state: leader="Node at 5734fc47-f636-8665-780c-90a00ec2398a [Leader]"
2025-01-25T14:28:17.824Z [INFO]  server-5734fc47-f636-8665-780c-90a00ec2398a: added peer, starting replication: peer=server-e5108ba7-450f-e118-c1b0-c48038aef9e4
2025-01-25T14:28:17.824Z [INFO]  server-5734fc47-f636-8665-780c-90a00ec2398a: added peer, starting replication: peer=server-404920e1-e475-3b28-01bd-ec1e4c8213ac
2025-01-25T14:28:17.825Z [INFO]  server-5734fc47-f636-8665-780c-90a00ec2398a: pipelining replication: peer="{Voter server-404920e1-e475-3b28-01bd-ec1e4c8213ac 404920e1-e475-3b28-01bd-ec1e4c8213ac}"
2025-01-25T14:28:17.825Z [INFO]  server-e5108ba7-450f-e118-c1b0-c48038aef9e4: entering follower state: follower="Node at e5108ba7-450f-e118-c1b0-c48038aef9e4 [Follower]" leader-address= leader-id=
2025-01-25T14:28:17.825Z [INFO]  server-5734fc47-f636-8665-780c-90a00ec2398a: pipelining replication: peer="{Voter server-e5108ba7-450f-e118-c1b0-c48038aef9e4 e5108ba7-450f-e118-c1b0-c48038aef9e4}"
2025-01-25T14:28:17.930Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:17.825252678 +0000 UTC m=+7.679577660 (1 nodes), 172.969656ms from start of poll, 173.629673ms from cluster start. Timeout at 2025-01-25 14:28:17.930275637 +0000 UTC m=+7.784600619, 105.022959ms after stability
2025-01-25T14:28:17.930Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:18.035Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:17.930699523 +0000 UTC m=+7.785024505 (2 nodes), 2.8µs from start of poll, 279.076518ms from cluster start. Timeout at 2025-01-25 14:28:18.035735921 +0000 UTC m=+7.890060903, 105.036398ms after stability
2025-01-25T14:28:18.036Z [ERROR] server-5734fc47-f636-8665-780c-90a00ec2398a: failed to pipeline appendEntries: peer="{Voter server-e5108ba7-450f-e118-c1b0-c48038aef9e4 e5108ba7-450f-e118-c1b0-c48038aef9e4}" error="append pipeline closed"
2025-01-25T14:28:18.036Z [INFO]  server-5734fc47-f636-8665-780c-90a00ec2398a: aborting pipeline replication: peer="{Voter server-e5108ba7-450f-e118-c1b0-c48038aef9e4 e5108ba7-450f-e118-c1b0-c48038aef9e4}"
2025-01-25T14:28:18.036Z [ERROR] server-5734fc47-f636-8665-780c-90a00ec2398a: failed to appendEntries to: peer="{Voter server-e5108ba7-450f-e118-c1b0-c48038aef9e4 e5108ba7-450f-e118-c1b0-c48038aef9e4}" error="failed to connect to peer: e5108ba7-450f-e118-c1b0-c48038aef9e4"
2025-01-25T14:28:18.043Z [ERROR] server-5734fc47-f636-8665-780c-90a00ec2398a: failed to heartbeat to: peer=e5108ba7-450f-e118-c1b0-c48038aef9e4 backoff time=10ms error="failed to connect to peer: e5108ba7-450f-e118-c1b0-c48038aef9e4"
    raft_test.go:459: [INFO] Finished apply without behind follower
2025-01-25T14:28:18.046Z [INFO]  server-5734fc47-f636-8665-780c-90a00ec2398a: pipelining replication: peer="{Voter server-e5108ba7-450f-e118-c1b0-c48038aef9e4 e5108ba7-450f-e118-c1b0-c48038aef9e4}"
2025-01-25T14:28:18.054Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:18.160Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:18.054920973 +0000 UTC m=+7.909245955 (1 nodes), 1.68µs from start of poll, 403.297968ms from cluster start. Timeout at 2025-01-25 14:28:18.159960572 +0000 UTC m=+8.014285674, 105.039719ms after stability
2025-01-25T14:28:18.160Z [INFO]  server-5734fc47-f636-8665-780c-90a00ec2398a: aborting pipeline replication: peer="{Voter server-404920e1-e475-3b28-01bd-ec1e4c8213ac 404920e1-e475-3b28-01bd-ec1e4c8213ac}"
2025-01-25T14:28:18.160Z [INFO]  server-5734fc47-f636-8665-780c-90a00ec2398a: aborting pipeline replication: peer="{Voter server-e5108ba7-450f-e118-c1b0-c48038aef9e4 e5108ba7-450f-e118-c1b0-c48038aef9e4}"
--- PASS: TestRaft_BehindFollower (0.51s)
=== RUN   TestRaft_ApplyNonLeader
2025-01-25T14:28:18.164Z [INFO]  server-07a949fa-857e-58dc-6852-8e2424b37ceb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07a949fa-857e-58dc-6852-8e2424b37ceb Address:07a949fa-857e-58dc-6852-8e2424b37ceb} {Suffrage:Voter ID:server-9e32af90-93ed-156e-02b6-5267982b6d22 Address:9e32af90-93ed-156e-02b6-5267982b6d22} {Suffrage:Voter ID:server-b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71 Address:b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71}]"
2025-01-25T14:28:18.164Z [INFO]  server-07a949fa-857e-58dc-6852-8e2424b37ceb: entering follower state: follower="Node at 07a949fa-857e-58dc-6852-8e2424b37ceb [Follower]" leader-address= leader-id=
2025-01-25T14:28:18.165Z [INFO]  server-9e32af90-93ed-156e-02b6-5267982b6d22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07a949fa-857e-58dc-6852-8e2424b37ceb Address:07a949fa-857e-58dc-6852-8e2424b37ceb} {Suffrage:Voter ID:server-9e32af90-93ed-156e-02b6-5267982b6d22 Address:9e32af90-93ed-156e-02b6-5267982b6d22} {Suffrage:Voter ID:server-b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71 Address:b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71}]"
2025-01-25T14:28:18.165Z [INFO]  server-b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07a949fa-857e-58dc-6852-8e2424b37ceb Address:07a949fa-857e-58dc-6852-8e2424b37ceb} {Suffrage:Voter ID:server-9e32af90-93ed-156e-02b6-5267982b6d22 Address:9e32af90-93ed-156e-02b6-5267982b6d22} {Suffrage:Voter ID:server-b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71 Address:b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71}]"
2025-01-25T14:28:18.165Z [INFO]  server-9e32af90-93ed-156e-02b6-5267982b6d22: entering follower state: follower="Node at 9e32af90-93ed-156e-02b6-5267982b6d22 [Follower]" leader-address= leader-id=
2025-01-25T14:28:18.165Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:18.165Z [INFO]  server-b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71: entering follower state: follower="Node at b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71 [Follower]" leader-address= leader-id=
2025-01-25T14:28:18.223Z [WARN]  server-b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:18.223Z [INFO]  server-b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71: entering candidate state: node="Node at b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71 [Candidate]" term=2
2025-01-25T14:28:18.223Z [INFO]  server-b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71: election won: term=2 tally=2
2025-01-25T14:28:18.223Z [INFO]  server-b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71: entering leader state: leader="Node at b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71 [Leader]"
2025-01-25T14:28:18.223Z [INFO]  server-b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71: added peer, starting replication: peer=server-07a949fa-857e-58dc-6852-8e2424b37ceb
2025-01-25T14:28:18.223Z [INFO]  server-b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71: added peer, starting replication: peer=server-9e32af90-93ed-156e-02b6-5267982b6d22
2025-01-25T14:28:18.223Z [INFO]  server-b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71: pipelining replication: peer="{Voter server-9e32af90-93ed-156e-02b6-5267982b6d22 9e32af90-93ed-156e-02b6-5267982b6d22}"
2025-01-25T14:28:18.223Z [INFO]  server-b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71: pipelining replication: peer="{Voter server-07a949fa-857e-58dc-6852-8e2424b37ceb 07a949fa-857e-58dc-6852-8e2424b37ceb}"
2025-01-25T14:28:18.329Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:18.223870374 +0000 UTC m=+8.078195356 (1 nodes), 57.872278ms from start of poll, 60.021761ms from cluster start. Timeout at 2025-01-25 14:28:18.328872814 +0000 UTC m=+8.183197756, 105.0024ms after stability
2025-01-25T14:28:18.330Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:18.435Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:18.330051873 +0000 UTC m=+8.184376855 (2 nodes), 2.84µs from start of poll, 166.20326ms from cluster start. Timeout at 2025-01-25 14:28:18.435085432 +0000 UTC m=+8.289410414, 105.033559ms after stability
2025-01-25T14:28:18.436Z [INFO]  server-b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71: aborting pipeline replication: peer="{Voter server-9e32af90-93ed-156e-02b6-5267982b6d22 9e32af90-93ed-156e-02b6-5267982b6d22}"
2025-01-25T14:28:18.436Z [INFO]  server-b9d6c7d3-7840-3f7d-50ba-a0ff69e55a71: aborting pipeline replication: peer="{Voter server-07a949fa-857e-58dc-6852-8e2424b37ceb 07a949fa-857e-58dc-6852-8e2424b37ceb}"
--- PASS: TestRaft_ApplyNonLeader (0.28s)
=== RUN   TestRaft_ApplyConcurrent
2025-01-25T14:28:18.438Z [INFO]  server-6af0f32e-f7a7-263c-a0bc-632980e4ab2b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6af0f32e-f7a7-263c-a0bc-632980e4ab2b Address:6af0f32e-f7a7-263c-a0bc-632980e4ab2b} {Suffrage:Voter ID:server-67949996-9e51-fe6a-f647-52f1a2f95191 Address:67949996-9e51-fe6a-f647-52f1a2f95191} {Suffrage:Voter ID:server-2d8e96d3-58c2-81f2-c49a-8240ce5df7d8 Address:2d8e96d3-58c2-81f2-c49a-8240ce5df7d8}]"
2025-01-25T14:28:18.438Z [INFO]  server-67949996-9e51-fe6a-f647-52f1a2f95191: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6af0f32e-f7a7-263c-a0bc-632980e4ab2b Address:6af0f32e-f7a7-263c-a0bc-632980e4ab2b} {Suffrage:Voter ID:server-67949996-9e51-fe6a-f647-52f1a2f95191 Address:67949996-9e51-fe6a-f647-52f1a2f95191} {Suffrage:Voter ID:server-2d8e96d3-58c2-81f2-c49a-8240ce5df7d8 Address:2d8e96d3-58c2-81f2-c49a-8240ce5df7d8}]"
2025-01-25T14:28:18.438Z [INFO]  server-6af0f32e-f7a7-263c-a0bc-632980e4ab2b: entering follower state: follower="Node at 6af0f32e-f7a7-263c-a0bc-632980e4ab2b [Follower]" leader-address= leader-id=
2025-01-25T14:28:18.438Z [INFO]  server-67949996-9e51-fe6a-f647-52f1a2f95191: entering follower state: follower="Node at 67949996-9e51-fe6a-f647-52f1a2f95191 [Follower]" leader-address= leader-id=
2025-01-25T14:28:18.438Z [INFO]  server-2d8e96d3-58c2-81f2-c49a-8240ce5df7d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6af0f32e-f7a7-263c-a0bc-632980e4ab2b Address:6af0f32e-f7a7-263c-a0bc-632980e4ab2b} {Suffrage:Voter ID:server-67949996-9e51-fe6a-f647-52f1a2f95191 Address:67949996-9e51-fe6a-f647-52f1a2f95191} {Suffrage:Voter ID:server-2d8e96d3-58c2-81f2-c49a-8240ce5df7d8 Address:2d8e96d3-58c2-81f2-c49a-8240ce5df7d8}]"
2025-01-25T14:28:18.438Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:18.438Z [INFO]  server-2d8e96d3-58c2-81f2-c49a-8240ce5df7d8: entering follower state: follower="Node at 2d8e96d3-58c2-81f2-c49a-8240ce5df7d8 [Follower]" leader-address= leader-id=
2025-01-25T14:28:18.597Z [WARN]  server-6af0f32e-f7a7-263c-a0bc-632980e4ab2b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:18.597Z [INFO]  server-6af0f32e-f7a7-263c-a0bc-632980e4ab2b: entering candidate state: node="Node at 6af0f32e-f7a7-263c-a0bc-632980e4ab2b [Candidate]" term=2
2025-01-25T14:28:18.597Z [INFO]  server-6af0f32e-f7a7-263c-a0bc-632980e4ab2b: election won: term=2 tally=2
2025-01-25T14:28:18.597Z [INFO]  server-6af0f32e-f7a7-263c-a0bc-632980e4ab2b: entering leader state: leader="Node at 6af0f32e-f7a7-263c-a0bc-632980e4ab2b [Leader]"
2025-01-25T14:28:18.597Z [INFO]  server-6af0f32e-f7a7-263c-a0bc-632980e4ab2b: added peer, starting replication: peer=server-67949996-9e51-fe6a-f647-52f1a2f95191
2025-01-25T14:28:18.597Z [INFO]  server-6af0f32e-f7a7-263c-a0bc-632980e4ab2b: added peer, starting replication: peer=server-2d8e96d3-58c2-81f2-c49a-8240ce5df7d8
2025-01-25T14:28:18.597Z [INFO]  server-6af0f32e-f7a7-263c-a0bc-632980e4ab2b: pipelining replication: peer="{Voter server-67949996-9e51-fe6a-f647-52f1a2f95191 67949996-9e51-fe6a-f647-52f1a2f95191}"
2025-01-25T14:28:18.597Z [INFO]  server-6af0f32e-f7a7-263c-a0bc-632980e4ab2b: pipelining replication: peer="{Voter server-2d8e96d3-58c2-81f2-c49a-8240ce5df7d8 2d8e96d3-58c2-81f2-c49a-8240ce5df7d8}"
2025-01-25T14:28:18.803Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:18.597715561 +0000 UTC m=+8.452040543 (1 nodes), 158.852324ms from start of poll, 159.52102ms from cluster start. Timeout at 2025-01-25 14:28:18.802719001 +0000 UTC m=+8.657043943, 205.0034ms after stability
2025-01-25T14:28:18.818Z [INFO]  server-6af0f32e-f7a7-263c-a0bc-632980e4ab2b: aborting pipeline replication: peer="{Voter server-2d8e96d3-58c2-81f2-c49a-8240ce5df7d8 2d8e96d3-58c2-81f2-c49a-8240ce5df7d8}"
2025-01-25T14:28:18.818Z [INFO]  server-6af0f32e-f7a7-263c-a0bc-632980e4ab2b: aborting pipeline replication: peer="{Voter server-67949996-9e51-fe6a-f647-52f1a2f95191 67949996-9e51-fe6a-f647-52f1a2f95191}"
--- PASS: TestRaft_ApplyConcurrent (0.38s)
=== RUN   TestRaft_ApplyConcurrent_Timeout
2025-01-25T14:28:18.819Z [INFO]  server-0754073f-4eb0-1938-6b44-664ad4d119cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0754073f-4eb0-1938-6b44-664ad4d119cf Address:0754073f-4eb0-1938-6b44-664ad4d119cf}]"
2025-01-25T14:28:18.820Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:18.820Z [INFO]  server-0754073f-4eb0-1938-6b44-664ad4d119cf: entering follower state: follower="Node at 0754073f-4eb0-1938-6b44-664ad4d119cf [Follower]" leader-address= leader-id=
2025-01-25T14:28:18.934Z [WARN]  server-0754073f-4eb0-1938-6b44-664ad4d119cf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:18.934Z [INFO]  server-0754073f-4eb0-1938-6b44-664ad4d119cf: entering candidate state: node="Node at 0754073f-4eb0-1938-6b44-664ad4d119cf [Candidate]" term=2
2025-01-25T14:28:18.934Z [INFO]  server-0754073f-4eb0-1938-6b44-664ad4d119cf: election won: term=2 tally=1
2025-01-25T14:28:18.934Z [INFO]  server-0754073f-4eb0-1938-6b44-664ad4d119cf: entering leader state: leader="Node at 0754073f-4eb0-1938-6b44-664ad4d119cf [Leader]"
2025-01-25T14:28:19.136Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:18.934531001 +0000 UTC m=+8.788855983 (1 nodes), 114.474841ms from start of poll, 114.77503ms from cluster start. Timeout at 2025-01-25 14:28:19.135536201 +0000 UTC m=+8.989861143, 201.00516ms after stability
--- PASS: TestRaft_ApplyConcurrent_Timeout (0.32s)
=== RUN   TestRaft_JoinNode
2025-01-25T14:28:19.138Z [INFO]  server-0c9b1edb-7cf4-7768-93b3-f988d8cdb637: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c9b1edb-7cf4-7768-93b3-f988d8cdb637 Address:0c9b1edb-7cf4-7768-93b3-f988d8cdb637} {Suffrage:Voter ID:server-0b1e82db-67f2-3ee1-2ebd-8a9e60a63702 Address:0b1e82db-67f2-3ee1-2ebd-8a9e60a63702}]"
2025-01-25T14:28:19.139Z [INFO]  server-0c9b1edb-7cf4-7768-93b3-f988d8cdb637: entering follower state: follower="Node at 0c9b1edb-7cf4-7768-93b3-f988d8cdb637 [Follower]" leader-address= leader-id=
2025-01-25T14:28:19.139Z [INFO]  server-0b1e82db-67f2-3ee1-2ebd-8a9e60a63702: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c9b1edb-7cf4-7768-93b3-f988d8cdb637 Address:0c9b1edb-7cf4-7768-93b3-f988d8cdb637} {Suffrage:Voter ID:server-0b1e82db-67f2-3ee1-2ebd-8a9e60a63702 Address:0b1e82db-67f2-3ee1-2ebd-8a9e60a63702}]"
2025-01-25T14:28:19.139Z [INFO]  server-0b1e82db-67f2-3ee1-2ebd-8a9e60a63702: entering follower state: follower="Node at 0b1e82db-67f2-3ee1-2ebd-8a9e60a63702 [Follower]" leader-address= leader-id=
2025-01-25T14:28:19.139Z [INFO]  server-37ee8ec6-3b94-752c-20f1-4b78d967bdb6: initial configuration: index=0 servers=[]
2025-01-25T14:28:19.139Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:19.140Z [INFO]  server-37ee8ec6-3b94-752c-20f1-4b78d967bdb6: entering follower state: follower="Node at 37ee8ec6-3b94-752c-20f1-4b78d967bdb6 [Follower]" leader-address= leader-id=
2025-01-25T14:28:19.198Z [WARN]  server-37ee8ec6-3b94-752c-20f1-4b78d967bdb6: no known peers, aborting election
2025-01-25T14:28:19.205Z [WARN]  server-0c9b1edb-7cf4-7768-93b3-f988d8cdb637: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:19.205Z [INFO]  server-0c9b1edb-7cf4-7768-93b3-f988d8cdb637: entering candidate state: node="Node at 0c9b1edb-7cf4-7768-93b3-f988d8cdb637 [Candidate]" term=2
2025-01-25T14:28:19.205Z [INFO]  server-0c9b1edb-7cf4-7768-93b3-f988d8cdb637: election won: term=2 tally=2
2025-01-25T14:28:19.205Z [INFO]  server-0c9b1edb-7cf4-7768-93b3-f988d8cdb637: entering leader state: leader="Node at 0c9b1edb-7cf4-7768-93b3-f988d8cdb637 [Leader]"
2025-01-25T14:28:19.205Z [INFO]  server-0c9b1edb-7cf4-7768-93b3-f988d8cdb637: added peer, starting replication: peer=server-0b1e82db-67f2-3ee1-2ebd-8a9e60a63702
2025-01-25T14:28:19.205Z [INFO]  server-0c9b1edb-7cf4-7768-93b3-f988d8cdb637: pipelining replication: peer="{Voter server-0b1e82db-67f2-3ee1-2ebd-8a9e60a63702 0b1e82db-67f2-3ee1-2ebd-8a9e60a63702}"
2025-01-25T14:28:19.311Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:19.206024352 +0000 UTC m=+9.060349334 (1 nodes), 66.010225ms from start of poll, 67.235261ms from cluster start. Timeout at 2025-01-25 14:28:19.311028432 +0000 UTC m=+9.165353414, 105.00408ms after stability
2025-01-25T14:28:19.311Z [INFO]  server-0c9b1edb-7cf4-7768-93b3-f988d8cdb637: updating configuration: command=AddVoter server-id=server-37ee8ec6-3b94-752c-20f1-4b78d967bdb6 server-addr=37ee8ec6-3b94-752c-20f1-4b78d967bdb6 servers="[{Suffrage:Voter ID:server-0c9b1edb-7cf4-7768-93b3-f988d8cdb637 Address:0c9b1edb-7cf4-7768-93b3-f988d8cdb637} {Suffrage:Voter ID:server-0b1e82db-67f2-3ee1-2ebd-8a9e60a63702 Address:0b1e82db-67f2-3ee1-2ebd-8a9e60a63702} {Suffrage:Voter ID:server-37ee8ec6-3b94-752c-20f1-4b78d967bdb6 Address:37ee8ec6-3b94-752c-20f1-4b78d967bdb6}]"
2025-01-25T14:28:19.311Z [INFO]  server-0c9b1edb-7cf4-7768-93b3-f988d8cdb637: added peer, starting replication: peer=server-37ee8ec6-3b94-752c-20f1-4b78d967bdb6
2025-01-25T14:28:19.311Z [WARN]  server-37ee8ec6-3b94-752c-20f1-4b78d967bdb6: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-01-25T14:28:19.311Z [WARN]  server-0c9b1edb-7cf4-7768-93b3-f988d8cdb637: appendEntries rejected, sending older logs: peer="{Voter server-37ee8ec6-3b94-752c-20f1-4b78d967bdb6 37ee8ec6-3b94-752c-20f1-4b78d967bdb6}" next=1
2025-01-25T14:28:19.312Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:19.312Z [INFO]  server-0c9b1edb-7cf4-7768-93b3-f988d8cdb637: pipelining replication: peer="{Voter server-37ee8ec6-3b94-752c-20f1-4b78d967bdb6 37ee8ec6-3b94-752c-20f1-4b78d967bdb6}"
2025-01-25T14:28:19.417Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:19.312204811 +0000 UTC m=+9.166529793 (1 nodes), 1.32µs from start of poll, 173.41572ms from cluster start. Timeout at 2025-01-25 14:28:19.41722325 +0000 UTC m=+9.271548232, 105.018439ms after stability
2025-01-25T14:28:19.417Z [INFO]  server-0c9b1edb-7cf4-7768-93b3-f988d8cdb637: aborting pipeline replication: peer="{Voter server-37ee8ec6-3b94-752c-20f1-4b78d967bdb6 37ee8ec6-3b94-752c-20f1-4b78d967bdb6}"
2025-01-25T14:28:19.417Z [INFO]  server-0c9b1edb-7cf4-7768-93b3-f988d8cdb637: aborting pipeline replication: peer="{Voter server-0b1e82db-67f2-3ee1-2ebd-8a9e60a63702 0b1e82db-67f2-3ee1-2ebd-8a9e60a63702}"
--- PASS: TestRaft_JoinNode (0.28s)
=== RUN   TestRaft_JoinNode_ConfigStore
2025-01-25T14:28:19.419Z [INFO]  server-9faaba19-5697-0983-daae-aedf305f130d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9faaba19-5697-0983-daae-aedf305f130d Address:9faaba19-5697-0983-daae-aedf305f130d}]"
2025-01-25T14:28:19.419Z [INFO]  server-9faaba19-5697-0983-daae-aedf305f130d: entering follower state: follower="Node at 9faaba19-5697-0983-daae-aedf305f130d [Follower]" leader-address= leader-id=
2025-01-25T14:28:19.420Z [INFO]  server-7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef: initial configuration: index=0 servers=[]
2025-01-25T14:28:19.420Z [INFO]  server-7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef: entering follower state: follower="Node at 7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef [Follower]" leader-address= leader-id=
2025-01-25T14:28:19.420Z [INFO]  server-4d9f8495-f09e-c8e0-a12d-9f809ef17a48: initial configuration: index=0 servers=[]
2025-01-25T14:28:19.420Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:19.420Z [INFO]  server-4d9f8495-f09e-c8e0-a12d-9f809ef17a48: entering follower state: follower="Node at 4d9f8495-f09e-c8e0-a12d-9f809ef17a48 [Follower]" leader-address= leader-id=
2025-01-25T14:28:19.493Z [WARN]  server-4d9f8495-f09e-c8e0-a12d-9f809ef17a48: no known peers, aborting election
2025-01-25T14:28:19.501Z [WARN]  server-9faaba19-5697-0983-daae-aedf305f130d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:19.501Z [INFO]  server-9faaba19-5697-0983-daae-aedf305f130d: entering candidate state: node="Node at 9faaba19-5697-0983-daae-aedf305f130d [Candidate]" term=2
2025-01-25T14:28:19.501Z [INFO]  server-9faaba19-5697-0983-daae-aedf305f130d: election won: term=2 tally=1
2025-01-25T14:28:19.501Z [INFO]  server-9faaba19-5697-0983-daae-aedf305f130d: entering leader state: leader="Node at 9faaba19-5697-0983-daae-aedf305f130d [Leader]"
2025-01-25T14:28:19.515Z [WARN]  server-7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef: no known peers, aborting election
2025-01-25T14:28:19.607Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:19.501948703 +0000 UTC m=+9.356273685 (1 nodes), 81.196278ms from start of poll, 82.348677ms from cluster start. Timeout at 2025-01-25 14:28:19.606952943 +0000 UTC m=+9.461277885, 105.0042ms after stability
2025-01-25T14:28:19.607Z [INFO]  server-9faaba19-5697-0983-daae-aedf305f130d: updating configuration: command=AddVoter server-id=server-7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef server-addr=7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef servers="[{Suffrage:Voter ID:server-9faaba19-5697-0983-daae-aedf305f130d Address:9faaba19-5697-0983-daae-aedf305f130d} {Suffrage:Voter ID:server-7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef Address:7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef}]"
2025-01-25T14:28:19.607Z [INFO]  server-9faaba19-5697-0983-daae-aedf305f130d: added peer, starting replication: peer=server-7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef
2025-01-25T14:28:19.607Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:19.607Z [WARN]  server-7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-01-25T14:28:19.608Z [WARN]  server-9faaba19-5697-0983-daae-aedf305f130d: appendEntries rejected, sending older logs: peer="{Voter server-7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef 7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef}" next=1
2025-01-25T14:28:19.608Z [INFO]  server-9faaba19-5697-0983-daae-aedf305f130d: pipelining replication: peer="{Voter server-7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef 7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef}"
2025-01-25T14:28:19.713Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:19.607864252 +0000 UTC m=+9.462189234 (1 nodes), 1.36µs from start of poll, 188.264226ms from cluster start. Timeout at 2025-01-25 14:28:19.712880772 +0000 UTC m=+9.567205834, 105.0166ms after stability
2025-01-25T14:28:19.713Z [INFO]  server-9faaba19-5697-0983-daae-aedf305f130d: updating configuration: command=AddVoter server-id=server-4d9f8495-f09e-c8e0-a12d-9f809ef17a48 server-addr=4d9f8495-f09e-c8e0-a12d-9f809ef17a48 servers="[{Suffrage:Voter ID:server-9faaba19-5697-0983-daae-aedf305f130d Address:9faaba19-5697-0983-daae-aedf305f130d} {Suffrage:Voter ID:server-7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef Address:7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef} {Suffrage:Voter ID:server-4d9f8495-f09e-c8e0-a12d-9f809ef17a48 Address:4d9f8495-f09e-c8e0-a12d-9f809ef17a48}]"
2025-01-25T14:28:19.714Z [INFO]  server-9faaba19-5697-0983-daae-aedf305f130d: added peer, starting replication: peer=server-4d9f8495-f09e-c8e0-a12d-9f809ef17a48
2025-01-25T14:28:19.714Z [WARN]  server-4d9f8495-f09e-c8e0-a12d-9f809ef17a48: failed to get previous log: previous-index=4 last-index=0 error="log not found"
2025-01-25T14:28:19.714Z [WARN]  server-9faaba19-5697-0983-daae-aedf305f130d: appendEntries rejected, sending older logs: peer="{Voter server-4d9f8495-f09e-c8e0-a12d-9f809ef17a48 4d9f8495-f09e-c8e0-a12d-9f809ef17a48}" next=1
2025-01-25T14:28:19.714Z [INFO]  server-9faaba19-5697-0983-daae-aedf305f130d: pipelining replication: peer="{Voter server-4d9f8495-f09e-c8e0-a12d-9f809ef17a48 4d9f8495-f09e-c8e0-a12d-9f809ef17a48}"
2025-01-25T14:28:19.714Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:19.820Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:19.714872801 +0000 UTC m=+9.569197783 (1 nodes), 1.24µs from start of poll, 295.272775ms from cluster start. Timeout at 2025-01-25 14:28:19.819915759 +0000 UTC m=+9.674240741, 105.042958ms after stability
2025-01-25T14:28:19.820Z [INFO]  server-9faaba19-5697-0983-daae-aedf305f130d: aborting pipeline replication: peer="{Voter server-7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef 7ece0a12-22fc-e71e-3ff5-2fb2f56d1aef}"
2025-01-25T14:28:19.820Z [INFO]  server-9faaba19-5697-0983-daae-aedf305f130d: aborting pipeline replication: peer="{Voter server-4d9f8495-f09e-c8e0-a12d-9f809ef17a48 4d9f8495-f09e-c8e0-a12d-9f809ef17a48}"
--- PASS: TestRaft_JoinNode_ConfigStore (0.40s)
=== RUN   TestRaft_RemoveFollower
2025-01-25T14:28:19.823Z [INFO]  server-63bd4c4e-e9a3-464e-93df-d626cdcef492: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63bd4c4e-e9a3-464e-93df-d626cdcef492 Address:63bd4c4e-e9a3-464e-93df-d626cdcef492} {Suffrage:Voter ID:server-e3a7e5b7-62c1-7793-de55-2969fd575b18 Address:e3a7e5b7-62c1-7793-de55-2969fd575b18} {Suffrage:Voter ID:server-65520df9-2c38-c96a-22f2-65d49ef3852d Address:65520df9-2c38-c96a-22f2-65d49ef3852d}]"
2025-01-25T14:28:19.824Z [INFO]  server-e3a7e5b7-62c1-7793-de55-2969fd575b18: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63bd4c4e-e9a3-464e-93df-d626cdcef492 Address:63bd4c4e-e9a3-464e-93df-d626cdcef492} {Suffrage:Voter ID:server-e3a7e5b7-62c1-7793-de55-2969fd575b18 Address:e3a7e5b7-62c1-7793-de55-2969fd575b18} {Suffrage:Voter ID:server-65520df9-2c38-c96a-22f2-65d49ef3852d Address:65520df9-2c38-c96a-22f2-65d49ef3852d}]"
2025-01-25T14:28:19.824Z [INFO]  server-65520df9-2c38-c96a-22f2-65d49ef3852d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63bd4c4e-e9a3-464e-93df-d626cdcef492 Address:63bd4c4e-e9a3-464e-93df-d626cdcef492} {Suffrage:Voter ID:server-e3a7e5b7-62c1-7793-de55-2969fd575b18 Address:e3a7e5b7-62c1-7793-de55-2969fd575b18} {Suffrage:Voter ID:server-65520df9-2c38-c96a-22f2-65d49ef3852d Address:65520df9-2c38-c96a-22f2-65d49ef3852d}]"
2025-01-25T14:28:19.824Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:19.824Z [INFO]  server-63bd4c4e-e9a3-464e-93df-d626cdcef492: entering follower state: follower="Node at 63bd4c4e-e9a3-464e-93df-d626cdcef492 [Follower]" leader-address= leader-id=
2025-01-25T14:28:19.824Z [INFO]  server-e3a7e5b7-62c1-7793-de55-2969fd575b18: entering follower state: follower="Node at e3a7e5b7-62c1-7793-de55-2969fd575b18 [Follower]" leader-address= leader-id=
2025-01-25T14:28:19.824Z [INFO]  server-65520df9-2c38-c96a-22f2-65d49ef3852d: entering follower state: follower="Node at 65520df9-2c38-c96a-22f2-65d49ef3852d [Follower]" leader-address= leader-id=
2025-01-25T14:28:19.895Z [WARN]  server-63bd4c4e-e9a3-464e-93df-d626cdcef492: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:19.895Z [WARN]  server-65520df9-2c38-c96a-22f2-65d49ef3852d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:19.895Z [INFO]  server-63bd4c4e-e9a3-464e-93df-d626cdcef492: entering candidate state: node="Node at 63bd4c4e-e9a3-464e-93df-d626cdcef492 [Candidate]" term=2
2025-01-25T14:28:19.895Z [INFO]  server-65520df9-2c38-c96a-22f2-65d49ef3852d: entering candidate state: node="Node at 65520df9-2c38-c96a-22f2-65d49ef3852d [Candidate]" term=2
2025-01-25T14:28:19.895Z [INFO]  server-65520df9-2c38-c96a-22f2-65d49ef3852d: duplicate requestVote for same term: term=2
2025-01-25T14:28:19.895Z [INFO]  server-63bd4c4e-e9a3-464e-93df-d626cdcef492: duplicate requestVote for same term: term=2
2025-01-25T14:28:19.895Z [INFO]  server-63bd4c4e-e9a3-464e-93df-d626cdcef492: election won: term=2 tally=2
2025-01-25T14:28:19.895Z [INFO]  server-63bd4c4e-e9a3-464e-93df-d626cdcef492: entering leader state: leader="Node at 63bd4c4e-e9a3-464e-93df-d626cdcef492 [Leader]"
2025-01-25T14:28:19.895Z [INFO]  server-e3a7e5b7-62c1-7793-de55-2969fd575b18: duplicate requestVote for same term: term=2
2025-01-25T14:28:19.895Z [INFO]  server-63bd4c4e-e9a3-464e-93df-d626cdcef492: added peer, starting replication: peer=server-e3a7e5b7-62c1-7793-de55-2969fd575b18
2025-01-25T14:28:19.895Z [INFO]  server-63bd4c4e-e9a3-464e-93df-d626cdcef492: added peer, starting replication: peer=server-65520df9-2c38-c96a-22f2-65d49ef3852d
2025-01-25T14:28:19.895Z [INFO]  server-65520df9-2c38-c96a-22f2-65d49ef3852d: entering follower state: follower="Node at 65520df9-2c38-c96a-22f2-65d49ef3852d [Follower]" leader-address=63bd4c4e-e9a3-464e-93df-d626cdcef492 leader-id=server-63bd4c4e-e9a3-464e-93df-d626cdcef492
2025-01-25T14:28:19.895Z [INFO]  server-63bd4c4e-e9a3-464e-93df-d626cdcef492: pipelining replication: peer="{Voter server-e3a7e5b7-62c1-7793-de55-2969fd575b18 e3a7e5b7-62c1-7793-de55-2969fd575b18}"
2025-01-25T14:28:19.896Z [INFO]  server-63bd4c4e-e9a3-464e-93df-d626cdcef492: pipelining replication: peer="{Voter server-65520df9-2c38-c96a-22f2-65d49ef3852d 65520df9-2c38-c96a-22f2-65d49ef3852d}"
2025-01-25T14:28:20.002Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:19.896077361 +0000 UTC m=+9.750402343 (1 nodes), 71.552066ms from start of poll, 72.390435ms from cluster start. Timeout at 2025-01-25 14:28:20.001080801 +0000 UTC m=+9.855405943, 105.0036ms after stability
2025-01-25T14:28:20.007Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:20.113Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:20.007846099 +0000 UTC m=+9.862171081 (2 nodes), 4.04µs from start of poll, 184.159173ms from cluster start. Timeout at 2025-01-25 14:28:20.112877138 +0000 UTC m=+9.967202240, 105.031159ms after stability
2025-01-25T14:28:20.114Z [INFO]  server-63bd4c4e-e9a3-464e-93df-d626cdcef492: updating configuration: command=RemoveServer server-id=server-e3a7e5b7-62c1-7793-de55-2969fd575b18 server-addr= servers="[{Suffrage:Voter ID:server-63bd4c4e-e9a3-464e-93df-d626cdcef492 Address:63bd4c4e-e9a3-464e-93df-d626cdcef492} {Suffrage:Voter ID:server-65520df9-2c38-c96a-22f2-65d49ef3852d Address:65520df9-2c38-c96a-22f2-65d49ef3852d}]"
2025-01-25T14:28:20.114Z [INFO]  server-63bd4c4e-e9a3-464e-93df-d626cdcef492: removed peer, stopping replication: peer=server-e3a7e5b7-62c1-7793-de55-2969fd575b18 last-index=3
2025-01-25T14:28:20.114Z [INFO]  server-63bd4c4e-e9a3-464e-93df-d626cdcef492: aborting pipeline replication: peer="{Voter server-e3a7e5b7-62c1-7793-de55-2969fd575b18 e3a7e5b7-62c1-7793-de55-2969fd575b18}"
2025-01-25T14:28:20.212Z [WARN]  server-e3a7e5b7-62c1-7793-de55-2969fd575b18: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-01-25T14:28:20.220Z [INFO]  server-63bd4c4e-e9a3-464e-93df-d626cdcef492: aborting pipeline replication: peer="{Voter server-65520df9-2c38-c96a-22f2-65d49ef3852d 65520df9-2c38-c96a-22f2-65d49ef3852d}"
--- PASS: TestRaft_RemoveFollower (0.40s)
=== RUN   TestRaft_RemoveLeader
2025-01-25T14:28:20.223Z [INFO]  server-7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365 Address:7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365} {Suffrage:Voter ID:server-e4d9ce19-7e7e-5748-7c0d-00213f306dee Address:e4d9ce19-7e7e-5748-7c0d-00213f306dee} {Suffrage:Voter ID:server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e Address:f7b71b2a-f14d-846b-0f6c-095d91c0fd8e}]"
2025-01-25T14:28:20.223Z [INFO]  server-e4d9ce19-7e7e-5748-7c0d-00213f306dee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365 Address:7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365} {Suffrage:Voter ID:server-e4d9ce19-7e7e-5748-7c0d-00213f306dee Address:e4d9ce19-7e7e-5748-7c0d-00213f306dee} {Suffrage:Voter ID:server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e Address:f7b71b2a-f14d-846b-0f6c-095d91c0fd8e}]"
2025-01-25T14:28:20.223Z [INFO]  server-7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365: entering follower state: follower="Node at 7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365 [Follower]" leader-address= leader-id=
2025-01-25T14:28:20.223Z [INFO]  server-e4d9ce19-7e7e-5748-7c0d-00213f306dee: entering follower state: follower="Node at e4d9ce19-7e7e-5748-7c0d-00213f306dee [Follower]" leader-address= leader-id=
2025-01-25T14:28:20.224Z [INFO]  server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365 Address:7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365} {Suffrage:Voter ID:server-e4d9ce19-7e7e-5748-7c0d-00213f306dee Address:e4d9ce19-7e7e-5748-7c0d-00213f306dee} {Suffrage:Voter ID:server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e Address:f7b71b2a-f14d-846b-0f6c-095d91c0fd8e}]"
2025-01-25T14:28:20.224Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:20.224Z [INFO]  server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e: entering follower state: follower="Node at f7b71b2a-f14d-846b-0f6c-095d91c0fd8e [Follower]" leader-address= leader-id=
2025-01-25T14:28:20.300Z [WARN]  server-e4d9ce19-7e7e-5748-7c0d-00213f306dee: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:20.300Z [INFO]  server-e4d9ce19-7e7e-5748-7c0d-00213f306dee: entering candidate state: node="Node at e4d9ce19-7e7e-5748-7c0d-00213f306dee [Candidate]" term=2
2025-01-25T14:28:20.300Z [INFO]  server-e4d9ce19-7e7e-5748-7c0d-00213f306dee: election won: term=2 tally=2
2025-01-25T14:28:20.300Z [INFO]  server-e4d9ce19-7e7e-5748-7c0d-00213f306dee: entering leader state: leader="Node at e4d9ce19-7e7e-5748-7c0d-00213f306dee [Leader]"
2025-01-25T14:28:20.300Z [INFO]  server-e4d9ce19-7e7e-5748-7c0d-00213f306dee: added peer, starting replication: peer=server-7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365
2025-01-25T14:28:20.300Z [INFO]  server-e4d9ce19-7e7e-5748-7c0d-00213f306dee: added peer, starting replication: peer=server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e
2025-01-25T14:28:20.301Z [INFO]  server-e4d9ce19-7e7e-5748-7c0d-00213f306dee: pipelining replication: peer="{Voter server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e f7b71b2a-f14d-846b-0f6c-095d91c0fd8e}"
2025-01-25T14:28:20.301Z [INFO]  server-e4d9ce19-7e7e-5748-7c0d-00213f306dee: pipelining replication: peer="{Voter server-7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365 7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365}"
2025-01-25T14:28:20.407Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:20.301046348 +0000 UTC m=+10.155371330 (1 nodes), 76.881873ms from start of poll, 77.668085ms from cluster start. Timeout at 2025-01-25 14:28:20.406050508 +0000 UTC m=+10.260375490, 105.00416ms after stability
2025-01-25T14:28:20.413Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:20.518Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:20.413086597 +0000 UTC m=+10.267411579 (2 nodes), 2.24µs from start of poll, 189.708334ms from cluster start. Timeout at 2025-01-25 14:28:20.518138756 +0000 UTC m=+10.372463738, 105.052159ms after stability
2025-01-25T14:28:20.518Z [INFO]  server-e4d9ce19-7e7e-5748-7c0d-00213f306dee: updating configuration: command=RemoveServer server-id=server-e4d9ce19-7e7e-5748-7c0d-00213f306dee server-addr= servers="[{Suffrage:Voter ID:server-7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365 Address:7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365} {Suffrage:Voter ID:server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e Address:f7b71b2a-f14d-846b-0f6c-095d91c0fd8e}]"
2025-01-25T14:28:20.518Z [INFO]  server-e4d9ce19-7e7e-5748-7c0d-00213f306dee: removed ourself, shutting down
2025-01-25T14:28:20.518Z [INFO]  server-e4d9ce19-7e7e-5748-7c0d-00213f306dee: aborting pipeline replication: peer="{Voter server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e f7b71b2a-f14d-846b-0f6c-095d91c0fd8e}"
2025-01-25T14:28:20.518Z [INFO]  server-e4d9ce19-7e7e-5748-7c0d-00213f306dee: aborting pipeline replication: peer="{Voter server-7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365 7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365}"
2025-01-25T14:28:20.569Z [WARN]  server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e: heartbeat timeout reached, starting election: last-leader-addr=e4d9ce19-7e7e-5748-7c0d-00213f306dee last-leader-id=server-e4d9ce19-7e7e-5748-7c0d-00213f306dee
2025-01-25T14:28:20.569Z [INFO]  server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e: entering candidate state: node="Node at f7b71b2a-f14d-846b-0f6c-095d91c0fd8e [Candidate]" term=3
2025-01-25T14:28:20.569Z [WARN]  server-7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365: rejecting vote request since we have a leader: from=f7b71b2a-f14d-846b-0f6c-095d91c0fd8e leader=e4d9ce19-7e7e-5748-7c0d-00213f306dee leader-id=server-e4d9ce19-7e7e-5748-7c0d-00213f306dee
2025-01-25T14:28:20.623Z [WARN]  server-7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365: heartbeat timeout reached, starting election: last-leader-addr=e4d9ce19-7e7e-5748-7c0d-00213f306dee last-leader-id=server-e4d9ce19-7e7e-5748-7c0d-00213f306dee
2025-01-25T14:28:20.623Z [INFO]  server-7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365: entering candidate state: node="Node at 7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365 [Candidate]" term=3
2025-01-25T14:28:20.623Z [INFO]  server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e: duplicate requestVote for same term: term=3
2025-01-25T14:28:20.630Z [WARN]  server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e: Election timeout reached, restarting election
2025-01-25T14:28:20.630Z [INFO]  server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e: entering candidate state: node="Node at f7b71b2a-f14d-846b-0f6c-095d91c0fd8e [Candidate]" term=4
2025-01-25T14:28:20.630Z [INFO]  server-7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365: entering follower state: follower="Node at 7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365 [Follower]" leader-address= leader-id=
2025-01-25T14:28:20.630Z [INFO]  server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e: election won: term=4 tally=2
2025-01-25T14:28:20.630Z [INFO]  server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e: entering leader state: leader="Node at f7b71b2a-f14d-846b-0f6c-095d91c0fd8e [Leader]"
2025-01-25T14:28:20.630Z [INFO]  server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e: added peer, starting replication: peer=server-7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365
2025-01-25T14:28:20.630Z [INFO]  server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e: pipelining replication: peer="{Voter server-7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365 7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365}"
2025-01-25T14:28:20.729Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:20.836Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:20.729958714 +0000 UTC m=+10.584283656 (1 nodes), 126.195µs from start of poll, 506.580411ms from cluster start. Timeout at 2025-01-25 14:28:20.834994193 +0000 UTC m=+10.689319215, 105.035559ms after stability
2025-01-25T14:28:20.836Z [INFO]  server-f7b71b2a-f14d-846b-0f6c-095d91c0fd8e: aborting pipeline replication: peer="{Voter server-7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365 7ca8c4ed-46f5-f629-d8ea-3bc9f04a0365}"
--- PASS: TestRaft_RemoveLeader (0.62s)
=== RUN   TestRaft_RemoveLeader_NoShutdown
2025-01-25T14:28:20.838Z [INFO]  server-a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4 Address:a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4} {Suffrage:Voter ID:server-ac52c234-61a7-d637-6af5-5c84a6d32a15 Address:ac52c234-61a7-d637-6af5-5c84a6d32a15} {Suffrage:Voter ID:server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995 Address:17c25c52-0283-6ee2-e7cd-1bcd7ba2a995}]"
2025-01-25T14:28:20.838Z [INFO]  server-ac52c234-61a7-d637-6af5-5c84a6d32a15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4 Address:a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4} {Suffrage:Voter ID:server-ac52c234-61a7-d637-6af5-5c84a6d32a15 Address:ac52c234-61a7-d637-6af5-5c84a6d32a15} {Suffrage:Voter ID:server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995 Address:17c25c52-0283-6ee2-e7cd-1bcd7ba2a995}]"
2025-01-25T14:28:20.839Z [INFO]  server-a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4: entering follower state: follower="Node at a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4 [Follower]" leader-address= leader-id=
2025-01-25T14:28:20.839Z [INFO]  server-ac52c234-61a7-d637-6af5-5c84a6d32a15: entering follower state: follower="Node at ac52c234-61a7-d637-6af5-5c84a6d32a15 [Follower]" leader-address= leader-id=
2025-01-25T14:28:20.839Z [INFO]  server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4 Address:a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4} {Suffrage:Voter ID:server-ac52c234-61a7-d637-6af5-5c84a6d32a15 Address:ac52c234-61a7-d637-6af5-5c84a6d32a15} {Suffrage:Voter ID:server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995 Address:17c25c52-0283-6ee2-e7cd-1bcd7ba2a995}]"
2025-01-25T14:28:20.839Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:20.839Z [INFO]  server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995: entering follower state: follower="Node at 17c25c52-0283-6ee2-e7cd-1bcd7ba2a995 [Follower]" leader-address= leader-id=
2025-01-25T14:28:20.897Z [WARN]  server-ac52c234-61a7-d637-6af5-5c84a6d32a15: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:20.897Z [INFO]  server-ac52c234-61a7-d637-6af5-5c84a6d32a15: entering candidate state: node="Node at ac52c234-61a7-d637-6af5-5c84a6d32a15 [Candidate]" term=2
2025-01-25T14:28:20.897Z [INFO]  server-ac52c234-61a7-d637-6af5-5c84a6d32a15: election won: term=2 tally=2
2025-01-25T14:28:20.897Z [INFO]  server-ac52c234-61a7-d637-6af5-5c84a6d32a15: entering leader state: leader="Node at ac52c234-61a7-d637-6af5-5c84a6d32a15 [Leader]"
2025-01-25T14:28:20.897Z [INFO]  server-ac52c234-61a7-d637-6af5-5c84a6d32a15: added peer, starting replication: peer=server-a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4
2025-01-25T14:28:20.897Z [INFO]  server-ac52c234-61a7-d637-6af5-5c84a6d32a15: added peer, starting replication: peer=server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995
2025-01-25T14:28:20.897Z [INFO]  server-ac52c234-61a7-d637-6af5-5c84a6d32a15: pipelining replication: peer="{Voter server-a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4 a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4}"
2025-01-25T14:28:20.897Z [INFO]  server-ac52c234-61a7-d637-6af5-5c84a6d32a15: pipelining replication: peer="{Voter server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995 17c25c52-0283-6ee2-e7cd-1bcd7ba2a995}"
2025-01-25T14:28:21.002Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:20.897561523 +0000 UTC m=+10.751886505 (2 nodes), 58.139428ms from start of poll, 59.117513ms from cluster start. Timeout at 2025-01-25 14:28:21.002574322 +0000 UTC m=+10.856899264, 105.012759ms after stability
2025-01-25T14:28:21.003Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:21.108Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:21.003051887 +0000 UTC m=+10.857376869 (1 nodes), 1.52µs from start of poll, 164.607877ms from cluster start. Timeout at 2025-01-25 14:28:21.108079926 +0000 UTC m=+10.962404868, 105.027999ms after stability
2025-01-25T14:28:21.117Z [INFO]  server-ac52c234-61a7-d637-6af5-5c84a6d32a15: updating configuration: command=RemoveServer server-id=server-ac52c234-61a7-d637-6af5-5c84a6d32a15 server-addr= servers="[{Suffrage:Voter ID:server-a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4 Address:a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4} {Suffrage:Voter ID:server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995 Address:17c25c52-0283-6ee2-e7cd-1bcd7ba2a995}]"
2025-01-25T14:28:21.117Z [INFO]  server-ac52c234-61a7-d637-6af5-5c84a6d32a15: removed ourself, transitioning to follower
2025-01-25T14:28:21.117Z [INFO]  server-ac52c234-61a7-d637-6af5-5c84a6d32a15: entering follower state: follower="Node at ac52c234-61a7-d637-6af5-5c84a6d32a15 [Follower]" leader-address= leader-id=
2025-01-25T14:28:21.117Z [INFO]  server-ac52c234-61a7-d637-6af5-5c84a6d32a15: aborting pipeline replication: peer="{Voter server-a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4 a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4}"
2025-01-25T14:28:21.117Z [INFO]  server-ac52c234-61a7-d637-6af5-5c84a6d32a15: aborting pipeline replication: peer="{Voter server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995 17c25c52-0283-6ee2-e7cd-1bcd7ba2a995}"
2025-01-25T14:28:21.202Z [WARN]  server-ac52c234-61a7-d637-6af5-5c84a6d32a15: not part of stable configuration, aborting election
2025-01-25T14:28:21.212Z [WARN]  server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995: heartbeat timeout reached, starting election: last-leader-addr=ac52c234-61a7-d637-6af5-5c84a6d32a15 last-leader-id=server-ac52c234-61a7-d637-6af5-5c84a6d32a15
2025-01-25T14:28:21.212Z [INFO]  server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995: entering candidate state: node="Node at 17c25c52-0283-6ee2-e7cd-1bcd7ba2a995 [Candidate]" term=3
2025-01-25T14:28:21.212Z [WARN]  server-a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4: rejecting vote request since we have a leader: from=17c25c52-0283-6ee2-e7cd-1bcd7ba2a995 leader=ac52c234-61a7-d637-6af5-5c84a6d32a15 leader-id=server-ac52c234-61a7-d637-6af5-5c84a6d32a15
2025-01-25T14:28:21.215Z [WARN]  server-a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4: heartbeat timeout reached, starting election: last-leader-addr=ac52c234-61a7-d637-6af5-5c84a6d32a15 last-leader-id=server-ac52c234-61a7-d637-6af5-5c84a6d32a15
2025-01-25T14:28:21.215Z [INFO]  server-a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4: entering candidate state: node="Node at a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4 [Candidate]" term=3
2025-01-25T14:28:21.215Z [INFO]  server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995: duplicate requestVote for same term: term=3
2025-01-25T14:28:21.224Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:21.280Z [WARN]  server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995: Election timeout reached, restarting election
2025-01-25T14:28:21.280Z [INFO]  server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995: entering candidate state: node="Node at 17c25c52-0283-6ee2-e7cd-1bcd7ba2a995 [Candidate]" term=4
2025-01-25T14:28:21.280Z [INFO]  server-a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4: entering follower state: follower="Node at a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4 [Follower]" leader-address= leader-id=
2025-01-25T14:28:21.280Z [INFO]  server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995: election won: term=4 tally=2
2025-01-25T14:28:21.280Z [INFO]  server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995: entering leader state: leader="Node at 17c25c52-0283-6ee2-e7cd-1bcd7ba2a995 [Leader]"
2025-01-25T14:28:21.280Z [INFO]  server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995: added peer, starting replication: peer=server-a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4
2025-01-25T14:28:21.280Z [INFO]  server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995: pipelining replication: peer="{Voter server-a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4 a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4}"
2025-01-25T14:28:21.386Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:21.281004485 +0000 UTC m=+11.135329467 (1 nodes), 56.923871ms from start of poll, 442.560475ms from cluster start. Timeout at 2025-01-25 14:28:21.386008325 +0000 UTC m=+11.240333307, 105.00384ms after stability
2025-01-25T14:28:21.492Z [INFO]  server-17c25c52-0283-6ee2-e7cd-1bcd7ba2a995: aborting pipeline replication: peer="{Voter server-a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4 a0ef52de-67d2-e87f-ecf0-3a477bbdc7c4}"
--- PASS: TestRaft_RemoveLeader_NoShutdown (0.66s)
=== RUN   TestRaft_RemoveFollower_SplitCluster
2025-01-25T14:28:21.495Z [INFO]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6c58e15-48df-3152-f01b-b31b19b7aa67 Address:b6c58e15-48df-3152-f01b-b31b19b7aa67} {Suffrage:Voter ID:server-fb7430f8-6ee3-1406-dbe8-c946d0093ef2 Address:fb7430f8-6ee3-1406-dbe8-c946d0093ef2} {Suffrage:Voter ID:server-d8b4de19-dab6-d624-44b2-3032a71b6029 Address:d8b4de19-dab6-d624-44b2-3032a71b6029} {Suffrage:Voter ID:server-3ea6bb86-68ac-01f0-b227-2995e4ea1456 Address:3ea6bb86-68ac-01f0-b227-2995e4ea1456}]"
2025-01-25T14:28:21.495Z [INFO]  server-fb7430f8-6ee3-1406-dbe8-c946d0093ef2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6c58e15-48df-3152-f01b-b31b19b7aa67 Address:b6c58e15-48df-3152-f01b-b31b19b7aa67} {Suffrage:Voter ID:server-fb7430f8-6ee3-1406-dbe8-c946d0093ef2 Address:fb7430f8-6ee3-1406-dbe8-c946d0093ef2} {Suffrage:Voter ID:server-d8b4de19-dab6-d624-44b2-3032a71b6029 Address:d8b4de19-dab6-d624-44b2-3032a71b6029} {Suffrage:Voter ID:server-3ea6bb86-68ac-01f0-b227-2995e4ea1456 Address:3ea6bb86-68ac-01f0-b227-2995e4ea1456}]"
2025-01-25T14:28:21.495Z [INFO]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: entering follower state: follower="Node at b6c58e15-48df-3152-f01b-b31b19b7aa67 [Follower]" leader-address= leader-id=
2025-01-25T14:28:21.495Z [INFO]  server-d8b4de19-dab6-d624-44b2-3032a71b6029: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6c58e15-48df-3152-f01b-b31b19b7aa67 Address:b6c58e15-48df-3152-f01b-b31b19b7aa67} {Suffrage:Voter ID:server-fb7430f8-6ee3-1406-dbe8-c946d0093ef2 Address:fb7430f8-6ee3-1406-dbe8-c946d0093ef2} {Suffrage:Voter ID:server-d8b4de19-dab6-d624-44b2-3032a71b6029 Address:d8b4de19-dab6-d624-44b2-3032a71b6029} {Suffrage:Voter ID:server-3ea6bb86-68ac-01f0-b227-2995e4ea1456 Address:3ea6bb86-68ac-01f0-b227-2995e4ea1456}]"
2025-01-25T14:28:21.495Z [INFO]  server-fb7430f8-6ee3-1406-dbe8-c946d0093ef2: entering follower state: follower="Node at fb7430f8-6ee3-1406-dbe8-c946d0093ef2 [Follower]" leader-address= leader-id=
2025-01-25T14:28:21.495Z [INFO]  server-d8b4de19-dab6-d624-44b2-3032a71b6029: entering follower state: follower="Node at d8b4de19-dab6-d624-44b2-3032a71b6029 [Follower]" leader-address= leader-id=
2025-01-25T14:28:21.495Z [INFO]  server-3ea6bb86-68ac-01f0-b227-2995e4ea1456: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6c58e15-48df-3152-f01b-b31b19b7aa67 Address:b6c58e15-48df-3152-f01b-b31b19b7aa67} {Suffrage:Voter ID:server-fb7430f8-6ee3-1406-dbe8-c946d0093ef2 Address:fb7430f8-6ee3-1406-dbe8-c946d0093ef2} {Suffrage:Voter ID:server-d8b4de19-dab6-d624-44b2-3032a71b6029 Address:d8b4de19-dab6-d624-44b2-3032a71b6029} {Suffrage:Voter ID:server-3ea6bb86-68ac-01f0-b227-2995e4ea1456 Address:3ea6bb86-68ac-01f0-b227-2995e4ea1456}]"
2025-01-25T14:28:21.495Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:21.495Z [INFO]  server-3ea6bb86-68ac-01f0-b227-2995e4ea1456: entering follower state: follower="Node at 3ea6bb86-68ac-01f0-b227-2995e4ea1456 [Follower]" leader-address= leader-id=
2025-01-25T14:28:21.548Z [WARN]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:21.548Z [INFO]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: entering candidate state: node="Node at b6c58e15-48df-3152-f01b-b31b19b7aa67 [Candidate]" term=2
2025-01-25T14:28:21.548Z [INFO]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: election won: term=2 tally=3
2025-01-25T14:28:21.548Z [INFO]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: entering leader state: leader="Node at b6c58e15-48df-3152-f01b-b31b19b7aa67 [Leader]"
2025-01-25T14:28:21.548Z [INFO]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: added peer, starting replication: peer=server-fb7430f8-6ee3-1406-dbe8-c946d0093ef2
2025-01-25T14:28:21.548Z [INFO]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: added peer, starting replication: peer=server-d8b4de19-dab6-d624-44b2-3032a71b6029
2025-01-25T14:28:21.548Z [INFO]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: added peer, starting replication: peer=server-3ea6bb86-68ac-01f0-b227-2995e4ea1456
2025-01-25T14:28:21.549Z [INFO]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: pipelining replication: peer="{Voter server-3ea6bb86-68ac-01f0-b227-2995e4ea1456 3ea6bb86-68ac-01f0-b227-2995e4ea1456}"
2025-01-25T14:28:21.549Z [INFO]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: pipelining replication: peer="{Voter server-fb7430f8-6ee3-1406-dbe8-c946d0093ef2 fb7430f8-6ee3-1406-dbe8-c946d0093ef2}"
2025-01-25T14:28:21.549Z [INFO]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: pipelining replication: peer="{Voter server-d8b4de19-dab6-d624-44b2-3032a71b6029 d8b4de19-dab6-d624-44b2-3032a71b6029}"
2025-01-25T14:28:21.654Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:21.548977522 +0000 UTC m=+11.403302504 (1 nodes), 53.006212ms from start of poll, 53.963298ms from cluster start. Timeout at 2025-01-25 14:28:21.653982401 +0000 UTC m=+11.508307383, 105.004879ms after stability
2025-01-25T14:28:21.761Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:21.866Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:21.76108341 +0000 UTC m=+11.615408392 (3 nodes), 1.84µs from start of poll, 266.069186ms from cluster start. Timeout at 2025-01-25 14:28:21.866121649 +0000 UTC m=+11.720446591, 105.038199ms after stability
2025-01-25T14:28:21.866Z [INFO]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: updating configuration: command=RemoveServer server-id=server-3ea6bb86-68ac-01f0-b227-2995e4ea1456 server-addr= servers="[{Suffrage:Voter ID:server-b6c58e15-48df-3152-f01b-b31b19b7aa67 Address:b6c58e15-48df-3152-f01b-b31b19b7aa67} {Suffrage:Voter ID:server-fb7430f8-6ee3-1406-dbe8-c946d0093ef2 Address:fb7430f8-6ee3-1406-dbe8-c946d0093ef2} {Suffrage:Voter ID:server-d8b4de19-dab6-d624-44b2-3032a71b6029 Address:d8b4de19-dab6-d624-44b2-3032a71b6029}]"
2025-01-25T14:28:21.866Z [ERROR] server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to heartbeat to: peer=fb7430f8-6ee3-1406-dbe8-c946d0093ef2 backoff time=10ms error="failed to connect to peer: fb7430f8-6ee3-1406-dbe8-c946d0093ef2"
2025-01-25T14:28:21.866Z [ERROR] server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to pipeline appendEntries: peer="{Voter server-fb7430f8-6ee3-1406-dbe8-c946d0093ef2 fb7430f8-6ee3-1406-dbe8-c946d0093ef2}" error="append pipeline closed"
2025-01-25T14:28:21.866Z [INFO]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: aborting pipeline replication: peer="{Voter server-fb7430f8-6ee3-1406-dbe8-c946d0093ef2 fb7430f8-6ee3-1406-dbe8-c946d0093ef2}"
2025-01-25T14:28:21.866Z [INFO]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: removed peer, stopping replication: peer=server-3ea6bb86-68ac-01f0-b227-2995e4ea1456 last-index=3
2025-01-25T14:28:21.866Z [ERROR] server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to pipeline appendEntries: peer="{Voter server-d8b4de19-dab6-d624-44b2-3032a71b6029 d8b4de19-dab6-d624-44b2-3032a71b6029}" error="append pipeline closed"
2025-01-25T14:28:21.866Z [INFO]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: aborting pipeline replication: peer="{Voter server-d8b4de19-dab6-d624-44b2-3032a71b6029 d8b4de19-dab6-d624-44b2-3032a71b6029}"
2025-01-25T14:28:21.866Z [INFO]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: aborting pipeline replication: peer="{Voter server-3ea6bb86-68ac-01f0-b227-2995e4ea1456 3ea6bb86-68ac-01f0-b227-2995e4ea1456}"
2025-01-25T14:28:21.872Z [ERROR] server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to appendEntries to: peer="{Voter server-fb7430f8-6ee3-1406-dbe8-c946d0093ef2 fb7430f8-6ee3-1406-dbe8-c946d0093ef2}" error="failed to connect to peer: fb7430f8-6ee3-1406-dbe8-c946d0093ef2"
2025-01-25T14:28:21.879Z [ERROR] server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to heartbeat to: peer=d8b4de19-dab6-d624-44b2-3032a71b6029 backoff time=10ms error="failed to connect to peer: d8b4de19-dab6-d624-44b2-3032a71b6029"
2025-01-25T14:28:21.879Z [ERROR] server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to appendEntries to: peer="{Voter server-d8b4de19-dab6-d624-44b2-3032a71b6029 d8b4de19-dab6-d624-44b2-3032a71b6029}" error="failed to connect to peer: d8b4de19-dab6-d624-44b2-3032a71b6029"
2025-01-25T14:28:21.885Z [ERROR] server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to heartbeat to: peer=fb7430f8-6ee3-1406-dbe8-c946d0093ef2 backoff time=10ms error="failed to connect to peer: fb7430f8-6ee3-1406-dbe8-c946d0093ef2"
2025-01-25T14:28:21.893Z [ERROR] server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to appendEntries to: peer="{Voter server-fb7430f8-6ee3-1406-dbe8-c946d0093ef2 fb7430f8-6ee3-1406-dbe8-c946d0093ef2}" error="failed to connect to peer: fb7430f8-6ee3-1406-dbe8-c946d0093ef2"
2025-01-25T14:28:21.897Z [ERROR] server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to heartbeat to: peer=d8b4de19-dab6-d624-44b2-3032a71b6029 backoff time=10ms error="failed to connect to peer: d8b4de19-dab6-d624-44b2-3032a71b6029"
2025-01-25T14:28:21.897Z [ERROR] server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to appendEntries to: peer="{Voter server-d8b4de19-dab6-d624-44b2-3032a71b6029 d8b4de19-dab6-d624-44b2-3032a71b6029}" error="failed to connect to peer: d8b4de19-dab6-d624-44b2-3032a71b6029"
2025-01-25T14:28:21.904Z [ERROR] server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to heartbeat to: peer=fb7430f8-6ee3-1406-dbe8-c946d0093ef2 backoff time=10ms error="failed to connect to peer: fb7430f8-6ee3-1406-dbe8-c946d0093ef2"
2025-01-25T14:28:21.908Z [WARN]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to contact: server-id=server-fb7430f8-6ee3-1406-dbe8-c946d0093ef2 time=50.109157ms
2025-01-25T14:28:21.912Z [ERROR] server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to appendEntries to: peer="{Voter server-fb7430f8-6ee3-1406-dbe8-c946d0093ef2 fb7430f8-6ee3-1406-dbe8-c946d0093ef2}" error="failed to connect to peer: fb7430f8-6ee3-1406-dbe8-c946d0093ef2"
2025-01-25T14:28:21.915Z [ERROR] server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to heartbeat to: peer=d8b4de19-dab6-d624-44b2-3032a71b6029 backoff time=10ms error="failed to connect to peer: d8b4de19-dab6-d624-44b2-3032a71b6029"
2025-01-25T14:28:21.916Z [ERROR] server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to appendEntries to: peer="{Voter server-d8b4de19-dab6-d624-44b2-3032a71b6029 d8b4de19-dab6-d624-44b2-3032a71b6029}" error="failed to connect to peer: d8b4de19-dab6-d624-44b2-3032a71b6029"
2025-01-25T14:28:21.918Z [WARN]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to contact: server-id=server-fb7430f8-6ee3-1406-dbe8-c946d0093ef2 time=60.376027ms
2025-01-25T14:28:21.918Z [WARN]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to contact: server-id=server-d8b4de19-dab6-d624-44b2-3032a71b6029 time=52.440472ms
2025-01-25T14:28:21.918Z [WARN]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: failed to contact quorum of nodes, stepping down
2025-01-25T14:28:21.918Z [INFO]  server-b6c58e15-48df-3152-f01b-b31b19b7aa67: entering follower state: follower="Node at b6c58e15-48df-3152-f01b-b31b19b7aa67 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_RemoveFollower_SplitCluster (0.43s)
=== RUN   TestRaft_AddKnownPeer
2025-01-25T14:28:21.921Z [INFO]  server-90b306c5-c416-2a48-4f30-74063a43b43e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90b306c5-c416-2a48-4f30-74063a43b43e Address:90b306c5-c416-2a48-4f30-74063a43b43e} {Suffrage:Voter ID:server-8c5280e2-4471-9752-12ba-76c0ba9daf7f Address:8c5280e2-4471-9752-12ba-76c0ba9daf7f} {Suffrage:Voter ID:server-7b6b3118-69a9-ec91-912a-adcd732057cd Address:7b6b3118-69a9-ec91-912a-adcd732057cd}]"
2025-01-25T14:28:21.921Z [INFO]  server-90b306c5-c416-2a48-4f30-74063a43b43e: entering follower state: follower="Node at 90b306c5-c416-2a48-4f30-74063a43b43e [Follower]" leader-address= leader-id=
2025-01-25T14:28:21.921Z [INFO]  server-8c5280e2-4471-9752-12ba-76c0ba9daf7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90b306c5-c416-2a48-4f30-74063a43b43e Address:90b306c5-c416-2a48-4f30-74063a43b43e} {Suffrage:Voter ID:server-8c5280e2-4471-9752-12ba-76c0ba9daf7f Address:8c5280e2-4471-9752-12ba-76c0ba9daf7f} {Suffrage:Voter ID:server-7b6b3118-69a9-ec91-912a-adcd732057cd Address:7b6b3118-69a9-ec91-912a-adcd732057cd}]"
2025-01-25T14:28:21.922Z [INFO]  server-8c5280e2-4471-9752-12ba-76c0ba9daf7f: entering follower state: follower="Node at 8c5280e2-4471-9752-12ba-76c0ba9daf7f [Follower]" leader-address= leader-id=
2025-01-25T14:28:21.922Z [INFO]  server-7b6b3118-69a9-ec91-912a-adcd732057cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90b306c5-c416-2a48-4f30-74063a43b43e Address:90b306c5-c416-2a48-4f30-74063a43b43e} {Suffrage:Voter ID:server-8c5280e2-4471-9752-12ba-76c0ba9daf7f Address:8c5280e2-4471-9752-12ba-76c0ba9daf7f} {Suffrage:Voter ID:server-7b6b3118-69a9-ec91-912a-adcd732057cd Address:7b6b3118-69a9-ec91-912a-adcd732057cd}]"
2025-01-25T14:28:21.922Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:21.922Z [INFO]  server-7b6b3118-69a9-ec91-912a-adcd732057cd: entering follower state: follower="Node at 7b6b3118-69a9-ec91-912a-adcd732057cd [Follower]" leader-address= leader-id=
2025-01-25T14:28:21.989Z [WARN]  server-8c5280e2-4471-9752-12ba-76c0ba9daf7f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:21.990Z [INFO]  server-8c5280e2-4471-9752-12ba-76c0ba9daf7f: entering candidate state: node="Node at 8c5280e2-4471-9752-12ba-76c0ba9daf7f [Candidate]" term=2
2025-01-25T14:28:21.990Z [INFO]  server-8c5280e2-4471-9752-12ba-76c0ba9daf7f: election won: term=2 tally=2
2025-01-25T14:28:21.990Z [INFO]  server-8c5280e2-4471-9752-12ba-76c0ba9daf7f: entering leader state: leader="Node at 8c5280e2-4471-9752-12ba-76c0ba9daf7f [Leader]"
2025-01-25T14:28:21.990Z [INFO]  server-8c5280e2-4471-9752-12ba-76c0ba9daf7f: added peer, starting replication: peer=server-90b306c5-c416-2a48-4f30-74063a43b43e
2025-01-25T14:28:21.990Z [INFO]  server-8c5280e2-4471-9752-12ba-76c0ba9daf7f: added peer, starting replication: peer=server-7b6b3118-69a9-ec91-912a-adcd732057cd
2025-01-25T14:28:21.990Z [INFO]  server-8c5280e2-4471-9752-12ba-76c0ba9daf7f: pipelining replication: peer="{Voter server-90b306c5-c416-2a48-4f30-74063a43b43e 90b306c5-c416-2a48-4f30-74063a43b43e}"
2025-01-25T14:28:21.990Z [INFO]  server-8c5280e2-4471-9752-12ba-76c0ba9daf7f: pipelining replication: peer="{Voter server-7b6b3118-69a9-ec91-912a-adcd732057cd 7b6b3118-69a9-ec91-912a-adcd732057cd}"
2025-01-25T14:28:22.095Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:21.990438077 +0000 UTC m=+11.844763059 (1 nodes), 68.186186ms from start of poll, 68.91756ms from cluster start. Timeout at 2025-01-25 14:28:22.095443277 +0000 UTC m=+11.949768259, 105.0052ms after stability
2025-01-25T14:28:22.096Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:22.201Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:22.096095435 +0000 UTC m=+11.950420417 (2 nodes), 2.08µs from start of poll, 174.574918ms from cluster start. Timeout at 2025-01-25 14:28:22.201149273 +0000 UTC m=+12.055474215, 105.053798ms after stability
2025-01-25T14:28:22.201Z [INFO]  server-8c5280e2-4471-9752-12ba-76c0ba9daf7f: updating configuration: command=AddVoter server-id=server-90b306c5-c416-2a48-4f30-74063a43b43e server-addr=90b306c5-c416-2a48-4f30-74063a43b43e servers="[{Suffrage:Voter ID:server-90b306c5-c416-2a48-4f30-74063a43b43e Address:90b306c5-c416-2a48-4f30-74063a43b43e} {Suffrage:Voter ID:server-8c5280e2-4471-9752-12ba-76c0ba9daf7f Address:8c5280e2-4471-9752-12ba-76c0ba9daf7f} {Suffrage:Voter ID:server-7b6b3118-69a9-ec91-912a-adcd732057cd Address:7b6b3118-69a9-ec91-912a-adcd732057cd}]"
2025-01-25T14:28:22.202Z [INFO]  server-8c5280e2-4471-9752-12ba-76c0ba9daf7f: aborting pipeline replication: peer="{Voter server-7b6b3118-69a9-ec91-912a-adcd732057cd 7b6b3118-69a9-ec91-912a-adcd732057cd}"
2025-01-25T14:28:22.202Z [INFO]  server-8c5280e2-4471-9752-12ba-76c0ba9daf7f: aborting pipeline replication: peer="{Voter server-90b306c5-c416-2a48-4f30-74063a43b43e 90b306c5-c416-2a48-4f30-74063a43b43e}"
--- PASS: TestRaft_AddKnownPeer (0.28s)
=== RUN   TestRaft_RemoveUnknownPeer
2025-01-25T14:28:22.204Z [INFO]  server-60b2d22b-7c2c-3bb6-dbcc-d226b2a0b5ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60b2d22b-7c2c-3bb6-dbcc-d226b2a0b5ef Address:60b2d22b-7c2c-3bb6-dbcc-d226b2a0b5ef} {Suffrage:Voter ID:server-a4938ee5-c214-7fd7-8fa7-4334d89e43f7 Address:a4938ee5-c214-7fd7-8fa7-4334d89e43f7} {Suffrage:Voter ID:server-6e2ff882-ae08-3db9-39ea-342981b18dd3 Address:6e2ff882-ae08-3db9-39ea-342981b18dd3}]"
2025-01-25T14:28:22.204Z [INFO]  server-a4938ee5-c214-7fd7-8fa7-4334d89e43f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60b2d22b-7c2c-3bb6-dbcc-d226b2a0b5ef Address:60b2d22b-7c2c-3bb6-dbcc-d226b2a0b5ef} {Suffrage:Voter ID:server-a4938ee5-c214-7fd7-8fa7-4334d89e43f7 Address:a4938ee5-c214-7fd7-8fa7-4334d89e43f7} {Suffrage:Voter ID:server-6e2ff882-ae08-3db9-39ea-342981b18dd3 Address:6e2ff882-ae08-3db9-39ea-342981b18dd3}]"
2025-01-25T14:28:22.204Z [INFO]  server-6e2ff882-ae08-3db9-39ea-342981b18dd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60b2d22b-7c2c-3bb6-dbcc-d226b2a0b5ef Address:60b2d22b-7c2c-3bb6-dbcc-d226b2a0b5ef} {Suffrage:Voter ID:server-a4938ee5-c214-7fd7-8fa7-4334d89e43f7 Address:a4938ee5-c214-7fd7-8fa7-4334d89e43f7} {Suffrage:Voter ID:server-6e2ff882-ae08-3db9-39ea-342981b18dd3 Address:6e2ff882-ae08-3db9-39ea-342981b18dd3}]"
2025-01-25T14:28:22.204Z [INFO]  server-60b2d22b-7c2c-3bb6-dbcc-d226b2a0b5ef: entering follower state: follower="Node at 60b2d22b-7c2c-3bb6-dbcc-d226b2a0b5ef [Follower]" leader-address= leader-id=
2025-01-25T14:28:22.204Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:22.204Z [INFO]  server-a4938ee5-c214-7fd7-8fa7-4334d89e43f7: entering follower state: follower="Node at a4938ee5-c214-7fd7-8fa7-4334d89e43f7 [Follower]" leader-address= leader-id=
2025-01-25T14:28:22.204Z [INFO]  server-6e2ff882-ae08-3db9-39ea-342981b18dd3: entering follower state: follower="Node at 6e2ff882-ae08-3db9-39ea-342981b18dd3 [Follower]" leader-address= leader-id=
2025-01-25T14:28:22.257Z [WARN]  server-a4938ee5-c214-7fd7-8fa7-4334d89e43f7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:22.257Z [INFO]  server-a4938ee5-c214-7fd7-8fa7-4334d89e43f7: entering candidate state: node="Node at a4938ee5-c214-7fd7-8fa7-4334d89e43f7 [Candidate]" term=2
2025-01-25T14:28:22.257Z [INFO]  server-a4938ee5-c214-7fd7-8fa7-4334d89e43f7: election won: term=2 tally=2
2025-01-25T14:28:22.257Z [INFO]  server-a4938ee5-c214-7fd7-8fa7-4334d89e43f7: entering leader state: leader="Node at a4938ee5-c214-7fd7-8fa7-4334d89e43f7 [Leader]"
2025-01-25T14:28:22.257Z [INFO]  server-a4938ee5-c214-7fd7-8fa7-4334d89e43f7: added peer, starting replication: peer=server-60b2d22b-7c2c-3bb6-dbcc-d226b2a0b5ef
2025-01-25T14:28:22.257Z [INFO]  server-a4938ee5-c214-7fd7-8fa7-4334d89e43f7: added peer, starting replication: peer=server-6e2ff882-ae08-3db9-39ea-342981b18dd3
2025-01-25T14:28:22.257Z [INFO]  server-a4938ee5-c214-7fd7-8fa7-4334d89e43f7: pipelining replication: peer="{Voter server-6e2ff882-ae08-3db9-39ea-342981b18dd3 6e2ff882-ae08-3db9-39ea-342981b18dd3}"
2025-01-25T14:28:22.257Z [INFO]  server-a4938ee5-c214-7fd7-8fa7-4334d89e43f7: pipelining replication: peer="{Voter server-60b2d22b-7c2c-3bb6-dbcc-d226b2a0b5ef 60b2d22b-7c2c-3bb6-dbcc-d226b2a0b5ef}"
2025-01-25T14:28:22.363Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:22.257724139 +0000 UTC m=+12.112049121 (1 nodes), 52.912296ms from start of poll, 53.599311ms from cluster start. Timeout at 2025-01-25 14:28:22.362729099 +0000 UTC m=+12.217054201, 105.00508ms after stability
2025-01-25T14:28:22.364Z [INFO]  server-a4938ee5-c214-7fd7-8fa7-4334d89e43f7: updating configuration: command=RemoveServer server-id=984de5d9-5e76-c32a-ea13-66e7225d0626 server-addr= servers="[{Suffrage:Voter ID:server-60b2d22b-7c2c-3bb6-dbcc-d226b2a0b5ef Address:60b2d22b-7c2c-3bb6-dbcc-d226b2a0b5ef} {Suffrage:Voter ID:server-a4938ee5-c214-7fd7-8fa7-4334d89e43f7 Address:a4938ee5-c214-7fd7-8fa7-4334d89e43f7} {Suffrage:Voter ID:server-6e2ff882-ae08-3db9-39ea-342981b18dd3 Address:6e2ff882-ae08-3db9-39ea-342981b18dd3}]"
2025-01-25T14:28:22.364Z [INFO]  server-a4938ee5-c214-7fd7-8fa7-4334d89e43f7: aborting pipeline replication: peer="{Voter server-60b2d22b-7c2c-3bb6-dbcc-d226b2a0b5ef 60b2d22b-7c2c-3bb6-dbcc-d226b2a0b5ef}"
2025-01-25T14:28:22.364Z [INFO]  server-a4938ee5-c214-7fd7-8fa7-4334d89e43f7: aborting pipeline replication: peer="{Voter server-6e2ff882-ae08-3db9-39ea-342981b18dd3 6e2ff882-ae08-3db9-39ea-342981b18dd3}"
--- PASS: TestRaft_RemoveUnknownPeer (0.16s)
=== RUN   TestRaft_SnapshotRestore
2025-01-25T14:28:22.366Z [INFO]  server-39d61487-4038-aaf2-7a39-1c894b4ca10e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-39d61487-4038-aaf2-7a39-1c894b4ca10e Address:39d61487-4038-aaf2-7a39-1c894b4ca10e}]"
2025-01-25T14:28:22.366Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:22.366Z [INFO]  server-39d61487-4038-aaf2-7a39-1c894b4ca10e: entering follower state: follower="Node at 39d61487-4038-aaf2-7a39-1c894b4ca10e [Follower]" leader-address= leader-id=
2025-01-25T14:28:22.452Z [WARN]  server-39d61487-4038-aaf2-7a39-1c894b4ca10e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:22.452Z [INFO]  server-39d61487-4038-aaf2-7a39-1c894b4ca10e: entering candidate state: node="Node at 39d61487-4038-aaf2-7a39-1c894b4ca10e [Candidate]" term=2
2025-01-25T14:28:22.452Z [INFO]  server-39d61487-4038-aaf2-7a39-1c894b4ca10e: election won: term=2 tally=1
2025-01-25T14:28:22.452Z [INFO]  server-39d61487-4038-aaf2-7a39-1c894b4ca10e: entering leader state: leader="Node at 39d61487-4038-aaf2-7a39-1c894b4ca10e [Leader]"
2025-01-25T14:28:22.558Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:22.452888836 +0000 UTC m=+12.307213818 (1 nodes), 86.600524ms from start of poll, 86.858955ms from cluster start. Timeout at 2025-01-25 14:28:22.557892716 +0000 UTC m=+12.412217698, 105.00388ms after stability
2025-01-25T14:28:22.564Z [INFO]  server-39d61487-4038-aaf2-7a39-1c894b4ca10e: starting snapshot up to: index=102
2025-01-25T14:28:22.564Z [INFO]  creating new snapshot: path=/tmp/raft3455121960/snapshots/2-102-1737815302564.tmp
2025-01-25T14:28:22.565Z [INFO]  server-39d61487-4038-aaf2-7a39-1c894b4ca10e: compacting logs: from=1 to=92
2025-01-25T14:28:22.565Z [INFO]  server-39d61487-4038-aaf2-7a39-1c894b4ca10e: snapshot complete up to: index=102
2025-01-25T14:28:22.566Z [INFO]  server-39d61487-4038-aaf2-7a39-1c894b4ca10e: starting restore from snapshot: id=2-102-1737815302564 last-index=102 last-term=2 size-in-bytes=693
2025-01-25T14:28:22.566Z [INFO]  server-39d61487-4038-aaf2-7a39-1c894b4ca10e: snapshot restore progress: id=2-102-1737815302564 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-01-25T14:28:22.566Z [INFO]  server-39d61487-4038-aaf2-7a39-1c894b4ca10e: restored from snapshot: id=2-102-1737815302564 last-index=102 last-term=2 size-in-bytes=693
2025-01-25T14:28:22.566Z [INFO]  server-39d61487-4038-aaf2-7a39-1c894b4ca10e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-39d61487-4038-aaf2-7a39-1c894b4ca10e Address:39d61487-4038-aaf2-7a39-1c894b4ca10e}]"
2025-01-25T14:28:22.566Z [INFO]  server-39d61487-4038-aaf2-7a39-1c894b4ca10e: entering follower state: follower="Node at 39d61487-4038-aaf2-7a39-1c894b4ca10e [Shutdown]" leader-address= leader-id=
--- PASS: TestRaft_SnapshotRestore (0.21s)
=== RUN   TestRaft_RestoreSnapshotOnStartup_Monotonic
2025-01-25T14:28:22.573Z [INFO]  server-41e82818-87ba-968e-4a42-ddaf71a2b4d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41e82818-87ba-968e-4a42-ddaf71a2b4d8 Address:41e82818-87ba-968e-4a42-ddaf71a2b4d8}]"
2025-01-25T14:28:22.573Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:22.573Z [INFO]  server-41e82818-87ba-968e-4a42-ddaf71a2b4d8: entering follower state: follower="Node at 41e82818-87ba-968e-4a42-ddaf71a2b4d8 [Follower]" leader-address= leader-id=
2025-01-25T14:28:22.660Z [WARN]  server-41e82818-87ba-968e-4a42-ddaf71a2b4d8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:22.660Z [INFO]  server-41e82818-87ba-968e-4a42-ddaf71a2b4d8: entering candidate state: node="Node at 41e82818-87ba-968e-4a42-ddaf71a2b4d8 [Candidate]" term=2
2025-01-25T14:28:22.660Z [INFO]  server-41e82818-87ba-968e-4a42-ddaf71a2b4d8: election won: term=2 tally=1
2025-01-25T14:28:22.660Z [INFO]  server-41e82818-87ba-968e-4a42-ddaf71a2b4d8: entering leader state: leader="Node at 41e82818-87ba-968e-4a42-ddaf71a2b4d8 [Leader]"
2025-01-25T14:28:22.766Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:22.661093504 +0000 UTC m=+12.515418486 (1 nodes), 87.700244ms from start of poll, 88.018993ms from cluster start. Timeout at 2025-01-25 14:28:22.766097344 +0000 UTC m=+12.620422326, 105.00384ms after stability
2025-01-25T14:28:22.772Z [INFO]  server-41e82818-87ba-968e-4a42-ddaf71a2b4d8: starting snapshot up to: index=102
2025-01-25T14:28:22.772Z [INFO]  creating new snapshot: path=/tmp/raft1921148889/snapshots/2-102-1737815302772.tmp
2025-01-25T14:28:22.773Z [INFO]  server-41e82818-87ba-968e-4a42-ddaf71a2b4d8: compacting logs: from=1 to=92
2025-01-25T14:28:22.773Z [INFO]  server-41e82818-87ba-968e-4a42-ddaf71a2b4d8: snapshot complete up to: index=102
2025-01-25T14:28:22.774Z [INFO]  server-41e82818-87ba-968e-4a42-ddaf71a2b4d8: starting restore from snapshot: id=2-102-1737815302772 last-index=102 last-term=2 size-in-bytes=693
2025-01-25T14:28:22.774Z [INFO]  server-41e82818-87ba-968e-4a42-ddaf71a2b4d8: snapshot restore progress: id=2-102-1737815302772 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-01-25T14:28:22.774Z [INFO]  server-41e82818-87ba-968e-4a42-ddaf71a2b4d8: restored from snapshot: id=2-102-1737815302772 last-index=102 last-term=2 size-in-bytes=693
2025-01-25T14:28:22.774Z [INFO]  server-41e82818-87ba-968e-4a42-ddaf71a2b4d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41e82818-87ba-968e-4a42-ddaf71a2b4d8 Address:41e82818-87ba-968e-4a42-ddaf71a2b4d8}]"
--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.20s)
=== RUN   TestRaft_SnapshotRestore_Progress
2025-01-25T14:28:22.776Z [INFO]  server-a0be593a-bac6-86af-d40a-24fc9f66e6bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0be593a-bac6-86af-d40a-24fc9f66e6bf Address:a0be593a-bac6-86af-d40a-24fc9f66e6bf}]"
2025-01-25T14:28:22.776Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:22.776Z [INFO]  server-a0be593a-bac6-86af-d40a-24fc9f66e6bf: entering follower state: follower="Node at a0be593a-bac6-86af-d40a-24fc9f66e6bf [Follower]" leader-address= leader-id=
2025-01-25T14:28:22.846Z [WARN]  server-a0be593a-bac6-86af-d40a-24fc9f66e6bf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:22.846Z [INFO]  server-a0be593a-bac6-86af-d40a-24fc9f66e6bf: entering candidate state: node="Node at a0be593a-bac6-86af-d40a-24fc9f66e6bf [Candidate]" term=2
2025-01-25T14:28:22.846Z [INFO]  server-a0be593a-bac6-86af-d40a-24fc9f66e6bf: election won: term=2 tally=1
2025-01-25T14:28:22.846Z [INFO]  server-a0be593a-bac6-86af-d40a-24fc9f66e6bf: entering leader state: leader="Node at a0be593a-bac6-86af-d40a-24fc9f66e6bf [Leader]"
2025-01-25T14:28:22.952Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:22.847166848 +0000 UTC m=+12.701491830 (1 nodes), 70.798572ms from start of poll, 71.056163ms from cluster start. Timeout at 2025-01-25 14:28:22.952175647 +0000 UTC m=+12.806500749, 105.008919ms after stability
2025-01-25T14:28:22.957Z [INFO]  server-a0be593a-bac6-86af-d40a-24fc9f66e6bf: starting snapshot up to: index=102
2025-01-25T14:28:22.957Z [INFO]  creating new snapshot: path=/tmp/raft1795339862/snapshots/2-102-1737815302957.tmp
2025-01-25T14:28:22.958Z [INFO]  server-a0be593a-bac6-86af-d40a-24fc9f66e6bf: compacting logs: from=1 to=92
2025-01-25T14:28:22.958Z [INFO]  server-a0be593a-bac6-86af-d40a-24fc9f66e6bf: snapshot complete up to: index=102
--- PASS: TestRaft_SnapshotRestore_Progress (0.19s)
=== RUN   TestRaft_NoRestoreOnStart
2025-01-25T14:28:22.963Z [INFO]  server-2d5b6e6e-5630-52bf-b163-99cccce47836: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d5b6e6e-5630-52bf-b163-99cccce47836 Address:2d5b6e6e-5630-52bf-b163-99cccce47836}]"
2025-01-25T14:28:22.963Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:22.963Z [INFO]  server-2d5b6e6e-5630-52bf-b163-99cccce47836: entering follower state: follower="Node at 2d5b6e6e-5630-52bf-b163-99cccce47836 [Follower]" leader-address= leader-id=
2025-01-25T14:28:23.028Z [WARN]  server-2d5b6e6e-5630-52bf-b163-99cccce47836: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:23.028Z [INFO]  server-2d5b6e6e-5630-52bf-b163-99cccce47836: entering candidate state: node="Node at 2d5b6e6e-5630-52bf-b163-99cccce47836 [Candidate]" term=2
2025-01-25T14:28:23.028Z [INFO]  server-2d5b6e6e-5630-52bf-b163-99cccce47836: election won: term=2 tally=1
2025-01-25T14:28:23.028Z [INFO]  server-2d5b6e6e-5630-52bf-b163-99cccce47836: entering leader state: leader="Node at 2d5b6e6e-5630-52bf-b163-99cccce47836 [Leader]"
2025-01-25T14:28:23.135Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:23.029070422 +0000 UTC m=+12.883395364 (1 nodes), 65.146135ms from start of poll, 65.446604ms from cluster start. Timeout at 2025-01-25 14:28:23.134072582 +0000 UTC m=+12.988397564, 105.0022ms after stability
2025-01-25T14:28:23.141Z [INFO]  server-2d5b6e6e-5630-52bf-b163-99cccce47836: starting snapshot up to: index=102
2025-01-25T14:28:23.141Z [INFO]  creating new snapshot: path=/tmp/raft1170098183/snapshots/2-102-1737815303141.tmp
2025-01-25T14:28:23.142Z [INFO]  server-2d5b6e6e-5630-52bf-b163-99cccce47836: compacting logs: from=1 to=92
2025-01-25T14:28:23.142Z [INFO]  server-2d5b6e6e-5630-52bf-b163-99cccce47836: snapshot complete up to: index=102
2025-01-25T14:28:23.143Z [INFO]  server-2d5b6e6e-5630-52bf-b163-99cccce47836: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d5b6e6e-5630-52bf-b163-99cccce47836 Address:2d5b6e6e-5630-52bf-b163-99cccce47836}]"
--- PASS: TestRaft_NoRestoreOnStart (0.18s)
=== RUN   TestRaft_SnapshotRestore_PeerChange
    raft_test.go:1266: Skipping unreliable test
--- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
=== RUN   TestRaft_AutoSnapshot
2025-01-25T14:28:23.143Z [INFO]  server-2d5b6e6e-5630-52bf-b163-99cccce47836: entering follower state: follower="Node at 2d5b6e6e-5630-52bf-b163-99cccce47836 [Follower]" leader-address= leader-id=
2025-01-25T14:28:23.144Z [INFO]  server-d8221df0-c784-f02f-9292-ab5bb8771427: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8221df0-c784-f02f-9292-ab5bb8771427 Address:d8221df0-c784-f02f-9292-ab5bb8771427}]"
2025-01-25T14:28:23.144Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:23.144Z [INFO]  server-d8221df0-c784-f02f-9292-ab5bb8771427: entering follower state: follower="Node at d8221df0-c784-f02f-9292-ab5bb8771427 [Follower]" leader-address= leader-id=
2025-01-25T14:28:23.202Z [WARN]  server-d8221df0-c784-f02f-9292-ab5bb8771427: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:23.202Z [INFO]  server-d8221df0-c784-f02f-9292-ab5bb8771427: entering candidate state: node="Node at d8221df0-c784-f02f-9292-ab5bb8771427 [Candidate]" term=2
2025-01-25T14:28:23.202Z [INFO]  server-d8221df0-c784-f02f-9292-ab5bb8771427: election won: term=2 tally=1
2025-01-25T14:28:23.202Z [INFO]  server-d8221df0-c784-f02f-9292-ab5bb8771427: entering leader state: leader="Node at d8221df0-c784-f02f-9292-ab5bb8771427 [Leader]"
2025-01-25T14:28:23.224Z [WARN]  server-2d5b6e6e-5630-52bf-b163-99cccce47836: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:23.224Z [INFO]  server-2d5b6e6e-5630-52bf-b163-99cccce47836: entering candidate state: node="Node at 2d5b6e6e-5630-52bf-b163-99cccce47836 [Candidate]" term=3
2025-01-25T14:28:23.224Z [INFO]  server-2d5b6e6e-5630-52bf-b163-99cccce47836: election won: term=3 tally=1
2025-01-25T14:28:23.224Z [INFO]  server-2d5b6e6e-5630-52bf-b163-99cccce47836: entering leader state: leader="Node at 2d5b6e6e-5630-52bf-b163-99cccce47836 [Leader]"
2025-01-25T14:28:23.308Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:23.202428304 +0000 UTC m=+13.056753246 (1 nodes), 58.031991ms from start of poll, 58.268623ms from cluster start. Timeout at 2025-01-25 14:28:23.307439904 +0000 UTC m=+13.161764846, 105.0116ms after stability
2025-01-25T14:28:23.320Z [INFO]  server-d8221df0-c784-f02f-9292-ab5bb8771427: starting snapshot up to: index=102
2025-01-25T14:28:23.320Z [INFO]  creating new snapshot: path=/tmp/raft1349550150/snapshots/2-102-1737815303320.tmp
2025-01-25T14:28:23.322Z [INFO]  server-d8221df0-c784-f02f-9292-ab5bb8771427: compacting logs: from=1 to=92
2025-01-25T14:28:23.323Z [INFO]  server-d8221df0-c784-f02f-9292-ab5bb8771427: snapshot complete up to: index=102
--- PASS: TestRaft_AutoSnapshot (0.28s)
=== RUN   TestRaft_UserSnapshot
2025-01-25T14:28:23.421Z [INFO]  server-2e478ced-bfbd-ab8f-40e4-f18b73616d3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e478ced-bfbd-ab8f-40e4-f18b73616d3f Address:2e478ced-bfbd-ab8f-40e4-f18b73616d3f}]"
2025-01-25T14:28:23.421Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:23.421Z [INFO]  server-2e478ced-bfbd-ab8f-40e4-f18b73616d3f: entering follower state: follower="Node at 2e478ced-bfbd-ab8f-40e4-f18b73616d3f [Follower]" leader-address= leader-id=
2025-01-25T14:28:23.474Z [WARN]  server-2e478ced-bfbd-ab8f-40e4-f18b73616d3f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:23.474Z [INFO]  server-2e478ced-bfbd-ab8f-40e4-f18b73616d3f: entering candidate state: node="Node at 2e478ced-bfbd-ab8f-40e4-f18b73616d3f [Candidate]" term=2
2025-01-25T14:28:23.474Z [INFO]  server-2e478ced-bfbd-ab8f-40e4-f18b73616d3f: election won: term=2 tally=1
2025-01-25T14:28:23.474Z [INFO]  server-2e478ced-bfbd-ab8f-40e4-f18b73616d3f: entering leader state: leader="Node at 2e478ced-bfbd-ab8f-40e4-f18b73616d3f [Leader]"
2025-01-25T14:28:23.580Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:23.4748913 +0000 UTC m=+13.329216322 (1 nodes), 53.220965ms from start of poll, 53.498395ms from cluster start. Timeout at 2025-01-25 14:28:23.57990298 +0000 UTC m=+13.434228082, 105.01176ms after stability
2025-01-25T14:28:23.580Z [ERROR] server-2e478ced-bfbd-ab8f-40e4-f18b73616d3f: failed to take snapshot: error="nothing new to snapshot"
2025-01-25T14:28:23.581Z [INFO]  server-2e478ced-bfbd-ab8f-40e4-f18b73616d3f: starting snapshot up to: index=12
2025-01-25T14:28:23.581Z [INFO]  creating new snapshot: path=/tmp/raft2371625306/snapshots/2-12-1737815303581.tmp
2025-01-25T14:28:23.582Z [INFO]  server-2e478ced-bfbd-ab8f-40e4-f18b73616d3f: compacting logs: from=1 to=2
2025-01-25T14:28:23.582Z [INFO]  server-2e478ced-bfbd-ab8f-40e4-f18b73616d3f: snapshot complete up to: index=12
--- PASS: TestRaft_UserSnapshot (0.16s)
=== RUN   TestRaft_UserRestore
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false
2025-01-25T14:28:23.585Z [INFO]  server-f5f038c3-0c91-3db2-f9a9-7b07b30e36b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5f038c3-0c91-3db2-f9a9-7b07b30e36b8 Address:f5f038c3-0c91-3db2-f9a9-7b07b30e36b8} {Suffrage:Voter ID:server-da7a359e-d96a-ff91-fd39-ec315a69c951 Address:da7a359e-d96a-ff91-fd39-ec315a69c951} {Suffrage:Voter ID:server-3e88375d-24de-1f19-4836-cbee105fa881 Address:3e88375d-24de-1f19-4836-cbee105fa881}]"
2025-01-25T14:28:23.585Z [INFO]  server-f5f038c3-0c91-3db2-f9a9-7b07b30e36b8: entering follower state: follower="Node at f5f038c3-0c91-3db2-f9a9-7b07b30e36b8 [Follower]" leader-address= leader-id=
2025-01-25T14:28:23.585Z [INFO]  server-da7a359e-d96a-ff91-fd39-ec315a69c951: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5f038c3-0c91-3db2-f9a9-7b07b30e36b8 Address:f5f038c3-0c91-3db2-f9a9-7b07b30e36b8} {Suffrage:Voter ID:server-da7a359e-d96a-ff91-fd39-ec315a69c951 Address:da7a359e-d96a-ff91-fd39-ec315a69c951} {Suffrage:Voter ID:server-3e88375d-24de-1f19-4836-cbee105fa881 Address:3e88375d-24de-1f19-4836-cbee105fa881}]"
2025-01-25T14:28:23.585Z [INFO]  server-da7a359e-d96a-ff91-fd39-ec315a69c951: entering follower state: follower="Node at da7a359e-d96a-ff91-fd39-ec315a69c951 [Follower]" leader-address= leader-id=
2025-01-25T14:28:23.585Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5f038c3-0c91-3db2-f9a9-7b07b30e36b8 Address:f5f038c3-0c91-3db2-f9a9-7b07b30e36b8} {Suffrage:Voter ID:server-da7a359e-d96a-ff91-fd39-ec315a69c951 Address:da7a359e-d96a-ff91-fd39-ec315a69c951} {Suffrage:Voter ID:server-3e88375d-24de-1f19-4836-cbee105fa881 Address:3e88375d-24de-1f19-4836-cbee105fa881}]"
2025-01-25T14:28:23.586Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:23.586Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: entering follower state: follower="Node at 3e88375d-24de-1f19-4836-cbee105fa881 [Follower]" leader-address= leader-id=
2025-01-25T14:28:24.254Z [WARN]  server-3e88375d-24de-1f19-4836-cbee105fa881: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:24.254Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: entering candidate state: node="Node at 3e88375d-24de-1f19-4836-cbee105fa881 [Candidate]" term=2
2025-01-25T14:28:24.254Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: election won: term=2 tally=2
2025-01-25T14:28:24.254Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: entering leader state: leader="Node at 3e88375d-24de-1f19-4836-cbee105fa881 [Leader]"
2025-01-25T14:28:24.254Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: added peer, starting replication: peer=server-f5f038c3-0c91-3db2-f9a9-7b07b30e36b8
2025-01-25T14:28:24.254Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: added peer, starting replication: peer=server-da7a359e-d96a-ff91-fd39-ec315a69c951
2025-01-25T14:28:24.254Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: pipelining replication: peer="{Voter server-da7a359e-d96a-ff91-fd39-ec315a69c951 da7a359e-d96a-ff91-fd39-ec315a69c951}"
2025-01-25T14:28:24.255Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: pipelining replication: peer="{Voter server-f5f038c3-0c91-3db2-f9a9-7b07b30e36b8 f5f038c3-0c91-3db2-f9a9-7b07b30e36b8}"
2025-01-25T14:28:25.260Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:24.255166942 +0000 UTC m=+14.109491924 (1 nodes), 669.082763ms from start of poll, 669.950852ms from cluster start. Timeout at 2025-01-25 14:28:25.260169942 +0000 UTC m=+15.114494924, 1.005003s after stability
2025-01-25T14:28:25.261Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: starting snapshot up to: index=12
2025-01-25T14:28:25.261Z [INFO]  creating new snapshot: path=/tmp/raft3249874888/snapshots/2-12-1737815305261.tmp
2025-01-25T14:28:25.263Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: snapshot complete up to: index=12
2025-01-25T14:28:25.264Z [INFO]  creating new snapshot: path=/tmp/raft3249874888/snapshots/2-23-1737815305264.tmp
2025-01-25T14:28:25.265Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: copied to local snapshot: bytes=71
2025-01-25T14:28:25.266Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: snapshot restore progress: id=2-23-1737815305264 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:25.266Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: restored user snapshot: index=23
2025-01-25T14:28:25.266Z [ERROR] server-3e88375d-24de-1f19-4836-cbee105fa881: failed to get log: index=23 error="log not found"
2025-01-25T14:28:25.266Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: aborting pipeline replication: peer="{Voter server-da7a359e-d96a-ff91-fd39-ec315a69c951 da7a359e-d96a-ff91-fd39-ec315a69c951}"
2025-01-25T14:28:25.266Z [ERROR] server-3e88375d-24de-1f19-4836-cbee105fa881: failed to get log: index=23 error="log not found"
2025-01-25T14:28:25.266Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: aborting pipeline replication: peer="{Voter server-f5f038c3-0c91-3db2-f9a9-7b07b30e36b8 f5f038c3-0c91-3db2-f9a9-7b07b30e36b8}"
2025-01-25T14:28:25.273Z [ERROR] server-3e88375d-24de-1f19-4836-cbee105fa881: failed to get log: index=23 error="log not found"
2025-01-25T14:28:25.273Z [INFO]  creating new snapshot: path=/tmp/raft1302280046/snapshots/2-23-1737815305273.tmp
2025-01-25T14:28:25.274Z [INFO]  server-f5f038c3-0c91-3db2-f9a9-7b07b30e36b8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:25.275Z [INFO]  server-f5f038c3-0c91-3db2-f9a9-7b07b30e36b8: copied to local snapshot: bytes=71
2025-01-25T14:28:25.275Z [ERROR] server-3e88375d-24de-1f19-4836-cbee105fa881: failed to get log: index=23 error="log not found"
2025-01-25T14:28:25.275Z [INFO]  server-f5f038c3-0c91-3db2-f9a9-7b07b30e36b8: snapshot restore progress: id=2-23-1737815305273 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:25.275Z [INFO]  server-f5f038c3-0c91-3db2-f9a9-7b07b30e36b8: Installed remote snapshot
2025-01-25T14:28:25.275Z [WARN]  server-f5f038c3-0c91-3db2-f9a9-7b07b30e36b8: skipping application of old log: index=22
2025-01-25T14:28:25.275Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: pipelining replication: peer="{Voter server-f5f038c3-0c91-3db2-f9a9-7b07b30e36b8 f5f038c3-0c91-3db2-f9a9-7b07b30e36b8}"
2025-01-25T14:28:25.276Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:25.276Z [INFO]  creating new snapshot: path=/tmp/raft2831107949/snapshots/2-23-1737815305276.tmp
2025-01-25T14:28:25.276Z [INFO]  server-da7a359e-d96a-ff91-fd39-ec315a69c951: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:25.277Z [INFO]  server-da7a359e-d96a-ff91-fd39-ec315a69c951: copied to local snapshot: bytes=71
2025-01-25T14:28:25.278Z [INFO]  server-da7a359e-d96a-ff91-fd39-ec315a69c951: snapshot restore progress: id=2-23-1737815305276 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:25.278Z [INFO]  server-da7a359e-d96a-ff91-fd39-ec315a69c951: Installed remote snapshot
2025-01-25T14:28:25.278Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: pipelining replication: peer="{Voter server-da7a359e-d96a-ff91-fd39-ec315a69c951 da7a359e-d96a-ff91-fd39-ec315a69c951}"
2025-01-25T14:28:26.282Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:25.276075046 +0000 UTC m=+15.130400028 (2 nodes), 2.4µs from start of poll, 1.690858956s from cluster start. Timeout at 2025-01-25 14:28:26.281093005 +0000 UTC m=+16.135417987, 1.005017959s after stability
2025-01-25T14:28:26.294Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: aborting pipeline replication: peer="{Voter server-f5f038c3-0c91-3db2-f9a9-7b07b30e36b8 f5f038c3-0c91-3db2-f9a9-7b07b30e36b8}"
2025-01-25T14:28:26.294Z [INFO]  server-3e88375d-24de-1f19-4836-cbee105fa881: aborting pipeline replication: peer="{Voter server-da7a359e-d96a-ff91-fd39-ec315a69c951 da7a359e-d96a-ff91-fd39-ec315a69c951}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false
2025-01-25T14:28:26.300Z [INFO]  server-2a0d79cf-9cfb-0500-3f63-44160acf1b11: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a0d79cf-9cfb-0500-3f63-44160acf1b11 Address:2a0d79cf-9cfb-0500-3f63-44160acf1b11} {Suffrage:Voter ID:server-32597975-50ca-6368-9b67-126543c1b1eb Address:32597975-50ca-6368-9b67-126543c1b1eb} {Suffrage:Voter ID:server-164f35b8-acee-1a04-a74d-8841a8c45ed6 Address:164f35b8-acee-1a04-a74d-8841a8c45ed6}]"
2025-01-25T14:28:26.300Z [INFO]  server-32597975-50ca-6368-9b67-126543c1b1eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a0d79cf-9cfb-0500-3f63-44160acf1b11 Address:2a0d79cf-9cfb-0500-3f63-44160acf1b11} {Suffrage:Voter ID:server-32597975-50ca-6368-9b67-126543c1b1eb Address:32597975-50ca-6368-9b67-126543c1b1eb} {Suffrage:Voter ID:server-164f35b8-acee-1a04-a74d-8841a8c45ed6 Address:164f35b8-acee-1a04-a74d-8841a8c45ed6}]"
2025-01-25T14:28:26.300Z [INFO]  server-2a0d79cf-9cfb-0500-3f63-44160acf1b11: entering follower state: follower="Node at 2a0d79cf-9cfb-0500-3f63-44160acf1b11 [Follower]" leader-address= leader-id=
2025-01-25T14:28:26.300Z [INFO]  server-32597975-50ca-6368-9b67-126543c1b1eb: entering follower state: follower="Node at 32597975-50ca-6368-9b67-126543c1b1eb [Follower]" leader-address= leader-id=
2025-01-25T14:28:26.300Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a0d79cf-9cfb-0500-3f63-44160acf1b11 Address:2a0d79cf-9cfb-0500-3f63-44160acf1b11} {Suffrage:Voter ID:server-32597975-50ca-6368-9b67-126543c1b1eb Address:32597975-50ca-6368-9b67-126543c1b1eb} {Suffrage:Voter ID:server-164f35b8-acee-1a04-a74d-8841a8c45ed6 Address:164f35b8-acee-1a04-a74d-8841a8c45ed6}]"
2025-01-25T14:28:26.300Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:26.300Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: entering follower state: follower="Node at 164f35b8-acee-1a04-a74d-8841a8c45ed6 [Follower]" leader-address= leader-id=
2025-01-25T14:28:26.833Z [WARN]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:26.833Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: entering candidate state: node="Node at 164f35b8-acee-1a04-a74d-8841a8c45ed6 [Candidate]" term=2
2025-01-25T14:28:26.833Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: election won: term=2 tally=2
2025-01-25T14:28:26.833Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: entering leader state: leader="Node at 164f35b8-acee-1a04-a74d-8841a8c45ed6 [Leader]"
2025-01-25T14:28:26.833Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: added peer, starting replication: peer=server-2a0d79cf-9cfb-0500-3f63-44160acf1b11
2025-01-25T14:28:26.833Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: added peer, starting replication: peer=server-32597975-50ca-6368-9b67-126543c1b1eb
2025-01-25T14:28:26.833Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: pipelining replication: peer="{Voter server-32597975-50ca-6368-9b67-126543c1b1eb 32597975-50ca-6368-9b67-126543c1b1eb}"
2025-01-25T14:28:26.833Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: pipelining replication: peer="{Voter server-2a0d79cf-9cfb-0500-3f63-44160acf1b11 2a0d79cf-9cfb-0500-3f63-44160acf1b11}"
2025-01-25T14:28:27.839Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:26.833754913 +0000 UTC m=+16.688079895 (1 nodes), 533.055978ms from start of poll, 533.84227ms from cluster start. Timeout at 2025-01-25 14:28:27.838757513 +0000 UTC m=+17.693082495, 1.0050026s after stability
2025-01-25T14:28:27.840Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: starting snapshot up to: index=12
2025-01-25T14:28:27.840Z [INFO]  creating new snapshot: path=/tmp/raft1058140866/snapshots/2-12-1737815307840.tmp
2025-01-25T14:28:27.841Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: snapshot complete up to: index=12
2025-01-25T14:28:27.844Z [INFO]  creating new snapshot: path=/tmp/raft1058140866/snapshots/2-23-1737815307844.tmp
2025-01-25T14:28:27.845Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: copied to local snapshot: bytes=71
2025-01-25T14:28:27.845Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: snapshot restore progress: id=2-23-1737815307844 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:27.845Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: removing all old logs from log store
2025-01-25T14:28:27.845Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: compacting logs: from=1 to=22
2025-01-25T14:28:27.846Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: restored user snapshot: index=23
2025-01-25T14:28:27.846Z [ERROR] server-164f35b8-acee-1a04-a74d-8841a8c45ed6: failed to get log: index=22 error="log not found"
2025-01-25T14:28:27.846Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: aborting pipeline replication: peer="{Voter server-32597975-50ca-6368-9b67-126543c1b1eb 32597975-50ca-6368-9b67-126543c1b1eb}"
2025-01-25T14:28:27.846Z [ERROR] server-164f35b8-acee-1a04-a74d-8841a8c45ed6: failed to get log: index=22 error="log not found"
2025-01-25T14:28:27.846Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: aborting pipeline replication: peer="{Voter server-2a0d79cf-9cfb-0500-3f63-44160acf1b11 2a0d79cf-9cfb-0500-3f63-44160acf1b11}"
2025-01-25T14:28:27.853Z [ERROR] server-164f35b8-acee-1a04-a74d-8841a8c45ed6: failed to get log: index=22 error="log not found"
2025-01-25T14:28:27.853Z [ERROR] server-164f35b8-acee-1a04-a74d-8841a8c45ed6: failed to get log: index=22 error="log not found"
2025-01-25T14:28:27.854Z [INFO]  creating new snapshot: path=/tmp/raft204205405/snapshots/2-23-1737815307854.tmp
2025-01-25T14:28:27.854Z [INFO]  creating new snapshot: path=/tmp/raft15249334/snapshots/2-23-1737815307854.tmp
2025-01-25T14:28:27.854Z [INFO]  server-32597975-50ca-6368-9b67-126543c1b1eb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:27.855Z [INFO]  server-2a0d79cf-9cfb-0500-3f63-44160acf1b11: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:27.855Z [INFO]  server-32597975-50ca-6368-9b67-126543c1b1eb: copied to local snapshot: bytes=71
2025-01-25T14:28:27.855Z [INFO]  server-32597975-50ca-6368-9b67-126543c1b1eb: snapshot restore progress: id=2-23-1737815307854 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:27.855Z [INFO]  server-2a0d79cf-9cfb-0500-3f63-44160acf1b11: copied to local snapshot: bytes=71
2025-01-25T14:28:27.855Z [INFO]  server-32597975-50ca-6368-9b67-126543c1b1eb: removing all old logs from log store
2025-01-25T14:28:27.856Z [INFO]  server-32597975-50ca-6368-9b67-126543c1b1eb: compacting logs: from=1 to=22
2025-01-25T14:28:27.856Z [INFO]  server-32597975-50ca-6368-9b67-126543c1b1eb: Installed remote snapshot
2025-01-25T14:28:27.856Z [WARN]  server-32597975-50ca-6368-9b67-126543c1b1eb: skipping application of old log: index=22
2025-01-25T14:28:27.856Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: pipelining replication: peer="{Voter server-32597975-50ca-6368-9b67-126543c1b1eb 32597975-50ca-6368-9b67-126543c1b1eb}"
2025-01-25T14:28:27.856Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:27.856Z [INFO]  server-2a0d79cf-9cfb-0500-3f63-44160acf1b11: snapshot restore progress: id=2-23-1737815307854 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:27.856Z [INFO]  server-2a0d79cf-9cfb-0500-3f63-44160acf1b11: removing all old logs from log store
2025-01-25T14:28:27.856Z [INFO]  server-2a0d79cf-9cfb-0500-3f63-44160acf1b11: compacting logs: from=1 to=22
2025-01-25T14:28:27.856Z [INFO]  server-2a0d79cf-9cfb-0500-3f63-44160acf1b11: Installed remote snapshot
2025-01-25T14:28:27.856Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: pipelining replication: peer="{Voter server-2a0d79cf-9cfb-0500-3f63-44160acf1b11 2a0d79cf-9cfb-0500-3f63-44160acf1b11}"
2025-01-25T14:28:28.862Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:27.856358676 +0000 UTC m=+17.710683658 (2 nodes), 1.92µs from start of poll, 1.556446033s from cluster start. Timeout at 2025-01-25 14:28:28.861381835 +0000 UTC m=+18.715706937, 1.005023279s after stability
2025-01-25T14:28:28.875Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: aborting pipeline replication: peer="{Voter server-32597975-50ca-6368-9b67-126543c1b1eb 32597975-50ca-6368-9b67-126543c1b1eb}"
2025-01-25T14:28:28.876Z [INFO]  server-164f35b8-acee-1a04-a74d-8841a8c45ed6: aborting pipeline replication: peer="{Voter server-2a0d79cf-9cfb-0500-3f63-44160acf1b11 2a0d79cf-9cfb-0500-3f63-44160acf1b11}"
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true
2025-01-25T14:28:28.878Z [INFO]  server-dd0b3047-2bc0-c9c9-a469-a4a4a4d223f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd0b3047-2bc0-c9c9-a469-a4a4a4d223f0 Address:dd0b3047-2bc0-c9c9-a469-a4a4a4d223f0} {Suffrage:Voter ID:server-a640e938-62dd-b60f-a2a4-83e2d644b629 Address:a640e938-62dd-b60f-a2a4-83e2d644b629} {Suffrage:Voter ID:server-95b0d786-5552-3df0-7e2a-df0bbd4203df Address:95b0d786-5552-3df0-7e2a-df0bbd4203df}]"
2025-01-25T14:28:28.879Z [INFO]  server-dd0b3047-2bc0-c9c9-a469-a4a4a4d223f0: entering follower state: follower="Node at dd0b3047-2bc0-c9c9-a469-a4a4a4d223f0 [Follower]" leader-address= leader-id=
2025-01-25T14:28:28.879Z [INFO]  server-a640e938-62dd-b60f-a2a4-83e2d644b629: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd0b3047-2bc0-c9c9-a469-a4a4a4d223f0 Address:dd0b3047-2bc0-c9c9-a469-a4a4a4d223f0} {Suffrage:Voter ID:server-a640e938-62dd-b60f-a2a4-83e2d644b629 Address:a640e938-62dd-b60f-a2a4-83e2d644b629} {Suffrage:Voter ID:server-95b0d786-5552-3df0-7e2a-df0bbd4203df Address:95b0d786-5552-3df0-7e2a-df0bbd4203df}]"
2025-01-25T14:28:28.879Z [INFO]  server-a640e938-62dd-b60f-a2a4-83e2d644b629: entering follower state: follower="Node at a640e938-62dd-b60f-a2a4-83e2d644b629 [Follower]" leader-address= leader-id=
2025-01-25T14:28:28.879Z [INFO]  server-95b0d786-5552-3df0-7e2a-df0bbd4203df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd0b3047-2bc0-c9c9-a469-a4a4a4d223f0 Address:dd0b3047-2bc0-c9c9-a469-a4a4a4d223f0} {Suffrage:Voter ID:server-a640e938-62dd-b60f-a2a4-83e2d644b629 Address:a640e938-62dd-b60f-a2a4-83e2d644b629} {Suffrage:Voter ID:server-95b0d786-5552-3df0-7e2a-df0bbd4203df Address:95b0d786-5552-3df0-7e2a-df0bbd4203df}]"
2025-01-25T14:28:28.879Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:28.879Z [INFO]  server-95b0d786-5552-3df0-7e2a-df0bbd4203df: entering follower state: follower="Node at 95b0d786-5552-3df0-7e2a-df0bbd4203df [Follower]" leader-address= leader-id=
2025-01-25T14:28:29.390Z [WARN]  server-95b0d786-5552-3df0-7e2a-df0bbd4203df: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:29.390Z [INFO]  server-95b0d786-5552-3df0-7e2a-df0bbd4203df: entering candidate state: node="Node at 95b0d786-5552-3df0-7e2a-df0bbd4203df [Candidate]" term=2
2025-01-25T14:28:29.390Z [INFO]  server-95b0d786-5552-3df0-7e2a-df0bbd4203df: election won: term=2 tally=2
2025-01-25T14:28:29.390Z [INFO]  server-95b0d786-5552-3df0-7e2a-df0bbd4203df: entering leader state: leader="Node at 95b0d786-5552-3df0-7e2a-df0bbd4203df [Leader]"
2025-01-25T14:28:29.390Z [INFO]  server-95b0d786-5552-3df0-7e2a-df0bbd4203df: added peer, starting replication: peer=server-dd0b3047-2bc0-c9c9-a469-a4a4a4d223f0
2025-01-25T14:28:29.390Z [INFO]  server-95b0d786-5552-3df0-7e2a-df0bbd4203df: added peer, starting replication: peer=server-a640e938-62dd-b60f-a2a4-83e2d644b629
2025-01-25T14:28:29.390Z [INFO]  server-95b0d786-5552-3df0-7e2a-df0bbd4203df: pipelining replication: peer="{Voter server-dd0b3047-2bc0-c9c9-a469-a4a4a4d223f0 dd0b3047-2bc0-c9c9-a469-a4a4a4d223f0}"
2025-01-25T14:28:29.390Z [INFO]  server-95b0d786-5552-3df0-7e2a-df0bbd4203df: pipelining replication: peer="{Voter server-a640e938-62dd-b60f-a2a4-83e2d644b629 a640e938-62dd-b60f-a2a4-83e2d644b629}"
2025-01-25T14:28:30.396Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:29.390756741 +0000 UTC m=+19.245081723 (1 nodes), 511.150566ms from start of poll, 512.079973ms from cluster start. Timeout at 2025-01-25 14:28:30.39576994 +0000 UTC m=+20.250094882, 1.005013159s after stability
2025-01-25T14:28:30.398Z [INFO]  server-95b0d786-5552-3df0-7e2a-df0bbd4203df: starting snapshot up to: index=12
2025-01-25T14:28:30.398Z [INFO]  creating new snapshot: path=/tmp/raft77379406/snapshots/2-12-1737815310398.tmp
2025-01-25T14:28:30.399Z [INFO]  server-95b0d786-5552-3df0-7e2a-df0bbd4203df: snapshot complete up to: index=12
2025-01-25T14:28:30.402Z [INFO]  server-eef17de3-64ee-8f3e-7579-3c1c746e4acc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eef17de3-64ee-8f3e-7579-3c1c746e4acc Address:eef17de3-64ee-8f3e-7579-3c1c746e4acc} {Suffrage:Voter ID:server-66ece0d9-ebca-02c7-a98d-14a1ce7a503b Address:66ece0d9-ebca-02c7-a98d-14a1ce7a503b} {Suffrage:Voter ID:server-972da02a-ccc4-5e53-1f90-1452016ead72 Address:972da02a-ccc4-5e53-1f90-1452016ead72}]"
2025-01-25T14:28:30.402Z [INFO]  server-66ece0d9-ebca-02c7-a98d-14a1ce7a503b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eef17de3-64ee-8f3e-7579-3c1c746e4acc Address:eef17de3-64ee-8f3e-7579-3c1c746e4acc} {Suffrage:Voter ID:server-66ece0d9-ebca-02c7-a98d-14a1ce7a503b Address:66ece0d9-ebca-02c7-a98d-14a1ce7a503b} {Suffrage:Voter ID:server-972da02a-ccc4-5e53-1f90-1452016ead72 Address:972da02a-ccc4-5e53-1f90-1452016ead72}]"
2025-01-25T14:28:30.402Z [INFO]  server-eef17de3-64ee-8f3e-7579-3c1c746e4acc: entering follower state: follower="Node at eef17de3-64ee-8f3e-7579-3c1c746e4acc [Follower]" leader-address= leader-id=
2025-01-25T14:28:30.402Z [INFO]  server-66ece0d9-ebca-02c7-a98d-14a1ce7a503b: entering follower state: follower="Node at 66ece0d9-ebca-02c7-a98d-14a1ce7a503b [Follower]" leader-address= leader-id=
2025-01-25T14:28:30.402Z [INFO]  server-972da02a-ccc4-5e53-1f90-1452016ead72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eef17de3-64ee-8f3e-7579-3c1c746e4acc Address:eef17de3-64ee-8f3e-7579-3c1c746e4acc} {Suffrage:Voter ID:server-66ece0d9-ebca-02c7-a98d-14a1ce7a503b Address:66ece0d9-ebca-02c7-a98d-14a1ce7a503b} {Suffrage:Voter ID:server-972da02a-ccc4-5e53-1f90-1452016ead72 Address:972da02a-ccc4-5e53-1f90-1452016ead72}]"
2025-01-25T14:28:30.402Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:30.402Z [INFO]  server-972da02a-ccc4-5e53-1f90-1452016ead72: entering follower state: follower="Node at 972da02a-ccc4-5e53-1f90-1452016ead72 [Follower]" leader-address= leader-id=
2025-01-25T14:28:30.916Z [WARN]  server-972da02a-ccc4-5e53-1f90-1452016ead72: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:30.916Z [INFO]  server-972da02a-ccc4-5e53-1f90-1452016ead72: entering candidate state: node="Node at 972da02a-ccc4-5e53-1f90-1452016ead72 [Candidate]" term=2
2025-01-25T14:28:30.916Z [INFO]  server-972da02a-ccc4-5e53-1f90-1452016ead72: election won: term=2 tally=2
2025-01-25T14:28:30.916Z [INFO]  server-972da02a-ccc4-5e53-1f90-1452016ead72: entering leader state: leader="Node at 972da02a-ccc4-5e53-1f90-1452016ead72 [Leader]"
2025-01-25T14:28:30.916Z [INFO]  server-972da02a-ccc4-5e53-1f90-1452016ead72: added peer, starting replication: peer=server-eef17de3-64ee-8f3e-7579-3c1c746e4acc
2025-01-25T14:28:30.917Z [INFO]  server-972da02a-ccc4-5e53-1f90-1452016ead72: added peer, starting replication: peer=server-66ece0d9-ebca-02c7-a98d-14a1ce7a503b
2025-01-25T14:28:30.917Z [INFO]  server-972da02a-ccc4-5e53-1f90-1452016ead72: pipelining replication: peer="{Voter server-66ece0d9-ebca-02c7-a98d-14a1ce7a503b 66ece0d9-ebca-02c7-a98d-14a1ce7a503b}"
2025-01-25T14:28:30.917Z [INFO]  server-972da02a-ccc4-5e53-1f90-1452016ead72: pipelining replication: peer="{Voter server-eef17de3-64ee-8f3e-7579-3c1c746e4acc eef17de3-64ee-8f3e-7579-3c1c746e4acc}"
2025-01-25T14:28:31.922Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:30.917218332 +0000 UTC m=+20.771543314 (1 nodes), 514.605922ms from start of poll, 515.461651ms from cluster start. Timeout at 2025-01-25 14:28:31.922233251 +0000 UTC m=+21.776558233, 1.005014919s after stability
2025-01-25T14:28:31.922Z [INFO]  creating new snapshot: path=/tmp/raft830560830/snapshots/2-13-1737815311922.tmp
2025-01-25T14:28:31.924Z [INFO]  server-972da02a-ccc4-5e53-1f90-1452016ead72: copied to local snapshot: bytes=71
2025-01-25T14:28:31.924Z [INFO]  server-972da02a-ccc4-5e53-1f90-1452016ead72: snapshot restore progress: id=2-13-1737815311922 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:31.924Z [INFO]  server-972da02a-ccc4-5e53-1f90-1452016ead72: restored user snapshot: index=13
2025-01-25T14:28:31.924Z [ERROR] server-972da02a-ccc4-5e53-1f90-1452016ead72: failed to get log: index=3 error="log not found"
2025-01-25T14:28:31.924Z [ERROR] server-972da02a-ccc4-5e53-1f90-1452016ead72: failed to get log: index=3 error="log not found"
2025-01-25T14:28:31.924Z [INFO]  server-972da02a-ccc4-5e53-1f90-1452016ead72: aborting pipeline replication: peer="{Voter server-eef17de3-64ee-8f3e-7579-3c1c746e4acc eef17de3-64ee-8f3e-7579-3c1c746e4acc}"
2025-01-25T14:28:31.924Z [INFO]  server-972da02a-ccc4-5e53-1f90-1452016ead72: aborting pipeline replication: peer="{Voter server-66ece0d9-ebca-02c7-a98d-14a1ce7a503b 66ece0d9-ebca-02c7-a98d-14a1ce7a503b}"
2025-01-25T14:28:31.931Z [ERROR] server-972da02a-ccc4-5e53-1f90-1452016ead72: failed to get log: index=3 error="log not found"
2025-01-25T14:28:31.932Z [INFO]  creating new snapshot: path=/tmp/raft1915430452/snapshots/2-13-1737815311932.tmp
2025-01-25T14:28:31.933Z [ERROR] server-972da02a-ccc4-5e53-1f90-1452016ead72: failed to get log: index=3 error="log not found"
2025-01-25T14:28:31.934Z [INFO]  server-66ece0d9-ebca-02c7-a98d-14a1ce7a503b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:31.935Z [INFO]  creating new snapshot: path=/tmp/raft1345354572/snapshots/2-13-1737815311935.tmp
2025-01-25T14:28:31.935Z [INFO]  server-66ece0d9-ebca-02c7-a98d-14a1ce7a503b: copied to local snapshot: bytes=71
2025-01-25T14:28:31.935Z [INFO]  server-eef17de3-64ee-8f3e-7579-3c1c746e4acc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:31.936Z [INFO]  server-66ece0d9-ebca-02c7-a98d-14a1ce7a503b: snapshot restore progress: id=2-13-1737815311932 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:31.936Z [INFO]  server-66ece0d9-ebca-02c7-a98d-14a1ce7a503b: Installed remote snapshot
2025-01-25T14:28:31.936Z [INFO]  server-eef17de3-64ee-8f3e-7579-3c1c746e4acc: copied to local snapshot: bytes=71
2025-01-25T14:28:31.936Z [INFO]  server-972da02a-ccc4-5e53-1f90-1452016ead72: pipelining replication: peer="{Voter server-66ece0d9-ebca-02c7-a98d-14a1ce7a503b 66ece0d9-ebca-02c7-a98d-14a1ce7a503b}"
2025-01-25T14:28:31.936Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:31.936Z [INFO]  server-eef17de3-64ee-8f3e-7579-3c1c746e4acc: snapshot restore progress: id=2-13-1737815311935 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:31.936Z [INFO]  server-eef17de3-64ee-8f3e-7579-3c1c746e4acc: Installed remote snapshot
2025-01-25T14:28:31.936Z [INFO]  server-972da02a-ccc4-5e53-1f90-1452016ead72: pipelining replication: peer="{Voter server-eef17de3-64ee-8f3e-7579-3c1c746e4acc eef17de3-64ee-8f3e-7579-3c1c746e4acc}"
2025-01-25T14:28:32.941Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:31.936556932 +0000 UTC m=+21.790881914 (2 nodes), 1.8µs from start of poll, 1.534800251s from cluster start. Timeout at 2025-01-25 14:28:32.941595051 +0000 UTC m=+22.795920033, 1.005038119s after stability
2025-01-25T14:28:32.954Z [INFO]  server-95b0d786-5552-3df0-7e2a-df0bbd4203df: aborting pipeline replication: peer="{Voter server-a640e938-62dd-b60f-a2a4-83e2d644b629 a640e938-62dd-b60f-a2a4-83e2d644b629}"
2025-01-25T14:28:32.954Z [INFO]  server-95b0d786-5552-3df0-7e2a-df0bbd4203df: aborting pipeline replication: peer="{Voter server-dd0b3047-2bc0-c9c9-a469-a4a4a4d223f0 dd0b3047-2bc0-c9c9-a469-a4a4a4d223f0}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true
2025-01-25T14:28:32.957Z [INFO]  server-79269506-73d6-fa20-351f-cbab4553a4b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79269506-73d6-fa20-351f-cbab4553a4b1 Address:79269506-73d6-fa20-351f-cbab4553a4b1} {Suffrage:Voter ID:server-2d05640c-3080-5f64-fa40-3f7a458d415d Address:2d05640c-3080-5f64-fa40-3f7a458d415d} {Suffrage:Voter ID:server-5550ac9f-b445-20c7-7776-71e81e3228c3 Address:5550ac9f-b445-20c7-7776-71e81e3228c3}]"
2025-01-25T14:28:32.957Z [INFO]  server-79269506-73d6-fa20-351f-cbab4553a4b1: entering follower state: follower="Node at 79269506-73d6-fa20-351f-cbab4553a4b1 [Follower]" leader-address= leader-id=
2025-01-25T14:28:32.957Z [INFO]  server-2d05640c-3080-5f64-fa40-3f7a458d415d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79269506-73d6-fa20-351f-cbab4553a4b1 Address:79269506-73d6-fa20-351f-cbab4553a4b1} {Suffrage:Voter ID:server-2d05640c-3080-5f64-fa40-3f7a458d415d Address:2d05640c-3080-5f64-fa40-3f7a458d415d} {Suffrage:Voter ID:server-5550ac9f-b445-20c7-7776-71e81e3228c3 Address:5550ac9f-b445-20c7-7776-71e81e3228c3}]"
2025-01-25T14:28:32.957Z [INFO]  server-2d05640c-3080-5f64-fa40-3f7a458d415d: entering follower state: follower="Node at 2d05640c-3080-5f64-fa40-3f7a458d415d [Follower]" leader-address= leader-id=
2025-01-25T14:28:32.957Z [INFO]  server-5550ac9f-b445-20c7-7776-71e81e3228c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79269506-73d6-fa20-351f-cbab4553a4b1 Address:79269506-73d6-fa20-351f-cbab4553a4b1} {Suffrage:Voter ID:server-2d05640c-3080-5f64-fa40-3f7a458d415d Address:2d05640c-3080-5f64-fa40-3f7a458d415d} {Suffrage:Voter ID:server-5550ac9f-b445-20c7-7776-71e81e3228c3 Address:5550ac9f-b445-20c7-7776-71e81e3228c3}]"
2025-01-25T14:28:32.957Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:32.957Z [INFO]  server-5550ac9f-b445-20c7-7776-71e81e3228c3: entering follower state: follower="Node at 5550ac9f-b445-20c7-7776-71e81e3228c3 [Follower]" leader-address= leader-id=
2025-01-25T14:28:33.657Z [WARN]  server-79269506-73d6-fa20-351f-cbab4553a4b1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:33.657Z [INFO]  server-79269506-73d6-fa20-351f-cbab4553a4b1: entering candidate state: node="Node at 79269506-73d6-fa20-351f-cbab4553a4b1 [Candidate]" term=2
2025-01-25T14:28:33.657Z [INFO]  server-79269506-73d6-fa20-351f-cbab4553a4b1: election won: term=2 tally=2
2025-01-25T14:28:33.657Z [INFO]  server-79269506-73d6-fa20-351f-cbab4553a4b1: entering leader state: leader="Node at 79269506-73d6-fa20-351f-cbab4553a4b1 [Leader]"
2025-01-25T14:28:33.657Z [INFO]  server-79269506-73d6-fa20-351f-cbab4553a4b1: added peer, starting replication: peer=server-2d05640c-3080-5f64-fa40-3f7a458d415d
2025-01-25T14:28:33.657Z [INFO]  server-79269506-73d6-fa20-351f-cbab4553a4b1: added peer, starting replication: peer=server-5550ac9f-b445-20c7-7776-71e81e3228c3
2025-01-25T14:28:33.658Z [INFO]  server-79269506-73d6-fa20-351f-cbab4553a4b1: pipelining replication: peer="{Voter server-5550ac9f-b445-20c7-7776-71e81e3228c3 5550ac9f-b445-20c7-7776-71e81e3228c3}"
2025-01-25T14:28:33.658Z [INFO]  server-79269506-73d6-fa20-351f-cbab4553a4b1: pipelining replication: peer="{Voter server-2d05640c-3080-5f64-fa40-3f7a458d415d 2d05640c-3080-5f64-fa40-3f7a458d415d}"
2025-01-25T14:28:34.663Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:33.657867511 +0000 UTC m=+23.512192493 (1 nodes), 700.121886ms from start of poll, 700.824861ms from cluster start. Timeout at 2025-01-25 14:28:34.662876271 +0000 UTC m=+24.517201253, 1.00500876s after stability
2025-01-25T14:28:34.665Z [INFO]  server-79269506-73d6-fa20-351f-cbab4553a4b1: starting snapshot up to: index=12
2025-01-25T14:28:34.665Z [INFO]  creating new snapshot: path=/tmp/raft1583965139/snapshots/2-12-1737815314665.tmp
2025-01-25T14:28:34.666Z [INFO]  server-79269506-73d6-fa20-351f-cbab4553a4b1: snapshot complete up to: index=12
2025-01-25T14:28:34.668Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e302c46-c6a3-3d89-52e8-d43528206fe0 Address:1e302c46-c6a3-3d89-52e8-d43528206fe0} {Suffrage:Voter ID:server-af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2 Address:af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2} {Suffrage:Voter ID:server-23ad6e03-93c7-f0a6-af79-eec31ce61721 Address:23ad6e03-93c7-f0a6-af79-eec31ce61721}]"
2025-01-25T14:28:34.669Z [INFO]  server-af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e302c46-c6a3-3d89-52e8-d43528206fe0 Address:1e302c46-c6a3-3d89-52e8-d43528206fe0} {Suffrage:Voter ID:server-af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2 Address:af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2} {Suffrage:Voter ID:server-23ad6e03-93c7-f0a6-af79-eec31ce61721 Address:23ad6e03-93c7-f0a6-af79-eec31ce61721}]"
2025-01-25T14:28:34.669Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: entering follower state: follower="Node at 1e302c46-c6a3-3d89-52e8-d43528206fe0 [Follower]" leader-address= leader-id=
2025-01-25T14:28:34.669Z [INFO]  server-af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2: entering follower state: follower="Node at af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2 [Follower]" leader-address= leader-id=
2025-01-25T14:28:34.669Z [INFO]  server-23ad6e03-93c7-f0a6-af79-eec31ce61721: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e302c46-c6a3-3d89-52e8-d43528206fe0 Address:1e302c46-c6a3-3d89-52e8-d43528206fe0} {Suffrage:Voter ID:server-af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2 Address:af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2} {Suffrage:Voter ID:server-23ad6e03-93c7-f0a6-af79-eec31ce61721 Address:23ad6e03-93c7-f0a6-af79-eec31ce61721}]"
2025-01-25T14:28:34.669Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:34.669Z [INFO]  server-23ad6e03-93c7-f0a6-af79-eec31ce61721: entering follower state: follower="Node at 23ad6e03-93c7-f0a6-af79-eec31ce61721 [Follower]" leader-address= leader-id=
2025-01-25T14:28:35.171Z [WARN]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:35.171Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: entering candidate state: node="Node at 1e302c46-c6a3-3d89-52e8-d43528206fe0 [Candidate]" term=2
2025-01-25T14:28:35.171Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: election won: term=2 tally=2
2025-01-25T14:28:35.171Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: entering leader state: leader="Node at 1e302c46-c6a3-3d89-52e8-d43528206fe0 [Leader]"
2025-01-25T14:28:35.171Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: added peer, starting replication: peer=server-af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2
2025-01-25T14:28:35.172Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: added peer, starting replication: peer=server-23ad6e03-93c7-f0a6-af79-eec31ce61721
2025-01-25T14:28:35.172Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: pipelining replication: peer="{Voter server-af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2 af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2}"
2025-01-25T14:28:35.172Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: pipelining replication: peer="{Voter server-23ad6e03-93c7-f0a6-af79-eec31ce61721 23ad6e03-93c7-f0a6-af79-eec31ce61721}"
2025-01-25T14:28:36.177Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:35.172051464 +0000 UTC m=+25.026376446 (1 nodes), 502.632713ms from start of poll, 503.361287ms from cluster start. Timeout at 2025-01-25 14:28:36.177059824 +0000 UTC m=+26.031384806, 1.00500836s after stability
2025-01-25T14:28:36.178Z [INFO]  creating new snapshot: path=/tmp/raft1132033211/snapshots/2-13-1737815316178.tmp
2025-01-25T14:28:36.179Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: copied to local snapshot: bytes=71
2025-01-25T14:28:36.180Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: snapshot restore progress: id=2-13-1737815316178 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:36.180Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: removing all old logs from log store
2025-01-25T14:28:36.180Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: compacting logs: from=1 to=2
2025-01-25T14:28:36.180Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: restored user snapshot: index=13
2025-01-25T14:28:36.180Z [ERROR] server-1e302c46-c6a3-3d89-52e8-d43528206fe0: failed to get log: index=2 error="log not found"
2025-01-25T14:28:36.180Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: aborting pipeline replication: peer="{Voter server-23ad6e03-93c7-f0a6-af79-eec31ce61721 23ad6e03-93c7-f0a6-af79-eec31ce61721}"
2025-01-25T14:28:36.180Z [ERROR] server-1e302c46-c6a3-3d89-52e8-d43528206fe0: failed to get log: index=2 error="log not found"
2025-01-25T14:28:36.180Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: aborting pipeline replication: peer="{Voter server-af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2 af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2}"
2025-01-25T14:28:36.187Z [ERROR] server-1e302c46-c6a3-3d89-52e8-d43528206fe0: failed to get log: index=2 error="log not found"
2025-01-25T14:28:36.187Z [INFO]  creating new snapshot: path=/tmp/raft1872886137/snapshots/2-13-1737815316187.tmp
2025-01-25T14:28:36.188Z [INFO]  server-23ad6e03-93c7-f0a6-af79-eec31ce61721: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:36.189Z [INFO]  server-23ad6e03-93c7-f0a6-af79-eec31ce61721: copied to local snapshot: bytes=71
2025-01-25T14:28:36.189Z [ERROR] server-1e302c46-c6a3-3d89-52e8-d43528206fe0: failed to get log: index=2 error="log not found"
2025-01-25T14:28:36.189Z [INFO]  server-23ad6e03-93c7-f0a6-af79-eec31ce61721: snapshot restore progress: id=2-13-1737815316187 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:36.189Z [INFO]  server-23ad6e03-93c7-f0a6-af79-eec31ce61721: removing all old logs from log store
2025-01-25T14:28:36.189Z [INFO]  server-23ad6e03-93c7-f0a6-af79-eec31ce61721: compacting logs: from=1 to=2
2025-01-25T14:28:36.189Z [INFO]  server-23ad6e03-93c7-f0a6-af79-eec31ce61721: Installed remote snapshot
2025-01-25T14:28:36.190Z [INFO]  creating new snapshot: path=/tmp/raft1346027855/snapshots/2-13-1737815316190.tmp
2025-01-25T14:28:36.190Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: pipelining replication: peer="{Voter server-23ad6e03-93c7-f0a6-af79-eec31ce61721 23ad6e03-93c7-f0a6-af79-eec31ce61721}"
2025-01-25T14:28:36.190Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:36.190Z [INFO]  server-af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:36.191Z [INFO]  server-af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2: copied to local snapshot: bytes=71
2025-01-25T14:28:36.191Z [INFO]  server-af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2: snapshot restore progress: id=2-13-1737815316190 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:36.191Z [INFO]  server-af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2: removing all old logs from log store
2025-01-25T14:28:36.191Z [INFO]  server-af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2: compacting logs: from=1 to=2
2025-01-25T14:28:36.191Z [INFO]  server-af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2: Installed remote snapshot
2025-01-25T14:28:36.192Z [INFO]  server-1e302c46-c6a3-3d89-52e8-d43528206fe0: pipelining replication: peer="{Voter server-af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2 af0e5466-4af7-3e6e-0d9f-d32f2ffeddb2}"
2025-01-25T14:28:37.196Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:36.190309302 +0000 UTC m=+26.044634284 (2 nodes), 1.84µs from start of poll, 1.521619125s from cluster start. Timeout at 2025-01-25 14:28:37.1953529 +0000 UTC m=+27.049677882, 1.005043598s after stability
2025-01-25T14:28:37.208Z [INFO]  server-79269506-73d6-fa20-351f-cbab4553a4b1: aborting pipeline replication: peer="{Voter server-5550ac9f-b445-20c7-7776-71e81e3228c3 5550ac9f-b445-20c7-7776-71e81e3228c3}"
2025-01-25T14:28:37.208Z [INFO]  server-79269506-73d6-fa20-351f-cbab4553a4b1: aborting pipeline replication: peer="{Voter server-2d05640c-3080-5f64-fa40-3f7a458d415d 2d05640c-3080-5f64-fa40-3f7a458d415d}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false
2025-01-25T14:28:37.217Z [INFO]  server-68d235ef-4029-115e-d7fc-8ba5151dcce7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68d235ef-4029-115e-d7fc-8ba5151dcce7 Address:68d235ef-4029-115e-d7fc-8ba5151dcce7} {Suffrage:Voter ID:server-46503a29-1cb2-fd58-9484-a783c709e240 Address:46503a29-1cb2-fd58-9484-a783c709e240} {Suffrage:Voter ID:server-4fb5308e-41f8-8c84-e79b-dba926c1efdf Address:4fb5308e-41f8-8c84-e79b-dba926c1efdf}]"
2025-01-25T14:28:37.217Z [INFO]  server-68d235ef-4029-115e-d7fc-8ba5151dcce7: entering follower state: follower="Node at 68d235ef-4029-115e-d7fc-8ba5151dcce7 [Follower]" leader-address= leader-id=
2025-01-25T14:28:37.217Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68d235ef-4029-115e-d7fc-8ba5151dcce7 Address:68d235ef-4029-115e-d7fc-8ba5151dcce7} {Suffrage:Voter ID:server-46503a29-1cb2-fd58-9484-a783c709e240 Address:46503a29-1cb2-fd58-9484-a783c709e240} {Suffrage:Voter ID:server-4fb5308e-41f8-8c84-e79b-dba926c1efdf Address:4fb5308e-41f8-8c84-e79b-dba926c1efdf}]"
2025-01-25T14:28:37.217Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: entering follower state: follower="Node at 46503a29-1cb2-fd58-9484-a783c709e240 [Follower]" leader-address= leader-id=
2025-01-25T14:28:37.217Z [INFO]  server-4fb5308e-41f8-8c84-e79b-dba926c1efdf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68d235ef-4029-115e-d7fc-8ba5151dcce7 Address:68d235ef-4029-115e-d7fc-8ba5151dcce7} {Suffrage:Voter ID:server-46503a29-1cb2-fd58-9484-a783c709e240 Address:46503a29-1cb2-fd58-9484-a783c709e240} {Suffrage:Voter ID:server-4fb5308e-41f8-8c84-e79b-dba926c1efdf Address:4fb5308e-41f8-8c84-e79b-dba926c1efdf}]"
2025-01-25T14:28:37.217Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:37.217Z [INFO]  server-4fb5308e-41f8-8c84-e79b-dba926c1efdf: entering follower state: follower="Node at 4fb5308e-41f8-8c84-e79b-dba926c1efdf [Follower]" leader-address= leader-id=
2025-01-25T14:28:38.072Z [WARN]  server-46503a29-1cb2-fd58-9484-a783c709e240: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:38.072Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: entering candidate state: node="Node at 46503a29-1cb2-fd58-9484-a783c709e240 [Candidate]" term=2
2025-01-25T14:28:38.072Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: election won: term=2 tally=2
2025-01-25T14:28:38.072Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: entering leader state: leader="Node at 46503a29-1cb2-fd58-9484-a783c709e240 [Leader]"
2025-01-25T14:28:38.072Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: added peer, starting replication: peer=server-68d235ef-4029-115e-d7fc-8ba5151dcce7
2025-01-25T14:28:38.072Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: added peer, starting replication: peer=server-4fb5308e-41f8-8c84-e79b-dba926c1efdf
2025-01-25T14:28:38.072Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: pipelining replication: peer="{Voter server-68d235ef-4029-115e-d7fc-8ba5151dcce7 68d235ef-4029-115e-d7fc-8ba5151dcce7}"
2025-01-25T14:28:38.073Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: pipelining replication: peer="{Voter server-4fb5308e-41f8-8c84-e79b-dba926c1efdf 4fb5308e-41f8-8c84-e79b-dba926c1efdf}"
2025-01-25T14:28:39.078Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:38.072863039 +0000 UTC m=+27.927188021 (1 nodes), 855.042472ms from start of poll, 855.893721ms from cluster start. Timeout at 2025-01-25 14:28:39.077869359 +0000 UTC m=+28.932194341, 1.00500632s after stability
2025-01-25T14:28:39.079Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: starting snapshot up to: index=12
2025-01-25T14:28:39.079Z [INFO]  creating new snapshot: path=/tmp/raft4141175798/snapshots/2-12-1737815319079.tmp
2025-01-25T14:28:39.080Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: snapshot complete up to: index=12
2025-01-25T14:28:39.082Z [INFO]  creating new snapshot: path=/tmp/raft4141175798/snapshots/2-23-1737815319082.tmp
2025-01-25T14:28:39.083Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: copied to local snapshot: bytes=71
2025-01-25T14:28:39.083Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: snapshot restore progress: id=2-23-1737815319082 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:39.083Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: restored user snapshot: index=23
2025-01-25T14:28:39.083Z [ERROR] server-46503a29-1cb2-fd58-9484-a783c709e240: failed to get log: index=23 error="log not found"
2025-01-25T14:28:39.083Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: aborting pipeline replication: peer="{Voter server-4fb5308e-41f8-8c84-e79b-dba926c1efdf 4fb5308e-41f8-8c84-e79b-dba926c1efdf}"
2025-01-25T14:28:39.083Z [ERROR] server-46503a29-1cb2-fd58-9484-a783c709e240: failed to get log: index=23 error="log not found"
2025-01-25T14:28:39.083Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: aborting pipeline replication: peer="{Voter server-68d235ef-4029-115e-d7fc-8ba5151dcce7 68d235ef-4029-115e-d7fc-8ba5151dcce7}"
2025-01-25T14:28:39.092Z [ERROR] server-46503a29-1cb2-fd58-9484-a783c709e240: failed to get log: index=23 error="log not found"
2025-01-25T14:28:39.093Z [INFO]  creating new snapshot: path=/tmp/raft2560695296/snapshots/2-23-1737815319093.tmp
2025-01-25T14:28:39.093Z [ERROR] server-46503a29-1cb2-fd58-9484-a783c709e240: failed to get log: index=23 error="log not found"
2025-01-25T14:28:39.094Z [INFO]  server-4fb5308e-41f8-8c84-e79b-dba926c1efdf: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:39.094Z [INFO]  creating new snapshot: path=/tmp/raft4041373679/snapshots/2-23-1737815319094.tmp
2025-01-25T14:28:39.094Z [INFO]  server-4fb5308e-41f8-8c84-e79b-dba926c1efdf: copied to local snapshot: bytes=71
2025-01-25T14:28:39.094Z [INFO]  server-68d235ef-4029-115e-d7fc-8ba5151dcce7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:39.095Z [INFO]  server-4fb5308e-41f8-8c84-e79b-dba926c1efdf: snapshot restore progress: id=2-23-1737815319093 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:39.095Z [INFO]  server-68d235ef-4029-115e-d7fc-8ba5151dcce7: copied to local snapshot: bytes=71
2025-01-25T14:28:39.095Z [INFO]  server-4fb5308e-41f8-8c84-e79b-dba926c1efdf: Installed remote snapshot
2025-01-25T14:28:39.095Z [WARN]  server-4fb5308e-41f8-8c84-e79b-dba926c1efdf: skipping application of old log: index=22
2025-01-25T14:28:39.095Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: pipelining replication: peer="{Voter server-4fb5308e-41f8-8c84-e79b-dba926c1efdf 4fb5308e-41f8-8c84-e79b-dba926c1efdf}"
2025-01-25T14:28:39.095Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:39.095Z [INFO]  server-68d235ef-4029-115e-d7fc-8ba5151dcce7: snapshot restore progress: id=2-23-1737815319094 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:39.095Z [INFO]  server-68d235ef-4029-115e-d7fc-8ba5151dcce7: Installed remote snapshot
2025-01-25T14:28:39.096Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: pipelining replication: peer="{Voter server-68d235ef-4029-115e-d7fc-8ba5151dcce7 68d235ef-4029-115e-d7fc-8ba5151dcce7}"
2025-01-25T14:28:40.101Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:39.095867306 +0000 UTC m=+28.950192288 (2 nodes), 1.64µs from start of poll, 1.878897988s from cluster start. Timeout at 2025-01-25 14:28:40.100938263 +0000 UTC m=+29.955263205, 1.005070917s after stability
2025-01-25T14:28:40.117Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: aborting pipeline replication: peer="{Voter server-4fb5308e-41f8-8c84-e79b-dba926c1efdf 4fb5308e-41f8-8c84-e79b-dba926c1efdf}"
2025-01-25T14:28:40.117Z [INFO]  server-46503a29-1cb2-fd58-9484-a783c709e240: aborting pipeline replication: peer="{Voter server-68d235ef-4029-115e-d7fc-8ba5151dcce7 68d235ef-4029-115e-d7fc-8ba5151dcce7}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false
2025-01-25T14:28:40.120Z [INFO]  server-6f11e85b-dd51-5a6d-b735-f79e3596b986: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f11e85b-dd51-5a6d-b735-f79e3596b986 Address:6f11e85b-dd51-5a6d-b735-f79e3596b986} {Suffrage:Voter ID:server-e3ff4c4f-117d-646c-624b-67a40bc7b260 Address:e3ff4c4f-117d-646c-624b-67a40bc7b260} {Suffrage:Voter ID:server-27903de3-84e1-954a-c9ab-0f496cc530ed Address:27903de3-84e1-954a-c9ab-0f496cc530ed}]"
2025-01-25T14:28:40.120Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f11e85b-dd51-5a6d-b735-f79e3596b986 Address:6f11e85b-dd51-5a6d-b735-f79e3596b986} {Suffrage:Voter ID:server-e3ff4c4f-117d-646c-624b-67a40bc7b260 Address:e3ff4c4f-117d-646c-624b-67a40bc7b260} {Suffrage:Voter ID:server-27903de3-84e1-954a-c9ab-0f496cc530ed Address:27903de3-84e1-954a-c9ab-0f496cc530ed}]"
2025-01-25T14:28:40.120Z [INFO]  server-6f11e85b-dd51-5a6d-b735-f79e3596b986: entering follower state: follower="Node at 6f11e85b-dd51-5a6d-b735-f79e3596b986 [Follower]" leader-address= leader-id=
2025-01-25T14:28:40.121Z [INFO]  server-27903de3-84e1-954a-c9ab-0f496cc530ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f11e85b-dd51-5a6d-b735-f79e3596b986 Address:6f11e85b-dd51-5a6d-b735-f79e3596b986} {Suffrage:Voter ID:server-e3ff4c4f-117d-646c-624b-67a40bc7b260 Address:e3ff4c4f-117d-646c-624b-67a40bc7b260} {Suffrage:Voter ID:server-27903de3-84e1-954a-c9ab-0f496cc530ed Address:27903de3-84e1-954a-c9ab-0f496cc530ed}]"
2025-01-25T14:28:40.121Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: entering follower state: follower="Node at e3ff4c4f-117d-646c-624b-67a40bc7b260 [Follower]" leader-address= leader-id=
2025-01-25T14:28:40.121Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:40.121Z [INFO]  server-27903de3-84e1-954a-c9ab-0f496cc530ed: entering follower state: follower="Node at 27903de3-84e1-954a-c9ab-0f496cc530ed [Follower]" leader-address= leader-id=
2025-01-25T14:28:40.678Z [WARN]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:40.678Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: entering candidate state: node="Node at e3ff4c4f-117d-646c-624b-67a40bc7b260 [Candidate]" term=2
2025-01-25T14:28:40.678Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: election won: term=2 tally=2
2025-01-25T14:28:40.678Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: entering leader state: leader="Node at e3ff4c4f-117d-646c-624b-67a40bc7b260 [Leader]"
2025-01-25T14:28:40.678Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: added peer, starting replication: peer=server-6f11e85b-dd51-5a6d-b735-f79e3596b986
2025-01-25T14:28:40.678Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: added peer, starting replication: peer=server-27903de3-84e1-954a-c9ab-0f496cc530ed
2025-01-25T14:28:40.678Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: pipelining replication: peer="{Voter server-6f11e85b-dd51-5a6d-b735-f79e3596b986 6f11e85b-dd51-5a6d-b735-f79e3596b986}"
2025-01-25T14:28:40.678Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: pipelining replication: peer="{Voter server-27903de3-84e1-954a-c9ab-0f496cc530ed 27903de3-84e1-954a-c9ab-0f496cc530ed}"
2025-01-25T14:28:41.684Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:40.678730267 +0000 UTC m=+30.533055249 (1 nodes), 557.632414ms from start of poll, 558.266191ms from cluster start. Timeout at 2025-01-25 14:28:41.683734867 +0000 UTC m=+31.538059809, 1.00500456s after stability
2025-01-25T14:28:41.685Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: starting snapshot up to: index=12
2025-01-25T14:28:41.685Z [INFO]  creating new snapshot: path=/tmp/raft2218033496/snapshots/2-12-1737815321685.tmp
2025-01-25T14:28:41.686Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: snapshot complete up to: index=12
2025-01-25T14:28:41.687Z [INFO]  creating new snapshot: path=/tmp/raft2218033496/snapshots/2-23-1737815321687.tmp
2025-01-25T14:28:41.688Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: copied to local snapshot: bytes=71
2025-01-25T14:28:41.689Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: snapshot restore progress: id=2-23-1737815321687 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:41.689Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: removing all old logs from log store
2025-01-25T14:28:41.689Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: compacting logs: from=1 to=22
2025-01-25T14:28:41.689Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: restored user snapshot: index=23
2025-01-25T14:28:41.689Z [ERROR] server-e3ff4c4f-117d-646c-624b-67a40bc7b260: failed to get log: index=22 error="log not found"
2025-01-25T14:28:41.689Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: aborting pipeline replication: peer="{Voter server-27903de3-84e1-954a-c9ab-0f496cc530ed 27903de3-84e1-954a-c9ab-0f496cc530ed}"
2025-01-25T14:28:41.689Z [ERROR] server-e3ff4c4f-117d-646c-624b-67a40bc7b260: failed to get log: index=22 error="log not found"
2025-01-25T14:28:41.689Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: aborting pipeline replication: peer="{Voter server-6f11e85b-dd51-5a6d-b735-f79e3596b986 6f11e85b-dd51-5a6d-b735-f79e3596b986}"
2025-01-25T14:28:41.696Z [ERROR] server-e3ff4c4f-117d-646c-624b-67a40bc7b260: failed to get log: index=22 error="log not found"
2025-01-25T14:28:41.697Z [INFO]  creating new snapshot: path=/tmp/raft1304655243/snapshots/2-23-1737815321697.tmp
2025-01-25T14:28:41.697Z [ERROR] server-e3ff4c4f-117d-646c-624b-67a40bc7b260: failed to get log: index=22 error="log not found"
2025-01-25T14:28:41.697Z [INFO]  server-6f11e85b-dd51-5a6d-b735-f79e3596b986: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:41.698Z [INFO]  creating new snapshot: path=/tmp/raft1978040639/snapshots/2-23-1737815321698.tmp
2025-01-25T14:28:41.698Z [INFO]  server-6f11e85b-dd51-5a6d-b735-f79e3596b986: copied to local snapshot: bytes=71
2025-01-25T14:28:41.698Z [INFO]  server-27903de3-84e1-954a-c9ab-0f496cc530ed: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:41.699Z [INFO]  server-6f11e85b-dd51-5a6d-b735-f79e3596b986: snapshot restore progress: id=2-23-1737815321697 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:41.699Z [INFO]  server-6f11e85b-dd51-5a6d-b735-f79e3596b986: removing all old logs from log store
2025-01-25T14:28:41.699Z [INFO]  server-6f11e85b-dd51-5a6d-b735-f79e3596b986: compacting logs: from=1 to=22
2025-01-25T14:28:41.699Z [INFO]  server-6f11e85b-dd51-5a6d-b735-f79e3596b986: Installed remote snapshot
2025-01-25T14:28:41.700Z [INFO]  server-27903de3-84e1-954a-c9ab-0f496cc530ed: copied to local snapshot: bytes=71
2025-01-25T14:28:41.700Z [WARN]  server-6f11e85b-dd51-5a6d-b735-f79e3596b986: skipping application of old log: index=22
2025-01-25T14:28:41.700Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: pipelining replication: peer="{Voter server-6f11e85b-dd51-5a6d-b735-f79e3596b986 6f11e85b-dd51-5a6d-b735-f79e3596b986}"
2025-01-25T14:28:41.700Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:41.701Z [INFO]  server-27903de3-84e1-954a-c9ab-0f496cc530ed: snapshot restore progress: id=2-23-1737815321698 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:41.701Z [INFO]  server-27903de3-84e1-954a-c9ab-0f496cc530ed: removing all old logs from log store
2025-01-25T14:28:41.701Z [INFO]  server-27903de3-84e1-954a-c9ab-0f496cc530ed: compacting logs: from=1 to=22
2025-01-25T14:28:41.701Z [INFO]  server-27903de3-84e1-954a-c9ab-0f496cc530ed: Installed remote snapshot
2025-01-25T14:28:41.702Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: pipelining replication: peer="{Voter server-27903de3-84e1-954a-c9ab-0f496cc530ed 27903de3-84e1-954a-c9ab-0f496cc530ed}"
2025-01-25T14:28:42.706Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:41.701034199 +0000 UTC m=+31.555359141 (2 nodes), 3.2µs from start of poll, 1.580570083s from cluster start. Timeout at 2025-01-25 14:28:42.706082197 +0000 UTC m=+32.560407259, 1.005048118s after stability
2025-01-25T14:28:42.723Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: aborting pipeline replication: peer="{Voter server-27903de3-84e1-954a-c9ab-0f496cc530ed 27903de3-84e1-954a-c9ab-0f496cc530ed}"
2025-01-25T14:28:42.723Z [INFO]  server-e3ff4c4f-117d-646c-624b-67a40bc7b260: aborting pipeline replication: peer="{Voter server-6f11e85b-dd51-5a6d-b735-f79e3596b986 6f11e85b-dd51-5a6d-b735-f79e3596b986}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true
2025-01-25T14:28:42.726Z [INFO]  server-4a9d605f-d0a6-dbff-0ad6-70e49d73585c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a9d605f-d0a6-dbff-0ad6-70e49d73585c Address:4a9d605f-d0a6-dbff-0ad6-70e49d73585c} {Suffrage:Voter ID:server-b854593a-a3cd-cb49-f242-46ef9085ebe8 Address:b854593a-a3cd-cb49-f242-46ef9085ebe8} {Suffrage:Voter ID:server-a367d3ea-d4f2-66de-82fd-f95fe55af871 Address:a367d3ea-d4f2-66de-82fd-f95fe55af871}]"
2025-01-25T14:28:42.727Z [INFO]  server-b854593a-a3cd-cb49-f242-46ef9085ebe8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a9d605f-d0a6-dbff-0ad6-70e49d73585c Address:4a9d605f-d0a6-dbff-0ad6-70e49d73585c} {Suffrage:Voter ID:server-b854593a-a3cd-cb49-f242-46ef9085ebe8 Address:b854593a-a3cd-cb49-f242-46ef9085ebe8} {Suffrage:Voter ID:server-a367d3ea-d4f2-66de-82fd-f95fe55af871 Address:a367d3ea-d4f2-66de-82fd-f95fe55af871}]"
2025-01-25T14:28:42.727Z [INFO]  server-4a9d605f-d0a6-dbff-0ad6-70e49d73585c: entering follower state: follower="Node at 4a9d605f-d0a6-dbff-0ad6-70e49d73585c [Follower]" leader-address= leader-id=
2025-01-25T14:28:42.727Z [INFO]  server-b854593a-a3cd-cb49-f242-46ef9085ebe8: entering follower state: follower="Node at b854593a-a3cd-cb49-f242-46ef9085ebe8 [Follower]" leader-address= leader-id=
2025-01-25T14:28:42.727Z [INFO]  server-a367d3ea-d4f2-66de-82fd-f95fe55af871: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a9d605f-d0a6-dbff-0ad6-70e49d73585c Address:4a9d605f-d0a6-dbff-0ad6-70e49d73585c} {Suffrage:Voter ID:server-b854593a-a3cd-cb49-f242-46ef9085ebe8 Address:b854593a-a3cd-cb49-f242-46ef9085ebe8} {Suffrage:Voter ID:server-a367d3ea-d4f2-66de-82fd-f95fe55af871 Address:a367d3ea-d4f2-66de-82fd-f95fe55af871}]"
2025-01-25T14:28:42.727Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:42.727Z [INFO]  server-a367d3ea-d4f2-66de-82fd-f95fe55af871: entering follower state: follower="Node at a367d3ea-d4f2-66de-82fd-f95fe55af871 [Follower]" leader-address= leader-id=
2025-01-25T14:28:43.615Z [WARN]  server-4a9d605f-d0a6-dbff-0ad6-70e49d73585c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:43.615Z [INFO]  server-4a9d605f-d0a6-dbff-0ad6-70e49d73585c: entering candidate state: node="Node at 4a9d605f-d0a6-dbff-0ad6-70e49d73585c [Candidate]" term=2
2025-01-25T14:28:43.615Z [INFO]  server-4a9d605f-d0a6-dbff-0ad6-70e49d73585c: election won: term=2 tally=2
2025-01-25T14:28:43.615Z [INFO]  server-4a9d605f-d0a6-dbff-0ad6-70e49d73585c: entering leader state: leader="Node at 4a9d605f-d0a6-dbff-0ad6-70e49d73585c [Leader]"
2025-01-25T14:28:43.615Z [INFO]  server-4a9d605f-d0a6-dbff-0ad6-70e49d73585c: added peer, starting replication: peer=server-b854593a-a3cd-cb49-f242-46ef9085ebe8
2025-01-25T14:28:43.615Z [INFO]  server-4a9d605f-d0a6-dbff-0ad6-70e49d73585c: added peer, starting replication: peer=server-a367d3ea-d4f2-66de-82fd-f95fe55af871
2025-01-25T14:28:43.616Z [INFO]  server-4a9d605f-d0a6-dbff-0ad6-70e49d73585c: pipelining replication: peer="{Voter server-b854593a-a3cd-cb49-f242-46ef9085ebe8 b854593a-a3cd-cb49-f242-46ef9085ebe8}"
2025-01-25T14:28:43.616Z [INFO]  server-4a9d605f-d0a6-dbff-0ad6-70e49d73585c: pipelining replication: peer="{Voter server-a367d3ea-d4f2-66de-82fd-f95fe55af871 a367d3ea-d4f2-66de-82fd-f95fe55af871}"
2025-01-25T14:28:44.621Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:43.61598481 +0000 UTC m=+33.470309752 (1 nodes), 888.42259ms from start of poll, 889.26304ms from cluster start. Timeout at 2025-01-25 14:28:44.62098765 +0000 UTC m=+34.475312752, 1.005003s after stability
2025-01-25T14:28:44.623Z [INFO]  server-4a9d605f-d0a6-dbff-0ad6-70e49d73585c: starting snapshot up to: index=12
2025-01-25T14:28:44.623Z [INFO]  creating new snapshot: path=/tmp/raft3100709629/snapshots/2-12-1737815324623.tmp
2025-01-25T14:28:44.624Z [INFO]  server-4a9d605f-d0a6-dbff-0ad6-70e49d73585c: snapshot complete up to: index=12
2025-01-25T14:28:44.627Z [INFO]  server-dc031cf1-d77b-1ffc-db35-1bf7abed0a49: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc031cf1-d77b-1ffc-db35-1bf7abed0a49 Address:dc031cf1-d77b-1ffc-db35-1bf7abed0a49} {Suffrage:Voter ID:server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59 Address:caf2aa5b-052a-bfa5-7114-4d7ab5885b59} {Suffrage:Voter ID:server-14c2fa7d-c8fc-a947-d9ed-6001de72e3f1 Address:14c2fa7d-c8fc-a947-d9ed-6001de72e3f1}]"
2025-01-25T14:28:44.627Z [INFO]  server-dc031cf1-d77b-1ffc-db35-1bf7abed0a49: entering follower state: follower="Node at dc031cf1-d77b-1ffc-db35-1bf7abed0a49 [Follower]" leader-address= leader-id=
2025-01-25T14:28:44.627Z [INFO]  server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc031cf1-d77b-1ffc-db35-1bf7abed0a49 Address:dc031cf1-d77b-1ffc-db35-1bf7abed0a49} {Suffrage:Voter ID:server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59 Address:caf2aa5b-052a-bfa5-7114-4d7ab5885b59} {Suffrage:Voter ID:server-14c2fa7d-c8fc-a947-d9ed-6001de72e3f1 Address:14c2fa7d-c8fc-a947-d9ed-6001de72e3f1}]"
2025-01-25T14:28:44.627Z [INFO]  server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: entering follower state: follower="Node at caf2aa5b-052a-bfa5-7114-4d7ab5885b59 [Follower]" leader-address= leader-id=
2025-01-25T14:28:44.627Z [INFO]  server-14c2fa7d-c8fc-a947-d9ed-6001de72e3f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc031cf1-d77b-1ffc-db35-1bf7abed0a49 Address:dc031cf1-d77b-1ffc-db35-1bf7abed0a49} {Suffrage:Voter ID:server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59 Address:caf2aa5b-052a-bfa5-7114-4d7ab5885b59} {Suffrage:Voter ID:server-14c2fa7d-c8fc-a947-d9ed-6001de72e3f1 Address:14c2fa7d-c8fc-a947-d9ed-6001de72e3f1}]"
2025-01-25T14:28:44.627Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:44.627Z [INFO]  server-14c2fa7d-c8fc-a947-d9ed-6001de72e3f1: entering follower state: follower="Node at 14c2fa7d-c8fc-a947-d9ed-6001de72e3f1 [Follower]" leader-address= leader-id=
2025-01-25T14:28:45.131Z [WARN]  server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:45.131Z [INFO]  server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: entering candidate state: node="Node at caf2aa5b-052a-bfa5-7114-4d7ab5885b59 [Candidate]" term=2
2025-01-25T14:28:45.131Z [INFO]  server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: election won: term=2 tally=2
2025-01-25T14:28:45.131Z [INFO]  server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: entering leader state: leader="Node at caf2aa5b-052a-bfa5-7114-4d7ab5885b59 [Leader]"
2025-01-25T14:28:45.131Z [INFO]  server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: added peer, starting replication: peer=server-dc031cf1-d77b-1ffc-db35-1bf7abed0a49
2025-01-25T14:28:45.131Z [INFO]  server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: added peer, starting replication: peer=server-14c2fa7d-c8fc-a947-d9ed-6001de72e3f1
2025-01-25T14:28:45.131Z [INFO]  server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: pipelining replication: peer="{Voter server-dc031cf1-d77b-1ffc-db35-1bf7abed0a49 dc031cf1-d77b-1ffc-db35-1bf7abed0a49}"
2025-01-25T14:28:45.131Z [INFO]  server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: pipelining replication: peer="{Voter server-14c2fa7d-c8fc-a947-d9ed-6001de72e3f1 14c2fa7d-c8fc-a947-d9ed-6001de72e3f1}"
2025-01-25T14:28:46.136Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:45.131540473 +0000 UTC m=+34.985865455 (1 nodes), 503.742313ms from start of poll, 504.39169ms from cluster start. Timeout at 2025-01-25 14:28:46.136551192 +0000 UTC m=+35.990876174, 1.005010719s after stability
2025-01-25T14:28:46.137Z [INFO]  creating new snapshot: path=/tmp/raft934595688/snapshots/2-14-1737815326137.tmp
2025-01-25T14:28:46.138Z [INFO]  server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: copied to local snapshot: bytes=71
2025-01-25T14:28:46.138Z [INFO]  server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: snapshot restore progress: id=2-14-1737815326137 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:46.139Z [INFO]  server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: restored user snapshot: index=14
2025-01-25T14:28:46.139Z [ERROR] server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: failed to get log: index=3 error="log not found"
2025-01-25T14:28:46.139Z [INFO]  server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: aborting pipeline replication: peer="{Voter server-14c2fa7d-c8fc-a947-d9ed-6001de72e3f1 14c2fa7d-c8fc-a947-d9ed-6001de72e3f1}"
2025-01-25T14:28:46.139Z [ERROR] server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: failed to get log: index=3 error="log not found"
2025-01-25T14:28:46.139Z [INFO]  server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: aborting pipeline replication: peer="{Voter server-dc031cf1-d77b-1ffc-db35-1bf7abed0a49 dc031cf1-d77b-1ffc-db35-1bf7abed0a49}"
2025-01-25T14:28:46.139Z [ERROR] server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: failed to get log: index=3 error="log not found"
2025-01-25T14:28:46.139Z [INFO]  creating new snapshot: path=/tmp/raft2186777723/snapshots/2-14-1737815326139.tmp
2025-01-25T14:28:46.140Z [INFO]  server-14c2fa7d-c8fc-a947-d9ed-6001de72e3f1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:46.140Z [INFO]  server-14c2fa7d-c8fc-a947-d9ed-6001de72e3f1: copied to local snapshot: bytes=71
2025-01-25T14:28:46.141Z [INFO]  server-14c2fa7d-c8fc-a947-d9ed-6001de72e3f1: snapshot restore progress: id=2-14-1737815326139 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:46.141Z [INFO]  server-14c2fa7d-c8fc-a947-d9ed-6001de72e3f1: Installed remote snapshot
2025-01-25T14:28:46.141Z [INFO]  server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: pipelining replication: peer="{Voter server-14c2fa7d-c8fc-a947-d9ed-6001de72e3f1 14c2fa7d-c8fc-a947-d9ed-6001de72e3f1}"
2025-01-25T14:28:46.141Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:46.148Z [ERROR] server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: failed to get log: index=3 error="log not found"
2025-01-25T14:28:46.149Z [INFO]  creating new snapshot: path=/tmp/raft1490453568/snapshots/2-14-1737815326149.tmp
2025-01-25T14:28:46.150Z [INFO]  server-dc031cf1-d77b-1ffc-db35-1bf7abed0a49: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:46.150Z [INFO]  server-dc031cf1-d77b-1ffc-db35-1bf7abed0a49: copied to local snapshot: bytes=71
2025-01-25T14:28:46.151Z [INFO]  server-dc031cf1-d77b-1ffc-db35-1bf7abed0a49: snapshot restore progress: id=2-14-1737815326149 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:46.151Z [INFO]  server-dc031cf1-d77b-1ffc-db35-1bf7abed0a49: Installed remote snapshot
2025-01-25T14:28:46.151Z [INFO]  server-caf2aa5b-052a-bfa5-7114-4d7ab5885b59: pipelining replication: peer="{Voter server-dc031cf1-d77b-1ffc-db35-1bf7abed0a49 dc031cf1-d77b-1ffc-db35-1bf7abed0a49}"
2025-01-25T14:28:47.147Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:46.141900955 +0000 UTC m=+35.996225937 (2 nodes), 1.96µs from start of poll, 1.514752172s from cluster start. Timeout at 2025-01-25 14:28:47.146975033 +0000 UTC m=+37.001299975, 1.005074038s after stability
2025-01-25T14:28:47.158Z [INFO]  server-4a9d605f-d0a6-dbff-0ad6-70e49d73585c: aborting pipeline replication: peer="{Voter server-b854593a-a3cd-cb49-f242-46ef9085ebe8 b854593a-a3cd-cb49-f242-46ef9085ebe8}"
2025-01-25T14:28:47.158Z [INFO]  server-4a9d605f-d0a6-dbff-0ad6-70e49d73585c: aborting pipeline replication: peer="{Voter server-a367d3ea-d4f2-66de-82fd-f95fe55af871 a367d3ea-d4f2-66de-82fd-f95fe55af871}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true
2025-01-25T14:28:47.161Z [INFO]  server-ad3dd07a-2371-1554-368e-4c8027140051: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad3dd07a-2371-1554-368e-4c8027140051 Address:ad3dd07a-2371-1554-368e-4c8027140051} {Suffrage:Voter ID:server-5c29a50a-eb0a-ac2f-4837-7066db5b1f4f Address:5c29a50a-eb0a-ac2f-4837-7066db5b1f4f} {Suffrage:Voter ID:server-fb073403-dd4f-7a68-d87b-25cf16a08e4a Address:fb073403-dd4f-7a68-d87b-25cf16a08e4a}]"
2025-01-25T14:28:47.161Z [INFO]  server-5c29a50a-eb0a-ac2f-4837-7066db5b1f4f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad3dd07a-2371-1554-368e-4c8027140051 Address:ad3dd07a-2371-1554-368e-4c8027140051} {Suffrage:Voter ID:server-5c29a50a-eb0a-ac2f-4837-7066db5b1f4f Address:5c29a50a-eb0a-ac2f-4837-7066db5b1f4f} {Suffrage:Voter ID:server-fb073403-dd4f-7a68-d87b-25cf16a08e4a Address:fb073403-dd4f-7a68-d87b-25cf16a08e4a}]"
2025-01-25T14:28:47.161Z [INFO]  server-ad3dd07a-2371-1554-368e-4c8027140051: entering follower state: follower="Node at ad3dd07a-2371-1554-368e-4c8027140051 [Follower]" leader-address= leader-id=
2025-01-25T14:28:47.161Z [INFO]  server-5c29a50a-eb0a-ac2f-4837-7066db5b1f4f: entering follower state: follower="Node at 5c29a50a-eb0a-ac2f-4837-7066db5b1f4f [Follower]" leader-address= leader-id=
2025-01-25T14:28:47.161Z [INFO]  server-fb073403-dd4f-7a68-d87b-25cf16a08e4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad3dd07a-2371-1554-368e-4c8027140051 Address:ad3dd07a-2371-1554-368e-4c8027140051} {Suffrage:Voter ID:server-5c29a50a-eb0a-ac2f-4837-7066db5b1f4f Address:5c29a50a-eb0a-ac2f-4837-7066db5b1f4f} {Suffrage:Voter ID:server-fb073403-dd4f-7a68-d87b-25cf16a08e4a Address:fb073403-dd4f-7a68-d87b-25cf16a08e4a}]"
2025-01-25T14:28:47.161Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:47.161Z [INFO]  server-fb073403-dd4f-7a68-d87b-25cf16a08e4a: entering follower state: follower="Node at fb073403-dd4f-7a68-d87b-25cf16a08e4a [Follower]" leader-address= leader-id=
2025-01-25T14:28:47.749Z [WARN]  server-5c29a50a-eb0a-ac2f-4837-7066db5b1f4f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:47.749Z [INFO]  server-5c29a50a-eb0a-ac2f-4837-7066db5b1f4f: entering candidate state: node="Node at 5c29a50a-eb0a-ac2f-4837-7066db5b1f4f [Candidate]" term=2
2025-01-25T14:28:47.750Z [INFO]  server-5c29a50a-eb0a-ac2f-4837-7066db5b1f4f: election won: term=2 tally=2
2025-01-25T14:28:47.750Z [INFO]  server-5c29a50a-eb0a-ac2f-4837-7066db5b1f4f: entering leader state: leader="Node at 5c29a50a-eb0a-ac2f-4837-7066db5b1f4f [Leader]"
2025-01-25T14:28:47.750Z [INFO]  server-5c29a50a-eb0a-ac2f-4837-7066db5b1f4f: added peer, starting replication: peer=server-ad3dd07a-2371-1554-368e-4c8027140051
2025-01-25T14:28:47.750Z [INFO]  server-5c29a50a-eb0a-ac2f-4837-7066db5b1f4f: added peer, starting replication: peer=server-fb073403-dd4f-7a68-d87b-25cf16a08e4a
2025-01-25T14:28:47.750Z [INFO]  server-5c29a50a-eb0a-ac2f-4837-7066db5b1f4f: pipelining replication: peer="{Voter server-fb073403-dd4f-7a68-d87b-25cf16a08e4a fb073403-dd4f-7a68-d87b-25cf16a08e4a}"
2025-01-25T14:28:47.750Z [INFO]  server-5c29a50a-eb0a-ac2f-4837-7066db5b1f4f: pipelining replication: peer="{Voter server-ad3dd07a-2371-1554-368e-4c8027140051 ad3dd07a-2371-1554-368e-4c8027140051}"
2025-01-25T14:28:48.756Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:47.750277837 +0000 UTC m=+37.604602819 (1 nodes), 588.30427ms from start of poll, 588.970286ms from cluster start. Timeout at 2025-01-25 14:28:48.755283357 +0000 UTC m=+38.609608339, 1.00500552s after stability
2025-01-25T14:28:48.757Z [INFO]  server-5c29a50a-eb0a-ac2f-4837-7066db5b1f4f: starting snapshot up to: index=12
2025-01-25T14:28:48.757Z [INFO]  creating new snapshot: path=/tmp/raft1153494203/snapshots/2-12-1737815328757.tmp
2025-01-25T14:28:48.758Z [INFO]  server-5c29a50a-eb0a-ac2f-4837-7066db5b1f4f: snapshot complete up to: index=12
2025-01-25T14:28:48.760Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8 Address:f73d587f-e8b3-a88d-bb5a-b5b1658d93b8} {Suffrage:Voter ID:server-ae0659f9-1b2d-87f0-d6ea-7f158bcbea49 Address:ae0659f9-1b2d-87f0-d6ea-7f158bcbea49} {Suffrage:Voter ID:server-47a2e944-0095-2788-ca16-cfe24d926a1e Address:47a2e944-0095-2788-ca16-cfe24d926a1e}]"
2025-01-25T14:28:48.760Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: entering follower state: follower="Node at f73d587f-e8b3-a88d-bb5a-b5b1658d93b8 [Follower]" leader-address= leader-id=
2025-01-25T14:28:48.760Z [INFO]  server-ae0659f9-1b2d-87f0-d6ea-7f158bcbea49: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8 Address:f73d587f-e8b3-a88d-bb5a-b5b1658d93b8} {Suffrage:Voter ID:server-ae0659f9-1b2d-87f0-d6ea-7f158bcbea49 Address:ae0659f9-1b2d-87f0-d6ea-7f158bcbea49} {Suffrage:Voter ID:server-47a2e944-0095-2788-ca16-cfe24d926a1e Address:47a2e944-0095-2788-ca16-cfe24d926a1e}]"
2025-01-25T14:28:48.761Z [INFO]  server-ae0659f9-1b2d-87f0-d6ea-7f158bcbea49: entering follower state: follower="Node at ae0659f9-1b2d-87f0-d6ea-7f158bcbea49 [Follower]" leader-address= leader-id=
2025-01-25T14:28:48.761Z [INFO]  server-47a2e944-0095-2788-ca16-cfe24d926a1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8 Address:f73d587f-e8b3-a88d-bb5a-b5b1658d93b8} {Suffrage:Voter ID:server-ae0659f9-1b2d-87f0-d6ea-7f158bcbea49 Address:ae0659f9-1b2d-87f0-d6ea-7f158bcbea49} {Suffrage:Voter ID:server-47a2e944-0095-2788-ca16-cfe24d926a1e Address:47a2e944-0095-2788-ca16-cfe24d926a1e}]"
2025-01-25T14:28:48.761Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:48.761Z [INFO]  server-47a2e944-0095-2788-ca16-cfe24d926a1e: entering follower state: follower="Node at 47a2e944-0095-2788-ca16-cfe24d926a1e [Follower]" leader-address= leader-id=
2025-01-25T14:28:49.675Z [WARN]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:49.675Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: entering candidate state: node="Node at f73d587f-e8b3-a88d-bb5a-b5b1658d93b8 [Candidate]" term=2
2025-01-25T14:28:49.675Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: election won: term=2 tally=2
2025-01-25T14:28:49.675Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: entering leader state: leader="Node at f73d587f-e8b3-a88d-bb5a-b5b1658d93b8 [Leader]"
2025-01-25T14:28:49.675Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: added peer, starting replication: peer=server-ae0659f9-1b2d-87f0-d6ea-7f158bcbea49
2025-01-25T14:28:49.675Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: added peer, starting replication: peer=server-47a2e944-0095-2788-ca16-cfe24d926a1e
2025-01-25T14:28:49.676Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: pipelining replication: peer="{Voter server-47a2e944-0095-2788-ca16-cfe24d926a1e 47a2e944-0095-2788-ca16-cfe24d926a1e}"
2025-01-25T14:28:49.676Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: pipelining replication: peer="{Voter server-ae0659f9-1b2d-87f0-d6ea-7f158bcbea49 ae0659f9-1b2d-87f0-d6ea-7f158bcbea49}"
2025-01-25T14:28:50.680Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:49.675808627 +0000 UTC m=+39.530133609 (1 nodes), 914.631327ms from start of poll, 915.268384ms from cluster start. Timeout at 2025-01-25 14:28:50.680810227 +0000 UTC m=+40.535135169, 1.00500156s after stability
2025-01-25T14:28:50.682Z [INFO]  creating new snapshot: path=/tmp/raft3488167176/snapshots/2-14-1737815330682.tmp
2025-01-25T14:28:50.684Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: copied to local snapshot: bytes=71
2025-01-25T14:28:50.685Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: snapshot restore progress: id=2-14-1737815330682 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:50.685Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: removing all old logs from log store
2025-01-25T14:28:50.685Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: compacting logs: from=1 to=2
2025-01-25T14:28:50.685Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: restored user snapshot: index=14
2025-01-25T14:28:50.685Z [ERROR] server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: failed to get log: index=2 error="log not found"
2025-01-25T14:28:50.685Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: aborting pipeline replication: peer="{Voter server-47a2e944-0095-2788-ca16-cfe24d926a1e 47a2e944-0095-2788-ca16-cfe24d926a1e}"
2025-01-25T14:28:50.685Z [ERROR] server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: failed to get log: index=2 error="log not found"
2025-01-25T14:28:50.685Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: aborting pipeline replication: peer="{Voter server-ae0659f9-1b2d-87f0-d6ea-7f158bcbea49 ae0659f9-1b2d-87f0-d6ea-7f158bcbea49}"
2025-01-25T14:28:50.693Z [ERROR] server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: failed to get log: index=2 error="log not found"
2025-01-25T14:28:50.693Z [ERROR] server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: failed to get log: index=2 error="log not found"
2025-01-25T14:28:50.694Z [INFO]  creating new snapshot: path=/tmp/raft739605567/snapshots/2-14-1737815330694.tmp
2025-01-25T14:28:50.694Z [INFO]  creating new snapshot: path=/tmp/raft743504276/snapshots/2-14-1737815330694.tmp
2025-01-25T14:28:50.695Z [INFO]  server-ae0659f9-1b2d-87f0-d6ea-7f158bcbea49: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:50.695Z [INFO]  server-47a2e944-0095-2788-ca16-cfe24d926a1e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:50.695Z [INFO]  server-47a2e944-0095-2788-ca16-cfe24d926a1e: copied to local snapshot: bytes=71
2025-01-25T14:28:50.696Z [INFO]  server-ae0659f9-1b2d-87f0-d6ea-7f158bcbea49: copied to local snapshot: bytes=71
2025-01-25T14:28:50.696Z [INFO]  server-47a2e944-0095-2788-ca16-cfe24d926a1e: snapshot restore progress: id=2-14-1737815330694 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:50.696Z [INFO]  server-47a2e944-0095-2788-ca16-cfe24d926a1e: removing all old logs from log store
2025-01-25T14:28:50.696Z [INFO]  server-47a2e944-0095-2788-ca16-cfe24d926a1e: compacting logs: from=1 to=2
2025-01-25T14:28:50.696Z [INFO]  server-47a2e944-0095-2788-ca16-cfe24d926a1e: Installed remote snapshot
2025-01-25T14:28:50.696Z [INFO]  server-ae0659f9-1b2d-87f0-d6ea-7f158bcbea49: snapshot restore progress: id=2-14-1737815330694 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:50.696Z [INFO]  server-ae0659f9-1b2d-87f0-d6ea-7f158bcbea49: removing all old logs from log store
2025-01-25T14:28:50.696Z [INFO]  server-ae0659f9-1b2d-87f0-d6ea-7f158bcbea49: compacting logs: from=1 to=2
2025-01-25T14:28:50.696Z [INFO]  server-ae0659f9-1b2d-87f0-d6ea-7f158bcbea49: Installed remote snapshot
2025-01-25T14:28:50.696Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: pipelining replication: peer="{Voter server-47a2e944-0095-2788-ca16-cfe24d926a1e 47a2e944-0095-2788-ca16-cfe24d926a1e}"
2025-01-25T14:28:50.696Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:50.696Z [WARN]  server-ae0659f9-1b2d-87f0-d6ea-7f158bcbea49: skipping application of old log: index=14
2025-01-25T14:28:50.697Z [INFO]  server-f73d587f-e8b3-a88d-bb5a-b5b1658d93b8: pipelining replication: peer="{Voter server-ae0659f9-1b2d-87f0-d6ea-7f158bcbea49 ae0659f9-1b2d-87f0-d6ea-7f158bcbea49}"
2025-01-25T14:28:51.703Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:50.696902722 +0000 UTC m=+40.551227704 (2 nodes), 1.52µs from start of poll, 1.936362479s from cluster start. Timeout at 2025-01-25 14:28:51.702006879 +0000 UTC m=+41.556331821, 1.005104117s after stability
2025-01-25T14:28:51.716Z [INFO]  server-5c29a50a-eb0a-ac2f-4837-7066db5b1f4f: aborting pipeline replication: peer="{Voter server-fb073403-dd4f-7a68-d87b-25cf16a08e4a fb073403-dd4f-7a68-d87b-25cf16a08e4a}"
2025-01-25T14:28:51.716Z [INFO]  server-5c29a50a-eb0a-ac2f-4837-7066db5b1f4f: aborting pipeline replication: peer="{Voter server-ad3dd07a-2371-1554-368e-4c8027140051 ad3dd07a-2371-1554-368e-4c8027140051}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false
2025-01-25T14:28:51.718Z [INFO]  server-908fbfd9-b228-2fec-6428-268069742cda: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-908fbfd9-b228-2fec-6428-268069742cda Address:908fbfd9-b228-2fec-6428-268069742cda} {Suffrage:Voter ID:server-3dd21ff8-240f-f532-b9f2-77487d6b4b82 Address:3dd21ff8-240f-f532-b9f2-77487d6b4b82} {Suffrage:Voter ID:server-24dd8be8-8623-ee91-0673-2a99a935066a Address:24dd8be8-8623-ee91-0673-2a99a935066a}]"
2025-01-25T14:28:51.718Z [INFO]  server-908fbfd9-b228-2fec-6428-268069742cda: entering follower state: follower="Node at 908fbfd9-b228-2fec-6428-268069742cda [Follower]" leader-address= leader-id=
2025-01-25T14:28:51.719Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-908fbfd9-b228-2fec-6428-268069742cda Address:908fbfd9-b228-2fec-6428-268069742cda} {Suffrage:Voter ID:server-3dd21ff8-240f-f532-b9f2-77487d6b4b82 Address:3dd21ff8-240f-f532-b9f2-77487d6b4b82} {Suffrage:Voter ID:server-24dd8be8-8623-ee91-0673-2a99a935066a Address:24dd8be8-8623-ee91-0673-2a99a935066a}]"
2025-01-25T14:28:51.719Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: entering follower state: follower="Node at 3dd21ff8-240f-f532-b9f2-77487d6b4b82 [Follower]" leader-address= leader-id=
2025-01-25T14:28:51.719Z [INFO]  server-24dd8be8-8623-ee91-0673-2a99a935066a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-908fbfd9-b228-2fec-6428-268069742cda Address:908fbfd9-b228-2fec-6428-268069742cda} {Suffrage:Voter ID:server-3dd21ff8-240f-f532-b9f2-77487d6b4b82 Address:3dd21ff8-240f-f532-b9f2-77487d6b4b82} {Suffrage:Voter ID:server-24dd8be8-8623-ee91-0673-2a99a935066a Address:24dd8be8-8623-ee91-0673-2a99a935066a}]"
2025-01-25T14:28:51.719Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:51.719Z [INFO]  server-24dd8be8-8623-ee91-0673-2a99a935066a: entering follower state: follower="Node at 24dd8be8-8623-ee91-0673-2a99a935066a [Follower]" leader-address= leader-id=
2025-01-25T14:28:52.221Z [WARN]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:52.221Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: entering candidate state: node="Node at 3dd21ff8-240f-f532-b9f2-77487d6b4b82 [Candidate]" term=2
2025-01-25T14:28:52.221Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: election won: term=2 tally=2
2025-01-25T14:28:52.221Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: entering leader state: leader="Node at 3dd21ff8-240f-f532-b9f2-77487d6b4b82 [Leader]"
2025-01-25T14:28:52.221Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: added peer, starting replication: peer=server-908fbfd9-b228-2fec-6428-268069742cda
2025-01-25T14:28:52.221Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: added peer, starting replication: peer=server-24dd8be8-8623-ee91-0673-2a99a935066a
2025-01-25T14:28:52.221Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: pipelining replication: peer="{Voter server-24dd8be8-8623-ee91-0673-2a99a935066a 24dd8be8-8623-ee91-0673-2a99a935066a}"
2025-01-25T14:28:52.221Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: pipelining replication: peer="{Voter server-908fbfd9-b228-2fec-6428-268069742cda 908fbfd9-b228-2fec-6428-268069742cda}"
2025-01-25T14:28:53.226Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:52.2214919 +0000 UTC m=+42.075816842 (1 nodes), 502.076053ms from start of poll, 502.819506ms from cluster start. Timeout at 2025-01-25 14:28:53.226494379 +0000 UTC m=+43.080819361, 1.005002519s after stability
2025-01-25T14:28:53.227Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: starting snapshot up to: index=12
2025-01-25T14:28:53.227Z [INFO]  creating new snapshot: path=/tmp/raft1488112060/snapshots/2-12-1737815333227.tmp
2025-01-25T14:28:53.229Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: snapshot complete up to: index=12
2025-01-25T14:28:53.230Z [INFO]  creating new snapshot: path=/tmp/raft1488112060/snapshots/2-23-1737815333230.tmp
2025-01-25T14:28:53.232Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: copied to local snapshot: bytes=71
2025-01-25T14:28:53.232Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: snapshot restore progress: id=2-23-1737815333230 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:53.232Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: restored user snapshot: index=23
2025-01-25T14:28:53.232Z [ERROR] server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: failed to get log: index=23 error="log not found"
2025-01-25T14:28:53.232Z [ERROR] server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: failed to get log: index=23 error="log not found"
2025-01-25T14:28:53.232Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: aborting pipeline replication: peer="{Voter server-908fbfd9-b228-2fec-6428-268069742cda 908fbfd9-b228-2fec-6428-268069742cda}"
2025-01-25T14:28:53.232Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: aborting pipeline replication: peer="{Voter server-24dd8be8-8623-ee91-0673-2a99a935066a 24dd8be8-8623-ee91-0673-2a99a935066a}"
2025-01-25T14:28:53.238Z [ERROR] server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: failed to get log: index=23 error="log not found"
2025-01-25T14:28:53.239Z [INFO]  creating new snapshot: path=/tmp/raft1728878160/snapshots/2-23-1737815333239.tmp
2025-01-25T14:28:53.240Z [INFO]  server-24dd8be8-8623-ee91-0673-2a99a935066a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:53.240Z [INFO]  server-24dd8be8-8623-ee91-0673-2a99a935066a: copied to local snapshot: bytes=71
2025-01-25T14:28:53.241Z [INFO]  server-24dd8be8-8623-ee91-0673-2a99a935066a: snapshot restore progress: id=2-23-1737815333239 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:53.241Z [INFO]  server-24dd8be8-8623-ee91-0673-2a99a935066a: Installed remote snapshot
2025-01-25T14:28:53.241Z [WARN]  server-24dd8be8-8623-ee91-0673-2a99a935066a: skipping application of old log: index=22
2025-01-25T14:28:53.241Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: pipelining replication: peer="{Voter server-24dd8be8-8623-ee91-0673-2a99a935066a 24dd8be8-8623-ee91-0673-2a99a935066a}"
2025-01-25T14:28:53.241Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:53.243Z [ERROR] server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: failed to get log: index=23 error="log not found"
2025-01-25T14:28:53.244Z [INFO]  creating new snapshot: path=/tmp/raft3682433092/snapshots/2-23-1737815333244.tmp
2025-01-25T14:28:53.244Z [INFO]  server-908fbfd9-b228-2fec-6428-268069742cda: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:53.245Z [INFO]  server-908fbfd9-b228-2fec-6428-268069742cda: copied to local snapshot: bytes=71
2025-01-25T14:28:53.245Z [INFO]  server-908fbfd9-b228-2fec-6428-268069742cda: snapshot restore progress: id=2-23-1737815333244 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:53.245Z [INFO]  server-908fbfd9-b228-2fec-6428-268069742cda: Installed remote snapshot
2025-01-25T14:28:53.246Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: pipelining replication: peer="{Voter server-908fbfd9-b228-2fec-6428-268069742cda 908fbfd9-b228-2fec-6428-268069742cda}"
2025-01-25T14:28:54.247Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:53.241958618 +0000 UTC m=+43.096283600 (2 nodes), 1.56µs from start of poll, 1.523286264s from cluster start. Timeout at 2025-01-25 14:28:54.247063974 +0000 UTC m=+44.101388996, 1.005105396s after stability
2025-01-25T14:28:54.263Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: aborting pipeline replication: peer="{Voter server-24dd8be8-8623-ee91-0673-2a99a935066a 24dd8be8-8623-ee91-0673-2a99a935066a}"
2025-01-25T14:28:54.263Z [INFO]  server-3dd21ff8-240f-f532-b9f2-77487d6b4b82: aborting pipeline replication: peer="{Voter server-908fbfd9-b228-2fec-6428-268069742cda 908fbfd9-b228-2fec-6428-268069742cda}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false
2025-01-25T14:28:54.266Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d815684-c949-f313-f177-342ae959dca0 Address:4d815684-c949-f313-f177-342ae959dca0} {Suffrage:Voter ID:server-6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c Address:6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c} {Suffrage:Voter ID:server-4ed9fa5c-8311-d15e-920b-14b47705590d Address:4ed9fa5c-8311-d15e-920b-14b47705590d}]"
2025-01-25T14:28:54.266Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: entering follower state: follower="Node at 4d815684-c949-f313-f177-342ae959dca0 [Follower]" leader-address= leader-id=
2025-01-25T14:28:54.266Z [INFO]  server-6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d815684-c949-f313-f177-342ae959dca0 Address:4d815684-c949-f313-f177-342ae959dca0} {Suffrage:Voter ID:server-6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c Address:6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c} {Suffrage:Voter ID:server-4ed9fa5c-8311-d15e-920b-14b47705590d Address:4ed9fa5c-8311-d15e-920b-14b47705590d}]"
2025-01-25T14:28:54.267Z [INFO]  server-6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c: entering follower state: follower="Node at 6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c [Follower]" leader-address= leader-id=
2025-01-25T14:28:54.267Z [INFO]  server-4ed9fa5c-8311-d15e-920b-14b47705590d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d815684-c949-f313-f177-342ae959dca0 Address:4d815684-c949-f313-f177-342ae959dca0} {Suffrage:Voter ID:server-6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c Address:6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c} {Suffrage:Voter ID:server-4ed9fa5c-8311-d15e-920b-14b47705590d Address:4ed9fa5c-8311-d15e-920b-14b47705590d}]"
2025-01-25T14:28:54.267Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:54.267Z [INFO]  server-4ed9fa5c-8311-d15e-920b-14b47705590d: entering follower state: follower="Node at 4ed9fa5c-8311-d15e-920b-14b47705590d [Follower]" leader-address= leader-id=
2025-01-25T14:28:54.835Z [WARN]  server-4d815684-c949-f313-f177-342ae959dca0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:54.835Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: entering candidate state: node="Node at 4d815684-c949-f313-f177-342ae959dca0 [Candidate]" term=2
2025-01-25T14:28:54.836Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: election won: term=2 tally=2
2025-01-25T14:28:54.836Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: entering leader state: leader="Node at 4d815684-c949-f313-f177-342ae959dca0 [Leader]"
2025-01-25T14:28:54.836Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: added peer, starting replication: peer=server-6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c
2025-01-25T14:28:54.836Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: added peer, starting replication: peer=server-4ed9fa5c-8311-d15e-920b-14b47705590d
2025-01-25T14:28:54.836Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: pipelining replication: peer="{Voter server-4ed9fa5c-8311-d15e-920b-14b47705590d 4ed9fa5c-8311-d15e-920b-14b47705590d}"
2025-01-25T14:28:54.836Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: pipelining replication: peer="{Voter server-6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c 6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c}"
2025-01-25T14:28:55.842Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:54.836387162 +0000 UTC m=+44.690712144 (1 nodes), 569.175198ms from start of poll, 569.904372ms from cluster start. Timeout at 2025-01-25 14:28:55.841481039 +0000 UTC m=+45.695806061, 1.005093917s after stability
2025-01-25T14:28:55.843Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: starting snapshot up to: index=12
2025-01-25T14:28:55.843Z [INFO]  creating new snapshot: path=/tmp/raft1812817363/snapshots/2-12-1737815335843.tmp
2025-01-25T14:28:55.844Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: snapshot complete up to: index=12
2025-01-25T14:28:55.846Z [INFO]  creating new snapshot: path=/tmp/raft1812817363/snapshots/2-23-1737815335846.tmp
2025-01-25T14:28:55.847Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: copied to local snapshot: bytes=71
2025-01-25T14:28:55.847Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: snapshot restore progress: id=2-23-1737815335846 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:55.848Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: removing all old logs from log store
2025-01-25T14:28:55.848Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: compacting logs: from=1 to=22
2025-01-25T14:28:55.848Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: restored user snapshot: index=23
2025-01-25T14:28:55.848Z [ERROR] server-4d815684-c949-f313-f177-342ae959dca0: failed to get log: index=22 error="log not found"
2025-01-25T14:28:55.848Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: aborting pipeline replication: peer="{Voter server-4ed9fa5c-8311-d15e-920b-14b47705590d 4ed9fa5c-8311-d15e-920b-14b47705590d}"
2025-01-25T14:28:55.848Z [ERROR] server-4d815684-c949-f313-f177-342ae959dca0: failed to get log: index=22 error="log not found"
2025-01-25T14:28:55.848Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: aborting pipeline replication: peer="{Voter server-6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c 6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c}"
2025-01-25T14:28:55.855Z [ERROR] server-4d815684-c949-f313-f177-342ae959dca0: failed to get log: index=22 error="log not found"
2025-01-25T14:28:55.856Z [INFO]  creating new snapshot: path=/tmp/raft2739535423/snapshots/2-23-1737815335856.tmp
2025-01-25T14:28:55.856Z [INFO]  server-6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:55.857Z [INFO]  server-6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c: copied to local snapshot: bytes=71
2025-01-25T14:28:55.858Z [INFO]  server-6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c: snapshot restore progress: id=2-23-1737815335856 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:55.858Z [INFO]  server-6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c: removing all old logs from log store
2025-01-25T14:28:55.858Z [INFO]  server-6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c: compacting logs: from=1 to=22
2025-01-25T14:28:55.858Z [INFO]  server-6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c: Installed remote snapshot
2025-01-25T14:28:55.858Z [ERROR] server-4d815684-c949-f313-f177-342ae959dca0: failed to get log: index=22 error="log not found"
2025-01-25T14:28:55.858Z [WARN]  server-6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c: skipping application of old log: index=22
2025-01-25T14:28:55.858Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: pipelining replication: peer="{Voter server-6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c 6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c}"
2025-01-25T14:28:55.858Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:28:55.859Z [INFO]  creating new snapshot: path=/tmp/raft2573522237/snapshots/2-23-1737815335859.tmp
2025-01-25T14:28:55.859Z [INFO]  server-4ed9fa5c-8311-d15e-920b-14b47705590d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:55.860Z [INFO]  server-4ed9fa5c-8311-d15e-920b-14b47705590d: copied to local snapshot: bytes=71
2025-01-25T14:28:55.861Z [INFO]  server-4ed9fa5c-8311-d15e-920b-14b47705590d: snapshot restore progress: id=2-23-1737815335859 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:28:55.861Z [INFO]  server-4ed9fa5c-8311-d15e-920b-14b47705590d: removing all old logs from log store
2025-01-25T14:28:55.861Z [INFO]  server-4ed9fa5c-8311-d15e-920b-14b47705590d: compacting logs: from=1 to=22
2025-01-25T14:28:55.861Z [INFO]  server-4ed9fa5c-8311-d15e-920b-14b47705590d: Installed remote snapshot
2025-01-25T14:28:55.861Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: pipelining replication: peer="{Voter server-4ed9fa5c-8311-d15e-920b-14b47705590d 4ed9fa5c-8311-d15e-920b-14b47705590d}"
2025-01-25T14:28:56.864Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:28:55.858537259 +0000 UTC m=+45.712862241 (2 nodes), 2.92µs from start of poll, 1.592054469s from cluster start. Timeout at 2025-01-25 14:28:56.863646256 +0000 UTC m=+46.717971198, 1.005108957s after stability
2025-01-25T14:28:56.877Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: aborting pipeline replication: peer="{Voter server-4ed9fa5c-8311-d15e-920b-14b47705590d 4ed9fa5c-8311-d15e-920b-14b47705590d}"
2025-01-25T14:28:56.877Z [INFO]  server-4d815684-c949-f313-f177-342ae959dca0: aborting pipeline replication: peer="{Voter server-6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c 6b939789-b4ce-8c7b-b999-1b7fdd2c2f4c}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true
2025-01-25T14:28:56.880Z [INFO]  server-c9862a21-3a64-5896-7082-4cab656e5f00: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9862a21-3a64-5896-7082-4cab656e5f00 Address:c9862a21-3a64-5896-7082-4cab656e5f00} {Suffrage:Voter ID:server-3adee50b-dc2a-6a94-7e1d-fc1222d447ed Address:3adee50b-dc2a-6a94-7e1d-fc1222d447ed} {Suffrage:Voter ID:server-b576e696-78c3-27da-8bda-ff1db8feca07 Address:b576e696-78c3-27da-8bda-ff1db8feca07}]"
2025-01-25T14:28:56.880Z [INFO]  server-c9862a21-3a64-5896-7082-4cab656e5f00: entering follower state: follower="Node at c9862a21-3a64-5896-7082-4cab656e5f00 [Follower]" leader-address= leader-id=
2025-01-25T14:28:56.880Z [INFO]  server-3adee50b-dc2a-6a94-7e1d-fc1222d447ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9862a21-3a64-5896-7082-4cab656e5f00 Address:c9862a21-3a64-5896-7082-4cab656e5f00} {Suffrage:Voter ID:server-3adee50b-dc2a-6a94-7e1d-fc1222d447ed Address:3adee50b-dc2a-6a94-7e1d-fc1222d447ed} {Suffrage:Voter ID:server-b576e696-78c3-27da-8bda-ff1db8feca07 Address:b576e696-78c3-27da-8bda-ff1db8feca07}]"
2025-01-25T14:28:56.880Z [INFO]  server-3adee50b-dc2a-6a94-7e1d-fc1222d447ed: entering follower state: follower="Node at 3adee50b-dc2a-6a94-7e1d-fc1222d447ed [Follower]" leader-address= leader-id=
2025-01-25T14:28:56.880Z [INFO]  server-b576e696-78c3-27da-8bda-ff1db8feca07: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9862a21-3a64-5896-7082-4cab656e5f00 Address:c9862a21-3a64-5896-7082-4cab656e5f00} {Suffrage:Voter ID:server-3adee50b-dc2a-6a94-7e1d-fc1222d447ed Address:3adee50b-dc2a-6a94-7e1d-fc1222d447ed} {Suffrage:Voter ID:server-b576e696-78c3-27da-8bda-ff1db8feca07 Address:b576e696-78c3-27da-8bda-ff1db8feca07}]"
2025-01-25T14:28:56.880Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:56.880Z [INFO]  server-b576e696-78c3-27da-8bda-ff1db8feca07: entering follower state: follower="Node at b576e696-78c3-27da-8bda-ff1db8feca07 [Follower]" leader-address= leader-id=
2025-01-25T14:28:57.454Z [WARN]  server-b576e696-78c3-27da-8bda-ff1db8feca07: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:57.454Z [INFO]  server-b576e696-78c3-27da-8bda-ff1db8feca07: entering candidate state: node="Node at b576e696-78c3-27da-8bda-ff1db8feca07 [Candidate]" term=2
2025-01-25T14:28:57.454Z [INFO]  server-b576e696-78c3-27da-8bda-ff1db8feca07: election won: term=2 tally=2
2025-01-25T14:28:57.454Z [INFO]  server-b576e696-78c3-27da-8bda-ff1db8feca07: entering leader state: leader="Node at b576e696-78c3-27da-8bda-ff1db8feca07 [Leader]"
2025-01-25T14:28:57.454Z [INFO]  server-b576e696-78c3-27da-8bda-ff1db8feca07: added peer, starting replication: peer=server-c9862a21-3a64-5896-7082-4cab656e5f00
2025-01-25T14:28:57.454Z [INFO]  server-b576e696-78c3-27da-8bda-ff1db8feca07: added peer, starting replication: peer=server-3adee50b-dc2a-6a94-7e1d-fc1222d447ed
2025-01-25T14:28:57.454Z [INFO]  server-b576e696-78c3-27da-8bda-ff1db8feca07: pipelining replication: peer="{Voter server-3adee50b-dc2a-6a94-7e1d-fc1222d447ed 3adee50b-dc2a-6a94-7e1d-fc1222d447ed}"
2025-01-25T14:28:57.455Z [INFO]  server-b576e696-78c3-27da-8bda-ff1db8feca07: pipelining replication: peer="{Voter server-c9862a21-3a64-5896-7082-4cab656e5f00 c9862a21-3a64-5896-7082-4cab656e5f00}"
2025-01-25T14:28:58.459Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:57.454889974 +0000 UTC m=+47.309214956 (1 nodes), 574.158219ms from start of poll, 574.807196ms from cluster start. Timeout at 2025-01-25 14:28:58.459894294 +0000 UTC m=+48.314219236, 1.00500428s after stability
2025-01-25T14:28:58.461Z [INFO]  server-b576e696-78c3-27da-8bda-ff1db8feca07: starting snapshot up to: index=12
2025-01-25T14:28:58.461Z [INFO]  creating new snapshot: path=/tmp/raft1181848739/snapshots/2-12-1737815338461.tmp
2025-01-25T14:28:58.462Z [INFO]  server-b576e696-78c3-27da-8bda-ff1db8feca07: snapshot complete up to: index=12
2025-01-25T14:28:58.464Z [INFO]  server-eb5e61bd-f0ca-a289-e33d-226a0a786170: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eb5e61bd-f0ca-a289-e33d-226a0a786170 Address:eb5e61bd-f0ca-a289-e33d-226a0a786170} {Suffrage:Voter ID:server-e0a8ce1e-d964-e763-1c82-d2ddc66b713b Address:e0a8ce1e-d964-e763-1c82-d2ddc66b713b} {Suffrage:Voter ID:server-9b42780b-af98-44d1-e0a3-220357490865 Address:9b42780b-af98-44d1-e0a3-220357490865}]"
2025-01-25T14:28:58.464Z [INFO]  server-e0a8ce1e-d964-e763-1c82-d2ddc66b713b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eb5e61bd-f0ca-a289-e33d-226a0a786170 Address:eb5e61bd-f0ca-a289-e33d-226a0a786170} {Suffrage:Voter ID:server-e0a8ce1e-d964-e763-1c82-d2ddc66b713b Address:e0a8ce1e-d964-e763-1c82-d2ddc66b713b} {Suffrage:Voter ID:server-9b42780b-af98-44d1-e0a3-220357490865 Address:9b42780b-af98-44d1-e0a3-220357490865}]"
2025-01-25T14:28:58.464Z [INFO]  server-eb5e61bd-f0ca-a289-e33d-226a0a786170: entering follower state: follower="Node at eb5e61bd-f0ca-a289-e33d-226a0a786170 [Follower]" leader-address= leader-id=
2025-01-25T14:28:58.464Z [INFO]  server-e0a8ce1e-d964-e763-1c82-d2ddc66b713b: entering follower state: follower="Node at e0a8ce1e-d964-e763-1c82-d2ddc66b713b [Follower]" leader-address= leader-id=
2025-01-25T14:28:58.464Z [INFO]  server-9b42780b-af98-44d1-e0a3-220357490865: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eb5e61bd-f0ca-a289-e33d-226a0a786170 Address:eb5e61bd-f0ca-a289-e33d-226a0a786170} {Suffrage:Voter ID:server-e0a8ce1e-d964-e763-1c82-d2ddc66b713b Address:e0a8ce1e-d964-e763-1c82-d2ddc66b713b} {Suffrage:Voter ID:server-9b42780b-af98-44d1-e0a3-220357490865 Address:9b42780b-af98-44d1-e0a3-220357490865}]"
2025-01-25T14:28:58.465Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:28:58.465Z [INFO]  server-9b42780b-af98-44d1-e0a3-220357490865: entering follower state: follower="Node at 9b42780b-af98-44d1-e0a3-220357490865 [Follower]" leader-address= leader-id=
2025-01-25T14:28:59.045Z [WARN]  server-eb5e61bd-f0ca-a289-e33d-226a0a786170: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:28:59.045Z [INFO]  server-eb5e61bd-f0ca-a289-e33d-226a0a786170: entering candidate state: node="Node at eb5e61bd-f0ca-a289-e33d-226a0a786170 [Candidate]" term=2
2025-01-25T14:28:59.045Z [INFO]  server-eb5e61bd-f0ca-a289-e33d-226a0a786170: election won: term=2 tally=2
2025-01-25T14:28:59.045Z [INFO]  server-eb5e61bd-f0ca-a289-e33d-226a0a786170: entering leader state: leader="Node at eb5e61bd-f0ca-a289-e33d-226a0a786170 [Leader]"
2025-01-25T14:28:59.045Z [INFO]  server-eb5e61bd-f0ca-a289-e33d-226a0a786170: added peer, starting replication: peer=server-e0a8ce1e-d964-e763-1c82-d2ddc66b713b
2025-01-25T14:28:59.045Z [INFO]  server-eb5e61bd-f0ca-a289-e33d-226a0a786170: added peer, starting replication: peer=server-9b42780b-af98-44d1-e0a3-220357490865
2025-01-25T14:28:59.045Z [INFO]  server-eb5e61bd-f0ca-a289-e33d-226a0a786170: pipelining replication: peer="{Voter server-e0a8ce1e-d964-e763-1c82-d2ddc66b713b e0a8ce1e-d964-e763-1c82-d2ddc66b713b}"
2025-01-25T14:28:59.045Z [INFO]  server-eb5e61bd-f0ca-a289-e33d-226a0a786170: pipelining replication: peer="{Voter server-9b42780b-af98-44d1-e0a3-220357490865 9b42780b-af98-44d1-e0a3-220357490865}"
2025-01-25T14:29:00.050Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:28:59.045437097 +0000 UTC m=+48.899762079 (1 nodes), 580.395594ms from start of poll, 581.134768ms from cluster start. Timeout at 2025-01-25 14:29:00.050442457 +0000 UTC m=+49.904767439, 1.00500536s after stability
2025-01-25T14:29:00.051Z [INFO]  creating new snapshot: path=/tmp/raft345270667/snapshots/2-15-1737815340051.tmp
2025-01-25T14:29:00.052Z [INFO]  server-eb5e61bd-f0ca-a289-e33d-226a0a786170: copied to local snapshot: bytes=71
2025-01-25T14:29:00.053Z [INFO]  server-eb5e61bd-f0ca-a289-e33d-226a0a786170: snapshot restore progress: id=2-15-1737815340051 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:00.053Z [INFO]  server-eb5e61bd-f0ca-a289-e33d-226a0a786170: restored user snapshot: index=15
2025-01-25T14:29:00.053Z [ERROR] server-eb5e61bd-f0ca-a289-e33d-226a0a786170: failed to get log: index=3 error="log not found"
2025-01-25T14:29:00.053Z [INFO]  server-eb5e61bd-f0ca-a289-e33d-226a0a786170: aborting pipeline replication: peer="{Voter server-9b42780b-af98-44d1-e0a3-220357490865 9b42780b-af98-44d1-e0a3-220357490865}"
2025-01-25T14:29:00.053Z [ERROR] server-eb5e61bd-f0ca-a289-e33d-226a0a786170: failed to get log: index=3 error="log not found"
2025-01-25T14:29:00.053Z [INFO]  server-eb5e61bd-f0ca-a289-e33d-226a0a786170: aborting pipeline replication: peer="{Voter server-e0a8ce1e-d964-e763-1c82-d2ddc66b713b e0a8ce1e-d964-e763-1c82-d2ddc66b713b}"
2025-01-25T14:29:00.062Z [ERROR] server-eb5e61bd-f0ca-a289-e33d-226a0a786170: failed to get log: index=3 error="log not found"
2025-01-25T14:29:00.063Z [ERROR] server-eb5e61bd-f0ca-a289-e33d-226a0a786170: failed to get log: index=3 error="log not found"
2025-01-25T14:29:00.063Z [INFO]  creating new snapshot: path=/tmp/raft2194126534/snapshots/2-15-1737815340063.tmp
2025-01-25T14:29:00.064Z [INFO]  creating new snapshot: path=/tmp/raft3376604060/snapshots/2-15-1737815340064.tmp
2025-01-25T14:29:00.064Z [INFO]  server-9b42780b-af98-44d1-e0a3-220357490865: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:00.064Z [INFO]  server-e0a8ce1e-d964-e763-1c82-d2ddc66b713b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:00.066Z [INFO]  server-9b42780b-af98-44d1-e0a3-220357490865: copied to local snapshot: bytes=71
2025-01-25T14:29:00.068Z [INFO]  server-9b42780b-af98-44d1-e0a3-220357490865: snapshot restore progress: id=2-15-1737815340063 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:00.068Z [INFO]  server-9b42780b-af98-44d1-e0a3-220357490865: Installed remote snapshot
2025-01-25T14:29:00.068Z [INFO]  server-eb5e61bd-f0ca-a289-e33d-226a0a786170: pipelining replication: peer="{Voter server-9b42780b-af98-44d1-e0a3-220357490865 9b42780b-af98-44d1-e0a3-220357490865}"
2025-01-25T14:29:00.068Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:00.069Z [INFO]  server-e0a8ce1e-d964-e763-1c82-d2ddc66b713b: copied to local snapshot: bytes=71
2025-01-25T14:29:00.071Z [INFO]  server-e0a8ce1e-d964-e763-1c82-d2ddc66b713b: snapshot restore progress: id=2-15-1737815340064 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:00.071Z [INFO]  server-e0a8ce1e-d964-e763-1c82-d2ddc66b713b: Installed remote snapshot
2025-01-25T14:29:00.071Z [INFO]  server-eb5e61bd-f0ca-a289-e33d-226a0a786170: pipelining replication: peer="{Voter server-e0a8ce1e-d964-e763-1c82-d2ddc66b713b e0a8ce1e-d964-e763-1c82-d2ddc66b713b}"
2025-01-25T14:29:01.074Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:00.069029502 +0000 UTC m=+49.923354484 (2 nodes), 2.28µs from start of poll, 1.604727173s from cluster start. Timeout at 2025-01-25 14:29:01.074120779 +0000 UTC m=+50.928445761, 1.005091277s after stability
2025-01-25T14:29:01.086Z [INFO]  server-b576e696-78c3-27da-8bda-ff1db8feca07: aborting pipeline replication: peer="{Voter server-3adee50b-dc2a-6a94-7e1d-fc1222d447ed 3adee50b-dc2a-6a94-7e1d-fc1222d447ed}"
2025-01-25T14:29:01.086Z [INFO]  server-b576e696-78c3-27da-8bda-ff1db8feca07: aborting pipeline replication: peer="{Voter server-c9862a21-3a64-5896-7082-4cab656e5f00 c9862a21-3a64-5896-7082-4cab656e5f00}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true
2025-01-25T14:29:01.090Z [INFO]  server-81a5ded8-c6bc-e8f7-620f-68829f653f2b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81a5ded8-c6bc-e8f7-620f-68829f653f2b Address:81a5ded8-c6bc-e8f7-620f-68829f653f2b} {Suffrage:Voter ID:server-6b72411d-f485-bd29-ad17-20e98c399204 Address:6b72411d-f485-bd29-ad17-20e98c399204} {Suffrage:Voter ID:server-7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81 Address:7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81}]"
2025-01-25T14:29:01.090Z [INFO]  server-6b72411d-f485-bd29-ad17-20e98c399204: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81a5ded8-c6bc-e8f7-620f-68829f653f2b Address:81a5ded8-c6bc-e8f7-620f-68829f653f2b} {Suffrage:Voter ID:server-6b72411d-f485-bd29-ad17-20e98c399204 Address:6b72411d-f485-bd29-ad17-20e98c399204} {Suffrage:Voter ID:server-7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81 Address:7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81}]"
2025-01-25T14:29:01.090Z [INFO]  server-7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81a5ded8-c6bc-e8f7-620f-68829f653f2b Address:81a5ded8-c6bc-e8f7-620f-68829f653f2b} {Suffrage:Voter ID:server-6b72411d-f485-bd29-ad17-20e98c399204 Address:6b72411d-f485-bd29-ad17-20e98c399204} {Suffrage:Voter ID:server-7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81 Address:7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81}]"
2025-01-25T14:29:01.090Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:01.095Z [INFO]  server-81a5ded8-c6bc-e8f7-620f-68829f653f2b: entering follower state: follower="Node at 81a5ded8-c6bc-e8f7-620f-68829f653f2b [Follower]" leader-address= leader-id=
2025-01-25T14:29:01.095Z [INFO]  server-6b72411d-f485-bd29-ad17-20e98c399204: entering follower state: follower="Node at 6b72411d-f485-bd29-ad17-20e98c399204 [Follower]" leader-address= leader-id=
2025-01-25T14:29:01.095Z [INFO]  server-7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81: entering follower state: follower="Node at 7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81 [Follower]" leader-address= leader-id=
2025-01-25T14:29:01.695Z [WARN]  server-7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:01.695Z [INFO]  server-7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81: entering candidate state: node="Node at 7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81 [Candidate]" term=2
2025-01-25T14:29:01.695Z [INFO]  server-7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81: election won: term=2 tally=2
2025-01-25T14:29:01.695Z [INFO]  server-7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81: entering leader state: leader="Node at 7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81 [Leader]"
2025-01-25T14:29:01.695Z [INFO]  server-7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81: added peer, starting replication: peer=server-81a5ded8-c6bc-e8f7-620f-68829f653f2b
2025-01-25T14:29:01.695Z [INFO]  server-7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81: added peer, starting replication: peer=server-6b72411d-f485-bd29-ad17-20e98c399204
2025-01-25T14:29:01.695Z [INFO]  server-7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81: pipelining replication: peer="{Voter server-6b72411d-f485-bd29-ad17-20e98c399204 6b72411d-f485-bd29-ad17-20e98c399204}"
2025-01-25T14:29:01.695Z [INFO]  server-7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81: pipelining replication: peer="{Voter server-81a5ded8-c6bc-e8f7-620f-68829f653f2b 81a5ded8-c6bc-e8f7-620f-68829f653f2b}"
2025-01-25T14:29:02.700Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:01.695723765 +0000 UTC m=+51.550048747 (1 nodes), 604.95899ms from start of poll, 606.36286ms from cluster start. Timeout at 2025-01-25 14:29:02.700805282 +0000 UTC m=+52.555130384, 1.005081637s after stability
2025-01-25T14:29:02.702Z [INFO]  server-7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81: starting snapshot up to: index=12
2025-01-25T14:29:02.702Z [INFO]  creating new snapshot: path=/tmp/raft359287863/snapshots/2-12-1737815342702.tmp
2025-01-25T14:29:02.704Z [INFO]  server-7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81: snapshot complete up to: index=12
2025-01-25T14:29:02.707Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ea6adc0-08bd-23b6-ada9-e03113001cd6 Address:9ea6adc0-08bd-23b6-ada9-e03113001cd6} {Suffrage:Voter ID:server-478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb Address:478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb} {Suffrage:Voter ID:server-33ba07e9-155d-372e-58cb-89ce36575d4d Address:33ba07e9-155d-372e-58cb-89ce36575d4d}]"
2025-01-25T14:29:02.708Z [INFO]  server-478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ea6adc0-08bd-23b6-ada9-e03113001cd6 Address:9ea6adc0-08bd-23b6-ada9-e03113001cd6} {Suffrage:Voter ID:server-478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb Address:478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb} {Suffrage:Voter ID:server-33ba07e9-155d-372e-58cb-89ce36575d4d Address:33ba07e9-155d-372e-58cb-89ce36575d4d}]"
2025-01-25T14:29:02.708Z [INFO]  server-33ba07e9-155d-372e-58cb-89ce36575d4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ea6adc0-08bd-23b6-ada9-e03113001cd6 Address:9ea6adc0-08bd-23b6-ada9-e03113001cd6} {Suffrage:Voter ID:server-478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb Address:478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb} {Suffrage:Voter ID:server-33ba07e9-155d-372e-58cb-89ce36575d4d Address:33ba07e9-155d-372e-58cb-89ce36575d4d}]"
2025-01-25T14:29:02.708Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:02.709Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: entering follower state: follower="Node at 9ea6adc0-08bd-23b6-ada9-e03113001cd6 [Follower]" leader-address= leader-id=
2025-01-25T14:29:02.709Z [INFO]  server-478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb: entering follower state: follower="Node at 478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb [Follower]" leader-address= leader-id=
2025-01-25T14:29:02.710Z [INFO]  server-33ba07e9-155d-372e-58cb-89ce36575d4d: entering follower state: follower="Node at 33ba07e9-155d-372e-58cb-89ce36575d4d [Follower]" leader-address= leader-id=
2025-01-25T14:29:03.264Z [WARN]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:03.264Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: entering candidate state: node="Node at 9ea6adc0-08bd-23b6-ada9-e03113001cd6 [Candidate]" term=2
2025-01-25T14:29:03.264Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: election won: term=2 tally=2
2025-01-25T14:29:03.264Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: entering leader state: leader="Node at 9ea6adc0-08bd-23b6-ada9-e03113001cd6 [Leader]"
2025-01-25T14:29:03.264Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: added peer, starting replication: peer=server-478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb
2025-01-25T14:29:03.264Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: added peer, starting replication: peer=server-33ba07e9-155d-372e-58cb-89ce36575d4d
2025-01-25T14:29:03.264Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: pipelining replication: peer="{Voter server-33ba07e9-155d-372e-58cb-89ce36575d4d 33ba07e9-155d-372e-58cb-89ce36575d4d}"
2025-01-25T14:29:03.265Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: pipelining replication: peer="{Voter server-478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb 478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb}"
2025-01-25T14:29:04.271Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:03.265123529 +0000 UTC m=+53.119448511 (1 nodes), 556.103348ms from start of poll, 557.651773ms from cluster start. Timeout at 2025-01-25 14:29:04.270145888 +0000 UTC m=+54.124470870, 1.005022359s after stability
2025-01-25T14:29:04.272Z [INFO]  creating new snapshot: path=/tmp/raft3325980079/snapshots/2-15-1737815344272.tmp
2025-01-25T14:29:04.273Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: copied to local snapshot: bytes=71
2025-01-25T14:29:04.273Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: snapshot restore progress: id=2-15-1737815344272 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:04.273Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: removing all old logs from log store
2025-01-25T14:29:04.273Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: compacting logs: from=1 to=2
2025-01-25T14:29:04.273Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: restored user snapshot: index=15
2025-01-25T14:29:04.273Z [ERROR] server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: failed to get log: index=2 error="log not found"
2025-01-25T14:29:04.273Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: aborting pipeline replication: peer="{Voter server-33ba07e9-155d-372e-58cb-89ce36575d4d 33ba07e9-155d-372e-58cb-89ce36575d4d}"
2025-01-25T14:29:04.273Z [ERROR] server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: failed to get log: index=2 error="log not found"
2025-01-25T14:29:04.274Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: aborting pipeline replication: peer="{Voter server-478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb 478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb}"
2025-01-25T14:29:04.281Z [ERROR] server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: failed to get log: index=2 error="log not found"
2025-01-25T14:29:04.282Z [INFO]  creating new snapshot: path=/tmp/raft2141987800/snapshots/2-15-1737815344282.tmp
2025-01-25T14:29:04.282Z [INFO]  server-33ba07e9-155d-372e-58cb-89ce36575d4d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:04.283Z [ERROR] server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: failed to get log: index=2 error="log not found"
2025-01-25T14:29:04.283Z [INFO]  server-33ba07e9-155d-372e-58cb-89ce36575d4d: copied to local snapshot: bytes=71
2025-01-25T14:29:04.283Z [INFO]  creating new snapshot: path=/tmp/raft3942468882/snapshots/2-15-1737815344283.tmp
2025-01-25T14:29:04.284Z [INFO]  server-33ba07e9-155d-372e-58cb-89ce36575d4d: snapshot restore progress: id=2-15-1737815344282 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:04.284Z [INFO]  server-33ba07e9-155d-372e-58cb-89ce36575d4d: removing all old logs from log store
2025-01-25T14:29:04.284Z [INFO]  server-33ba07e9-155d-372e-58cb-89ce36575d4d: compacting logs: from=1 to=2
2025-01-25T14:29:04.284Z [INFO]  server-33ba07e9-155d-372e-58cb-89ce36575d4d: Installed remote snapshot
2025-01-25T14:29:04.284Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: pipelining replication: peer="{Voter server-33ba07e9-155d-372e-58cb-89ce36575d4d 33ba07e9-155d-372e-58cb-89ce36575d4d}"
2025-01-25T14:29:04.284Z [INFO]  server-478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:04.284Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:04.284Z [INFO]  server-478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb: copied to local snapshot: bytes=71
2025-01-25T14:29:04.285Z [INFO]  server-478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb: snapshot restore progress: id=2-15-1737815344283 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:04.285Z [INFO]  server-478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb: removing all old logs from log store
2025-01-25T14:29:04.285Z [INFO]  server-478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb: compacting logs: from=1 to=2
2025-01-25T14:29:04.285Z [INFO]  server-478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb: Installed remote snapshot
2025-01-25T14:29:04.285Z [INFO]  server-9ea6adc0-08bd-23b6-ada9-e03113001cd6: pipelining replication: peer="{Voter server-478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb 478ea1b6-6ebc-7753-81e5-cd82f2dbbbdb}"
2025-01-25T14:29:05.290Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:04.284499286 +0000 UTC m=+54.138824268 (2 nodes), 1.6µs from start of poll, 1.57702753s from cluster start. Timeout at 2025-01-25 14:29:05.289619722 +0000 UTC m=+55.143944664, 1.005120396s after stability
2025-01-25T14:29:05.302Z [INFO]  server-7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81: aborting pipeline replication: peer="{Voter server-6b72411d-f485-bd29-ad17-20e98c399204 6b72411d-f485-bd29-ad17-20e98c399204}"
2025-01-25T14:29:05.303Z [INFO]  server-7f3bea51-66dc-f7d3-4fa6-21da2d1d0c81: aborting pipeline replication: peer="{Voter server-81a5ded8-c6bc-e8f7-620f-68829f653f2b 81a5ded8-c6bc-e8f7-620f-68829f653f2b}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false
2025-01-25T14:29:05.305Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60a0aeb6-2bbb-e914-e26b-58c2a774307f Address:60a0aeb6-2bbb-e914-e26b-58c2a774307f} {Suffrage:Voter ID:server-8f192e0d-4f48-540f-b398-63441088583c Address:8f192e0d-4f48-540f-b398-63441088583c} {Suffrage:Voter ID:server-0fdc1750-f512-61e0-7d96-61c5d384ae0a Address:0fdc1750-f512-61e0-7d96-61c5d384ae0a}]"
2025-01-25T14:29:05.305Z [INFO]  server-8f192e0d-4f48-540f-b398-63441088583c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60a0aeb6-2bbb-e914-e26b-58c2a774307f Address:60a0aeb6-2bbb-e914-e26b-58c2a774307f} {Suffrage:Voter ID:server-8f192e0d-4f48-540f-b398-63441088583c Address:8f192e0d-4f48-540f-b398-63441088583c} {Suffrage:Voter ID:server-0fdc1750-f512-61e0-7d96-61c5d384ae0a Address:0fdc1750-f512-61e0-7d96-61c5d384ae0a}]"
2025-01-25T14:29:05.306Z [INFO]  server-0fdc1750-f512-61e0-7d96-61c5d384ae0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60a0aeb6-2bbb-e914-e26b-58c2a774307f Address:60a0aeb6-2bbb-e914-e26b-58c2a774307f} {Suffrage:Voter ID:server-8f192e0d-4f48-540f-b398-63441088583c Address:8f192e0d-4f48-540f-b398-63441088583c} {Suffrage:Voter ID:server-0fdc1750-f512-61e0-7d96-61c5d384ae0a Address:0fdc1750-f512-61e0-7d96-61c5d384ae0a}]"
2025-01-25T14:29:05.306Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:05.306Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: entering follower state: follower="Node at 60a0aeb6-2bbb-e914-e26b-58c2a774307f [Follower]" leader-address= leader-id=
2025-01-25T14:29:05.306Z [INFO]  server-0fdc1750-f512-61e0-7d96-61c5d384ae0a: entering follower state: follower="Node at 0fdc1750-f512-61e0-7d96-61c5d384ae0a [Follower]" leader-address= leader-id=
2025-01-25T14:29:05.306Z [INFO]  server-8f192e0d-4f48-540f-b398-63441088583c: entering follower state: follower="Node at 8f192e0d-4f48-540f-b398-63441088583c [Follower]" leader-address= leader-id=
2025-01-25T14:29:05.872Z [WARN]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:05.872Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: entering candidate state: node="Node at 60a0aeb6-2bbb-e914-e26b-58c2a774307f [Candidate]" term=2
2025-01-25T14:29:05.872Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: election won: term=2 tally=2
2025-01-25T14:29:05.872Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: entering leader state: leader="Node at 60a0aeb6-2bbb-e914-e26b-58c2a774307f [Leader]"
2025-01-25T14:29:05.872Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: added peer, starting replication: peer=server-8f192e0d-4f48-540f-b398-63441088583c
2025-01-25T14:29:05.872Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: added peer, starting replication: peer=server-0fdc1750-f512-61e0-7d96-61c5d384ae0a
2025-01-25T14:29:05.872Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: pipelining replication: peer="{Voter server-8f192e0d-4f48-540f-b398-63441088583c 8f192e0d-4f48-540f-b398-63441088583c}"
2025-01-25T14:29:05.872Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: pipelining replication: peer="{Voter server-0fdc1750-f512-61e0-7d96-61c5d384ae0a 0fdc1750-f512-61e0-7d96-61c5d384ae0a}"
2025-01-25T14:29:06.878Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:05.872391984 +0000 UTC m=+55.726716966 (1 nodes), 566.185825ms from start of poll, 566.88652ms from cluster start. Timeout at 2025-01-25 14:29:06.877396344 +0000 UTC m=+56.731721326, 1.00500436s after stability
2025-01-25T14:29:06.879Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: starting snapshot up to: index=12
2025-01-25T14:29:06.879Z [INFO]  creating new snapshot: path=/tmp/raft1578067068/snapshots/2-12-1737815346879.tmp
2025-01-25T14:29:06.883Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: snapshot complete up to: index=12
2025-01-25T14:29:06.885Z [INFO]  creating new snapshot: path=/tmp/raft1578067068/snapshots/2-113-1737815346885.tmp
2025-01-25T14:29:06.886Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: copied to local snapshot: bytes=71
2025-01-25T14:29:06.886Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: snapshot restore progress: id=2-113-1737815346885 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:06.886Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: restored user snapshot: index=113
2025-01-25T14:29:06.886Z [ERROR] server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: failed to get log: index=23 error="log not found"
2025-01-25T14:29:06.886Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: aborting pipeline replication: peer="{Voter server-0fdc1750-f512-61e0-7d96-61c5d384ae0a 0fdc1750-f512-61e0-7d96-61c5d384ae0a}"
2025-01-25T14:29:06.886Z [ERROR] server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: failed to get log: index=23 error="log not found"
2025-01-25T14:29:06.887Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: aborting pipeline replication: peer="{Voter server-8f192e0d-4f48-540f-b398-63441088583c 8f192e0d-4f48-540f-b398-63441088583c}"
2025-01-25T14:29:06.893Z [ERROR] server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: failed to get log: index=23 error="log not found"
2025-01-25T14:29:06.894Z [INFO]  creating new snapshot: path=/tmp/raft1507187908/snapshots/2-113-1737815346894.tmp
2025-01-25T14:29:06.894Z [INFO]  server-8f192e0d-4f48-540f-b398-63441088583c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:06.895Z [INFO]  server-8f192e0d-4f48-540f-b398-63441088583c: copied to local snapshot: bytes=71
2025-01-25T14:29:06.895Z [ERROR] server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: failed to get log: index=23 error="log not found"
2025-01-25T14:29:06.895Z [INFO]  server-8f192e0d-4f48-540f-b398-63441088583c: snapshot restore progress: id=2-113-1737815346894 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:06.896Z [INFO]  server-8f192e0d-4f48-540f-b398-63441088583c: Installed remote snapshot
2025-01-25T14:29:06.896Z [WARN]  server-8f192e0d-4f48-540f-b398-63441088583c: skipping application of old log: index=22
2025-01-25T14:29:06.896Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: pipelining replication: peer="{Voter server-8f192e0d-4f48-540f-b398-63441088583c 8f192e0d-4f48-540f-b398-63441088583c}"
2025-01-25T14:29:06.896Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:06.896Z [INFO]  creating new snapshot: path=/tmp/raft1148884927/snapshots/2-113-1737815346896.tmp
2025-01-25T14:29:06.897Z [INFO]  server-0fdc1750-f512-61e0-7d96-61c5d384ae0a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:06.897Z [INFO]  server-0fdc1750-f512-61e0-7d96-61c5d384ae0a: copied to local snapshot: bytes=71
2025-01-25T14:29:06.898Z [INFO]  server-0fdc1750-f512-61e0-7d96-61c5d384ae0a: snapshot restore progress: id=2-113-1737815346896 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:06.898Z [INFO]  server-0fdc1750-f512-61e0-7d96-61c5d384ae0a: Installed remote snapshot
2025-01-25T14:29:06.899Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: pipelining replication: peer="{Voter server-0fdc1750-f512-61e0-7d96-61c5d384ae0a 0fdc1750-f512-61e0-7d96-61c5d384ae0a}"
2025-01-25T14:29:07.901Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:06.896420374 +0000 UTC m=+56.750745356 (2 nodes), 1.92µs from start of poll, 1.59091491s from cluster start. Timeout at 2025-01-25 14:29:07.90153801 +0000 UTC m=+57.755862992, 1.005117636s after stability
2025-01-25T14:29:07.913Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: aborting pipeline replication: peer="{Voter server-0fdc1750-f512-61e0-7d96-61c5d384ae0a 0fdc1750-f512-61e0-7d96-61c5d384ae0a}"
2025-01-25T14:29:07.913Z [INFO]  server-60a0aeb6-2bbb-e914-e26b-58c2a774307f: aborting pipeline replication: peer="{Voter server-8f192e0d-4f48-540f-b398-63441088583c 8f192e0d-4f48-540f-b398-63441088583c}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false
2025-01-25T14:29:07.916Z [INFO]  server-47e6297d-dcde-53b0-1987-3d1bf19aee22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47e6297d-dcde-53b0-1987-3d1bf19aee22 Address:47e6297d-dcde-53b0-1987-3d1bf19aee22} {Suffrage:Voter ID:server-87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d Address:87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d} {Suffrage:Voter ID:server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240 Address:709cdcd2-8b8f-49ad-a0ec-e70d24c41240}]"
2025-01-25T14:29:07.916Z [INFO]  server-87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47e6297d-dcde-53b0-1987-3d1bf19aee22 Address:47e6297d-dcde-53b0-1987-3d1bf19aee22} {Suffrage:Voter ID:server-87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d Address:87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d} {Suffrage:Voter ID:server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240 Address:709cdcd2-8b8f-49ad-a0ec-e70d24c41240}]"
2025-01-25T14:29:07.916Z [INFO]  server-47e6297d-dcde-53b0-1987-3d1bf19aee22: entering follower state: follower="Node at 47e6297d-dcde-53b0-1987-3d1bf19aee22 [Follower]" leader-address= leader-id=
2025-01-25T14:29:07.916Z [INFO]  server-87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d: entering follower state: follower="Node at 87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d [Follower]" leader-address= leader-id=
2025-01-25T14:29:07.916Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47e6297d-dcde-53b0-1987-3d1bf19aee22 Address:47e6297d-dcde-53b0-1987-3d1bf19aee22} {Suffrage:Voter ID:server-87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d Address:87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d} {Suffrage:Voter ID:server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240 Address:709cdcd2-8b8f-49ad-a0ec-e70d24c41240}]"
2025-01-25T14:29:07.917Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:07.917Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: entering follower state: follower="Node at 709cdcd2-8b8f-49ad-a0ec-e70d24c41240 [Follower]" leader-address= leader-id=
2025-01-25T14:29:08.648Z [WARN]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:08.648Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: entering candidate state: node="Node at 709cdcd2-8b8f-49ad-a0ec-e70d24c41240 [Candidate]" term=2
2025-01-25T14:29:08.648Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: election won: term=2 tally=2
2025-01-25T14:29:08.648Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: entering leader state: leader="Node at 709cdcd2-8b8f-49ad-a0ec-e70d24c41240 [Leader]"
2025-01-25T14:29:08.648Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: added peer, starting replication: peer=server-47e6297d-dcde-53b0-1987-3d1bf19aee22
2025-01-25T14:29:08.648Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: added peer, starting replication: peer=server-87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d
2025-01-25T14:29:08.648Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: pipelining replication: peer="{Voter server-47e6297d-dcde-53b0-1987-3d1bf19aee22 47e6297d-dcde-53b0-1987-3d1bf19aee22}"
2025-01-25T14:29:08.648Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: pipelining replication: peer="{Voter server-87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d 87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d}"
2025-01-25T14:29:09.654Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:08.648464325 +0000 UTC m=+58.502789307 (1 nodes), 731.416879ms from start of poll, 732.175772ms from cluster start. Timeout at 2025-01-25 14:29:09.653467525 +0000 UTC m=+59.507792467, 1.00500316s after stability
2025-01-25T14:29:09.655Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: starting snapshot up to: index=12
2025-01-25T14:29:09.655Z [INFO]  creating new snapshot: path=/tmp/raft3744132961/snapshots/2-12-1737815349655.tmp
2025-01-25T14:29:09.657Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: snapshot complete up to: index=12
2025-01-25T14:29:09.658Z [INFO]  creating new snapshot: path=/tmp/raft3744132961/snapshots/2-113-1737815349658.tmp
2025-01-25T14:29:09.659Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: copied to local snapshot: bytes=71
2025-01-25T14:29:09.659Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: snapshot restore progress: id=2-113-1737815349658 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:09.660Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: removing all old logs from log store
2025-01-25T14:29:09.660Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: compacting logs: from=1 to=22
2025-01-25T14:29:09.660Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: restored user snapshot: index=113
2025-01-25T14:29:09.660Z [ERROR] server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: failed to get log: index=22 error="log not found"
2025-01-25T14:29:09.660Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: aborting pipeline replication: peer="{Voter server-87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d 87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d}"
2025-01-25T14:29:09.660Z [ERROR] server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: failed to get log: index=22 error="log not found"
2025-01-25T14:29:09.660Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: aborting pipeline replication: peer="{Voter server-47e6297d-dcde-53b0-1987-3d1bf19aee22 47e6297d-dcde-53b0-1987-3d1bf19aee22}"
2025-01-25T14:29:09.668Z [ERROR] server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: failed to get log: index=22 error="log not found"
2025-01-25T14:29:09.670Z [ERROR] server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: failed to get log: index=22 error="log not found"
2025-01-25T14:29:09.673Z [INFO]  creating new snapshot: path=/tmp/raft3954771927/snapshots/2-113-1737815349673.tmp
2025-01-25T14:29:09.674Z [INFO]  creating new snapshot: path=/tmp/raft3553680851/snapshots/2-113-1737815349674.tmp
2025-01-25T14:29:09.674Z [INFO]  server-87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:09.674Z [INFO]  server-47e6297d-dcde-53b0-1987-3d1bf19aee22: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:09.674Z [INFO]  server-87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d: copied to local snapshot: bytes=71
2025-01-25T14:29:09.675Z [INFO]  server-87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d: snapshot restore progress: id=2-113-1737815349673 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:09.675Z [INFO]  server-87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d: removing all old logs from log store
2025-01-25T14:29:09.675Z [INFO]  server-87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d: compacting logs: from=1 to=22
2025-01-25T14:29:09.675Z [INFO]  server-87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d: Installed remote snapshot
2025-01-25T14:29:09.675Z [INFO]  server-47e6297d-dcde-53b0-1987-3d1bf19aee22: copied to local snapshot: bytes=71
2025-01-25T14:29:09.675Z [WARN]  server-87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d: skipping application of old log: index=22
2025-01-25T14:29:09.675Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: pipelining replication: peer="{Voter server-87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d 87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d}"
2025-01-25T14:29:09.675Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:09.676Z [INFO]  server-47e6297d-dcde-53b0-1987-3d1bf19aee22: snapshot restore progress: id=2-113-1737815349674 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:09.676Z [INFO]  server-47e6297d-dcde-53b0-1987-3d1bf19aee22: removing all old logs from log store
2025-01-25T14:29:09.676Z [INFO]  server-47e6297d-dcde-53b0-1987-3d1bf19aee22: compacting logs: from=1 to=22
2025-01-25T14:29:09.676Z [INFO]  server-47e6297d-dcde-53b0-1987-3d1bf19aee22: Installed remote snapshot
2025-01-25T14:29:09.676Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: pipelining replication: peer="{Voter server-47e6297d-dcde-53b0-1987-3d1bf19aee22 47e6297d-dcde-53b0-1987-3d1bf19aee22}"
2025-01-25T14:29:10.682Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:09.675995109 +0000 UTC m=+59.530320091 (2 nodes), 1.8µs from start of poll, 1.759706556s from cluster start. Timeout at 2025-01-25 14:29:10.681126305 +0000 UTC m=+60.535451287, 1.005131196s after stability
2025-01-25T14:29:10.696Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: aborting pipeline replication: peer="{Voter server-87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d 87895c1b-cd49-9b7e-8c44-87a3a6fdbf1d}"
2025-01-25T14:29:10.696Z [INFO]  server-709cdcd2-8b8f-49ad-a0ec-e70d24c41240: aborting pipeline replication: peer="{Voter server-47e6297d-dcde-53b0-1987-3d1bf19aee22 47e6297d-dcde-53b0-1987-3d1bf19aee22}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true
2025-01-25T14:29:10.700Z [INFO]  server-2e270c31-33c1-8d85-3ca6-32058f0cfb45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e270c31-33c1-8d85-3ca6-32058f0cfb45 Address:2e270c31-33c1-8d85-3ca6-32058f0cfb45} {Suffrage:Voter ID:server-d13575af-1ae0-6f27-be3b-ce0f235f7a45 Address:d13575af-1ae0-6f27-be3b-ce0f235f7a45} {Suffrage:Voter ID:server-0d12f722-391f-8b5c-3382-260ef915d052 Address:0d12f722-391f-8b5c-3382-260ef915d052}]"
2025-01-25T14:29:10.700Z [INFO]  server-d13575af-1ae0-6f27-be3b-ce0f235f7a45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e270c31-33c1-8d85-3ca6-32058f0cfb45 Address:2e270c31-33c1-8d85-3ca6-32058f0cfb45} {Suffrage:Voter ID:server-d13575af-1ae0-6f27-be3b-ce0f235f7a45 Address:d13575af-1ae0-6f27-be3b-ce0f235f7a45} {Suffrage:Voter ID:server-0d12f722-391f-8b5c-3382-260ef915d052 Address:0d12f722-391f-8b5c-3382-260ef915d052}]"
2025-01-25T14:29:10.700Z [INFO]  server-0d12f722-391f-8b5c-3382-260ef915d052: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e270c31-33c1-8d85-3ca6-32058f0cfb45 Address:2e270c31-33c1-8d85-3ca6-32058f0cfb45} {Suffrage:Voter ID:server-d13575af-1ae0-6f27-be3b-ce0f235f7a45 Address:d13575af-1ae0-6f27-be3b-ce0f235f7a45} {Suffrage:Voter ID:server-0d12f722-391f-8b5c-3382-260ef915d052 Address:0d12f722-391f-8b5c-3382-260ef915d052}]"
2025-01-25T14:29:10.700Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:10.700Z [INFO]  server-2e270c31-33c1-8d85-3ca6-32058f0cfb45: entering follower state: follower="Node at 2e270c31-33c1-8d85-3ca6-32058f0cfb45 [Follower]" leader-address= leader-id=
2025-01-25T14:29:10.701Z [INFO]  server-d13575af-1ae0-6f27-be3b-ce0f235f7a45: entering follower state: follower="Node at d13575af-1ae0-6f27-be3b-ce0f235f7a45 [Follower]" leader-address= leader-id=
2025-01-25T14:29:10.701Z [INFO]  server-0d12f722-391f-8b5c-3382-260ef915d052: entering follower state: follower="Node at 0d12f722-391f-8b5c-3382-260ef915d052 [Follower]" leader-address= leader-id=
2025-01-25T14:29:11.261Z [WARN]  server-d13575af-1ae0-6f27-be3b-ce0f235f7a45: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:11.261Z [INFO]  server-d13575af-1ae0-6f27-be3b-ce0f235f7a45: entering candidate state: node="Node at d13575af-1ae0-6f27-be3b-ce0f235f7a45 [Candidate]" term=2
2025-01-25T14:29:11.261Z [INFO]  server-d13575af-1ae0-6f27-be3b-ce0f235f7a45: election won: term=2 tally=2
2025-01-25T14:29:11.262Z [INFO]  server-d13575af-1ae0-6f27-be3b-ce0f235f7a45: entering leader state: leader="Node at d13575af-1ae0-6f27-be3b-ce0f235f7a45 [Leader]"
2025-01-25T14:29:11.262Z [INFO]  server-d13575af-1ae0-6f27-be3b-ce0f235f7a45: added peer, starting replication: peer=server-2e270c31-33c1-8d85-3ca6-32058f0cfb45
2025-01-25T14:29:11.262Z [INFO]  server-d13575af-1ae0-6f27-be3b-ce0f235f7a45: added peer, starting replication: peer=server-0d12f722-391f-8b5c-3382-260ef915d052
2025-01-25T14:29:11.262Z [INFO]  server-d13575af-1ae0-6f27-be3b-ce0f235f7a45: pipelining replication: peer="{Voter server-2e270c31-33c1-8d85-3ca6-32058f0cfb45 2e270c31-33c1-8d85-3ca6-32058f0cfb45}"
2025-01-25T14:29:11.262Z [INFO]  server-d13575af-1ae0-6f27-be3b-ce0f235f7a45: pipelining replication: peer="{Voter server-0d12f722-391f-8b5c-3382-260ef915d052 0d12f722-391f-8b5c-3382-260ef915d052}"
2025-01-25T14:29:12.268Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:11.262031714 +0000 UTC m=+61.116356696 (1 nodes), 561.115888ms from start of poll, 561.790104ms from cluster start. Timeout at 2025-01-25 14:29:12.267046953 +0000 UTC m=+62.121371935, 1.005015239s after stability
2025-01-25T14:29:12.269Z [INFO]  server-d13575af-1ae0-6f27-be3b-ce0f235f7a45: starting snapshot up to: index=12
2025-01-25T14:29:12.269Z [INFO]  creating new snapshot: path=/tmp/raft3277779116/snapshots/2-12-1737815352269.tmp
2025-01-25T14:29:12.270Z [INFO]  server-d13575af-1ae0-6f27-be3b-ce0f235f7a45: snapshot complete up to: index=12
2025-01-25T14:29:12.273Z [INFO]  server-4c93850d-69b2-b675-eec5-d238b7c865c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c93850d-69b2-b675-eec5-d238b7c865c6 Address:4c93850d-69b2-b675-eec5-d238b7c865c6} {Suffrage:Voter ID:server-b7356e8b-ceb7-e553-a82d-832d4d96dcb8 Address:b7356e8b-ceb7-e553-a82d-832d4d96dcb8} {Suffrage:Voter ID:server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd Address:5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd}]"
2025-01-25T14:29:12.273Z [INFO]  server-4c93850d-69b2-b675-eec5-d238b7c865c6: entering follower state: follower="Node at 4c93850d-69b2-b675-eec5-d238b7c865c6 [Follower]" leader-address= leader-id=
2025-01-25T14:29:12.273Z [INFO]  server-b7356e8b-ceb7-e553-a82d-832d4d96dcb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c93850d-69b2-b675-eec5-d238b7c865c6 Address:4c93850d-69b2-b675-eec5-d238b7c865c6} {Suffrage:Voter ID:server-b7356e8b-ceb7-e553-a82d-832d4d96dcb8 Address:b7356e8b-ceb7-e553-a82d-832d4d96dcb8} {Suffrage:Voter ID:server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd Address:5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd}]"
2025-01-25T14:29:12.273Z [INFO]  server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c93850d-69b2-b675-eec5-d238b7c865c6 Address:4c93850d-69b2-b675-eec5-d238b7c865c6} {Suffrage:Voter ID:server-b7356e8b-ceb7-e553-a82d-832d4d96dcb8 Address:b7356e8b-ceb7-e553-a82d-832d4d96dcb8} {Suffrage:Voter ID:server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd Address:5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd}]"
2025-01-25T14:29:12.273Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:12.273Z [INFO]  server-b7356e8b-ceb7-e553-a82d-832d4d96dcb8: entering follower state: follower="Node at b7356e8b-ceb7-e553-a82d-832d4d96dcb8 [Follower]" leader-address= leader-id=
2025-01-25T14:29:12.273Z [INFO]  server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: entering follower state: follower="Node at 5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd [Follower]" leader-address= leader-id=
2025-01-25T14:29:12.788Z [WARN]  server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:12.788Z [INFO]  server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: entering candidate state: node="Node at 5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd [Candidate]" term=2
2025-01-25T14:29:12.789Z [INFO]  server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: election won: term=2 tally=2
2025-01-25T14:29:12.789Z [INFO]  server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: entering leader state: leader="Node at 5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd [Leader]"
2025-01-25T14:29:12.789Z [INFO]  server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: added peer, starting replication: peer=server-4c93850d-69b2-b675-eec5-d238b7c865c6
2025-01-25T14:29:12.789Z [INFO]  server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: added peer, starting replication: peer=server-b7356e8b-ceb7-e553-a82d-832d4d96dcb8
2025-01-25T14:29:12.789Z [INFO]  server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: pipelining replication: peer="{Voter server-b7356e8b-ceb7-e553-a82d-832d4d96dcb8 b7356e8b-ceb7-e553-a82d-832d4d96dcb8}"
2025-01-25T14:29:12.789Z [INFO]  server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: pipelining replication: peer="{Voter server-4c93850d-69b2-b675-eec5-d238b7c865c6 4c93850d-69b2-b675-eec5-d238b7c865c6}"
2025-01-25T14:29:13.794Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:12.789430749 +0000 UTC m=+62.643755691 (1 nodes), 515.863397ms from start of poll, 516.556292ms from cluster start. Timeout at 2025-01-25 14:29:13.794432909 +0000 UTC m=+63.648757891, 1.0050022s after stability
2025-01-25T14:29:13.795Z [INFO]  creating new snapshot: path=/tmp/raft3385829941/snapshots/2-113-1737815353795.tmp
2025-01-25T14:29:13.796Z [INFO]  server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: copied to local snapshot: bytes=71
2025-01-25T14:29:13.797Z [INFO]  server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: snapshot restore progress: id=2-113-1737815353795 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:13.797Z [INFO]  server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: restored user snapshot: index=113
2025-01-25T14:29:13.797Z [ERROR] server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: failed to get log: index=3 error="log not found"
2025-01-25T14:29:13.797Z [INFO]  server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: aborting pipeline replication: peer="{Voter server-b7356e8b-ceb7-e553-a82d-832d4d96dcb8 b7356e8b-ceb7-e553-a82d-832d4d96dcb8}"
2025-01-25T14:29:13.797Z [ERROR] server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: failed to get log: index=3 error="log not found"
2025-01-25T14:29:13.797Z [INFO]  server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: aborting pipeline replication: peer="{Voter server-4c93850d-69b2-b675-eec5-d238b7c865c6 4c93850d-69b2-b675-eec5-d238b7c865c6}"
2025-01-25T14:29:13.807Z [ERROR] server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: failed to get log: index=3 error="log not found"
2025-01-25T14:29:13.807Z [ERROR] server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: failed to get log: index=3 error="log not found"
2025-01-25T14:29:13.808Z [INFO]  creating new snapshot: path=/tmp/raft1627652999/snapshots/2-113-1737815353807.tmp
2025-01-25T14:29:13.808Z [INFO]  creating new snapshot: path=/tmp/raft1111282116/snapshots/2-113-1737815353808.tmp
2025-01-25T14:29:13.808Z [INFO]  server-b7356e8b-ceb7-e553-a82d-832d4d96dcb8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:13.812Z [INFO]  server-b7356e8b-ceb7-e553-a82d-832d4d96dcb8: copied to local snapshot: bytes=71
2025-01-25T14:29:13.815Z [INFO]  server-4c93850d-69b2-b675-eec5-d238b7c865c6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:13.816Z [INFO]  server-b7356e8b-ceb7-e553-a82d-832d4d96dcb8: snapshot restore progress: id=2-113-1737815353807 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:13.816Z [INFO]  server-b7356e8b-ceb7-e553-a82d-832d4d96dcb8: Installed remote snapshot
2025-01-25T14:29:13.816Z [INFO]  server-4c93850d-69b2-b675-eec5-d238b7c865c6: copied to local snapshot: bytes=71
2025-01-25T14:29:13.816Z [INFO]  server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: pipelining replication: peer="{Voter server-b7356e8b-ceb7-e553-a82d-832d4d96dcb8 b7356e8b-ceb7-e553-a82d-832d4d96dcb8}"
2025-01-25T14:29:13.817Z [INFO]  server-4c93850d-69b2-b675-eec5-d238b7c865c6: snapshot restore progress: id=2-113-1737815353808 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:13.817Z [INFO]  server-4c93850d-69b2-b675-eec5-d238b7c865c6: Installed remote snapshot
2025-01-25T14:29:13.817Z [INFO]  server-5c30a755-0453-9d1e-e6c1-9e7b81e9fbdd: pipelining replication: peer="{Voter server-4c93850d-69b2-b675-eec5-d238b7c865c6 4c93850d-69b2-b675-eec5-d238b7c865c6}"
2025-01-25T14:29:13.817Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:14.823Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:13.817759343 +0000 UTC m=+63.672084325 (2 nodes), 1.56µs from start of poll, 1.544884926s from cluster start. Timeout at 2025-01-25 14:29:14.822879139 +0000 UTC m=+64.677204081, 1.005119756s after stability
2025-01-25T14:29:14.835Z [INFO]  server-d13575af-1ae0-6f27-be3b-ce0f235f7a45: aborting pipeline replication: peer="{Voter server-0d12f722-391f-8b5c-3382-260ef915d052 0d12f722-391f-8b5c-3382-260ef915d052}"
2025-01-25T14:29:14.835Z [INFO]  server-d13575af-1ae0-6f27-be3b-ce0f235f7a45: aborting pipeline replication: peer="{Voter server-2e270c31-33c1-8d85-3ca6-32058f0cfb45 2e270c31-33c1-8d85-3ca6-32058f0cfb45}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true
2025-01-25T14:29:14.838Z [INFO]  server-15c5dab3-22d4-23b7-e103-bf5a63a3ccd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15c5dab3-22d4-23b7-e103-bf5a63a3ccd1 Address:15c5dab3-22d4-23b7-e103-bf5a63a3ccd1} {Suffrage:Voter ID:server-977f50ab-3940-9f74-c6ca-9e4cffe8927a Address:977f50ab-3940-9f74-c6ca-9e4cffe8927a} {Suffrage:Voter ID:server-65501e75-bc18-c9d2-9e54-6a5136e6decd Address:65501e75-bc18-c9d2-9e54-6a5136e6decd}]"
2025-01-25T14:29:14.838Z [INFO]  server-977f50ab-3940-9f74-c6ca-9e4cffe8927a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15c5dab3-22d4-23b7-e103-bf5a63a3ccd1 Address:15c5dab3-22d4-23b7-e103-bf5a63a3ccd1} {Suffrage:Voter ID:server-977f50ab-3940-9f74-c6ca-9e4cffe8927a Address:977f50ab-3940-9f74-c6ca-9e4cffe8927a} {Suffrage:Voter ID:server-65501e75-bc18-c9d2-9e54-6a5136e6decd Address:65501e75-bc18-c9d2-9e54-6a5136e6decd}]"
2025-01-25T14:29:14.838Z [INFO]  server-15c5dab3-22d4-23b7-e103-bf5a63a3ccd1: entering follower state: follower="Node at 15c5dab3-22d4-23b7-e103-bf5a63a3ccd1 [Follower]" leader-address= leader-id=
2025-01-25T14:29:14.838Z [INFO]  server-977f50ab-3940-9f74-c6ca-9e4cffe8927a: entering follower state: follower="Node at 977f50ab-3940-9f74-c6ca-9e4cffe8927a [Follower]" leader-address= leader-id=
2025-01-25T14:29:14.838Z [INFO]  server-65501e75-bc18-c9d2-9e54-6a5136e6decd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15c5dab3-22d4-23b7-e103-bf5a63a3ccd1 Address:15c5dab3-22d4-23b7-e103-bf5a63a3ccd1} {Suffrage:Voter ID:server-977f50ab-3940-9f74-c6ca-9e4cffe8927a Address:977f50ab-3940-9f74-c6ca-9e4cffe8927a} {Suffrage:Voter ID:server-65501e75-bc18-c9d2-9e54-6a5136e6decd Address:65501e75-bc18-c9d2-9e54-6a5136e6decd}]"
2025-01-25T14:29:14.838Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:14.838Z [INFO]  server-65501e75-bc18-c9d2-9e54-6a5136e6decd: entering follower state: follower="Node at 65501e75-bc18-c9d2-9e54-6a5136e6decd [Follower]" leader-address= leader-id=
2025-01-25T14:29:15.341Z [WARN]  server-65501e75-bc18-c9d2-9e54-6a5136e6decd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:15.341Z [INFO]  server-65501e75-bc18-c9d2-9e54-6a5136e6decd: entering candidate state: node="Node at 65501e75-bc18-c9d2-9e54-6a5136e6decd [Candidate]" term=2
2025-01-25T14:29:15.341Z [INFO]  server-65501e75-bc18-c9d2-9e54-6a5136e6decd: election won: term=2 tally=2
2025-01-25T14:29:15.341Z [INFO]  server-65501e75-bc18-c9d2-9e54-6a5136e6decd: entering leader state: leader="Node at 65501e75-bc18-c9d2-9e54-6a5136e6decd [Leader]"
2025-01-25T14:29:15.341Z [INFO]  server-65501e75-bc18-c9d2-9e54-6a5136e6decd: added peer, starting replication: peer=server-15c5dab3-22d4-23b7-e103-bf5a63a3ccd1
2025-01-25T14:29:15.341Z [INFO]  server-65501e75-bc18-c9d2-9e54-6a5136e6decd: added peer, starting replication: peer=server-977f50ab-3940-9f74-c6ca-9e4cffe8927a
2025-01-25T14:29:15.342Z [INFO]  server-65501e75-bc18-c9d2-9e54-6a5136e6decd: pipelining replication: peer="{Voter server-15c5dab3-22d4-23b7-e103-bf5a63a3ccd1 15c5dab3-22d4-23b7-e103-bf5a63a3ccd1}"
2025-01-25T14:29:15.342Z [INFO]  server-65501e75-bc18-c9d2-9e54-6a5136e6decd: pipelining replication: peer="{Voter server-977f50ab-3940-9f74-c6ca-9e4cffe8927a 977f50ab-3940-9f74-c6ca-9e4cffe8927a}"
2025-01-25T14:29:16.347Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:15.341867136 +0000 UTC m=+65.196192078 (1 nodes), 503.021378ms from start of poll, 503.717313ms from cluster start. Timeout at 2025-01-25 14:29:16.346876376 +0000 UTC m=+66.201201318, 1.00500924s after stability
2025-01-25T14:29:16.348Z [INFO]  server-65501e75-bc18-c9d2-9e54-6a5136e6decd: starting snapshot up to: index=12
2025-01-25T14:29:16.348Z [INFO]  creating new snapshot: path=/tmp/raft2135019359/snapshots/2-12-1737815356348.tmp
2025-01-25T14:29:16.349Z [INFO]  server-65501e75-bc18-c9d2-9e54-6a5136e6decd: snapshot complete up to: index=12
2025-01-25T14:29:16.352Z [INFO]  server-8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3 Address:8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3} {Suffrage:Voter ID:server-f956397f-36b4-3bf2-464a-844f7e6dbe06 Address:f956397f-36b4-3bf2-464a-844f7e6dbe06} {Suffrage:Voter ID:server-b78d0a09-bc3b-fa5d-bf7c-e150112bf577 Address:b78d0a09-bc3b-fa5d-bf7c-e150112bf577}]"
2025-01-25T14:29:16.352Z [INFO]  server-8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3: entering follower state: follower="Node at 8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3 [Follower]" leader-address= leader-id=
2025-01-25T14:29:16.352Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3 Address:8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3} {Suffrage:Voter ID:server-f956397f-36b4-3bf2-464a-844f7e6dbe06 Address:f956397f-36b4-3bf2-464a-844f7e6dbe06} {Suffrage:Voter ID:server-b78d0a09-bc3b-fa5d-bf7c-e150112bf577 Address:b78d0a09-bc3b-fa5d-bf7c-e150112bf577}]"
2025-01-25T14:29:16.352Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: entering follower state: follower="Node at f956397f-36b4-3bf2-464a-844f7e6dbe06 [Follower]" leader-address= leader-id=
2025-01-25T14:29:16.352Z [INFO]  server-b78d0a09-bc3b-fa5d-bf7c-e150112bf577: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3 Address:8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3} {Suffrage:Voter ID:server-f956397f-36b4-3bf2-464a-844f7e6dbe06 Address:f956397f-36b4-3bf2-464a-844f7e6dbe06} {Suffrage:Voter ID:server-b78d0a09-bc3b-fa5d-bf7c-e150112bf577 Address:b78d0a09-bc3b-fa5d-bf7c-e150112bf577}]"
2025-01-25T14:29:16.352Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:16.352Z [INFO]  server-b78d0a09-bc3b-fa5d-bf7c-e150112bf577: entering follower state: follower="Node at b78d0a09-bc3b-fa5d-bf7c-e150112bf577 [Follower]" leader-address= leader-id=
2025-01-25T14:29:16.995Z [WARN]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:16.995Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: entering candidate state: node="Node at f956397f-36b4-3bf2-464a-844f7e6dbe06 [Candidate]" term=2
2025-01-25T14:29:16.995Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: election won: term=2 tally=2
2025-01-25T14:29:16.995Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: entering leader state: leader="Node at f956397f-36b4-3bf2-464a-844f7e6dbe06 [Leader]"
2025-01-25T14:29:16.995Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: added peer, starting replication: peer=server-8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3
2025-01-25T14:29:16.995Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: added peer, starting replication: peer=server-b78d0a09-bc3b-fa5d-bf7c-e150112bf577
2025-01-25T14:29:16.995Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: pipelining replication: peer="{Voter server-b78d0a09-bc3b-fa5d-bf7c-e150112bf577 b78d0a09-bc3b-fa5d-bf7c-e150112bf577}"
2025-01-25T14:29:16.995Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: pipelining replication: peer="{Voter server-8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3 8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3}"
2025-01-25T14:29:18.000Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:16.995606785 +0000 UTC m=+66.849931767 (1 nodes), 643.095538ms from start of poll, 643.748314ms from cluster start. Timeout at 2025-01-25 14:29:18.000615224 +0000 UTC m=+67.854940246, 1.005008479s after stability
2025-01-25T14:29:18.001Z [INFO]  creating new snapshot: path=/tmp/raft719616705/snapshots/2-113-1737815358001.tmp
2025-01-25T14:29:18.002Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: copied to local snapshot: bytes=71
2025-01-25T14:29:18.002Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: snapshot restore progress: id=2-113-1737815358001 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:18.003Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: removing all old logs from log store
2025-01-25T14:29:18.003Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: compacting logs: from=1 to=2
2025-01-25T14:29:18.003Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: restored user snapshot: index=113
2025-01-25T14:29:18.003Z [ERROR] server-f956397f-36b4-3bf2-464a-844f7e6dbe06: failed to get log: index=2 error="log not found"
2025-01-25T14:29:18.003Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: aborting pipeline replication: peer="{Voter server-8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3 8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3}"
2025-01-25T14:29:18.003Z [ERROR] server-f956397f-36b4-3bf2-464a-844f7e6dbe06: failed to get log: index=2 error="log not found"
2025-01-25T14:29:18.003Z [ERROR] server-f956397f-36b4-3bf2-464a-844f7e6dbe06: failed to get log: index=2 error="log not found"
2025-01-25T14:29:18.003Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: aborting pipeline replication: peer="{Voter server-b78d0a09-bc3b-fa5d-bf7c-e150112bf577 b78d0a09-bc3b-fa5d-bf7c-e150112bf577}"
2025-01-25T14:29:18.004Z [INFO]  creating new snapshot: path=/tmp/raft4026948096/snapshots/2-113-1737815358004.tmp
2025-01-25T14:29:18.004Z [INFO]  server-8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:18.005Z [INFO]  server-8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3: copied to local snapshot: bytes=71
2025-01-25T14:29:18.005Z [INFO]  server-8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3: snapshot restore progress: id=2-113-1737815358004 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:18.005Z [INFO]  server-8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3: removing all old logs from log store
2025-01-25T14:29:18.005Z [INFO]  server-8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3: compacting logs: from=1 to=2
2025-01-25T14:29:18.005Z [INFO]  server-8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3: Installed remote snapshot
2025-01-25T14:29:18.006Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: pipelining replication: peer="{Voter server-8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3 8b4378bc-30e0-cfe0-f4a3-123ea14f8cf3}"
2025-01-25T14:29:18.006Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:18.011Z [ERROR] server-f956397f-36b4-3bf2-464a-844f7e6dbe06: failed to get log: index=2 error="log not found"
2025-01-25T14:29:18.012Z [INFO]  creating new snapshot: path=/tmp/raft2371475202/snapshots/2-113-1737815358012.tmp
2025-01-25T14:29:18.012Z [INFO]  server-b78d0a09-bc3b-fa5d-bf7c-e150112bf577: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:18.013Z [INFO]  server-b78d0a09-bc3b-fa5d-bf7c-e150112bf577: copied to local snapshot: bytes=71
2025-01-25T14:29:18.014Z [INFO]  server-b78d0a09-bc3b-fa5d-bf7c-e150112bf577: snapshot restore progress: id=2-113-1737815358012 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:18.014Z [INFO]  server-b78d0a09-bc3b-fa5d-bf7c-e150112bf577: removing all old logs from log store
2025-01-25T14:29:18.014Z [INFO]  server-b78d0a09-bc3b-fa5d-bf7c-e150112bf577: compacting logs: from=1 to=2
2025-01-25T14:29:18.014Z [INFO]  server-b78d0a09-bc3b-fa5d-bf7c-e150112bf577: Installed remote snapshot
2025-01-25T14:29:18.014Z [INFO]  server-f956397f-36b4-3bf2-464a-844f7e6dbe06: pipelining replication: peer="{Voter server-b78d0a09-bc3b-fa5d-bf7c-e150112bf577 b78d0a09-bc3b-fa5d-bf7c-e150112bf577}"
2025-01-25T14:29:19.011Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:18.006275775 +0000 UTC m=+67.860600757 (2 nodes), 1.64µs from start of poll, 1.654417304s from cluster start. Timeout at 2025-01-25 14:29:19.01142569 +0000 UTC m=+68.865750672, 1.005149915s after stability
2025-01-25T14:29:19.023Z [INFO]  server-65501e75-bc18-c9d2-9e54-6a5136e6decd: aborting pipeline replication: peer="{Voter server-977f50ab-3940-9f74-c6ca-9e4cffe8927a 977f50ab-3940-9f74-c6ca-9e4cffe8927a}"
2025-01-25T14:29:19.023Z [INFO]  server-65501e75-bc18-c9d2-9e54-6a5136e6decd: aborting pipeline replication: peer="{Voter server-15c5dab3-22d4-23b7-e103-bf5a63a3ccd1 15c5dab3-22d4-23b7-e103-bf5a63a3ccd1}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false
2025-01-25T14:29:19.026Z [INFO]  server-a4fc1263-f4dd-5b81-3e5e-143819cefc59: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4fc1263-f4dd-5b81-3e5e-143819cefc59 Address:a4fc1263-f4dd-5b81-3e5e-143819cefc59} {Suffrage:Voter ID:server-ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275 Address:ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275} {Suffrage:Voter ID:server-3f2a5ce6-8faf-d30a-b490-3e69de51833c Address:3f2a5ce6-8faf-d30a-b490-3e69de51833c}]"
2025-01-25T14:29:19.026Z [INFO]  server-ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4fc1263-f4dd-5b81-3e5e-143819cefc59 Address:a4fc1263-f4dd-5b81-3e5e-143819cefc59} {Suffrage:Voter ID:server-ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275 Address:ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275} {Suffrage:Voter ID:server-3f2a5ce6-8faf-d30a-b490-3e69de51833c Address:3f2a5ce6-8faf-d30a-b490-3e69de51833c}]"
2025-01-25T14:29:19.026Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4fc1263-f4dd-5b81-3e5e-143819cefc59 Address:a4fc1263-f4dd-5b81-3e5e-143819cefc59} {Suffrage:Voter ID:server-ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275 Address:ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275} {Suffrage:Voter ID:server-3f2a5ce6-8faf-d30a-b490-3e69de51833c Address:3f2a5ce6-8faf-d30a-b490-3e69de51833c}]"
2025-01-25T14:29:19.026Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:19.026Z [INFO]  server-a4fc1263-f4dd-5b81-3e5e-143819cefc59: entering follower state: follower="Node at a4fc1263-f4dd-5b81-3e5e-143819cefc59 [Follower]" leader-address= leader-id=
2025-01-25T14:29:19.027Z [INFO]  server-ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275: entering follower state: follower="Node at ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275 [Follower]" leader-address= leader-id=
2025-01-25T14:29:19.027Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: entering follower state: follower="Node at 3f2a5ce6-8faf-d30a-b490-3e69de51833c [Follower]" leader-address= leader-id=
2025-01-25T14:29:19.531Z [WARN]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:19.531Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: entering candidate state: node="Node at 3f2a5ce6-8faf-d30a-b490-3e69de51833c [Candidate]" term=2
2025-01-25T14:29:19.531Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: election won: term=2 tally=2
2025-01-25T14:29:19.531Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: entering leader state: leader="Node at 3f2a5ce6-8faf-d30a-b490-3e69de51833c [Leader]"
2025-01-25T14:29:19.531Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: added peer, starting replication: peer=server-a4fc1263-f4dd-5b81-3e5e-143819cefc59
2025-01-25T14:29:19.531Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: added peer, starting replication: peer=server-ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275
2025-01-25T14:29:19.531Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: pipelining replication: peer="{Voter server-a4fc1263-f4dd-5b81-3e5e-143819cefc59 a4fc1263-f4dd-5b81-3e5e-143819cefc59}"
2025-01-25T14:29:19.531Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: pipelining replication: peer="{Voter server-ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275 ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275}"
2025-01-25T14:29:20.537Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:19.531558205 +0000 UTC m=+69.385883187 (1 nodes), 504.801514ms from start of poll, 505.393892ms from cluster start. Timeout at 2025-01-25 14:29:20.536563365 +0000 UTC m=+70.390888347, 1.00500516s after stability
2025-01-25T14:29:20.539Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: starting snapshot up to: index=12
2025-01-25T14:29:20.539Z [INFO]  creating new snapshot: path=/tmp/raft4170409730/snapshots/2-12-1737815360539.tmp
2025-01-25T14:29:20.540Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: snapshot complete up to: index=12
2025-01-25T14:29:20.541Z [INFO]  creating new snapshot: path=/tmp/raft4170409730/snapshots/2-1013-1737815360541.tmp
2025-01-25T14:29:20.543Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: copied to local snapshot: bytes=71
2025-01-25T14:29:20.543Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: snapshot restore progress: id=2-1013-1737815360541 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:20.543Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: restored user snapshot: index=1013
2025-01-25T14:29:20.543Z [ERROR] server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: failed to get log: index=23 error="log not found"
2025-01-25T14:29:20.543Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: aborting pipeline replication: peer="{Voter server-ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275 ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275}"
2025-01-25T14:29:20.543Z [ERROR] server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: failed to get log: index=23 error="log not found"
2025-01-25T14:29:20.543Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: aborting pipeline replication: peer="{Voter server-a4fc1263-f4dd-5b81-3e5e-143819cefc59 a4fc1263-f4dd-5b81-3e5e-143819cefc59}"
2025-01-25T14:29:20.550Z [ERROR] server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: failed to get log: index=23 error="log not found"
2025-01-25T14:29:20.551Z [INFO]  creating new snapshot: path=/tmp/raft2983815109/snapshots/2-1013-1737815360551.tmp
2025-01-25T14:29:20.552Z [INFO]  server-ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:20.553Z [ERROR] server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: failed to get log: index=23 error="log not found"
2025-01-25T14:29:20.553Z [INFO]  server-ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275: copied to local snapshot: bytes=71
2025-01-25T14:29:20.555Z [INFO]  creating new snapshot: path=/tmp/raft2505429516/snapshots/2-1013-1737815360555.tmp
2025-01-25T14:29:20.556Z [INFO]  server-ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275: snapshot restore progress: id=2-1013-1737815360551 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:20.556Z [INFO]  server-ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275: Installed remote snapshot
2025-01-25T14:29:20.557Z [WARN]  server-ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275: skipping application of old log: index=22
2025-01-25T14:29:20.557Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: pipelining replication: peer="{Voter server-ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275 ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275}"
2025-01-25T14:29:20.557Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:20.557Z [INFO]  server-a4fc1263-f4dd-5b81-3e5e-143819cefc59: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:20.558Z [INFO]  server-a4fc1263-f4dd-5b81-3e5e-143819cefc59: copied to local snapshot: bytes=71
2025-01-25T14:29:20.559Z [INFO]  server-a4fc1263-f4dd-5b81-3e5e-143819cefc59: snapshot restore progress: id=2-1013-1737815360555 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:20.559Z [INFO]  server-a4fc1263-f4dd-5b81-3e5e-143819cefc59: Installed remote snapshot
2025-01-25T14:29:20.559Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: pipelining replication: peer="{Voter server-a4fc1263-f4dd-5b81-3e5e-143819cefc59 a4fc1263-f4dd-5b81-3e5e-143819cefc59}"
2025-01-25T14:29:21.563Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:20.557842233 +0000 UTC m=+70.412167215 (2 nodes), 1.44µs from start of poll, 1.53167792s from cluster start. Timeout at 2025-01-25 14:29:21.562989267 +0000 UTC m=+71.417314249, 1.005147034s after stability
2025-01-25T14:29:21.581Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: aborting pipeline replication: peer="{Voter server-ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275 ef0b42df-2ed6-b5ce-48ea-d4ec9a38b275}"
2025-01-25T14:29:21.581Z [INFO]  server-3f2a5ce6-8faf-d30a-b490-3e69de51833c: aborting pipeline replication: peer="{Voter server-a4fc1263-f4dd-5b81-3e5e-143819cefc59 a4fc1263-f4dd-5b81-3e5e-143819cefc59}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false
2025-01-25T14:29:21.584Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524 Address:87f24fd1-dcea-5868-dbe7-1c7d10a8f524} {Suffrage:Voter ID:server-d6ffaeb2-f826-a3d6-e63e-e5b8b7427335 Address:d6ffaeb2-f826-a3d6-e63e-e5b8b7427335} {Suffrage:Voter ID:server-1c34bbb4-c6eb-17ae-08c8-3329b23ec47a Address:1c34bbb4-c6eb-17ae-08c8-3329b23ec47a}]"
2025-01-25T14:29:21.584Z [INFO]  server-d6ffaeb2-f826-a3d6-e63e-e5b8b7427335: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524 Address:87f24fd1-dcea-5868-dbe7-1c7d10a8f524} {Suffrage:Voter ID:server-d6ffaeb2-f826-a3d6-e63e-e5b8b7427335 Address:d6ffaeb2-f826-a3d6-e63e-e5b8b7427335} {Suffrage:Voter ID:server-1c34bbb4-c6eb-17ae-08c8-3329b23ec47a Address:1c34bbb4-c6eb-17ae-08c8-3329b23ec47a}]"
2025-01-25T14:29:21.585Z [INFO]  server-1c34bbb4-c6eb-17ae-08c8-3329b23ec47a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524 Address:87f24fd1-dcea-5868-dbe7-1c7d10a8f524} {Suffrage:Voter ID:server-d6ffaeb2-f826-a3d6-e63e-e5b8b7427335 Address:d6ffaeb2-f826-a3d6-e63e-e5b8b7427335} {Suffrage:Voter ID:server-1c34bbb4-c6eb-17ae-08c8-3329b23ec47a Address:1c34bbb4-c6eb-17ae-08c8-3329b23ec47a}]"
2025-01-25T14:29:21.585Z [INFO]  server-d6ffaeb2-f826-a3d6-e63e-e5b8b7427335: entering follower state: follower="Node at d6ffaeb2-f826-a3d6-e63e-e5b8b7427335 [Follower]" leader-address= leader-id=
2025-01-25T14:29:21.585Z [INFO]  server-1c34bbb4-c6eb-17ae-08c8-3329b23ec47a: entering follower state: follower="Node at 1c34bbb4-c6eb-17ae-08c8-3329b23ec47a [Follower]" leader-address= leader-id=
2025-01-25T14:29:21.584Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: entering follower state: follower="Node at 87f24fd1-dcea-5868-dbe7-1c7d10a8f524 [Follower]" leader-address= leader-id=
2025-01-25T14:29:21.585Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:22.163Z [WARN]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:22.163Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: entering candidate state: node="Node at 87f24fd1-dcea-5868-dbe7-1c7d10a8f524 [Candidate]" term=2
2025-01-25T14:29:22.163Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: election won: term=2 tally=2
2025-01-25T14:29:22.163Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: entering leader state: leader="Node at 87f24fd1-dcea-5868-dbe7-1c7d10a8f524 [Leader]"
2025-01-25T14:29:22.163Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: added peer, starting replication: peer=server-d6ffaeb2-f826-a3d6-e63e-e5b8b7427335
2025-01-25T14:29:22.163Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: added peer, starting replication: peer=server-1c34bbb4-c6eb-17ae-08c8-3329b23ec47a
2025-01-25T14:29:22.163Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: pipelining replication: peer="{Voter server-1c34bbb4-c6eb-17ae-08c8-3329b23ec47a 1c34bbb4-c6eb-17ae-08c8-3329b23ec47a}"
2025-01-25T14:29:22.163Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: pipelining replication: peer="{Voter server-d6ffaeb2-f826-a3d6-e63e-e5b8b7427335 d6ffaeb2-f826-a3d6-e63e-e5b8b7427335}"
2025-01-25T14:29:23.169Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:22.163517051 +0000 UTC m=+72.017842033 (1 nodes), 578.334748ms from start of poll, 579.053802ms from cluster start. Timeout at 2025-01-25 14:29:23.168524131 +0000 UTC m=+73.022849113, 1.00500708s after stability
2025-01-25T14:29:23.175Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: starting snapshot up to: index=12
2025-01-25T14:29:23.175Z [INFO]  creating new snapshot: path=/tmp/raft3204552013/snapshots/2-12-1737815363175.tmp
2025-01-25T14:29:23.176Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: snapshot complete up to: index=12
2025-01-25T14:29:23.178Z [INFO]  creating new snapshot: path=/tmp/raft3204552013/snapshots/2-1013-1737815363178.tmp
2025-01-25T14:29:23.180Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: copied to local snapshot: bytes=71
2025-01-25T14:29:23.180Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: snapshot restore progress: id=2-1013-1737815363178 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:23.180Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: removing all old logs from log store
2025-01-25T14:29:23.180Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: compacting logs: from=1 to=22
2025-01-25T14:29:23.180Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: restored user snapshot: index=1013
2025-01-25T14:29:23.180Z [ERROR] server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: failed to get log: index=22 error="log not found"
2025-01-25T14:29:23.180Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: aborting pipeline replication: peer="{Voter server-1c34bbb4-c6eb-17ae-08c8-3329b23ec47a 1c34bbb4-c6eb-17ae-08c8-3329b23ec47a}"
2025-01-25T14:29:23.180Z [ERROR] server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: failed to get log: index=22 error="log not found"
2025-01-25T14:29:23.180Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: aborting pipeline replication: peer="{Voter server-d6ffaeb2-f826-a3d6-e63e-e5b8b7427335 d6ffaeb2-f826-a3d6-e63e-e5b8b7427335}"
2025-01-25T14:29:23.186Z [ERROR] server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: failed to get log: index=22 error="log not found"
2025-01-25T14:29:23.186Z [ERROR] server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: failed to get log: index=22 error="log not found"
2025-01-25T14:29:23.186Z [INFO]  creating new snapshot: path=/tmp/raft1608611309/snapshots/2-1013-1737815363186.tmp
2025-01-25T14:29:23.187Z [INFO]  creating new snapshot: path=/tmp/raft3898775036/snapshots/2-1013-1737815363187.tmp
2025-01-25T14:29:23.187Z [INFO]  server-1c34bbb4-c6eb-17ae-08c8-3329b23ec47a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:23.188Z [INFO]  server-d6ffaeb2-f826-a3d6-e63e-e5b8b7427335: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:23.188Z [INFO]  server-1c34bbb4-c6eb-17ae-08c8-3329b23ec47a: copied to local snapshot: bytes=71
2025-01-25T14:29:23.188Z [INFO]  server-d6ffaeb2-f826-a3d6-e63e-e5b8b7427335: copied to local snapshot: bytes=71
2025-01-25T14:29:23.189Z [INFO]  server-1c34bbb4-c6eb-17ae-08c8-3329b23ec47a: snapshot restore progress: id=2-1013-1737815363186 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:23.189Z [INFO]  server-1c34bbb4-c6eb-17ae-08c8-3329b23ec47a: removing all old logs from log store
2025-01-25T14:29:23.189Z [INFO]  server-d6ffaeb2-f826-a3d6-e63e-e5b8b7427335: snapshot restore progress: id=2-1013-1737815363187 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:23.189Z [INFO]  server-1c34bbb4-c6eb-17ae-08c8-3329b23ec47a: compacting logs: from=1 to=22
2025-01-25T14:29:23.189Z [INFO]  server-d6ffaeb2-f826-a3d6-e63e-e5b8b7427335: removing all old logs from log store
2025-01-25T14:29:23.189Z [INFO]  server-d6ffaeb2-f826-a3d6-e63e-e5b8b7427335: compacting logs: from=1 to=22
2025-01-25T14:29:23.189Z [INFO]  server-1c34bbb4-c6eb-17ae-08c8-3329b23ec47a: Installed remote snapshot
2025-01-25T14:29:23.189Z [INFO]  server-d6ffaeb2-f826-a3d6-e63e-e5b8b7427335: Installed remote snapshot
2025-01-25T14:29:23.189Z [WARN]  server-d6ffaeb2-f826-a3d6-e63e-e5b8b7427335: skipping application of old log: index=1013
2025-01-25T14:29:23.189Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: pipelining replication: peer="{Voter server-d6ffaeb2-f826-a3d6-e63e-e5b8b7427335 d6ffaeb2-f826-a3d6-e63e-e5b8b7427335}"
2025-01-25T14:29:23.189Z [WARN]  server-1c34bbb4-c6eb-17ae-08c8-3329b23ec47a: skipping application of old log: index=22
2025-01-25T14:29:23.189Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:23.189Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: pipelining replication: peer="{Voter server-1c34bbb4-c6eb-17ae-08c8-3329b23ec47a 1c34bbb4-c6eb-17ae-08c8-3329b23ec47a}"
2025-01-25T14:29:24.194Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:23.189732721 +0000 UTC m=+73.044057703 (2 nodes), 1.48µs from start of poll, 1.605269472s from cluster start. Timeout at 2025-01-25 14:29:24.194866956 +0000 UTC m=+74.049191938, 1.005134235s after stability
2025-01-25T14:29:24.208Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: aborting pipeline replication: peer="{Voter server-1c34bbb4-c6eb-17ae-08c8-3329b23ec47a 1c34bbb4-c6eb-17ae-08c8-3329b23ec47a}"
2025-01-25T14:29:24.208Z [INFO]  server-87f24fd1-dcea-5868-dbe7-1c7d10a8f524: aborting pipeline replication: peer="{Voter server-d6ffaeb2-f826-a3d6-e63e-e5b8b7427335 d6ffaeb2-f826-a3d6-e63e-e5b8b7427335}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true
2025-01-25T14:29:24.212Z [INFO]  server-c6b73303-998f-ce4b-f5dd-bffc533ff251: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c6b73303-998f-ce4b-f5dd-bffc533ff251 Address:c6b73303-998f-ce4b-f5dd-bffc533ff251} {Suffrage:Voter ID:server-d2cba660-3a81-b8dc-0633-a287f4965a9f Address:d2cba660-3a81-b8dc-0633-a287f4965a9f} {Suffrage:Voter ID:server-1ac1e9af-e75c-22d1-90a3-59a684cfc086 Address:1ac1e9af-e75c-22d1-90a3-59a684cfc086}]"
2025-01-25T14:29:24.212Z [INFO]  server-c6b73303-998f-ce4b-f5dd-bffc533ff251: entering follower state: follower="Node at c6b73303-998f-ce4b-f5dd-bffc533ff251 [Follower]" leader-address= leader-id=
2025-01-25T14:29:24.212Z [INFO]  server-d2cba660-3a81-b8dc-0633-a287f4965a9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c6b73303-998f-ce4b-f5dd-bffc533ff251 Address:c6b73303-998f-ce4b-f5dd-bffc533ff251} {Suffrage:Voter ID:server-d2cba660-3a81-b8dc-0633-a287f4965a9f Address:d2cba660-3a81-b8dc-0633-a287f4965a9f} {Suffrage:Voter ID:server-1ac1e9af-e75c-22d1-90a3-59a684cfc086 Address:1ac1e9af-e75c-22d1-90a3-59a684cfc086}]"
2025-01-25T14:29:24.212Z [INFO]  server-d2cba660-3a81-b8dc-0633-a287f4965a9f: entering follower state: follower="Node at d2cba660-3a81-b8dc-0633-a287f4965a9f [Follower]" leader-address= leader-id=
2025-01-25T14:29:24.212Z [INFO]  server-1ac1e9af-e75c-22d1-90a3-59a684cfc086: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c6b73303-998f-ce4b-f5dd-bffc533ff251 Address:c6b73303-998f-ce4b-f5dd-bffc533ff251} {Suffrage:Voter ID:server-d2cba660-3a81-b8dc-0633-a287f4965a9f Address:d2cba660-3a81-b8dc-0633-a287f4965a9f} {Suffrage:Voter ID:server-1ac1e9af-e75c-22d1-90a3-59a684cfc086 Address:1ac1e9af-e75c-22d1-90a3-59a684cfc086}]"
2025-01-25T14:29:24.212Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:24.212Z [INFO]  server-1ac1e9af-e75c-22d1-90a3-59a684cfc086: entering follower state: follower="Node at 1ac1e9af-e75c-22d1-90a3-59a684cfc086 [Follower]" leader-address= leader-id=
2025-01-25T14:29:24.780Z [WARN]  server-c6b73303-998f-ce4b-f5dd-bffc533ff251: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:24.780Z [INFO]  server-c6b73303-998f-ce4b-f5dd-bffc533ff251: entering candidate state: node="Node at c6b73303-998f-ce4b-f5dd-bffc533ff251 [Candidate]" term=2
2025-01-25T14:29:24.780Z [INFO]  server-c6b73303-998f-ce4b-f5dd-bffc533ff251: election won: term=2 tally=2
2025-01-25T14:29:24.780Z [INFO]  server-c6b73303-998f-ce4b-f5dd-bffc533ff251: entering leader state: leader="Node at c6b73303-998f-ce4b-f5dd-bffc533ff251 [Leader]"
2025-01-25T14:29:24.780Z [INFO]  server-c6b73303-998f-ce4b-f5dd-bffc533ff251: added peer, starting replication: peer=server-d2cba660-3a81-b8dc-0633-a287f4965a9f
2025-01-25T14:29:24.780Z [INFO]  server-c6b73303-998f-ce4b-f5dd-bffc533ff251: added peer, starting replication: peer=server-1ac1e9af-e75c-22d1-90a3-59a684cfc086
2025-01-25T14:29:24.781Z [INFO]  server-c6b73303-998f-ce4b-f5dd-bffc533ff251: pipelining replication: peer="{Voter server-d2cba660-3a81-b8dc-0633-a287f4965a9f d2cba660-3a81-b8dc-0633-a287f4965a9f}"
2025-01-25T14:29:24.781Z [INFO]  server-c6b73303-998f-ce4b-f5dd-bffc533ff251: pipelining replication: peer="{Voter server-1ac1e9af-e75c-22d1-90a3-59a684cfc086 1ac1e9af-e75c-22d1-90a3-59a684cfc086}"
2025-01-25T14:29:25.785Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:24.780739347 +0000 UTC m=+74.635064329 (1 nodes), 567.756209ms from start of poll, 568.606418ms from cluster start. Timeout at 2025-01-25 14:29:25.785741507 +0000 UTC m=+75.640066449, 1.00500212s after stability
2025-01-25T14:29:25.786Z [INFO]  server-c6b73303-998f-ce4b-f5dd-bffc533ff251: starting snapshot up to: index=12
2025-01-25T14:29:25.786Z [INFO]  creating new snapshot: path=/tmp/raft1575114220/snapshots/2-12-1737815365786.tmp
2025-01-25T14:29:25.788Z [INFO]  server-c6b73303-998f-ce4b-f5dd-bffc533ff251: snapshot complete up to: index=12
2025-01-25T14:29:25.790Z [INFO]  server-ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7 Address:ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7} {Suffrage:Voter ID:server-df5fff9a-770c-262a-937e-8fe7e82babf9 Address:df5fff9a-770c-262a-937e-8fe7e82babf9} {Suffrage:Voter ID:server-d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f Address:d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f}]"
2025-01-25T14:29:25.790Z [INFO]  server-ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7: entering follower state: follower="Node at ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7 [Follower]" leader-address= leader-id=
2025-01-25T14:29:25.791Z [INFO]  server-df5fff9a-770c-262a-937e-8fe7e82babf9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7 Address:ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7} {Suffrage:Voter ID:server-df5fff9a-770c-262a-937e-8fe7e82babf9 Address:df5fff9a-770c-262a-937e-8fe7e82babf9} {Suffrage:Voter ID:server-d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f Address:d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f}]"
2025-01-25T14:29:25.791Z [INFO]  server-df5fff9a-770c-262a-937e-8fe7e82babf9: entering follower state: follower="Node at df5fff9a-770c-262a-937e-8fe7e82babf9 [Follower]" leader-address= leader-id=
2025-01-25T14:29:25.791Z [INFO]  server-d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7 Address:ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7} {Suffrage:Voter ID:server-df5fff9a-770c-262a-937e-8fe7e82babf9 Address:df5fff9a-770c-262a-937e-8fe7e82babf9} {Suffrage:Voter ID:server-d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f Address:d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f}]"
2025-01-25T14:29:25.791Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:25.791Z [INFO]  server-d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f: entering follower state: follower="Node at d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f [Follower]" leader-address= leader-id=
2025-01-25T14:29:26.301Z [WARN]  server-df5fff9a-770c-262a-937e-8fe7e82babf9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:26.301Z [INFO]  server-df5fff9a-770c-262a-937e-8fe7e82babf9: entering candidate state: node="Node at df5fff9a-770c-262a-937e-8fe7e82babf9 [Candidate]" term=2
2025-01-25T14:29:26.307Z [INFO]  server-df5fff9a-770c-262a-937e-8fe7e82babf9: election won: term=2 tally=2
2025-01-25T14:29:26.308Z [INFO]  server-df5fff9a-770c-262a-937e-8fe7e82babf9: entering leader state: leader="Node at df5fff9a-770c-262a-937e-8fe7e82babf9 [Leader]"
2025-01-25T14:29:26.308Z [INFO]  server-df5fff9a-770c-262a-937e-8fe7e82babf9: added peer, starting replication: peer=server-ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7
2025-01-25T14:29:26.308Z [INFO]  server-df5fff9a-770c-262a-937e-8fe7e82babf9: added peer, starting replication: peer=server-d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f
2025-01-25T14:29:26.308Z [INFO]  server-df5fff9a-770c-262a-937e-8fe7e82babf9: pipelining replication: peer="{Voter server-d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f}"
2025-01-25T14:29:26.308Z [WARN]  server-ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7: rejecting vote request since our last term is greater: candidate=df5fff9a-770c-262a-937e-8fe7e82babf9 last-term=2 last-candidate-term=1
2025-01-25T14:29:26.309Z [INFO]  server-df5fff9a-770c-262a-937e-8fe7e82babf9: pipelining replication: peer="{Voter server-ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7 ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7}"
2025-01-25T14:29:27.314Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:26.308962951 +0000 UTC m=+76.163287933 (1 nodes), 517.497697ms from start of poll, 518.427383ms from cluster start. Timeout at 2025-01-25 14:29:27.313964871 +0000 UTC m=+77.168289853, 1.00500192s after stability
2025-01-25T14:29:27.316Z [INFO]  creating new snapshot: path=/tmp/raft3102055552/snapshots/2-1013-1737815367316.tmp
2025-01-25T14:29:27.317Z [INFO]  server-df5fff9a-770c-262a-937e-8fe7e82babf9: copied to local snapshot: bytes=71
2025-01-25T14:29:27.317Z [INFO]  server-df5fff9a-770c-262a-937e-8fe7e82babf9: snapshot restore progress: id=2-1013-1737815367316 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:27.318Z [INFO]  server-df5fff9a-770c-262a-937e-8fe7e82babf9: restored user snapshot: index=1013
2025-01-25T14:29:27.318Z [ERROR] server-df5fff9a-770c-262a-937e-8fe7e82babf9: failed to get log: index=3 error="log not found"
2025-01-25T14:29:27.318Z [INFO]  server-df5fff9a-770c-262a-937e-8fe7e82babf9: aborting pipeline replication: peer="{Voter server-d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f}"
2025-01-25T14:29:27.318Z [ERROR] server-df5fff9a-770c-262a-937e-8fe7e82babf9: failed to get log: index=3 error="log not found"
2025-01-25T14:29:27.318Z [INFO]  server-df5fff9a-770c-262a-937e-8fe7e82babf9: aborting pipeline replication: peer="{Voter server-ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7 ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7}"
2025-01-25T14:29:27.325Z [ERROR] server-df5fff9a-770c-262a-937e-8fe7e82babf9: failed to get log: index=3 error="log not found"
2025-01-25T14:29:27.326Z [INFO]  creating new snapshot: path=/tmp/raft563240831/snapshots/2-1013-1737815367326.tmp
2025-01-25T14:29:27.326Z [INFO]  server-d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:27.326Z [ERROR] server-df5fff9a-770c-262a-937e-8fe7e82babf9: failed to get log: index=3 error="log not found"
2025-01-25T14:29:27.327Z [INFO]  server-d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f: copied to local snapshot: bytes=71
2025-01-25T14:29:27.327Z [INFO]  creating new snapshot: path=/tmp/raft819682955/snapshots/2-1013-1737815367327.tmp
2025-01-25T14:29:27.327Z [INFO]  server-d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f: snapshot restore progress: id=2-1013-1737815367326 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:27.328Z [INFO]  server-d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f: Installed remote snapshot
2025-01-25T14:29:27.328Z [INFO]  server-df5fff9a-770c-262a-937e-8fe7e82babf9: pipelining replication: peer="{Voter server-d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f d7eceecb-fa8d-b19f-218a-ebb7a4dc4e9f}"
2025-01-25T14:29:27.328Z [INFO]  server-ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:27.328Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:27.328Z [INFO]  server-ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7: copied to local snapshot: bytes=71
2025-01-25T14:29:27.329Z [INFO]  server-ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7: snapshot restore progress: id=2-1013-1737815367327 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:27.329Z [INFO]  server-ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7: Installed remote snapshot
2025-01-25T14:29:27.329Z [INFO]  server-df5fff9a-770c-262a-937e-8fe7e82babf9: pipelining replication: peer="{Voter server-ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7 ec2ddbed-adf2-a8fc-f94b-a00c4ae311f7}"
2025-01-25T14:29:28.333Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:27.328450463 +0000 UTC m=+77.182775445 (2 nodes), 1.72µs from start of poll, 1.537914895s from cluster start. Timeout at 2025-01-25 14:29:28.333590658 +0000 UTC m=+78.187915680, 1.005140235s after stability
2025-01-25T14:29:28.346Z [INFO]  server-c6b73303-998f-ce4b-f5dd-bffc533ff251: aborting pipeline replication: peer="{Voter server-d2cba660-3a81-b8dc-0633-a287f4965a9f d2cba660-3a81-b8dc-0633-a287f4965a9f}"
2025-01-25T14:29:28.346Z [INFO]  server-c6b73303-998f-ce4b-f5dd-bffc533ff251: aborting pipeline replication: peer="{Voter server-1ac1e9af-e75c-22d1-90a3-59a684cfc086 1ac1e9af-e75c-22d1-90a3-59a684cfc086}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true
2025-01-25T14:29:28.348Z [INFO]  server-deac9130-839c-f54e-185d-246a460b2f82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-deac9130-839c-f54e-185d-246a460b2f82 Address:deac9130-839c-f54e-185d-246a460b2f82} {Suffrage:Voter ID:server-edd04b33-d20b-5083-c904-d62c59699a66 Address:edd04b33-d20b-5083-c904-d62c59699a66} {Suffrage:Voter ID:server-27d2c5f2-30d3-851d-cb8e-504c5b5f4446 Address:27d2c5f2-30d3-851d-cb8e-504c5b5f4446}]"
2025-01-25T14:29:28.349Z [INFO]  server-edd04b33-d20b-5083-c904-d62c59699a66: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-deac9130-839c-f54e-185d-246a460b2f82 Address:deac9130-839c-f54e-185d-246a460b2f82} {Suffrage:Voter ID:server-edd04b33-d20b-5083-c904-d62c59699a66 Address:edd04b33-d20b-5083-c904-d62c59699a66} {Suffrage:Voter ID:server-27d2c5f2-30d3-851d-cb8e-504c5b5f4446 Address:27d2c5f2-30d3-851d-cb8e-504c5b5f4446}]"
2025-01-25T14:29:28.349Z [INFO]  server-deac9130-839c-f54e-185d-246a460b2f82: entering follower state: follower="Node at deac9130-839c-f54e-185d-246a460b2f82 [Follower]" leader-address= leader-id=
2025-01-25T14:29:28.349Z [INFO]  server-edd04b33-d20b-5083-c904-d62c59699a66: entering follower state: follower="Node at edd04b33-d20b-5083-c904-d62c59699a66 [Follower]" leader-address= leader-id=
2025-01-25T14:29:28.349Z [INFO]  server-27d2c5f2-30d3-851d-cb8e-504c5b5f4446: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-deac9130-839c-f54e-185d-246a460b2f82 Address:deac9130-839c-f54e-185d-246a460b2f82} {Suffrage:Voter ID:server-edd04b33-d20b-5083-c904-d62c59699a66 Address:edd04b33-d20b-5083-c904-d62c59699a66} {Suffrage:Voter ID:server-27d2c5f2-30d3-851d-cb8e-504c5b5f4446 Address:27d2c5f2-30d3-851d-cb8e-504c5b5f4446}]"
2025-01-25T14:29:28.349Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:28.349Z [INFO]  server-27d2c5f2-30d3-851d-cb8e-504c5b5f4446: entering follower state: follower="Node at 27d2c5f2-30d3-851d-cb8e-504c5b5f4446 [Follower]" leader-address= leader-id=
2025-01-25T14:29:28.860Z [WARN]  server-27d2c5f2-30d3-851d-cb8e-504c5b5f4446: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:28.860Z [INFO]  server-27d2c5f2-30d3-851d-cb8e-504c5b5f4446: entering candidate state: node="Node at 27d2c5f2-30d3-851d-cb8e-504c5b5f4446 [Candidate]" term=2
2025-01-25T14:29:28.860Z [INFO]  server-27d2c5f2-30d3-851d-cb8e-504c5b5f4446: election won: term=2 tally=2
2025-01-25T14:29:28.860Z [INFO]  server-27d2c5f2-30d3-851d-cb8e-504c5b5f4446: entering leader state: leader="Node at 27d2c5f2-30d3-851d-cb8e-504c5b5f4446 [Leader]"
2025-01-25T14:29:28.860Z [INFO]  server-27d2c5f2-30d3-851d-cb8e-504c5b5f4446: added peer, starting replication: peer=server-deac9130-839c-f54e-185d-246a460b2f82
2025-01-25T14:29:28.861Z [INFO]  server-27d2c5f2-30d3-851d-cb8e-504c5b5f4446: added peer, starting replication: peer=server-edd04b33-d20b-5083-c904-d62c59699a66
2025-01-25T14:29:28.861Z [INFO]  server-27d2c5f2-30d3-851d-cb8e-504c5b5f4446: pipelining replication: peer="{Voter server-deac9130-839c-f54e-185d-246a460b2f82 deac9130-839c-f54e-185d-246a460b2f82}"
2025-01-25T14:29:28.861Z [INFO]  server-27d2c5f2-30d3-851d-cb8e-504c5b5f4446: pipelining replication: peer="{Voter server-edd04b33-d20b-5083-c904-d62c59699a66 edd04b33-d20b-5083-c904-d62c59699a66}"
2025-01-25T14:29:29.866Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:28.861012632 +0000 UTC m=+78.715337614 (1 nodes), 511.563271ms from start of poll, 512.236327ms from cluster start. Timeout at 2025-01-25 14:29:29.866015672 +0000 UTC m=+79.720340614, 1.005003s after stability
2025-01-25T14:29:29.872Z [INFO]  server-27d2c5f2-30d3-851d-cb8e-504c5b5f4446: starting snapshot up to: index=12
2025-01-25T14:29:29.872Z [INFO]  creating new snapshot: path=/tmp/raft2053591696/snapshots/2-12-1737815369872.tmp
2025-01-25T14:29:29.874Z [INFO]  server-27d2c5f2-30d3-851d-cb8e-504c5b5f4446: snapshot complete up to: index=12
2025-01-25T14:29:29.876Z [INFO]  server-75afaf31-1898-20fb-9d23-a94b8b344e23: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75afaf31-1898-20fb-9d23-a94b8b344e23 Address:75afaf31-1898-20fb-9d23-a94b8b344e23} {Suffrage:Voter ID:server-a627ff05-dcb8-1937-5bd4-fdc8d3c04893 Address:a627ff05-dcb8-1937-5bd4-fdc8d3c04893} {Suffrage:Voter ID:server-16902e3d-3e90-0290-977b-4c48d469fc00 Address:16902e3d-3e90-0290-977b-4c48d469fc00}]"
2025-01-25T14:29:29.877Z [INFO]  server-75afaf31-1898-20fb-9d23-a94b8b344e23: entering follower state: follower="Node at 75afaf31-1898-20fb-9d23-a94b8b344e23 [Follower]" leader-address= leader-id=
2025-01-25T14:29:29.877Z [INFO]  server-a627ff05-dcb8-1937-5bd4-fdc8d3c04893: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75afaf31-1898-20fb-9d23-a94b8b344e23 Address:75afaf31-1898-20fb-9d23-a94b8b344e23} {Suffrage:Voter ID:server-a627ff05-dcb8-1937-5bd4-fdc8d3c04893 Address:a627ff05-dcb8-1937-5bd4-fdc8d3c04893} {Suffrage:Voter ID:server-16902e3d-3e90-0290-977b-4c48d469fc00 Address:16902e3d-3e90-0290-977b-4c48d469fc00}]"
2025-01-25T14:29:29.877Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75afaf31-1898-20fb-9d23-a94b8b344e23 Address:75afaf31-1898-20fb-9d23-a94b8b344e23} {Suffrage:Voter ID:server-a627ff05-dcb8-1937-5bd4-fdc8d3c04893 Address:a627ff05-dcb8-1937-5bd4-fdc8d3c04893} {Suffrage:Voter ID:server-16902e3d-3e90-0290-977b-4c48d469fc00 Address:16902e3d-3e90-0290-977b-4c48d469fc00}]"
2025-01-25T14:29:29.877Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:29.877Z [INFO]  server-a627ff05-dcb8-1937-5bd4-fdc8d3c04893: entering follower state: follower="Node at a627ff05-dcb8-1937-5bd4-fdc8d3c04893 [Follower]" leader-address= leader-id=
2025-01-25T14:29:29.877Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: entering follower state: follower="Node at 16902e3d-3e90-0290-977b-4c48d469fc00 [Follower]" leader-address= leader-id=
2025-01-25T14:29:30.433Z [WARN]  server-16902e3d-3e90-0290-977b-4c48d469fc00: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:30.433Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: entering candidate state: node="Node at 16902e3d-3e90-0290-977b-4c48d469fc00 [Candidate]" term=2
2025-01-25T14:29:30.434Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: election won: term=2 tally=2
2025-01-25T14:29:30.434Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: entering leader state: leader="Node at 16902e3d-3e90-0290-977b-4c48d469fc00 [Leader]"
2025-01-25T14:29:30.434Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: added peer, starting replication: peer=server-75afaf31-1898-20fb-9d23-a94b8b344e23
2025-01-25T14:29:30.434Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: added peer, starting replication: peer=server-a627ff05-dcb8-1937-5bd4-fdc8d3c04893
2025-01-25T14:29:30.434Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: pipelining replication: peer="{Voter server-a627ff05-dcb8-1937-5bd4-fdc8d3c04893 a627ff05-dcb8-1937-5bd4-fdc8d3c04893}"
2025-01-25T14:29:30.434Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: pipelining replication: peer="{Voter server-75afaf31-1898-20fb-9d23-a94b8b344e23 75afaf31-1898-20fb-9d23-a94b8b344e23}"
2025-01-25T14:29:31.439Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:30.434184579 +0000 UTC m=+80.288509561 (1 nodes), 556.775244ms from start of poll, 557.458579ms from cluster start. Timeout at 2025-01-25 14:29:31.439187539 +0000 UTC m=+81.293512521, 1.00500296s after stability
2025-01-25T14:29:31.440Z [INFO]  creating new snapshot: path=/tmp/raft3524174746/snapshots/2-1013-1737815371440.tmp
2025-01-25T14:29:31.441Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: copied to local snapshot: bytes=71
2025-01-25T14:29:31.441Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: snapshot restore progress: id=2-1013-1737815371440 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:31.441Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: removing all old logs from log store
2025-01-25T14:29:31.441Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: compacting logs: from=1 to=2
2025-01-25T14:29:31.441Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: restored user snapshot: index=1013
2025-01-25T14:29:31.441Z [ERROR] server-16902e3d-3e90-0290-977b-4c48d469fc00: failed to get log: index=2 error="log not found"
2025-01-25T14:29:31.441Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: aborting pipeline replication: peer="{Voter server-a627ff05-dcb8-1937-5bd4-fdc8d3c04893 a627ff05-dcb8-1937-5bd4-fdc8d3c04893}"
2025-01-25T14:29:31.441Z [ERROR] server-16902e3d-3e90-0290-977b-4c48d469fc00: failed to get log: index=2 error="log not found"
2025-01-25T14:29:31.441Z [ERROR] server-16902e3d-3e90-0290-977b-4c48d469fc00: failed to get log: index=2 error="log not found"
2025-01-25T14:29:31.441Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: aborting pipeline replication: peer="{Voter server-75afaf31-1898-20fb-9d23-a94b8b344e23 75afaf31-1898-20fb-9d23-a94b8b344e23}"
2025-01-25T14:29:31.442Z [INFO]  creating new snapshot: path=/tmp/raft1353825785/snapshots/2-1013-1737815371442.tmp
2025-01-25T14:29:31.443Z [INFO]  server-a627ff05-dcb8-1937-5bd4-fdc8d3c04893: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:31.443Z [INFO]  server-a627ff05-dcb8-1937-5bd4-fdc8d3c04893: copied to local snapshot: bytes=71
2025-01-25T14:29:31.444Z [INFO]  server-a627ff05-dcb8-1937-5bd4-fdc8d3c04893: snapshot restore progress: id=2-1013-1737815371442 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:31.444Z [INFO]  server-a627ff05-dcb8-1937-5bd4-fdc8d3c04893: removing all old logs from log store
2025-01-25T14:29:31.444Z [INFO]  server-a627ff05-dcb8-1937-5bd4-fdc8d3c04893: compacting logs: from=1 to=2
2025-01-25T14:29:31.444Z [INFO]  server-a627ff05-dcb8-1937-5bd4-fdc8d3c04893: Installed remote snapshot
2025-01-25T14:29:31.444Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: pipelining replication: peer="{Voter server-a627ff05-dcb8-1937-5bd4-fdc8d3c04893 a627ff05-dcb8-1937-5bd4-fdc8d3c04893}"
2025-01-25T14:29:31.444Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:31.452Z [ERROR] server-16902e3d-3e90-0290-977b-4c48d469fc00: failed to get log: index=2 error="log not found"
2025-01-25T14:29:31.455Z [INFO]  creating new snapshot: path=/tmp/raft3794580954/snapshots/2-1013-1737815371455.tmp
2025-01-25T14:29:31.456Z [INFO]  server-75afaf31-1898-20fb-9d23-a94b8b344e23: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:31.456Z [INFO]  server-75afaf31-1898-20fb-9d23-a94b8b344e23: copied to local snapshot: bytes=71
2025-01-25T14:29:31.457Z [INFO]  server-75afaf31-1898-20fb-9d23-a94b8b344e23: snapshot restore progress: id=2-1013-1737815371455 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:31.457Z [INFO]  server-75afaf31-1898-20fb-9d23-a94b8b344e23: removing all old logs from log store
2025-01-25T14:29:31.457Z [INFO]  server-75afaf31-1898-20fb-9d23-a94b8b344e23: compacting logs: from=1 to=2
2025-01-25T14:29:31.457Z [INFO]  server-75afaf31-1898-20fb-9d23-a94b8b344e23: Installed remote snapshot
2025-01-25T14:29:31.457Z [INFO]  server-16902e3d-3e90-0290-977b-4c48d469fc00: pipelining replication: peer="{Voter server-75afaf31-1898-20fb-9d23-a94b8b344e23 75afaf31-1898-20fb-9d23-a94b8b344e23}"
2025-01-25T14:29:32.451Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:31.444969204 +0000 UTC m=+81.299294186 (2 nodes), 1.48µs from start of poll, 1.568243204s from cluster start. Timeout at 2025-01-25 14:29:32.450147718 +0000 UTC m=+82.304472660, 1.005178474s after stability
2025-01-25T14:29:32.466Z [INFO]  server-27d2c5f2-30d3-851d-cb8e-504c5b5f4446: aborting pipeline replication: peer="{Voter server-edd04b33-d20b-5083-c904-d62c59699a66 edd04b33-d20b-5083-c904-d62c59699a66}"
2025-01-25T14:29:32.466Z [INFO]  server-27d2c5f2-30d3-851d-cb8e-504c5b5f4446: aborting pipeline replication: peer="{Voter server-deac9130-839c-f54e-185d-246a460b2f82 deac9130-839c-f54e-185d-246a460b2f82}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false
2025-01-25T14:29:32.469Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-23e3abb6-f312-acc9-15cd-330a48fc9f67 Address:23e3abb6-f312-acc9-15cd-330a48fc9f67} {Suffrage:Voter ID:server-95f24c46-890b-9e09-638e-f956eeb2397b Address:95f24c46-890b-9e09-638e-f956eeb2397b} {Suffrage:Voter ID:server-c4659b27-d1b5-5fde-eee3-1131794b82d3 Address:c4659b27-d1b5-5fde-eee3-1131794b82d3}]"
2025-01-25T14:29:32.469Z [INFO]  server-95f24c46-890b-9e09-638e-f956eeb2397b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-23e3abb6-f312-acc9-15cd-330a48fc9f67 Address:23e3abb6-f312-acc9-15cd-330a48fc9f67} {Suffrage:Voter ID:server-95f24c46-890b-9e09-638e-f956eeb2397b Address:95f24c46-890b-9e09-638e-f956eeb2397b} {Suffrage:Voter ID:server-c4659b27-d1b5-5fde-eee3-1131794b82d3 Address:c4659b27-d1b5-5fde-eee3-1131794b82d3}]"
2025-01-25T14:29:32.470Z [INFO]  server-c4659b27-d1b5-5fde-eee3-1131794b82d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-23e3abb6-f312-acc9-15cd-330a48fc9f67 Address:23e3abb6-f312-acc9-15cd-330a48fc9f67} {Suffrage:Voter ID:server-95f24c46-890b-9e09-638e-f956eeb2397b Address:95f24c46-890b-9e09-638e-f956eeb2397b} {Suffrage:Voter ID:server-c4659b27-d1b5-5fde-eee3-1131794b82d3 Address:c4659b27-d1b5-5fde-eee3-1131794b82d3}]"
2025-01-25T14:29:32.470Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:32.470Z [INFO]  server-95f24c46-890b-9e09-638e-f956eeb2397b: entering follower state: follower="Node at 95f24c46-890b-9e09-638e-f956eeb2397b [Follower]" leader-address= leader-id=
2025-01-25T14:29:32.470Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: entering follower state: follower="Node at 23e3abb6-f312-acc9-15cd-330a48fc9f67 [Follower]" leader-address= leader-id=
2025-01-25T14:29:32.470Z [INFO]  server-c4659b27-d1b5-5fde-eee3-1131794b82d3: entering follower state: follower="Node at c4659b27-d1b5-5fde-eee3-1131794b82d3 [Follower]" leader-address= leader-id=
2025-01-25T14:29:33.201Z [WARN]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:33.201Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: entering candidate state: node="Node at 23e3abb6-f312-acc9-15cd-330a48fc9f67 [Candidate]" term=2
2025-01-25T14:29:33.201Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: election won: term=2 tally=2
2025-01-25T14:29:33.201Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: entering leader state: leader="Node at 23e3abb6-f312-acc9-15cd-330a48fc9f67 [Leader]"
2025-01-25T14:29:33.201Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: added peer, starting replication: peer=server-95f24c46-890b-9e09-638e-f956eeb2397b
2025-01-25T14:29:33.201Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: added peer, starting replication: peer=server-c4659b27-d1b5-5fde-eee3-1131794b82d3
2025-01-25T14:29:33.201Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: pipelining replication: peer="{Voter server-c4659b27-d1b5-5fde-eee3-1131794b82d3 c4659b27-d1b5-5fde-eee3-1131794b82d3}"
2025-01-25T14:29:33.201Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: pipelining replication: peer="{Voter server-95f24c46-890b-9e09-638e-f956eeb2397b 95f24c46-890b-9e09-638e-f956eeb2397b}"
2025-01-25T14:29:34.208Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:33.201780462 +0000 UTC m=+83.056105444 (1 nodes), 731.64711ms from start of poll, 732.361605ms from cluster start. Timeout at 2025-01-25 14:29:34.206785422 +0000 UTC m=+84.061110444, 1.005005s after stability
2025-01-25T14:29:34.208Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: starting snapshot up to: index=12
2025-01-25T14:29:34.209Z [INFO]  creating new snapshot: path=/tmp/raft322268888/snapshots/2-12-1737815374209.tmp
2025-01-25T14:29:34.210Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: snapshot complete up to: index=12
2025-01-25T14:29:34.211Z [INFO]  creating new snapshot: path=/tmp/raft322268888/snapshots/2-10013-1737815374211.tmp
2025-01-25T14:29:34.212Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: copied to local snapshot: bytes=71
2025-01-25T14:29:34.213Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: snapshot restore progress: id=2-10013-1737815374211 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:34.213Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: restored user snapshot: index=10013
2025-01-25T14:29:34.213Z [ERROR] server-23e3abb6-f312-acc9-15cd-330a48fc9f67: failed to get log: index=23 error="log not found"
2025-01-25T14:29:34.213Z [ERROR] server-23e3abb6-f312-acc9-15cd-330a48fc9f67: failed to get log: index=23 error="log not found"
2025-01-25T14:29:34.213Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: aborting pipeline replication: peer="{Voter server-c4659b27-d1b5-5fde-eee3-1131794b82d3 c4659b27-d1b5-5fde-eee3-1131794b82d3}"
2025-01-25T14:29:34.213Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: aborting pipeline replication: peer="{Voter server-95f24c46-890b-9e09-638e-f956eeb2397b 95f24c46-890b-9e09-638e-f956eeb2397b}"
2025-01-25T14:29:34.223Z [ERROR] server-23e3abb6-f312-acc9-15cd-330a48fc9f67: failed to get log: index=23 error="log not found"
2025-01-25T14:29:34.223Z [ERROR] server-23e3abb6-f312-acc9-15cd-330a48fc9f67: failed to get log: index=23 error="log not found"
2025-01-25T14:29:34.224Z [INFO]  creating new snapshot: path=/tmp/raft1531839359/snapshots/2-10013-1737815374224.tmp
2025-01-25T14:29:34.224Z [INFO]  creating new snapshot: path=/tmp/raft1184432685/snapshots/2-10013-1737815374224.tmp
2025-01-25T14:29:34.224Z [INFO]  server-95f24c46-890b-9e09-638e-f956eeb2397b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:34.225Z [INFO]  server-c4659b27-d1b5-5fde-eee3-1131794b82d3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:34.225Z [INFO]  server-95f24c46-890b-9e09-638e-f956eeb2397b: copied to local snapshot: bytes=71
2025-01-25T14:29:34.225Z [INFO]  server-c4659b27-d1b5-5fde-eee3-1131794b82d3: copied to local snapshot: bytes=71
2025-01-25T14:29:34.226Z [INFO]  server-95f24c46-890b-9e09-638e-f956eeb2397b: snapshot restore progress: id=2-10013-1737815374224 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:34.226Z [INFO]  server-95f24c46-890b-9e09-638e-f956eeb2397b: Installed remote snapshot
2025-01-25T14:29:34.226Z [WARN]  server-95f24c46-890b-9e09-638e-f956eeb2397b: skipping application of old log: index=22
2025-01-25T14:29:34.226Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: pipelining replication: peer="{Voter server-95f24c46-890b-9e09-638e-f956eeb2397b 95f24c46-890b-9e09-638e-f956eeb2397b}"
2025-01-25T14:29:34.226Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:34.227Z [INFO]  server-c4659b27-d1b5-5fde-eee3-1131794b82d3: snapshot restore progress: id=2-10013-1737815374224 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:34.227Z [INFO]  server-c4659b27-d1b5-5fde-eee3-1131794b82d3: Installed remote snapshot
2025-01-25T14:29:34.227Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: pipelining replication: peer="{Voter server-c4659b27-d1b5-5fde-eee3-1131794b82d3 c4659b27-d1b5-5fde-eee3-1131794b82d3}"
2025-01-25T14:29:35.232Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:34.226898851 +0000 UTC m=+84.081223833 (2 nodes), 1.68µs from start of poll, 1.757479994s from cluster start. Timeout at 2025-01-25 14:29:35.232051526 +0000 UTC m=+85.086376468, 1.005152635s after stability
2025-01-25T14:29:35.245Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: aborting pipeline replication: peer="{Voter server-c4659b27-d1b5-5fde-eee3-1131794b82d3 c4659b27-d1b5-5fde-eee3-1131794b82d3}"
2025-01-25T14:29:35.245Z [INFO]  server-23e3abb6-f312-acc9-15cd-330a48fc9f67: aborting pipeline replication: peer="{Voter server-95f24c46-890b-9e09-638e-f956eeb2397b 95f24c46-890b-9e09-638e-f956eeb2397b}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false
2025-01-25T14:29:35.248Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b Address:4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b} {Suffrage:Voter ID:server-3a7a655e-2090-5c6c-0f06-ecd30723cde6 Address:3a7a655e-2090-5c6c-0f06-ecd30723cde6} {Suffrage:Voter ID:server-b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca Address:b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca}]"
2025-01-25T14:29:35.248Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: entering follower state: follower="Node at 4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b [Follower]" leader-address= leader-id=
2025-01-25T14:29:35.248Z [INFO]  server-3a7a655e-2090-5c6c-0f06-ecd30723cde6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b Address:4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b} {Suffrage:Voter ID:server-3a7a655e-2090-5c6c-0f06-ecd30723cde6 Address:3a7a655e-2090-5c6c-0f06-ecd30723cde6} {Suffrage:Voter ID:server-b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca Address:b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca}]"
2025-01-25T14:29:35.248Z [INFO]  server-3a7a655e-2090-5c6c-0f06-ecd30723cde6: entering follower state: follower="Node at 3a7a655e-2090-5c6c-0f06-ecd30723cde6 [Follower]" leader-address= leader-id=
2025-01-25T14:29:35.248Z [INFO]  server-b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b Address:4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b} {Suffrage:Voter ID:server-3a7a655e-2090-5c6c-0f06-ecd30723cde6 Address:3a7a655e-2090-5c6c-0f06-ecd30723cde6} {Suffrage:Voter ID:server-b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca Address:b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca}]"
2025-01-25T14:29:35.248Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:35.248Z [INFO]  server-b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca: entering follower state: follower="Node at b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca [Follower]" leader-address= leader-id=
2025-01-25T14:29:35.834Z [WARN]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:35.834Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: entering candidate state: node="Node at 4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b [Candidate]" term=2
2025-01-25T14:29:35.834Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: election won: term=2 tally=2
2025-01-25T14:29:35.834Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: entering leader state: leader="Node at 4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b [Leader]"
2025-01-25T14:29:35.834Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: added peer, starting replication: peer=server-3a7a655e-2090-5c6c-0f06-ecd30723cde6
2025-01-25T14:29:35.834Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: added peer, starting replication: peer=server-b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca
2025-01-25T14:29:35.835Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: pipelining replication: peer="{Voter server-b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca}"
2025-01-25T14:29:35.835Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: pipelining replication: peer="{Voter server-3a7a655e-2090-5c6c-0f06-ecd30723cde6 3a7a655e-2090-5c6c-0f06-ecd30723cde6}"
2025-01-25T14:29:36.840Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:35.834949263 +0000 UTC m=+85.689274205 (1 nodes), 586.06011ms from start of poll, 586.778604ms from cluster start. Timeout at 2025-01-25 14:29:36.839952103 +0000 UTC m=+86.694277765, 1.00500356s after stability
2025-01-25T14:29:36.841Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: starting snapshot up to: index=12
2025-01-25T14:29:36.841Z [INFO]  creating new snapshot: path=/tmp/raft824119493/snapshots/2-12-1737815376841.tmp
2025-01-25T14:29:36.842Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: snapshot complete up to: index=12
2025-01-25T14:29:36.844Z [INFO]  creating new snapshot: path=/tmp/raft824119493/snapshots/2-10013-1737815376844.tmp
2025-01-25T14:29:36.845Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: copied to local snapshot: bytes=71
2025-01-25T14:29:36.845Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: snapshot restore progress: id=2-10013-1737815376844 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:36.845Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: removing all old logs from log store
2025-01-25T14:29:36.845Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: compacting logs: from=1 to=22
2025-01-25T14:29:36.845Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: restored user snapshot: index=10013
2025-01-25T14:29:36.845Z [ERROR] server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: failed to get log: index=22 error="log not found"
2025-01-25T14:29:36.845Z [ERROR] server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: failed to get log: index=22 error="log not found"
2025-01-25T14:29:36.845Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: aborting pipeline replication: peer="{Voter server-3a7a655e-2090-5c6c-0f06-ecd30723cde6 3a7a655e-2090-5c6c-0f06-ecd30723cde6}"
2025-01-25T14:29:36.845Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: aborting pipeline replication: peer="{Voter server-b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca}"
2025-01-25T14:29:36.851Z [ERROR] server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: failed to get log: index=22 error="log not found"
2025-01-25T14:29:36.851Z [INFO]  creating new snapshot: path=/tmp/raft1759923620/snapshots/2-10013-1737815376851.tmp
2025-01-25T14:29:36.852Z [INFO]  server-b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:36.853Z [INFO]  server-b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca: copied to local snapshot: bytes=71
2025-01-25T14:29:36.853Z [INFO]  server-b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca: snapshot restore progress: id=2-10013-1737815376851 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:36.853Z [INFO]  server-b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca: removing all old logs from log store
2025-01-25T14:29:36.853Z [INFO]  server-b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca: compacting logs: from=1 to=22
2025-01-25T14:29:36.853Z [INFO]  server-b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca: Installed remote snapshot
2025-01-25T14:29:36.853Z [WARN]  server-b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca: skipping application of old log: index=22
2025-01-25T14:29:36.853Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: pipelining replication: peer="{Voter server-b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca}"
2025-01-25T14:29:36.854Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:36.855Z [ERROR] server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: failed to get log: index=22 error="log not found"
2025-01-25T14:29:36.856Z [INFO]  creating new snapshot: path=/tmp/raft1530357578/snapshots/2-10013-1737815376856.tmp
2025-01-25T14:29:36.856Z [INFO]  server-3a7a655e-2090-5c6c-0f06-ecd30723cde6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:36.857Z [INFO]  server-3a7a655e-2090-5c6c-0f06-ecd30723cde6: copied to local snapshot: bytes=71
2025-01-25T14:29:36.857Z [INFO]  server-3a7a655e-2090-5c6c-0f06-ecd30723cde6: snapshot restore progress: id=2-10013-1737815376856 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:36.857Z [INFO]  server-3a7a655e-2090-5c6c-0f06-ecd30723cde6: removing all old logs from log store
2025-01-25T14:29:36.857Z [INFO]  server-3a7a655e-2090-5c6c-0f06-ecd30723cde6: compacting logs: from=1 to=22
2025-01-25T14:29:36.857Z [INFO]  server-3a7a655e-2090-5c6c-0f06-ecd30723cde6: Installed remote snapshot
2025-01-25T14:29:36.858Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: pipelining replication: peer="{Voter server-3a7a655e-2090-5c6c-0f06-ecd30723cde6 3a7a655e-2090-5c6c-0f06-ecd30723cde6}"
2025-01-25T14:29:37.859Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:36.854139825 +0000 UTC m=+86.708464807 (2 nodes), 1.56µs from start of poll, 1.605969206s from cluster start. Timeout at 2025-01-25 14:29:37.85930522 +0000 UTC m=+87.713630202, 1.005165395s after stability
2025-01-25T14:29:37.872Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: aborting pipeline replication: peer="{Voter server-b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca b4acdb5e-008f-1684-0d80-4b1b1a2cd6ca}"
2025-01-25T14:29:37.872Z [INFO]  server-4d9d5fe9-fb2a-23bd-c840-122e74f9cb9b: aborting pipeline replication: peer="{Voter server-3a7a655e-2090-5c6c-0f06-ecd30723cde6 3a7a655e-2090-5c6c-0f06-ecd30723cde6}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true
2025-01-25T14:29:37.876Z [INFO]  server-1ec4db90-9040-e72f-d2cd-01b640a11559: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ec4db90-9040-e72f-d2cd-01b640a11559 Address:1ec4db90-9040-e72f-d2cd-01b640a11559} {Suffrage:Voter ID:server-5544a573-752b-b9a4-f76d-b65b3d6d03b7 Address:5544a573-752b-b9a4-f76d-b65b3d6d03b7} {Suffrage:Voter ID:server-cb831d4f-3295-6d3c-d0dd-bbcc930dae56 Address:cb831d4f-3295-6d3c-d0dd-bbcc930dae56}]"
2025-01-25T14:29:37.876Z [INFO]  server-1ec4db90-9040-e72f-d2cd-01b640a11559: entering follower state: follower="Node at 1ec4db90-9040-e72f-d2cd-01b640a11559 [Follower]" leader-address= leader-id=
2025-01-25T14:29:37.876Z [INFO]  server-5544a573-752b-b9a4-f76d-b65b3d6d03b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ec4db90-9040-e72f-d2cd-01b640a11559 Address:1ec4db90-9040-e72f-d2cd-01b640a11559} {Suffrage:Voter ID:server-5544a573-752b-b9a4-f76d-b65b3d6d03b7 Address:5544a573-752b-b9a4-f76d-b65b3d6d03b7} {Suffrage:Voter ID:server-cb831d4f-3295-6d3c-d0dd-bbcc930dae56 Address:cb831d4f-3295-6d3c-d0dd-bbcc930dae56}]"
2025-01-25T14:29:37.876Z [INFO]  server-5544a573-752b-b9a4-f76d-b65b3d6d03b7: entering follower state: follower="Node at 5544a573-752b-b9a4-f76d-b65b3d6d03b7 [Follower]" leader-address= leader-id=
2025-01-25T14:29:37.876Z [INFO]  server-cb831d4f-3295-6d3c-d0dd-bbcc930dae56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ec4db90-9040-e72f-d2cd-01b640a11559 Address:1ec4db90-9040-e72f-d2cd-01b640a11559} {Suffrage:Voter ID:server-5544a573-752b-b9a4-f76d-b65b3d6d03b7 Address:5544a573-752b-b9a4-f76d-b65b3d6d03b7} {Suffrage:Voter ID:server-cb831d4f-3295-6d3c-d0dd-bbcc930dae56 Address:cb831d4f-3295-6d3c-d0dd-bbcc930dae56}]"
2025-01-25T14:29:37.876Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:37.876Z [INFO]  server-cb831d4f-3295-6d3c-d0dd-bbcc930dae56: entering follower state: follower="Node at cb831d4f-3295-6d3c-d0dd-bbcc930dae56 [Follower]" leader-address= leader-id=
2025-01-25T14:29:38.426Z [WARN]  server-cb831d4f-3295-6d3c-d0dd-bbcc930dae56: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:38.426Z [INFO]  server-cb831d4f-3295-6d3c-d0dd-bbcc930dae56: entering candidate state: node="Node at cb831d4f-3295-6d3c-d0dd-bbcc930dae56 [Candidate]" term=2
2025-01-25T14:29:38.426Z [INFO]  server-cb831d4f-3295-6d3c-d0dd-bbcc930dae56: election won: term=2 tally=2
2025-01-25T14:29:38.426Z [INFO]  server-cb831d4f-3295-6d3c-d0dd-bbcc930dae56: entering leader state: leader="Node at cb831d4f-3295-6d3c-d0dd-bbcc930dae56 [Leader]"
2025-01-25T14:29:38.426Z [INFO]  server-cb831d4f-3295-6d3c-d0dd-bbcc930dae56: added peer, starting replication: peer=server-1ec4db90-9040-e72f-d2cd-01b640a11559
2025-01-25T14:29:38.426Z [INFO]  server-cb831d4f-3295-6d3c-d0dd-bbcc930dae56: added peer, starting replication: peer=server-5544a573-752b-b9a4-f76d-b65b3d6d03b7
2025-01-25T14:29:38.426Z [INFO]  server-cb831d4f-3295-6d3c-d0dd-bbcc930dae56: pipelining replication: peer="{Voter server-5544a573-752b-b9a4-f76d-b65b3d6d03b7 5544a573-752b-b9a4-f76d-b65b3d6d03b7}"
2025-01-25T14:29:38.427Z [INFO]  server-cb831d4f-3295-6d3c-d0dd-bbcc930dae56: pipelining replication: peer="{Voter server-1ec4db90-9040-e72f-d2cd-01b640a11559 1ec4db90-9040-e72f-d2cd-01b640a11559}"
2025-01-25T14:29:39.432Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:38.426879507 +0000 UTC m=+88.281204489 (1 nodes), 550.399033ms from start of poll, 551.098327ms from cluster start. Timeout at 2025-01-25 14:29:39.431885187 +0000 UTC m=+89.286210129, 1.00500564s after stability
2025-01-25T14:29:39.433Z [INFO]  server-cb831d4f-3295-6d3c-d0dd-bbcc930dae56: starting snapshot up to: index=12
2025-01-25T14:29:39.433Z [INFO]  creating new snapshot: path=/tmp/raft3090776554/snapshots/2-12-1737815379433.tmp
2025-01-25T14:29:39.434Z [INFO]  server-cb831d4f-3295-6d3c-d0dd-bbcc930dae56: snapshot complete up to: index=12
2025-01-25T14:29:39.437Z [INFO]  server-66b15622-b3e2-29c3-9e5e-3ad3d8464cdc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66b15622-b3e2-29c3-9e5e-3ad3d8464cdc Address:66b15622-b3e2-29c3-9e5e-3ad3d8464cdc} {Suffrage:Voter ID:server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f Address:0e9b5732-88e1-d4bd-f2e7-e25aa13be55f} {Suffrage:Voter ID:server-9288e411-bf17-9328-4b25-c3fa1906c367 Address:9288e411-bf17-9328-4b25-c3fa1906c367}]"
2025-01-25T14:29:39.437Z [INFO]  server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66b15622-b3e2-29c3-9e5e-3ad3d8464cdc Address:66b15622-b3e2-29c3-9e5e-3ad3d8464cdc} {Suffrage:Voter ID:server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f Address:0e9b5732-88e1-d4bd-f2e7-e25aa13be55f} {Suffrage:Voter ID:server-9288e411-bf17-9328-4b25-c3fa1906c367 Address:9288e411-bf17-9328-4b25-c3fa1906c367}]"
2025-01-25T14:29:39.437Z [INFO]  server-66b15622-b3e2-29c3-9e5e-3ad3d8464cdc: entering follower state: follower="Node at 66b15622-b3e2-29c3-9e5e-3ad3d8464cdc [Follower]" leader-address= leader-id=
2025-01-25T14:29:39.437Z [INFO]  server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: entering follower state: follower="Node at 0e9b5732-88e1-d4bd-f2e7-e25aa13be55f [Follower]" leader-address= leader-id=
2025-01-25T14:29:39.437Z [INFO]  server-9288e411-bf17-9328-4b25-c3fa1906c367: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66b15622-b3e2-29c3-9e5e-3ad3d8464cdc Address:66b15622-b3e2-29c3-9e5e-3ad3d8464cdc} {Suffrage:Voter ID:server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f Address:0e9b5732-88e1-d4bd-f2e7-e25aa13be55f} {Suffrage:Voter ID:server-9288e411-bf17-9328-4b25-c3fa1906c367 Address:9288e411-bf17-9328-4b25-c3fa1906c367}]"
2025-01-25T14:29:39.437Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:39.437Z [INFO]  server-9288e411-bf17-9328-4b25-c3fa1906c367: entering follower state: follower="Node at 9288e411-bf17-9328-4b25-c3fa1906c367 [Follower]" leader-address= leader-id=
2025-01-25T14:29:40.177Z [WARN]  server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:40.177Z [INFO]  server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: entering candidate state: node="Node at 0e9b5732-88e1-d4bd-f2e7-e25aa13be55f [Candidate]" term=2
2025-01-25T14:29:40.177Z [INFO]  server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: election won: term=2 tally=2
2025-01-25T14:29:40.177Z [INFO]  server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: entering leader state: leader="Node at 0e9b5732-88e1-d4bd-f2e7-e25aa13be55f [Leader]"
2025-01-25T14:29:40.177Z [INFO]  server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: added peer, starting replication: peer=server-66b15622-b3e2-29c3-9e5e-3ad3d8464cdc
2025-01-25T14:29:40.177Z [INFO]  server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: added peer, starting replication: peer=server-9288e411-bf17-9328-4b25-c3fa1906c367
2025-01-25T14:29:40.177Z [INFO]  server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: pipelining replication: peer="{Voter server-66b15622-b3e2-29c3-9e5e-3ad3d8464cdc 66b15622-b3e2-29c3-9e5e-3ad3d8464cdc}"
2025-01-25T14:29:40.177Z [INFO]  server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: pipelining replication: peer="{Voter server-9288e411-bf17-9328-4b25-c3fa1906c367 9288e411-bf17-9328-4b25-c3fa1906c367}"
2025-01-25T14:29:41.182Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:40.177590345 +0000 UTC m=+90.031915287 (1 nodes), 739.775058ms from start of poll, 740.487553ms from cluster start. Timeout at 2025-01-25 14:29:41.182601745 +0000 UTC m=+91.036926687, 1.0050114s after stability
2025-01-25T14:29:41.183Z [INFO]  creating new snapshot: path=/tmp/raft1459660841/snapshots/2-10013-1737815381183.tmp
2025-01-25T14:29:41.184Z [INFO]  server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: copied to local snapshot: bytes=71
2025-01-25T14:29:41.184Z [INFO]  server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: snapshot restore progress: id=2-10013-1737815381183 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:41.184Z [INFO]  server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: restored user snapshot: index=10013
2025-01-25T14:29:41.185Z [ERROR] server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: failed to get log: index=3 error="log not found"
2025-01-25T14:29:41.185Z [INFO]  server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: aborting pipeline replication: peer="{Voter server-9288e411-bf17-9328-4b25-c3fa1906c367 9288e411-bf17-9328-4b25-c3fa1906c367}"
2025-01-25T14:29:41.185Z [ERROR] server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: failed to get log: index=3 error="log not found"
2025-01-25T14:29:41.185Z [INFO]  server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: aborting pipeline replication: peer="{Voter server-66b15622-b3e2-29c3-9e5e-3ad3d8464cdc 66b15622-b3e2-29c3-9e5e-3ad3d8464cdc}"
2025-01-25T14:29:41.190Z [ERROR] server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: failed to get log: index=3 error="log not found"
2025-01-25T14:29:41.191Z [INFO]  creating new snapshot: path=/tmp/raft1722530050/snapshots/2-10013-1737815381191.tmp
2025-01-25T14:29:41.191Z [ERROR] server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: failed to get log: index=3 error="log not found"
2025-01-25T14:29:41.191Z [INFO]  server-9288e411-bf17-9328-4b25-c3fa1906c367: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:41.192Z [INFO]  server-9288e411-bf17-9328-4b25-c3fa1906c367: copied to local snapshot: bytes=71
2025-01-25T14:29:41.192Z [INFO]  creating new snapshot: path=/tmp/raft14806582/snapshots/2-10013-1737815381192.tmp
2025-01-25T14:29:41.192Z [INFO]  server-9288e411-bf17-9328-4b25-c3fa1906c367: snapshot restore progress: id=2-10013-1737815381191 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:41.192Z [INFO]  server-9288e411-bf17-9328-4b25-c3fa1906c367: Installed remote snapshot
2025-01-25T14:29:41.192Z [INFO]  server-66b15622-b3e2-29c3-9e5e-3ad3d8464cdc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:41.193Z [INFO]  server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: pipelining replication: peer="{Voter server-9288e411-bf17-9328-4b25-c3fa1906c367 9288e411-bf17-9328-4b25-c3fa1906c367}"
2025-01-25T14:29:41.193Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:41.193Z [INFO]  server-66b15622-b3e2-29c3-9e5e-3ad3d8464cdc: copied to local snapshot: bytes=71
2025-01-25T14:29:41.194Z [INFO]  server-66b15622-b3e2-29c3-9e5e-3ad3d8464cdc: snapshot restore progress: id=2-10013-1737815381192 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:41.194Z [INFO]  server-66b15622-b3e2-29c3-9e5e-3ad3d8464cdc: Installed remote snapshot
2025-01-25T14:29:41.194Z [INFO]  server-0e9b5732-88e1-d4bd-f2e7-e25aa13be55f: pipelining replication: peer="{Voter server-66b15622-b3e2-29c3-9e5e-3ad3d8464cdc 66b15622-b3e2-29c3-9e5e-3ad3d8464cdc}"
2025-01-25T14:29:42.199Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:41.193207756 +0000 UTC m=+91.047532738 (2 nodes), 1.56µs from start of poll, 1.756105004s from cluster start. Timeout at 2025-01-25 14:29:42.198430988 +0000 UTC m=+92.052755970, 1.005223232s after stability
2025-01-25T14:29:42.211Z [INFO]  server-cb831d4f-3295-6d3c-d0dd-bbcc930dae56: aborting pipeline replication: peer="{Voter server-1ec4db90-9040-e72f-d2cd-01b640a11559 1ec4db90-9040-e72f-d2cd-01b640a11559}"
2025-01-25T14:29:42.211Z [INFO]  server-cb831d4f-3295-6d3c-d0dd-bbcc930dae56: aborting pipeline replication: peer="{Voter server-5544a573-752b-b9a4-f76d-b65b3d6d03b7 5544a573-752b-b9a4-f76d-b65b3d6d03b7}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true
2025-01-25T14:29:42.214Z [INFO]  server-2d4bbf4a-b755-51fe-7de1-eb34e5a1c47e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d4bbf4a-b755-51fe-7de1-eb34e5a1c47e Address:2d4bbf4a-b755-51fe-7de1-eb34e5a1c47e} {Suffrage:Voter ID:server-04ef2b72-148b-89d7-9562-cc9d60eba792 Address:04ef2b72-148b-89d7-9562-cc9d60eba792} {Suffrage:Voter ID:server-777f47a4-665c-dae8-ed72-6302f2e20c9c Address:777f47a4-665c-dae8-ed72-6302f2e20c9c}]"
2025-01-25T14:29:42.214Z [INFO]  server-04ef2b72-148b-89d7-9562-cc9d60eba792: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d4bbf4a-b755-51fe-7de1-eb34e5a1c47e Address:2d4bbf4a-b755-51fe-7de1-eb34e5a1c47e} {Suffrage:Voter ID:server-04ef2b72-148b-89d7-9562-cc9d60eba792 Address:04ef2b72-148b-89d7-9562-cc9d60eba792} {Suffrage:Voter ID:server-777f47a4-665c-dae8-ed72-6302f2e20c9c Address:777f47a4-665c-dae8-ed72-6302f2e20c9c}]"
2025-01-25T14:29:42.214Z [INFO]  server-2d4bbf4a-b755-51fe-7de1-eb34e5a1c47e: entering follower state: follower="Node at 2d4bbf4a-b755-51fe-7de1-eb34e5a1c47e [Follower]" leader-address= leader-id=
2025-01-25T14:29:42.214Z [INFO]  server-777f47a4-665c-dae8-ed72-6302f2e20c9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d4bbf4a-b755-51fe-7de1-eb34e5a1c47e Address:2d4bbf4a-b755-51fe-7de1-eb34e5a1c47e} {Suffrage:Voter ID:server-04ef2b72-148b-89d7-9562-cc9d60eba792 Address:04ef2b72-148b-89d7-9562-cc9d60eba792} {Suffrage:Voter ID:server-777f47a4-665c-dae8-ed72-6302f2e20c9c Address:777f47a4-665c-dae8-ed72-6302f2e20c9c}]"
2025-01-25T14:29:42.214Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:42.214Z [INFO]  server-777f47a4-665c-dae8-ed72-6302f2e20c9c: entering follower state: follower="Node at 777f47a4-665c-dae8-ed72-6302f2e20c9c [Follower]" leader-address= leader-id=
2025-01-25T14:29:42.214Z [INFO]  server-04ef2b72-148b-89d7-9562-cc9d60eba792: entering follower state: follower="Node at 04ef2b72-148b-89d7-9562-cc9d60eba792 [Follower]" leader-address= leader-id=
2025-01-25T14:29:42.792Z [WARN]  server-04ef2b72-148b-89d7-9562-cc9d60eba792: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:42.792Z [INFO]  server-04ef2b72-148b-89d7-9562-cc9d60eba792: entering candidate state: node="Node at 04ef2b72-148b-89d7-9562-cc9d60eba792 [Candidate]" term=2
2025-01-25T14:29:42.792Z [INFO]  server-04ef2b72-148b-89d7-9562-cc9d60eba792: election won: term=2 tally=2
2025-01-25T14:29:42.792Z [INFO]  server-04ef2b72-148b-89d7-9562-cc9d60eba792: entering leader state: leader="Node at 04ef2b72-148b-89d7-9562-cc9d60eba792 [Leader]"
2025-01-25T14:29:42.792Z [INFO]  server-04ef2b72-148b-89d7-9562-cc9d60eba792: added peer, starting replication: peer=server-2d4bbf4a-b755-51fe-7de1-eb34e5a1c47e
2025-01-25T14:29:42.792Z [INFO]  server-04ef2b72-148b-89d7-9562-cc9d60eba792: added peer, starting replication: peer=server-777f47a4-665c-dae8-ed72-6302f2e20c9c
2025-01-25T14:29:42.793Z [INFO]  server-04ef2b72-148b-89d7-9562-cc9d60eba792: pipelining replication: peer="{Voter server-777f47a4-665c-dae8-ed72-6302f2e20c9c 777f47a4-665c-dae8-ed72-6302f2e20c9c}"
2025-01-25T14:29:42.793Z [INFO]  server-04ef2b72-148b-89d7-9562-cc9d60eba792: pipelining replication: peer="{Voter server-2d4bbf4a-b755-51fe-7de1-eb34e5a1c47e 2d4bbf4a-b755-51fe-7de1-eb34e5a1c47e}"
2025-01-25T14:29:43.798Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:42.793489846 +0000 UTC m=+92.647814828 (1 nodes), 578.80433ms from start of poll, 579.508425ms from cluster start. Timeout at 2025-01-25 14:29:43.798502566 +0000 UTC m=+93.652827548, 1.00501272s after stability
2025-01-25T14:29:43.799Z [INFO]  server-04ef2b72-148b-89d7-9562-cc9d60eba792: starting snapshot up to: index=12
2025-01-25T14:29:43.799Z [INFO]  creating new snapshot: path=/tmp/raft2785895767/snapshots/2-12-1737815383799.tmp
2025-01-25T14:29:43.800Z [INFO]  server-04ef2b72-148b-89d7-9562-cc9d60eba792: snapshot complete up to: index=12
2025-01-25T14:29:43.803Z [INFO]  server-ab73106a-aebb-13bf-c751-17b2f518886c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ab73106a-aebb-13bf-c751-17b2f518886c Address:ab73106a-aebb-13bf-c751-17b2f518886c} {Suffrage:Voter ID:server-ddd30d47-14a1-11c1-b864-16ac8f6862b7 Address:ddd30d47-14a1-11c1-b864-16ac8f6862b7} {Suffrage:Voter ID:server-678fc9c7-1974-ff73-c888-d3fb679aa89c Address:678fc9c7-1974-ff73-c888-d3fb679aa89c}]"
2025-01-25T14:29:43.803Z [INFO]  server-ab73106a-aebb-13bf-c751-17b2f518886c: entering follower state: follower="Node at ab73106a-aebb-13bf-c751-17b2f518886c [Follower]" leader-address= leader-id=
2025-01-25T14:29:43.803Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ab73106a-aebb-13bf-c751-17b2f518886c Address:ab73106a-aebb-13bf-c751-17b2f518886c} {Suffrage:Voter ID:server-ddd30d47-14a1-11c1-b864-16ac8f6862b7 Address:ddd30d47-14a1-11c1-b864-16ac8f6862b7} {Suffrage:Voter ID:server-678fc9c7-1974-ff73-c888-d3fb679aa89c Address:678fc9c7-1974-ff73-c888-d3fb679aa89c}]"
2025-01-25T14:29:43.803Z [INFO]  server-678fc9c7-1974-ff73-c888-d3fb679aa89c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ab73106a-aebb-13bf-c751-17b2f518886c Address:ab73106a-aebb-13bf-c751-17b2f518886c} {Suffrage:Voter ID:server-ddd30d47-14a1-11c1-b864-16ac8f6862b7 Address:ddd30d47-14a1-11c1-b864-16ac8f6862b7} {Suffrage:Voter ID:server-678fc9c7-1974-ff73-c888-d3fb679aa89c Address:678fc9c7-1974-ff73-c888-d3fb679aa89c}]"
2025-01-25T14:29:43.803Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: entering follower state: follower="Node at ddd30d47-14a1-11c1-b864-16ac8f6862b7 [Follower]" leader-address= leader-id=
2025-01-25T14:29:43.803Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:43.803Z [INFO]  server-678fc9c7-1974-ff73-c888-d3fb679aa89c: entering follower state: follower="Node at 678fc9c7-1974-ff73-c888-d3fb679aa89c [Follower]" leader-address= leader-id=
2025-01-25T14:29:44.352Z [WARN]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:44.352Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: entering candidate state: node="Node at ddd30d47-14a1-11c1-b864-16ac8f6862b7 [Candidate]" term=2
2025-01-25T14:29:44.353Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: election won: term=2 tally=2
2025-01-25T14:29:44.353Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: entering leader state: leader="Node at ddd30d47-14a1-11c1-b864-16ac8f6862b7 [Leader]"
2025-01-25T14:29:44.353Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: added peer, starting replication: peer=server-ab73106a-aebb-13bf-c751-17b2f518886c
2025-01-25T14:29:44.353Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: added peer, starting replication: peer=server-678fc9c7-1974-ff73-c888-d3fb679aa89c
2025-01-25T14:29:44.353Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: pipelining replication: peer="{Voter server-678fc9c7-1974-ff73-c888-d3fb679aa89c 678fc9c7-1974-ff73-c888-d3fb679aa89c}"
2025-01-25T14:29:44.353Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: pipelining replication: peer="{Voter server-ab73106a-aebb-13bf-c751-17b2f518886c ab73106a-aebb-13bf-c751-17b2f518886c}"
2025-01-25T14:29:45.359Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:44.353157318 +0000 UTC m=+94.207482300 (1 nodes), 549.339951ms from start of poll, 550.18792ms from cluster start. Timeout at 2025-01-25 14:29:45.358160238 +0000 UTC m=+95.212485220, 1.00500292s after stability
2025-01-25T14:29:45.360Z [INFO]  creating new snapshot: path=/tmp/raft3009688639/snapshots/2-10013-1737815385360.tmp
2025-01-25T14:29:45.361Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: copied to local snapshot: bytes=71
2025-01-25T14:29:45.361Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: snapshot restore progress: id=2-10013-1737815385360 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:45.362Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: removing all old logs from log store
2025-01-25T14:29:45.362Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: compacting logs: from=1 to=2
2025-01-25T14:29:45.362Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: restored user snapshot: index=10013
2025-01-25T14:29:45.362Z [ERROR] server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: failed to get log: index=2 error="log not found"
2025-01-25T14:29:45.362Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: aborting pipeline replication: peer="{Voter server-678fc9c7-1974-ff73-c888-d3fb679aa89c 678fc9c7-1974-ff73-c888-d3fb679aa89c}"
2025-01-25T14:29:45.362Z [ERROR] server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: failed to get log: index=2 error="log not found"
2025-01-25T14:29:45.362Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: aborting pipeline replication: peer="{Voter server-ab73106a-aebb-13bf-c751-17b2f518886c ab73106a-aebb-13bf-c751-17b2f518886c}"
2025-01-25T14:29:45.368Z [ERROR] server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: failed to get log: index=2 error="log not found"
2025-01-25T14:29:45.368Z [INFO]  creating new snapshot: path=/tmp/raft2273850496/snapshots/2-10013-1737815385368.tmp
2025-01-25T14:29:45.369Z [INFO]  server-ab73106a-aebb-13bf-c751-17b2f518886c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:45.370Z [INFO]  server-ab73106a-aebb-13bf-c751-17b2f518886c: copied to local snapshot: bytes=71
2025-01-25T14:29:45.370Z [INFO]  server-ab73106a-aebb-13bf-c751-17b2f518886c: snapshot restore progress: id=2-10013-1737815385368 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:45.370Z [INFO]  server-ab73106a-aebb-13bf-c751-17b2f518886c: removing all old logs from log store
2025-01-25T14:29:45.370Z [INFO]  server-ab73106a-aebb-13bf-c751-17b2f518886c: compacting logs: from=1 to=2
2025-01-25T14:29:45.370Z [INFO]  server-ab73106a-aebb-13bf-c751-17b2f518886c: Installed remote snapshot
2025-01-25T14:29:45.371Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: pipelining replication: peer="{Voter server-ab73106a-aebb-13bf-c751-17b2f518886c ab73106a-aebb-13bf-c751-17b2f518886c}"
2025-01-25T14:29:45.371Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:45.371Z [ERROR] server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: failed to get log: index=2 error="log not found"
2025-01-25T14:29:45.372Z [INFO]  creating new snapshot: path=/tmp/raft1251686499/snapshots/2-10013-1737815385372.tmp
2025-01-25T14:29:45.372Z [INFO]  server-678fc9c7-1974-ff73-c888-d3fb679aa89c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:45.373Z [INFO]  server-678fc9c7-1974-ff73-c888-d3fb679aa89c: copied to local snapshot: bytes=71
2025-01-25T14:29:45.373Z [INFO]  server-678fc9c7-1974-ff73-c888-d3fb679aa89c: snapshot restore progress: id=2-10013-1737815385372 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-25T14:29:45.374Z [INFO]  server-678fc9c7-1974-ff73-c888-d3fb679aa89c: removing all old logs from log store
2025-01-25T14:29:45.374Z [INFO]  server-678fc9c7-1974-ff73-c888-d3fb679aa89c: compacting logs: from=1 to=2
2025-01-25T14:29:45.374Z [INFO]  server-678fc9c7-1974-ff73-c888-d3fb679aa89c: Installed remote snapshot
2025-01-25T14:29:45.374Z [INFO]  server-ddd30d47-14a1-11c1-b864-16ac8f6862b7: pipelining replication: peer="{Voter server-678fc9c7-1974-ff73-c888-d3fb679aa89c 678fc9c7-1974-ff73-c888-d3fb679aa89c}"
2025-01-25T14:29:46.376Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:45.371324557 +0000 UTC m=+95.225649499 (2 nodes), 1.96µs from start of poll, 1.568355119s from cluster start. Timeout at 2025-01-25 14:29:46.376580507 +0000 UTC m=+96.230905489, 1.00525599s after stability
2025-01-25T14:29:46.388Z [INFO]  server-04ef2b72-148b-89d7-9562-cc9d60eba792: aborting pipeline replication: peer="{Voter server-777f47a4-665c-dae8-ed72-6302f2e20c9c 777f47a4-665c-dae8-ed72-6302f2e20c9c}"
2025-01-25T14:29:46.388Z [INFO]  server-04ef2b72-148b-89d7-9562-cc9d60eba792: aborting pipeline replication: peer="{Voter server-2d4bbf4a-b755-51fe-7de1-eb34e5a1c47e 2d4bbf4a-b755-51fe-7de1-eb34e5a1c47e}"
--- PASS: TestRaft_UserRestore (82.81s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.71s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.58s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.08s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.26s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.90s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.43s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.56s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.55s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.21s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.22s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.78s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.14s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.19s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.56s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.63s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.14s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.12s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.78s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.63s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.34s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.18s)
=== RUN   TestRaft_SendSnapshotFollower
2025-01-25T14:29:46.398Z [INFO]  server-df7d3529-fcf7-edee-33ba-4dc1c04ad324: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df7d3529-fcf7-edee-33ba-4dc1c04ad324 Address:df7d3529-fcf7-edee-33ba-4dc1c04ad324} {Suffrage:Voter ID:server-558d93a0-fba7-31db-1efc-5f3d1d067252 Address:558d93a0-fba7-31db-1efc-5f3d1d067252} {Suffrage:Voter ID:server-966e830d-4b6e-96dd-70de-88075b220f08 Address:966e830d-4b6e-96dd-70de-88075b220f08}]"
2025-01-25T14:29:46.401Z [INFO]  server-df7d3529-fcf7-edee-33ba-4dc1c04ad324: entering follower state: follower="Node at df7d3529-fcf7-edee-33ba-4dc1c04ad324 [Follower]" leader-address= leader-id=
2025-01-25T14:29:46.401Z [INFO]  server-558d93a0-fba7-31db-1efc-5f3d1d067252: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df7d3529-fcf7-edee-33ba-4dc1c04ad324 Address:df7d3529-fcf7-edee-33ba-4dc1c04ad324} {Suffrage:Voter ID:server-558d93a0-fba7-31db-1efc-5f3d1d067252 Address:558d93a0-fba7-31db-1efc-5f3d1d067252} {Suffrage:Voter ID:server-966e830d-4b6e-96dd-70de-88075b220f08 Address:966e830d-4b6e-96dd-70de-88075b220f08}]"
2025-01-25T14:29:46.401Z [INFO]  server-966e830d-4b6e-96dd-70de-88075b220f08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df7d3529-fcf7-edee-33ba-4dc1c04ad324 Address:df7d3529-fcf7-edee-33ba-4dc1c04ad324} {Suffrage:Voter ID:server-558d93a0-fba7-31db-1efc-5f3d1d067252 Address:558d93a0-fba7-31db-1efc-5f3d1d067252} {Suffrage:Voter ID:server-966e830d-4b6e-96dd-70de-88075b220f08 Address:966e830d-4b6e-96dd-70de-88075b220f08}]"
2025-01-25T14:29:46.401Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:46.401Z [INFO]  server-558d93a0-fba7-31db-1efc-5f3d1d067252: entering follower state: follower="Node at 558d93a0-fba7-31db-1efc-5f3d1d067252 [Follower]" leader-address= leader-id=
2025-01-25T14:29:46.401Z [INFO]  server-966e830d-4b6e-96dd-70de-88075b220f08: entering follower state: follower="Node at 966e830d-4b6e-96dd-70de-88075b220f08 [Follower]" leader-address= leader-id=
2025-01-25T14:29:46.460Z [WARN]  server-966e830d-4b6e-96dd-70de-88075b220f08: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:46.460Z [INFO]  server-966e830d-4b6e-96dd-70de-88075b220f08: entering candidate state: node="Node at 966e830d-4b6e-96dd-70de-88075b220f08 [Candidate]" term=2
2025-01-25T14:29:46.460Z [INFO]  server-966e830d-4b6e-96dd-70de-88075b220f08: election won: term=2 tally=2
2025-01-25T14:29:46.460Z [INFO]  server-966e830d-4b6e-96dd-70de-88075b220f08: entering leader state: leader="Node at 966e830d-4b6e-96dd-70de-88075b220f08 [Leader]"
2025-01-25T14:29:46.460Z [INFO]  server-966e830d-4b6e-96dd-70de-88075b220f08: added peer, starting replication: peer=server-df7d3529-fcf7-edee-33ba-4dc1c04ad324
2025-01-25T14:29:46.460Z [INFO]  server-966e830d-4b6e-96dd-70de-88075b220f08: added peer, starting replication: peer=server-558d93a0-fba7-31db-1efc-5f3d1d067252
2025-01-25T14:29:46.460Z [INFO]  server-966e830d-4b6e-96dd-70de-88075b220f08: pipelining replication: peer="{Voter server-558d93a0-fba7-31db-1efc-5f3d1d067252 558d93a0-fba7-31db-1efc-5f3d1d067252}"
2025-01-25T14:29:46.460Z [INFO]  server-966e830d-4b6e-96dd-70de-88075b220f08: pipelining replication: peer="{Voter server-df7d3529-fcf7-edee-33ba-4dc1c04ad324 df7d3529-fcf7-edee-33ba-4dc1c04ad324}"
2025-01-25T14:29:46.566Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:46.460646636 +0000 UTC m=+96.314971618 (2 nodes), 58.799244ms from start of poll, 68.37718ms from cluster start. Timeout at 2025-01-25 14:29:46.565650316 +0000 UTC m=+96.419975298, 105.00368ms after stability
2025-01-25T14:29:46.566Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:46.672Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:46.566366431 +0000 UTC m=+96.420691413 (1 nodes), 1.68µs from start of poll, 174.096975ms from cluster start. Timeout at 2025-01-25 14:29:46.671521466 +0000 UTC m=+96.525846488, 105.155075ms after stability
2025-01-25T14:29:46.674Z [ERROR] server-966e830d-4b6e-96dd-70de-88075b220f08: failed to pipeline appendEntries: peer="{Voter server-df7d3529-fcf7-edee-33ba-4dc1c04ad324 df7d3529-fcf7-edee-33ba-4dc1c04ad324}" error="append pipeline closed"
2025-01-25T14:29:46.674Z [INFO]  server-966e830d-4b6e-96dd-70de-88075b220f08: aborting pipeline replication: peer="{Voter server-df7d3529-fcf7-edee-33ba-4dc1c04ad324 df7d3529-fcf7-edee-33ba-4dc1c04ad324}"
2025-01-25T14:29:46.674Z [ERROR] server-966e830d-4b6e-96dd-70de-88075b220f08: failed to appendEntries to: peer="{Voter server-df7d3529-fcf7-edee-33ba-4dc1c04ad324 df7d3529-fcf7-edee-33ba-4dc1c04ad324}" error="failed to connect to peer: df7d3529-fcf7-edee-33ba-4dc1c04ad324"
2025-01-25T14:29:46.676Z [ERROR] server-966e830d-4b6e-96dd-70de-88075b220f08: failed to heartbeat to: peer=df7d3529-fcf7-edee-33ba-4dc1c04ad324 backoff time=10ms error="failed to connect to peer: df7d3529-fcf7-edee-33ba-4dc1c04ad324"
    raft_test.go:1627: [INFO] Finished apply without behind follower
2025-01-25T14:29:46.683Z [ERROR] server-df7d3529-fcf7-edee-33ba-4dc1c04ad324: failed to take snapshot: error="nothing new to snapshot"
2025-01-25T14:29:46.683Z [INFO]  server-558d93a0-fba7-31db-1efc-5f3d1d067252: starting snapshot up to: index=80
2025-01-25T14:29:46.683Z [INFO]  creating new snapshot: path=/tmp/raft213672628/snapshots/2-80-1737815386683.tmp
2025-01-25T14:29:46.685Z [ERROR] server-966e830d-4b6e-96dd-70de-88075b220f08: failed to appendEntries to: peer="{Voter server-df7d3529-fcf7-edee-33ba-4dc1c04ad324 df7d3529-fcf7-edee-33ba-4dc1c04ad324}" error="failed to connect to peer: df7d3529-fcf7-edee-33ba-4dc1c04ad324"
2025-01-25T14:29:46.694Z [INFO]  server-558d93a0-fba7-31db-1efc-5f3d1d067252: compacting logs: from=1 to=80
2025-01-25T14:29:46.694Z [INFO]  server-558d93a0-fba7-31db-1efc-5f3d1d067252: snapshot complete up to: index=80
2025-01-25T14:29:46.695Z [ERROR] server-966e830d-4b6e-96dd-70de-88075b220f08: failed to appendEntries to: peer="{Voter server-df7d3529-fcf7-edee-33ba-4dc1c04ad324 df7d3529-fcf7-edee-33ba-4dc1c04ad324}" error="failed to connect to peer: df7d3529-fcf7-edee-33ba-4dc1c04ad324"
2025-01-25T14:29:46.696Z [INFO]  server-966e830d-4b6e-96dd-70de-88075b220f08: starting snapshot up to: index=102
2025-01-25T14:29:46.696Z [INFO]  creating new snapshot: path=/tmp/raft408683412/snapshots/2-102-1737815386696.tmp
2025-01-25T14:29:46.699Z [INFO]  server-966e830d-4b6e-96dd-70de-88075b220f08: compacting logs: from=1 to=92
2025-01-25T14:29:46.699Z [INFO]  server-966e830d-4b6e-96dd-70de-88075b220f08: snapshot complete up to: index=102
2025-01-25T14:29:46.725Z [ERROR] server-966e830d-4b6e-96dd-70de-88075b220f08: failed to get log: index=2 error="log not found"
2025-01-25T14:29:46.726Z [INFO]  creating new snapshot: path=/tmp/raft2294705873/snapshots/2-102-1737815386726.tmp
2025-01-25T14:29:46.726Z [INFO]  server-df7d3529-fcf7-edee-33ba-4dc1c04ad324: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-01-25T14:29:46.727Z [INFO]  server-df7d3529-fcf7-edee-33ba-4dc1c04ad324: copied to local snapshot: bytes=693
2025-01-25T14:29:46.727Z [INFO]  server-df7d3529-fcf7-edee-33ba-4dc1c04ad324: snapshot restore progress: id=2-102-1737815386726 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-01-25T14:29:46.728Z [INFO]  server-df7d3529-fcf7-edee-33ba-4dc1c04ad324: Installed remote snapshot
2025-01-25T14:29:46.728Z [INFO]  server-966e830d-4b6e-96dd-70de-88075b220f08: aborting pipeline replication: peer="{Voter server-558d93a0-fba7-31db-1efc-5f3d1d067252 558d93a0-fba7-31db-1efc-5f3d1d067252}"
--- PASS: TestRaft_SendSnapshotFollower (0.34s)
=== RUN   TestRaft_SendSnapshotAndLogsFollower
2025-01-25T14:29:46.732Z [INFO]  server-3ac21012-71e3-68a6-a999-f85fbd9242ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ac21012-71e3-68a6-a999-f85fbd9242ad Address:3ac21012-71e3-68a6-a999-f85fbd9242ad} {Suffrage:Voter ID:server-c62c5dcf-3dac-4404-82e6-619ff7dba9f8 Address:c62c5dcf-3dac-4404-82e6-619ff7dba9f8} {Suffrage:Voter ID:server-16af2563-94ff-cd32-6c94-112692f1aa3d Address:16af2563-94ff-cd32-6c94-112692f1aa3d}]"
2025-01-25T14:29:46.732Z [INFO]  server-3ac21012-71e3-68a6-a999-f85fbd9242ad: entering follower state: follower="Node at 3ac21012-71e3-68a6-a999-f85fbd9242ad [Follower]" leader-address= leader-id=
2025-01-25T14:29:46.732Z [INFO]  server-c62c5dcf-3dac-4404-82e6-619ff7dba9f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ac21012-71e3-68a6-a999-f85fbd9242ad Address:3ac21012-71e3-68a6-a999-f85fbd9242ad} {Suffrage:Voter ID:server-c62c5dcf-3dac-4404-82e6-619ff7dba9f8 Address:c62c5dcf-3dac-4404-82e6-619ff7dba9f8} {Suffrage:Voter ID:server-16af2563-94ff-cd32-6c94-112692f1aa3d Address:16af2563-94ff-cd32-6c94-112692f1aa3d}]"
2025-01-25T14:29:46.732Z [INFO]  server-c62c5dcf-3dac-4404-82e6-619ff7dba9f8: entering follower state: follower="Node at c62c5dcf-3dac-4404-82e6-619ff7dba9f8 [Follower]" leader-address= leader-id=
2025-01-25T14:29:46.732Z [INFO]  server-16af2563-94ff-cd32-6c94-112692f1aa3d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ac21012-71e3-68a6-a999-f85fbd9242ad Address:3ac21012-71e3-68a6-a999-f85fbd9242ad} {Suffrage:Voter ID:server-c62c5dcf-3dac-4404-82e6-619ff7dba9f8 Address:c62c5dcf-3dac-4404-82e6-619ff7dba9f8} {Suffrage:Voter ID:server-16af2563-94ff-cd32-6c94-112692f1aa3d Address:16af2563-94ff-cd32-6c94-112692f1aa3d}]"
2025-01-25T14:29:46.732Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:46.732Z [INFO]  server-16af2563-94ff-cd32-6c94-112692f1aa3d: entering follower state: follower="Node at 16af2563-94ff-cd32-6c94-112692f1aa3d [Follower]" leader-address= leader-id=
2025-01-25T14:29:46.795Z [WARN]  server-16af2563-94ff-cd32-6c94-112692f1aa3d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:46.795Z [INFO]  server-16af2563-94ff-cd32-6c94-112692f1aa3d: entering candidate state: node="Node at 16af2563-94ff-cd32-6c94-112692f1aa3d [Candidate]" term=2
2025-01-25T14:29:46.796Z [INFO]  server-16af2563-94ff-cd32-6c94-112692f1aa3d: election won: term=2 tally=2
2025-01-25T14:29:46.796Z [INFO]  server-16af2563-94ff-cd32-6c94-112692f1aa3d: entering leader state: leader="Node at 16af2563-94ff-cd32-6c94-112692f1aa3d [Leader]"
2025-01-25T14:29:46.796Z [INFO]  server-16af2563-94ff-cd32-6c94-112692f1aa3d: added peer, starting replication: peer=server-3ac21012-71e3-68a6-a999-f85fbd9242ad
2025-01-25T14:29:46.796Z [INFO]  server-16af2563-94ff-cd32-6c94-112692f1aa3d: added peer, starting replication: peer=server-c62c5dcf-3dac-4404-82e6-619ff7dba9f8
2025-01-25T14:29:46.796Z [INFO]  server-16af2563-94ff-cd32-6c94-112692f1aa3d: pipelining replication: peer="{Voter server-3ac21012-71e3-68a6-a999-f85fbd9242ad 3ac21012-71e3-68a6-a999-f85fbd9242ad}"
2025-01-25T14:29:46.796Z [INFO]  server-16af2563-94ff-cd32-6c94-112692f1aa3d: pipelining replication: peer="{Voter server-c62c5dcf-3dac-4404-82e6-619ff7dba9f8 c62c5dcf-3dac-4404-82e6-619ff7dba9f8}"
2025-01-25T14:29:46.901Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:46.796246038 +0000 UTC m=+96.650571020 (2 nodes), 63.650909ms from start of poll, 64.317965ms from cluster start. Timeout at 2025-01-25 14:29:46.901251478 +0000 UTC m=+96.755576460, 105.00544ms after stability
2025-01-25T14:29:46.901Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:47.007Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:46.901430973 +0000 UTC m=+96.755755915 (1 nodes), 2.12µs from start of poll, 169.50286ms from cluster start. Timeout at 2025-01-25 14:29:47.006659445 +0000 UTC m=+96.860984387, 105.228472ms after stability
2025-01-25T14:29:47.007Z [ERROR] server-16af2563-94ff-cd32-6c94-112692f1aa3d: failed to pipeline appendEntries: peer="{Voter server-3ac21012-71e3-68a6-a999-f85fbd9242ad 3ac21012-71e3-68a6-a999-f85fbd9242ad}" error="append pipeline closed"
2025-01-25T14:29:47.007Z [INFO]  server-16af2563-94ff-cd32-6c94-112692f1aa3d: aborting pipeline replication: peer="{Voter server-3ac21012-71e3-68a6-a999-f85fbd9242ad 3ac21012-71e3-68a6-a999-f85fbd9242ad}"
2025-01-25T14:29:47.007Z [ERROR] server-16af2563-94ff-cd32-6c94-112692f1aa3d: failed to appendEntries to: peer="{Voter server-3ac21012-71e3-68a6-a999-f85fbd9242ad 3ac21012-71e3-68a6-a999-f85fbd9242ad}" error="failed to connect to peer: 3ac21012-71e3-68a6-a999-f85fbd9242ad"
2025-01-25T14:29:47.009Z [ERROR] server-16af2563-94ff-cd32-6c94-112692f1aa3d: failed to heartbeat to: peer=3ac21012-71e3-68a6-a999-f85fbd9242ad backoff time=10ms error="failed to connect to peer: 3ac21012-71e3-68a6-a999-f85fbd9242ad"
    raft_test.go:1669: [INFO] Finished apply without behind follower
2025-01-25T14:29:47.018Z [ERROR] server-3ac21012-71e3-68a6-a999-f85fbd9242ad: failed to take snapshot: error="nothing new to snapshot"
2025-01-25T14:29:47.018Z [INFO]  server-c62c5dcf-3dac-4404-82e6-619ff7dba9f8: starting snapshot up to: index=97
2025-01-25T14:29:47.018Z [INFO]  creating new snapshot: path=/tmp/raft2276402575/snapshots/2-97-1737815387018.tmp
2025-01-25T14:29:47.018Z [ERROR] server-16af2563-94ff-cd32-6c94-112692f1aa3d: failed to appendEntries to: peer="{Voter server-3ac21012-71e3-68a6-a999-f85fbd9242ad 3ac21012-71e3-68a6-a999-f85fbd9242ad}" error="failed to connect to peer: 3ac21012-71e3-68a6-a999-f85fbd9242ad"
2025-01-25T14:29:47.019Z [INFO]  server-c62c5dcf-3dac-4404-82e6-619ff7dba9f8: compacting logs: from=1 to=92
2025-01-25T14:29:47.019Z [INFO]  server-c62c5dcf-3dac-4404-82e6-619ff7dba9f8: snapshot complete up to: index=97
2025-01-25T14:29:47.019Z [INFO]  server-16af2563-94ff-cd32-6c94-112692f1aa3d: starting snapshot up to: index=102
2025-01-25T14:29:47.020Z [INFO]  creating new snapshot: path=/tmp/raft2369143322/snapshots/2-102-1737815387019.tmp
2025-01-25T14:29:47.021Z [INFO]  server-16af2563-94ff-cd32-6c94-112692f1aa3d: compacting logs: from=1 to=92
2025-01-25T14:29:47.021Z [INFO]  server-16af2563-94ff-cd32-6c94-112692f1aa3d: snapshot complete up to: index=102
2025-01-25T14:29:47.035Z [ERROR] server-16af2563-94ff-cd32-6c94-112692f1aa3d: failed to heartbeat to: peer=3ac21012-71e3-68a6-a999-f85fbd9242ad backoff time=10ms error="failed to connect to peer: 3ac21012-71e3-68a6-a999-f85fbd9242ad"
2025-01-25T14:29:47.038Z [ERROR] server-16af2563-94ff-cd32-6c94-112692f1aa3d: failed to get log: index=2 error="log not found"
    raft_test.go:1690: [INFO] Finished apply without behind follower
2025-01-25T14:29:47.041Z [INFO]  creating new snapshot: path=/tmp/raft2386578095/snapshots/2-102-1737815387041.tmp
2025-01-25T14:29:47.041Z [INFO]  server-3ac21012-71e3-68a6-a999-f85fbd9242ad: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-01-25T14:29:47.043Z [INFO]  server-3ac21012-71e3-68a6-a999-f85fbd9242ad: copied to local snapshot: bytes=693
2025-01-25T14:29:47.043Z [INFO]  server-3ac21012-71e3-68a6-a999-f85fbd9242ad: snapshot restore progress: id=2-102-1737815387041 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-01-25T14:29:47.044Z [INFO]  server-3ac21012-71e3-68a6-a999-f85fbd9242ad: Installed remote snapshot
2025-01-25T14:29:47.044Z [INFO]  server-16af2563-94ff-cd32-6c94-112692f1aa3d: pipelining replication: peer="{Voter server-3ac21012-71e3-68a6-a999-f85fbd9242ad 3ac21012-71e3-68a6-a999-f85fbd9242ad}"
2025-01-25T14:29:47.052Z [INFO]  server-16af2563-94ff-cd32-6c94-112692f1aa3d: aborting pipeline replication: peer="{Voter server-c62c5dcf-3dac-4404-82e6-619ff7dba9f8 c62c5dcf-3dac-4404-82e6-619ff7dba9f8}"
2025-01-25T14:29:47.052Z [INFO]  server-16af2563-94ff-cd32-6c94-112692f1aa3d: aborting pipeline replication: peer="{Voter server-3ac21012-71e3-68a6-a999-f85fbd9242ad 3ac21012-71e3-68a6-a999-f85fbd9242ad}"
--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.32s)
=== RUN   TestRaft_ReJoinFollower
2025-01-25T14:29:47.054Z [INFO]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8d59acf-d76b-52ee-086a-cdd42ce1445f Address:a8d59acf-d76b-52ee-086a-cdd42ce1445f} {Suffrage:Voter ID:server-e2404cea-3363-8a08-25b1-5ab57c105b86 Address:e2404cea-3363-8a08-25b1-5ab57c105b86} {Suffrage:Voter ID:server-4c6a32da-d560-903c-980f-ddd2a15bfa6d Address:4c6a32da-d560-903c-980f-ddd2a15bfa6d}]"
2025-01-25T14:29:47.055Z [INFO]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: entering follower state: follower="Node at a8d59acf-d76b-52ee-086a-cdd42ce1445f [Follower]" leader-address= leader-id=
2025-01-25T14:29:47.055Z [INFO]  server-e2404cea-3363-8a08-25b1-5ab57c105b86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8d59acf-d76b-52ee-086a-cdd42ce1445f Address:a8d59acf-d76b-52ee-086a-cdd42ce1445f} {Suffrage:Voter ID:server-e2404cea-3363-8a08-25b1-5ab57c105b86 Address:e2404cea-3363-8a08-25b1-5ab57c105b86} {Suffrage:Voter ID:server-4c6a32da-d560-903c-980f-ddd2a15bfa6d Address:4c6a32da-d560-903c-980f-ddd2a15bfa6d}]"
2025-01-25T14:29:47.055Z [INFO]  server-e2404cea-3363-8a08-25b1-5ab57c105b86: entering follower state: follower="Node at e2404cea-3363-8a08-25b1-5ab57c105b86 [Follower]" leader-address= leader-id=
2025-01-25T14:29:47.055Z [INFO]  server-4c6a32da-d560-903c-980f-ddd2a15bfa6d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8d59acf-d76b-52ee-086a-cdd42ce1445f Address:a8d59acf-d76b-52ee-086a-cdd42ce1445f} {Suffrage:Voter ID:server-e2404cea-3363-8a08-25b1-5ab57c105b86 Address:e2404cea-3363-8a08-25b1-5ab57c105b86} {Suffrage:Voter ID:server-4c6a32da-d560-903c-980f-ddd2a15bfa6d Address:4c6a32da-d560-903c-980f-ddd2a15bfa6d}]"
2025-01-25T14:29:47.055Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:47.055Z [INFO]  server-4c6a32da-d560-903c-980f-ddd2a15bfa6d: entering follower state: follower="Node at 4c6a32da-d560-903c-980f-ddd2a15bfa6d [Follower]" leader-address= leader-id=
2025-01-25T14:29:47.108Z [WARN]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:47.108Z [INFO]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: entering candidate state: node="Node at a8d59acf-d76b-52ee-086a-cdd42ce1445f [Candidate]" term=2
2025-01-25T14:29:47.108Z [INFO]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: election won: term=2 tally=2
2025-01-25T14:29:47.108Z [INFO]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: entering leader state: leader="Node at a8d59acf-d76b-52ee-086a-cdd42ce1445f [Leader]"
2025-01-25T14:29:47.108Z [INFO]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: added peer, starting replication: peer=server-e2404cea-3363-8a08-25b1-5ab57c105b86
2025-01-25T14:29:47.108Z [INFO]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: added peer, starting replication: peer=server-4c6a32da-d560-903c-980f-ddd2a15bfa6d
2025-01-25T14:29:47.109Z [INFO]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: pipelining replication: peer="{Voter server-4c6a32da-d560-903c-980f-ddd2a15bfa6d 4c6a32da-d560-903c-980f-ddd2a15bfa6d}"
2025-01-25T14:29:47.109Z [INFO]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: pipelining replication: peer="{Voter server-e2404cea-3363-8a08-25b1-5ab57c105b86 e2404cea-3363-8a08-25b1-5ab57c105b86}"
2025-01-25T14:29:47.215Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:47.108916186 +0000 UTC m=+96.963241168 (1 nodes), 53.022492ms from start of poll, 54.104773ms from cluster start. Timeout at 2025-01-25 14:29:47.213931666 +0000 UTC m=+97.068256608, 105.01544ms after stability
2025-01-25T14:29:47.220Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:47.327Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:47.220946715 +0000 UTC m=+97.075271697 (2 nodes), 2.24µs from start of poll, 166.135302ms from cluster start. Timeout at 2025-01-25 14:29:47.326166147 +0000 UTC m=+97.180491129, 105.219432ms after stability
2025-01-25T14:29:47.327Z [INFO]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: updating configuration: command=RemoveServer server-id=server-e2404cea-3363-8a08-25b1-5ab57c105b86 server-addr= servers="[{Suffrage:Voter ID:server-a8d59acf-d76b-52ee-086a-cdd42ce1445f Address:a8d59acf-d76b-52ee-086a-cdd42ce1445f} {Suffrage:Voter ID:server-4c6a32da-d560-903c-980f-ddd2a15bfa6d Address:4c6a32da-d560-903c-980f-ddd2a15bfa6d}]"
2025-01-25T14:29:47.327Z [INFO]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: removed peer, stopping replication: peer=server-e2404cea-3363-8a08-25b1-5ab57c105b86 last-index=3
2025-01-25T14:29:47.327Z [INFO]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: aborting pipeline replication: peer="{Voter server-e2404cea-3363-8a08-25b1-5ab57c105b86 e2404cea-3363-8a08-25b1-5ab57c105b86}"
2025-01-25T14:29:47.433Z [INFO]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: updating configuration: command=AddVoter server-id=server-e2404cea-3363-8a08-25b1-5ab57c105b86 server-addr=e2404cea-3363-8a08-25b1-5ab57c105b86 servers="[{Suffrage:Voter ID:server-a8d59acf-d76b-52ee-086a-cdd42ce1445f Address:a8d59acf-d76b-52ee-086a-cdd42ce1445f} {Suffrage:Voter ID:server-4c6a32da-d560-903c-980f-ddd2a15bfa6d Address:4c6a32da-d560-903c-980f-ddd2a15bfa6d} {Suffrage:Voter ID:server-e2404cea-3363-8a08-25b1-5ab57c105b86 Address:e2404cea-3363-8a08-25b1-5ab57c105b86}]"
2025-01-25T14:29:47.433Z [INFO]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: added peer, starting replication: peer=server-e2404cea-3363-8a08-25b1-5ab57c105b86
2025-01-25T14:29:47.433Z [WARN]  server-e2404cea-3363-8a08-25b1-5ab57c105b86: failed to get previous log: previous-index=4 last-index=3 error="log not found"
2025-01-25T14:29:47.434Z [WARN]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: appendEntries rejected, sending older logs: peer="{Voter server-e2404cea-3363-8a08-25b1-5ab57c105b86 e2404cea-3363-8a08-25b1-5ab57c105b86}" next=4
2025-01-25T14:29:47.434Z [INFO]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: pipelining replication: peer="{Voter server-e2404cea-3363-8a08-25b1-5ab57c105b86 e2404cea-3363-8a08-25b1-5ab57c105b86}"
2025-01-25T14:29:47.434Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:47.540Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:47.434352795 +0000 UTC m=+97.288677777 (1 nodes), 1.12µs from start of poll, 379.541382ms from cluster start. Timeout at 2025-01-25 14:29:47.539541948 +0000 UTC m=+97.393866930, 105.189153ms after stability
2025-01-25T14:29:47.540Z [INFO]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: aborting pipeline replication: peer="{Voter server-4c6a32da-d560-903c-980f-ddd2a15bfa6d 4c6a32da-d560-903c-980f-ddd2a15bfa6d}"
2025-01-25T14:29:47.540Z [INFO]  server-a8d59acf-d76b-52ee-086a-cdd42ce1445f: aborting pipeline replication: peer="{Voter server-e2404cea-3363-8a08-25b1-5ab57c105b86 e2404cea-3363-8a08-25b1-5ab57c105b86}"
--- PASS: TestRaft_ReJoinFollower (0.49s)
=== RUN   TestRaft_LeaderLeaseExpire
2025-01-25T14:29:47.542Z [INFO]  server-25ccac42-f31d-35a1-5318-6bc169312cec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25ccac42-f31d-35a1-5318-6bc169312cec Address:25ccac42-f31d-35a1-5318-6bc169312cec} {Suffrage:Voter ID:server-68c8e1c7-8678-32c2-504d-98f1683e16c6 Address:68c8e1c7-8678-32c2-504d-98f1683e16c6}]"
2025-01-25T14:29:47.542Z [INFO]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25ccac42-f31d-35a1-5318-6bc169312cec Address:25ccac42-f31d-35a1-5318-6bc169312cec} {Suffrage:Voter ID:server-68c8e1c7-8678-32c2-504d-98f1683e16c6 Address:68c8e1c7-8678-32c2-504d-98f1683e16c6}]"
2025-01-25T14:29:47.542Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:47.542Z [INFO]  server-25ccac42-f31d-35a1-5318-6bc169312cec: entering follower state: follower="Node at 25ccac42-f31d-35a1-5318-6bc169312cec [Follower]" leader-address= leader-id=
2025-01-25T14:29:47.542Z [INFO]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: entering follower state: follower="Node at 68c8e1c7-8678-32c2-504d-98f1683e16c6 [Follower]" leader-address= leader-id=
2025-01-25T14:29:47.606Z [WARN]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:47.606Z [INFO]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: entering candidate state: node="Node at 68c8e1c7-8678-32c2-504d-98f1683e16c6 [Candidate]" term=2
2025-01-25T14:29:47.606Z [INFO]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: election won: term=2 tally=2
2025-01-25T14:29:47.606Z [INFO]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: entering leader state: leader="Node at 68c8e1c7-8678-32c2-504d-98f1683e16c6 [Leader]"
2025-01-25T14:29:47.606Z [INFO]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: added peer, starting replication: peer=server-25ccac42-f31d-35a1-5318-6bc169312cec
2025-01-25T14:29:47.606Z [INFO]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: pipelining replication: peer="{Voter server-25ccac42-f31d-35a1-5318-6bc169312cec 25ccac42-f31d-35a1-5318-6bc169312cec}"
2025-01-25T14:29:47.711Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:47.606626155 +0000 UTC m=+97.460951137 (1 nodes), 63.870781ms from start of poll, 64.359084ms from cluster start. Timeout at 2025-01-25 14:29:47.711628714 +0000 UTC m=+97.565953696, 105.002559ms after stability
2025-01-25T14:29:47.717Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:47.822Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:47.717510205 +0000 UTC m=+97.571835187 (1 nodes), 1.76µs from start of poll, 175.243134ms from cluster start. Timeout at 2025-01-25 14:29:47.822705478 +0000 UTC m=+97.677030500, 105.195313ms after stability
    raft_test.go:1799: [INFO] Disconnecting Node at 25ccac42-f31d-35a1-5318-6bc169312cec [Follower]
2025-01-25T14:29:47.826Z [ERROR] server-68c8e1c7-8678-32c2-504d-98f1683e16c6: failed to pipeline appendEntries: peer="{Voter server-25ccac42-f31d-35a1-5318-6bc169312cec 25ccac42-f31d-35a1-5318-6bc169312cec}" error="append pipeline closed"
2025-01-25T14:29:47.826Z [INFO]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: aborting pipeline replication: peer="{Voter server-25ccac42-f31d-35a1-5318-6bc169312cec 25ccac42-f31d-35a1-5318-6bc169312cec}"
2025-01-25T14:29:47.829Z [ERROR] server-68c8e1c7-8678-32c2-504d-98f1683e16c6: failed to heartbeat to: peer=25ccac42-f31d-35a1-5318-6bc169312cec backoff time=10ms error="failed to connect to peer: 25ccac42-f31d-35a1-5318-6bc169312cec"
2025-01-25T14:29:47.832Z [ERROR] server-68c8e1c7-8678-32c2-504d-98f1683e16c6: failed to appendEntries to: peer="{Voter server-25ccac42-f31d-35a1-5318-6bc169312cec 25ccac42-f31d-35a1-5318-6bc169312cec}" error="failed to connect to peer: 25ccac42-f31d-35a1-5318-6bc169312cec"
2025-01-25T14:29:47.848Z [ERROR] server-68c8e1c7-8678-32c2-504d-98f1683e16c6: failed to heartbeat to: peer=25ccac42-f31d-35a1-5318-6bc169312cec backoff time=10ms error="failed to connect to peer: 25ccac42-f31d-35a1-5318-6bc169312cec"
2025-01-25T14:29:47.851Z [ERROR] server-68c8e1c7-8678-32c2-504d-98f1683e16c6: failed to appendEntries to: peer="{Voter server-25ccac42-f31d-35a1-5318-6bc169312cec 25ccac42-f31d-35a1-5318-6bc169312cec}" error="failed to connect to peer: 25ccac42-f31d-35a1-5318-6bc169312cec"
2025-01-25T14:29:47.865Z [ERROR] server-68c8e1c7-8678-32c2-504d-98f1683e16c6: failed to heartbeat to: peer=25ccac42-f31d-35a1-5318-6bc169312cec backoff time=10ms error="failed to connect to peer: 25ccac42-f31d-35a1-5318-6bc169312cec"
2025-01-25T14:29:47.867Z [ERROR] server-68c8e1c7-8678-32c2-504d-98f1683e16c6: failed to appendEntries to: peer="{Voter server-25ccac42-f31d-35a1-5318-6bc169312cec 25ccac42-f31d-35a1-5318-6bc169312cec}" error="failed to connect to peer: 25ccac42-f31d-35a1-5318-6bc169312cec"
2025-01-25T14:29:47.872Z [WARN]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: failed to contact: server-id=server-25ccac42-f31d-35a1-5318-6bc169312cec time=50.499982ms
2025-01-25T14:29:47.872Z [WARN]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: failed to contact quorum of nodes, stepping down
2025-01-25T14:29:47.872Z [INFO]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: entering follower state: follower="Node at 68c8e1c7-8678-32c2-504d-98f1683e16c6 [Follower]" leader-address= leader-id=
2025-01-25T14:29:47.872Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:47.876Z [WARN]  server-25ccac42-f31d-35a1-5318-6bc169312cec: heartbeat timeout reached, starting election: last-leader-addr=68c8e1c7-8678-32c2-504d-98f1683e16c6 last-leader-id=server-68c8e1c7-8678-32c2-504d-98f1683e16c6
2025-01-25T14:29:47.876Z [INFO]  server-25ccac42-f31d-35a1-5318-6bc169312cec: entering candidate state: node="Node at 25ccac42-f31d-35a1-5318-6bc169312cec [Candidate]" term=3
2025-01-25T14:29:47.876Z [ERROR] server-25ccac42-f31d-35a1-5318-6bc169312cec: failed to make requestVote RPC: target="{Voter server-68c8e1c7-8678-32c2-504d-98f1683e16c6 68c8e1c7-8678-32c2-504d-98f1683e16c6}" error="failed to connect to peer: 68c8e1c7-8678-32c2-504d-98f1683e16c6" term=3
2025-01-25T14:29:47.939Z [WARN]  server-25ccac42-f31d-35a1-5318-6bc169312cec: Election timeout reached, restarting election
2025-01-25T14:29:47.939Z [INFO]  server-25ccac42-f31d-35a1-5318-6bc169312cec: entering candidate state: node="Node at 25ccac42-f31d-35a1-5318-6bc169312cec [Candidate]" term=4
2025-01-25T14:29:47.939Z [ERROR] server-25ccac42-f31d-35a1-5318-6bc169312cec: failed to make requestVote RPC: target="{Voter server-68c8e1c7-8678-32c2-504d-98f1683e16c6 68c8e1c7-8678-32c2-504d-98f1683e16c6}" error="failed to connect to peer: 68c8e1c7-8678-32c2-504d-98f1683e16c6" term=4
2025-01-25T14:29:47.942Z [WARN]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:47.942Z [INFO]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: entering candidate state: node="Node at 68c8e1c7-8678-32c2-504d-98f1683e16c6 [Candidate]" term=3
2025-01-25T14:29:47.942Z [ERROR] server-68c8e1c7-8678-32c2-504d-98f1683e16c6: failed to make requestVote RPC: target="{Voter server-25ccac42-f31d-35a1-5318-6bc169312cec 25ccac42-f31d-35a1-5318-6bc169312cec}" error="failed to connect to peer: 25ccac42-f31d-35a1-5318-6bc169312cec" term=3
2025-01-25T14:29:47.993Z [WARN]  server-25ccac42-f31d-35a1-5318-6bc169312cec: Election timeout reached, restarting election
2025-01-25T14:29:47.993Z [INFO]  server-25ccac42-f31d-35a1-5318-6bc169312cec: entering candidate state: node="Node at 25ccac42-f31d-35a1-5318-6bc169312cec [Candidate]" term=5
2025-01-25T14:29:47.993Z [ERROR] server-25ccac42-f31d-35a1-5318-6bc169312cec: failed to make requestVote RPC: target="{Voter server-68c8e1c7-8678-32c2-504d-98f1683e16c6 68c8e1c7-8678-32c2-504d-98f1683e16c6}" error="failed to connect to peer: 68c8e1c7-8678-32c2-504d-98f1683e16c6" term=5
2025-01-25T14:29:48.031Z [WARN]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: Election timeout reached, restarting election
2025-01-25T14:29:48.031Z [INFO]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: entering candidate state: node="Node at 68c8e1c7-8678-32c2-504d-98f1683e16c6 [Candidate]" term=4
2025-01-25T14:29:48.031Z [ERROR] server-68c8e1c7-8678-32c2-504d-98f1683e16c6: failed to make requestVote RPC: target="{Voter server-25ccac42-f31d-35a1-5318-6bc169312cec 25ccac42-f31d-35a1-5318-6bc169312cec}" error="failed to connect to peer: 25ccac42-f31d-35a1-5318-6bc169312cec" term=4
2025-01-25T14:29:48.048Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:47.942665622 +0000 UTC m=+97.796990604 (0 nodes), 69.835007ms from start of poll, 400.398551ms from cluster start. Timeout at 2025-01-25 14:29:48.047668902 +0000 UTC m=+97.901993884, 105.00328ms after stability
2025-01-25T14:29:48.077Z [WARN]  server-25ccac42-f31d-35a1-5318-6bc169312cec: Election timeout reached, restarting election
2025-01-25T14:29:48.077Z [INFO]  server-25ccac42-f31d-35a1-5318-6bc169312cec: entering candidate state: node="Node at 25ccac42-f31d-35a1-5318-6bc169312cec [Candidate]" term=6
2025-01-25T14:29:48.077Z [ERROR] server-25ccac42-f31d-35a1-5318-6bc169312cec: failed to make requestVote RPC: target="{Voter server-68c8e1c7-8678-32c2-504d-98f1683e16c6 68c8e1c7-8678-32c2-504d-98f1683e16c6}" error="failed to connect to peer: 68c8e1c7-8678-32c2-504d-98f1683e16c6" term=6
2025-01-25T14:29:48.092Z [WARN]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: Election timeout reached, restarting election
2025-01-25T14:29:48.092Z [INFO]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: entering candidate state: node="Node at 68c8e1c7-8678-32c2-504d-98f1683e16c6 [Candidate]" term=5
2025-01-25T14:29:48.092Z [ERROR] server-68c8e1c7-8678-32c2-504d-98f1683e16c6: failed to make requestVote RPC: target="{Voter server-25ccac42-f31d-35a1-5318-6bc169312cec 25ccac42-f31d-35a1-5318-6bc169312cec}" error="failed to connect to peer: 25ccac42-f31d-35a1-5318-6bc169312cec" term=5
2025-01-25T14:29:48.149Z [WARN]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: Election timeout reached, restarting election
2025-01-25T14:29:48.149Z [INFO]  server-68c8e1c7-8678-32c2-504d-98f1683e16c6: entering candidate state: node="Node at 68c8e1c7-8678-32c2-504d-98f1683e16c6 [Candidate]" term=6
2025-01-25T14:29:48.149Z [ERROR] server-68c8e1c7-8678-32c2-504d-98f1683e16c6: failed to make requestVote RPC: target="{Voter server-25ccac42-f31d-35a1-5318-6bc169312cec 25ccac42-f31d-35a1-5318-6bc169312cec}" error="failed to connect to peer: 25ccac42-f31d-35a1-5318-6bc169312cec" term=6
--- PASS: TestRaft_LeaderLeaseExpire (0.61s)
=== RUN   TestRaft_Barrier
2025-01-25T14:29:48.156Z [INFO]  server-141d2975-5452-45bc-1596-0b8528b97cf2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-141d2975-5452-45bc-1596-0b8528b97cf2 Address:141d2975-5452-45bc-1596-0b8528b97cf2} {Suffrage:Voter ID:server-a992ec2a-25c9-0cec-a38f-b3ac52a83f0d Address:a992ec2a-25c9-0cec-a38f-b3ac52a83f0d} {Suffrage:Voter ID:server-4562e26a-b643-1f1f-c30c-50cb7a04bae2 Address:4562e26a-b643-1f1f-c30c-50cb7a04bae2}]"
2025-01-25T14:29:48.156Z [INFO]  server-a992ec2a-25c9-0cec-a38f-b3ac52a83f0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-141d2975-5452-45bc-1596-0b8528b97cf2 Address:141d2975-5452-45bc-1596-0b8528b97cf2} {Suffrage:Voter ID:server-a992ec2a-25c9-0cec-a38f-b3ac52a83f0d Address:a992ec2a-25c9-0cec-a38f-b3ac52a83f0d} {Suffrage:Voter ID:server-4562e26a-b643-1f1f-c30c-50cb7a04bae2 Address:4562e26a-b643-1f1f-c30c-50cb7a04bae2}]"
2025-01-25T14:29:48.157Z [INFO]  server-4562e26a-b643-1f1f-c30c-50cb7a04bae2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-141d2975-5452-45bc-1596-0b8528b97cf2 Address:141d2975-5452-45bc-1596-0b8528b97cf2} {Suffrage:Voter ID:server-a992ec2a-25c9-0cec-a38f-b3ac52a83f0d Address:a992ec2a-25c9-0cec-a38f-b3ac52a83f0d} {Suffrage:Voter ID:server-4562e26a-b643-1f1f-c30c-50cb7a04bae2 Address:4562e26a-b643-1f1f-c30c-50cb7a04bae2}]"
2025-01-25T14:29:48.157Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:48.157Z [INFO]  server-a992ec2a-25c9-0cec-a38f-b3ac52a83f0d: entering follower state: follower="Node at a992ec2a-25c9-0cec-a38f-b3ac52a83f0d [Follower]" leader-address= leader-id=
2025-01-25T14:29:48.157Z [INFO]  server-141d2975-5452-45bc-1596-0b8528b97cf2: entering follower state: follower="Node at 141d2975-5452-45bc-1596-0b8528b97cf2 [Follower]" leader-address= leader-id=
2025-01-25T14:29:48.157Z [INFO]  server-4562e26a-b643-1f1f-c30c-50cb7a04bae2: entering follower state: follower="Node at 4562e26a-b643-1f1f-c30c-50cb7a04bae2 [Follower]" leader-address= leader-id=
2025-01-25T14:29:48.221Z [WARN]  server-4562e26a-b643-1f1f-c30c-50cb7a04bae2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:48.221Z [INFO]  server-4562e26a-b643-1f1f-c30c-50cb7a04bae2: entering candidate state: node="Node at 4562e26a-b643-1f1f-c30c-50cb7a04bae2 [Candidate]" term=2
2025-01-25T14:29:48.221Z [INFO]  server-4562e26a-b643-1f1f-c30c-50cb7a04bae2: election won: term=2 tally=2
2025-01-25T14:29:48.222Z [INFO]  server-4562e26a-b643-1f1f-c30c-50cb7a04bae2: entering leader state: leader="Node at 4562e26a-b643-1f1f-c30c-50cb7a04bae2 [Leader]"
2025-01-25T14:29:48.222Z [INFO]  server-4562e26a-b643-1f1f-c30c-50cb7a04bae2: added peer, starting replication: peer=server-141d2975-5452-45bc-1596-0b8528b97cf2
2025-01-25T14:29:48.222Z [INFO]  server-4562e26a-b643-1f1f-c30c-50cb7a04bae2: added peer, starting replication: peer=server-a992ec2a-25c9-0cec-a38f-b3ac52a83f0d
2025-01-25T14:29:48.222Z [INFO]  server-4562e26a-b643-1f1f-c30c-50cb7a04bae2: pipelining replication: peer="{Voter server-a992ec2a-25c9-0cec-a38f-b3ac52a83f0d a992ec2a-25c9-0cec-a38f-b3ac52a83f0d}"
2025-01-25T14:29:48.222Z [INFO]  server-4562e26a-b643-1f1f-c30c-50cb7a04bae2: pipelining replication: peer="{Voter server-141d2975-5452-45bc-1596-0b8528b97cf2 141d2975-5452-45bc-1596-0b8528b97cf2}"
2025-01-25T14:29:48.327Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:48.222125405 +0000 UTC m=+98.076450387 (1 nodes), 64.920424ms from start of poll, 65.563841ms from cluster start. Timeout at 2025-01-25 14:29:48.327130124 +0000 UTC m=+98.181455106, 105.004719ms after stability
2025-01-25T14:29:48.352Z [INFO]  server-4562e26a-b643-1f1f-c30c-50cb7a04bae2: aborting pipeline replication: peer="{Voter server-a992ec2a-25c9-0cec-a38f-b3ac52a83f0d a992ec2a-25c9-0cec-a38f-b3ac52a83f0d}"
2025-01-25T14:29:48.352Z [INFO]  server-4562e26a-b643-1f1f-c30c-50cb7a04bae2: aborting pipeline replication: peer="{Voter server-141d2975-5452-45bc-1596-0b8528b97cf2 141d2975-5452-45bc-1596-0b8528b97cf2}"
--- PASS: TestRaft_Barrier (0.20s)
=== RUN   TestRaft_VerifyLeader
2025-01-25T14:29:48.355Z [INFO]  server-b40c6640-37f9-2e08-55ba-4e899981f78e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b40c6640-37f9-2e08-55ba-4e899981f78e Address:b40c6640-37f9-2e08-55ba-4e899981f78e} {Suffrage:Voter ID:server-cf4885b8-2887-6f72-3406-23e40561e3aa Address:cf4885b8-2887-6f72-3406-23e40561e3aa} {Suffrage:Voter ID:server-8de764b5-dd07-04dc-34c0-cdf4de098bf0 Address:8de764b5-dd07-04dc-34c0-cdf4de098bf0}]"
2025-01-25T14:29:48.355Z [INFO]  server-b40c6640-37f9-2e08-55ba-4e899981f78e: entering follower state: follower="Node at b40c6640-37f9-2e08-55ba-4e899981f78e [Follower]" leader-address= leader-id=
2025-01-25T14:29:48.355Z [INFO]  server-cf4885b8-2887-6f72-3406-23e40561e3aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b40c6640-37f9-2e08-55ba-4e899981f78e Address:b40c6640-37f9-2e08-55ba-4e899981f78e} {Suffrage:Voter ID:server-cf4885b8-2887-6f72-3406-23e40561e3aa Address:cf4885b8-2887-6f72-3406-23e40561e3aa} {Suffrage:Voter ID:server-8de764b5-dd07-04dc-34c0-cdf4de098bf0 Address:8de764b5-dd07-04dc-34c0-cdf4de098bf0}]"
2025-01-25T14:29:48.355Z [INFO]  server-cf4885b8-2887-6f72-3406-23e40561e3aa: entering follower state: follower="Node at cf4885b8-2887-6f72-3406-23e40561e3aa [Follower]" leader-address= leader-id=
2025-01-25T14:29:48.356Z [INFO]  server-8de764b5-dd07-04dc-34c0-cdf4de098bf0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b40c6640-37f9-2e08-55ba-4e899981f78e Address:b40c6640-37f9-2e08-55ba-4e899981f78e} {Suffrage:Voter ID:server-cf4885b8-2887-6f72-3406-23e40561e3aa Address:cf4885b8-2887-6f72-3406-23e40561e3aa} {Suffrage:Voter ID:server-8de764b5-dd07-04dc-34c0-cdf4de098bf0 Address:8de764b5-dd07-04dc-34c0-cdf4de098bf0}]"
2025-01-25T14:29:48.356Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:48.356Z [INFO]  server-8de764b5-dd07-04dc-34c0-cdf4de098bf0: entering follower state: follower="Node at 8de764b5-dd07-04dc-34c0-cdf4de098bf0 [Follower]" leader-address= leader-id=
2025-01-25T14:29:48.430Z [WARN]  server-b40c6640-37f9-2e08-55ba-4e899981f78e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:48.430Z [INFO]  server-b40c6640-37f9-2e08-55ba-4e899981f78e: entering candidate state: node="Node at b40c6640-37f9-2e08-55ba-4e899981f78e [Candidate]" term=2
2025-01-25T14:29:48.431Z [INFO]  server-b40c6640-37f9-2e08-55ba-4e899981f78e: election won: term=2 tally=2
2025-01-25T14:29:48.431Z [INFO]  server-b40c6640-37f9-2e08-55ba-4e899981f78e: entering leader state: leader="Node at b40c6640-37f9-2e08-55ba-4e899981f78e [Leader]"
2025-01-25T14:29:48.431Z [INFO]  server-b40c6640-37f9-2e08-55ba-4e899981f78e: added peer, starting replication: peer=server-cf4885b8-2887-6f72-3406-23e40561e3aa
2025-01-25T14:29:48.431Z [INFO]  server-b40c6640-37f9-2e08-55ba-4e899981f78e: added peer, starting replication: peer=server-8de764b5-dd07-04dc-34c0-cdf4de098bf0
2025-01-25T14:29:48.431Z [WARN]  server-8de764b5-dd07-04dc-34c0-cdf4de098bf0: rejecting vote request since our last term is greater: candidate=b40c6640-37f9-2e08-55ba-4e899981f78e last-term=2 last-candidate-term=1
2025-01-25T14:29:48.431Z [INFO]  server-b40c6640-37f9-2e08-55ba-4e899981f78e: pipelining replication: peer="{Voter server-cf4885b8-2887-6f72-3406-23e40561e3aa cf4885b8-2887-6f72-3406-23e40561e3aa}"
2025-01-25T14:29:48.431Z [INFO]  server-b40c6640-37f9-2e08-55ba-4e899981f78e: pipelining replication: peer="{Voter server-8de764b5-dd07-04dc-34c0-cdf4de098bf0 8de764b5-dd07-04dc-34c0-cdf4de098bf0}"
2025-01-25T14:29:48.537Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:48.431617746 +0000 UTC m=+98.285942688 (1 nodes), 75.504203ms from start of poll, 76.262656ms from cluster start. Timeout at 2025-01-25 14:29:48.536630746 +0000 UTC m=+98.390955688, 105.013ms after stability
2025-01-25T14:29:48.537Z [INFO]  server-b40c6640-37f9-2e08-55ba-4e899981f78e: aborting pipeline replication: peer="{Voter server-8de764b5-dd07-04dc-34c0-cdf4de098bf0 8de764b5-dd07-04dc-34c0-cdf4de098bf0}"
2025-01-25T14:29:48.537Z [INFO]  server-b40c6640-37f9-2e08-55ba-4e899981f78e: aborting pipeline replication: peer="{Voter server-cf4885b8-2887-6f72-3406-23e40561e3aa cf4885b8-2887-6f72-3406-23e40561e3aa}"
--- PASS: TestRaft_VerifyLeader (0.18s)
=== RUN   TestRaft_VerifyLeader_Single
2025-01-25T14:29:48.539Z [INFO]  server-9d692c05-8179-c61c-5f9b-441a1663aa5b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d692c05-8179-c61c-5f9b-441a1663aa5b Address:9d692c05-8179-c61c-5f9b-441a1663aa5b}]"
2025-01-25T14:29:48.539Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:48.539Z [INFO]  server-9d692c05-8179-c61c-5f9b-441a1663aa5b: entering follower state: follower="Node at 9d692c05-8179-c61c-5f9b-441a1663aa5b [Follower]" leader-address= leader-id=
2025-01-25T14:29:48.602Z [WARN]  server-9d692c05-8179-c61c-5f9b-441a1663aa5b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:48.602Z [INFO]  server-9d692c05-8179-c61c-5f9b-441a1663aa5b: entering candidate state: node="Node at 9d692c05-8179-c61c-5f9b-441a1663aa5b [Candidate]" term=2
2025-01-25T14:29:48.602Z [INFO]  server-9d692c05-8179-c61c-5f9b-441a1663aa5b: election won: term=2 tally=1
2025-01-25T14:29:48.602Z [INFO]  server-9d692c05-8179-c61c-5f9b-441a1663aa5b: entering leader state: leader="Node at 9d692c05-8179-c61c-5f9b-441a1663aa5b [Leader]"
2025-01-25T14:29:48.707Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:48.602275284 +0000 UTC m=+98.456600266 (1 nodes), 62.654505ms from start of poll, 63.34016ms from cluster start. Timeout at 2025-01-25 14:29:48.707282804 +0000 UTC m=+98.561607786, 105.00752ms after stability
--- PASS: TestRaft_VerifyLeader_Single (0.17s)
=== RUN   TestRaft_VerifyLeader_Fail
2025-01-25T14:29:48.710Z [INFO]  server-a3c10448-efa2-4c57-99b1-abb6032f5eb4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3c10448-efa2-4c57-99b1-abb6032f5eb4 Address:a3c10448-efa2-4c57-99b1-abb6032f5eb4} {Suffrage:Voter ID:server-f1ee4f06-f9bc-d6e0-713a-7423eb2722d0 Address:f1ee4f06-f9bc-d6e0-713a-7423eb2722d0}]"
2025-01-25T14:29:48.710Z [INFO]  server-f1ee4f06-f9bc-d6e0-713a-7423eb2722d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3c10448-efa2-4c57-99b1-abb6032f5eb4 Address:a3c10448-efa2-4c57-99b1-abb6032f5eb4} {Suffrage:Voter ID:server-f1ee4f06-f9bc-d6e0-713a-7423eb2722d0 Address:f1ee4f06-f9bc-d6e0-713a-7423eb2722d0}]"
2025-01-25T14:29:48.710Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:48.710Z [INFO]  server-a3c10448-efa2-4c57-99b1-abb6032f5eb4: entering follower state: follower="Node at a3c10448-efa2-4c57-99b1-abb6032f5eb4 [Follower]" leader-address= leader-id=
2025-01-25T14:29:48.710Z [INFO]  server-f1ee4f06-f9bc-d6e0-713a-7423eb2722d0: entering follower state: follower="Node at f1ee4f06-f9bc-d6e0-713a-7423eb2722d0 [Follower]" leader-address= leader-id=
2025-01-25T14:29:48.764Z [WARN]  server-f1ee4f06-f9bc-d6e0-713a-7423eb2722d0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:48.764Z [INFO]  server-f1ee4f06-f9bc-d6e0-713a-7423eb2722d0: entering candidate state: node="Node at f1ee4f06-f9bc-d6e0-713a-7423eb2722d0 [Candidate]" term=2
2025-01-25T14:29:48.764Z [INFO]  server-f1ee4f06-f9bc-d6e0-713a-7423eb2722d0: election won: term=2 tally=2
2025-01-25T14:29:48.764Z [INFO]  server-f1ee4f06-f9bc-d6e0-713a-7423eb2722d0: entering leader state: leader="Node at f1ee4f06-f9bc-d6e0-713a-7423eb2722d0 [Leader]"
2025-01-25T14:29:48.764Z [INFO]  server-f1ee4f06-f9bc-d6e0-713a-7423eb2722d0: added peer, starting replication: peer=server-a3c10448-efa2-4c57-99b1-abb6032f5eb4
2025-01-25T14:29:48.765Z [INFO]  server-f1ee4f06-f9bc-d6e0-713a-7423eb2722d0: pipelining replication: peer="{Voter server-a3c10448-efa2-4c57-99b1-abb6032f5eb4 a3c10448-efa2-4c57-99b1-abb6032f5eb4}"
2025-01-25T14:29:48.870Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:48.765272698 +0000 UTC m=+98.619597680 (1 nodes), 54.868225ms from start of poll, 55.367727ms from cluster start. Timeout at 2025-01-25 14:29:48.870276578 +0000 UTC m=+98.724601560, 105.00388ms after stability
2025-01-25T14:29:48.870Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:48.976Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:48.87082918 +0000 UTC m=+98.725154162 (1 nodes), 1.32µs from start of poll, 160.924209ms from cluster start. Timeout at 2025-01-25 14:29:48.976021853 +0000 UTC m=+98.830346835, 105.192673ms after stability
2025-01-25T14:29:48.978Z [ERROR] server-f1ee4f06-f9bc-d6e0-713a-7423eb2722d0: peer has newer term, stopping replication: peer="{Voter server-a3c10448-efa2-4c57-99b1-abb6032f5eb4 a3c10448-efa2-4c57-99b1-abb6032f5eb4}"
2025-01-25T14:29:48.978Z [INFO]  server-f1ee4f06-f9bc-d6e0-713a-7423eb2722d0: aborting pipeline replication: peer="{Voter server-a3c10448-efa2-4c57-99b1-abb6032f5eb4 a3c10448-efa2-4c57-99b1-abb6032f5eb4}"
2025-01-25T14:29:48.978Z [INFO]  server-f1ee4f06-f9bc-d6e0-713a-7423eb2722d0: entering follower state: follower="Node at f1ee4f06-f9bc-d6e0-713a-7423eb2722d0 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VerifyLeader_Fail (0.27s)
=== RUN   TestRaft_VerifyLeader_PartialConnect
2025-01-25T14:29:48.981Z [INFO]  server-51eeb3cc-b3e1-5b59-cc9f-3886c06f4123: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-51eeb3cc-b3e1-5b59-cc9f-3886c06f4123 Address:51eeb3cc-b3e1-5b59-cc9f-3886c06f4123} {Suffrage:Voter ID:server-a78708f3-8e3e-b88e-73c1-51063f4f36cb Address:a78708f3-8e3e-b88e-73c1-51063f4f36cb} {Suffrage:Voter ID:server-7d8797c8-f722-9ede-6de0-0b4d6c7b1041 Address:7d8797c8-f722-9ede-6de0-0b4d6c7b1041}]"
2025-01-25T14:29:48.981Z [INFO]  server-a78708f3-8e3e-b88e-73c1-51063f4f36cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-51eeb3cc-b3e1-5b59-cc9f-3886c06f4123 Address:51eeb3cc-b3e1-5b59-cc9f-3886c06f4123} {Suffrage:Voter ID:server-a78708f3-8e3e-b88e-73c1-51063f4f36cb Address:a78708f3-8e3e-b88e-73c1-51063f4f36cb} {Suffrage:Voter ID:server-7d8797c8-f722-9ede-6de0-0b4d6c7b1041 Address:7d8797c8-f722-9ede-6de0-0b4d6c7b1041}]"
2025-01-25T14:29:48.981Z [INFO]  server-51eeb3cc-b3e1-5b59-cc9f-3886c06f4123: entering follower state: follower="Node at 51eeb3cc-b3e1-5b59-cc9f-3886c06f4123 [Follower]" leader-address= leader-id=
2025-01-25T14:29:48.981Z [INFO]  server-a78708f3-8e3e-b88e-73c1-51063f4f36cb: entering follower state: follower="Node at a78708f3-8e3e-b88e-73c1-51063f4f36cb [Follower]" leader-address= leader-id=
2025-01-25T14:29:48.981Z [INFO]  server-7d8797c8-f722-9ede-6de0-0b4d6c7b1041: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-51eeb3cc-b3e1-5b59-cc9f-3886c06f4123 Address:51eeb3cc-b3e1-5b59-cc9f-3886c06f4123} {Suffrage:Voter ID:server-a78708f3-8e3e-b88e-73c1-51063f4f36cb Address:a78708f3-8e3e-b88e-73c1-51063f4f36cb} {Suffrage:Voter ID:server-7d8797c8-f722-9ede-6de0-0b4d6c7b1041 Address:7d8797c8-f722-9ede-6de0-0b4d6c7b1041}]"
2025-01-25T14:29:48.981Z [INFO]  server-7d8797c8-f722-9ede-6de0-0b4d6c7b1041: entering follower state: follower="Node at 7d8797c8-f722-9ede-6de0-0b4d6c7b1041 [Follower]" leader-address= leader-id=
2025-01-25T14:29:48.981Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:49.051Z [WARN]  server-7d8797c8-f722-9ede-6de0-0b4d6c7b1041: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:49.052Z [INFO]  server-7d8797c8-f722-9ede-6de0-0b4d6c7b1041: entering candidate state: node="Node at 7d8797c8-f722-9ede-6de0-0b4d6c7b1041 [Candidate]" term=2
2025-01-25T14:29:49.052Z [INFO]  server-7d8797c8-f722-9ede-6de0-0b4d6c7b1041: election won: term=2 tally=2
2025-01-25T14:29:49.052Z [INFO]  server-7d8797c8-f722-9ede-6de0-0b4d6c7b1041: entering leader state: leader="Node at 7d8797c8-f722-9ede-6de0-0b4d6c7b1041 [Leader]"
2025-01-25T14:29:49.053Z [INFO]  server-7d8797c8-f722-9ede-6de0-0b4d6c7b1041: added peer, starting replication: peer=server-51eeb3cc-b3e1-5b59-cc9f-3886c06f4123
2025-01-25T14:29:49.053Z [INFO]  server-7d8797c8-f722-9ede-6de0-0b4d6c7b1041: added peer, starting replication: peer=server-a78708f3-8e3e-b88e-73c1-51063f4f36cb
2025-01-25T14:29:49.053Z [INFO]  server-7d8797c8-f722-9ede-6de0-0b4d6c7b1041: pipelining replication: peer="{Voter server-a78708f3-8e3e-b88e-73c1-51063f4f36cb a78708f3-8e3e-b88e-73c1-51063f4f36cb}"
2025-01-25T14:29:49.056Z [INFO]  server-7d8797c8-f722-9ede-6de0-0b4d6c7b1041: pipelining replication: peer="{Voter server-51eeb3cc-b3e1-5b59-cc9f-3886c06f4123 51eeb3cc-b3e1-5b59-cc9f-3886c06f4123}"
2025-01-25T14:29:49.160Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:49.055654528 +0000 UTC m=+98.909979510 (1 nodes), 73.707947ms from start of poll, 74.405282ms from cluster start. Timeout at 2025-01-25 14:29:49.160658688 +0000 UTC m=+99.014983670, 105.00416ms after stability
2025-01-25T14:29:49.166Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:49.272Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:49.16676305 +0000 UTC m=+99.021088032 (2 nodes), 1.96µs from start of poll, 185.513804ms from cluster start. Timeout at 2025-01-25 14:29:49.271973082 +0000 UTC m=+99.126298104, 105.210072ms after stability
    raft_test.go:1968: [INFO] Disconnecting Node at 51eeb3cc-b3e1-5b59-cc9f-3886c06f4123 [Follower]
2025-01-25T14:29:49.272Z [ERROR] server-7d8797c8-f722-9ede-6de0-0b4d6c7b1041: failed to heartbeat to: peer=51eeb3cc-b3e1-5b59-cc9f-3886c06f4123 backoff time=10ms error="failed to connect to peer: 51eeb3cc-b3e1-5b59-cc9f-3886c06f4123"
2025-01-25T14:29:49.273Z [INFO]  server-7d8797c8-f722-9ede-6de0-0b4d6c7b1041: aborting pipeline replication: peer="{Voter server-a78708f3-8e3e-b88e-73c1-51063f4f36cb a78708f3-8e3e-b88e-73c1-51063f4f36cb}"
2025-01-25T14:29:49.273Z [INFO]  server-7d8797c8-f722-9ede-6de0-0b4d6c7b1041: aborting pipeline replication: peer="{Voter server-51eeb3cc-b3e1-5b59-cc9f-3886c06f4123 51eeb3cc-b3e1-5b59-cc9f-3886c06f4123}"
--- PASS: TestRaft_VerifyLeader_PartialConnect (0.29s)
=== RUN   TestRaft_NotifyCh
2025-01-25T14:29:49.274Z [INFO]  server-d3105446-26dd-4d5c-a665-d43a0fb786a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3105446-26dd-4d5c-a665-d43a0fb786a2 Address:d3105446-26dd-4d5c-a665-d43a0fb786a2}]"
2025-01-25T14:29:49.274Z [INFO]  server-d3105446-26dd-4d5c-a665-d43a0fb786a2: entering follower state: follower="Node at d3105446-26dd-4d5c-a665-d43a0fb786a2 [Follower]" leader-address= leader-id=
2025-01-25T14:29:49.337Z [WARN]  server-d3105446-26dd-4d5c-a665-d43a0fb786a2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:49.337Z [INFO]  server-d3105446-26dd-4d5c-a665-d43a0fb786a2: entering candidate state: node="Node at d3105446-26dd-4d5c-a665-d43a0fb786a2 [Candidate]" term=2
2025-01-25T14:29:49.338Z [INFO]  server-d3105446-26dd-4d5c-a665-d43a0fb786a2: election won: term=2 tally=1
2025-01-25T14:29:49.338Z [INFO]  server-d3105446-26dd-4d5c-a665-d43a0fb786a2: entering leader state: leader="Node at d3105446-26dd-4d5c-a665-d43a0fb786a2 [Leader]"
--- PASS: TestRaft_NotifyCh (0.06s)
=== RUN   TestRaft_AppendEntry
2025-01-25T14:29:49.340Z [INFO]  server-94acb135-878e-eb19-b4ab-718c571a6bdb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94acb135-878e-eb19-b4ab-718c571a6bdb Address:94acb135-878e-eb19-b4ab-718c571a6bdb} {Suffrage:Voter ID:server-242e5f70-7bd4-d6d3-4f9e-98828be08e1a Address:242e5f70-7bd4-d6d3-4f9e-98828be08e1a} {Suffrage:Voter ID:server-1e3277e5-461d-8989-fa76-45160a8b152e Address:1e3277e5-461d-8989-fa76-45160a8b152e}]"
2025-01-25T14:29:49.340Z [INFO]  server-242e5f70-7bd4-d6d3-4f9e-98828be08e1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94acb135-878e-eb19-b4ab-718c571a6bdb Address:94acb135-878e-eb19-b4ab-718c571a6bdb} {Suffrage:Voter ID:server-242e5f70-7bd4-d6d3-4f9e-98828be08e1a Address:242e5f70-7bd4-d6d3-4f9e-98828be08e1a} {Suffrage:Voter ID:server-1e3277e5-461d-8989-fa76-45160a8b152e Address:1e3277e5-461d-8989-fa76-45160a8b152e}]"
2025-01-25T14:29:49.340Z [INFO]  server-94acb135-878e-eb19-b4ab-718c571a6bdb: entering follower state: follower="Node at 94acb135-878e-eb19-b4ab-718c571a6bdb [Follower]" leader-address= leader-id=
2025-01-25T14:29:49.340Z [INFO]  server-242e5f70-7bd4-d6d3-4f9e-98828be08e1a: entering follower state: follower="Node at 242e5f70-7bd4-d6d3-4f9e-98828be08e1a [Follower]" leader-address= leader-id=
2025-01-25T14:29:49.340Z [INFO]  server-1e3277e5-461d-8989-fa76-45160a8b152e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94acb135-878e-eb19-b4ab-718c571a6bdb Address:94acb135-878e-eb19-b4ab-718c571a6bdb} {Suffrage:Voter ID:server-242e5f70-7bd4-d6d3-4f9e-98828be08e1a Address:242e5f70-7bd4-d6d3-4f9e-98828be08e1a} {Suffrage:Voter ID:server-1e3277e5-461d-8989-fa76-45160a8b152e Address:1e3277e5-461d-8989-fa76-45160a8b152e}]"
2025-01-25T14:29:49.340Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:49.341Z [INFO]  server-1e3277e5-461d-8989-fa76-45160a8b152e: entering follower state: follower="Node at 1e3277e5-461d-8989-fa76-45160a8b152e [Follower]" leader-address= leader-id=
2025-01-25T14:29:49.391Z [WARN]  server-1e3277e5-461d-8989-fa76-45160a8b152e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:49.391Z [INFO]  server-1e3277e5-461d-8989-fa76-45160a8b152e: entering candidate state: node="Node at 1e3277e5-461d-8989-fa76-45160a8b152e [Candidate]" term=2
2025-01-25T14:29:49.391Z [INFO]  server-1e3277e5-461d-8989-fa76-45160a8b152e: election won: term=2 tally=2
2025-01-25T14:29:49.391Z [INFO]  server-1e3277e5-461d-8989-fa76-45160a8b152e: entering leader state: leader="Node at 1e3277e5-461d-8989-fa76-45160a8b152e [Leader]"
2025-01-25T14:29:49.391Z [INFO]  server-1e3277e5-461d-8989-fa76-45160a8b152e: added peer, starting replication: peer=server-94acb135-878e-eb19-b4ab-718c571a6bdb
2025-01-25T14:29:49.391Z [INFO]  server-1e3277e5-461d-8989-fa76-45160a8b152e: added peer, starting replication: peer=server-242e5f70-7bd4-d6d3-4f9e-98828be08e1a
2025-01-25T14:29:49.392Z [INFO]  server-1e3277e5-461d-8989-fa76-45160a8b152e: pipelining replication: peer="{Voter server-242e5f70-7bd4-d6d3-4f9e-98828be08e1a 242e5f70-7bd4-d6d3-4f9e-98828be08e1a}"
2025-01-25T14:29:49.392Z [INFO]  server-1e3277e5-461d-8989-fa76-45160a8b152e: pipelining replication: peer="{Voter server-94acb135-878e-eb19-b4ab-718c571a6bdb 94acb135-878e-eb19-b4ab-718c571a6bdb}"
2025-01-25T14:29:49.497Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:49.392011144 +0000 UTC m=+99.246336126 (2 nodes), 50.976926ms from start of poll, 51.7025ms from cluster start. Timeout at 2025-01-25 14:29:49.497015183 +0000 UTC m=+99.351340165, 105.004039ms after stability
2025-01-25T14:29:49.497Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:49.603Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:49.497622503 +0000 UTC m=+99.351947485 (1 nodes), 2.48µs from start of poll, 157.313859ms from cluster start. Timeout at 2025-01-25 14:29:49.602885333 +0000 UTC m=+99.457210355, 105.26287ms after stability
2025-01-25T14:29:49.603Z [WARN]  server-94acb135-878e-eb19-b4ab-718c571a6bdb: clearing log suffix: from=1 to=2
2025-01-25T14:29:49.603Z [WARN]  server-94acb135-878e-eb19-b4ab-718c571a6bdb: skipping application of old log: index=1
2025-01-25T14:29:49.603Z [WARN]  server-94acb135-878e-eb19-b4ab-718c571a6bdb: skipping application of old log: index=1
2025-01-25T14:29:49.603Z [INFO]  server-1e3277e5-461d-8989-fa76-45160a8b152e: aborting pipeline replication: peer="{Voter server-242e5f70-7bd4-d6d3-4f9e-98828be08e1a 242e5f70-7bd4-d6d3-4f9e-98828be08e1a}"
2025-01-25T14:29:49.603Z [INFO]  server-1e3277e5-461d-8989-fa76-45160a8b152e: aborting pipeline replication: peer="{Voter server-94acb135-878e-eb19-b4ab-718c571a6bdb 94acb135-878e-eb19-b4ab-718c571a6bdb}"
--- PASS: TestRaft_AppendEntry (0.27s)
=== RUN   TestRaft_VotingGrant_WhenLeaderAvailable
2025-01-25T14:29:49.605Z [INFO]  server-e35948a9-9d50-b227-eb75-61073348b35e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e35948a9-9d50-b227-eb75-61073348b35e Address:e35948a9-9d50-b227-eb75-61073348b35e} {Suffrage:Voter ID:server-07fa1e12-081c-37c5-dbd9-0194a7bf379a Address:07fa1e12-081c-37c5-dbd9-0194a7bf379a} {Suffrage:Voter ID:server-57aeb195-dcc1-871a-fba0-c2771a5fef27 Address:57aeb195-dcc1-871a-fba0-c2771a5fef27}]"
2025-01-25T14:29:49.605Z [INFO]  server-e35948a9-9d50-b227-eb75-61073348b35e: entering follower state: follower="Node at e35948a9-9d50-b227-eb75-61073348b35e [Follower]" leader-address= leader-id=
2025-01-25T14:29:49.606Z [INFO]  server-07fa1e12-081c-37c5-dbd9-0194a7bf379a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e35948a9-9d50-b227-eb75-61073348b35e Address:e35948a9-9d50-b227-eb75-61073348b35e} {Suffrage:Voter ID:server-07fa1e12-081c-37c5-dbd9-0194a7bf379a Address:07fa1e12-081c-37c5-dbd9-0194a7bf379a} {Suffrage:Voter ID:server-57aeb195-dcc1-871a-fba0-c2771a5fef27 Address:57aeb195-dcc1-871a-fba0-c2771a5fef27}]"
2025-01-25T14:29:49.606Z [INFO]  server-07fa1e12-081c-37c5-dbd9-0194a7bf379a: entering follower state: follower="Node at 07fa1e12-081c-37c5-dbd9-0194a7bf379a [Follower]" leader-address= leader-id=
2025-01-25T14:29:49.606Z [INFO]  server-57aeb195-dcc1-871a-fba0-c2771a5fef27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e35948a9-9d50-b227-eb75-61073348b35e Address:e35948a9-9d50-b227-eb75-61073348b35e} {Suffrage:Voter ID:server-07fa1e12-081c-37c5-dbd9-0194a7bf379a Address:07fa1e12-081c-37c5-dbd9-0194a7bf379a} {Suffrage:Voter ID:server-57aeb195-dcc1-871a-fba0-c2771a5fef27 Address:57aeb195-dcc1-871a-fba0-c2771a5fef27}]"
2025-01-25T14:29:49.606Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:49.606Z [INFO]  server-57aeb195-dcc1-871a-fba0-c2771a5fef27: entering follower state: follower="Node at 57aeb195-dcc1-871a-fba0-c2771a5fef27 [Follower]" leader-address= leader-id=
2025-01-25T14:29:49.663Z [WARN]  server-57aeb195-dcc1-871a-fba0-c2771a5fef27: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:49.663Z [INFO]  server-57aeb195-dcc1-871a-fba0-c2771a5fef27: entering candidate state: node="Node at 57aeb195-dcc1-871a-fba0-c2771a5fef27 [Candidate]" term=2
2025-01-25T14:29:49.663Z [INFO]  server-57aeb195-dcc1-871a-fba0-c2771a5fef27: election won: term=2 tally=2
2025-01-25T14:29:49.663Z [INFO]  server-57aeb195-dcc1-871a-fba0-c2771a5fef27: entering leader state: leader="Node at 57aeb195-dcc1-871a-fba0-c2771a5fef27 [Leader]"
2025-01-25T14:29:49.663Z [INFO]  server-57aeb195-dcc1-871a-fba0-c2771a5fef27: added peer, starting replication: peer=server-e35948a9-9d50-b227-eb75-61073348b35e
2025-01-25T14:29:49.663Z [INFO]  server-57aeb195-dcc1-871a-fba0-c2771a5fef27: added peer, starting replication: peer=server-07fa1e12-081c-37c5-dbd9-0194a7bf379a
2025-01-25T14:29:49.664Z [INFO]  server-57aeb195-dcc1-871a-fba0-c2771a5fef27: pipelining replication: peer="{Voter server-e35948a9-9d50-b227-eb75-61073348b35e e35948a9-9d50-b227-eb75-61073348b35e}"
2025-01-25T14:29:49.664Z [INFO]  server-57aeb195-dcc1-871a-fba0-c2771a5fef27: pipelining replication: peer="{Voter server-07fa1e12-081c-37c5-dbd9-0194a7bf379a 07fa1e12-081c-37c5-dbd9-0194a7bf379a}"
2025-01-25T14:29:49.769Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:49.663853081 +0000 UTC m=+99.518178023 (2 nodes), 57.535769ms from start of poll, 58.194186ms from cluster start. Timeout at 2025-01-25 14:29:49.768858081 +0000 UTC m=+99.623183063, 105.00504ms after stability
2025-01-25T14:29:49.769Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:49.874Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:49.769557477 +0000 UTC m=+99.623882459 (1 nodes), 1.92µs from start of poll, 163.898622ms from cluster start. Timeout at 2025-01-25 14:29:49.874764429 +0000 UTC m=+99.729089571, 105.207112ms after stability
2025-01-25T14:29:49.875Z [WARN]  server-07fa1e12-081c-37c5-dbd9-0194a7bf379a: rejecting vote request since we have a leader: from=e35948a9-9d50-b227-eb75-61073348b35e leader=57aeb195-dcc1-871a-fba0-c2771a5fef27 leader-id=server-57aeb195-dcc1-871a-fba0-c2771a5fef27
2025-01-25T14:29:49.875Z [ERROR] server-57aeb195-dcc1-871a-fba0-c2771a5fef27: peer has newer term, stopping replication: peer="{Voter server-e35948a9-9d50-b227-eb75-61073348b35e e35948a9-9d50-b227-eb75-61073348b35e}"
2025-01-25T14:29:49.875Z [INFO]  server-57aeb195-dcc1-871a-fba0-c2771a5fef27: aborting pipeline replication: peer="{Voter server-e35948a9-9d50-b227-eb75-61073348b35e e35948a9-9d50-b227-eb75-61073348b35e}"
2025-01-25T14:29:49.875Z [INFO]  server-57aeb195-dcc1-871a-fba0-c2771a5fef27: aborting pipeline replication: peer="{Voter server-07fa1e12-081c-37c5-dbd9-0194a7bf379a 07fa1e12-081c-37c5-dbd9-0194a7bf379a}"
--- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.27s)
=== RUN   TestRaft_ProtocolVersion_RejectRPC
2025-01-25T14:29:49.877Z [INFO]  server-869b7e59-f8f1-fce0-5626-fd59907b12b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-869b7e59-f8f1-fce0-5626-fd59907b12b1 Address:869b7e59-f8f1-fce0-5626-fd59907b12b1} {Suffrage:Voter ID:server-bbb55135-deaa-e666-8235-97f50969b083 Address:bbb55135-deaa-e666-8235-97f50969b083} {Suffrage:Voter ID:server-9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce Address:9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce}]"
2025-01-25T14:29:49.877Z [INFO]  server-869b7e59-f8f1-fce0-5626-fd59907b12b1: entering follower state: follower="Node at 869b7e59-f8f1-fce0-5626-fd59907b12b1 [Follower]" leader-address= leader-id=
2025-01-25T14:29:49.878Z [INFO]  server-bbb55135-deaa-e666-8235-97f50969b083: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-869b7e59-f8f1-fce0-5626-fd59907b12b1 Address:869b7e59-f8f1-fce0-5626-fd59907b12b1} {Suffrage:Voter ID:server-bbb55135-deaa-e666-8235-97f50969b083 Address:bbb55135-deaa-e666-8235-97f50969b083} {Suffrage:Voter ID:server-9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce Address:9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce}]"
2025-01-25T14:29:49.878Z [INFO]  server-bbb55135-deaa-e666-8235-97f50969b083: entering follower state: follower="Node at bbb55135-deaa-e666-8235-97f50969b083 [Follower]" leader-address= leader-id=
2025-01-25T14:29:49.878Z [INFO]  server-9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-869b7e59-f8f1-fce0-5626-fd59907b12b1 Address:869b7e59-f8f1-fce0-5626-fd59907b12b1} {Suffrage:Voter ID:server-bbb55135-deaa-e666-8235-97f50969b083 Address:bbb55135-deaa-e666-8235-97f50969b083} {Suffrage:Voter ID:server-9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce Address:9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce}]"
2025-01-25T14:29:49.878Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:49.878Z [INFO]  server-9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce: entering follower state: follower="Node at 9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce [Follower]" leader-address= leader-id=
2025-01-25T14:29:49.937Z [WARN]  server-9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:49.937Z [INFO]  server-9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce: entering candidate state: node="Node at 9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce [Candidate]" term=2
2025-01-25T14:29:49.937Z [INFO]  server-9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce: election won: term=2 tally=2
2025-01-25T14:29:49.937Z [INFO]  server-9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce: entering leader state: leader="Node at 9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce [Leader]"
2025-01-25T14:29:49.937Z [INFO]  server-9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce: added peer, starting replication: peer=server-869b7e59-f8f1-fce0-5626-fd59907b12b1
2025-01-25T14:29:49.937Z [INFO]  server-9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce: added peer, starting replication: peer=server-bbb55135-deaa-e666-8235-97f50969b083
2025-01-25T14:29:49.937Z [INFO]  server-9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce: pipelining replication: peer="{Voter server-bbb55135-deaa-e666-8235-97f50969b083 bbb55135-deaa-e666-8235-97f50969b083}"
2025-01-25T14:29:49.937Z [INFO]  server-9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce: pipelining replication: peer="{Voter server-869b7e59-f8f1-fce0-5626-fd59907b12b1 869b7e59-f8f1-fce0-5626-fd59907b12b1}"
2025-01-25T14:29:50.043Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:49.93757635 +0000 UTC m=+99.791901332 (2 nodes), 59.293626ms from start of poll, 59.968081ms from cluster start. Timeout at 2025-01-25 14:29:50.042615509 +0000 UTC m=+99.896940491, 105.039159ms after stability
2025-01-25T14:29:50.044Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:50.150Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:50.044039399 +0000 UTC m=+99.898364381 (1 nodes), 2.52µs from start of poll, 166.43113ms from cluster start. Timeout at 2025-01-25 14:29:50.149198193 +0000 UTC m=+100.003523175, 105.158794ms after stability
2025-01-25T14:29:50.150Z [INFO]  server-9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce: aborting pipeline replication: peer="{Voter server-bbb55135-deaa-e666-8235-97f50969b083 bbb55135-deaa-e666-8235-97f50969b083}"
2025-01-25T14:29:50.150Z [INFO]  server-9c8d64aa-c31b-ae1b-00b5-3da07b4cd2ce: aborting pipeline replication: peer="{Voter server-869b7e59-f8f1-fce0-5626-fd59907b12b1 869b7e59-f8f1-fce0-5626-fd59907b12b1}"
--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_1_2
    raft_test.go:2151: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_2_3
    raft_test.go:2196: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s)
=== RUN   TestRaft_LeaderID_Propagated
2025-01-25T14:29:50.153Z [INFO]  server-65b71926-a98d-c4c2-708d-4c8ee5270da2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65b71926-a98d-c4c2-708d-4c8ee5270da2 Address:65b71926-a98d-c4c2-708d-4c8ee5270da2} {Suffrage:Voter ID:server-b24b02b8-59db-e4a4-e245-819a9768d206 Address:b24b02b8-59db-e4a4-e245-819a9768d206} {Suffrage:Voter ID:server-7e4b5d00-4cac-567f-df73-374479edd8e7 Address:7e4b5d00-4cac-567f-df73-374479edd8e7}]"
2025-01-25T14:29:50.153Z [INFO]  server-65b71926-a98d-c4c2-708d-4c8ee5270da2: entering follower state: follower="Node at 65b71926-a98d-c4c2-708d-4c8ee5270da2 [Follower]" leader-address= leader-id=
2025-01-25T14:29:50.153Z [INFO]  server-b24b02b8-59db-e4a4-e245-819a9768d206: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65b71926-a98d-c4c2-708d-4c8ee5270da2 Address:65b71926-a98d-c4c2-708d-4c8ee5270da2} {Suffrage:Voter ID:server-b24b02b8-59db-e4a4-e245-819a9768d206 Address:b24b02b8-59db-e4a4-e245-819a9768d206} {Suffrage:Voter ID:server-7e4b5d00-4cac-567f-df73-374479edd8e7 Address:7e4b5d00-4cac-567f-df73-374479edd8e7}]"
2025-01-25T14:29:50.153Z [INFO]  server-b24b02b8-59db-e4a4-e245-819a9768d206: entering follower state: follower="Node at b24b02b8-59db-e4a4-e245-819a9768d206 [Follower]" leader-address= leader-id=
2025-01-25T14:29:50.153Z [INFO]  server-7e4b5d00-4cac-567f-df73-374479edd8e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65b71926-a98d-c4c2-708d-4c8ee5270da2 Address:65b71926-a98d-c4c2-708d-4c8ee5270da2} {Suffrage:Voter ID:server-b24b02b8-59db-e4a4-e245-819a9768d206 Address:b24b02b8-59db-e4a4-e245-819a9768d206} {Suffrage:Voter ID:server-7e4b5d00-4cac-567f-df73-374479edd8e7 Address:7e4b5d00-4cac-567f-df73-374479edd8e7}]"
2025-01-25T14:29:50.153Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:50.153Z [INFO]  server-7e4b5d00-4cac-567f-df73-374479edd8e7: entering follower state: follower="Node at 7e4b5d00-4cac-567f-df73-374479edd8e7 [Follower]" leader-address= leader-id=
2025-01-25T14:29:50.215Z [WARN]  server-65b71926-a98d-c4c2-708d-4c8ee5270da2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:50.216Z [INFO]  server-65b71926-a98d-c4c2-708d-4c8ee5270da2: entering candidate state: node="Node at 65b71926-a98d-c4c2-708d-4c8ee5270da2 [Candidate]" term=2
2025-01-25T14:29:50.216Z [INFO]  server-65b71926-a98d-c4c2-708d-4c8ee5270da2: election won: term=2 tally=2
2025-01-25T14:29:50.216Z [INFO]  server-65b71926-a98d-c4c2-708d-4c8ee5270da2: entering leader state: leader="Node at 65b71926-a98d-c4c2-708d-4c8ee5270da2 [Leader]"
2025-01-25T14:29:50.216Z [INFO]  server-65b71926-a98d-c4c2-708d-4c8ee5270da2: added peer, starting replication: peer=server-b24b02b8-59db-e4a4-e245-819a9768d206
2025-01-25T14:29:50.216Z [INFO]  server-65b71926-a98d-c4c2-708d-4c8ee5270da2: added peer, starting replication: peer=server-7e4b5d00-4cac-567f-df73-374479edd8e7
2025-01-25T14:29:50.216Z [INFO]  server-65b71926-a98d-c4c2-708d-4c8ee5270da2: pipelining replication: peer="{Voter server-7e4b5d00-4cac-567f-df73-374479edd8e7 7e4b5d00-4cac-567f-df73-374479edd8e7}"
2025-01-25T14:29:50.216Z [INFO]  server-65b71926-a98d-c4c2-708d-4c8ee5270da2: pipelining replication: peer="{Voter server-b24b02b8-59db-e4a4-e245-819a9768d206 b24b02b8-59db-e4a4-e245-819a9768d206}"
2025-01-25T14:29:50.322Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:50.216462554 +0000 UTC m=+100.070787536 (1 nodes), 62.473672ms from start of poll, 63.189846ms from cluster start. Timeout at 2025-01-25 14:29:50.321466513 +0000 UTC m=+100.175791535, 105.003999ms after stability
2025-01-25T14:29:50.322Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:50.429Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:50.32271633 +0000 UTC m=+100.177041272 (1 nodes), 1.24µs from start of poll, 169.443582ms from cluster start. Timeout at 2025-01-25 14:29:50.42798972 +0000 UTC m=+100.282314662, 105.27339ms after stability
2025-01-25T14:29:50.429Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:50.536Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:50.429667521 +0000 UTC m=+100.283992463 (1 nodes), 1.2µs from start of poll, 276.394773ms from cluster start. Timeout at 2025-01-25 14:29:50.534836075 +0000 UTC m=+100.389161017, 105.168554ms after stability
2025-01-25T14:29:50.536Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:50.642Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:50.536649871 +0000 UTC m=+100.390974853 (1 nodes), 1.88µs from start of poll, 383.377163ms from cluster start. Timeout at 2025-01-25 14:29:50.641891062 +0000 UTC m=+100.496216084, 105.241231ms after stability
2025-01-25T14:29:50.642Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:50.748Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:50.642428985 +0000 UTC m=+100.496753967 (1 nodes), 1.36µs from start of poll, 489.156277ms from cluster start. Timeout at 2025-01-25 14:29:50.747654297 +0000 UTC m=+100.601979279, 105.225312ms after stability
2025-01-25T14:29:50.748Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:50.854Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:50.748400291 +0000 UTC m=+100.602725273 (1 nodes), 1.12µs from start of poll, 595.127583ms from cluster start. Timeout at 2025-01-25 14:29:50.853571604 +0000 UTC m=+100.707896586, 105.171313ms after stability
2025-01-25T14:29:50.960Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:51.066Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:50.960212948 +0000 UTC m=+100.814537930 (1 nodes), 1.159µs from start of poll, 806.94024ms from cluster start. Timeout at 2025-01-25 14:29:51.065431581 +0000 UTC m=+100.919756563, 105.218633ms after stability
2025-01-25T14:29:51.066Z [INFO]  server-65b71926-a98d-c4c2-708d-4c8ee5270da2: aborting pipeline replication: peer="{Voter server-b24b02b8-59db-e4a4-e245-819a9768d206 b24b02b8-59db-e4a4-e245-819a9768d206}"
2025-01-25T14:29:51.066Z [INFO]  server-65b71926-a98d-c4c2-708d-4c8ee5270da2: aborting pipeline replication: peer="{Voter server-7e4b5d00-4cac-567f-df73-374479edd8e7 7e4b5d00-4cac-567f-df73-374479edd8e7}"
--- PASS: TestRaft_LeaderID_Propagated (0.92s)
=== RUN   TestRaft_LeadershipTransferInProgress
--- PASS: TestRaft_LeadershipTransferInProgress (0.00s)
=== RUN   TestRaft_LeadershipTransferPickServer
--- PASS: TestRaft_LeadershipTransferPickServer (0.00s)
=== RUN   TestRaft_LeadershipTransfer
2025-01-25T14:29:51.070Z [INFO]  server-06c471f4-1096-676e-f79e-cba8e1f25656: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06c471f4-1096-676e-f79e-cba8e1f25656 Address:06c471f4-1096-676e-f79e-cba8e1f25656} {Suffrage:Voter ID:server-a65baebb-6749-785b-1027-899b4cbba662 Address:a65baebb-6749-785b-1027-899b4cbba662} {Suffrage:Voter ID:server-e1f0691e-0ba0-c022-c1c5-175e73e93f7a Address:e1f0691e-0ba0-c022-c1c5-175e73e93f7a}]"
2025-01-25T14:29:51.070Z [INFO]  server-06c471f4-1096-676e-f79e-cba8e1f25656: entering follower state: follower="Node at 06c471f4-1096-676e-f79e-cba8e1f25656 [Follower]" leader-address= leader-id=
2025-01-25T14:29:51.070Z [INFO]  server-a65baebb-6749-785b-1027-899b4cbba662: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06c471f4-1096-676e-f79e-cba8e1f25656 Address:06c471f4-1096-676e-f79e-cba8e1f25656} {Suffrage:Voter ID:server-a65baebb-6749-785b-1027-899b4cbba662 Address:a65baebb-6749-785b-1027-899b4cbba662} {Suffrage:Voter ID:server-e1f0691e-0ba0-c022-c1c5-175e73e93f7a Address:e1f0691e-0ba0-c022-c1c5-175e73e93f7a}]"
2025-01-25T14:29:51.070Z [INFO]  server-a65baebb-6749-785b-1027-899b4cbba662: entering follower state: follower="Node at a65baebb-6749-785b-1027-899b4cbba662 [Follower]" leader-address= leader-id=
2025-01-25T14:29:51.070Z [INFO]  server-e1f0691e-0ba0-c022-c1c5-175e73e93f7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06c471f4-1096-676e-f79e-cba8e1f25656 Address:06c471f4-1096-676e-f79e-cba8e1f25656} {Suffrage:Voter ID:server-a65baebb-6749-785b-1027-899b4cbba662 Address:a65baebb-6749-785b-1027-899b4cbba662} {Suffrage:Voter ID:server-e1f0691e-0ba0-c022-c1c5-175e73e93f7a Address:e1f0691e-0ba0-c022-c1c5-175e73e93f7a}]"
2025-01-25T14:29:51.070Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:51.070Z [INFO]  server-e1f0691e-0ba0-c022-c1c5-175e73e93f7a: entering follower state: follower="Node at e1f0691e-0ba0-c022-c1c5-175e73e93f7a [Follower]" leader-address= leader-id=
2025-01-25T14:29:51.131Z [WARN]  server-06c471f4-1096-676e-f79e-cba8e1f25656: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:51.131Z [INFO]  server-06c471f4-1096-676e-f79e-cba8e1f25656: entering candidate state: node="Node at 06c471f4-1096-676e-f79e-cba8e1f25656 [Candidate]" term=2
2025-01-25T14:29:51.131Z [INFO]  server-06c471f4-1096-676e-f79e-cba8e1f25656: election won: term=2 tally=2
2025-01-25T14:29:51.131Z [INFO]  server-06c471f4-1096-676e-f79e-cba8e1f25656: entering leader state: leader="Node at 06c471f4-1096-676e-f79e-cba8e1f25656 [Leader]"
2025-01-25T14:29:51.131Z [INFO]  server-06c471f4-1096-676e-f79e-cba8e1f25656: added peer, starting replication: peer=server-a65baebb-6749-785b-1027-899b4cbba662
2025-01-25T14:29:51.131Z [INFO]  server-06c471f4-1096-676e-f79e-cba8e1f25656: added peer, starting replication: peer=server-e1f0691e-0ba0-c022-c1c5-175e73e93f7a
2025-01-25T14:29:51.131Z [INFO]  server-06c471f4-1096-676e-f79e-cba8e1f25656: pipelining replication: peer="{Voter server-e1f0691e-0ba0-c022-c1c5-175e73e93f7a e1f0691e-0ba0-c022-c1c5-175e73e93f7a}"
2025-01-25T14:29:51.131Z [INFO]  server-06c471f4-1096-676e-f79e-cba8e1f25656: pipelining replication: peer="{Voter server-a65baebb-6749-785b-1027-899b4cbba662 a65baebb-6749-785b-1027-899b4cbba662}"
2025-01-25T14:29:51.236Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:51.131475065 +0000 UTC m=+100.985800047 (1 nodes), 60.656497ms from start of poll, 61.428709ms from cluster start. Timeout at 2025-01-25 14:29:51.236480425 +0000 UTC m=+101.090805367, 105.00532ms after stability
2025-01-25T14:29:51.236Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:51.343Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:51.236797595 +0000 UTC m=+101.091122577 (1 nodes), 2.88µs from start of poll, 166.751239ms from cluster start. Timeout at 2025-01-25 14:29:51.342000708 +0000 UTC m=+101.196325690, 105.203113ms after stability
2025-01-25T14:29:51.344Z [INFO]  server-a65baebb-6749-785b-1027-899b4cbba662: entering candidate state: node="Node at a65baebb-6749-785b-1027-899b4cbba662 [Candidate]" term=3
2025-01-25T14:29:51.344Z [INFO]  server-a65baebb-6749-785b-1027-899b4cbba662: election won: term=3 tally=2
2025-01-25T14:29:51.344Z [INFO]  server-a65baebb-6749-785b-1027-899b4cbba662: entering leader state: leader="Node at a65baebb-6749-785b-1027-899b4cbba662 [Leader]"
2025-01-25T14:29:51.344Z [INFO]  server-a65baebb-6749-785b-1027-899b4cbba662: added peer, starting replication: peer=server-06c471f4-1096-676e-f79e-cba8e1f25656
2025-01-25T14:29:51.344Z [INFO]  server-a65baebb-6749-785b-1027-899b4cbba662: added peer, starting replication: peer=server-e1f0691e-0ba0-c022-c1c5-175e73e93f7a
2025-01-25T14:29:51.344Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:51.344Z [INFO]  server-a65baebb-6749-785b-1027-899b4cbba662: pipelining replication: peer="{Voter server-e1f0691e-0ba0-c022-c1c5-175e73e93f7a e1f0691e-0ba0-c022-c1c5-175e73e93f7a}"
2025-01-25T14:29:51.344Z [INFO]  server-06c471f4-1096-676e-f79e-cba8e1f25656: entering follower state: follower="Node at 06c471f4-1096-676e-f79e-cba8e1f25656 [Follower]" leader-address= leader-id=
2025-01-25T14:29:51.344Z [INFO]  server-06c471f4-1096-676e-f79e-cba8e1f25656: aborting pipeline replication: peer="{Voter server-a65baebb-6749-785b-1027-899b4cbba662 a65baebb-6749-785b-1027-899b4cbba662}"
2025-01-25T14:29:51.344Z [INFO]  server-06c471f4-1096-676e-f79e-cba8e1f25656: aborting pipeline replication: peer="{Voter server-e1f0691e-0ba0-c022-c1c5-175e73e93f7a e1f0691e-0ba0-c022-c1c5-175e73e93f7a}"
2025-01-25T14:29:51.344Z [INFO]  server-a65baebb-6749-785b-1027-899b4cbba662: pipelining replication: peer="{Voter server-06c471f4-1096-676e-f79e-cba8e1f25656 06c471f4-1096-676e-f79e-cba8e1f25656}"
2025-01-25T14:29:51.450Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:51.34474427 +0000 UTC m=+101.199069252 (1 nodes), 276.71µs from start of poll, 274.697914ms from cluster start. Timeout at 2025-01-25 14:29:51.44974591 +0000 UTC m=+101.304070892, 105.00164ms after stability
2025-01-25T14:29:51.450Z [INFO]  server-a65baebb-6749-785b-1027-899b4cbba662: aborting pipeline replication: peer="{Voter server-e1f0691e-0ba0-c022-c1c5-175e73e93f7a e1f0691e-0ba0-c022-c1c5-175e73e93f7a}"
2025-01-25T14:29:51.450Z [INFO]  server-a65baebb-6749-785b-1027-899b4cbba662: aborting pipeline replication: peer="{Voter server-06c471f4-1096-676e-f79e-cba8e1f25656 06c471f4-1096-676e-f79e-cba8e1f25656}"
--- PASS: TestRaft_LeadershipTransfer (0.38s)
=== RUN   TestRaft_LeadershipTransferWithOneNode
2025-01-25T14:29:51.452Z [INFO]  server-c52787a1-0dbf-1315-a4e7-b6d887cfd5b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c52787a1-0dbf-1315-a4e7-b6d887cfd5b8 Address:c52787a1-0dbf-1315-a4e7-b6d887cfd5b8}]"
2025-01-25T14:29:51.452Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:51.452Z [INFO]  server-c52787a1-0dbf-1315-a4e7-b6d887cfd5b8: entering follower state: follower="Node at c52787a1-0dbf-1315-a4e7-b6d887cfd5b8 [Follower]" leader-address= leader-id=
2025-01-25T14:29:51.534Z [WARN]  server-c52787a1-0dbf-1315-a4e7-b6d887cfd5b8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:51.534Z [INFO]  server-c52787a1-0dbf-1315-a4e7-b6d887cfd5b8: entering candidate state: node="Node at c52787a1-0dbf-1315-a4e7-b6d887cfd5b8 [Candidate]" term=2
2025-01-25T14:29:51.534Z [INFO]  server-c52787a1-0dbf-1315-a4e7-b6d887cfd5b8: election won: term=2 tally=1
2025-01-25T14:29:51.534Z [INFO]  server-c52787a1-0dbf-1315-a4e7-b6d887cfd5b8: entering leader state: leader="Node at c52787a1-0dbf-1315-a4e7-b6d887cfd5b8 [Leader]"
2025-01-25T14:29:51.641Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:51.535146778 +0000 UTC m=+101.389471720 (1 nodes), 82.649506ms from start of poll, 82.890337ms from cluster start. Timeout at 2025-01-25 14:29:51.640151698 +0000 UTC m=+101.494476640, 105.00492ms after stability
--- PASS: TestRaft_LeadershipTransferWithOneNode (0.19s)
=== RUN   TestRaft_LeadershipTransferWithSevenNodes
2025-01-25T14:29:51.645Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1ef68a1-157e-004f-2992-86f439700735 Address:b1ef68a1-157e-004f-2992-86f439700735} {Suffrage:Voter ID:server-63dfeab7-b1db-8c9d-e8ff-9e5a455187ef Address:63dfeab7-b1db-8c9d-e8ff-9e5a455187ef} {Suffrage:Voter ID:server-8a095541-f276-2986-14c9-f10f3a3f537b Address:8a095541-f276-2986-14c9-f10f3a3f537b} {Suffrage:Voter ID:server-c0207010-c564-e72f-006e-a76fcc4ba9e4 Address:c0207010-c564-e72f-006e-a76fcc4ba9e4} {Suffrage:Voter ID:server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d Address:ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d} {Suffrage:Voter ID:server-7731092c-1bb9-a1a7-f8d8-14c99df84e0e Address:7731092c-1bb9-a1a7-f8d8-14c99df84e0e} {Suffrage:Voter ID:server-8c2b8647-cd78-d3a2-15db-31bb4a415706 Address:8c2b8647-cd78-d3a2-15db-31bb4a415706}]"
2025-01-25T14:29:51.645Z [INFO]  server-63dfeab7-b1db-8c9d-e8ff-9e5a455187ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1ef68a1-157e-004f-2992-86f439700735 Address:b1ef68a1-157e-004f-2992-86f439700735} {Suffrage:Voter ID:server-63dfeab7-b1db-8c9d-e8ff-9e5a455187ef Address:63dfeab7-b1db-8c9d-e8ff-9e5a455187ef} {Suffrage:Voter ID:server-8a095541-f276-2986-14c9-f10f3a3f537b Address:8a095541-f276-2986-14c9-f10f3a3f537b} {Suffrage:Voter ID:server-c0207010-c564-e72f-006e-a76fcc4ba9e4 Address:c0207010-c564-e72f-006e-a76fcc4ba9e4} {Suffrage:Voter ID:server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d Address:ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d} {Suffrage:Voter ID:server-7731092c-1bb9-a1a7-f8d8-14c99df84e0e Address:7731092c-1bb9-a1a7-f8d8-14c99df84e0e} {Suffrage:Voter ID:server-8c2b8647-cd78-d3a2-15db-31bb4a415706 Address:8c2b8647-cd78-d3a2-15db-31bb4a415706}]"
2025-01-25T14:29:51.645Z [INFO]  server-8a095541-f276-2986-14c9-f10f3a3f537b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1ef68a1-157e-004f-2992-86f439700735 Address:b1ef68a1-157e-004f-2992-86f439700735} {Suffrage:Voter ID:server-63dfeab7-b1db-8c9d-e8ff-9e5a455187ef Address:63dfeab7-b1db-8c9d-e8ff-9e5a455187ef} {Suffrage:Voter ID:server-8a095541-f276-2986-14c9-f10f3a3f537b Address:8a095541-f276-2986-14c9-f10f3a3f537b} {Suffrage:Voter ID:server-c0207010-c564-e72f-006e-a76fcc4ba9e4 Address:c0207010-c564-e72f-006e-a76fcc4ba9e4} {Suffrage:Voter ID:server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d Address:ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d} {Suffrage:Voter ID:server-7731092c-1bb9-a1a7-f8d8-14c99df84e0e Address:7731092c-1bb9-a1a7-f8d8-14c99df84e0e} {Suffrage:Voter ID:server-8c2b8647-cd78-d3a2-15db-31bb4a415706 Address:8c2b8647-cd78-d3a2-15db-31bb4a415706}]"
2025-01-25T14:29:51.646Z [INFO]  server-c0207010-c564-e72f-006e-a76fcc4ba9e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1ef68a1-157e-004f-2992-86f439700735 Address:b1ef68a1-157e-004f-2992-86f439700735} {Suffrage:Voter ID:server-63dfeab7-b1db-8c9d-e8ff-9e5a455187ef Address:63dfeab7-b1db-8c9d-e8ff-9e5a455187ef} {Suffrage:Voter ID:server-8a095541-f276-2986-14c9-f10f3a3f537b Address:8a095541-f276-2986-14c9-f10f3a3f537b} {Suffrage:Voter ID:server-c0207010-c564-e72f-006e-a76fcc4ba9e4 Address:c0207010-c564-e72f-006e-a76fcc4ba9e4} {Suffrage:Voter ID:server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d Address:ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d} {Suffrage:Voter ID:server-7731092c-1bb9-a1a7-f8d8-14c99df84e0e Address:7731092c-1bb9-a1a7-f8d8-14c99df84e0e} {Suffrage:Voter ID:server-8c2b8647-cd78-d3a2-15db-31bb4a415706 Address:8c2b8647-cd78-d3a2-15db-31bb4a415706}]"
2025-01-25T14:29:51.646Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1ef68a1-157e-004f-2992-86f439700735 Address:b1ef68a1-157e-004f-2992-86f439700735} {Suffrage:Voter ID:server-63dfeab7-b1db-8c9d-e8ff-9e5a455187ef Address:63dfeab7-b1db-8c9d-e8ff-9e5a455187ef} {Suffrage:Voter ID:server-8a095541-f276-2986-14c9-f10f3a3f537b Address:8a095541-f276-2986-14c9-f10f3a3f537b} {Suffrage:Voter ID:server-c0207010-c564-e72f-006e-a76fcc4ba9e4 Address:c0207010-c564-e72f-006e-a76fcc4ba9e4} {Suffrage:Voter ID:server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d Address:ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d} {Suffrage:Voter ID:server-7731092c-1bb9-a1a7-f8d8-14c99df84e0e Address:7731092c-1bb9-a1a7-f8d8-14c99df84e0e} {Suffrage:Voter ID:server-8c2b8647-cd78-d3a2-15db-31bb4a415706 Address:8c2b8647-cd78-d3a2-15db-31bb4a415706}]"
2025-01-25T14:29:51.646Z [INFO]  server-7731092c-1bb9-a1a7-f8d8-14c99df84e0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1ef68a1-157e-004f-2992-86f439700735 Address:b1ef68a1-157e-004f-2992-86f439700735} {Suffrage:Voter ID:server-63dfeab7-b1db-8c9d-e8ff-9e5a455187ef Address:63dfeab7-b1db-8c9d-e8ff-9e5a455187ef} {Suffrage:Voter ID:server-8a095541-f276-2986-14c9-f10f3a3f537b Address:8a095541-f276-2986-14c9-f10f3a3f537b} {Suffrage:Voter ID:server-c0207010-c564-e72f-006e-a76fcc4ba9e4 Address:c0207010-c564-e72f-006e-a76fcc4ba9e4} {Suffrage:Voter ID:server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d Address:ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d} {Suffrage:Voter ID:server-7731092c-1bb9-a1a7-f8d8-14c99df84e0e Address:7731092c-1bb9-a1a7-f8d8-14c99df84e0e} {Suffrage:Voter ID:server-8c2b8647-cd78-d3a2-15db-31bb4a415706 Address:8c2b8647-cd78-d3a2-15db-31bb4a415706}]"
2025-01-25T14:29:51.647Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: entering follower state: follower="Node at b1ef68a1-157e-004f-2992-86f439700735 [Follower]" leader-address= leader-id=
2025-01-25T14:29:51.647Z [INFO]  server-63dfeab7-b1db-8c9d-e8ff-9e5a455187ef: entering follower state: follower="Node at 63dfeab7-b1db-8c9d-e8ff-9e5a455187ef [Follower]" leader-address= leader-id=
2025-01-25T14:29:51.647Z [INFO]  server-8a095541-f276-2986-14c9-f10f3a3f537b: entering follower state: follower="Node at 8a095541-f276-2986-14c9-f10f3a3f537b [Follower]" leader-address= leader-id=
2025-01-25T14:29:51.647Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: entering follower state: follower="Node at ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d [Follower]" leader-address= leader-id=
2025-01-25T14:29:51.647Z [INFO]  server-c0207010-c564-e72f-006e-a76fcc4ba9e4: entering follower state: follower="Node at c0207010-c564-e72f-006e-a76fcc4ba9e4 [Follower]" leader-address= leader-id=
2025-01-25T14:29:51.647Z [INFO]  server-7731092c-1bb9-a1a7-f8d8-14c99df84e0e: entering follower state: follower="Node at 7731092c-1bb9-a1a7-f8d8-14c99df84e0e [Follower]" leader-address= leader-id=
2025-01-25T14:29:51.648Z [INFO]  server-8c2b8647-cd78-d3a2-15db-31bb4a415706: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1ef68a1-157e-004f-2992-86f439700735 Address:b1ef68a1-157e-004f-2992-86f439700735} {Suffrage:Voter ID:server-63dfeab7-b1db-8c9d-e8ff-9e5a455187ef Address:63dfeab7-b1db-8c9d-e8ff-9e5a455187ef} {Suffrage:Voter ID:server-8a095541-f276-2986-14c9-f10f3a3f537b Address:8a095541-f276-2986-14c9-f10f3a3f537b} {Suffrage:Voter ID:server-c0207010-c564-e72f-006e-a76fcc4ba9e4 Address:c0207010-c564-e72f-006e-a76fcc4ba9e4} {Suffrage:Voter ID:server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d Address:ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d} {Suffrage:Voter ID:server-7731092c-1bb9-a1a7-f8d8-14c99df84e0e Address:7731092c-1bb9-a1a7-f8d8-14c99df84e0e} {Suffrage:Voter ID:server-8c2b8647-cd78-d3a2-15db-31bb4a415706 Address:8c2b8647-cd78-d3a2-15db-31bb4a415706}]"
2025-01-25T14:29:51.648Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:51.648Z [INFO]  server-8c2b8647-cd78-d3a2-15db-31bb4a415706: entering follower state: follower="Node at 8c2b8647-cd78-d3a2-15db-31bb4a415706 [Follower]" leader-address= leader-id=
2025-01-25T14:29:51.698Z [WARN]  server-8a095541-f276-2986-14c9-f10f3a3f537b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:51.698Z [WARN]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:51.698Z [INFO]  server-8a095541-f276-2986-14c9-f10f3a3f537b: entering candidate state: node="Node at 8a095541-f276-2986-14c9-f10f3a3f537b [Candidate]" term=2
2025-01-25T14:29:51.698Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: entering candidate state: node="Node at ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d [Candidate]" term=2
2025-01-25T14:29:51.698Z [INFO]  server-8c2b8647-cd78-d3a2-15db-31bb4a415706: duplicate requestVote for same term: term=2
2025-01-25T14:29:51.699Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: duplicate requestVote for same term: term=2
2025-01-25T14:29:51.699Z [INFO]  server-8a095541-f276-2986-14c9-f10f3a3f537b: duplicate requestVote for same term: term=2
2025-01-25T14:29:51.699Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: election won: term=2 tally=4
2025-01-25T14:29:51.699Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: entering leader state: leader="Node at ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d [Leader]"
2025-01-25T14:29:51.699Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: added peer, starting replication: peer=server-b1ef68a1-157e-004f-2992-86f439700735
2025-01-25T14:29:51.699Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: added peer, starting replication: peer=server-63dfeab7-b1db-8c9d-e8ff-9e5a455187ef
2025-01-25T14:29:51.699Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: added peer, starting replication: peer=server-8a095541-f276-2986-14c9-f10f3a3f537b
2025-01-25T14:29:51.699Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: added peer, starting replication: peer=server-c0207010-c564-e72f-006e-a76fcc4ba9e4
2025-01-25T14:29:51.699Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: added peer, starting replication: peer=server-7731092c-1bb9-a1a7-f8d8-14c99df84e0e
2025-01-25T14:29:51.699Z [INFO]  server-63dfeab7-b1db-8c9d-e8ff-9e5a455187ef: duplicate requestVote for same term: term=2
2025-01-25T14:29:51.699Z [INFO]  server-c0207010-c564-e72f-006e-a76fcc4ba9e4: duplicate requestVote for same term: term=2
2025-01-25T14:29:51.699Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: added peer, starting replication: peer=server-8c2b8647-cd78-d3a2-15db-31bb4a415706
2025-01-25T14:29:51.699Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: pipelining replication: peer="{Voter server-b1ef68a1-157e-004f-2992-86f439700735 b1ef68a1-157e-004f-2992-86f439700735}"
2025-01-25T14:29:51.699Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: pipelining replication: peer="{Voter server-8c2b8647-cd78-d3a2-15db-31bb4a415706 8c2b8647-cd78-d3a2-15db-31bb4a415706}"
2025-01-25T14:29:51.699Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: pipelining replication: peer="{Voter server-63dfeab7-b1db-8c9d-e8ff-9e5a455187ef 63dfeab7-b1db-8c9d-e8ff-9e5a455187ef}"
2025-01-25T14:29:51.699Z [WARN]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: rejecting vote request since we have a leader: from=8a095541-f276-2986-14c9-f10f3a3f537b leader=ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d leader-id=server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d
2025-01-25T14:29:51.699Z [INFO]  server-7731092c-1bb9-a1a7-f8d8-14c99df84e0e: duplicate requestVote for same term: term=2
2025-01-25T14:29:51.700Z [INFO]  server-8a095541-f276-2986-14c9-f10f3a3f537b: entering follower state: follower="Node at 8a095541-f276-2986-14c9-f10f3a3f537b [Follower]" leader-address=ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d leader-id=server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d
2025-01-25T14:29:51.700Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: pipelining replication: peer="{Voter server-7731092c-1bb9-a1a7-f8d8-14c99df84e0e 7731092c-1bb9-a1a7-f8d8-14c99df84e0e}"
2025-01-25T14:29:51.700Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: pipelining replication: peer="{Voter server-c0207010-c564-e72f-006e-a76fcc4ba9e4 c0207010-c564-e72f-006e-a76fcc4ba9e4}"
2025-01-25T14:29:51.700Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: pipelining replication: peer="{Voter server-8a095541-f276-2986-14c9-f10f3a3f537b 8a095541-f276-2986-14c9-f10f3a3f537b}"
2025-01-25T14:29:51.805Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:51.700383072 +0000 UTC m=+101.554708054 (1 nodes), 51.805336ms from start of poll, 55.322569ms from cluster start. Timeout at 2025-01-25 14:29:51.805385512 +0000 UTC m=+101.659710494, 105.00244ms after stability
2025-01-25T14:29:51.805Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:51.911Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:51.805791358 +0000 UTC m=+101.660116340 (6 nodes), 3.84µs from start of poll, 160.730855ms from cluster start. Timeout at 2025-01-25 14:29:51.911079228 +0000 UTC m=+101.765404170, 105.28783ms after stability
2025-01-25T14:29:51.911Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:52.017Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:51.911826622 +0000 UTC m=+101.766151604 (1 nodes), 1.92µs from start of poll, 266.766119ms from cluster start. Timeout at 2025-01-25 14:29:52.017041054 +0000 UTC m=+101.871366036, 105.214432ms after stability
2025-01-25T14:29:52.017Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: entering candidate state: node="Node at b1ef68a1-157e-004f-2992-86f439700735 [Candidate]" term=3
2025-01-25T14:29:52.017Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:52.018Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: entering follower state: follower="Node at ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d [Follower]" leader-address= leader-id=
2025-01-25T14:29:52.018Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: aborting pipeline replication: peer="{Voter server-b1ef68a1-157e-004f-2992-86f439700735 b1ef68a1-157e-004f-2992-86f439700735}"
2025-01-25T14:29:52.018Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: election won: term=3 tally=4
2025-01-25T14:29:52.018Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: entering leader state: leader="Node at b1ef68a1-157e-004f-2992-86f439700735 [Leader]"
2025-01-25T14:29:52.018Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: added peer, starting replication: peer=server-63dfeab7-b1db-8c9d-e8ff-9e5a455187ef
2025-01-25T14:29:52.018Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: added peer, starting replication: peer=server-8a095541-f276-2986-14c9-f10f3a3f537b
2025-01-25T14:29:52.018Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: added peer, starting replication: peer=server-c0207010-c564-e72f-006e-a76fcc4ba9e4
2025-01-25T14:29:52.018Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: added peer, starting replication: peer=server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d
2025-01-25T14:29:52.018Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: added peer, starting replication: peer=server-7731092c-1bb9-a1a7-f8d8-14c99df84e0e
2025-01-25T14:29:52.018Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: added peer, starting replication: peer=server-8c2b8647-cd78-d3a2-15db-31bb4a415706
2025-01-25T14:29:52.019Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: aborting pipeline replication: peer="{Voter server-63dfeab7-b1db-8c9d-e8ff-9e5a455187ef 63dfeab7-b1db-8c9d-e8ff-9e5a455187ef}"
2025-01-25T14:29:52.019Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: aborting pipeline replication: peer="{Voter server-8a095541-f276-2986-14c9-f10f3a3f537b 8a095541-f276-2986-14c9-f10f3a3f537b}"
2025-01-25T14:29:52.019Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: aborting pipeline replication: peer="{Voter server-c0207010-c564-e72f-006e-a76fcc4ba9e4 c0207010-c564-e72f-006e-a76fcc4ba9e4}"
2025-01-25T14:29:52.019Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: aborting pipeline replication: peer="{Voter server-7731092c-1bb9-a1a7-f8d8-14c99df84e0e 7731092c-1bb9-a1a7-f8d8-14c99df84e0e}"
2025-01-25T14:29:52.019Z [INFO]  server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d: aborting pipeline replication: peer="{Voter server-8c2b8647-cd78-d3a2-15db-31bb4a415706 8c2b8647-cd78-d3a2-15db-31bb4a415706}"
2025-01-25T14:29:52.019Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: pipelining replication: peer="{Voter server-c0207010-c564-e72f-006e-a76fcc4ba9e4 c0207010-c564-e72f-006e-a76fcc4ba9e4}"
2025-01-25T14:29:52.019Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: pipelining replication: peer="{Voter server-7731092c-1bb9-a1a7-f8d8-14c99df84e0e 7731092c-1bb9-a1a7-f8d8-14c99df84e0e}"
2025-01-25T14:29:52.020Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: pipelining replication: peer="{Voter server-8c2b8647-cd78-d3a2-15db-31bb4a415706 8c2b8647-cd78-d3a2-15db-31bb4a415706}"
2025-01-25T14:29:52.021Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: pipelining replication: peer="{Voter server-63dfeab7-b1db-8c9d-e8ff-9e5a455187ef 63dfeab7-b1db-8c9d-e8ff-9e5a455187ef}"
2025-01-25T14:29:52.021Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: pipelining replication: peer="{Voter server-8a095541-f276-2986-14c9-f10f3a3f537b 8a095541-f276-2986-14c9-f10f3a3f537b}"
2025-01-25T14:29:52.021Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: pipelining replication: peer="{Voter server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d}"
2025-01-25T14:29:52.124Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:52.018952447 +0000 UTC m=+101.873277389 (1 nodes), 993.204µs from start of poll, 373.891904ms from cluster start. Timeout at 2025-01-25 14:29:52.123985045 +0000 UTC m=+101.978309987, 105.032598ms after stability
2025-01-25T14:29:52.125Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: aborting pipeline replication: peer="{Voter server-8a095541-f276-2986-14c9-f10f3a3f537b 8a095541-f276-2986-14c9-f10f3a3f537b}"
2025-01-25T14:29:52.126Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: aborting pipeline replication: peer="{Voter server-8c2b8647-cd78-d3a2-15db-31bb4a415706 8c2b8647-cd78-d3a2-15db-31bb4a415706}"
2025-01-25T14:29:52.126Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: aborting pipeline replication: peer="{Voter server-ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d ef6fc62f-910c-6fae-4bd8-d03f5ebdc08d}"
2025-01-25T14:29:52.126Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: aborting pipeline replication: peer="{Voter server-7731092c-1bb9-a1a7-f8d8-14c99df84e0e 7731092c-1bb9-a1a7-f8d8-14c99df84e0e}"
2025-01-25T14:29:52.126Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: aborting pipeline replication: peer="{Voter server-c0207010-c564-e72f-006e-a76fcc4ba9e4 c0207010-c564-e72f-006e-a76fcc4ba9e4}"
2025-01-25T14:29:52.127Z [INFO]  server-b1ef68a1-157e-004f-2992-86f439700735: aborting pipeline replication: peer="{Voter server-63dfeab7-b1db-8c9d-e8ff-9e5a455187ef 63dfeab7-b1db-8c9d-e8ff-9e5a455187ef}"
2025-01-25T14:29:52.626Z [ERROR] server-b1ef68a1-157e-004f-2992-86f439700735: failed to heartbeat to: peer=7731092c-1bb9-a1a7-f8d8-14c99df84e0e backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.99s)
=== RUN   TestRaft_LeadershipTransferToInvalidID
2025-01-25T14:29:52.630Z [INFO]  server-fa0bc42c-ea77-39ac-5ad5-e2f983e94abd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa0bc42c-ea77-39ac-5ad5-e2f983e94abd Address:fa0bc42c-ea77-39ac-5ad5-e2f983e94abd} {Suffrage:Voter ID:server-cb47b2e9-0a53-82d1-9818-08887336521b Address:cb47b2e9-0a53-82d1-9818-08887336521b} {Suffrage:Voter ID:server-34148589-7b42-c3d5-7425-c8788b1e81d4 Address:34148589-7b42-c3d5-7425-c8788b1e81d4}]"
2025-01-25T14:29:52.630Z [INFO]  server-fa0bc42c-ea77-39ac-5ad5-e2f983e94abd: entering follower state: follower="Node at fa0bc42c-ea77-39ac-5ad5-e2f983e94abd [Follower]" leader-address= leader-id=
2025-01-25T14:29:52.630Z [INFO]  server-cb47b2e9-0a53-82d1-9818-08887336521b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa0bc42c-ea77-39ac-5ad5-e2f983e94abd Address:fa0bc42c-ea77-39ac-5ad5-e2f983e94abd} {Suffrage:Voter ID:server-cb47b2e9-0a53-82d1-9818-08887336521b Address:cb47b2e9-0a53-82d1-9818-08887336521b} {Suffrage:Voter ID:server-34148589-7b42-c3d5-7425-c8788b1e81d4 Address:34148589-7b42-c3d5-7425-c8788b1e81d4}]"
2025-01-25T14:29:52.630Z [INFO]  server-cb47b2e9-0a53-82d1-9818-08887336521b: entering follower state: follower="Node at cb47b2e9-0a53-82d1-9818-08887336521b [Follower]" leader-address= leader-id=
2025-01-25T14:29:52.630Z [INFO]  server-34148589-7b42-c3d5-7425-c8788b1e81d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa0bc42c-ea77-39ac-5ad5-e2f983e94abd Address:fa0bc42c-ea77-39ac-5ad5-e2f983e94abd} {Suffrage:Voter ID:server-cb47b2e9-0a53-82d1-9818-08887336521b Address:cb47b2e9-0a53-82d1-9818-08887336521b} {Suffrage:Voter ID:server-34148589-7b42-c3d5-7425-c8788b1e81d4 Address:34148589-7b42-c3d5-7425-c8788b1e81d4}]"
2025-01-25T14:29:52.630Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:52.630Z [INFO]  server-34148589-7b42-c3d5-7425-c8788b1e81d4: entering follower state: follower="Node at 34148589-7b42-c3d5-7425-c8788b1e81d4 [Follower]" leader-address= leader-id=
2025-01-25T14:29:52.695Z [WARN]  server-cb47b2e9-0a53-82d1-9818-08887336521b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:52.695Z [INFO]  server-cb47b2e9-0a53-82d1-9818-08887336521b: entering candidate state: node="Node at cb47b2e9-0a53-82d1-9818-08887336521b [Candidate]" term=2
2025-01-25T14:29:52.695Z [INFO]  server-cb47b2e9-0a53-82d1-9818-08887336521b: election won: term=2 tally=2
2025-01-25T14:29:52.695Z [INFO]  server-cb47b2e9-0a53-82d1-9818-08887336521b: entering leader state: leader="Node at cb47b2e9-0a53-82d1-9818-08887336521b [Leader]"
2025-01-25T14:29:52.695Z [INFO]  server-cb47b2e9-0a53-82d1-9818-08887336521b: added peer, starting replication: peer=server-fa0bc42c-ea77-39ac-5ad5-e2f983e94abd
2025-01-25T14:29:52.695Z [INFO]  server-cb47b2e9-0a53-82d1-9818-08887336521b: added peer, starting replication: peer=server-34148589-7b42-c3d5-7425-c8788b1e81d4
2025-01-25T14:29:52.695Z [INFO]  server-cb47b2e9-0a53-82d1-9818-08887336521b: pipelining replication: peer="{Voter server-34148589-7b42-c3d5-7425-c8788b1e81d4 34148589-7b42-c3d5-7425-c8788b1e81d4}"
2025-01-25T14:29:52.696Z [INFO]  server-cb47b2e9-0a53-82d1-9818-08887336521b: pipelining replication: peer="{Voter server-fa0bc42c-ea77-39ac-5ad5-e2f983e94abd fa0bc42c-ea77-39ac-5ad5-e2f983e94abd}"
2025-01-25T14:29:52.801Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:52.695890886 +0000 UTC m=+102.550215868 (1 nodes), 65.205373ms from start of poll, 65.886989ms from cluster start. Timeout at 2025-01-25 14:29:52.800894406 +0000 UTC m=+102.655219388, 105.00352ms after stability
2025-01-25T14:29:52.801Z [INFO]  server-cb47b2e9-0a53-82d1-9818-08887336521b: aborting pipeline replication: peer="{Voter server-34148589-7b42-c3d5-7425-c8788b1e81d4 34148589-7b42-c3d5-7425-c8788b1e81d4}"
2025-01-25T14:29:52.801Z [INFO]  server-cb47b2e9-0a53-82d1-9818-08887336521b: aborting pipeline replication: peer="{Voter server-fa0bc42c-ea77-39ac-5ad5-e2f983e94abd fa0bc42c-ea77-39ac-5ad5-e2f983e94abd}"
--- PASS: TestRaft_LeadershipTransferToInvalidID (0.17s)
=== RUN   TestRaft_LeadershipTransferToInvalidAddress
2025-01-25T14:29:52.804Z [INFO]  server-43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a Address:43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a} {Suffrage:Voter ID:server-275a0f32-4be4-3289-2502-faa6a76e0355 Address:275a0f32-4be4-3289-2502-faa6a76e0355} {Suffrage:Voter ID:server-07525e83-5e99-69ea-b3e6-2bff0c91ebe6 Address:07525e83-5e99-69ea-b3e6-2bff0c91ebe6}]"
2025-01-25T14:29:52.804Z [INFO]  server-275a0f32-4be4-3289-2502-faa6a76e0355: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a Address:43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a} {Suffrage:Voter ID:server-275a0f32-4be4-3289-2502-faa6a76e0355 Address:275a0f32-4be4-3289-2502-faa6a76e0355} {Suffrage:Voter ID:server-07525e83-5e99-69ea-b3e6-2bff0c91ebe6 Address:07525e83-5e99-69ea-b3e6-2bff0c91ebe6}]"
2025-01-25T14:29:52.804Z [INFO]  server-07525e83-5e99-69ea-b3e6-2bff0c91ebe6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a Address:43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a} {Suffrage:Voter ID:server-275a0f32-4be4-3289-2502-faa6a76e0355 Address:275a0f32-4be4-3289-2502-faa6a76e0355} {Suffrage:Voter ID:server-07525e83-5e99-69ea-b3e6-2bff0c91ebe6 Address:07525e83-5e99-69ea-b3e6-2bff0c91ebe6}]"
2025-01-25T14:29:52.804Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:52.804Z [INFO]  server-43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a: entering follower state: follower="Node at 43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a [Follower]" leader-address= leader-id=
2025-01-25T14:29:52.805Z [INFO]  server-275a0f32-4be4-3289-2502-faa6a76e0355: entering follower state: follower="Node at 275a0f32-4be4-3289-2502-faa6a76e0355 [Follower]" leader-address= leader-id=
2025-01-25T14:29:52.805Z [INFO]  server-07525e83-5e99-69ea-b3e6-2bff0c91ebe6: entering follower state: follower="Node at 07525e83-5e99-69ea-b3e6-2bff0c91ebe6 [Follower]" leader-address= leader-id=
2025-01-25T14:29:52.894Z [WARN]  server-275a0f32-4be4-3289-2502-faa6a76e0355: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:52.894Z [INFO]  server-275a0f32-4be4-3289-2502-faa6a76e0355: entering candidate state: node="Node at 275a0f32-4be4-3289-2502-faa6a76e0355 [Candidate]" term=2
2025-01-25T14:29:52.894Z [INFO]  server-275a0f32-4be4-3289-2502-faa6a76e0355: election won: term=2 tally=2
2025-01-25T14:29:52.894Z [INFO]  server-275a0f32-4be4-3289-2502-faa6a76e0355: entering leader state: leader="Node at 275a0f32-4be4-3289-2502-faa6a76e0355 [Leader]"
2025-01-25T14:29:52.894Z [INFO]  server-275a0f32-4be4-3289-2502-faa6a76e0355: added peer, starting replication: peer=server-43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a
2025-01-25T14:29:52.894Z [INFO]  server-275a0f32-4be4-3289-2502-faa6a76e0355: added peer, starting replication: peer=server-07525e83-5e99-69ea-b3e6-2bff0c91ebe6
2025-01-25T14:29:52.894Z [INFO]  server-275a0f32-4be4-3289-2502-faa6a76e0355: pipelining replication: peer="{Voter server-07525e83-5e99-69ea-b3e6-2bff0c91ebe6 07525e83-5e99-69ea-b3e6-2bff0c91ebe6}"
2025-01-25T14:29:52.895Z [WARN]  server-43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:52.895Z [INFO]  server-43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a: entering candidate state: node="Node at 43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a [Candidate]" term=2
2025-01-25T14:29:52.895Z [INFO]  server-43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a: duplicate requestVote for same term: term=2
2025-01-25T14:29:52.895Z [WARN]  server-275a0f32-4be4-3289-2502-faa6a76e0355: rejecting vote request since we have a leader: from=43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a leader=275a0f32-4be4-3289-2502-faa6a76e0355 leader-id=server-275a0f32-4be4-3289-2502-faa6a76e0355
2025-01-25T14:29:52.895Z [INFO]  server-43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a: entering follower state: follower="Node at 43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a [Follower]" leader-address=275a0f32-4be4-3289-2502-faa6a76e0355 leader-id=server-275a0f32-4be4-3289-2502-faa6a76e0355
2025-01-25T14:29:52.895Z [WARN]  server-07525e83-5e99-69ea-b3e6-2bff0c91ebe6: rejecting vote request since we have a leader: from=43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a leader=275a0f32-4be4-3289-2502-faa6a76e0355 leader-id=server-275a0f32-4be4-3289-2502-faa6a76e0355
2025-01-25T14:29:52.895Z [INFO]  server-275a0f32-4be4-3289-2502-faa6a76e0355: pipelining replication: peer="{Voter server-43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a 43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a}"
2025-01-25T14:29:53.001Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:52.895814091 +0000 UTC m=+102.750139073 (2 nodes), 90.935367ms from start of poll, 91.629462ms from cluster start. Timeout at 2025-01-25 14:29:53.000840891 +0000 UTC m=+102.855165833, 105.02676ms after stability
2025-01-25T14:29:53.001Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:53.109Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:53.001128741 +0000 UTC m=+102.855453723 (1 nodes), 1.76µs from start of poll, 196.944112ms from cluster start. Timeout at 2025-01-25 14:29:53.106382092 +0000 UTC m=+102.960707194, 105.253471ms after stability
2025-01-25T14:29:53.109Z [INFO]  server-275a0f32-4be4-3289-2502-faa6a76e0355: aborting pipeline replication: peer="{Voter server-43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a 43e6afaa-ba92-b4a6-edc1-c76bc05ffb2a}"
2025-01-25T14:29:53.109Z [INFO]  server-275a0f32-4be4-3289-2502-faa6a76e0355: aborting pipeline replication: peer="{Voter server-07525e83-5e99-69ea-b3e6-2bff0c91ebe6 07525e83-5e99-69ea-b3e6-2bff0c91ebe6}"
--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.31s)
=== RUN   TestRaft_LeadershipTransferToBehindServer
2025-01-25T14:29:53.112Z [INFO]  server-3f106e1c-4394-bec0-f499-d1b4ddf4bd1d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f106e1c-4394-bec0-f499-d1b4ddf4bd1d Address:3f106e1c-4394-bec0-f499-d1b4ddf4bd1d} {Suffrage:Voter ID:server-67e445b5-76a8-7f31-47ae-597854d04e3d Address:67e445b5-76a8-7f31-47ae-597854d04e3d} {Suffrage:Voter ID:server-b0e3192f-d65a-3d81-283a-fef84d9d77b4 Address:b0e3192f-d65a-3d81-283a-fef84d9d77b4}]"
2025-01-25T14:29:53.112Z [INFO]  server-67e445b5-76a8-7f31-47ae-597854d04e3d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f106e1c-4394-bec0-f499-d1b4ddf4bd1d Address:3f106e1c-4394-bec0-f499-d1b4ddf4bd1d} {Suffrage:Voter ID:server-67e445b5-76a8-7f31-47ae-597854d04e3d Address:67e445b5-76a8-7f31-47ae-597854d04e3d} {Suffrage:Voter ID:server-b0e3192f-d65a-3d81-283a-fef84d9d77b4 Address:b0e3192f-d65a-3d81-283a-fef84d9d77b4}]"
2025-01-25T14:29:53.113Z [INFO]  server-67e445b5-76a8-7f31-47ae-597854d04e3d: entering follower state: follower="Node at 67e445b5-76a8-7f31-47ae-597854d04e3d [Follower]" leader-address= leader-id=
2025-01-25T14:29:53.112Z [INFO]  server-3f106e1c-4394-bec0-f499-d1b4ddf4bd1d: entering follower state: follower="Node at 3f106e1c-4394-bec0-f499-d1b4ddf4bd1d [Follower]" leader-address= leader-id=
2025-01-25T14:29:53.113Z [INFO]  server-b0e3192f-d65a-3d81-283a-fef84d9d77b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f106e1c-4394-bec0-f499-d1b4ddf4bd1d Address:3f106e1c-4394-bec0-f499-d1b4ddf4bd1d} {Suffrage:Voter ID:server-67e445b5-76a8-7f31-47ae-597854d04e3d Address:67e445b5-76a8-7f31-47ae-597854d04e3d} {Suffrage:Voter ID:server-b0e3192f-d65a-3d81-283a-fef84d9d77b4 Address:b0e3192f-d65a-3d81-283a-fef84d9d77b4}]"
2025-01-25T14:29:53.113Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:53.113Z [INFO]  server-b0e3192f-d65a-3d81-283a-fef84d9d77b4: entering follower state: follower="Node at b0e3192f-d65a-3d81-283a-fef84d9d77b4 [Follower]" leader-address= leader-id=
2025-01-25T14:29:53.169Z [WARN]  server-b0e3192f-d65a-3d81-283a-fef84d9d77b4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:53.170Z [INFO]  server-b0e3192f-d65a-3d81-283a-fef84d9d77b4: entering candidate state: node="Node at b0e3192f-d65a-3d81-283a-fef84d9d77b4 [Candidate]" term=2
2025-01-25T14:29:53.170Z [INFO]  server-b0e3192f-d65a-3d81-283a-fef84d9d77b4: election won: term=2 tally=2
2025-01-25T14:29:53.170Z [INFO]  server-b0e3192f-d65a-3d81-283a-fef84d9d77b4: entering leader state: leader="Node at b0e3192f-d65a-3d81-283a-fef84d9d77b4 [Leader]"
2025-01-25T14:29:53.170Z [INFO]  server-b0e3192f-d65a-3d81-283a-fef84d9d77b4: added peer, starting replication: peer=server-3f106e1c-4394-bec0-f499-d1b4ddf4bd1d
2025-01-25T14:29:53.170Z [INFO]  server-b0e3192f-d65a-3d81-283a-fef84d9d77b4: added peer, starting replication: peer=server-67e445b5-76a8-7f31-47ae-597854d04e3d
2025-01-25T14:29:53.171Z [INFO]  server-b0e3192f-d65a-3d81-283a-fef84d9d77b4: pipelining replication: peer="{Voter server-67e445b5-76a8-7f31-47ae-597854d04e3d 67e445b5-76a8-7f31-47ae-597854d04e3d}"
2025-01-25T14:29:53.171Z [INFO]  server-b0e3192f-d65a-3d81-283a-fef84d9d77b4: pipelining replication: peer="{Voter server-3f106e1c-4394-bec0-f499-d1b4ddf4bd1d 3f106e1c-4394-bec0-f499-d1b4ddf4bd1d}"
2025-01-25T14:29:53.276Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:53.171477051 +0000 UTC m=+103.025802033 (1 nodes), 58.192946ms from start of poll, 58.981518ms from cluster start. Timeout at 2025-01-25 14:29:53.276479371 +0000 UTC m=+103.130804353, 105.00232ms after stability
2025-01-25T14:29:53.276Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:53.382Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:53.276642106 +0000 UTC m=+103.130967088 (2 nodes), 1.68µs from start of poll, 164.146573ms from cluster start. Timeout at 2025-01-25 14:29:53.381924256 +0000 UTC m=+103.236249198, 105.28211ms after stability
2025-01-25T14:29:53.534Z [INFO]  server-3f106e1c-4394-bec0-f499-d1b4ddf4bd1d: entering candidate state: node="Node at 3f106e1c-4394-bec0-f499-d1b4ddf4bd1d [Candidate]" term=3
2025-01-25T14:29:53.534Z [INFO]  server-b0e3192f-d65a-3d81-283a-fef84d9d77b4: entering follower state: follower="Node at b0e3192f-d65a-3d81-283a-fef84d9d77b4 [Follower]" leader-address= leader-id=
2025-01-25T14:29:53.534Z [INFO]  server-b0e3192f-d65a-3d81-283a-fef84d9d77b4: aborting pipeline replication: peer="{Voter server-67e445b5-76a8-7f31-47ae-597854d04e3d 67e445b5-76a8-7f31-47ae-597854d04e3d}"
2025-01-25T14:29:53.534Z [INFO]  server-3f106e1c-4394-bec0-f499-d1b4ddf4bd1d: election won: term=3 tally=2
2025-01-25T14:29:53.534Z [INFO]  server-3f106e1c-4394-bec0-f499-d1b4ddf4bd1d: entering leader state: leader="Node at 3f106e1c-4394-bec0-f499-d1b4ddf4bd1d [Leader]"
2025-01-25T14:29:53.535Z [INFO]  server-3f106e1c-4394-bec0-f499-d1b4ddf4bd1d: added peer, starting replication: peer=server-67e445b5-76a8-7f31-47ae-597854d04e3d
2025-01-25T14:29:53.535Z [INFO]  server-3f106e1c-4394-bec0-f499-d1b4ddf4bd1d: added peer, starting replication: peer=server-b0e3192f-d65a-3d81-283a-fef84d9d77b4
2025-01-25T14:29:53.535Z [INFO]  server-b0e3192f-d65a-3d81-283a-fef84d9d77b4: aborting pipeline replication: peer="{Voter server-3f106e1c-4394-bec0-f499-d1b4ddf4bd1d 3f106e1c-4394-bec0-f499-d1b4ddf4bd1d}"
2025-01-25T14:29:53.535Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:53.535Z [INFO]  server-3f106e1c-4394-bec0-f499-d1b4ddf4bd1d: pipelining replication: peer="{Voter server-67e445b5-76a8-7f31-47ae-597854d04e3d 67e445b5-76a8-7f31-47ae-597854d04e3d}"
2025-01-25T14:29:53.535Z [INFO]  server-3f106e1c-4394-bec0-f499-d1b4ddf4bd1d: pipelining replication: peer="{Voter server-b0e3192f-d65a-3d81-283a-fef84d9d77b4 b0e3192f-d65a-3d81-283a-fef84d9d77b4}"
2025-01-25T14:29:53.640Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:53.535462672 +0000 UTC m=+103.389787654 (1 nodes), 54.598µs from start of poll, 422.967139ms from cluster start. Timeout at 2025-01-25 14:29:53.640637626 +0000 UTC m=+103.494962768, 105.175114ms after stability
2025-01-25T14:29:53.640Z [INFO]  server-3f106e1c-4394-bec0-f499-d1b4ddf4bd1d: aborting pipeline replication: peer="{Voter server-b0e3192f-d65a-3d81-283a-fef84d9d77b4 b0e3192f-d65a-3d81-283a-fef84d9d77b4}"
2025-01-25T14:29:53.640Z [INFO]  server-3f106e1c-4394-bec0-f499-d1b4ddf4bd1d: aborting pipeline replication: peer="{Voter server-67e445b5-76a8-7f31-47ae-597854d04e3d 67e445b5-76a8-7f31-47ae-597854d04e3d}"
--- PASS: TestRaft_LeadershipTransferToBehindServer (0.53s)
=== RUN   TestRaft_LeadershipTransferToItself
2025-01-25T14:29:53.644Z [INFO]  server-229aa372-85fe-e732-e6ba-ebb33cc9ed7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-229aa372-85fe-e732-e6ba-ebb33cc9ed7f Address:229aa372-85fe-e732-e6ba-ebb33cc9ed7f} {Suffrage:Voter ID:server-b07a38de-155a-2936-1c63-8fa8864b9eb5 Address:b07a38de-155a-2936-1c63-8fa8864b9eb5} {Suffrage:Voter ID:server-e225ea80-f715-4bd1-9db0-6ff70c578a05 Address:e225ea80-f715-4bd1-9db0-6ff70c578a05}]"
2025-01-25T14:29:53.645Z [INFO]  server-b07a38de-155a-2936-1c63-8fa8864b9eb5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-229aa372-85fe-e732-e6ba-ebb33cc9ed7f Address:229aa372-85fe-e732-e6ba-ebb33cc9ed7f} {Suffrage:Voter ID:server-b07a38de-155a-2936-1c63-8fa8864b9eb5 Address:b07a38de-155a-2936-1c63-8fa8864b9eb5} {Suffrage:Voter ID:server-e225ea80-f715-4bd1-9db0-6ff70c578a05 Address:e225ea80-f715-4bd1-9db0-6ff70c578a05}]"
2025-01-25T14:29:53.645Z [INFO]  server-e225ea80-f715-4bd1-9db0-6ff70c578a05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-229aa372-85fe-e732-e6ba-ebb33cc9ed7f Address:229aa372-85fe-e732-e6ba-ebb33cc9ed7f} {Suffrage:Voter ID:server-b07a38de-155a-2936-1c63-8fa8864b9eb5 Address:b07a38de-155a-2936-1c63-8fa8864b9eb5} {Suffrage:Voter ID:server-e225ea80-f715-4bd1-9db0-6ff70c578a05 Address:e225ea80-f715-4bd1-9db0-6ff70c578a05}]"
2025-01-25T14:29:53.645Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:53.645Z [INFO]  server-229aa372-85fe-e732-e6ba-ebb33cc9ed7f: entering follower state: follower="Node at 229aa372-85fe-e732-e6ba-ebb33cc9ed7f [Follower]" leader-address= leader-id=
2025-01-25T14:29:53.645Z [INFO]  server-b07a38de-155a-2936-1c63-8fa8864b9eb5: entering follower state: follower="Node at b07a38de-155a-2936-1c63-8fa8864b9eb5 [Follower]" leader-address= leader-id=
2025-01-25T14:29:53.653Z [INFO]  server-e225ea80-f715-4bd1-9db0-6ff70c578a05: entering follower state: follower="Node at e225ea80-f715-4bd1-9db0-6ff70c578a05 [Follower]" leader-address= leader-id=
2025-01-25T14:29:53.710Z [WARN]  server-229aa372-85fe-e732-e6ba-ebb33cc9ed7f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:53.711Z [INFO]  server-229aa372-85fe-e732-e6ba-ebb33cc9ed7f: entering candidate state: node="Node at 229aa372-85fe-e732-e6ba-ebb33cc9ed7f [Candidate]" term=2
2025-01-25T14:29:53.711Z [INFO]  server-229aa372-85fe-e732-e6ba-ebb33cc9ed7f: election won: term=2 tally=2
2025-01-25T14:29:53.711Z [INFO]  server-229aa372-85fe-e732-e6ba-ebb33cc9ed7f: entering leader state: leader="Node at 229aa372-85fe-e732-e6ba-ebb33cc9ed7f [Leader]"
2025-01-25T14:29:53.711Z [INFO]  server-229aa372-85fe-e732-e6ba-ebb33cc9ed7f: added peer, starting replication: peer=server-b07a38de-155a-2936-1c63-8fa8864b9eb5
2025-01-25T14:29:53.711Z [INFO]  server-229aa372-85fe-e732-e6ba-ebb33cc9ed7f: added peer, starting replication: peer=server-e225ea80-f715-4bd1-9db0-6ff70c578a05
2025-01-25T14:29:53.711Z [INFO]  server-229aa372-85fe-e732-e6ba-ebb33cc9ed7f: pipelining replication: peer="{Voter server-e225ea80-f715-4bd1-9db0-6ff70c578a05 e225ea80-f715-4bd1-9db0-6ff70c578a05}"
2025-01-25T14:29:53.712Z [INFO]  server-229aa372-85fe-e732-e6ba-ebb33cc9ed7f: pipelining replication: peer="{Voter server-b07a38de-155a-2936-1c63-8fa8864b9eb5 b07a38de-155a-2936-1c63-8fa8864b9eb5}"
2025-01-25T14:29:53.817Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:53.712147034 +0000 UTC m=+103.566472015 (1 nodes), 66.626442ms from start of poll, 67.421453ms from cluster start. Timeout at 2025-01-25 14:29:53.817167592 +0000 UTC m=+103.671492574, 105.020559ms after stability
2025-01-25T14:29:53.817Z [INFO]  server-229aa372-85fe-e732-e6ba-ebb33cc9ed7f: cannot transfer leadership to itself
2025-01-25T14:29:53.817Z [INFO]  server-229aa372-85fe-e732-e6ba-ebb33cc9ed7f: aborting pipeline replication: peer="{Voter server-e225ea80-f715-4bd1-9db0-6ff70c578a05 e225ea80-f715-4bd1-9db0-6ff70c578a05}"
2025-01-25T14:29:53.817Z [INFO]  server-229aa372-85fe-e732-e6ba-ebb33cc9ed7f: aborting pipeline replication: peer="{Voter server-b07a38de-155a-2936-1c63-8fa8864b9eb5 b07a38de-155a-2936-1c63-8fa8864b9eb5}"
--- PASS: TestRaft_LeadershipTransferToItself (0.18s)
=== RUN   TestRaft_LeadershipTransferLeaderRejectsClientRequests
2025-01-25T14:29:53.821Z [INFO]  server-39a99784-482f-8024-85d3-c62d58ad144c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-39a99784-482f-8024-85d3-c62d58ad144c Address:39a99784-482f-8024-85d3-c62d58ad144c} {Suffrage:Voter ID:server-1cfdfc44-782a-7f48-b01d-242982286025 Address:1cfdfc44-782a-7f48-b01d-242982286025} {Suffrage:Voter ID:server-fbd285d9-b5b5-0f40-0a5a-ec87214cadbb Address:fbd285d9-b5b5-0f40-0a5a-ec87214cadbb}]"
2025-01-25T14:29:53.821Z [INFO]  server-1cfdfc44-782a-7f48-b01d-242982286025: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-39a99784-482f-8024-85d3-c62d58ad144c Address:39a99784-482f-8024-85d3-c62d58ad144c} {Suffrage:Voter ID:server-1cfdfc44-782a-7f48-b01d-242982286025 Address:1cfdfc44-782a-7f48-b01d-242982286025} {Suffrage:Voter ID:server-fbd285d9-b5b5-0f40-0a5a-ec87214cadbb Address:fbd285d9-b5b5-0f40-0a5a-ec87214cadbb}]"
2025-01-25T14:29:53.821Z [INFO]  server-fbd285d9-b5b5-0f40-0a5a-ec87214cadbb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-39a99784-482f-8024-85d3-c62d58ad144c Address:39a99784-482f-8024-85d3-c62d58ad144c} {Suffrage:Voter ID:server-1cfdfc44-782a-7f48-b01d-242982286025 Address:1cfdfc44-782a-7f48-b01d-242982286025} {Suffrage:Voter ID:server-fbd285d9-b5b5-0f40-0a5a-ec87214cadbb Address:fbd285d9-b5b5-0f40-0a5a-ec87214cadbb}]"
2025-01-25T14:29:53.821Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:53.821Z [INFO]  server-39a99784-482f-8024-85d3-c62d58ad144c: entering follower state: follower="Node at 39a99784-482f-8024-85d3-c62d58ad144c [Follower]" leader-address= leader-id=
2025-01-25T14:29:53.821Z [INFO]  server-1cfdfc44-782a-7f48-b01d-242982286025: entering follower state: follower="Node at 1cfdfc44-782a-7f48-b01d-242982286025 [Follower]" leader-address= leader-id=
2025-01-25T14:29:53.821Z [INFO]  server-fbd285d9-b5b5-0f40-0a5a-ec87214cadbb: entering follower state: follower="Node at fbd285d9-b5b5-0f40-0a5a-ec87214cadbb [Follower]" leader-address= leader-id=
2025-01-25T14:29:53.876Z [WARN]  server-fbd285d9-b5b5-0f40-0a5a-ec87214cadbb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:53.876Z [INFO]  server-fbd285d9-b5b5-0f40-0a5a-ec87214cadbb: entering candidate state: node="Node at fbd285d9-b5b5-0f40-0a5a-ec87214cadbb [Candidate]" term=2
2025-01-25T14:29:53.876Z [INFO]  server-fbd285d9-b5b5-0f40-0a5a-ec87214cadbb: election won: term=2 tally=2
2025-01-25T14:29:53.876Z [INFO]  server-fbd285d9-b5b5-0f40-0a5a-ec87214cadbb: entering leader state: leader="Node at fbd285d9-b5b5-0f40-0a5a-ec87214cadbb [Leader]"
2025-01-25T14:29:53.876Z [INFO]  server-fbd285d9-b5b5-0f40-0a5a-ec87214cadbb: added peer, starting replication: peer=server-39a99784-482f-8024-85d3-c62d58ad144c
2025-01-25T14:29:53.876Z [INFO]  server-fbd285d9-b5b5-0f40-0a5a-ec87214cadbb: added peer, starting replication: peer=server-1cfdfc44-782a-7f48-b01d-242982286025
2025-01-25T14:29:53.876Z [INFO]  server-fbd285d9-b5b5-0f40-0a5a-ec87214cadbb: pipelining replication: peer="{Voter server-39a99784-482f-8024-85d3-c62d58ad144c 39a99784-482f-8024-85d3-c62d58ad144c}"
2025-01-25T14:29:53.876Z [INFO]  server-fbd285d9-b5b5-0f40-0a5a-ec87214cadbb: pipelining replication: peer="{Voter server-1cfdfc44-782a-7f48-b01d-242982286025 1cfdfc44-782a-7f48-b01d-242982286025}"
2025-01-25T14:29:53.988Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:53.876713551 +0000 UTC m=+103.731038533 (1 nodes), 55.058018ms from start of poll, 55.797712ms from cluster start. Timeout at 2025-01-25 14:29:53.981721911 +0000 UTC m=+103.836046893, 105.00836ms after stability
    raft_test.go:2457: waiting on future 0
    raft_test.go:2457: waiting on future 1
    raft_test.go:2457: waiting on future 2
    raft_test.go:2457: waiting on future 3
    raft_test.go:2457: waiting on future 4
    raft_test.go:2457: waiting on future 5
    raft_test.go:2457: waiting on future 6
2025-01-25T14:29:53.989Z [INFO]  server-fbd285d9-b5b5-0f40-0a5a-ec87214cadbb: aborting pipeline replication: peer="{Voter server-1cfdfc44-782a-7f48-b01d-242982286025 1cfdfc44-782a-7f48-b01d-242982286025}"
2025-01-25T14:29:53.989Z [INFO]  server-fbd285d9-b5b5-0f40-0a5a-ec87214cadbb: aborting pipeline replication: peer="{Voter server-39a99784-482f-8024-85d3-c62d58ad144c 39a99784-482f-8024-85d3-c62d58ad144c}"
--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.17s)
=== RUN   TestRaft_LeadershipTransferLeaderReplicationTimeout
2025-01-25T14:29:53.991Z [INFO]  server-b5e68fd4-766a-c95e-e2f7-4c6862d2c343: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5e68fd4-766a-c95e-e2f7-4c6862d2c343 Address:b5e68fd4-766a-c95e-e2f7-4c6862d2c343} {Suffrage:Voter ID:server-bae24b4a-8101-b75e-b80f-181136becf8b Address:bae24b4a-8101-b75e-b80f-181136becf8b} {Suffrage:Voter ID:server-c919d1a7-c505-c3db-d672-60269f366cf0 Address:c919d1a7-c505-c3db-d672-60269f366cf0}]"
2025-01-25T14:29:53.992Z [INFO]  server-bae24b4a-8101-b75e-b80f-181136becf8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5e68fd4-766a-c95e-e2f7-4c6862d2c343 Address:b5e68fd4-766a-c95e-e2f7-4c6862d2c343} {Suffrage:Voter ID:server-bae24b4a-8101-b75e-b80f-181136becf8b Address:bae24b4a-8101-b75e-b80f-181136becf8b} {Suffrage:Voter ID:server-c919d1a7-c505-c3db-d672-60269f366cf0 Address:c919d1a7-c505-c3db-d672-60269f366cf0}]"
2025-01-25T14:29:53.992Z [INFO]  server-c919d1a7-c505-c3db-d672-60269f366cf0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5e68fd4-766a-c95e-e2f7-4c6862d2c343 Address:b5e68fd4-766a-c95e-e2f7-4c6862d2c343} {Suffrage:Voter ID:server-bae24b4a-8101-b75e-b80f-181136becf8b Address:bae24b4a-8101-b75e-b80f-181136becf8b} {Suffrage:Voter ID:server-c919d1a7-c505-c3db-d672-60269f366cf0 Address:c919d1a7-c505-c3db-d672-60269f366cf0}]"
2025-01-25T14:29:53.992Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:53.992Z [INFO]  server-b5e68fd4-766a-c95e-e2f7-4c6862d2c343: entering follower state: follower="Node at b5e68fd4-766a-c95e-e2f7-4c6862d2c343 [Follower]" leader-address= leader-id=
2025-01-25T14:29:53.992Z [INFO]  server-bae24b4a-8101-b75e-b80f-181136becf8b: entering follower state: follower="Node at bae24b4a-8101-b75e-b80f-181136becf8b [Follower]" leader-address= leader-id=
2025-01-25T14:29:53.992Z [INFO]  server-c919d1a7-c505-c3db-d672-60269f366cf0: entering follower state: follower="Node at c919d1a7-c505-c3db-d672-60269f366cf0 [Follower]" leader-address= leader-id=
2025-01-25T14:29:54.047Z [WARN]  server-bae24b4a-8101-b75e-b80f-181136becf8b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:54.047Z [INFO]  server-bae24b4a-8101-b75e-b80f-181136becf8b: entering candidate state: node="Node at bae24b4a-8101-b75e-b80f-181136becf8b [Candidate]" term=2
2025-01-25T14:29:54.047Z [INFO]  server-bae24b4a-8101-b75e-b80f-181136becf8b: election won: term=2 tally=2
2025-01-25T14:29:54.047Z [INFO]  server-bae24b4a-8101-b75e-b80f-181136becf8b: entering leader state: leader="Node at bae24b4a-8101-b75e-b80f-181136becf8b [Leader]"
2025-01-25T14:29:54.047Z [INFO]  server-bae24b4a-8101-b75e-b80f-181136becf8b: added peer, starting replication: peer=server-b5e68fd4-766a-c95e-e2f7-4c6862d2c343
2025-01-25T14:29:54.047Z [INFO]  server-bae24b4a-8101-b75e-b80f-181136becf8b: added peer, starting replication: peer=server-c919d1a7-c505-c3db-d672-60269f366cf0
2025-01-25T14:29:54.048Z [INFO]  server-bae24b4a-8101-b75e-b80f-181136becf8b: pipelining replication: peer="{Voter server-b5e68fd4-766a-c95e-e2f7-4c6862d2c343 b5e68fd4-766a-c95e-e2f7-4c6862d2c343}"
2025-01-25T14:29:54.048Z [INFO]  server-bae24b4a-8101-b75e-b80f-181136becf8b: pipelining replication: peer="{Voter server-c919d1a7-c505-c3db-d672-60269f366cf0 c919d1a7-c505-c3db-d672-60269f366cf0}"
2025-01-25T14:29:54.152Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:54.047763796 +0000 UTC m=+103.902088778 (1 nodes), 55.349449ms from start of poll, 56.022184ms from cluster start. Timeout at 2025-01-25 14:29:54.152776634 +0000 UTC m=+104.007101616, 105.012838ms after stability
2025-01-25T14:29:54.153Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:54.264Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:54.153079045 +0000 UTC m=+104.007403987 (2 nodes), 2.119µs from start of poll, 161.337393ms from cluster start. Timeout at 2025-01-25 14:29:54.258348276 +0000 UTC m=+104.112673218, 105.269231ms after stability
2025-01-25T14:29:55.643Z [INFO]  server-bae24b4a-8101-b75e-b80f-181136becf8b: aborting pipeline replication: peer="{Voter server-c919d1a7-c505-c3db-d672-60269f366cf0 c919d1a7-c505-c3db-d672-60269f366cf0}"
2025-01-25T14:29:55.643Z [INFO]  server-bae24b4a-8101-b75e-b80f-181136becf8b: aborting pipeline replication: peer="{Voter server-b5e68fd4-766a-c95e-e2f7-4c6862d2c343 b5e68fd4-766a-c95e-e2f7-4c6862d2c343}"
2025-01-25T14:29:56.140Z [ERROR] server-bae24b4a-8101-b75e-b80f-181136becf8b: failed to heartbeat to: peer=c919d1a7-c505-c3db-d672-60269f366cf0 backoff time=10ms error="command timed out"
2025-01-25T14:29:56.144Z [ERROR] server-bae24b4a-8101-b75e-b80f-181136becf8b: failed to heartbeat to: peer=b5e68fd4-766a-c95e-e2f7-4c6862d2c343 backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (2.22s)
=== RUN   TestRaft_LeadershipTransferIgnoresNonvoters
2025-01-25T14:29:56.207Z [INFO]  server-cb4e076e-a78b-e0fb-3804-7ea5fddf3287: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb4e076e-a78b-e0fb-3804-7ea5fddf3287 Address:cb4e076e-a78b-e0fb-3804-7ea5fddf3287} {Suffrage:Voter ID:server-3b500cf7-47d8-babe-6855-2ef8dd28a6da Address:3b500cf7-47d8-babe-6855-2ef8dd28a6da}]"
2025-01-25T14:29:56.207Z [INFO]  server-cb4e076e-a78b-e0fb-3804-7ea5fddf3287: entering follower state: follower="Node at cb4e076e-a78b-e0fb-3804-7ea5fddf3287 [Follower]" leader-address= leader-id=
2025-01-25T14:29:56.207Z [INFO]  server-3b500cf7-47d8-babe-6855-2ef8dd28a6da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb4e076e-a78b-e0fb-3804-7ea5fddf3287 Address:cb4e076e-a78b-e0fb-3804-7ea5fddf3287} {Suffrage:Voter ID:server-3b500cf7-47d8-babe-6855-2ef8dd28a6da Address:3b500cf7-47d8-babe-6855-2ef8dd28a6da}]"
2025-01-25T14:29:56.207Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:56.207Z [INFO]  server-3b500cf7-47d8-babe-6855-2ef8dd28a6da: entering follower state: follower="Node at 3b500cf7-47d8-babe-6855-2ef8dd28a6da [Follower]" leader-address= leader-id=
2025-01-25T14:29:56.277Z [WARN]  server-3b500cf7-47d8-babe-6855-2ef8dd28a6da: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:56.278Z [INFO]  server-3b500cf7-47d8-babe-6855-2ef8dd28a6da: entering candidate state: node="Node at 3b500cf7-47d8-babe-6855-2ef8dd28a6da [Candidate]" term=2
2025-01-25T14:29:56.278Z [INFO]  server-3b500cf7-47d8-babe-6855-2ef8dd28a6da: election won: term=2 tally=2
2025-01-25T14:29:56.278Z [INFO]  server-3b500cf7-47d8-babe-6855-2ef8dd28a6da: entering leader state: leader="Node at 3b500cf7-47d8-babe-6855-2ef8dd28a6da [Leader]"
2025-01-25T14:29:56.278Z [INFO]  server-3b500cf7-47d8-babe-6855-2ef8dd28a6da: added peer, starting replication: peer=server-cb4e076e-a78b-e0fb-3804-7ea5fddf3287
2025-01-25T14:29:56.278Z [INFO]  server-3b500cf7-47d8-babe-6855-2ef8dd28a6da: pipelining replication: peer="{Voter server-cb4e076e-a78b-e0fb-3804-7ea5fddf3287 cb4e076e-a78b-e0fb-3804-7ea5fddf3287}"
2025-01-25T14:29:56.383Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:56.278334723 +0000 UTC m=+106.132659705 (1 nodes), 70.751494ms from start of poll, 71.315394ms from cluster start. Timeout at 2025-01-25 14:29:56.383337883 +0000 UTC m=+106.237662825, 105.00312ms after stability
2025-01-25T14:29:56.383Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:56.489Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:56.383859325 +0000 UTC m=+106.238184307 (1 nodes), 1.48µs from start of poll, 176.839996ms from cluster start. Timeout at 2025-01-25 14:29:56.489096036 +0000 UTC m=+106.343420978, 105.236671ms after stability
2025-01-25T14:29:56.490Z [INFO]  server-3b500cf7-47d8-babe-6855-2ef8dd28a6da: updating configuration: command=DemoteVoter server-id=server-cb4e076e-a78b-e0fb-3804-7ea5fddf3287 server-addr= servers="[{Suffrage:Nonvoter ID:server-cb4e076e-a78b-e0fb-3804-7ea5fddf3287 Address:cb4e076e-a78b-e0fb-3804-7ea5fddf3287} {Suffrage:Voter ID:server-3b500cf7-47d8-babe-6855-2ef8dd28a6da Address:3b500cf7-47d8-babe-6855-2ef8dd28a6da}]"
2025-01-25T14:29:56.490Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:56.596Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:56.490416771 +0000 UTC m=+106.344741713 (1 nodes), 1.24µs from start of poll, 283.397402ms from cluster start. Timeout at 2025-01-25 14:29:56.59573696 +0000 UTC m=+106.450061902, 105.320189ms after stability
2025-01-25T14:29:56.597Z [INFO]  server-3b500cf7-47d8-babe-6855-2ef8dd28a6da: aborting pipeline replication: peer="{Voter server-cb4e076e-a78b-e0fb-3804-7ea5fddf3287 cb4e076e-a78b-e0fb-3804-7ea5fddf3287}"
--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.39s)
=== RUN   TestRaft_LeadershipTransferStopRightAway
--- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s)
=== RUN   TestRaft_GetConfigurationNoBootstrap
2025-01-25T14:29:56.599Z [INFO]  server-f27a6ea9-eaf4-4af9-ed74-33118fe1544a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f27a6ea9-eaf4-4af9-ed74-33118fe1544a Address:f27a6ea9-eaf4-4af9-ed74-33118fe1544a} {Suffrage:Voter ID:server-c431c6c0-2334-02c8-cb62-9b03ddf0b899 Address:c431c6c0-2334-02c8-cb62-9b03ddf0b899}]"
2025-01-25T14:29:56.600Z [INFO]  server-c431c6c0-2334-02c8-cb62-9b03ddf0b899: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f27a6ea9-eaf4-4af9-ed74-33118fe1544a Address:f27a6ea9-eaf4-4af9-ed74-33118fe1544a} {Suffrage:Voter ID:server-c431c6c0-2334-02c8-cb62-9b03ddf0b899 Address:c431c6c0-2334-02c8-cb62-9b03ddf0b899}]"
2025-01-25T14:29:56.600Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:56.600Z [INFO]  server-f27a6ea9-eaf4-4af9-ed74-33118fe1544a: entering follower state: follower="Node at f27a6ea9-eaf4-4af9-ed74-33118fe1544a [Follower]" leader-address= leader-id=
2025-01-25T14:29:56.600Z [INFO]  server-c431c6c0-2334-02c8-cb62-9b03ddf0b899: entering follower state: follower="Node at c431c6c0-2334-02c8-cb62-9b03ddf0b899 [Follower]" leader-address= leader-id=
2025-01-25T14:29:56.698Z [WARN]  server-c431c6c0-2334-02c8-cb62-9b03ddf0b899: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:56.698Z [INFO]  server-c431c6c0-2334-02c8-cb62-9b03ddf0b899: entering candidate state: node="Node at c431c6c0-2334-02c8-cb62-9b03ddf0b899 [Candidate]" term=2
2025-01-25T14:29:56.698Z [INFO]  server-c431c6c0-2334-02c8-cb62-9b03ddf0b899: election won: term=2 tally=2
2025-01-25T14:29:56.698Z [INFO]  server-c431c6c0-2334-02c8-cb62-9b03ddf0b899: entering leader state: leader="Node at c431c6c0-2334-02c8-cb62-9b03ddf0b899 [Leader]"
2025-01-25T14:29:56.698Z [INFO]  server-c431c6c0-2334-02c8-cb62-9b03ddf0b899: added peer, starting replication: peer=server-f27a6ea9-eaf4-4af9-ed74-33118fe1544a
2025-01-25T14:29:56.698Z [INFO]  server-c431c6c0-2334-02c8-cb62-9b03ddf0b899: pipelining replication: peer="{Voter server-f27a6ea9-eaf4-4af9-ed74-33118fe1544a f27a6ea9-eaf4-4af9-ed74-33118fe1544a}"
2025-01-25T14:29:56.803Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:56.698518361 +0000 UTC m=+106.552843343 (1 nodes), 98.34514ms from start of poll, 98.865522ms from cluster start. Timeout at 2025-01-25 14:29:56.803520041 +0000 UTC m=+106.657845023, 105.00168ms after stability
2025-01-25T14:29:56.803Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:56.909Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:56.803716896 +0000 UTC m=+106.658041878 (1 nodes), 1.28µs from start of poll, 204.064057ms from cluster start. Timeout at 2025-01-25 14:29:56.908948208 +0000 UTC m=+106.763273190, 105.231312ms after stability
2025-01-25T14:29:56.921Z [INFO]  server-c431c6c0-2334-02c8-cb62-9b03ddf0b899: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f27a6ea9-eaf4-4af9-ed74-33118fe1544a Address:f27a6ea9-eaf4-4af9-ed74-33118fe1544a} {Suffrage:Voter ID:server-c431c6c0-2334-02c8-cb62-9b03ddf0b899 Address:c431c6c0-2334-02c8-cb62-9b03ddf0b899}]"
2025-01-25T14:29:56.921Z [INFO]  server-c431c6c0-2334-02c8-cb62-9b03ddf0b899: aborting pipeline replication: peer="{Voter server-f27a6ea9-eaf4-4af9-ed74-33118fe1544a f27a6ea9-eaf4-4af9-ed74-33118fe1544a}"
--- PASS: TestRaft_GetConfigurationNoBootstrap (0.32s)
=== RUN   TestRaft_LogStoreIsMonotonic
2025-01-25T14:29:56.923Z [INFO]  server-66710fe2-7ffd-67d0-6cdc-b6e8a57fb895: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66710fe2-7ffd-67d0-6cdc-b6e8a57fb895 Address:66710fe2-7ffd-67d0-6cdc-b6e8a57fb895}]"
2025-01-25T14:29:56.923Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:56.923Z [INFO]  server-66710fe2-7ffd-67d0-6cdc-b6e8a57fb895: entering follower state: follower="Node at 66710fe2-7ffd-67d0-6cdc-b6e8a57fb895 [Follower]" leader-address= leader-id=
2025-01-25T14:29:57.009Z [WARN]  server-66710fe2-7ffd-67d0-6cdc-b6e8a57fb895: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:57.010Z [INFO]  server-66710fe2-7ffd-67d0-6cdc-b6e8a57fb895: entering candidate state: node="Node at 66710fe2-7ffd-67d0-6cdc-b6e8a57fb895 [Candidate]" term=2
2025-01-25T14:29:57.010Z [INFO]  server-66710fe2-7ffd-67d0-6cdc-b6e8a57fb895: election won: term=2 tally=1
2025-01-25T14:29:57.010Z [INFO]  server-66710fe2-7ffd-67d0-6cdc-b6e8a57fb895: entering leader state: leader="Node at 66710fe2-7ffd-67d0-6cdc-b6e8a57fb895 [Leader]"
2025-01-25T14:29:57.115Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:57.010241103 +0000 UTC m=+106.864566085 (1 nodes), 87.020748ms from start of poll, 87.310258ms from cluster start. Timeout at 2025-01-25 14:29:57.115248743 +0000 UTC m=+106.969573725, 105.00764ms after stability
--- PASS: TestRaft_LogStoreIsMonotonic (0.19s)
=== RUN   TestRaft_CacheLogWithStoreError
2025-01-25T14:29:57.118Z [INFO]  server-dd96b427-3c80-ef12-ac75-22a3c69d7d06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd96b427-3c80-ef12-ac75-22a3c69d7d06 Address:dd96b427-3c80-ef12-ac75-22a3c69d7d06} {Suffrage:Voter ID:server-fbbfa707-236e-d103-b6ef-0e4c13712e9d Address:fbbfa707-236e-d103-b6ef-0e4c13712e9d}]"
2025-01-25T14:29:57.118Z [INFO]  server-fbbfa707-236e-d103-b6ef-0e4c13712e9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd96b427-3c80-ef12-ac75-22a3c69d7d06 Address:dd96b427-3c80-ef12-ac75-22a3c69d7d06} {Suffrage:Voter ID:server-fbbfa707-236e-d103-b6ef-0e4c13712e9d Address:fbbfa707-236e-d103-b6ef-0e4c13712e9d}]"
2025-01-25T14:29:57.118Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:57.118Z [INFO]  server-dd96b427-3c80-ef12-ac75-22a3c69d7d06: entering follower state: follower="Node at dd96b427-3c80-ef12-ac75-22a3c69d7d06 [Follower]" leader-address= leader-id=
2025-01-25T14:29:57.119Z [INFO]  server-fbbfa707-236e-d103-b6ef-0e4c13712e9d: entering follower state: follower="Node at fbbfa707-236e-d103-b6ef-0e4c13712e9d [Follower]" leader-address= leader-id=
2025-01-25T14:29:57.171Z [WARN]  server-dd96b427-3c80-ef12-ac75-22a3c69d7d06: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:57.171Z [INFO]  server-dd96b427-3c80-ef12-ac75-22a3c69d7d06: entering candidate state: node="Node at dd96b427-3c80-ef12-ac75-22a3c69d7d06 [Candidate]" term=2
2025-01-25T14:29:57.171Z [INFO]  server-dd96b427-3c80-ef12-ac75-22a3c69d7d06: election won: term=2 tally=2
2025-01-25T14:29:57.171Z [INFO]  server-dd96b427-3c80-ef12-ac75-22a3c69d7d06: entering leader state: leader="Node at dd96b427-3c80-ef12-ac75-22a3c69d7d06 [Leader]"
2025-01-25T14:29:57.171Z [INFO]  server-dd96b427-3c80-ef12-ac75-22a3c69d7d06: added peer, starting replication: peer=server-fbbfa707-236e-d103-b6ef-0e4c13712e9d
2025-01-25T14:29:57.172Z [INFO]  server-dd96b427-3c80-ef12-ac75-22a3c69d7d06: pipelining replication: peer="{Voter server-fbbfa707-236e-d103-b6ef-0e4c13712e9d fbbfa707-236e-d103-b6ef-0e4c13712e9d}"
2025-01-25T14:29:57.277Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:57.171961123 +0000 UTC m=+107.026286105 (1 nodes), 53.405438ms from start of poll, 53.925459ms from cluster start. Timeout at 2025-01-25 14:29:57.276966163 +0000 UTC m=+107.131291145, 105.00504ms after stability
2025-01-25T14:29:57.277Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:57.383Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:57.277624681 +0000 UTC m=+107.131949663 (1 nodes), 1.64µs from start of poll, 159.589017ms from cluster start. Timeout at 2025-01-25 14:29:57.38290703 +0000 UTC m=+107.237232012, 105.282349ms after stability
2025-01-25T14:29:57.384Z [ERROR] server-fbbfa707-236e-d103-b6ef-0e4c13712e9d: failed to append to logs: error="unable to store logs within log store, err: "some error""
2025-01-25T14:29:57.385Z [WARN]  server-fbbfa707-236e-d103-b6ef-0e4c13712e9d: failed to get previous log: previous-index=9 last-index=7 error="log not found"
2025-01-25T14:29:57.385Z [INFO]  server-dd96b427-3c80-ef12-ac75-22a3c69d7d06: aborting pipeline replication: peer="{Voter server-fbbfa707-236e-d103-b6ef-0e4c13712e9d fbbfa707-236e-d103-b6ef-0e4c13712e9d}"
2025-01-25T14:29:57.394Z [INFO]  server-dd96b427-3c80-ef12-ac75-22a3c69d7d06: pipelining replication: peer="{Voter server-fbbfa707-236e-d103-b6ef-0e4c13712e9d fbbfa707-236e-d103-b6ef-0e4c13712e9d}"
2025-01-25T14:29:57.395Z [INFO]  server-fbbfa707-236e-d103-b6ef-0e4c13712e9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd96b427-3c80-ef12-ac75-22a3c69d7d06 Address:dd96b427-3c80-ef12-ac75-22a3c69d7d06} {Suffrage:Voter ID:server-fbbfa707-236e-d103-b6ef-0e4c13712e9d Address:fbbfa707-236e-d103-b6ef-0e4c13712e9d}]"
2025-01-25T14:29:57.396Z [INFO]  server-dd96b427-3c80-ef12-ac75-22a3c69d7d06: aborting pipeline replication: peer="{Voter server-fbbfa707-236e-d103-b6ef-0e4c13712e9d fbbfa707-236e-d103-b6ef-0e4c13712e9d}"
--- PASS: TestRaft_CacheLogWithStoreError (0.28s)
=== RUN   TestRaft_ReloadConfig
2025-01-25T14:29:57.397Z [INFO]  server-e2808c35-767c-9e30-fb74-bf3df4ce3505: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2808c35-767c-9e30-fb74-bf3df4ce3505 Address:e2808c35-767c-9e30-fb74-bf3df4ce3505}]"
--- PASS: TestRaft_ReloadConfig (0.00s)
=== RUN   TestRaft_ReloadConfigValidates
2025-01-25T14:29:57.398Z [INFO]  server-037f4600-4e4a-125a-a7c3-a26b9121a9fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-037f4600-4e4a-125a-a7c3-a26b9121a9fb Address:037f4600-4e4a-125a-a7c3-a26b9121a9fb}]"
--- PASS: TestRaft_ReloadConfigValidates (0.00s)
=== RUN   TestRaft_InstallSnapshot_InvalidPeers
2025-01-25T14:29:57.401Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)"
--- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s)
=== RUN   TestRaft_VoteNotGranted_WhenNodeNotInCluster
2025-01-25T14:29:57.402Z [INFO]  server-db30f276-75c4-c6c3-032d-31803158d704: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db30f276-75c4-c6c3-032d-31803158d704 Address:db30f276-75c4-c6c3-032d-31803158d704} {Suffrage:Voter ID:server-4f82702b-2dac-21b8-1eac-da805577c8d4 Address:4f82702b-2dac-21b8-1eac-da805577c8d4} {Suffrage:Voter ID:server-0575ecae-99f8-0ded-5a67-999ef60160c9 Address:0575ecae-99f8-0ded-5a67-999ef60160c9}]"
2025-01-25T14:29:57.402Z [INFO]  server-4f82702b-2dac-21b8-1eac-da805577c8d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db30f276-75c4-c6c3-032d-31803158d704 Address:db30f276-75c4-c6c3-032d-31803158d704} {Suffrage:Voter ID:server-4f82702b-2dac-21b8-1eac-da805577c8d4 Address:4f82702b-2dac-21b8-1eac-da805577c8d4} {Suffrage:Voter ID:server-0575ecae-99f8-0ded-5a67-999ef60160c9 Address:0575ecae-99f8-0ded-5a67-999ef60160c9}]"
2025-01-25T14:29:57.402Z [INFO]  server-0575ecae-99f8-0ded-5a67-999ef60160c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db30f276-75c4-c6c3-032d-31803158d704 Address:db30f276-75c4-c6c3-032d-31803158d704} {Suffrage:Voter ID:server-4f82702b-2dac-21b8-1eac-da805577c8d4 Address:4f82702b-2dac-21b8-1eac-da805577c8d4} {Suffrage:Voter ID:server-0575ecae-99f8-0ded-5a67-999ef60160c9 Address:0575ecae-99f8-0ded-5a67-999ef60160c9}]"
2025-01-25T14:29:57.403Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:57.403Z [INFO]  server-4f82702b-2dac-21b8-1eac-da805577c8d4: entering follower state: follower="Node at 4f82702b-2dac-21b8-1eac-da805577c8d4 [Follower]" leader-address= leader-id=
2025-01-25T14:29:57.403Z [INFO]  server-db30f276-75c4-c6c3-032d-31803158d704: entering follower state: follower="Node at db30f276-75c4-c6c3-032d-31803158d704 [Follower]" leader-address= leader-id=
2025-01-25T14:29:57.403Z [INFO]  server-0575ecae-99f8-0ded-5a67-999ef60160c9: entering follower state: follower="Node at 0575ecae-99f8-0ded-5a67-999ef60160c9 [Follower]" leader-address= leader-id=
2025-01-25T14:29:57.469Z [WARN]  server-4f82702b-2dac-21b8-1eac-da805577c8d4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:57.469Z [INFO]  server-4f82702b-2dac-21b8-1eac-da805577c8d4: entering candidate state: node="Node at 4f82702b-2dac-21b8-1eac-da805577c8d4 [Candidate]" term=2
2025-01-25T14:29:57.470Z [INFO]  server-4f82702b-2dac-21b8-1eac-da805577c8d4: election won: term=2 tally=2
2025-01-25T14:29:57.470Z [INFO]  server-4f82702b-2dac-21b8-1eac-da805577c8d4: entering leader state: leader="Node at 4f82702b-2dac-21b8-1eac-da805577c8d4 [Leader]"
2025-01-25T14:29:57.470Z [INFO]  server-4f82702b-2dac-21b8-1eac-da805577c8d4: added peer, starting replication: peer=server-db30f276-75c4-c6c3-032d-31803158d704
2025-01-25T14:29:57.470Z [INFO]  server-4f82702b-2dac-21b8-1eac-da805577c8d4: added peer, starting replication: peer=server-0575ecae-99f8-0ded-5a67-999ef60160c9
2025-01-25T14:29:57.470Z [WARN]  server-0575ecae-99f8-0ded-5a67-999ef60160c9: rejecting vote request since our last term is greater: candidate=4f82702b-2dac-21b8-1eac-da805577c8d4 last-term=2 last-candidate-term=1
2025-01-25T14:29:57.470Z [INFO]  server-4f82702b-2dac-21b8-1eac-da805577c8d4: pipelining replication: peer="{Voter server-db30f276-75c4-c6c3-032d-31803158d704 db30f276-75c4-c6c3-032d-31803158d704}"
2025-01-25T14:29:57.470Z [INFO]  server-4f82702b-2dac-21b8-1eac-da805577c8d4: pipelining replication: peer="{Voter server-0575ecae-99f8-0ded-5a67-999ef60160c9 0575ecae-99f8-0ded-5a67-999ef60160c9}"
2025-01-25T14:29:57.575Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:57.470599216 +0000 UTC m=+107.324924198 (1 nodes), 66.803596ms from start of poll, 68.205305ms from cluster start. Timeout at 2025-01-25 14:29:57.575609336 +0000 UTC m=+107.429934398, 105.0102ms after stability
2025-01-25T14:29:57.581Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:57.687Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:57.581459787 +0000 UTC m=+107.435784729 (2 nodes), 2.56µs from start of poll, 179.065836ms from cluster start. Timeout at 2025-01-25 14:29:57.686678019 +0000 UTC m=+107.541002961, 105.218232ms after stability
2025-01-25T14:29:57.687Z [INFO]  server-4f82702b-2dac-21b8-1eac-da805577c8d4: updating configuration: command=RemoveServer server-id=server-db30f276-75c4-c6c3-032d-31803158d704 server-addr= servers="[{Suffrage:Voter ID:server-4f82702b-2dac-21b8-1eac-da805577c8d4 Address:4f82702b-2dac-21b8-1eac-da805577c8d4} {Suffrage:Voter ID:server-0575ecae-99f8-0ded-5a67-999ef60160c9 Address:0575ecae-99f8-0ded-5a67-999ef60160c9}]"
2025-01-25T14:29:57.687Z [INFO]  server-4f82702b-2dac-21b8-1eac-da805577c8d4: removed peer, stopping replication: peer=server-db30f276-75c4-c6c3-032d-31803158d704 last-index=3
2025-01-25T14:29:57.687Z [INFO]  server-4f82702b-2dac-21b8-1eac-da805577c8d4: aborting pipeline replication: peer="{Voter server-db30f276-75c4-c6c3-032d-31803158d704 db30f276-75c4-c6c3-032d-31803158d704}"
2025-01-25T14:29:57.743Z [WARN]  server-db30f276-75c4-c6c3-032d-31803158d704: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-01-25T14:29:57.807Z [ERROR] server-4f82702b-2dac-21b8-1eac-da805577c8d4: failed to heartbeat to: peer=0575ecae-99f8-0ded-5a67-999ef60160c9 backoff time=10ms error="failed to connect to peer: 0575ecae-99f8-0ded-5a67-999ef60160c9"
2025-01-25T14:29:57.810Z [ERROR] server-4f82702b-2dac-21b8-1eac-da805577c8d4: failed to pipeline appendEntries: peer="{Voter server-0575ecae-99f8-0ded-5a67-999ef60160c9 0575ecae-99f8-0ded-5a67-999ef60160c9}" error="append pipeline closed"
2025-01-25T14:29:57.811Z [INFO]  server-4f82702b-2dac-21b8-1eac-da805577c8d4: aborting pipeline replication: peer="{Voter server-0575ecae-99f8-0ded-5a67-999ef60160c9 0575ecae-99f8-0ded-5a67-999ef60160c9}"
2025-01-25T14:29:57.817Z [ERROR] server-4f82702b-2dac-21b8-1eac-da805577c8d4: failed to appendEntries to: peer="{Voter server-0575ecae-99f8-0ded-5a67-999ef60160c9 0575ecae-99f8-0ded-5a67-999ef60160c9}" error="failed to connect to peer: 0575ecae-99f8-0ded-5a67-999ef60160c9"
2025-01-25T14:29:57.827Z [ERROR] server-4f82702b-2dac-21b8-1eac-da805577c8d4: failed to heartbeat to: peer=0575ecae-99f8-0ded-5a67-999ef60160c9 backoff time=10ms error="failed to connect to peer: 0575ecae-99f8-0ded-5a67-999ef60160c9"
2025-01-25T14:29:57.839Z [ERROR] server-4f82702b-2dac-21b8-1eac-da805577c8d4: failed to appendEntries to: peer="{Voter server-0575ecae-99f8-0ded-5a67-999ef60160c9 0575ecae-99f8-0ded-5a67-999ef60160c9}" error="failed to connect to peer: 0575ecae-99f8-0ded-5a67-999ef60160c9"
2025-01-25T14:29:57.848Z [ERROR] server-4f82702b-2dac-21b8-1eac-da805577c8d4: failed to heartbeat to: peer=0575ecae-99f8-0ded-5a67-999ef60160c9 backoff time=10ms error="failed to connect to peer: 0575ecae-99f8-0ded-5a67-999ef60160c9"
2025-01-25T14:29:57.853Z [WARN]  server-4f82702b-2dac-21b8-1eac-da805577c8d4: failed to contact: server-id=server-0575ecae-99f8-0ded-5a67-999ef60160c9 time=51.099201ms
2025-01-25T14:29:57.853Z [WARN]  server-4f82702b-2dac-21b8-1eac-da805577c8d4: failed to contact quorum of nodes, stepping down
2025-01-25T14:29:57.853Z [INFO]  server-4f82702b-2dac-21b8-1eac-da805577c8d4: entering follower state: follower="Node at 4f82702b-2dac-21b8-1eac-da805577c8d4 [Follower]" leader-address= leader-id=
2025-01-25T14:29:57.855Z [ERROR] server-4f82702b-2dac-21b8-1eac-da805577c8d4: failed to appendEntries to: peer="{Voter server-0575ecae-99f8-0ded-5a67-999ef60160c9 0575ecae-99f8-0ded-5a67-999ef60160c9}" error="failed to connect to peer: 0575ecae-99f8-0ded-5a67-999ef60160c9"
2025-01-25T14:29:57.915Z [WARN]  server-0575ecae-99f8-0ded-5a67-999ef60160c9: heartbeat timeout reached, starting election: last-leader-addr=4f82702b-2dac-21b8-1eac-da805577c8d4 last-leader-id=server-4f82702b-2dac-21b8-1eac-da805577c8d4
2025-01-25T14:29:57.915Z [INFO]  server-0575ecae-99f8-0ded-5a67-999ef60160c9: entering candidate state: node="Node at 0575ecae-99f8-0ded-5a67-999ef60160c9 [Candidate]" term=3
2025-01-25T14:29:57.915Z [ERROR] server-0575ecae-99f8-0ded-5a67-999ef60160c9: failed to make requestVote RPC: target="{Voter server-4f82702b-2dac-21b8-1eac-da805577c8d4 4f82702b-2dac-21b8-1eac-da805577c8d4}" error="failed to connect to peer: 4f82702b-2dac-21b8-1eac-da805577c8d4" term=3
2025-01-25T14:29:57.917Z [WARN]  server-0575ecae-99f8-0ded-5a67-999ef60160c9: rejecting vote request since node is not in configuration: from=db30f276-75c4-c6c3-032d-31803158d704
--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.52s)
=== RUN   TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin
2025-01-25T14:29:57.920Z [INFO]  server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 Address:a66f840e-ffe7-9536-7fc6-7f7da39d6cb3} {Suffrage:Voter ID:server-05bd9c38-1487-3899-8245-14972d6639ff Address:05bd9c38-1487-3899-8245-14972d6639ff} {Suffrage:Voter ID:server-96992678-6d6d-da45-643c-8131ddef872a Address:96992678-6d6d-da45-643c-8131ddef872a}]"
2025-01-25T14:29:57.920Z [INFO]  server-05bd9c38-1487-3899-8245-14972d6639ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 Address:a66f840e-ffe7-9536-7fc6-7f7da39d6cb3} {Suffrage:Voter ID:server-05bd9c38-1487-3899-8245-14972d6639ff Address:05bd9c38-1487-3899-8245-14972d6639ff} {Suffrage:Voter ID:server-96992678-6d6d-da45-643c-8131ddef872a Address:96992678-6d6d-da45-643c-8131ddef872a}]"
2025-01-25T14:29:57.920Z [INFO]  server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3: entering follower state: follower="Node at a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 [Follower]" leader-address= leader-id=
2025-01-25T14:29:57.920Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 Address:a66f840e-ffe7-9536-7fc6-7f7da39d6cb3} {Suffrage:Voter ID:server-05bd9c38-1487-3899-8245-14972d6639ff Address:05bd9c38-1487-3899-8245-14972d6639ff} {Suffrage:Voter ID:server-96992678-6d6d-da45-643c-8131ddef872a Address:96992678-6d6d-da45-643c-8131ddef872a}]"
2025-01-25T14:29:57.920Z [INFO]  server-05bd9c38-1487-3899-8245-14972d6639ff: entering follower state: follower="Node at 05bd9c38-1487-3899-8245-14972d6639ff [Follower]" leader-address= leader-id=
2025-01-25T14:29:57.920Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:57.920Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: entering follower state: follower="Node at 96992678-6d6d-da45-643c-8131ddef872a [Follower]" leader-address= leader-id=
2025-01-25T14:29:57.979Z [WARN]  server-96992678-6d6d-da45-643c-8131ddef872a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:57.979Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: entering candidate state: node="Node at 96992678-6d6d-da45-643c-8131ddef872a [Candidate]" term=2
2025-01-25T14:29:57.979Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: election won: term=2 tally=2
2025-01-25T14:29:57.979Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: entering leader state: leader="Node at 96992678-6d6d-da45-643c-8131ddef872a [Leader]"
2025-01-25T14:29:57.979Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: added peer, starting replication: peer=server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3
2025-01-25T14:29:57.979Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: added peer, starting replication: peer=server-05bd9c38-1487-3899-8245-14972d6639ff
2025-01-25T14:29:57.979Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: pipelining replication: peer="{Voter server-05bd9c38-1487-3899-8245-14972d6639ff 05bd9c38-1487-3899-8245-14972d6639ff}"
2025-01-25T14:29:57.980Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: pipelining replication: peer="{Voter server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}"
2025-01-25T14:29:58.086Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:57.979859049 +0000 UTC m=+107.834183991 (1 nodes), 58.935159ms from start of poll, 59.720331ms from cluster start. Timeout at 2025-01-25 14:29:58.084867249 +0000 UTC m=+107.939192231, 105.00824ms after stability
2025-01-25T14:29:58.091Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:29:58.214Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:29:58.091901377 +0000 UTC m=+107.946226359 (2 nodes), 2.48µs from start of poll, 171.762699ms from cluster start. Timeout at 2025-01-25 14:29:58.197279764 +0000 UTC m=+108.051604746, 105.378387ms after stability
2025-01-25T14:29:58.220Z [ERROR] server-96992678-6d6d-da45-643c-8131ddef872a: failed to pipeline appendEntries: peer="{Voter server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}" error="append pipeline closed"
2025-01-25T14:29:58.220Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: aborting pipeline replication: peer="{Voter server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}"
2025-01-25T14:29:58.222Z [ERROR] server-96992678-6d6d-da45-643c-8131ddef872a: failed to heartbeat to: peer=a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 backoff time=10ms error="failed to connect to peer: a66f840e-ffe7-9536-7fc6-7f7da39d6cb3"
2025-01-25T14:29:58.227Z [ERROR] server-96992678-6d6d-da45-643c-8131ddef872a: failed to appendEntries to: peer="{Voter server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}" error="failed to connect to peer: a66f840e-ffe7-9536-7fc6-7f7da39d6cb3"
2025-01-25T14:29:58.240Z [ERROR] server-96992678-6d6d-da45-643c-8131ddef872a: failed to heartbeat to: peer=a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 backoff time=10ms error="failed to connect to peer: a66f840e-ffe7-9536-7fc6-7f7da39d6cb3"
2025-01-25T14:29:58.244Z [ERROR] server-96992678-6d6d-da45-643c-8131ddef872a: failed to appendEntries to: peer="{Voter server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}" error="failed to connect to peer: a66f840e-ffe7-9536-7fc6-7f7da39d6cb3"
2025-01-25T14:29:58.259Z [ERROR] server-96992678-6d6d-da45-643c-8131ddef872a: failed to heartbeat to: peer=a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 backoff time=10ms error="failed to connect to peer: a66f840e-ffe7-9536-7fc6-7f7da39d6cb3"
2025-01-25T14:29:58.262Z [ERROR] server-96992678-6d6d-da45-643c-8131ddef872a: failed to appendEntries to: peer="{Voter server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}" error="failed to connect to peer: a66f840e-ffe7-9536-7fc6-7f7da39d6cb3"
2025-01-25T14:29:58.275Z [WARN]  server-96992678-6d6d-da45-643c-8131ddef872a: failed to contact: server-id=server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 time=60.891249ms
2025-01-25T14:29:58.277Z [ERROR] server-96992678-6d6d-da45-643c-8131ddef872a: failed to heartbeat to: peer=a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 backoff time=20ms error="failed to connect to peer: a66f840e-ffe7-9536-7fc6-7f7da39d6cb3"
2025-01-25T14:29:58.292Z [ERROR] server-96992678-6d6d-da45-643c-8131ddef872a: failed to appendEntries to: peer="{Voter server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}" error="failed to connect to peer: a66f840e-ffe7-9536-7fc6-7f7da39d6cb3"
2025-01-25T14:29:58.308Z [ERROR] server-96992678-6d6d-da45-643c-8131ddef872a: failed to heartbeat to: peer=a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 backoff time=25ms error="failed to connect to peer: a66f840e-ffe7-9536-7fc6-7f7da39d6cb3"
2025-01-25T14:29:58.319Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: updating configuration: command=RemoveServer server-id=server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 server-addr= servers="[{Suffrage:Voter ID:server-05bd9c38-1487-3899-8245-14972d6639ff Address:05bd9c38-1487-3899-8245-14972d6639ff} {Suffrage:Voter ID:server-96992678-6d6d-da45-643c-8131ddef872a Address:96992678-6d6d-da45-643c-8131ddef872a}]"
2025-01-25T14:29:58.320Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: removed peer, stopping replication: peer=server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 last-index=3
2025-01-25T14:29:58.320Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: updating configuration: command=AddNonvoter server-id=server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 server-addr=a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 servers="[{Suffrage:Voter ID:server-05bd9c38-1487-3899-8245-14972d6639ff Address:05bd9c38-1487-3899-8245-14972d6639ff} {Suffrage:Voter ID:server-96992678-6d6d-da45-643c-8131ddef872a Address:96992678-6d6d-da45-643c-8131ddef872a} {Suffrage:Nonvoter ID:server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 Address:a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}]"
2025-01-25T14:29:58.320Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: added peer, starting replication: peer=server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3
2025-01-25T14:29:58.320Z [ERROR] server-96992678-6d6d-da45-643c-8131ddef872a: failed to appendEntries to: peer="{Nonvoter server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}" error="failed to connect to peer: a66f840e-ffe7-9536-7fc6-7f7da39d6cb3"
2025-01-25T14:29:58.321Z [WARN]  server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3: heartbeat timeout reached, starting election: last-leader-addr=96992678-6d6d-da45-643c-8131ddef872a last-leader-id=server-96992678-6d6d-da45-643c-8131ddef872a
2025-01-25T14:29:58.321Z [INFO]  server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3: entering candidate state: node="Node at a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 [Candidate]" term=23
2025-01-25T14:29:58.321Z [WARN]  server-96992678-6d6d-da45-643c-8131ddef872a: rejecting vote request since we have a leader: from=a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 leader=96992678-6d6d-da45-643c-8131ddef872a leader-id=server-96992678-6d6d-da45-643c-8131ddef872a
2025-01-25T14:29:58.321Z [WARN]  server-05bd9c38-1487-3899-8245-14972d6639ff: rejecting vote request since we have a leader: from=a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 leader=96992678-6d6d-da45-643c-8131ddef872a leader-id=server-96992678-6d6d-da45-643c-8131ddef872a
2025-01-25T14:29:58.340Z [ERROR] server-96992678-6d6d-da45-643c-8131ddef872a: peer has newer term, stopping replication: peer="{Nonvoter server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}"
2025-01-25T14:29:58.340Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: entering follower state: follower="Node at 96992678-6d6d-da45-643c-8131ddef872a [Follower]" leader-address= leader-id=
2025-01-25T14:29:58.340Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: aborting pipeline replication: peer="{Voter server-05bd9c38-1487-3899-8245-14972d6639ff 05bd9c38-1487-3899-8245-14972d6639ff}"
2025-01-25T14:29:58.343Z [ERROR] server-96992678-6d6d-da45-643c-8131ddef872a: peer has newer term, stopping replication: peer="{Voter server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}"
2025-01-25T14:29:58.406Z [WARN]  server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3: Election timeout reached, restarting election
2025-01-25T14:29:58.407Z [INFO]  server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3: entering candidate state: node="Node at a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 [Candidate]" term=24
2025-01-25T14:29:58.407Z [WARN]  server-96992678-6d6d-da45-643c-8131ddef872a: rejecting vote request since node is not a voter: from=a66f840e-ffe7-9536-7fc6-7f7da39d6cb3
2025-01-25T14:29:58.407Z [WARN]  server-05bd9c38-1487-3899-8245-14972d6639ff: rejecting vote request since we have a leader: from=a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 leader=96992678-6d6d-da45-643c-8131ddef872a leader-id=server-96992678-6d6d-da45-643c-8131ddef872a
2025-01-25T14:29:58.426Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:58.434Z [WARN]  server-96992678-6d6d-da45-643c-8131ddef872a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:58.434Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: entering candidate state: node="Node at 96992678-6d6d-da45-643c-8131ddef872a [Candidate]" term=25
2025-01-25T14:29:58.434Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: election won: term=25 tally=2
2025-01-25T14:29:58.434Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: entering leader state: leader="Node at 96992678-6d6d-da45-643c-8131ddef872a [Leader]"
2025-01-25T14:29:58.434Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: added peer, starting replication: peer=server-05bd9c38-1487-3899-8245-14972d6639ff
2025-01-25T14:29:58.434Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: added peer, starting replication: peer=server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3
2025-01-25T14:29:58.435Z [WARN]  server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3: failed to get previous log: previous-index=4 last-index=2 error="log not found"
2025-01-25T14:29:58.435Z [INFO]  server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3: entering follower state: follower="Node at a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 [Follower]" leader-address=96992678-6d6d-da45-643c-8131ddef872a leader-id=server-96992678-6d6d-da45-643c-8131ddef872a
2025-01-25T14:29:58.435Z [WARN]  server-96992678-6d6d-da45-643c-8131ddef872a: appendEntries rejected, sending older logs: peer="{Nonvoter server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}" next=3
2025-01-25T14:29:58.435Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: pipelining replication: peer="{Voter server-05bd9c38-1487-3899-8245-14972d6639ff 05bd9c38-1487-3899-8245-14972d6639ff}"
2025-01-25T14:29:58.436Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: pipelining replication: peer="{Nonvoter server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}"
2025-01-25T14:29:58.543Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:58.435502092 +0000 UTC m=+108.289827074 (1 nodes), 8.9168ms from start of poll, 515.363414ms from cluster start. Timeout at 2025-01-25 14:29:58.54054785 +0000 UTC m=+108.394872832, 105.045758ms after stability
2025-01-25T14:29:58.613Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: updating configuration: command=RemoveServer server-id=server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 server-addr= servers="[{Suffrage:Voter ID:server-05bd9c38-1487-3899-8245-14972d6639ff Address:05bd9c38-1487-3899-8245-14972d6639ff} {Suffrage:Voter ID:server-96992678-6d6d-da45-643c-8131ddef872a Address:96992678-6d6d-da45-643c-8131ddef872a}]"
2025-01-25T14:29:58.613Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: removed peer, stopping replication: peer=server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 last-index=106
2025-01-25T14:29:58.613Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: aborting pipeline replication: peer="{Nonvoter server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}"
2025-01-25T14:29:58.613Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: updating configuration: command=AddVoter server-id=server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 server-addr=a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 servers="[{Suffrage:Voter ID:server-05bd9c38-1487-3899-8245-14972d6639ff Address:05bd9c38-1487-3899-8245-14972d6639ff} {Suffrage:Voter ID:server-96992678-6d6d-da45-643c-8131ddef872a Address:96992678-6d6d-da45-643c-8131ddef872a} {Suffrage:Voter ID:server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 Address:a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}]"
2025-01-25T14:29:58.613Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: added peer, starting replication: peer=server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3
2025-01-25T14:29:58.614Z [WARN]  server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3: failed to get previous log: previous-index=107 last-index=106 error="log not found"
2025-01-25T14:29:58.614Z [WARN]  server-96992678-6d6d-da45-643c-8131ddef872a: appendEntries rejected, sending older logs: peer="{Voter server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}" next=107
2025-01-25T14:29:58.614Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: pipelining replication: peer="{Voter server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}"
2025-01-25T14:29:59.728Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:59.833Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:59.728071735 +0000 UTC m=+109.582396717 (1 nodes), 1.76µs from start of poll, 1.807933057s from cluster start. Timeout at 2025-01-25 14:29:59.833370444 +0000 UTC m=+109.687695426, 105.298709ms after stability
2025-01-25T14:29:59.833Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: aborting pipeline replication: peer="{Voter server-a66f840e-ffe7-9536-7fc6-7f7da39d6cb3 a66f840e-ffe7-9536-7fc6-7f7da39d6cb3}"
2025-01-25T14:29:59.833Z [INFO]  server-96992678-6d6d-da45-643c-8131ddef872a: aborting pipeline replication: peer="{Voter server-05bd9c38-1487-3899-8245-14972d6639ff 05bd9c38-1487-3899-8245-14972d6639ff}"
--- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.92s)
=== RUN   TestRaft_FollowerRemovalNoElection
2025-01-25T14:29:59.836Z [INFO]  server-2a23e391-6f61-7ee9-2d06-7913f83a7468: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a23e391-6f61-7ee9-2d06-7913f83a7468 Address:2a23e391-6f61-7ee9-2d06-7913f83a7468} {Suffrage:Voter ID:server-da86c863-436d-7519-d7f0-91d1a582634b Address:da86c863-436d-7519-d7f0-91d1a582634b} {Suffrage:Voter ID:server-9c2edd62-05a0-2906-f869-9b6a134db5cf Address:9c2edd62-05a0-2906-f869-9b6a134db5cf}]"
2025-01-25T14:29:59.836Z [INFO]  server-2a23e391-6f61-7ee9-2d06-7913f83a7468: entering follower state: follower="Node at 2a23e391-6f61-7ee9-2d06-7913f83a7468 [Follower]" leader-address= leader-id=
2025-01-25T14:29:59.836Z [INFO]  server-da86c863-436d-7519-d7f0-91d1a582634b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a23e391-6f61-7ee9-2d06-7913f83a7468 Address:2a23e391-6f61-7ee9-2d06-7913f83a7468} {Suffrage:Voter ID:server-da86c863-436d-7519-d7f0-91d1a582634b Address:da86c863-436d-7519-d7f0-91d1a582634b} {Suffrage:Voter ID:server-9c2edd62-05a0-2906-f869-9b6a134db5cf Address:9c2edd62-05a0-2906-f869-9b6a134db5cf}]"
2025-01-25T14:29:59.836Z [INFO]  server-da86c863-436d-7519-d7f0-91d1a582634b: entering follower state: follower="Node at da86c863-436d-7519-d7f0-91d1a582634b [Follower]" leader-address= leader-id=
2025-01-25T14:29:59.836Z [INFO]  server-9c2edd62-05a0-2906-f869-9b6a134db5cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a23e391-6f61-7ee9-2d06-7913f83a7468 Address:2a23e391-6f61-7ee9-2d06-7913f83a7468} {Suffrage:Voter ID:server-da86c863-436d-7519-d7f0-91d1a582634b Address:da86c863-436d-7519-d7f0-91d1a582634b} {Suffrage:Voter ID:server-9c2edd62-05a0-2906-f869-9b6a134db5cf Address:9c2edd62-05a0-2906-f869-9b6a134db5cf}]"
2025-01-25T14:29:59.837Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:29:59.837Z [INFO]  server-9c2edd62-05a0-2906-f869-9b6a134db5cf: entering follower state: follower="Node at 9c2edd62-05a0-2906-f869-9b6a134db5cf [Follower]" leader-address= leader-id=
2025-01-25T14:29:59.985Z [WARN]  server-2a23e391-6f61-7ee9-2d06-7913f83a7468: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:59.985Z [INFO]  server-2a23e391-6f61-7ee9-2d06-7913f83a7468: entering candidate state: node="Node at 2a23e391-6f61-7ee9-2d06-7913f83a7468 [Candidate]" term=2
2025-01-25T14:29:59.984Z [WARN]  server-9c2edd62-05a0-2906-f869-9b6a134db5cf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:29:59.986Z [INFO]  server-9c2edd62-05a0-2906-f869-9b6a134db5cf: entering candidate state: node="Node at 9c2edd62-05a0-2906-f869-9b6a134db5cf [Candidate]" term=2
2025-01-25T14:29:59.987Z [INFO]  server-9c2edd62-05a0-2906-f869-9b6a134db5cf: duplicate requestVote for same term: term=2
2025-01-25T14:29:59.988Z [INFO]  server-2a23e391-6f61-7ee9-2d06-7913f83a7468: election won: term=2 tally=2
2025-01-25T14:29:59.988Z [INFO]  server-2a23e391-6f61-7ee9-2d06-7913f83a7468: entering leader state: leader="Node at 2a23e391-6f61-7ee9-2d06-7913f83a7468 [Leader]"
2025-01-25T14:29:59.988Z [INFO]  server-2a23e391-6f61-7ee9-2d06-7913f83a7468: added peer, starting replication: peer=server-da86c863-436d-7519-d7f0-91d1a582634b
2025-01-25T14:29:59.988Z [INFO]  server-2a23e391-6f61-7ee9-2d06-7913f83a7468: added peer, starting replication: peer=server-9c2edd62-05a0-2906-f869-9b6a134db5cf
2025-01-25T14:29:59.988Z [INFO]  server-9c2edd62-05a0-2906-f869-9b6a134db5cf: entering follower state: follower="Node at 9c2edd62-05a0-2906-f869-9b6a134db5cf [Follower]" leader-address=2a23e391-6f61-7ee9-2d06-7913f83a7468 leader-id=server-2a23e391-6f61-7ee9-2d06-7913f83a7468
2025-01-25T14:29:59.988Z [INFO]  server-2a23e391-6f61-7ee9-2d06-7913f83a7468: pipelining replication: peer="{Voter server-9c2edd62-05a0-2906-f869-9b6a134db5cf 9c2edd62-05a0-2906-f869-9b6a134db5cf}"
2025-01-25T14:29:59.988Z [INFO]  server-da86c863-436d-7519-d7f0-91d1a582634b: duplicate requestVote for same term: term=2
2025-01-25T14:29:59.988Z [INFO]  server-2a23e391-6f61-7ee9-2d06-7913f83a7468: pipelining replication: peer="{Voter server-da86c863-436d-7519-d7f0-91d1a582634b da86c863-436d-7519-d7f0-91d1a582634b}"
2025-01-25T14:29:59.988Z [WARN]  server-2a23e391-6f61-7ee9-2d06-7913f83a7468: rejecting vote request since we have a leader: from=9c2edd62-05a0-2906-f869-9b6a134db5cf leader=2a23e391-6f61-7ee9-2d06-7913f83a7468 leader-id=server-2a23e391-6f61-7ee9-2d06-7913f83a7468
2025-01-25T14:30:00.195Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:29:59.98886903 +0000 UTC m=+109.843193972 (1 nodes), 151.812017ms from start of poll, 152.746543ms from cluster start. Timeout at 2025-01-25 14:30:00.19387291 +0000 UTC m=+110.048197892, 205.00392ms after stability
2025-01-25T14:30:00.195Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:30:00.401Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:30:00.195201004 +0000 UTC m=+110.049525946 (1 nodes), 2.52µs from start of poll, 359.078517ms from cluster start. Timeout at 2025-01-25 14:30:00.400370758 +0000 UTC m=+110.254695780, 205.169834ms after stability
2025-01-25T14:30:00.407Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:30:00.612Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:30:00.40733233 +0000 UTC m=+110.261657312 (2 nodes), 3.28µs from start of poll, 571.209883ms from cluster start. Timeout at 2025-01-25 14:30:00.61259488 +0000 UTC m=+110.466919902, 205.26259ms after stability
2025-01-25T14:30:00.612Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:30:00.818Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:30:00.612767177 +0000 UTC m=+110.467092159 (1 nodes), 1.8µs from start of poll, 776.64473ms from cluster start. Timeout at 2025-01-25 14:30:00.818001489 +0000 UTC m=+110.672326431, 205.234272ms after stability
    raft_test.go:2961: [INFO] restarting Node at da86c863-436d-7519-d7f0-91d1a582634b [Follower]
2025-01-25T14:30:00.818Z [INFO]  server-da86c863-436d-7519-d7f0-91d1a582634b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a23e391-6f61-7ee9-2d06-7913f83a7468 Address:2a23e391-6f61-7ee9-2d06-7913f83a7468} {Suffrage:Voter ID:server-da86c863-436d-7519-d7f0-91d1a582634b Address:da86c863-436d-7519-d7f0-91d1a582634b} {Suffrage:Voter ID:server-9c2edd62-05a0-2906-f869-9b6a134db5cf Address:9c2edd62-05a0-2906-f869-9b6a134db5cf}]"
2025-01-25T14:30:00.819Z [INFO]  server-da86c863-436d-7519-d7f0-91d1a582634b: entering follower state: follower="Node at da86c863-436d-7519-d7f0-91d1a582634b [Follower]" leader-address= leader-id=
2025-01-25T14:30:01.069Z [INFO]  server-2a23e391-6f61-7ee9-2d06-7913f83a7468: aborting pipeline replication: peer="{Voter server-9c2edd62-05a0-2906-f869-9b6a134db5cf 9c2edd62-05a0-2906-f869-9b6a134db5cf}"
2025-01-25T14:30:01.452Z [ERROR] server-2a23e391-6f61-7ee9-2d06-7913f83a7468: failed to pipeline appendEntries: peer="{Voter server-da86c863-436d-7519-d7f0-91d1a582634b da86c863-436d-7519-d7f0-91d1a582634b}" error="command enqueue timeout"
2025-01-25T14:30:01.453Z [INFO]  server-2a23e391-6f61-7ee9-2d06-7913f83a7468: aborting pipeline replication: peer="{Voter server-da86c863-436d-7519-d7f0-91d1a582634b da86c863-436d-7519-d7f0-91d1a582634b}"
2025-01-25T14:30:01.578Z [ERROR] server-2a23e391-6f61-7ee9-2d06-7913f83a7468: failed to heartbeat to: peer=da86c863-436d-7519-d7f0-91d1a582634b backoff time=10ms error="command timed out"
--- PASS: TestRaft_FollowerRemovalNoElection (1.75s)
=== RUN   TestRaft_VoteWithNoIDNoAddr
2025-01-25T14:30:01.581Z [INFO]  server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1 Address:4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1} {Suffrage:Voter ID:server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1 Address:1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1} {Suffrage:Voter ID:server-a142183c-0374-2555-f5ca-8cd872388a26 Address:a142183c-0374-2555-f5ca-8cd872388a26}]"
2025-01-25T14:30:01.582Z [INFO]  server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1 Address:4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1} {Suffrage:Voter ID:server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1 Address:1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1} {Suffrage:Voter ID:server-a142183c-0374-2555-f5ca-8cd872388a26 Address:a142183c-0374-2555-f5ca-8cd872388a26}]"
2025-01-25T14:30:01.582Z [INFO]  server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1: entering follower state: follower="Node at 4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1 [Follower]" leader-address= leader-id=
2025-01-25T14:30:01.582Z [INFO]  server-a142183c-0374-2555-f5ca-8cd872388a26: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1 Address:4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1} {Suffrage:Voter ID:server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1 Address:1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1} {Suffrage:Voter ID:server-a142183c-0374-2555-f5ca-8cd872388a26 Address:a142183c-0374-2555-f5ca-8cd872388a26}]"
2025-01-25T14:30:01.582Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:30:01.582Z [INFO]  server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: entering follower state: follower="Node at 1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1 [Follower]" leader-address= leader-id=
2025-01-25T14:30:01.582Z [INFO]  server-a142183c-0374-2555-f5ca-8cd872388a26: entering follower state: follower="Node at a142183c-0374-2555-f5ca-8cd872388a26 [Follower]" leader-address= leader-id=
2025-01-25T14:30:01.640Z [WARN]  server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-25T14:30:01.640Z [INFO]  server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: entering candidate state: node="Node at 1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1 [Candidate]" term=2
2025-01-25T14:30:01.640Z [INFO]  server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: election won: term=2 tally=2
2025-01-25T14:30:01.640Z [INFO]  server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: entering leader state: leader="Node at 1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1 [Leader]"
2025-01-25T14:30:01.640Z [INFO]  server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: added peer, starting replication: peer=server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1
2025-01-25T14:30:01.640Z [INFO]  server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: added peer, starting replication: peer=server-a142183c-0374-2555-f5ca-8cd872388a26
2025-01-25T14:30:01.640Z [INFO]  server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: pipelining replication: peer="{Voter server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1 4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1}"
2025-01-25T14:30:01.640Z [INFO]  server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: pipelining replication: peer="{Voter server-a142183c-0374-2555-f5ca-8cd872388a26 a142183c-0374-2555-f5ca-8cd872388a26}"
2025-01-25T14:30:01.746Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:30:01.640504151 +0000 UTC m=+111.494829093 (1 nodes), 58.147747ms from start of poll, 58.876281ms from cluster start. Timeout at 2025-01-25 14:30:01.745508631 +0000 UTC m=+111.599833613, 105.00452ms after stability
2025-01-25T14:30:01.746Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-25T14:30:01.852Z [INFO]  cluster: stable state for Leader reached at 2025-01-25 14:30:01.746799526 +0000 UTC m=+111.601124508 (1 nodes), 1.52µs from start of poll, 165.171696ms from cluster start. Timeout at 2025-01-25 14:30:01.852002318 +0000 UTC m=+111.706327300, 105.202792ms after stability
2025-01-25T14:30:01.857Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-25T14:30:01.973Z [INFO]  cluster: stable state for Follower reached at 2025-01-25 14:30:01.85756422 +0000 UTC m=+111.711889202 (2 nodes), 2.36µs from start of poll, 275.93639ms from cluster start. Timeout at 2025-01-25 14:30:01.962809572 +0000 UTC m=+111.817134514, 105.245312ms after stability
2025-01-25T14:30:01.974Z [ERROR] server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: failed to pipeline appendEntries: peer="{Voter server-a142183c-0374-2555-f5ca-8cd872388a26 a142183c-0374-2555-f5ca-8cd872388a26}" error="append pipeline closed"
2025-01-25T14:30:01.974Z [INFO]  server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: aborting pipeline replication: peer="{Voter server-a142183c-0374-2555-f5ca-8cd872388a26 a142183c-0374-2555-f5ca-8cd872388a26}"
2025-01-25T14:30:01.982Z [ERROR] server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: failed to heartbeat to: peer=4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1 backoff time=10ms error="failed to connect to peer: 4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1"
2025-01-25T14:30:01.983Z [ERROR] server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: failed to pipeline appendEntries: peer="{Voter server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1 4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1}" error="append pipeline closed"
2025-01-25T14:30:01.983Z [INFO]  server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: aborting pipeline replication: peer="{Voter server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1 4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1}"
2025-01-25T14:30:01.983Z [ERROR] server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: failed to heartbeat to: peer=a142183c-0374-2555-f5ca-8cd872388a26 backoff time=10ms error="failed to connect to peer: a142183c-0374-2555-f5ca-8cd872388a26"
2025-01-25T14:30:01.983Z [ERROR] server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: failed to appendEntries to: peer="{Voter server-a142183c-0374-2555-f5ca-8cd872388a26 a142183c-0374-2555-f5ca-8cd872388a26}" error="failed to connect to peer: a142183c-0374-2555-f5ca-8cd872388a26"
2025-01-25T14:30:01.995Z [ERROR] server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: failed to appendEntries to: peer="{Voter server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1 4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1}" error="failed to connect to peer: 4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1"
2025-01-25T14:30:02.001Z [ERROR] server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: failed to heartbeat to: peer=a142183c-0374-2555-f5ca-8cd872388a26 backoff time=10ms error="failed to connect to peer: a142183c-0374-2555-f5ca-8cd872388a26"
2025-01-25T14:30:02.004Z [ERROR] server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: failed to heartbeat to: peer=4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1 backoff time=10ms error="failed to connect to peer: 4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1"
2025-01-25T14:30:02.006Z [ERROR] server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: failed to appendEntries to: peer="{Voter server-a142183c-0374-2555-f5ca-8cd872388a26 a142183c-0374-2555-f5ca-8cd872388a26}" error="failed to connect to peer: a142183c-0374-2555-f5ca-8cd872388a26"
2025-01-25T14:30:02.015Z [ERROR] server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: failed to appendEntries to: peer="{Voter server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1 4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1}" error="failed to connect to peer: 4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1"
2025-01-25T14:30:02.021Z [ERROR] server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: failed to heartbeat to: peer=a142183c-0374-2555-f5ca-8cd872388a26 backoff time=10ms error="failed to connect to peer: a142183c-0374-2555-f5ca-8cd872388a26"
2025-01-25T14:30:02.021Z [ERROR] server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: failed to heartbeat to: peer=4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1 backoff time=10ms error="failed to connect to peer: 4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1"
2025-01-25T14:30:02.027Z [WARN]  server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: failed to contact: server-id=server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1 time=53.462316ms
2025-01-25T14:30:02.027Z [WARN]  server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: failed to contact: server-id=server-a142183c-0374-2555-f5ca-8cd872388a26 time=52.444473ms
2025-01-25T14:30:02.027Z [WARN]  server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: failed to contact quorum of nodes, stepping down
2025-01-25T14:30:02.027Z [INFO]  server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: entering follower state: follower="Node at 1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1 [Follower]" leader-address= leader-id=
2025-01-25T14:30:02.027Z [ERROR] server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: failed to appendEntries to: peer="{Voter server-a142183c-0374-2555-f5ca-8cd872388a26 a142183c-0374-2555-f5ca-8cd872388a26}" error="failed to connect to peer: a142183c-0374-2555-f5ca-8cd872388a26"
2025-01-25T14:30:02.031Z [WARN]  server-a142183c-0374-2555-f5ca-8cd872388a26: heartbeat timeout reached, starting election: last-leader-addr=1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1 last-leader-id=server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1
2025-01-25T14:30:02.031Z [INFO]  server-a142183c-0374-2555-f5ca-8cd872388a26: entering candidate state: node="Node at a142183c-0374-2555-f5ca-8cd872388a26 [Candidate]" term=3
2025-01-25T14:30:02.031Z [ERROR] server-a142183c-0374-2555-f5ca-8cd872388a26: failed to make requestVote RPC: target="{Voter server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1 1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1}" error="failed to connect to peer: 1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1" term=3
2025-01-25T14:30:02.031Z [WARN]  server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1: rejecting vote request since we have a leader: from=a142183c-0374-2555-f5ca-8cd872388a26 leader=1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1 leader-id=server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1
2025-01-25T14:30:02.032Z [ERROR] server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1: failed to appendEntries to: peer="{Voter server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1 4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1}" error="failed to connect to peer: 4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1"
2025-01-25T14:30:02.040Z [WARN]  server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1: heartbeat timeout reached, starting election: last-leader-addr=1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1 last-leader-id=server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1
2025-01-25T14:30:02.040Z [INFO]  server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1: entering candidate state: node="Node at 4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1 [Candidate]" term=3
2025-01-25T14:30:02.041Z [ERROR] server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1: failed to make requestVote RPC: target="{Voter server-1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1 1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1}" error="failed to connect to peer: 1b9e86cf-9de8-ee67-75de-9cf5baa2e8e1" term=3
2025-01-25T14:30:02.041Z [INFO]  server-a142183c-0374-2555-f5ca-8cd872388a26: duplicate requestVote for same term: term=3
2025-01-25T14:30:02.079Z [INFO]  server-4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1: entering follower state: follower="Node at 4e4b88b5-d8e4-b7a0-8df0-362ecdf2a6c1 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VoteWithNoIDNoAddr (0.50s)
=== RUN   TestRaft_runFollower_State_Transition
=== RUN   TestRaft_runFollower_State_Transition/NonVoter
2025-01-25T14:30:02.082Z [INFO]  127.0.0.1:42887 :: starting node: addr=127.0.0.1:42887
2025-01-25T14:30:02.082Z [INFO]  127.0.0.1:42887 :: initial configuration: index=0 servers=[]
2025-01-25T14:30:02.082Z [INFO]  127.0.0.1:42887 :: entering follower state: follower="Node at 127.0.0.1:42887 [Follower]" leader-address= leader-id=
2025-01-25T14:30:02.145Z [WARN]  127.0.0.1:42887 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
=== RUN   TestRaft_runFollower_State_Transition/Voter
2025-01-25T14:30:02.233Z [INFO]  127.0.0.1:41847 :: starting node: addr=127.0.0.1:41847
2025-01-25T14:30:02.233Z [INFO]  127.0.0.1:41847 :: initial configuration: index=0 servers=[]
2025-01-25T14:30:02.233Z [INFO]  127.0.0.1:41847 :: entering follower state: follower="Node at 127.0.0.1:41847 [Follower]" leader-address= leader-id=
2025-01-25T14:30:02.320Z [WARN]  127.0.0.1:41847 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
=== RUN   TestRaft_runFollower_State_Transition/Not_in_Config
2025-01-25T14:30:02.384Z [INFO]  127.0.0.1:44491 :: starting node: addr=127.0.0.1:44491
2025-01-25T14:30:02.384Z [INFO]  127.0.0.1:44491 :: initial configuration: index=0 servers=[]
2025-01-25T14:30:02.384Z [INFO]  127.0.0.1:44491 :: entering follower state: follower="Node at 127.0.0.1:44491 [Follower]" leader-address= leader-id=
2025-01-25T14:30:02.442Z [WARN]  127.0.0.1:44491 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
--- PASS: TestRaft_runFollower_State_Transition (0.45s)
    --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s)
=== RUN   TestRaft_runFollower_ReloadTimeoutConfigs
2025-01-25T14:30:02.536Z [INFO]  127.0.0.1:39553 :: starting node: addr=127.0.0.1:39553
2025-01-25T14:30:02.536Z [INFO]  127.0.0.1:39553 :: initial configuration: index=0 servers=[]
2025-01-25T14:30:02.536Z [INFO]  127.0.0.1:39553 :: entering follower state: follower="Node at 127.0.0.1:39553 [Follower]" leader-address= leader-id=
2025-01-25T14:30:02.536Z [WARN]  127.0.0.1:39553 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
--- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.16s)
=== RUN   TestSaturationMetric
=== RUN   TestSaturationMetric/without_smoothing
--- PASS: TestSaturationMetric (0.00s)
    --- PASS: TestSaturationMetric/without_smoothing (0.00s)
=== RUN   TestSaturationMetric_IncorrectUsage
=== RUN   TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_first
--- PASS: TestSaturationMetric_IncorrectUsage (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s)
=== RUN   TestTCPTransport_BadAddr
--- PASS: TestTCPTransport_BadAddr (0.00s)
=== RUN   TestTCPTransport_EmptyAddr
--- PASS: TestTCPTransport_EmptyAddr (0.00s)
=== RUN   TestTCPTransport_WithAdvertise
--- PASS: TestTCPTransport_WithAdvertise (0.00s)
=== RUN   TestTransport_StartStop
--- PASS: TestTransport_StartStop (0.00s)
=== RUN   TestTransport_AppendEntries
--- PASS: TestTransport_AppendEntries (0.00s)
=== RUN   TestTransport_AppendEntriesPipeline
--- PASS: TestTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestTransport_RequestVote
--- PASS: TestTransport_RequestVote (0.00s)
=== RUN   TestTransport_InstallSnapshot
--- PASS: TestTransport_InstallSnapshot (0.00s)
=== RUN   TestTransport_EncodeDecode
--- PASS: TestTransport_EncodeDecode (0.00s)
=== RUN   TestRandomTimeout
--- PASS: TestRandomTimeout (0.00s)
=== RUN   TestNewSeed
--- PASS: TestNewSeed (0.00s)
=== RUN   TestRandomTimeout_NoTime
--- PASS: TestRandomTimeout_NoTime (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
=== RUN   TestMax
--- PASS: TestMax (0.00s)
=== RUN   TestGenerateUUID
--- PASS: TestGenerateUUID (0.01s)
=== RUN   TestBackoff
--- PASS: TestBackoff (0.00s)
=== RUN   TestOverrideNotifyBool
--- PASS: TestOverrideNotifyBool (0.00s)
PASS
ok  	github.com/hashicorp/raft	112.578s
make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
   create-stamp debian/debhelper-build-stamp
   dh_testroot -O--buildsystem=golang
   dh_prep -O--buildsystem=golang
   dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang
   dh_installdocs -O--buildsystem=golang
   dh_installchangelogs -O--buildsystem=golang
   dh_installsystemduser -O--buildsystem=golang
   dh_perl -O--buildsystem=golang
   dh_link -O--buildsystem=golang
   dh_strip_nondeterminism -O--buildsystem=golang
   dh_compress -O--buildsystem=golang
   dh_fixperms -O--buildsystem=golang
   dh_missing -O--buildsystem=golang
   dh_installdeb -O--buildsystem=golang
   dh_golang -O--buildsystem=golang
   dh_gencontrol -O--buildsystem=golang
   dh_md5sums -O--buildsystem=golang
   dh_builddeb -O--buildsystem=golang
dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'.
 dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_armhf.buildinfo
 dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_armhf.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-buildpackage: info: binary-only upload (no source included)
dpkg-genchanges: info: including full source code in upload
I: copying local configuration
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /srv/workspace/pbuilder/22464 and its subdirectories
I: Current time: Sat Jan 25 02:30:16 -12 2025
I: pbuilder-time-stamp: 1737815416
Sat Jan 25 14:30:24 UTC 2025  I: 1st build successful. Starting 2nd build on remote node ff64a-armhf-rb.debian.net.
Sat Jan 25 14:30:24 UTC 2025  I: Preparing to do remote build '2' on ff64a-armhf-rb.debian.net.
Sat Jan 25 14:41:30 UTC 2025  I: Deleting $TMPDIR on ff64a-armhf-rb.debian.net.
Sat Jan 25 14:41:33 UTC 2025  I: golang-github-hashicorp-raft_1.5.0-1_armhf.changes:
Format: 1.8
Date: Thu, 27 Jul 2023 15:11:32 +0000
Source: golang-github-hashicorp-raft
Binary: golang-github-hashicorp-raft-dev
Architecture: all
Version: 1.5.0-1
Distribution: unstable
Urgency: medium
Maintainer: Debian Go Packaging Team <team+pkg-go@tracker.debian.org>
Changed-By: Daniel Swarbrick <dswarbrick@debian.org>
Description:
 golang-github-hashicorp-raft-dev - implementation of the Raft consensus protocol
Changes:
 golang-github-hashicorp-raft (1.5.0-1) unstable; urgency=medium
 .
   * Team upload
   * New upstream release
   * Add flaky TestRaft_HasExistingState to skipped tests
Checksums-Sha1:
 661dc27cce54b2fa917be64df5d901ec5e69e04c 106468 golang-github-hashicorp-raft-dev_1.5.0-1_all.deb
 fbc6eac0fe3288fe7d696a2af1d0585f57805968 8323 golang-github-hashicorp-raft_1.5.0-1_armhf.buildinfo
Checksums-Sha256:
 a59f670467267462b33ea85847a52a174cc5c54327d3d38cf4352b0bddb2887e 106468 golang-github-hashicorp-raft-dev_1.5.0-1_all.deb
 19b456282f4589da5e22c1f880adcf1389abf4cfdd8e47331a2d17ede8333b32 8323 golang-github-hashicorp-raft_1.5.0-1_armhf.buildinfo
Files:
 6162adc0e7336a9201ced64e81c3ff6e 106468 golang optional golang-github-hashicorp-raft-dev_1.5.0-1_all.deb
 31ac82a63781d88782289142bbc43a0c 8323 golang optional golang-github-hashicorp-raft_1.5.0-1_armhf.buildinfo
Sat Jan 25 14:41:34 UTC 2025  I: diffoscope 285 will be used to compare the two builds:
Running as unit: rb-diffoscope-armhf_14-9529.service
# Profiling output for: /usr/bin/diffoscope --timeout 7200 --html /srv/reproducible-results/rbuild-debian/r-b-build.qVeudEaw/golang-github-hashicorp-raft_1.5.0-1.diffoscope.html --text /srv/reproducible-results/rbuild-debian/r-b-build.qVeudEaw/golang-github-hashicorp-raft_1.5.0-1.diffoscope.txt --json /srv/reproducible-results/rbuild-debian/r-b-build.qVeudEaw/golang-github-hashicorp-raft_1.5.0-1.diffoscope.json --profile=- /srv/reproducible-results/rbuild-debian/r-b-build.qVeudEaw/b1/golang-github-hashicorp-raft_1.5.0-1_armhf.changes /srv/reproducible-results/rbuild-debian/r-b-build.qVeudEaw/b2/golang-github-hashicorp-raft_1.5.0-1_armhf.changes

## command (total time: 0.000s)
       0.000s      1 call     cmp (internal)

## has_same_content_as (total time: 0.000s)
       0.000s      1 call     abc.DotChangesFile

## main (total time: 0.363s)
       0.363s      2 calls    outputs
       0.000s      1 call     cleanup

## recognizes (total time: 0.016s)
       0.016s     12 calls    diffoscope.comparators.binary.FilesystemFile

## specialize (total time: 0.000s)
       0.000s      1 call     specialize
Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 752ms
CPU time consumed: 754ms
Sat Jan 25 14:41:35 UTC 2025  I: diffoscope 285 found no differences in the changes files, and a .buildinfo file also exists.
Sat Jan 25 14:41:35 UTC 2025  I: golang-github-hashicorp-raft from trixie built successfully and reproducibly on armhf.
Sat Jan 25 14:41:37 UTC 2025  I: Submitting .buildinfo files to external archives:
Sat Jan 25 14:41:37 UTC 2025  I: Submitting 12K	b1/golang-github-hashicorp-raft_1.5.0-1_armhf.buildinfo.asc
Sat Jan 25 14:41:38 UTC 2025  I: Submitting 12K	b2/golang-github-hashicorp-raft_1.5.0-1_armhf.buildinfo.asc
Sat Jan 25 14:41:40 UTC 2025  I: Done submitting .buildinfo files to http://buildinfo.debian.net/api/submit.
Sat Jan 25 14:41:40 UTC 2025  I: Done submitting .buildinfo files.
Sat Jan 25 14:41:40 UTC 2025  I: Removing signed golang-github-hashicorp-raft_1.5.0-1_armhf.buildinfo.asc files:
removed './b1/golang-github-hashicorp-raft_1.5.0-1_armhf.buildinfo.asc'
removed './b2/golang-github-hashicorp-raft_1.5.0-1_armhf.buildinfo.asc'