Sun Feb 23 18:39:54 UTC 2025  I: starting to build golang-github-hashicorp-raft/trixie/i386 on jenkins on '2025-02-23 18:39'
Sun Feb 23 18:39:54 UTC 2025  I: The jenkins build log is/was available at https://jenkins.debian.net/userContent/reproducible/debian/build_service/i386_7/58677/console.log
Sun Feb 23 18:39:54 UTC 2025  I: Downloading source for trixie/golang-github-hashicorp-raft=1.5.0-1
--2025-02-23 18:39:55--  http://deb.debian.org/debian/pool/main/g/golang-github-hashicorp-raft/golang-github-hashicorp-raft_1.5.0-1.dsc
Connecting to 46.16.76.132:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2480 (2.4K) [text/prs.lines.tag]
Saving to: ‘golang-github-hashicorp-raft_1.5.0-1.dsc’

     0K ..                                                    100%  376M=0s

2025-02-23 18:39:55 (376 MB/s) - ‘golang-github-hashicorp-raft_1.5.0-1.dsc’ saved [2480/2480]

Sun Feb 23 18:39:55 UTC 2025  I: golang-github-hashicorp-raft_1.5.0-1.dsc
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 3.0 (quilt)
Source: golang-github-hashicorp-raft
Binary: golang-github-hashicorp-raft-dev
Architecture: all
Version: 1.5.0-1
Maintainer: Debian Go Packaging Team <team+pkg-go@tracker.debian.org>
Uploaders: Tim Potter <tpot@hpe.com>, Dmitry Smirnov <onlyjob@debian.org>,
Homepage: https://github.com/hashicorp/raft
Standards-Version: 4.6.2
Vcs-Browser: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft
Vcs-Git: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git
Testsuite: autopkgtest-pkg-go
Build-Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
Package-List:
 golang-github-hashicorp-raft-dev deb golang optional arch=all
Checksums-Sha1:
 cea1c93d584c6f8dfbf53aef3438ab18ec7e7222 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 015fac87182a135b38ce31d1eb99f68ca251ae23 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Checksums-Sha256:
 f4dc9aabc7784a1b9b5da36b85a3797ac37911f2a6dc54a6fd6cb60a6fd643e9 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 0d95ee03abc0b447cfb92e400d914ecee5113fd181a4ea1d51fa5b6c37288658 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Files:
 7ebc703c8f7e2d7fc02568216f2e3b0a 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 1f0ee65ab76bca0e05a75b8f9ad32d8d 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Go-Import-Path: github.com/hashicorp/raft

-----BEGIN PGP SIGNATURE-----

iQJKBAEBCgA0FiEEMD9oek78sa58GjWjtwAXP7uAWikFAmTCiiIWHGRzd2FyYnJp
Y2tAZGViaWFuLm9yZwAKCRC3ABc/u4BaKYS6D/wN81mTx6pt43be/9H7Ovy6z+5p
nv0+kgrOLtH5y4KUjNLZJuPljAp+uoELmotxp1wqkoLSYs1AwCZ11X4NVs8YoLXj
sLd+Ub+QcWb0Gejqjw7Nk9VZqqXjwfFJW54n8NC2Zy8D6OelCl47sDPmiu03v3uz
oRu4dp2DW/uKkC0vb4RczT81GBmJ19Tu2ZdDvtk7kwnvF48cqDhXMhUTQAEjoyVU
Q8kfY13blza+uXZk6xhWyYsc+wpTh6RHZIWsbpgejKmEYnZCoCdaR0UqOoau33Bs
T1OXVNM0P6vrmeXyoiEy66Q2XyHeei+HYRUgED16Ujg0qxPxSEVE5timSwcXHa/u
xb5ZcKQwqEoJISjWTFDG8gww8MBPOKNGW5BKRdQSdtDhNEsnmn58jl1lsRs5L6Xg
+DrufjU6KKqqYyOBS+H+s68KXleg2t/qP+K8GZjJl/eX3+Tw5xOdt3ZWDMWxB2qF
gS+ASmxYV0Uu/pd0MLWXjAFcllnl93MzPVkF14A8eYrghujLjN3+oUOD0RQzxpAb
LHc5b0iPnN4/YQtJX3fZi2y6c62BCZ5Slb5sB0bsgNPbk1T3qQrtMPWJXUfr32ys
VOow6eNA+VDxNxHZ3aSUiVC0lcUm6dJDYqcfk/PgDdBR/E7iigkTi3O/2rgW0Y7Z
14FQMVvbK31veGXJjA==
=okTI
-----END PGP SIGNATURE-----
Sun Feb 23 18:39:55 UTC 2025  I: Checking whether the package is not for us
Sun Feb 23 18:39:55 UTC 2025  I: Starting 1st build on remote node ionos12-i386.debian.net.
Sun Feb 23 18:39:55 UTC 2025  I: Preparing to do remote build '1' on ionos12-i386.debian.net.
Sun Feb 23 18:43:36 UTC 2025  I: Deleting $TMPDIR on ionos12-i386.debian.net.
I: pbuilder: network access will be disabled during build
I: Current time: Sun Feb 23 06:39:56 -12 2025
I: pbuilder-time-stamp: 1740335996
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/trixie-reproducible-base.tgz]
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: policy-rc.d already exists
I: using eatmydata during job
I: Copying source file
I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc]
I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz]
I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz]
I: Extracting source
dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: unsupported subcommand
dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying 0001-Skip-unreliable-tests.patch
I: Not using root during the build.
I: Installing the build-deps
I: user script /srv/workspace/pbuilder/30230/tmp/hooks/D02_print_environment starting
I: set
  BUILDDIR='/build/reproducible-path'
  BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other'
  BUILDUSERNAME='pbuilder1'
  BUILD_ARCH='i386'
  DEBIAN_FRONTEND='noninteractive'
  DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=11 '
  DISTRIBUTION='trixie'
  HOME='/root'
  HOST_ARCH='i386'
  IFS=' 	
  '
  INVOCATION_ID='87202869cecb4aa78a79764c664c9fa3'
  LANG='C'
  LANGUAGE='en_US:en'
  LC_ALL='C'
  LD_LIBRARY_PATH='/usr/lib/libeatmydata'
  LD_PRELOAD='libeatmydata.so'
  MAIL='/var/mail/root'
  OPTIND='1'
  PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games'
  PBCURRENTCOMMANDLINEOPERATION='build'
  PBUILDER_OPERATION='build'
  PBUILDER_PKGDATADIR='/usr/share/pbuilder'
  PBUILDER_PKGLIBDIR='/usr/lib/pbuilder'
  PBUILDER_SYSCONFDIR='/etc'
  PPID='30230'
  PS1='# '
  PS2='> '
  PS4='+ '
  PWD='/'
  SHELL='/bin/bash'
  SHLVL='2'
  SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.9TGEveym/pbuilderrc_YIPC --distribution trixie --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/trixie-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.9TGEveym/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc'
  SUDO_GID='112'
  SUDO_UID='107'
  SUDO_USER='jenkins'
  TERM='unknown'
  TZ='/usr/share/zoneinfo/Etc/GMT+12'
  USER='root'
  _='/usr/bin/systemd-run'
  http_proxy='http://46.16.76.132:3128'
I: uname -a
  Linux ionos12-i386 6.1.0-31-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.128-1 (2025-02-07) x86_64 GNU/Linux
I: ls -l /bin
  lrwxrwxrwx 1 root root 7 Nov 22 14:40 /bin -> usr/bin
I: user script /srv/workspace/pbuilder/30230/tmp/hooks/D02_print_environment finished
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: i386
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 19791 files and directories currently installed.)
Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ...
Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however:
  Package debhelper-compat is not installed.
 pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however:
  Package dh-sequence-golang is not installed.
 pbuilder-satisfydepends-dummy depends on golang-any; however:
  Package golang-any is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however:
  Package golang-github-armon-go-metrics-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however:
  Package golang-github-hashicorp-go-hclog-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however:
  Package golang-github-hashicorp-go-msgpack-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however:
  Package golang-github-stretchr-testify-dev is not installed.

Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
Building tag database...
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
The following NEW packages will be installed:
  autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.24-go{a} golang-1.24-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-prometheus-sigv4-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libunistring5{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} 
The following packages are RECOMMENDED but will NOT be installed:
  curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 
0 packages upgraded, 107 newly installed, 0 to remove and 0 not upgraded.
Need to get 129 MB of archives. After unpacking 699 MB will be used.
Writing extended state information...
Get: 1 http://deb.debian.org/debian trixie/main i386 sensible-utils all 0.0.24 [24.8 kB]
Get: 2 http://deb.debian.org/debian trixie/main i386 libmagic-mgc i386 1:5.45-3+b1 [314 kB]
Get: 3 http://deb.debian.org/debian trixie/main i386 libmagic1t64 i386 1:5.45-3+b1 [115 kB]
Get: 4 http://deb.debian.org/debian trixie/main i386 file i386 1:5.45-3+b1 [43.2 kB]
Get: 5 http://deb.debian.org/debian trixie/main i386 gettext-base i386 0.23.1-1 [245 kB]
Get: 6 http://deb.debian.org/debian trixie/main i386 libuchardet0 i386 0.0.8-1+b2 [69.2 kB]
Get: 7 http://deb.debian.org/debian trixie/main i386 groff-base i386 1.23.0-7 [1199 kB]
Get: 8 http://deb.debian.org/debian trixie/main i386 bsdextrautils i386 2.40.4-3 [96.2 kB]
Get: 9 http://deb.debian.org/debian trixie/main i386 libpipeline1 i386 1.5.8-1 [41.2 kB]
Get: 10 http://deb.debian.org/debian trixie/main i386 man-db i386 2.13.0-1 [1428 kB]
Get: 11 http://deb.debian.org/debian trixie/main i386 m4 i386 1.4.19-5 [301 kB]
Get: 12 http://deb.debian.org/debian trixie/main i386 autoconf all 2.72-3 [493 kB]
Get: 13 http://deb.debian.org/debian trixie/main i386 autotools-dev all 20220109.1 [51.6 kB]
Get: 14 http://deb.debian.org/debian trixie/main i386 automake all 1:1.17-3 [862 kB]
Get: 15 http://deb.debian.org/debian trixie/main i386 autopoint all 0.23.1-1 [770 kB]
Get: 16 http://deb.debian.org/debian trixie/main i386 libdebhelper-perl all 13.24.1 [90.9 kB]
Get: 17 http://deb.debian.org/debian trixie/main i386 libtool all 2.5.4-3 [539 kB]
Get: 18 http://deb.debian.org/debian trixie/main i386 dh-autoreconf all 20 [17.1 kB]
Get: 19 http://deb.debian.org/debian trixie/main i386 libarchive-zip-perl all 1.68-1 [104 kB]
Get: 20 http://deb.debian.org/debian trixie/main i386 libfile-stripnondeterminism-perl all 1.14.1-2 [19.7 kB]
Get: 21 http://deb.debian.org/debian trixie/main i386 dh-strip-nondeterminism all 1.14.1-2 [8620 B]
Get: 22 http://deb.debian.org/debian trixie/main i386 libelf1t64 i386 0.192-4 [195 kB]
Get: 23 http://deb.debian.org/debian trixie/main i386 dwz i386 0.15-1+b1 [116 kB]
Get: 24 http://deb.debian.org/debian trixie/main i386 libunistring5 i386 1.3-1 [458 kB]
Get: 25 http://deb.debian.org/debian trixie/main i386 libicu72 i386 72.1-6 [9582 kB]
Get: 26 http://deb.debian.org/debian trixie/main i386 libxml2 i386 2.12.7+dfsg+really2.9.14-0.2+b1 [734 kB]
Get: 27 http://deb.debian.org/debian trixie/main i386 gettext i386 0.23.1-1 [1714 kB]
Get: 28 http://deb.debian.org/debian trixie/main i386 intltool-debian all 0.35.0+20060710.6 [22.9 kB]
Get: 29 http://deb.debian.org/debian trixie/main i386 po-debconf all 1.0.21+nmu1 [248 kB]
Get: 30 http://deb.debian.org/debian trixie/main i386 debhelper all 13.24.1 [920 kB]
Get: 31 http://deb.debian.org/debian trixie/main i386 dh-golang all 1.63 [24.4 kB]
Get: 32 http://deb.debian.org/debian trixie/main i386 golang-1.24-src all 1.24.0-1 [20.3 MB]
Get: 33 http://deb.debian.org/debian trixie/main i386 golang-1.24-go i386 1.24.0-1 [26.8 MB]
Get: 34 http://deb.debian.org/debian trixie/main i386 golang-src all 2:1.24~2 [5136 B]
Get: 35 http://deb.debian.org/debian trixie/main i386 golang-go i386 2:1.24~2 [44.3 kB]
Get: 36 http://deb.debian.org/debian trixie/main i386 golang-any i386 2:1.24~2 [5216 B]
Get: 37 http://deb.debian.org/debian trixie/main i386 golang-github-alecthomas-units-dev all 0.0~git20240626.61d1e34-1 [9032 B]
Get: 38 http://deb.debian.org/debian trixie/main i386 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB]
Get: 39 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB]
Get: 40 http://deb.debian.org/debian trixie/main i386 golang-golang-x-sys-dev all 0.22.0-1 [431 kB]
Get: 41 http://deb.debian.org/debian trixie/main i386 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B]
Get: 42 http://deb.debian.org/debian trixie/main i386 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB]
Get: 43 http://deb.debian.org/debian trixie/main i386 golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB]
Get: 44 http://deb.debian.org/debian trixie/main i386 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB]
Get: 45 http://deb.debian.org/debian trixie/main i386 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB]
Get: 46 http://deb.debian.org/debian trixie/main i386 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB]
Get: 47 http://deb.debian.org/debian trixie/main i386 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB]
Get: 48 http://deb.debian.org/debian trixie/main i386 golang-github-stretchr-testify-dev all 1.9.0-1 [74.7 kB]
Get: 49 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB]
Get: 50 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB]
Get: 51 http://deb.debian.org/debian trixie/main i386 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB]
Get: 52 http://deb.debian.org/debian trixie/main i386 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B]
Get: 53 http://deb.debian.org/debian trixie/main i386 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB]
Get: 54 http://deb.debian.org/debian trixie/main i386 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB]
Get: 55 http://deb.debian.org/debian trixie/main i386 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB]
Get: 56 http://deb.debian.org/debian trixie/main i386 golang-google-protobuf-dev all 1.33.0-1 [625 kB]
Get: 57 http://deb.debian.org/debian trixie/main i386 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB]
Get: 58 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B]
Get: 59 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB]
Get: 60 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB]
Get: 61 http://deb.debian.org/debian trixie/main i386 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB]
Get: 62 http://deb.debian.org/debian trixie/main i386 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB]
Get: 63 http://deb.debian.org/debian trixie/main i386 golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB]
Get: 64 http://deb.debian.org/debian trixie/main i386 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B]
Get: 65 http://deb.debian.org/debian trixie/main i386 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB]
Get: 66 http://deb.debian.org/debian trixie/main i386 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB]
Get: 67 http://deb.debian.org/debian trixie/main i386 golang-github-klauspost-compress-dev all 1.17.11+ds1-1 [33.3 MB]
Get: 68 http://deb.debian.org/debian trixie/main i386 golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB]
Get: 69 http://deb.debian.org/debian trixie/main i386 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB]
Get: 70 http://deb.debian.org/debian trixie/main i386 golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB]
Get: 71 http://deb.debian.org/debian trixie/main i386 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB]
Get: 72 http://deb.debian.org/debian trixie/main i386 golang-golang-x-term-dev all 0.22.0-1 [14.8 kB]
Get: 73 http://deb.debian.org/debian trixie/main i386 golang-golang-x-text-dev all 0.22.0-1 [3995 kB]
Get: 74 http://deb.debian.org/debian trixie/main i386 golang-golang-x-crypto-dev all 1:0.25.0-1 [1682 kB]
Get: 75 http://deb.debian.org/debian trixie/main i386 golang-golang-x-net-dev all 1:0.27.0-1 [898 kB]
Get: 76 http://deb.debian.org/debian trixie/main i386 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB]
Get: 77 http://deb.debian.org/debian trixie/main i386 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB]
Get: 78 http://deb.debian.org/debian trixie/main i386 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB]
Get: 79 http://deb.debian.org/debian trixie/main i386 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB]
Get: 80 http://deb.debian.org/debian trixie/main i386 golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B]
Get: 81 http://deb.debian.org/debian trixie/main i386 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B]
Get: 82 http://deb.debian.org/debian trixie/main i386 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB]
Get: 83 http://deb.debian.org/debian trixie/main i386 golang-github-prometheus-sigv4-dev all 0.1.2-2 [8484 B]
Get: 84 http://deb.debian.org/debian trixie/main i386 golang-golang-x-oauth2-dev all 0.15.0-1 [38.0 kB]
Get: 85 http://deb.debian.org/debian trixie/main i386 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB]
Get: 86 http://deb.debian.org/debian trixie/main i386 golang-github-prometheus-common-dev all 0.62.0-3 [118 kB]
Get: 87 http://deb.debian.org/debian trixie/main i386 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB]
Get: 88 http://deb.debian.org/debian trixie/main i386 golang-github-prometheus-procfs-dev all 0.15.1-1 [184 kB]
Get: 89 http://deb.debian.org/debian trixie/main i386 golang-github-prometheus-client-golang-dev all 1.21.0-1 [195 kB]
Get: 90 http://deb.debian.org/debian trixie/main i386 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB]
Get: 91 http://deb.debian.org/debian trixie/main i386 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB]
Get: 92 http://deb.debian.org/debian trixie/main i386 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B]
Get: 93 http://deb.debian.org/debian trixie/main i386 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB]
Get: 94 http://deb.debian.org/debian trixie/main i386 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B]
Get: 95 http://deb.debian.org/debian trixie/main i386 libsasl2-modules-db i386 2.1.28+dfsg1-8+b1 [20.9 kB]
Get: 96 http://deb.debian.org/debian trixie/main i386 libsasl2-2 i386 2.1.28+dfsg1-8+b1 [61.3 kB]
Get: 97 http://deb.debian.org/debian trixie/main i386 libssl-dev i386 3.4.1-1 [2837 kB]
Get: 98 http://deb.debian.org/debian trixie/main i386 libsasl2-dev i386 2.1.28+dfsg1-8+b1 [248 kB]
Get: 99 http://deb.debian.org/debian trixie/main i386 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB]
Get: 100 http://deb.debian.org/debian trixie/main i386 golang-golang-x-mod-dev all 0.19.0-1 [104 kB]
Get: 101 http://deb.debian.org/debian trixie/main i386 golang-github-yuin-goldmark-dev all 1.7.4-1 [113 kB]
Get: 102 http://deb.debian.org/debian trixie/main i386 golang-golang-x-tools-dev all 1:0.23.0+ds-1 [1593 kB]
Get: 103 http://deb.debian.org/debian trixie/main i386 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB]
Get: 104 http://deb.debian.org/debian trixie/main i386 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB]
Get: 105 http://deb.debian.org/debian trixie/main i386 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB]
Get: 106 http://deb.debian.org/debian trixie/main i386 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB]
Get: 107 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB]
Fetched 129 MB in 3s (44.8 MB/s)
Preconfiguring packages ...
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 19791 files and directories currently installed.)
Preparing to unpack .../000-sensible-utils_0.0.24_all.deb ...
Unpacking sensible-utils (0.0.24) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3+b1_i386.deb ...
Unpacking libmagic-mgc (1:5.45-3+b1) ...
Selecting previously unselected package libmagic1t64:i386.
Preparing to unpack .../002-libmagic1t64_1%3a5.45-3+b1_i386.deb ...
Unpacking libmagic1t64:i386 (1:5.45-3+b1) ...
Selecting previously unselected package file.
Preparing to unpack .../003-file_1%3a5.45-3+b1_i386.deb ...
Unpacking file (1:5.45-3+b1) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../004-gettext-base_0.23.1-1_i386.deb ...
Unpacking gettext-base (0.23.1-1) ...
Selecting previously unselected package libuchardet0:i386.
Preparing to unpack .../005-libuchardet0_0.0.8-1+b2_i386.deb ...
Unpacking libuchardet0:i386 (0.0.8-1+b2) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../006-groff-base_1.23.0-7_i386.deb ...
Unpacking groff-base (1.23.0-7) ...
Selecting previously unselected package bsdextrautils.
Preparing to unpack .../007-bsdextrautils_2.40.4-3_i386.deb ...
Unpacking bsdextrautils (2.40.4-3) ...
Selecting previously unselected package libpipeline1:i386.
Preparing to unpack .../008-libpipeline1_1.5.8-1_i386.deb ...
Unpacking libpipeline1:i386 (1.5.8-1) ...
Selecting previously unselected package man-db.
Preparing to unpack .../009-man-db_2.13.0-1_i386.deb ...
Unpacking man-db (2.13.0-1) ...
Selecting previously unselected package m4.
Preparing to unpack .../010-m4_1.4.19-5_i386.deb ...
Unpacking m4 (1.4.19-5) ...
Selecting previously unselected package autoconf.
Preparing to unpack .../011-autoconf_2.72-3_all.deb ...
Unpacking autoconf (2.72-3) ...
Selecting previously unselected package autotools-dev.
Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ...
Unpacking autotools-dev (20220109.1) ...
Selecting previously unselected package automake.
Preparing to unpack .../013-automake_1%3a1.17-3_all.deb ...
Unpacking automake (1:1.17-3) ...
Selecting previously unselected package autopoint.
Preparing to unpack .../014-autopoint_0.23.1-1_all.deb ...
Unpacking autopoint (0.23.1-1) ...
Selecting previously unselected package libdebhelper-perl.
Preparing to unpack .../015-libdebhelper-perl_13.24.1_all.deb ...
Unpacking libdebhelper-perl (13.24.1) ...
Selecting previously unselected package libtool.
Preparing to unpack .../016-libtool_2.5.4-3_all.deb ...
Unpacking libtool (2.5.4-3) ...
Selecting previously unselected package dh-autoreconf.
Preparing to unpack .../017-dh-autoreconf_20_all.deb ...
Unpacking dh-autoreconf (20) ...
Selecting previously unselected package libarchive-zip-perl.
Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ...
Unpacking libarchive-zip-perl (1.68-1) ...
Selecting previously unselected package libfile-stripnondeterminism-perl.
Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.1-2_all.deb ...
Unpacking libfile-stripnondeterminism-perl (1.14.1-2) ...
Selecting previously unselected package dh-strip-nondeterminism.
Preparing to unpack .../020-dh-strip-nondeterminism_1.14.1-2_all.deb ...
Unpacking dh-strip-nondeterminism (1.14.1-2) ...
Selecting previously unselected package libelf1t64:i386.
Preparing to unpack .../021-libelf1t64_0.192-4_i386.deb ...
Unpacking libelf1t64:i386 (0.192-4) ...
Selecting previously unselected package dwz.
Preparing to unpack .../022-dwz_0.15-1+b1_i386.deb ...
Unpacking dwz (0.15-1+b1) ...
Selecting previously unselected package libunistring5:i386.
Preparing to unpack .../023-libunistring5_1.3-1_i386.deb ...
Unpacking libunistring5:i386 (1.3-1) ...
Selecting previously unselected package libicu72:i386.
Preparing to unpack .../024-libicu72_72.1-6_i386.deb ...
Unpacking libicu72:i386 (72.1-6) ...
Selecting previously unselected package libxml2:i386.
Preparing to unpack .../025-libxml2_2.12.7+dfsg+really2.9.14-0.2+b1_i386.deb ...
Unpacking libxml2:i386 (2.12.7+dfsg+really2.9.14-0.2+b1) ...
Selecting previously unselected package gettext.
Preparing to unpack .../026-gettext_0.23.1-1_i386.deb ...
Unpacking gettext (0.23.1-1) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../027-intltool-debian_0.35.0+20060710.6_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.6) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../028-po-debconf_1.0.21+nmu1_all.deb ...
Unpacking po-debconf (1.0.21+nmu1) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../029-debhelper_13.24.1_all.deb ...
Unpacking debhelper (13.24.1) ...
Selecting previously unselected package dh-golang.
Preparing to unpack .../030-dh-golang_1.63_all.deb ...
Unpacking dh-golang (1.63) ...
Selecting previously unselected package golang-1.24-src.
Preparing to unpack .../031-golang-1.24-src_1.24.0-1_all.deb ...
Unpacking golang-1.24-src (1.24.0-1) ...
Selecting previously unselected package golang-1.24-go.
Preparing to unpack .../032-golang-1.24-go_1.24.0-1_i386.deb ...
Unpacking golang-1.24-go (1.24.0-1) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../033-golang-src_2%3a1.24~2_all.deb ...
Unpacking golang-src (2:1.24~2) ...
Selecting previously unselected package golang-go:i386.
Preparing to unpack .../034-golang-go_2%3a1.24~2_i386.deb ...
Unpacking golang-go:i386 (2:1.24~2) ...
Selecting previously unselected package golang-any:i386.
Preparing to unpack .../035-golang-any_2%3a1.24~2_i386.deb ...
Unpacking golang-any:i386 (2:1.24~2) ...
Selecting previously unselected package golang-github-alecthomas-units-dev.
Preparing to unpack .../036-golang-github-alecthomas-units-dev_0.0~git20240626.61d1e34-1_all.deb ...
Unpacking golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ...
Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev.
Preparing to unpack .../037-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ...
Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev.
Preparing to unpack .../038-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ...
Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Selecting previously unselected package golang-golang-x-sys-dev.
Preparing to unpack .../039-golang-golang-x-sys-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-sys-dev (0.22.0-1) ...
Selecting previously unselected package golang-github-mattn-go-isatty-dev.
Preparing to unpack .../040-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ...
Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Selecting previously unselected package golang-github-mattn-go-colorable-dev.
Preparing to unpack .../041-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ...
Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Selecting previously unselected package golang-github-fatih-color-dev.
Preparing to unpack .../042-golang-github-fatih-color-dev_1.17.0-1_all.deb ...
Unpacking golang-github-fatih-color-dev (1.17.0-1) ...
Selecting previously unselected package golang-github-davecgh-go-spew-dev.
Preparing to unpack .../043-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ...
Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Selecting previously unselected package golang-github-pmezard-go-difflib-dev.
Preparing to unpack .../044-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ...
Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Selecting previously unselected package golang-github-stretchr-objx-dev.
Preparing to unpack .../045-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ...
Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ...
Selecting previously unselected package golang-gopkg-yaml.v3-dev.
Preparing to unpack .../046-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ...
Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Selecting previously unselected package golang-github-stretchr-testify-dev.
Preparing to unpack .../047-golang-github-stretchr-testify-dev_1.9.0-1_all.deb ...
Unpacking golang-github-stretchr-testify-dev (1.9.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-hclog-dev.
Preparing to unpack .../048-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ...
Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev.
Preparing to unpack .../049-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ...
Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Selecting previously unselected package golang-github-pkg-errors-dev.
Preparing to unpack .../050-golang-github-pkg-errors-dev_0.9.1-3_all.deb ...
Unpacking golang-github-pkg-errors-dev (0.9.1-3) ...
Selecting previously unselected package golang-github-tv42-httpunix-dev.
Preparing to unpack .../051-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ...
Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev.
Preparing to unpack .../052-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ...
Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Selecting previously unselected package golang-github-datadog-datadog-go-dev.
Preparing to unpack .../053-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ...
Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Selecting previously unselected package golang-github-google-go-cmp-dev.
Preparing to unpack .../054-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ...
Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ...
Selecting previously unselected package golang-google-protobuf-dev.
Preparing to unpack .../055-golang-google-protobuf-dev_1.33.0-1_all.deb ...
Unpacking golang-google-protobuf-dev (1.33.0-1) ...
Selecting previously unselected package golang-github-golang-protobuf-1-5-dev.
Preparing to unpack .../056-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ...
Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Selecting previously unselected package golang-github-hashicorp-go-uuid-dev.
Preparing to unpack .../057-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ...
Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Selecting previously unselected package golang-github-hashicorp-golang-lru-dev.
Preparing to unpack .../058-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ...
Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev.
Preparing to unpack .../059-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ...
Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Selecting previously unselected package golang-github-pascaldekloe-goe-dev.
Preparing to unpack .../060-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ...
Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Selecting previously unselected package golang-github-beorn7-perks-dev.
Preparing to unpack .../061-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ...
Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ...
Selecting previously unselected package golang-github-cespare-xxhash-dev.
Preparing to unpack .../062-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ...
Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ...
Selecting previously unselected package golang-github-modern-go-concurrent-dev.
Preparing to unpack .../063-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ...
Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Selecting previously unselected package golang-github-modern-go-reflect2-dev.
Preparing to unpack .../064-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ...
Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-json-iterator-go-dev.
Preparing to unpack .../065-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ...
Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ...
Selecting previously unselected package golang-github-klauspost-compress-dev.
Preparing to unpack .../066-golang-github-klauspost-compress-dev_1.17.11+ds1-1_all.deb ...
Unpacking golang-github-klauspost-compress-dev (1.17.11+ds1-1) ...
Selecting previously unselected package golang-github-kylelemons-godebug-dev.
Preparing to unpack .../067-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ...
Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Selecting previously unselected package golang-github-prometheus-client-model-dev.
Preparing to unpack .../068-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ...
Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ...
Selecting previously unselected package golang-gopkg-yaml.v2-dev.
Preparing to unpack .../069-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ...
Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Selecting previously unselected package golang-github-jmespath-go-jmespath-dev.
Preparing to unpack .../070-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ...
Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Selecting previously unselected package golang-golang-x-term-dev.
Preparing to unpack .../071-golang-golang-x-term-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-term-dev (0.22.0-1) ...
Selecting previously unselected package golang-golang-x-text-dev.
Preparing to unpack .../072-golang-golang-x-text-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-text-dev (0.22.0-1) ...
Selecting previously unselected package golang-golang-x-crypto-dev.
Preparing to unpack .../073-golang-golang-x-crypto-dev_1%3a0.25.0-1_all.deb ...
Unpacking golang-golang-x-crypto-dev (1:0.25.0-1) ...
Selecting previously unselected package golang-golang-x-net-dev.
Preparing to unpack .../074-golang-golang-x-net-dev_1%3a0.27.0-1_all.deb ...
Unpacking golang-golang-x-net-dev (1:0.27.0-1) ...
Selecting previously unselected package golang-github-aws-aws-sdk-go-dev.
Preparing to unpack .../075-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ...
Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Selecting previously unselected package golang-github-go-logfmt-logfmt-dev.
Preparing to unpack .../076-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ...
Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Selecting previously unselected package golang-github-go-kit-log-dev.
Preparing to unpack .../077-golang-github-go-kit-log-dev_0.2.1-1_all.deb ...
Unpacking golang-github-go-kit-log-dev (0.2.1-1) ...
Selecting previously unselected package golang-github-julienschmidt-httprouter-dev.
Preparing to unpack .../078-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ...
Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Selecting previously unselected package golang-github-munnerz-goautoneg-dev.
Preparing to unpack .../079-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ...
Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Selecting previously unselected package golang-github-jpillora-backoff-dev.
Preparing to unpack .../080-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ...
Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev.
Preparing to unpack .../081-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ...
Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Selecting previously unselected package golang-github-prometheus-sigv4-dev.
Preparing to unpack .../082-golang-github-prometheus-sigv4-dev_0.1.2-2_all.deb ...
Unpacking golang-github-prometheus-sigv4-dev (0.1.2-2) ...
Selecting previously unselected package golang-golang-x-oauth2-dev.
Preparing to unpack .../083-golang-golang-x-oauth2-dev_0.15.0-1_all.deb ...
Unpacking golang-golang-x-oauth2-dev (0.15.0-1) ...
Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev.
Preparing to unpack .../084-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ...
Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Selecting previously unselected package golang-github-prometheus-common-dev.
Preparing to unpack .../085-golang-github-prometheus-common-dev_0.62.0-3_all.deb ...
Unpacking golang-github-prometheus-common-dev (0.62.0-3) ...
Selecting previously unselected package golang-golang-x-sync-dev.
Preparing to unpack .../086-golang-golang-x-sync-dev_0.7.0-1_all.deb ...
Unpacking golang-golang-x-sync-dev (0.7.0-1) ...
Selecting previously unselected package golang-github-prometheus-procfs-dev.
Preparing to unpack .../087-golang-github-prometheus-procfs-dev_0.15.1-1_all.deb ...
Unpacking golang-github-prometheus-procfs-dev (0.15.1-1) ...
Selecting previously unselected package golang-github-prometheus-client-golang-dev.
Preparing to unpack .../088-golang-github-prometheus-client-golang-dev_1.21.0-1_all.deb ...
Unpacking golang-github-prometheus-client-golang-dev (1.21.0-1) ...
Selecting previously unselected package golang-github-armon-go-metrics-dev.
Preparing to unpack .../089-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ...
Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ...
Selecting previously unselected package golang-github-creack-pty-dev.
Preparing to unpack .../090-golang-github-creack-pty-dev_1.1.21-1_all.deb ...
Unpacking golang-github-creack-pty-dev (1.1.21-1) ...
Selecting previously unselected package golang-github-vmihailenco-tagparser-dev.
Preparing to unpack .../091-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ...
Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev.
Preparing to unpack .../092-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ...
Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Selecting previously unselected package golang-gopkg-tomb.v2-dev.
Preparing to unpack .../093-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ...
Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Selecting previously unselected package libsasl2-modules-db:i386.
Preparing to unpack .../094-libsasl2-modules-db_2.1.28+dfsg1-8+b1_i386.deb ...
Unpacking libsasl2-modules-db:i386 (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package libsasl2-2:i386.
Preparing to unpack .../095-libsasl2-2_2.1.28+dfsg1-8+b1_i386.deb ...
Unpacking libsasl2-2:i386 (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package libssl-dev:i386.
Preparing to unpack .../096-libssl-dev_3.4.1-1_i386.deb ...
Unpacking libssl-dev:i386 (3.4.1-1) ...
Selecting previously unselected package libsasl2-dev.
Preparing to unpack .../097-libsasl2-dev_2.1.28+dfsg1-8+b1_i386.deb ...
Unpacking libsasl2-dev (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package golang-github-kr-text-dev.
Preparing to unpack .../098-golang-github-kr-text-dev_0.2.0-2_all.deb ...
Unpacking golang-github-kr-text-dev (0.2.0-2) ...
Selecting previously unselected package golang-golang-x-mod-dev.
Preparing to unpack .../099-golang-golang-x-mod-dev_0.19.0-1_all.deb ...
Unpacking golang-golang-x-mod-dev (0.19.0-1) ...
Selecting previously unselected package golang-github-yuin-goldmark-dev.
Preparing to unpack .../100-golang-github-yuin-goldmark-dev_1.7.4-1_all.deb ...
Unpacking golang-github-yuin-goldmark-dev (1.7.4-1) ...
Selecting previously unselected package golang-golang-x-tools-dev.
Preparing to unpack .../101-golang-golang-x-tools-dev_1%3a0.23.0+ds-1_all.deb ...
Unpacking golang-golang-x-tools-dev (1:0.23.0+ds-1) ...
Selecting previously unselected package golang-github-rogpeppe-go-internal-dev.
Preparing to unpack .../102-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ...
Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Selecting previously unselected package golang-github-kr-pretty-dev.
Preparing to unpack .../103-golang-github-kr-pretty-dev_0.3.1-1_all.deb ...
Unpacking golang-github-kr-pretty-dev (0.3.1-1) ...
Selecting previously unselected package golang-gopkg-check.v1-dev.
Preparing to unpack .../104-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ...
Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Selecting previously unselected package golang-gopkg-mgo.v2-dev.
Preparing to unpack .../105-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ...
Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev.
Preparing to unpack .../106-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ...
Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-1.24-src (1.24.0-1) ...
Setting up libpipeline1:i386 (1.5.8-1) ...
Setting up golang-github-google-go-cmp-dev (0.6.0-1) ...
Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Setting up libicu72:i386 (72.1-6) ...
Setting up bsdextrautils (2.40.4-3) ...
Setting up golang-github-creack-pty-dev (1.1.21-1) ...
Setting up golang-github-stretchr-objx-dev (0.5.2-1) ...
Setting up libmagic-mgc (1:5.45-3+b1) ...
Setting up golang-github-pkg-errors-dev (0.9.1-3) ...
Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Setting up libarchive-zip-perl (1.68-1) ...
Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Setting up libdebhelper-perl (13.24.1) ...
Setting up golang-golang-x-sys-dev (0.22.0-1) ...
Setting up libmagic1t64:i386 (1:5.45-3+b1) ...
Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Setting up gettext-base (0.23.1-1) ...
Setting up m4 (1.4.19-5) ...
Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Setting up golang-golang-x-term-dev (0.22.0-1) ...
Setting up file (1:5.45-3+b1) ...
Setting up golang-github-beorn7-perks-dev (1.0.1-1) ...
Setting up libelf1t64:i386 (0.192-4) ...
Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Setting up libsasl2-modules-db:i386 (2.1.28+dfsg1-8+b1) ...
Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ...
Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Setting up autotools-dev (20220109.1) ...
Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Setting up golang-google-protobuf-dev (1.33.0-1) ...
Setting up golang-github-kr-text-dev (0.2.0-2) ...
Setting up golang-github-stretchr-testify-dev (1.9.0-1) ...
Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Setting up golang-github-yuin-goldmark-dev (1.7.4-1) ...
Setting up libunistring5:i386 (1.3-1) ...
Setting up libssl-dev:i386 (3.4.1-1) ...
Setting up golang-golang-x-sync-dev (0.7.0-1) ...
Setting up golang-golang-x-mod-dev (0.19.0-1) ...
Setting up autopoint (0.23.1-1) ...
Setting up golang-1.24-go (1.24.0-1) ...
Setting up golang-github-klauspost-compress-dev (1.17.11+ds1-1) ...
Setting up libsasl2-2:i386 (2.1.28+dfsg1-8+b1) ...
Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Setting up autoconf (2.72-3) ...
Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Setting up dwz (0.15-1+b1) ...
Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Setting up sensible-utils (0.0.24) ...
Setting up golang-golang-x-text-dev (0.22.0-1) ...
Setting up libuchardet0:i386 (0.0.8-1+b2) ...
Setting up golang-github-prometheus-procfs-dev (0.15.1-1) ...
Setting up golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ...
Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Setting up golang-golang-x-crypto-dev (1:0.25.0-1) ...
Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ...
Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Setting up golang-golang-x-oauth2-dev (0.15.0-1) ...
Setting up golang-src (2:1.24~2) ...
Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Setting up libxml2:i386 (2.12.7+dfsg+really2.9.14-0.2+b1) ...
Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Setting up automake (1:1.17-3) ...
update-alternatives: using /usr/bin/automake-1.17 to provide /usr/bin/automake (automake) in auto mode
Setting up libfile-stripnondeterminism-perl (1.14.1-2) ...
Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Setting up golang-github-go-kit-log-dev (0.2.1-1) ...
Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Setting up gettext (0.23.1-1) ...
Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Setting up libtool (2.5.4-3) ...
Setting up golang-go:i386 (2:1.24~2) ...
Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Setting up golang-github-json-iterator-go-dev (1.1.12-2) ...
Setting up golang-any:i386 (2:1.24~2) ...
Setting up intltool-debian (0.35.0+20060710.6) ...
Setting up dh-autoreconf (20) ...
Setting up libsasl2-dev (2.1.28+dfsg1-8+b1) ...
Setting up golang-golang-x-net-dev (1:0.27.0-1) ...
Setting up dh-strip-nondeterminism (1.14.1-2) ...
Setting up groff-base (1.23.0-7) ...
Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Setting up golang-github-fatih-color-dev (1.17.0-1) ...
Setting up po-debconf (1.0.21+nmu1) ...
Setting up golang-golang-x-tools-dev (1:0.23.0+ds-1) ...
Setting up man-db (2.13.0-1) ...
Not building database; man-db/auto-update is not 'true'.
Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Setting up golang-github-kr-pretty-dev (0.3.1-1) ...
Setting up debhelper (13.24.1) ...
Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Setting up dh-golang (1.63) ...
Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Setting up golang-github-prometheus-sigv4-dev (0.1.2-2) ...
Setting up golang-github-prometheus-common-dev (0.62.0-3) ...
Setting up golang-github-prometheus-client-golang-dev (1.21.0-1) ...
Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ...
Processing triggers for libc-bin (2.40-7) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Building tag database...
 -> Finished parsing the build-deps
I: Building the package
I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S  > ../golang-github-hashicorp-raft_1.5.0-1_source.changes
dpkg-buildpackage: info: source package golang-github-hashicorp-raft
dpkg-buildpackage: info: source version 1.5.0-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Daniel Swarbrick <dswarbrick@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture i386
 debian/rules clean
dh clean --buildsystem=golang
   dh_auto_clean -O--buildsystem=golang
   dh_autoreconf_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules binary
dh binary --buildsystem=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-i686-linux-gnu && go install -trimpath -v -p 11 github.com/hashicorp/raft github.com/hashicorp/raft/bench
internal/goarch
internal/byteorder
internal/unsafeheader
internal/goos
internal/coverage/rtcov
internal/goexperiment
internal/asan
internal/cpu
internal/profilerecord
internal/godebugs
internal/runtime/atomic
internal/msan
internal/abi
internal/runtime/math
internal/runtime/sys
internal/chacha8rand
internal/runtime/syscall
sync/atomic
math/bits
unicode
unicode/utf8
container/list
internal/itoa
crypto/internal/fips140/alias
math
crypto/internal/fips140deps/byteorder
cmp
crypto/internal/boring/sig
internal/runtime/exithook
encoding
internal/bytealg
crypto/internal/fips140deps/cpu
unicode/utf16
log/internal
internal/nettrace
vendor/golang.org/x/crypto/cryptobyte/asn1
vendor/golang.org/x/crypto/internal/alias
internal/stringslite
internal/race
internal/sync
internal/runtime/maps
runtime
iter
internal/reflectlite
sync
crypto/internal/fips140/subtle
weak
maps
slices
crypto/subtle
errors
sort
internal/testlog
internal/bisect
internal/singleflight
unique
internal/oserror
path
io
math/rand/v2
strconv
runtime/cgo
github.com/hashicorp/golang-lru/simplelru
vendor/golang.org/x/net/dns/dnsmessage
syscall
internal/godebug
strings
bytes
internal/saferio
hash
crypto/internal/randutil
hash/crc32
hash/crc64
crypto
crypto/internal/fips140deps/godebug
net/netip
math/rand
reflect
vendor/golang.org/x/text/transform
github.com/hashicorp/go-immutable-radix
bufio
crypto/internal/impl
net/http/internal/ascii
crypto/internal/fips140
regexp/syntax
html
crypto/internal/fips140/sha3
crypto/internal/fips140/sha256
crypto/internal/fips140/sha512
crypto/tls/internal/fips140tls
crypto/sha3
crypto/internal/fips140/hmac
crypto/internal/fips140/check
crypto/internal/fips140hash
internal/syscall/execenv
internal/syscall/unix
time
crypto/internal/fips140/edwards25519/field
crypto/internal/fips140/nistec/fiat
crypto/internal/fips140/aes
crypto/internal/fips140/bigmod
crypto/internal/fips140/hkdf
crypto/internal/fips140/tls12
crypto/internal/fips140/tls13
crypto/internal/fips140/edwards25519
regexp
context
io/fs
internal/poll
internal/filepathlite
os
internal/fmtsort
encoding/binary
crypto/internal/fips140/nistec
encoding/base64
vendor/golang.org/x/crypto/internal/poly1305
golang.org/x/sys/unix
encoding/pem
crypto/internal/sysrand
io/ioutil
os/signal
internal/sysinfo
path/filepath
net
fmt
crypto/internal/entropy
crypto/internal/fips140/drbg
crypto/internal/fips140only
crypto/internal/fips140/aes/gcm
crypto/internal/fips140/ed25519
crypto/internal/fips140/rsa
crypto/internal/fips140/mlkem
crypto/md5
crypto/rc4
crypto/cipher
crypto/internal/boring
crypto/des
vendor/golang.org/x/crypto/chacha20
log
crypto/internal/fips140/ecdh
encoding/json
crypto/internal/fips140/ecdsa
compress/flate
crypto/aes
math/big
crypto/sha512
crypto/hmac
vendor/golang.org/x/crypto/chacha20poly1305
crypto/sha1
crypto/ecdh
crypto/sha256
encoding/hex
net/url
vendor/golang.org/x/text/unicode/norm
vendor/golang.org/x/net/http2/hpack
vendor/golang.org/x/text/unicode/bidi
mime
mime/quotedprintable
compress/gzip
net/http/internal
encoding/gob
go/token
text/template/parse
flag
vendor/golang.org/x/text/secure/bidirule
runtime/debug
runtime/trace
github.com/mattn/go-isatty
testing
vendor/golang.org/x/net/idna
github.com/mattn/go-colorable
github.com/fatih/color
text/template
crypto/rand
crypto/internal/boring/bbig
crypto/elliptic
crypto/dsa
encoding/asn1
crypto/ed25519
crypto/internal/hpke
github.com/hashicorp/go-hclog
crypto/rsa
vendor/golang.org/x/crypto/cryptobyte
crypto/x509/pkix
html/template
crypto/ecdsa
net/textproto
vendor/golang.org/x/net/http/httpproxy
crypto/x509
vendor/golang.org/x/net/http/httpguts
mime/multipart
crypto/tls
net/http/httptrace
net/http
net/rpc
github.com/armon/go-metrics
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/raft
github.com/hashicorp/raft/bench
   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
dh_auto_test
	cd obj-i686-linux-gnu && go test -vet=off -v -p 11 github.com/hashicorp/raft github.com/hashicorp/raft/bench
=== RUN   TestCommitment_setVoters
--- PASS: TestCommitment_setVoters (0.00s)
=== RUN   TestCommitment_match_max
--- PASS: TestCommitment_match_max (0.00s)
=== RUN   TestCommitment_match_nonVoting
--- PASS: TestCommitment_match_nonVoting (0.00s)
=== RUN   TestCommitment_recalculate
--- PASS: TestCommitment_recalculate (0.00s)
=== RUN   TestCommitment_recalculate_startIndex
--- PASS: TestCommitment_recalculate_startIndex (0.00s)
=== RUN   TestCommitment_noVoterSanity
--- PASS: TestCommitment_noVoterSanity (0.00s)
=== RUN   TestCommitment_singleVoter
--- PASS: TestCommitment_singleVoter (0.00s)
=== RUN   TestConfiguration_Configuration_Clone
--- PASS: TestConfiguration_Configuration_Clone (0.00s)
=== RUN   TestConfiguration_configurations_Clone
--- PASS: TestConfiguration_configurations_Clone (0.00s)
=== RUN   TestConfiguration_hasVote
--- PASS: TestConfiguration_hasVote (0.00s)
=== RUN   TestConfiguration_checkConfiguration
--- PASS: TestConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_nextConfiguration_table
--- PASS: TestConfiguration_nextConfiguration_table (0.00s)
=== RUN   TestConfiguration_nextConfiguration_prevIndex
--- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
=== RUN   TestConfiguration_nextConfiguration_checkConfiguration
--- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_encodeDecodePeers
--- PASS: TestConfiguration_encodeDecodePeers (0.00s)
=== RUN   TestConfiguration_encodeDecodeConfiguration
--- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
=== RUN   TestDiscardSnapshotStoreImpl
--- PASS: TestDiscardSnapshotStoreImpl (0.00s)
=== RUN   TestDiscardSnapshotSinkImpl
--- PASS: TestDiscardSnapshotSinkImpl (0.00s)
=== RUN   TestFileSnapshotStoreImpl
--- PASS: TestFileSnapshotStoreImpl (0.00s)
=== RUN   TestFileSnapshotSinkImpl
--- PASS: TestFileSnapshotSinkImpl (0.00s)
=== RUN   TestFileSS_CreateSnapshotMissingParentDir
2025-02-23T18:41:35.706Z [INFO]  creating new snapshot: path=/tmp/raft349328007/raft132662650/snapshots/3-10-1740336095706.tmp
--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.01s)
=== RUN   TestFileSS_CreateSnapshot
2025-02-23T18:41:35.714Z [INFO]  creating new snapshot: path=/tmp/raft4100441017/snapshots/3-10-1740336095714.tmp
2025-02-23T18:41:35.720Z [WARN]  found temporary snapshot: name=3-10-1740336095714.tmp
--- PASS: TestFileSS_CreateSnapshot (0.01s)
=== RUN   TestFileSS_CancelSnapshot
2025-02-23T18:41:35.727Z [INFO]  creating new snapshot: path=/tmp/raft2017438400/snapshots/3-10-1740336095727.tmp
--- PASS: TestFileSS_CancelSnapshot (0.00s)
=== RUN   TestFileSS_Retention
2025-02-23T18:41:35.731Z [INFO]  creating new snapshot: path=/tmp/raft674548150/snapshots/3-10-1740336095731.tmp
2025-02-23T18:41:35.738Z [INFO]  creating new snapshot: path=/tmp/raft674548150/snapshots/3-11-1740336095738.tmp
2025-02-23T18:41:35.745Z [INFO]  creating new snapshot: path=/tmp/raft674548150/snapshots/3-12-1740336095745.tmp
2025-02-23T18:41:35.752Z [INFO]  reaping snapshot: path=/tmp/raft674548150/snapshots/3-10-1740336095731
2025-02-23T18:41:35.753Z [INFO]  creating new snapshot: path=/tmp/raft674548150/snapshots/3-13-1740336095753.tmp
2025-02-23T18:41:35.761Z [INFO]  reaping snapshot: path=/tmp/raft674548150/snapshots/3-11-1740336095738
2025-02-23T18:41:35.761Z [INFO]  creating new snapshot: path=/tmp/raft674548150/snapshots/3-14-1740336095761.tmp
2025-02-23T18:41:35.776Z [INFO]  reaping snapshot: path=/tmp/raft674548150/snapshots/3-12-1740336095745
--- PASS: TestFileSS_Retention (0.05s)
=== RUN   TestFileSS_BadPerm
--- PASS: TestFileSS_BadPerm (0.00s)
=== RUN   TestFileSS_MissingParentDir
--- PASS: TestFileSS_MissingParentDir (0.00s)
=== RUN   TestFileSS_Ordering
2025-02-23T18:41:35.778Z [INFO]  creating new snapshot: path=/tmp/raft1136465207/snapshots/5-130350-1740336095778.tmp
2025-02-23T18:41:35.785Z [INFO]  creating new snapshot: path=/tmp/raft1136465207/snapshots/36-204917-1740336095785.tmp
--- PASS: TestFileSS_Ordering (0.02s)
=== RUN   TestDeferFutureSuccess
--- PASS: TestDeferFutureSuccess (0.00s)
=== RUN   TestDeferFutureError
--- PASS: TestDeferFutureError (0.00s)
=== RUN   TestDeferFutureConcurrent
--- PASS: TestDeferFutureConcurrent (0.00s)
=== RUN   TestInmemSnapshotStoreImpl
--- PASS: TestInmemSnapshotStoreImpl (0.00s)
=== RUN   TestInmemSnapshotSinkImpl
--- PASS: TestInmemSnapshotSinkImpl (0.00s)
=== RUN   TestInmemSS_CreateSnapshot
--- PASS: TestInmemSS_CreateSnapshot (0.00s)
=== RUN   TestInmemSS_OpenSnapshotTwice
--- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
=== RUN   TestInmemTransportImpl
--- PASS: TestInmemTransportImpl (0.00s)
=== RUN   TestInmemTransportWriteTimeout
--- PASS: TestInmemTransportWriteTimeout (0.01s)
=== RUN   TestRaft_Integ
--- SKIP: TestRaft_Integ (0.00s)
=== RUN   TestRaft_RestartFollower_LongInitialHeartbeat
--- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s)
=== RUN   TestLogCache
--- PASS: TestLogCache (0.00s)
=== RUN   TestLogCacheWithBackendStoreError
--- PASS: TestLogCacheWithBackendStoreError (0.00s)
=== RUN   TestOldestLog
=== RUN   TestOldestLog/empty_logs
=== RUN   TestOldestLog/simple_case
--- PASS: TestOldestLog (0.00s)
    --- PASS: TestOldestLog/empty_logs (0.00s)
    --- PASS: TestOldestLog/simple_case (0.00s)
=== RUN   TestEmitsLogStoreMetrics
--- PASS: TestEmitsLogStoreMetrics (0.01s)
=== RUN   TestNetworkTransport_CloseStreams
--- PASS: TestNetworkTransport_CloseStreams (0.01s)
=== RUN   TestNetworkTransport_StartStop
--- PASS: TestNetworkTransport_StartStop (0.00s)
=== RUN   TestNetworkTransport_Heartbeat_FastPath
--- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s)
=== RUN   TestNetworkTransport_AppendEntries
--- PASS: TestNetworkTransport_AppendEntries (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline
--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.01s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_CloseStreams
--- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.02s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0
2025-02-23T18:41:35.907Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10
2025-02-23T18:41:35.959Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
2025-02-23T18:41:36.011Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s)
=== RUN   TestNetworkTransport_RequestVote
--- PASS: TestNetworkTransport_RequestVote (0.00s)
=== RUN   TestNetworkTransport_InstallSnapshot
--- PASS: TestNetworkTransport_InstallSnapshot (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode
--- PASS: TestNetworkTransport_EncodeDecode (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode_AddressProvider
--- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
=== RUN   TestNetworkTransport_PooledConn
--- PASS: TestNetworkTransport_PooledConn (0.01s)
=== RUN   TestNetworkTransport_ListenBackoff
    net_transport_test.go:839: countingWriter: 2025-02-23T18:41:36.029Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-02-23T18:41:36.034Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-02-23T18:41:36.045Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-02-23T18:41:36.065Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-02-23T18:41:36.106Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-02-23T18:41:36.186Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-02-23T18:41:36.347Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-02-23T18:41:36.668Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-02-23T18:41:37.310Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-02-23T18:41:38.312Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-02-23T18:41:39.314Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
--- PASS: TestNetworkTransport_ListenBackoff (4.00s)
=== RUN   TestPeersJSON_BadConfiguration
--- PASS: TestPeersJSON_BadConfiguration (0.00s)
=== RUN   TestPeersJSON_ReadPeersJSON
--- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
=== RUN   TestPeersJSON_ReadConfigJSON
--- PASS: TestPeersJSON_ReadConfigJSON (0.00s)
=== RUN   TestRaft_StartStop
2025-02-23T18:41:40.032Z [INFO]  server-f369e35e-7d8f-ff6f-c499-5dcbf56a86d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f369e35e-7d8f-ff6f-c499-5dcbf56a86d4 Address:f369e35e-7d8f-ff6f-c499-5dcbf56a86d4}]"
--- PASS: TestRaft_StartStop (0.00s)
=== RUN   TestRaft_AfterShutdown
2025-02-23T18:41:40.033Z [INFO]  server-447dda21-0962-a6f6-e77d-9d875a78ded1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-447dda21-0962-a6f6-e77d-9d875a78ded1 Address:447dda21-0962-a6f6-e77d-9d875a78ded1}]"
--- PASS: TestRaft_AfterShutdown (0.00s)
=== RUN   TestRaft_LiveBootstrap
2025-02-23T18:41:40.034Z [INFO]  server-40837b1a-c071-4beb-56d5-b1dcf22606da: initial configuration: index=0 servers=[]
2025-02-23T18:41:40.034Z [INFO]  server-490e9e26-8b8c-bd17-546a-723b6a77dc88: initial configuration: index=0 servers=[]
2025-02-23T18:41:40.034Z [INFO]  server-425e9235-57c2-5374-93ee-37a60f78c3c7: initial configuration: index=0 servers=[]
2025-02-23T18:41:40.034Z [INFO]  server-40837b1a-c071-4beb-56d5-b1dcf22606da: entering follower state: follower="Node at 40837b1a-c071-4beb-56d5-b1dcf22606da [Follower]" leader-address= leader-id=
2025-02-23T18:41:40.034Z [INFO]  server-490e9e26-8b8c-bd17-546a-723b6a77dc88: entering follower state: follower="Node at 490e9e26-8b8c-bd17-546a-723b6a77dc88 [Follower]" leader-address= leader-id=
2025-02-23T18:41:40.034Z [INFO]  server-425e9235-57c2-5374-93ee-37a60f78c3c7: entering follower state: follower="Node at 425e9235-57c2-5374-93ee-37a60f78c3c7 [Follower]" leader-address= leader-id=
2025-02-23T18:41:40.034Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:41:40.087Z [WARN]  server-40837b1a-c071-4beb-56d5-b1dcf22606da: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:40.087Z [INFO]  server-40837b1a-c071-4beb-56d5-b1dcf22606da: entering candidate state: node="Node at 40837b1a-c071-4beb-56d5-b1dcf22606da [Candidate]" term=2
2025-02-23T18:41:40.087Z [INFO]  server-40837b1a-c071-4beb-56d5-b1dcf22606da: election won: term=2 tally=2
2025-02-23T18:41:40.087Z [INFO]  server-40837b1a-c071-4beb-56d5-b1dcf22606da: entering leader state: leader="Node at 40837b1a-c071-4beb-56d5-b1dcf22606da [Leader]"
2025-02-23T18:41:40.087Z [INFO]  server-40837b1a-c071-4beb-56d5-b1dcf22606da: added peer, starting replication: peer=server-490e9e26-8b8c-bd17-546a-723b6a77dc88
2025-02-23T18:41:40.087Z [INFO]  server-40837b1a-c071-4beb-56d5-b1dcf22606da: added peer, starting replication: peer=server-425e9235-57c2-5374-93ee-37a60f78c3c7
2025-02-23T18:41:40.087Z [WARN]  server-490e9e26-8b8c-bd17-546a-723b6a77dc88: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-02-23T18:41:40.087Z [WARN]  server-425e9235-57c2-5374-93ee-37a60f78c3c7: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-02-23T18:41:40.087Z [WARN]  server-40837b1a-c071-4beb-56d5-b1dcf22606da: appendEntries rejected, sending older logs: peer="{Voter server-490e9e26-8b8c-bd17-546a-723b6a77dc88 490e9e26-8b8c-bd17-546a-723b6a77dc88}" next=1
2025-02-23T18:41:40.087Z [WARN]  server-40837b1a-c071-4beb-56d5-b1dcf22606da: appendEntries rejected, sending older logs: peer="{Voter server-425e9235-57c2-5374-93ee-37a60f78c3c7 425e9235-57c2-5374-93ee-37a60f78c3c7}" next=1
2025-02-23T18:41:40.087Z [INFO]  server-40837b1a-c071-4beb-56d5-b1dcf22606da: pipelining replication: peer="{Voter server-490e9e26-8b8c-bd17-546a-723b6a77dc88 490e9e26-8b8c-bd17-546a-723b6a77dc88}"
2025-02-23T18:41:40.087Z [INFO]  server-40837b1a-c071-4beb-56d5-b1dcf22606da: pipelining replication: peer="{Voter server-425e9235-57c2-5374-93ee-37a60f78c3c7 425e9235-57c2-5374-93ee-37a60f78c3c7}"
2025-02-23T18:41:40.192Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:41:40.087364248 +0000 UTC m=+4.385625265 (2 nodes), 52.656797ms from start of poll, 53.1567ms from cluster start. Timeout at 2025-02-23 18:41:40.192369022 +0000 UTC m=+4.490629720, 105.004455ms after stability
2025-02-23T18:41:40.192Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:40.298Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:40.192903133 +0000 UTC m=+4.491163866 (1 nodes), 3.083µs from start of poll, 158.695301ms from cluster start. Timeout at 2025-02-23 18:41:40.297916677 +0000 UTC m=+4.596177576, 105.01371ms after stability
2025-02-23T18:41:40.310Z [INFO]  server-40837b1a-c071-4beb-56d5-b1dcf22606da: aborting pipeline replication: peer="{Voter server-425e9235-57c2-5374-93ee-37a60f78c3c7 425e9235-57c2-5374-93ee-37a60f78c3c7}"
2025-02-23T18:41:40.310Z [INFO]  server-40837b1a-c071-4beb-56d5-b1dcf22606da: aborting pipeline replication: peer="{Voter server-490e9e26-8b8c-bd17-546a-723b6a77dc88 490e9e26-8b8c-bd17-546a-723b6a77dc88}"
--- PASS: TestRaft_LiveBootstrap (0.28s)
=== RUN   TestRaft_LiveBootstrap_From_NonVoter
2025-02-23T18:41:40.311Z [INFO]  server-0d62e9e5-b98f-6ac2-d8e8-1bb4fecb8cec: initial configuration: index=0 servers=[]
2025-02-23T18:41:40.311Z [INFO]  server-9a57bcf0-2d39-a833-4470-a1605265c229: initial configuration: index=0 servers=[]
2025-02-23T18:41:40.311Z [INFO]  server-0d62e9e5-b98f-6ac2-d8e8-1bb4fecb8cec: entering follower state: follower="Node at 0d62e9e5-b98f-6ac2-d8e8-1bb4fecb8cec [Follower]" leader-address= leader-id=
2025-02-23T18:41:40.311Z [INFO]  server-9a57bcf0-2d39-a833-4470-a1605265c229: entering follower state: follower="Node at 9a57bcf0-2d39-a833-4470-a1605265c229 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s)
=== RUN   TestRaft_RecoverCluster_NoState
2025-02-23T18:41:40.312Z [INFO]  server-9e8a8b45-64f4-cc3c-5547-1c499c14121c: initial configuration: index=0 servers=[]
--- PASS: TestRaft_RecoverCluster_NoState (0.00s)
=== RUN   TestRaft_RecoverCluster
=== RUN   TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs
2025-02-23T18:41:40.313Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ff73dd2-6316-1cf5-8843-4653f05dc080 Address:2ff73dd2-6316-1cf5-8843-4653f05dc080} {Suffrage:Voter ID:server-0543b0f9-f7ff-04da-ea99-67a64041e66f Address:0543b0f9-f7ff-04da-ea99-67a64041e66f} {Suffrage:Voter ID:server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5 Address:c9ce9c09-31f3-2d1e-395e-4298af3b53d5}]"
2025-02-23T18:41:40.314Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: entering follower state: follower="Node at 2ff73dd2-6316-1cf5-8843-4653f05dc080 [Follower]" leader-address= leader-id=
2025-02-23T18:41:40.314Z [INFO]  server-0543b0f9-f7ff-04da-ea99-67a64041e66f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ff73dd2-6316-1cf5-8843-4653f05dc080 Address:2ff73dd2-6316-1cf5-8843-4653f05dc080} {Suffrage:Voter ID:server-0543b0f9-f7ff-04da-ea99-67a64041e66f Address:0543b0f9-f7ff-04da-ea99-67a64041e66f} {Suffrage:Voter ID:server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5 Address:c9ce9c09-31f3-2d1e-395e-4298af3b53d5}]"
2025-02-23T18:41:40.314Z [INFO]  server-0543b0f9-f7ff-04da-ea99-67a64041e66f: entering follower state: follower="Node at 0543b0f9-f7ff-04da-ea99-67a64041e66f [Follower]" leader-address= leader-id=
2025-02-23T18:41:40.314Z [INFO]  server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ff73dd2-6316-1cf5-8843-4653f05dc080 Address:2ff73dd2-6316-1cf5-8843-4653f05dc080} {Suffrage:Voter ID:server-0543b0f9-f7ff-04da-ea99-67a64041e66f Address:0543b0f9-f7ff-04da-ea99-67a64041e66f} {Suffrage:Voter ID:server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5 Address:c9ce9c09-31f3-2d1e-395e-4298af3b53d5}]"
2025-02-23T18:41:40.314Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:40.314Z [INFO]  server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5: entering follower state: follower="Node at c9ce9c09-31f3-2d1e-395e-4298af3b53d5 [Follower]" leader-address= leader-id=
2025-02-23T18:41:40.377Z [WARN]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:40.377Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: entering candidate state: node="Node at 2ff73dd2-6316-1cf5-8843-4653f05dc080 [Candidate]" term=2
2025-02-23T18:41:40.377Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: election won: term=2 tally=2
2025-02-23T18:41:40.377Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: entering leader state: leader="Node at 2ff73dd2-6316-1cf5-8843-4653f05dc080 [Leader]"
2025-02-23T18:41:40.377Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: added peer, starting replication: peer=server-0543b0f9-f7ff-04da-ea99-67a64041e66f
2025-02-23T18:41:40.377Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: added peer, starting replication: peer=server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5
2025-02-23T18:41:40.378Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: pipelining replication: peer="{Voter server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5 c9ce9c09-31f3-2d1e-395e-4298af3b53d5}"
2025-02-23T18:41:40.378Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: pipelining replication: peer="{Voter server-0543b0f9-f7ff-04da-ea99-67a64041e66f 0543b0f9-f7ff-04da-ea99-67a64041e66f}"
2025-02-23T18:41:40.483Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:40.378010705 +0000 UTC m=+4.676271613 (1 nodes), 63.529808ms from start of poll, 64.203349ms from cluster start. Timeout at 2025-02-23 18:41:40.483021424 +0000 UTC m=+4.781282131, 105.010518ms after stability
2025-02-23T18:41:40.483Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: aborting pipeline replication: peer="{Voter server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5 c9ce9c09-31f3-2d1e-395e-4298af3b53d5}"
2025-02-23T18:41:40.483Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: aborting pipeline replication: peer="{Voter server-0543b0f9-f7ff-04da-ea99-67a64041e66f 0543b0f9-f7ff-04da-ea99-67a64041e66f}"
2025-02-23T18:41:40.484Z [INFO]  creating new snapshot: path=/tmp/raft2202915389/snapshots/2-2-1740336100484.tmp
2025-02-23T18:41:40.485Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: starting restore from snapshot: id=2-2-1740336100484 last-index=2 last-term=2 size-in-bytes=1
2025-02-23T18:41:40.485Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: snapshot restore progress: id=2-2-1740336100484 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-02-23T18:41:40.485Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: restored from snapshot: id=2-2-1740336100484 last-index=2 last-term=2 size-in-bytes=1
2025-02-23T18:41:40.485Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ff73dd2-6316-1cf5-8843-4653f05dc080 Address:2ff73dd2-6316-1cf5-8843-4653f05dc080} {Suffrage:Voter ID:server-0543b0f9-f7ff-04da-ea99-67a64041e66f Address:0543b0f9-f7ff-04da-ea99-67a64041e66f} {Suffrage:Voter ID:server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5 Address:c9ce9c09-31f3-2d1e-395e-4298af3b53d5}]"
2025-02-23T18:41:40.485Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: entering follower state: follower="Node at 2ff73dd2-6316-1cf5-8843-4653f05dc080 [Follower]" leader-address= leader-id=
2025-02-23T18:41:40.485Z [INFO]  creating new snapshot: path=/tmp/raft3949062445/snapshots/2-2-1740336100485.tmp
2025-02-23T18:41:40.486Z [INFO]  server-0543b0f9-f7ff-04da-ea99-67a64041e66f: starting restore from snapshot: id=2-2-1740336100485 last-index=2 last-term=2 size-in-bytes=1
2025-02-23T18:41:40.486Z [INFO]  server-0543b0f9-f7ff-04da-ea99-67a64041e66f: snapshot restore progress: id=2-2-1740336100485 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-02-23T18:41:40.487Z [INFO]  server-0543b0f9-f7ff-04da-ea99-67a64041e66f: restored from snapshot: id=2-2-1740336100485 last-index=2 last-term=2 size-in-bytes=1
2025-02-23T18:41:40.487Z [INFO]  server-0543b0f9-f7ff-04da-ea99-67a64041e66f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ff73dd2-6316-1cf5-8843-4653f05dc080 Address:2ff73dd2-6316-1cf5-8843-4653f05dc080} {Suffrage:Voter ID:server-0543b0f9-f7ff-04da-ea99-67a64041e66f Address:0543b0f9-f7ff-04da-ea99-67a64041e66f} {Suffrage:Voter ID:server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5 Address:c9ce9c09-31f3-2d1e-395e-4298af3b53d5}]"
2025-02-23T18:41:40.487Z [INFO]  server-0543b0f9-f7ff-04da-ea99-67a64041e66f: entering follower state: follower="Node at 0543b0f9-f7ff-04da-ea99-67a64041e66f [Follower]" leader-address= leader-id=
2025-02-23T18:41:40.487Z [INFO]  creating new snapshot: path=/tmp/raft1360374778/snapshots/2-2-1740336100487.tmp
2025-02-23T18:41:40.488Z [INFO]  server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5: starting restore from snapshot: id=2-2-1740336100487 last-index=2 last-term=2 size-in-bytes=1
2025-02-23T18:41:40.488Z [INFO]  server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5: snapshot restore progress: id=2-2-1740336100487 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-02-23T18:41:40.488Z [INFO]  server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5: restored from snapshot: id=2-2-1740336100487 last-index=2 last-term=2 size-in-bytes=1
2025-02-23T18:41:40.488Z [INFO]  server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ff73dd2-6316-1cf5-8843-4653f05dc080 Address:2ff73dd2-6316-1cf5-8843-4653f05dc080} {Suffrage:Voter ID:server-0543b0f9-f7ff-04da-ea99-67a64041e66f Address:0543b0f9-f7ff-04da-ea99-67a64041e66f} {Suffrage:Voter ID:server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5 Address:c9ce9c09-31f3-2d1e-395e-4298af3b53d5}]"
2025-02-23T18:41:40.488Z [INFO]  server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5: entering follower state: follower="Node at c9ce9c09-31f3-2d1e-395e-4298af3b53d5 [Follower]" leader-address= leader-id=
2025-02-23T18:41:40.539Z [WARN]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:40.539Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: entering candidate state: node="Node at 2ff73dd2-6316-1cf5-8843-4653f05dc080 [Candidate]" term=3
2025-02-23T18:41:40.539Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: election won: term=3 tally=2
2025-02-23T18:41:40.539Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: entering leader state: leader="Node at 2ff73dd2-6316-1cf5-8843-4653f05dc080 [Leader]"
2025-02-23T18:41:40.539Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: added peer, starting replication: peer=server-0543b0f9-f7ff-04da-ea99-67a64041e66f
2025-02-23T18:41:40.539Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: added peer, starting replication: peer=server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5
2025-02-23T18:41:40.539Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: pipelining replication: peer="{Voter server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5 c9ce9c09-31f3-2d1e-395e-4298af3b53d5}"
2025-02-23T18:41:40.540Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: pipelining replication: peer="{Voter server-0543b0f9-f7ff-04da-ea99-67a64041e66f 0543b0f9-f7ff-04da-ea99-67a64041e66f}"
2025-02-23T18:41:40.804Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:40.909Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:40.804528622 +0000 UTC m=+5.102789294 (1 nodes), 56.375µs from start of poll, 490.72103ms from cluster start. Timeout at 2025-02-23 18:41:40.909627101 +0000 UTC m=+5.207887901, 105.098607ms after stability
2025-02-23T18:41:40.909Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: aborting pipeline replication: peer="{Voter server-c9ce9c09-31f3-2d1e-395e-4298af3b53d5 c9ce9c09-31f3-2d1e-395e-4298af3b53d5}"
2025-02-23T18:41:40.909Z [INFO]  server-2ff73dd2-6316-1cf5-8843-4653f05dc080: aborting pipeline replication: peer="{Voter server-0543b0f9-f7ff-04da-ea99-67a64041e66f 0543b0f9-f7ff-04da-ea99-67a64041e66f}"
=== RUN   TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs
2025-02-23T18:41:40.916Z [INFO]  server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696 Address:2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696} {Suffrage:Voter ID:server-f1fde852-c749-e9ed-661d-7ec481998b6a Address:f1fde852-c749-e9ed-661d-7ec481998b6a} {Suffrage:Voter ID:server-a44db830-53a0-646d-f673-25542c398681 Address:a44db830-53a0-646d-f673-25542c398681}]"
2025-02-23T18:41:40.916Z [INFO]  server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696: entering follower state: follower="Node at 2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696 [Follower]" leader-address= leader-id=
2025-02-23T18:41:40.916Z [INFO]  server-f1fde852-c749-e9ed-661d-7ec481998b6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696 Address:2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696} {Suffrage:Voter ID:server-f1fde852-c749-e9ed-661d-7ec481998b6a Address:f1fde852-c749-e9ed-661d-7ec481998b6a} {Suffrage:Voter ID:server-a44db830-53a0-646d-f673-25542c398681 Address:a44db830-53a0-646d-f673-25542c398681}]"
2025-02-23T18:41:40.916Z [INFO]  server-f1fde852-c749-e9ed-661d-7ec481998b6a: entering follower state: follower="Node at f1fde852-c749-e9ed-661d-7ec481998b6a [Follower]" leader-address= leader-id=
2025-02-23T18:41:40.917Z [INFO]  server-a44db830-53a0-646d-f673-25542c398681: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696 Address:2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696} {Suffrage:Voter ID:server-f1fde852-c749-e9ed-661d-7ec481998b6a Address:f1fde852-c749-e9ed-661d-7ec481998b6a} {Suffrage:Voter ID:server-a44db830-53a0-646d-f673-25542c398681 Address:a44db830-53a0-646d-f673-25542c398681}]"
2025-02-23T18:41:40.917Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:40.917Z [INFO]  server-a44db830-53a0-646d-f673-25542c398681: entering follower state: follower="Node at a44db830-53a0-646d-f673-25542c398681 [Follower]" leader-address= leader-id=
2025-02-23T18:41:40.967Z [WARN]  server-a44db830-53a0-646d-f673-25542c398681: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:40.967Z [INFO]  server-a44db830-53a0-646d-f673-25542c398681: entering candidate state: node="Node at a44db830-53a0-646d-f673-25542c398681 [Candidate]" term=2
2025-02-23T18:41:40.968Z [INFO]  server-a44db830-53a0-646d-f673-25542c398681: election won: term=2 tally=2
2025-02-23T18:41:40.968Z [INFO]  server-a44db830-53a0-646d-f673-25542c398681: entering leader state: leader="Node at a44db830-53a0-646d-f673-25542c398681 [Leader]"
2025-02-23T18:41:40.968Z [INFO]  server-a44db830-53a0-646d-f673-25542c398681: added peer, starting replication: peer=server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696
2025-02-23T18:41:40.968Z [INFO]  server-a44db830-53a0-646d-f673-25542c398681: added peer, starting replication: peer=server-f1fde852-c749-e9ed-661d-7ec481998b6a
2025-02-23T18:41:40.968Z [INFO]  server-a44db830-53a0-646d-f673-25542c398681: pipelining replication: peer="{Voter server-f1fde852-c749-e9ed-661d-7ec481998b6a f1fde852-c749-e9ed-661d-7ec481998b6a}"
2025-02-23T18:41:40.968Z [INFO]  server-a44db830-53a0-646d-f673-25542c398681: pipelining replication: peer="{Voter server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696 2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696}"
2025-02-23T18:41:41.074Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:40.968156118 +0000 UTC m=+5.266416979 (1 nodes), 50.922595ms from start of poll, 51.64148ms from cluster start. Timeout at 2025-02-23 18:41:41.073165716 +0000 UTC m=+5.371426391, 105.009412ms after stability
2025-02-23T18:41:41.075Z [INFO]  server-a44db830-53a0-646d-f673-25542c398681: aborting pipeline replication: peer="{Voter server-f1fde852-c749-e9ed-661d-7ec481998b6a f1fde852-c749-e9ed-661d-7ec481998b6a}"
2025-02-23T18:41:41.075Z [INFO]  server-a44db830-53a0-646d-f673-25542c398681: aborting pipeline replication: peer="{Voter server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696 2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696}"
2025-02-23T18:41:41.075Z [INFO]  creating new snapshot: path=/tmp/raft2039988846/snapshots/2-6-1740336101075.tmp
2025-02-23T18:41:41.076Z [INFO]  server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696: starting restore from snapshot: id=2-6-1740336101075 last-index=6 last-term=2 size-in-bytes=25
2025-02-23T18:41:41.076Z [INFO]  server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696: snapshot restore progress: id=2-6-1740336101075 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-02-23T18:41:41.077Z [INFO]  server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696: restored from snapshot: id=2-6-1740336101075 last-index=6 last-term=2 size-in-bytes=25
2025-02-23T18:41:41.077Z [INFO]  server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696 Address:2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696} {Suffrage:Voter ID:server-f1fde852-c749-e9ed-661d-7ec481998b6a Address:f1fde852-c749-e9ed-661d-7ec481998b6a} {Suffrage:Voter ID:server-a44db830-53a0-646d-f673-25542c398681 Address:a44db830-53a0-646d-f673-25542c398681}]"
2025-02-23T18:41:41.077Z [INFO]  server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696: entering follower state: follower="Node at 2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696 [Follower]" leader-address= leader-id=
2025-02-23T18:41:41.077Z [INFO]  creating new snapshot: path=/tmp/raft779691393/snapshots/2-6-1740336101077.tmp
2025-02-23T18:41:41.078Z [INFO]  server-f1fde852-c749-e9ed-661d-7ec481998b6a: starting restore from snapshot: id=2-6-1740336101077 last-index=6 last-term=2 size-in-bytes=25
2025-02-23T18:41:41.078Z [INFO]  server-f1fde852-c749-e9ed-661d-7ec481998b6a: snapshot restore progress: id=2-6-1740336101077 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-02-23T18:41:41.078Z [INFO]  server-f1fde852-c749-e9ed-661d-7ec481998b6a: restored from snapshot: id=2-6-1740336101077 last-index=6 last-term=2 size-in-bytes=25
2025-02-23T18:41:41.078Z [INFO]  server-f1fde852-c749-e9ed-661d-7ec481998b6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696 Address:2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696} {Suffrage:Voter ID:server-f1fde852-c749-e9ed-661d-7ec481998b6a Address:f1fde852-c749-e9ed-661d-7ec481998b6a} {Suffrage:Voter ID:server-a44db830-53a0-646d-f673-25542c398681 Address:a44db830-53a0-646d-f673-25542c398681}]"
2025-02-23T18:41:41.078Z [INFO]  server-f1fde852-c749-e9ed-661d-7ec481998b6a: entering follower state: follower="Node at f1fde852-c749-e9ed-661d-7ec481998b6a [Follower]" leader-address= leader-id=
2025-02-23T18:41:41.078Z [INFO]  creating new snapshot: path=/tmp/raft692152663/snapshots/2-6-1740336101078.tmp
2025-02-23T18:41:41.079Z [INFO]  server-a44db830-53a0-646d-f673-25542c398681: starting restore from snapshot: id=2-6-1740336101078 last-index=6 last-term=2 size-in-bytes=25
2025-02-23T18:41:41.080Z [INFO]  server-a44db830-53a0-646d-f673-25542c398681: snapshot restore progress: id=2-6-1740336101078 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-02-23T18:41:41.080Z [INFO]  server-a44db830-53a0-646d-f673-25542c398681: restored from snapshot: id=2-6-1740336101078 last-index=6 last-term=2 size-in-bytes=25
2025-02-23T18:41:41.080Z [INFO]  server-a44db830-53a0-646d-f673-25542c398681: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696 Address:2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696} {Suffrage:Voter ID:server-f1fde852-c749-e9ed-661d-7ec481998b6a Address:f1fde852-c749-e9ed-661d-7ec481998b6a} {Suffrage:Voter ID:server-a44db830-53a0-646d-f673-25542c398681 Address:a44db830-53a0-646d-f673-25542c398681}]"
2025-02-23T18:41:41.081Z [INFO]  server-a44db830-53a0-646d-f673-25542c398681: entering follower state: follower="Node at a44db830-53a0-646d-f673-25542c398681 [Follower]" leader-address= leader-id=
2025-02-23T18:41:41.146Z [WARN]  server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:41.146Z [INFO]  server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696: entering candidate state: node="Node at 2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696 [Candidate]" term=3
2025-02-23T18:41:41.146Z [INFO]  server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696: election won: term=3 tally=2
2025-02-23T18:41:41.146Z [INFO]  server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696: entering leader state: leader="Node at 2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696 [Leader]"
2025-02-23T18:41:41.146Z [INFO]  server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696: added peer, starting replication: peer=server-f1fde852-c749-e9ed-661d-7ec481998b6a
2025-02-23T18:41:41.146Z [INFO]  server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696: added peer, starting replication: peer=server-a44db830-53a0-646d-f673-25542c398681
2025-02-23T18:41:41.146Z [INFO]  server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696: pipelining replication: peer="{Voter server-f1fde852-c749-e9ed-661d-7ec481998b6a f1fde852-c749-e9ed-661d-7ec481998b6a}"
2025-02-23T18:41:41.147Z [INFO]  server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696: pipelining replication: peer="{Voter server-a44db830-53a0-646d-f673-25542c398681 a44db830-53a0-646d-f673-25542c398681}"
2025-02-23T18:41:41.396Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:41.501Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:41.396654881 +0000 UTC m=+5.694915530 (1 nodes), 37.764µs from start of poll, 480.140031ms from cluster start. Timeout at 2025-02-23 18:41:41.501667533 +0000 UTC m=+5.799928196, 105.012666ms after stability
2025-02-23T18:41:41.502Z [INFO]  server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696: aborting pipeline replication: peer="{Voter server-a44db830-53a0-646d-f673-25542c398681 a44db830-53a0-646d-f673-25542c398681}"
2025-02-23T18:41:41.502Z [INFO]  server-2cc402bd-90a0-5a2f-9a3a-7a01b9fe5696: aborting pipeline replication: peer="{Voter server-f1fde852-c749-e9ed-661d-7ec481998b6a f1fde852-c749-e9ed-661d-7ec481998b6a}"
=== RUN   TestRaft_RecoverCluster/snapshot,_with_trailing_logs
2025-02-23T18:41:41.503Z [INFO]  server-32d73030-ed34-91d8-eb47-b770a2628c84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32d73030-ed34-91d8-eb47-b770a2628c84 Address:32d73030-ed34-91d8-eb47-b770a2628c84} {Suffrage:Voter ID:server-9e0b2fc7-94a1-8c91-3218-cb6842980d72 Address:9e0b2fc7-94a1-8c91-3218-cb6842980d72} {Suffrage:Voter ID:server-2efc6e51-f73a-caf2-c370-669d7d85534c Address:2efc6e51-f73a-caf2-c370-669d7d85534c}]"
2025-02-23T18:41:41.504Z [INFO]  server-9e0b2fc7-94a1-8c91-3218-cb6842980d72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32d73030-ed34-91d8-eb47-b770a2628c84 Address:32d73030-ed34-91d8-eb47-b770a2628c84} {Suffrage:Voter ID:server-9e0b2fc7-94a1-8c91-3218-cb6842980d72 Address:9e0b2fc7-94a1-8c91-3218-cb6842980d72} {Suffrage:Voter ID:server-2efc6e51-f73a-caf2-c370-669d7d85534c Address:2efc6e51-f73a-caf2-c370-669d7d85534c}]"
2025-02-23T18:41:41.504Z [INFO]  server-32d73030-ed34-91d8-eb47-b770a2628c84: entering follower state: follower="Node at 32d73030-ed34-91d8-eb47-b770a2628c84 [Follower]" leader-address= leader-id=
2025-02-23T18:41:41.504Z [INFO]  server-2efc6e51-f73a-caf2-c370-669d7d85534c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32d73030-ed34-91d8-eb47-b770a2628c84 Address:32d73030-ed34-91d8-eb47-b770a2628c84} {Suffrage:Voter ID:server-9e0b2fc7-94a1-8c91-3218-cb6842980d72 Address:9e0b2fc7-94a1-8c91-3218-cb6842980d72} {Suffrage:Voter ID:server-2efc6e51-f73a-caf2-c370-669d7d85534c Address:2efc6e51-f73a-caf2-c370-669d7d85534c}]"
2025-02-23T18:41:41.504Z [INFO]  server-9e0b2fc7-94a1-8c91-3218-cb6842980d72: entering follower state: follower="Node at 9e0b2fc7-94a1-8c91-3218-cb6842980d72 [Follower]" leader-address= leader-id=
2025-02-23T18:41:41.504Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:41.504Z [INFO]  server-2efc6e51-f73a-caf2-c370-669d7d85534c: entering follower state: follower="Node at 2efc6e51-f73a-caf2-c370-669d7d85534c [Follower]" leader-address= leader-id=
2025-02-23T18:41:41.554Z [WARN]  server-32d73030-ed34-91d8-eb47-b770a2628c84: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:41.554Z [INFO]  server-32d73030-ed34-91d8-eb47-b770a2628c84: entering candidate state: node="Node at 32d73030-ed34-91d8-eb47-b770a2628c84 [Candidate]" term=2
2025-02-23T18:41:41.554Z [INFO]  server-32d73030-ed34-91d8-eb47-b770a2628c84: election won: term=2 tally=2
2025-02-23T18:41:41.554Z [INFO]  server-32d73030-ed34-91d8-eb47-b770a2628c84: entering leader state: leader="Node at 32d73030-ed34-91d8-eb47-b770a2628c84 [Leader]"
2025-02-23T18:41:41.554Z [INFO]  server-32d73030-ed34-91d8-eb47-b770a2628c84: added peer, starting replication: peer=server-9e0b2fc7-94a1-8c91-3218-cb6842980d72
2025-02-23T18:41:41.554Z [INFO]  server-32d73030-ed34-91d8-eb47-b770a2628c84: added peer, starting replication: peer=server-2efc6e51-f73a-caf2-c370-669d7d85534c
2025-02-23T18:41:41.554Z [INFO]  server-32d73030-ed34-91d8-eb47-b770a2628c84: pipelining replication: peer="{Voter server-2efc6e51-f73a-caf2-c370-669d7d85534c 2efc6e51-f73a-caf2-c370-669d7d85534c}"
2025-02-23T18:41:41.555Z [INFO]  server-32d73030-ed34-91d8-eb47-b770a2628c84: pipelining replication: peer="{Voter server-9e0b2fc7-94a1-8c91-3218-cb6842980d72 9e0b2fc7-94a1-8c91-3218-cb6842980d72}"
2025-02-23T18:41:41.660Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:41.554801327 +0000 UTC m=+5.853062075 (1 nodes), 50.48906ms from start of poll, 50.940583ms from cluster start. Timeout at 2025-02-23 18:41:41.659806418 +0000 UTC m=+5.958067089, 105.005014ms after stability
2025-02-23T18:41:41.669Z [INFO]  server-32d73030-ed34-91d8-eb47-b770a2628c84: aborting pipeline replication: peer="{Voter server-2efc6e51-f73a-caf2-c370-669d7d85534c 2efc6e51-f73a-caf2-c370-669d7d85534c}"
2025-02-23T18:41:41.669Z [INFO]  server-32d73030-ed34-91d8-eb47-b770a2628c84: aborting pipeline replication: peer="{Voter server-9e0b2fc7-94a1-8c91-3218-cb6842980d72 9e0b2fc7-94a1-8c91-3218-cb6842980d72}"
2025-02-23T18:41:41.669Z [INFO]  creating new snapshot: path=/tmp/raft2395731783/snapshots/2-27-1740336101669.tmp
2025-02-23T18:41:41.670Z [INFO]  server-32d73030-ed34-91d8-eb47-b770a2628c84: starting restore from snapshot: id=2-27-1740336101669 last-index=27 last-term=2 size-in-bytes=168
2025-02-23T18:41:41.671Z [INFO]  server-32d73030-ed34-91d8-eb47-b770a2628c84: snapshot restore progress: id=2-27-1740336101669 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-02-23T18:41:41.671Z [INFO]  server-32d73030-ed34-91d8-eb47-b770a2628c84: restored from snapshot: id=2-27-1740336101669 last-index=27 last-term=2 size-in-bytes=168
2025-02-23T18:41:41.671Z [INFO]  server-32d73030-ed34-91d8-eb47-b770a2628c84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32d73030-ed34-91d8-eb47-b770a2628c84 Address:32d73030-ed34-91d8-eb47-b770a2628c84} {Suffrage:Voter ID:server-9e0b2fc7-94a1-8c91-3218-cb6842980d72 Address:9e0b2fc7-94a1-8c91-3218-cb6842980d72} {Suffrage:Voter ID:server-2efc6e51-f73a-caf2-c370-669d7d85534c Address:2efc6e51-f73a-caf2-c370-669d7d85534c}]"
2025-02-23T18:41:41.671Z [INFO]  server-32d73030-ed34-91d8-eb47-b770a2628c84: entering follower state: follower="Node at 32d73030-ed34-91d8-eb47-b770a2628c84 [Follower]" leader-address= leader-id=
2025-02-23T18:41:41.671Z [INFO]  creating new snapshot: path=/tmp/raft1854573533/snapshots/2-27-1740336101671.tmp
2025-02-23T18:41:41.672Z [INFO]  server-9e0b2fc7-94a1-8c91-3218-cb6842980d72: starting restore from snapshot: id=2-27-1740336101671 last-index=27 last-term=2 size-in-bytes=168
2025-02-23T18:41:41.672Z [INFO]  server-9e0b2fc7-94a1-8c91-3218-cb6842980d72: snapshot restore progress: id=2-27-1740336101671 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-02-23T18:41:41.672Z [INFO]  server-9e0b2fc7-94a1-8c91-3218-cb6842980d72: restored from snapshot: id=2-27-1740336101671 last-index=27 last-term=2 size-in-bytes=168
2025-02-23T18:41:41.672Z [INFO]  server-9e0b2fc7-94a1-8c91-3218-cb6842980d72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32d73030-ed34-91d8-eb47-b770a2628c84 Address:32d73030-ed34-91d8-eb47-b770a2628c84} {Suffrage:Voter ID:server-9e0b2fc7-94a1-8c91-3218-cb6842980d72 Address:9e0b2fc7-94a1-8c91-3218-cb6842980d72} {Suffrage:Voter ID:server-2efc6e51-f73a-caf2-c370-669d7d85534c Address:2efc6e51-f73a-caf2-c370-669d7d85534c}]"
2025-02-23T18:41:41.672Z [INFO]  server-9e0b2fc7-94a1-8c91-3218-cb6842980d72: entering follower state: follower="Node at 9e0b2fc7-94a1-8c91-3218-cb6842980d72 [Follower]" leader-address= leader-id=
2025-02-23T18:41:41.672Z [INFO]  creating new snapshot: path=/tmp/raft3066155408/snapshots/2-27-1740336101672.tmp
2025-02-23T18:41:41.673Z [INFO]  server-2efc6e51-f73a-caf2-c370-669d7d85534c: starting restore from snapshot: id=2-27-1740336101672 last-index=27 last-term=2 size-in-bytes=168
2025-02-23T18:41:41.673Z [INFO]  server-2efc6e51-f73a-caf2-c370-669d7d85534c: snapshot restore progress: id=2-27-1740336101672 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-02-23T18:41:41.673Z [INFO]  server-2efc6e51-f73a-caf2-c370-669d7d85534c: restored from snapshot: id=2-27-1740336101672 last-index=27 last-term=2 size-in-bytes=168
2025-02-23T18:41:41.673Z [INFO]  server-2efc6e51-f73a-caf2-c370-669d7d85534c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32d73030-ed34-91d8-eb47-b770a2628c84 Address:32d73030-ed34-91d8-eb47-b770a2628c84} {Suffrage:Voter ID:server-9e0b2fc7-94a1-8c91-3218-cb6842980d72 Address:9e0b2fc7-94a1-8c91-3218-cb6842980d72} {Suffrage:Voter ID:server-2efc6e51-f73a-caf2-c370-669d7d85534c Address:2efc6e51-f73a-caf2-c370-669d7d85534c}]"
2025-02-23T18:41:41.673Z [INFO]  server-2efc6e51-f73a-caf2-c370-669d7d85534c: entering follower state: follower="Node at 2efc6e51-f73a-caf2-c370-669d7d85534c [Follower]" leader-address= leader-id=
2025-02-23T18:41:41.725Z [WARN]  server-2efc6e51-f73a-caf2-c370-669d7d85534c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:41.725Z [INFO]  server-2efc6e51-f73a-caf2-c370-669d7d85534c: entering candidate state: node="Node at 2efc6e51-f73a-caf2-c370-669d7d85534c [Candidate]" term=3
2025-02-23T18:41:41.725Z [INFO]  server-2efc6e51-f73a-caf2-c370-669d7d85534c: election won: term=3 tally=2
2025-02-23T18:41:41.725Z [INFO]  server-2efc6e51-f73a-caf2-c370-669d7d85534c: entering leader state: leader="Node at 2efc6e51-f73a-caf2-c370-669d7d85534c [Leader]"
2025-02-23T18:41:41.725Z [INFO]  server-2efc6e51-f73a-caf2-c370-669d7d85534c: added peer, starting replication: peer=server-32d73030-ed34-91d8-eb47-b770a2628c84
2025-02-23T18:41:41.725Z [INFO]  server-2efc6e51-f73a-caf2-c370-669d7d85534c: added peer, starting replication: peer=server-9e0b2fc7-94a1-8c91-3218-cb6842980d72
2025-02-23T18:41:41.725Z [INFO]  server-2efc6e51-f73a-caf2-c370-669d7d85534c: pipelining replication: peer="{Voter server-32d73030-ed34-91d8-eb47-b770a2628c84 32d73030-ed34-91d8-eb47-b770a2628c84}"
2025-02-23T18:41:41.726Z [INFO]  server-2efc6e51-f73a-caf2-c370-669d7d85534c: pipelining replication: peer="{Voter server-9e0b2fc7-94a1-8c91-3218-cb6842980d72 9e0b2fc7-94a1-8c91-3218-cb6842980d72}"
2025-02-23T18:41:41.989Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:42.094Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:41.989205195 +0000 UTC m=+6.287465899 (1 nodes), 50.689µs from start of poll, 485.344407ms from cluster start. Timeout at 2025-02-23 18:41:42.094211954 +0000 UTC m=+6.392472619, 105.00672ms after stability
2025-02-23T18:41:42.094Z [INFO]  server-2efc6e51-f73a-caf2-c370-669d7d85534c: aborting pipeline replication: peer="{Voter server-9e0b2fc7-94a1-8c91-3218-cb6842980d72 9e0b2fc7-94a1-8c91-3218-cb6842980d72}"
2025-02-23T18:41:42.094Z [INFO]  server-2efc6e51-f73a-caf2-c370-669d7d85534c: aborting pipeline replication: peer="{Voter server-32d73030-ed34-91d8-eb47-b770a2628c84 32d73030-ed34-91d8-eb47-b770a2628c84}"
--- PASS: TestRaft_RecoverCluster (1.78s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.60s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.59s)
    --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.59s)
=== RUN   TestRaft_HasExistingState
    raft_test.go:258: Skipping unreliable test
--- SKIP: TestRaft_HasExistingState (0.00s)
=== RUN   TestRaft_SingleNode
2025-02-23T18:41:42.096Z [INFO]  server-9efffe66-af85-a49d-c98b-6b1ce51ce0ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9efffe66-af85-a49d-c98b-6b1ce51ce0ff Address:9efffe66-af85-a49d-c98b-6b1ce51ce0ff}]"
2025-02-23T18:41:42.096Z [INFO]  server-9efffe66-af85-a49d-c98b-6b1ce51ce0ff: entering follower state: follower="Node at 9efffe66-af85-a49d-c98b-6b1ce51ce0ff [Follower]" leader-address= leader-id=
2025-02-23T18:41:42.156Z [WARN]  server-9efffe66-af85-a49d-c98b-6b1ce51ce0ff: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:42.156Z [INFO]  server-9efffe66-af85-a49d-c98b-6b1ce51ce0ff: entering candidate state: node="Node at 9efffe66-af85-a49d-c98b-6b1ce51ce0ff [Candidate]" term=2
2025-02-23T18:41:42.156Z [INFO]  server-9efffe66-af85-a49d-c98b-6b1ce51ce0ff: election won: term=2 tally=1
2025-02-23T18:41:42.156Z [INFO]  server-9efffe66-af85-a49d-c98b-6b1ce51ce0ff: entering leader state: leader="Node at 9efffe66-af85-a49d-c98b-6b1ce51ce0ff [Leader]"
--- PASS: TestRaft_SingleNode (0.06s)
=== RUN   TestRaft_TripleNode
2025-02-23T18:41:42.157Z [INFO]  server-e15d26af-ef75-8705-ccf4-cdad37efa770: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e15d26af-ef75-8705-ccf4-cdad37efa770 Address:e15d26af-ef75-8705-ccf4-cdad37efa770} {Suffrage:Voter ID:server-d1999e54-6dee-96a5-4e4b-8af5593635ac Address:d1999e54-6dee-96a5-4e4b-8af5593635ac} {Suffrage:Voter ID:server-4c1798cf-0494-49b6-9163-31055b399547 Address:4c1798cf-0494-49b6-9163-31055b399547}]"
2025-02-23T18:41:42.157Z [INFO]  server-e15d26af-ef75-8705-ccf4-cdad37efa770: entering follower state: follower="Node at e15d26af-ef75-8705-ccf4-cdad37efa770 [Follower]" leader-address= leader-id=
2025-02-23T18:41:42.157Z [INFO]  server-d1999e54-6dee-96a5-4e4b-8af5593635ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e15d26af-ef75-8705-ccf4-cdad37efa770 Address:e15d26af-ef75-8705-ccf4-cdad37efa770} {Suffrage:Voter ID:server-d1999e54-6dee-96a5-4e4b-8af5593635ac Address:d1999e54-6dee-96a5-4e4b-8af5593635ac} {Suffrage:Voter ID:server-4c1798cf-0494-49b6-9163-31055b399547 Address:4c1798cf-0494-49b6-9163-31055b399547}]"
2025-02-23T18:41:42.157Z [INFO]  server-d1999e54-6dee-96a5-4e4b-8af5593635ac: entering follower state: follower="Node at d1999e54-6dee-96a5-4e4b-8af5593635ac [Follower]" leader-address= leader-id=
2025-02-23T18:41:42.157Z [INFO]  server-4c1798cf-0494-49b6-9163-31055b399547: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e15d26af-ef75-8705-ccf4-cdad37efa770 Address:e15d26af-ef75-8705-ccf4-cdad37efa770} {Suffrage:Voter ID:server-d1999e54-6dee-96a5-4e4b-8af5593635ac Address:d1999e54-6dee-96a5-4e4b-8af5593635ac} {Suffrage:Voter ID:server-4c1798cf-0494-49b6-9163-31055b399547 Address:4c1798cf-0494-49b6-9163-31055b399547}]"
2025-02-23T18:41:42.158Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:41:42.158Z [INFO]  server-4c1798cf-0494-49b6-9163-31055b399547: entering follower state: follower="Node at 4c1798cf-0494-49b6-9163-31055b399547 [Follower]" leader-address= leader-id=
2025-02-23T18:41:42.235Z [WARN]  server-4c1798cf-0494-49b6-9163-31055b399547: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:42.235Z [INFO]  server-4c1798cf-0494-49b6-9163-31055b399547: entering candidate state: node="Node at 4c1798cf-0494-49b6-9163-31055b399547 [Candidate]" term=2
2025-02-23T18:41:42.235Z [INFO]  server-4c1798cf-0494-49b6-9163-31055b399547: election won: term=2 tally=2
2025-02-23T18:41:42.235Z [INFO]  server-4c1798cf-0494-49b6-9163-31055b399547: entering leader state: leader="Node at 4c1798cf-0494-49b6-9163-31055b399547 [Leader]"
2025-02-23T18:41:42.235Z [INFO]  server-4c1798cf-0494-49b6-9163-31055b399547: added peer, starting replication: peer=server-e15d26af-ef75-8705-ccf4-cdad37efa770
2025-02-23T18:41:42.235Z [INFO]  server-4c1798cf-0494-49b6-9163-31055b399547: added peer, starting replication: peer=server-d1999e54-6dee-96a5-4e4b-8af5593635ac
2025-02-23T18:41:42.236Z [INFO]  server-4c1798cf-0494-49b6-9163-31055b399547: pipelining replication: peer="{Voter server-e15d26af-ef75-8705-ccf4-cdad37efa770 e15d26af-ef75-8705-ccf4-cdad37efa770}"
2025-02-23T18:41:42.236Z [INFO]  server-4c1798cf-0494-49b6-9163-31055b399547: pipelining replication: peer="{Voter server-d1999e54-6dee-96a5-4e4b-8af5593635ac d1999e54-6dee-96a5-4e4b-8af5593635ac}"
2025-02-23T18:41:42.340Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:41:42.235894211 +0000 UTC m=+6.534154812 (2 nodes), 77.840909ms from start of poll, 78.337978ms from cluster start. Timeout at 2025-02-23 18:41:42.340897765 +0000 UTC m=+6.639158408, 105.003596ms after stability
2025-02-23T18:41:42.341Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:42.447Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:42.341057853 +0000 UTC m=+6.639318470 (1 nodes), 1.868µs from start of poll, 183.501636ms from cluster start. Timeout at 2025-02-23 18:41:42.446069837 +0000 UTC m=+6.744330686, 105.012216ms after stability
2025-02-23T18:41:42.459Z [INFO]  server-4c1798cf-0494-49b6-9163-31055b399547: aborting pipeline replication: peer="{Voter server-d1999e54-6dee-96a5-4e4b-8af5593635ac d1999e54-6dee-96a5-4e4b-8af5593635ac}"
2025-02-23T18:41:42.459Z [INFO]  server-4c1798cf-0494-49b6-9163-31055b399547: aborting pipeline replication: peer="{Voter server-e15d26af-ef75-8705-ccf4-cdad37efa770 e15d26af-ef75-8705-ccf4-cdad37efa770}"
--- PASS: TestRaft_TripleNode (0.30s)
=== RUN   TestRaft_LeaderFail
2025-02-23T18:41:42.460Z [INFO]  server-20305761-4440-5a13-5506-aff1f331c1c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20305761-4440-5a13-5506-aff1f331c1c7 Address:20305761-4440-5a13-5506-aff1f331c1c7} {Suffrage:Voter ID:server-a17deb03-dfea-0f40-e02a-d5ee04ab2154 Address:a17deb03-dfea-0f40-e02a-d5ee04ab2154} {Suffrage:Voter ID:server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e Address:17a0bee8-7d1d-3fbe-c48c-35bcec31125e}]"
2025-02-23T18:41:42.460Z [INFO]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20305761-4440-5a13-5506-aff1f331c1c7 Address:20305761-4440-5a13-5506-aff1f331c1c7} {Suffrage:Voter ID:server-a17deb03-dfea-0f40-e02a-d5ee04ab2154 Address:a17deb03-dfea-0f40-e02a-d5ee04ab2154} {Suffrage:Voter ID:server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e Address:17a0bee8-7d1d-3fbe-c48c-35bcec31125e}]"
2025-02-23T18:41:42.461Z [INFO]  server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-20305761-4440-5a13-5506-aff1f331c1c7 Address:20305761-4440-5a13-5506-aff1f331c1c7} {Suffrage:Voter ID:server-a17deb03-dfea-0f40-e02a-d5ee04ab2154 Address:a17deb03-dfea-0f40-e02a-d5ee04ab2154} {Suffrage:Voter ID:server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e Address:17a0bee8-7d1d-3fbe-c48c-35bcec31125e}]"
2025-02-23T18:41:42.461Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:41:42.461Z [INFO]  server-20305761-4440-5a13-5506-aff1f331c1c7: entering follower state: follower="Node at 20305761-4440-5a13-5506-aff1f331c1c7 [Follower]" leader-address= leader-id=
2025-02-23T18:41:42.461Z [INFO]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: entering follower state: follower="Node at a17deb03-dfea-0f40-e02a-d5ee04ab2154 [Follower]" leader-address= leader-id=
2025-02-23T18:41:42.461Z [INFO]  server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e: entering follower state: follower="Node at 17a0bee8-7d1d-3fbe-c48c-35bcec31125e [Follower]" leader-address= leader-id=
2025-02-23T18:41:42.520Z [WARN]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:42.520Z [INFO]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: entering candidate state: node="Node at a17deb03-dfea-0f40-e02a-d5ee04ab2154 [Candidate]" term=2
2025-02-23T18:41:42.521Z [INFO]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: election won: term=2 tally=2
2025-02-23T18:41:42.521Z [INFO]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: entering leader state: leader="Node at a17deb03-dfea-0f40-e02a-d5ee04ab2154 [Leader]"
2025-02-23T18:41:42.521Z [INFO]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: added peer, starting replication: peer=server-20305761-4440-5a13-5506-aff1f331c1c7
2025-02-23T18:41:42.521Z [INFO]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: added peer, starting replication: peer=server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e
2025-02-23T18:41:42.521Z [INFO]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: pipelining replication: peer="{Voter server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e 17a0bee8-7d1d-3fbe-c48c-35bcec31125e}"
2025-02-23T18:41:42.521Z [INFO]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: pipelining replication: peer="{Voter server-20305761-4440-5a13-5506-aff1f331c1c7 20305761-4440-5a13-5506-aff1f331c1c7}"
2025-02-23T18:41:42.626Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:41:42.521170163 +0000 UTC m=+6.819430804 (2 nodes), 59.982308ms from start of poll, 60.445113ms from cluster start. Timeout at 2025-02-23 18:41:42.626176354 +0000 UTC m=+6.924437031, 105.006227ms after stability
2025-02-23T18:41:42.626Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:42.732Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:42.626807813 +0000 UTC m=+6.925068600 (1 nodes), 2.585µs from start of poll, 166.082909ms from cluster start. Timeout at 2025-02-23 18:41:42.731823737 +0000 UTC m=+7.030084444, 105.015844ms after stability
    raft_test.go:378: [INFO] Disconnecting Node at a17deb03-dfea-0f40-e02a-d5ee04ab2154 [Leader]
2025-02-23T18:41:42.747Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to pipeline appendEntries: peer="{Voter server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e 17a0bee8-7d1d-3fbe-c48c-35bcec31125e}" error="append pipeline closed"
2025-02-23T18:41:42.747Z [INFO]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: aborting pipeline replication: peer="{Voter server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e 17a0bee8-7d1d-3fbe-c48c-35bcec31125e}"
2025-02-23T18:41:42.749Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:42.749Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to heartbeat to: peer=17a0bee8-7d1d-3fbe-c48c-35bcec31125e backoff time=10ms error="failed to connect to peer: 17a0bee8-7d1d-3fbe-c48c-35bcec31125e"
2025-02-23T18:41:42.750Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to heartbeat to: peer=20305761-4440-5a13-5506-aff1f331c1c7 backoff time=10ms error="failed to connect to peer: 20305761-4440-5a13-5506-aff1f331c1c7"
2025-02-23T18:41:42.751Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to pipeline appendEntries: peer="{Voter server-20305761-4440-5a13-5506-aff1f331c1c7 20305761-4440-5a13-5506-aff1f331c1c7}" error="append pipeline closed"
2025-02-23T18:41:42.751Z [INFO]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: aborting pipeline replication: peer="{Voter server-20305761-4440-5a13-5506-aff1f331c1c7 20305761-4440-5a13-5506-aff1f331c1c7}"
2025-02-23T18:41:42.753Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to appendEntries to: peer="{Voter server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e 17a0bee8-7d1d-3fbe-c48c-35bcec31125e}" error="failed to connect to peer: 17a0bee8-7d1d-3fbe-c48c-35bcec31125e"
2025-02-23T18:41:42.757Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to appendEntries to: peer="{Voter server-20305761-4440-5a13-5506-aff1f331c1c7 20305761-4440-5a13-5506-aff1f331c1c7}" error="failed to connect to peer: 20305761-4440-5a13-5506-aff1f331c1c7"
2025-02-23T18:41:42.770Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to appendEntries to: peer="{Voter server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e 17a0bee8-7d1d-3fbe-c48c-35bcec31125e}" error="failed to connect to peer: 17a0bee8-7d1d-3fbe-c48c-35bcec31125e"
2025-02-23T18:41:42.770Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to heartbeat to: peer=20305761-4440-5a13-5506-aff1f331c1c7 backoff time=10ms error="failed to connect to peer: 20305761-4440-5a13-5506-aff1f331c1c7"
2025-02-23T18:41:42.770Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to heartbeat to: peer=17a0bee8-7d1d-3fbe-c48c-35bcec31125e backoff time=10ms error="failed to connect to peer: 17a0bee8-7d1d-3fbe-c48c-35bcec31125e"
2025-02-23T18:41:42.777Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to appendEntries to: peer="{Voter server-20305761-4440-5a13-5506-aff1f331c1c7 20305761-4440-5a13-5506-aff1f331c1c7}" error="failed to connect to peer: 20305761-4440-5a13-5506-aff1f331c1c7"
2025-02-23T18:41:42.785Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to heartbeat to: peer=17a0bee8-7d1d-3fbe-c48c-35bcec31125e backoff time=10ms error="failed to connect to peer: 17a0bee8-7d1d-3fbe-c48c-35bcec31125e"
2025-02-23T18:41:42.789Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to appendEntries to: peer="{Voter server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e 17a0bee8-7d1d-3fbe-c48c-35bcec31125e}" error="failed to connect to peer: 17a0bee8-7d1d-3fbe-c48c-35bcec31125e"
2025-02-23T18:41:42.790Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to heartbeat to: peer=20305761-4440-5a13-5506-aff1f331c1c7 backoff time=10ms error="failed to connect to peer: 20305761-4440-5a13-5506-aff1f331c1c7"
2025-02-23T18:41:42.792Z [WARN]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to contact: server-id=server-20305761-4440-5a13-5506-aff1f331c1c7 time=51.054929ms
2025-02-23T18:41:42.795Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to appendEntries to: peer="{Voter server-20305761-4440-5a13-5506-aff1f331c1c7 20305761-4440-5a13-5506-aff1f331c1c7}" error="failed to connect to peer: 20305761-4440-5a13-5506-aff1f331c1c7"
2025-02-23T18:41:42.801Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to heartbeat to: peer=17a0bee8-7d1d-3fbe-c48c-35bcec31125e backoff time=20ms error="failed to connect to peer: 17a0bee8-7d1d-3fbe-c48c-35bcec31125e"
2025-02-23T18:41:42.803Z [WARN]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to contact: server-id=server-20305761-4440-5a13-5506-aff1f331c1c7 time=61.233246ms
2025-02-23T18:41:42.803Z [WARN]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to contact: server-id=server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e time=60.155916ms
2025-02-23T18:41:42.803Z [WARN]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to contact quorum of nodes, stepping down
2025-02-23T18:41:42.803Z [INFO]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: entering follower state: follower="Node at a17deb03-dfea-0f40-e02a-d5ee04ab2154 [Follower]" leader-address= leader-id=
2025-02-23T18:41:42.807Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to heartbeat to: peer=20305761-4440-5a13-5506-aff1f331c1c7 backoff time=20ms error="failed to connect to peer: 20305761-4440-5a13-5506-aff1f331c1c7"
2025-02-23T18:41:42.816Z [WARN]  server-20305761-4440-5a13-5506-aff1f331c1c7: heartbeat timeout reached, starting election: last-leader-addr=a17deb03-dfea-0f40-e02a-d5ee04ab2154 last-leader-id=server-a17deb03-dfea-0f40-e02a-d5ee04ab2154
2025-02-23T18:41:42.816Z [INFO]  server-20305761-4440-5a13-5506-aff1f331c1c7: entering candidate state: node="Node at 20305761-4440-5a13-5506-aff1f331c1c7 [Candidate]" term=3
2025-02-23T18:41:42.816Z [WARN]  server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e: rejecting vote request since we have a leader: from=20305761-4440-5a13-5506-aff1f331c1c7 leader=a17deb03-dfea-0f40-e02a-d5ee04ab2154 leader-id=server-a17deb03-dfea-0f40-e02a-d5ee04ab2154
2025-02-23T18:41:42.816Z [ERROR] server-20305761-4440-5a13-5506-aff1f331c1c7: failed to make requestVote RPC: target="{Voter server-a17deb03-dfea-0f40-e02a-d5ee04ab2154 a17deb03-dfea-0f40-e02a-d5ee04ab2154}" error="failed to connect to peer: a17deb03-dfea-0f40-e02a-d5ee04ab2154" term=3
2025-02-23T18:41:42.818Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to appendEntries to: peer="{Voter server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e 17a0bee8-7d1d-3fbe-c48c-35bcec31125e}" error="failed to connect to peer: 17a0bee8-7d1d-3fbe-c48c-35bcec31125e"
2025-02-23T18:41:42.822Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to appendEntries to: peer="{Voter server-20305761-4440-5a13-5506-aff1f331c1c7 20305761-4440-5a13-5506-aff1f331c1c7}" error="failed to connect to peer: 20305761-4440-5a13-5506-aff1f331c1c7"
2025-02-23T18:41:42.856Z [WARN]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:42.856Z [INFO]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: entering candidate state: node="Node at a17deb03-dfea-0f40-e02a-d5ee04ab2154 [Candidate]" term=3
2025-02-23T18:41:42.856Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to make requestVote RPC: target="{Voter server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e 17a0bee8-7d1d-3fbe-c48c-35bcec31125e}" error="failed to connect to peer: 17a0bee8-7d1d-3fbe-c48c-35bcec31125e" term=3
2025-02-23T18:41:42.856Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to make requestVote RPC: target="{Voter server-20305761-4440-5a13-5506-aff1f331c1c7 20305761-4440-5a13-5506-aff1f331c1c7}" error="failed to connect to peer: 20305761-4440-5a13-5506-aff1f331c1c7" term=3
2025-02-23T18:41:42.857Z [WARN]  server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e: heartbeat timeout reached, starting election: last-leader-addr=a17deb03-dfea-0f40-e02a-d5ee04ab2154 last-leader-id=server-a17deb03-dfea-0f40-e02a-d5ee04ab2154
2025-02-23T18:41:42.857Z [INFO]  server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e: entering candidate state: node="Node at 17a0bee8-7d1d-3fbe-c48c-35bcec31125e [Candidate]" term=3
2025-02-23T18:41:42.857Z [ERROR] server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e: failed to make requestVote RPC: target="{Voter server-a17deb03-dfea-0f40-e02a-d5ee04ab2154 a17deb03-dfea-0f40-e02a-d5ee04ab2154}" error="failed to connect to peer: a17deb03-dfea-0f40-e02a-d5ee04ab2154" term=3
2025-02-23T18:41:42.857Z [INFO]  server-20305761-4440-5a13-5506-aff1f331c1c7: duplicate requestVote for same term: term=3
2025-02-23T18:41:42.894Z [WARN]  server-20305761-4440-5a13-5506-aff1f331c1c7: Election timeout reached, restarting election
2025-02-23T18:41:42.894Z [INFO]  server-20305761-4440-5a13-5506-aff1f331c1c7: entering candidate state: node="Node at 20305761-4440-5a13-5506-aff1f331c1c7 [Candidate]" term=4
2025-02-23T18:41:42.894Z [INFO]  server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e: entering follower state: follower="Node at 17a0bee8-7d1d-3fbe-c48c-35bcec31125e [Follower]" leader-address= leader-id=
2025-02-23T18:41:42.894Z [INFO]  server-20305761-4440-5a13-5506-aff1f331c1c7: election won: term=4 tally=2
2025-02-23T18:41:42.894Z [INFO]  server-20305761-4440-5a13-5506-aff1f331c1c7: entering leader state: leader="Node at 20305761-4440-5a13-5506-aff1f331c1c7 [Leader]"
2025-02-23T18:41:42.894Z [INFO]  server-20305761-4440-5a13-5506-aff1f331c1c7: added peer, starting replication: peer=server-a17deb03-dfea-0f40-e02a-d5ee04ab2154
2025-02-23T18:41:42.895Z [INFO]  server-20305761-4440-5a13-5506-aff1f331c1c7: added peer, starting replication: peer=server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e
2025-02-23T18:41:42.895Z [INFO]  server-20305761-4440-5a13-5506-aff1f331c1c7: pipelining replication: peer="{Voter server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e 17a0bee8-7d1d-3fbe-c48c-35bcec31125e}"
2025-02-23T18:41:42.895Z [ERROR] server-20305761-4440-5a13-5506-aff1f331c1c7: failed to make requestVote RPC: target="{Voter server-a17deb03-dfea-0f40-e02a-d5ee04ab2154 a17deb03-dfea-0f40-e02a-d5ee04ab2154}" error="failed to connect to peer: a17deb03-dfea-0f40-e02a-d5ee04ab2154" term=4
2025-02-23T18:41:42.895Z [ERROR] server-20305761-4440-5a13-5506-aff1f331c1c7: failed to appendEntries to: peer="{Voter server-a17deb03-dfea-0f40-e02a-d5ee04ab2154 a17deb03-dfea-0f40-e02a-d5ee04ab2154}" error="failed to connect to peer: a17deb03-dfea-0f40-e02a-d5ee04ab2154"
2025-02-23T18:41:42.905Z [ERROR] server-20305761-4440-5a13-5506-aff1f331c1c7: failed to heartbeat to: peer=a17deb03-dfea-0f40-e02a-d5ee04ab2154 backoff time=10ms error="failed to connect to peer: a17deb03-dfea-0f40-e02a-d5ee04ab2154"
2025-02-23T18:41:42.913Z [ERROR] server-20305761-4440-5a13-5506-aff1f331c1c7: failed to appendEntries to: peer="{Voter server-a17deb03-dfea-0f40-e02a-d5ee04ab2154 a17deb03-dfea-0f40-e02a-d5ee04ab2154}" error="failed to connect to peer: a17deb03-dfea-0f40-e02a-d5ee04ab2154"
2025-02-23T18:41:42.922Z [ERROR] server-20305761-4440-5a13-5506-aff1f331c1c7: failed to heartbeat to: peer=a17deb03-dfea-0f40-e02a-d5ee04ab2154 backoff time=10ms error="failed to connect to peer: a17deb03-dfea-0f40-e02a-d5ee04ab2154"
2025-02-23T18:41:42.930Z [ERROR] server-20305761-4440-5a13-5506-aff1f331c1c7: failed to appendEntries to: peer="{Voter server-a17deb03-dfea-0f40-e02a-d5ee04ab2154 a17deb03-dfea-0f40-e02a-d5ee04ab2154}" error="failed to connect to peer: a17deb03-dfea-0f40-e02a-d5ee04ab2154"
2025-02-23T18:41:42.940Z [ERROR] server-20305761-4440-5a13-5506-aff1f331c1c7: failed to heartbeat to: peer=a17deb03-dfea-0f40-e02a-d5ee04ab2154 backoff time=10ms error="failed to connect to peer: a17deb03-dfea-0f40-e02a-d5ee04ab2154"
2025-02-23T18:41:42.946Z [WARN]  server-20305761-4440-5a13-5506-aff1f331c1c7: failed to contact: server-id=server-a17deb03-dfea-0f40-e02a-d5ee04ab2154 time=51.170668ms
2025-02-23T18:41:42.948Z [WARN]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: Election timeout reached, restarting election
2025-02-23T18:41:42.948Z [INFO]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: entering candidate state: node="Node at a17deb03-dfea-0f40-e02a-d5ee04ab2154 [Candidate]" term=4
2025-02-23T18:41:42.948Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to make requestVote RPC: target="{Voter server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e 17a0bee8-7d1d-3fbe-c48c-35bcec31125e}" error="failed to connect to peer: 17a0bee8-7d1d-3fbe-c48c-35bcec31125e" term=4
2025-02-23T18:41:42.948Z [ERROR] server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: failed to make requestVote RPC: target="{Voter server-20305761-4440-5a13-5506-aff1f331c1c7 20305761-4440-5a13-5506-aff1f331c1c7}" error="failed to connect to peer: 20305761-4440-5a13-5506-aff1f331c1c7" term=4
2025-02-23T18:41:42.956Z [ERROR] server-20305761-4440-5a13-5506-aff1f331c1c7: failed to appendEntries to: peer="{Voter server-a17deb03-dfea-0f40-e02a-d5ee04ab2154 a17deb03-dfea-0f40-e02a-d5ee04ab2154}" error="failed to connect to peer: a17deb03-dfea-0f40-e02a-d5ee04ab2154"
2025-02-23T18:41:42.957Z [ERROR] server-20305761-4440-5a13-5506-aff1f331c1c7: failed to heartbeat to: peer=a17deb03-dfea-0f40-e02a-d5ee04ab2154 backoff time=20ms error="failed to connect to peer: a17deb03-dfea-0f40-e02a-d5ee04ab2154"
2025-02-23T18:41:42.988Z [ERROR] server-20305761-4440-5a13-5506-aff1f331c1c7: failed to heartbeat to: peer=a17deb03-dfea-0f40-e02a-d5ee04ab2154 backoff time=25ms error="failed to connect to peer: a17deb03-dfea-0f40-e02a-d5ee04ab2154"
2025-02-23T18:41:42.989Z [WARN]  server-20305761-4440-5a13-5506-aff1f331c1c7: failed to contact: server-id=server-a17deb03-dfea-0f40-e02a-d5ee04ab2154 time=94.523854ms
2025-02-23T18:41:43.001Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:42.895268911 +0000 UTC m=+7.193529550 (1 nodes), 145.489484ms from start of poll, 434.543859ms from cluster start. Timeout at 2025-02-23 18:41:43.000271791 +0000 UTC m=+7.298532409, 105.002859ms after stability
    raft_test.go:413: [INFO] Reconnecting Node at a17deb03-dfea-0f40-e02a-d5ee04ab2154 [Candidate]
2025-02-23T18:41:43.005Z [INFO]  server-a17deb03-dfea-0f40-e02a-d5ee04ab2154: entering follower state: follower="Node at a17deb03-dfea-0f40-e02a-d5ee04ab2154 [Follower]" leader-address=20305761-4440-5a13-5506-aff1f331c1c7 leader-id=server-20305761-4440-5a13-5506-aff1f331c1c7
2025-02-23T18:41:43.005Z [INFO]  server-20305761-4440-5a13-5506-aff1f331c1c7: pipelining replication: peer="{Voter server-a17deb03-dfea-0f40-e02a-d5ee04ab2154 a17deb03-dfea-0f40-e02a-d5ee04ab2154}"
2025-02-23T18:41:43.011Z [INFO]  server-20305761-4440-5a13-5506-aff1f331c1c7: aborting pipeline replication: peer="{Voter server-17a0bee8-7d1d-3fbe-c48c-35bcec31125e 17a0bee8-7d1d-3fbe-c48c-35bcec31125e}"
2025-02-23T18:41:43.011Z [INFO]  server-20305761-4440-5a13-5506-aff1f331c1c7: aborting pipeline replication: peer="{Voter server-a17deb03-dfea-0f40-e02a-d5ee04ab2154 a17deb03-dfea-0f40-e02a-d5ee04ab2154}"
--- PASS: TestRaft_LeaderFail (0.55s)
=== RUN   TestRaft_BehindFollower
2025-02-23T18:41:43.012Z [INFO]  server-a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749 Address:a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749} {Suffrage:Voter ID:server-7b61457b-6eff-dde8-5647-320c92f17d0a Address:7b61457b-6eff-dde8-5647-320c92f17d0a} {Suffrage:Voter ID:server-eb8949f8-4470-8a94-e930-741ae9c7ec2b Address:eb8949f8-4470-8a94-e930-741ae9c7ec2b}]"
2025-02-23T18:41:43.012Z [INFO]  server-7b61457b-6eff-dde8-5647-320c92f17d0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749 Address:a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749} {Suffrage:Voter ID:server-7b61457b-6eff-dde8-5647-320c92f17d0a Address:7b61457b-6eff-dde8-5647-320c92f17d0a} {Suffrage:Voter ID:server-eb8949f8-4470-8a94-e930-741ae9c7ec2b Address:eb8949f8-4470-8a94-e930-741ae9c7ec2b}]"
2025-02-23T18:41:43.013Z [INFO]  server-eb8949f8-4470-8a94-e930-741ae9c7ec2b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749 Address:a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749} {Suffrage:Voter ID:server-7b61457b-6eff-dde8-5647-320c92f17d0a Address:7b61457b-6eff-dde8-5647-320c92f17d0a} {Suffrage:Voter ID:server-eb8949f8-4470-8a94-e930-741ae9c7ec2b Address:eb8949f8-4470-8a94-e930-741ae9c7ec2b}]"
2025-02-23T18:41:43.013Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:43.013Z [INFO]  server-a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749: entering follower state: follower="Node at a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749 [Follower]" leader-address= leader-id=
2025-02-23T18:41:43.013Z [INFO]  server-7b61457b-6eff-dde8-5647-320c92f17d0a: entering follower state: follower="Node at 7b61457b-6eff-dde8-5647-320c92f17d0a [Follower]" leader-address= leader-id=
2025-02-23T18:41:43.013Z [INFO]  server-eb8949f8-4470-8a94-e930-741ae9c7ec2b: entering follower state: follower="Node at eb8949f8-4470-8a94-e930-741ae9c7ec2b [Follower]" leader-address= leader-id=
2025-02-23T18:41:43.081Z [WARN]  server-eb8949f8-4470-8a94-e930-741ae9c7ec2b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:43.081Z [INFO]  server-eb8949f8-4470-8a94-e930-741ae9c7ec2b: entering candidate state: node="Node at eb8949f8-4470-8a94-e930-741ae9c7ec2b [Candidate]" term=2
2025-02-23T18:41:43.081Z [INFO]  server-eb8949f8-4470-8a94-e930-741ae9c7ec2b: election won: term=2 tally=2
2025-02-23T18:41:43.081Z [INFO]  server-eb8949f8-4470-8a94-e930-741ae9c7ec2b: entering leader state: leader="Node at eb8949f8-4470-8a94-e930-741ae9c7ec2b [Leader]"
2025-02-23T18:41:43.081Z [INFO]  server-eb8949f8-4470-8a94-e930-741ae9c7ec2b: added peer, starting replication: peer=server-a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749
2025-02-23T18:41:43.081Z [INFO]  server-eb8949f8-4470-8a94-e930-741ae9c7ec2b: added peer, starting replication: peer=server-7b61457b-6eff-dde8-5647-320c92f17d0a
2025-02-23T18:41:43.082Z [INFO]  server-eb8949f8-4470-8a94-e930-741ae9c7ec2b: pipelining replication: peer="{Voter server-a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749 a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749}"
2025-02-23T18:41:43.082Z [INFO]  server-eb8949f8-4470-8a94-e930-741ae9c7ec2b: pipelining replication: peer="{Voter server-7b61457b-6eff-dde8-5647-320c92f17d0a 7b61457b-6eff-dde8-5647-320c92f17d0a}"
2025-02-23T18:41:43.187Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:43.081876782 +0000 UTC m=+7.380137582 (1 nodes), 68.736781ms from start of poll, 69.154732ms from cluster start. Timeout at 2025-02-23 18:41:43.186881946 +0000 UTC m=+7.485142576, 105.004994ms after stability
2025-02-23T18:41:43.187Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:41:43.292Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:41:43.187120377 +0000 UTC m=+7.485381270 (2 nodes), 3.254µs from start of poll, 174.39842ms from cluster start. Timeout at 2025-02-23 18:41:43.292127424 +0000 UTC m=+7.590387554, 105.006284ms after stability
2025-02-23T18:41:43.292Z [ERROR] server-eb8949f8-4470-8a94-e930-741ae9c7ec2b: failed to pipeline appendEntries: peer="{Voter server-a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749 a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749}" error="append pipeline closed"
2025-02-23T18:41:43.292Z [INFO]  server-eb8949f8-4470-8a94-e930-741ae9c7ec2b: aborting pipeline replication: peer="{Voter server-a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749 a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749}"
2025-02-23T18:41:43.292Z [ERROR] server-eb8949f8-4470-8a94-e930-741ae9c7ec2b: failed to appendEntries to: peer="{Voter server-a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749 a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749}" error="failed to connect to peer: a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749"
2025-02-23T18:41:43.295Z [ERROR] server-eb8949f8-4470-8a94-e930-741ae9c7ec2b: failed to heartbeat to: peer=a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749 backoff time=10ms error="failed to connect to peer: a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749"
    raft_test.go:459: [INFO] Finished apply without behind follower
2025-02-23T18:41:43.303Z [INFO]  server-eb8949f8-4470-8a94-e930-741ae9c7ec2b: pipelining replication: peer="{Voter server-a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749 a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749}"
2025-02-23T18:41:43.316Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:43.421Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:43.316662251 +0000 UTC m=+7.614922967 (1 nodes), 2.416µs from start of poll, 303.940117ms from cluster start. Timeout at 2025-02-23 18:41:43.42167879 +0000 UTC m=+7.719939559, 105.016592ms after stability
2025-02-23T18:41:43.421Z [INFO]  server-eb8949f8-4470-8a94-e930-741ae9c7ec2b: aborting pipeline replication: peer="{Voter server-7b61457b-6eff-dde8-5647-320c92f17d0a 7b61457b-6eff-dde8-5647-320c92f17d0a}"
2025-02-23T18:41:43.421Z [INFO]  server-eb8949f8-4470-8a94-e930-741ae9c7ec2b: aborting pipeline replication: peer="{Voter server-a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749 a9b29f4c-2ffc-eb3a-9dd7-8e9bc11de749}"
--- PASS: TestRaft_BehindFollower (0.41s)
=== RUN   TestRaft_ApplyNonLeader
2025-02-23T18:41:43.424Z [INFO]  server-44429264-f587-4059-86d5-1b4a3367bc30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44429264-f587-4059-86d5-1b4a3367bc30 Address:44429264-f587-4059-86d5-1b4a3367bc30} {Suffrage:Voter ID:server-135b1140-7f85-142c-b701-a3050f6100d1 Address:135b1140-7f85-142c-b701-a3050f6100d1} {Suffrage:Voter ID:server-b250184c-9643-366b-c5c9-fd7668ea7648 Address:b250184c-9643-366b-c5c9-fd7668ea7648}]"
2025-02-23T18:41:43.424Z [INFO]  server-135b1140-7f85-142c-b701-a3050f6100d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44429264-f587-4059-86d5-1b4a3367bc30 Address:44429264-f587-4059-86d5-1b4a3367bc30} {Suffrage:Voter ID:server-135b1140-7f85-142c-b701-a3050f6100d1 Address:135b1140-7f85-142c-b701-a3050f6100d1} {Suffrage:Voter ID:server-b250184c-9643-366b-c5c9-fd7668ea7648 Address:b250184c-9643-366b-c5c9-fd7668ea7648}]"
2025-02-23T18:41:43.424Z [INFO]  server-44429264-f587-4059-86d5-1b4a3367bc30: entering follower state: follower="Node at 44429264-f587-4059-86d5-1b4a3367bc30 [Follower]" leader-address= leader-id=
2025-02-23T18:41:43.424Z [INFO]  server-135b1140-7f85-142c-b701-a3050f6100d1: entering follower state: follower="Node at 135b1140-7f85-142c-b701-a3050f6100d1 [Follower]" leader-address= leader-id=
2025-02-23T18:41:43.425Z [INFO]  server-b250184c-9643-366b-c5c9-fd7668ea7648: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44429264-f587-4059-86d5-1b4a3367bc30 Address:44429264-f587-4059-86d5-1b4a3367bc30} {Suffrage:Voter ID:server-135b1140-7f85-142c-b701-a3050f6100d1 Address:135b1140-7f85-142c-b701-a3050f6100d1} {Suffrage:Voter ID:server-b250184c-9643-366b-c5c9-fd7668ea7648 Address:b250184c-9643-366b-c5c9-fd7668ea7648}]"
2025-02-23T18:41:43.425Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:43.425Z [INFO]  server-b250184c-9643-366b-c5c9-fd7668ea7648: entering follower state: follower="Node at b250184c-9643-366b-c5c9-fd7668ea7648 [Follower]" leader-address= leader-id=
2025-02-23T18:41:43.486Z [WARN]  server-b250184c-9643-366b-c5c9-fd7668ea7648: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:43.486Z [INFO]  server-b250184c-9643-366b-c5c9-fd7668ea7648: entering candidate state: node="Node at b250184c-9643-366b-c5c9-fd7668ea7648 [Candidate]" term=2
2025-02-23T18:41:43.486Z [INFO]  server-b250184c-9643-366b-c5c9-fd7668ea7648: election won: term=2 tally=2
2025-02-23T18:41:43.486Z [INFO]  server-b250184c-9643-366b-c5c9-fd7668ea7648: entering leader state: leader="Node at b250184c-9643-366b-c5c9-fd7668ea7648 [Leader]"
2025-02-23T18:41:43.486Z [INFO]  server-b250184c-9643-366b-c5c9-fd7668ea7648: added peer, starting replication: peer=server-44429264-f587-4059-86d5-1b4a3367bc30
2025-02-23T18:41:43.486Z [INFO]  server-b250184c-9643-366b-c5c9-fd7668ea7648: added peer, starting replication: peer=server-135b1140-7f85-142c-b701-a3050f6100d1
2025-02-23T18:41:43.487Z [INFO]  server-b250184c-9643-366b-c5c9-fd7668ea7648: pipelining replication: peer="{Voter server-44429264-f587-4059-86d5-1b4a3367bc30 44429264-f587-4059-86d5-1b4a3367bc30}"
2025-02-23T18:41:43.487Z [INFO]  server-b250184c-9643-366b-c5c9-fd7668ea7648: pipelining replication: peer="{Voter server-135b1140-7f85-142c-b701-a3050f6100d1 135b1140-7f85-142c-b701-a3050f6100d1}"
2025-02-23T18:41:43.592Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:43.486903242 +0000 UTC m=+7.785163372 (1 nodes), 61.25043ms from start of poll, 63.394881ms from cluster start. Timeout at 2025-02-23 18:41:43.591908662 +0000 UTC m=+7.890169644, 105.006272ms after stability
2025-02-23T18:41:43.592Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:41:43.698Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:41:43.592588236 +0000 UTC m=+7.890849079 (2 nodes), 2.564µs from start of poll, 169.080588ms from cluster start. Timeout at 2025-02-23 18:41:43.697611073 +0000 UTC m=+7.995871702, 105.022623ms after stability
2025-02-23T18:41:43.698Z [INFO]  server-b250184c-9643-366b-c5c9-fd7668ea7648: aborting pipeline replication: peer="{Voter server-135b1140-7f85-142c-b701-a3050f6100d1 135b1140-7f85-142c-b701-a3050f6100d1}"
2025-02-23T18:41:43.698Z [INFO]  server-b250184c-9643-366b-c5c9-fd7668ea7648: aborting pipeline replication: peer="{Voter server-44429264-f587-4059-86d5-1b4a3367bc30 44429264-f587-4059-86d5-1b4a3367bc30}"
--- PASS: TestRaft_ApplyNonLeader (0.28s)
=== RUN   TestRaft_ApplyConcurrent
2025-02-23T18:41:43.700Z [INFO]  server-7af551ea-ac45-49cb-8561-4bcf3f0ef251: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7af551ea-ac45-49cb-8561-4bcf3f0ef251 Address:7af551ea-ac45-49cb-8561-4bcf3f0ef251} {Suffrage:Voter ID:server-4dc12b92-8f7c-0a99-c706-7477241e3b9c Address:4dc12b92-8f7c-0a99-c706-7477241e3b9c} {Suffrage:Voter ID:server-98c1d4cb-155d-b0cb-3189-beb332ac932d Address:98c1d4cb-155d-b0cb-3189-beb332ac932d}]"
2025-02-23T18:41:43.700Z [INFO]  server-7af551ea-ac45-49cb-8561-4bcf3f0ef251: entering follower state: follower="Node at 7af551ea-ac45-49cb-8561-4bcf3f0ef251 [Follower]" leader-address= leader-id=
2025-02-23T18:41:43.700Z [INFO]  server-4dc12b92-8f7c-0a99-c706-7477241e3b9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7af551ea-ac45-49cb-8561-4bcf3f0ef251 Address:7af551ea-ac45-49cb-8561-4bcf3f0ef251} {Suffrage:Voter ID:server-4dc12b92-8f7c-0a99-c706-7477241e3b9c Address:4dc12b92-8f7c-0a99-c706-7477241e3b9c} {Suffrage:Voter ID:server-98c1d4cb-155d-b0cb-3189-beb332ac932d Address:98c1d4cb-155d-b0cb-3189-beb332ac932d}]"
2025-02-23T18:41:43.700Z [INFO]  server-4dc12b92-8f7c-0a99-c706-7477241e3b9c: entering follower state: follower="Node at 4dc12b92-8f7c-0a99-c706-7477241e3b9c [Follower]" leader-address= leader-id=
2025-02-23T18:41:43.700Z [INFO]  server-98c1d4cb-155d-b0cb-3189-beb332ac932d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7af551ea-ac45-49cb-8561-4bcf3f0ef251 Address:7af551ea-ac45-49cb-8561-4bcf3f0ef251} {Suffrage:Voter ID:server-4dc12b92-8f7c-0a99-c706-7477241e3b9c Address:4dc12b92-8f7c-0a99-c706-7477241e3b9c} {Suffrage:Voter ID:server-98c1d4cb-155d-b0cb-3189-beb332ac932d Address:98c1d4cb-155d-b0cb-3189-beb332ac932d}]"
2025-02-23T18:41:43.700Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:43.700Z [INFO]  server-98c1d4cb-155d-b0cb-3189-beb332ac932d: entering follower state: follower="Node at 98c1d4cb-155d-b0cb-3189-beb332ac932d [Follower]" leader-address= leader-id=
2025-02-23T18:41:43.836Z [WARN]  server-4dc12b92-8f7c-0a99-c706-7477241e3b9c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:43.836Z [INFO]  server-4dc12b92-8f7c-0a99-c706-7477241e3b9c: entering candidate state: node="Node at 4dc12b92-8f7c-0a99-c706-7477241e3b9c [Candidate]" term=2
2025-02-23T18:41:43.836Z [INFO]  server-4dc12b92-8f7c-0a99-c706-7477241e3b9c: election won: term=2 tally=2
2025-02-23T18:41:43.836Z [INFO]  server-4dc12b92-8f7c-0a99-c706-7477241e3b9c: entering leader state: leader="Node at 4dc12b92-8f7c-0a99-c706-7477241e3b9c [Leader]"
2025-02-23T18:41:43.836Z [INFO]  server-4dc12b92-8f7c-0a99-c706-7477241e3b9c: added peer, starting replication: peer=server-7af551ea-ac45-49cb-8561-4bcf3f0ef251
2025-02-23T18:41:43.836Z [INFO]  server-4dc12b92-8f7c-0a99-c706-7477241e3b9c: added peer, starting replication: peer=server-98c1d4cb-155d-b0cb-3189-beb332ac932d
2025-02-23T18:41:43.836Z [INFO]  server-4dc12b92-8f7c-0a99-c706-7477241e3b9c: pipelining replication: peer="{Voter server-98c1d4cb-155d-b0cb-3189-beb332ac932d 98c1d4cb-155d-b0cb-3189-beb332ac932d}"
2025-02-23T18:41:43.836Z [INFO]  server-4dc12b92-8f7c-0a99-c706-7477241e3b9c: pipelining replication: peer="{Voter server-7af551ea-ac45-49cb-8561-4bcf3f0ef251 7af551ea-ac45-49cb-8561-4bcf3f0ef251}"
2025-02-23T18:41:44.042Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:43.836453014 +0000 UTC m=+8.134713803 (1 nodes), 135.966832ms from start of poll, 136.431613ms from cluster start. Timeout at 2025-02-23 18:41:44.041466579 +0000 UTC m=+8.339727345, 205.013542ms after stability
2025-02-23T18:41:44.058Z [INFO]  server-4dc12b92-8f7c-0a99-c706-7477241e3b9c: aborting pipeline replication: peer="{Voter server-98c1d4cb-155d-b0cb-3189-beb332ac932d 98c1d4cb-155d-b0cb-3189-beb332ac932d}"
2025-02-23T18:41:44.058Z [INFO]  server-4dc12b92-8f7c-0a99-c706-7477241e3b9c: aborting pipeline replication: peer="{Voter server-7af551ea-ac45-49cb-8561-4bcf3f0ef251 7af551ea-ac45-49cb-8561-4bcf3f0ef251}"
--- PASS: TestRaft_ApplyConcurrent (0.36s)
=== RUN   TestRaft_ApplyConcurrent_Timeout
2025-02-23T18:41:44.059Z [INFO]  server-6b51b466-f99f-52ff-1e17-075d58b68acf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b51b466-f99f-52ff-1e17-075d58b68acf Address:6b51b466-f99f-52ff-1e17-075d58b68acf}]"
2025-02-23T18:41:44.059Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:44.059Z [INFO]  server-6b51b466-f99f-52ff-1e17-075d58b68acf: entering follower state: follower="Node at 6b51b466-f99f-52ff-1e17-075d58b68acf [Follower]" leader-address= leader-id=
2025-02-23T18:41:44.247Z [WARN]  server-6b51b466-f99f-52ff-1e17-075d58b68acf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:44.247Z [INFO]  server-6b51b466-f99f-52ff-1e17-075d58b68acf: entering candidate state: node="Node at 6b51b466-f99f-52ff-1e17-075d58b68acf [Candidate]" term=2
2025-02-23T18:41:44.248Z [INFO]  server-6b51b466-f99f-52ff-1e17-075d58b68acf: election won: term=2 tally=1
2025-02-23T18:41:44.248Z [INFO]  server-6b51b466-f99f-52ff-1e17-075d58b68acf: entering leader state: leader="Node at 6b51b466-f99f-52ff-1e17-075d58b68acf [Leader]"
2025-02-23T18:41:44.450Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:44.249722767 +0000 UTC m=+8.547982897 (1 nodes), 190.128392ms from start of poll, 190.390519ms from cluster start. Timeout at 2025-02-23 18:41:44.450722767 +0000 UTC m=+8.748982897, 201ms after stability
--- PASS: TestRaft_ApplyConcurrent_Timeout (0.40s)
=== RUN   TestRaft_JoinNode
2025-02-23T18:41:44.456Z [INFO]  server-f56b1b6a-b618-a17f-8d41-699c4fefcfb9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f56b1b6a-b618-a17f-8d41-699c4fefcfb9 Address:f56b1b6a-b618-a17f-8d41-699c4fefcfb9} {Suffrage:Voter ID:server-f817fa59-c09e-9267-a850-629be41825b2 Address:f817fa59-c09e-9267-a850-629be41825b2}]"
2025-02-23T18:41:44.456Z [INFO]  server-f56b1b6a-b618-a17f-8d41-699c4fefcfb9: entering follower state: follower="Node at f56b1b6a-b618-a17f-8d41-699c4fefcfb9 [Follower]" leader-address= leader-id=
2025-02-23T18:41:44.456Z [INFO]  server-f817fa59-c09e-9267-a850-629be41825b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f56b1b6a-b618-a17f-8d41-699c4fefcfb9 Address:f56b1b6a-b618-a17f-8d41-699c4fefcfb9} {Suffrage:Voter ID:server-f817fa59-c09e-9267-a850-629be41825b2 Address:f817fa59-c09e-9267-a850-629be41825b2}]"
2025-02-23T18:41:44.456Z [INFO]  server-f817fa59-c09e-9267-a850-629be41825b2: entering follower state: follower="Node at f817fa59-c09e-9267-a850-629be41825b2 [Follower]" leader-address= leader-id=
2025-02-23T18:41:44.456Z [INFO]  server-2d3c84be-d21b-7d1f-a366-eb43074253cf: initial configuration: index=0 servers=[]
2025-02-23T18:41:44.456Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:44.456Z [INFO]  server-2d3c84be-d21b-7d1f-a366-eb43074253cf: entering follower state: follower="Node at 2d3c84be-d21b-7d1f-a366-eb43074253cf [Follower]" leader-address= leader-id=
2025-02-23T18:41:44.519Z [WARN]  server-f56b1b6a-b618-a17f-8d41-699c4fefcfb9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:44.519Z [INFO]  server-f56b1b6a-b618-a17f-8d41-699c4fefcfb9: entering candidate state: node="Node at f56b1b6a-b618-a17f-8d41-699c4fefcfb9 [Candidate]" term=2
2025-02-23T18:41:44.519Z [INFO]  server-f56b1b6a-b618-a17f-8d41-699c4fefcfb9: election won: term=2 tally=2
2025-02-23T18:41:44.519Z [INFO]  server-f56b1b6a-b618-a17f-8d41-699c4fefcfb9: entering leader state: leader="Node at f56b1b6a-b618-a17f-8d41-699c4fefcfb9 [Leader]"
2025-02-23T18:41:44.519Z [INFO]  server-f56b1b6a-b618-a17f-8d41-699c4fefcfb9: added peer, starting replication: peer=server-f817fa59-c09e-9267-a850-629be41825b2
2025-02-23T18:41:44.519Z [INFO]  server-f56b1b6a-b618-a17f-8d41-699c4fefcfb9: pipelining replication: peer="{Voter server-f817fa59-c09e-9267-a850-629be41825b2 f817fa59-c09e-9267-a850-629be41825b2}"
2025-02-23T18:41:44.552Z [WARN]  server-2d3c84be-d21b-7d1f-a366-eb43074253cf: no known peers, aborting election
2025-02-23T18:41:44.625Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:44.519755456 +0000 UTC m=+8.818016187 (1 nodes), 62.98888ms from start of poll, 63.636654ms from cluster start. Timeout at 2025-02-23 18:41:44.624765985 +0000 UTC m=+8.923026763, 105.010576ms after stability
2025-02-23T18:41:44.625Z [INFO]  server-f56b1b6a-b618-a17f-8d41-699c4fefcfb9: updating configuration: command=AddVoter server-id=server-2d3c84be-d21b-7d1f-a366-eb43074253cf server-addr=2d3c84be-d21b-7d1f-a366-eb43074253cf servers="[{Suffrage:Voter ID:server-f56b1b6a-b618-a17f-8d41-699c4fefcfb9 Address:f56b1b6a-b618-a17f-8d41-699c4fefcfb9} {Suffrage:Voter ID:server-f817fa59-c09e-9267-a850-629be41825b2 Address:f817fa59-c09e-9267-a850-629be41825b2} {Suffrage:Voter ID:server-2d3c84be-d21b-7d1f-a366-eb43074253cf Address:2d3c84be-d21b-7d1f-a366-eb43074253cf}]"
2025-02-23T18:41:44.625Z [INFO]  server-f56b1b6a-b618-a17f-8d41-699c4fefcfb9: added peer, starting replication: peer=server-2d3c84be-d21b-7d1f-a366-eb43074253cf
2025-02-23T18:41:44.625Z [WARN]  server-2d3c84be-d21b-7d1f-a366-eb43074253cf: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-02-23T18:41:44.625Z [WARN]  server-f56b1b6a-b618-a17f-8d41-699c4fefcfb9: appendEntries rejected, sending older logs: peer="{Voter server-2d3c84be-d21b-7d1f-a366-eb43074253cf 2d3c84be-d21b-7d1f-a366-eb43074253cf}" next=1
2025-02-23T18:41:44.625Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:44.625Z [INFO]  server-f56b1b6a-b618-a17f-8d41-699c4fefcfb9: pipelining replication: peer="{Voter server-2d3c84be-d21b-7d1f-a366-eb43074253cf 2d3c84be-d21b-7d1f-a366-eb43074253cf}"
2025-02-23T18:41:44.731Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:44.625682374 +0000 UTC m=+8.923942990 (1 nodes), 1.782µs from start of poll, 169.563457ms from cluster start. Timeout at 2025-02-23 18:41:44.730693954 +0000 UTC m=+9.028954084, 105.011094ms after stability
2025-02-23T18:41:44.731Z [INFO]  server-f56b1b6a-b618-a17f-8d41-699c4fefcfb9: aborting pipeline replication: peer="{Voter server-2d3c84be-d21b-7d1f-a366-eb43074253cf 2d3c84be-d21b-7d1f-a366-eb43074253cf}"
2025-02-23T18:41:44.731Z [INFO]  server-f56b1b6a-b618-a17f-8d41-699c4fefcfb9: aborting pipeline replication: peer="{Voter server-f817fa59-c09e-9267-a850-629be41825b2 f817fa59-c09e-9267-a850-629be41825b2}"
--- PASS: TestRaft_JoinNode (0.28s)
=== RUN   TestRaft_JoinNode_ConfigStore
2025-02-23T18:41:44.733Z [INFO]  server-e2a4f563-0394-f1c2-9a8a-819c98840f1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2a4f563-0394-f1c2-9a8a-819c98840f1f Address:e2a4f563-0394-f1c2-9a8a-819c98840f1f}]"
2025-02-23T18:41:44.733Z [INFO]  server-e2a4f563-0394-f1c2-9a8a-819c98840f1f: entering follower state: follower="Node at e2a4f563-0394-f1c2-9a8a-819c98840f1f [Follower]" leader-address= leader-id=
2025-02-23T18:41:44.733Z [INFO]  server-33fb7852-0b4d-c480-8ffa-e5d946f73afe: initial configuration: index=0 servers=[]
2025-02-23T18:41:44.733Z [INFO]  server-33fb7852-0b4d-c480-8ffa-e5d946f73afe: entering follower state: follower="Node at 33fb7852-0b4d-c480-8ffa-e5d946f73afe [Follower]" leader-address= leader-id=
2025-02-23T18:41:44.733Z [INFO]  server-6ce296e8-ef68-7237-2d71-d9dde2a38dec: initial configuration: index=0 servers=[]
2025-02-23T18:41:44.733Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:44.733Z [INFO]  server-6ce296e8-ef68-7237-2d71-d9dde2a38dec: entering follower state: follower="Node at 6ce296e8-ef68-7237-2d71-d9dde2a38dec [Follower]" leader-address= leader-id=
2025-02-23T18:41:44.792Z [WARN]  server-33fb7852-0b4d-c480-8ffa-e5d946f73afe: no known peers, aborting election
2025-02-23T18:41:44.796Z [WARN]  server-6ce296e8-ef68-7237-2d71-d9dde2a38dec: no known peers, aborting election
2025-02-23T18:41:44.809Z [WARN]  server-e2a4f563-0394-f1c2-9a8a-819c98840f1f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:44.809Z [INFO]  server-e2a4f563-0394-f1c2-9a8a-819c98840f1f: entering candidate state: node="Node at e2a4f563-0394-f1c2-9a8a-819c98840f1f [Candidate]" term=2
2025-02-23T18:41:44.809Z [INFO]  server-e2a4f563-0394-f1c2-9a8a-819c98840f1f: election won: term=2 tally=1
2025-02-23T18:41:44.809Z [INFO]  server-e2a4f563-0394-f1c2-9a8a-819c98840f1f: entering leader state: leader="Node at e2a4f563-0394-f1c2-9a8a-819c98840f1f [Leader]"
2025-02-23T18:41:44.914Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:44.809913678 +0000 UTC m=+9.108174372 (1 nodes), 76.192373ms from start of poll, 76.961376ms from cluster start. Timeout at 2025-02-23 18:41:44.914919559 +0000 UTC m=+9.213180296, 105.005924ms after stability
2025-02-23T18:41:44.915Z [INFO]  server-e2a4f563-0394-f1c2-9a8a-819c98840f1f: updating configuration: command=AddVoter server-id=server-33fb7852-0b4d-c480-8ffa-e5d946f73afe server-addr=33fb7852-0b4d-c480-8ffa-e5d946f73afe servers="[{Suffrage:Voter ID:server-e2a4f563-0394-f1c2-9a8a-819c98840f1f Address:e2a4f563-0394-f1c2-9a8a-819c98840f1f} {Suffrage:Voter ID:server-33fb7852-0b4d-c480-8ffa-e5d946f73afe Address:33fb7852-0b4d-c480-8ffa-e5d946f73afe}]"
2025-02-23T18:41:44.915Z [INFO]  server-e2a4f563-0394-f1c2-9a8a-819c98840f1f: added peer, starting replication: peer=server-33fb7852-0b4d-c480-8ffa-e5d946f73afe
2025-02-23T18:41:44.915Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:44.915Z [WARN]  server-33fb7852-0b4d-c480-8ffa-e5d946f73afe: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-02-23T18:41:44.915Z [WARN]  server-e2a4f563-0394-f1c2-9a8a-819c98840f1f: appendEntries rejected, sending older logs: peer="{Voter server-33fb7852-0b4d-c480-8ffa-e5d946f73afe 33fb7852-0b4d-c480-8ffa-e5d946f73afe}" next=1
2025-02-23T18:41:44.915Z [INFO]  server-e2a4f563-0394-f1c2-9a8a-819c98840f1f: pipelining replication: peer="{Voter server-33fb7852-0b4d-c480-8ffa-e5d946f73afe 33fb7852-0b4d-c480-8ffa-e5d946f73afe}"
2025-02-23T18:41:45.020Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:44.915319603 +0000 UTC m=+9.213580561 (1 nodes), 1.956µs from start of poll, 182.367565ms from cluster start. Timeout at 2025-02-23 18:41:45.020332101 +0000 UTC m=+9.318592231, 105.01167ms after stability
2025-02-23T18:41:45.020Z [INFO]  server-e2a4f563-0394-f1c2-9a8a-819c98840f1f: updating configuration: command=AddVoter server-id=server-6ce296e8-ef68-7237-2d71-d9dde2a38dec server-addr=6ce296e8-ef68-7237-2d71-d9dde2a38dec servers="[{Suffrage:Voter ID:server-e2a4f563-0394-f1c2-9a8a-819c98840f1f Address:e2a4f563-0394-f1c2-9a8a-819c98840f1f} {Suffrage:Voter ID:server-33fb7852-0b4d-c480-8ffa-e5d946f73afe Address:33fb7852-0b4d-c480-8ffa-e5d946f73afe} {Suffrage:Voter ID:server-6ce296e8-ef68-7237-2d71-d9dde2a38dec Address:6ce296e8-ef68-7237-2d71-d9dde2a38dec}]"
2025-02-23T18:41:45.020Z [INFO]  server-e2a4f563-0394-f1c2-9a8a-819c98840f1f: added peer, starting replication: peer=server-6ce296e8-ef68-7237-2d71-d9dde2a38dec
2025-02-23T18:41:45.020Z [WARN]  server-6ce296e8-ef68-7237-2d71-d9dde2a38dec: failed to get previous log: previous-index=4 last-index=0 error="log not found"
2025-02-23T18:41:45.020Z [WARN]  server-e2a4f563-0394-f1c2-9a8a-819c98840f1f: appendEntries rejected, sending older logs: peer="{Voter server-6ce296e8-ef68-7237-2d71-d9dde2a38dec 6ce296e8-ef68-7237-2d71-d9dde2a38dec}" next=1
2025-02-23T18:41:45.020Z [INFO]  server-e2a4f563-0394-f1c2-9a8a-819c98840f1f: pipelining replication: peer="{Voter server-6ce296e8-ef68-7237-2d71-d9dde2a38dec 6ce296e8-ef68-7237-2d71-d9dde2a38dec}"
2025-02-23T18:41:45.021Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:45.126Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:45.021163957 +0000 UTC m=+9.319514946 (1 nodes), 92.453µs from start of poll, 288.30195ms from cluster start. Timeout at 2025-02-23 18:41:45.126275211 +0000 UTC m=+9.424535768, 105.020822ms after stability
2025-02-23T18:41:45.126Z [INFO]  server-e2a4f563-0394-f1c2-9a8a-819c98840f1f: aborting pipeline replication: peer="{Voter server-6ce296e8-ef68-7237-2d71-d9dde2a38dec 6ce296e8-ef68-7237-2d71-d9dde2a38dec}"
2025-02-23T18:41:45.126Z [INFO]  server-e2a4f563-0394-f1c2-9a8a-819c98840f1f: aborting pipeline replication: peer="{Voter server-33fb7852-0b4d-c480-8ffa-e5d946f73afe 33fb7852-0b4d-c480-8ffa-e5d946f73afe}"
--- PASS: TestRaft_JoinNode_ConfigStore (0.40s)
=== RUN   TestRaft_RemoveFollower
2025-02-23T18:41:45.128Z [INFO]  server-9758470f-afbf-4e49-f873-a2d274645cd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9758470f-afbf-4e49-f873-a2d274645cd3 Address:9758470f-afbf-4e49-f873-a2d274645cd3} {Suffrage:Voter ID:server-d871ecae-c7ee-1fa7-87eb-4c97ec841084 Address:d871ecae-c7ee-1fa7-87eb-4c97ec841084} {Suffrage:Voter ID:server-059e6ad3-8829-80ad-b6c5-f0e801338502 Address:059e6ad3-8829-80ad-b6c5-f0e801338502}]"
2025-02-23T18:41:45.128Z [INFO]  server-9758470f-afbf-4e49-f873-a2d274645cd3: entering follower state: follower="Node at 9758470f-afbf-4e49-f873-a2d274645cd3 [Follower]" leader-address= leader-id=
2025-02-23T18:41:45.128Z [INFO]  server-d871ecae-c7ee-1fa7-87eb-4c97ec841084: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9758470f-afbf-4e49-f873-a2d274645cd3 Address:9758470f-afbf-4e49-f873-a2d274645cd3} {Suffrage:Voter ID:server-d871ecae-c7ee-1fa7-87eb-4c97ec841084 Address:d871ecae-c7ee-1fa7-87eb-4c97ec841084} {Suffrage:Voter ID:server-059e6ad3-8829-80ad-b6c5-f0e801338502 Address:059e6ad3-8829-80ad-b6c5-f0e801338502}]"
2025-02-23T18:41:45.128Z [INFO]  server-d871ecae-c7ee-1fa7-87eb-4c97ec841084: entering follower state: follower="Node at d871ecae-c7ee-1fa7-87eb-4c97ec841084 [Follower]" leader-address= leader-id=
2025-02-23T18:41:45.128Z [INFO]  server-059e6ad3-8829-80ad-b6c5-f0e801338502: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9758470f-afbf-4e49-f873-a2d274645cd3 Address:9758470f-afbf-4e49-f873-a2d274645cd3} {Suffrage:Voter ID:server-d871ecae-c7ee-1fa7-87eb-4c97ec841084 Address:d871ecae-c7ee-1fa7-87eb-4c97ec841084} {Suffrage:Voter ID:server-059e6ad3-8829-80ad-b6c5-f0e801338502 Address:059e6ad3-8829-80ad-b6c5-f0e801338502}]"
2025-02-23T18:41:45.129Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:45.129Z [INFO]  server-059e6ad3-8829-80ad-b6c5-f0e801338502: entering follower state: follower="Node at 059e6ad3-8829-80ad-b6c5-f0e801338502 [Follower]" leader-address= leader-id=
2025-02-23T18:41:45.190Z [WARN]  server-059e6ad3-8829-80ad-b6c5-f0e801338502: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:45.190Z [INFO]  server-059e6ad3-8829-80ad-b6c5-f0e801338502: entering candidate state: node="Node at 059e6ad3-8829-80ad-b6c5-f0e801338502 [Candidate]" term=2
2025-02-23T18:41:45.190Z [INFO]  server-059e6ad3-8829-80ad-b6c5-f0e801338502: election won: term=2 tally=2
2025-02-23T18:41:45.190Z [INFO]  server-059e6ad3-8829-80ad-b6c5-f0e801338502: entering leader state: leader="Node at 059e6ad3-8829-80ad-b6c5-f0e801338502 [Leader]"
2025-02-23T18:41:45.191Z [INFO]  server-059e6ad3-8829-80ad-b6c5-f0e801338502: added peer, starting replication: peer=server-9758470f-afbf-4e49-f873-a2d274645cd3
2025-02-23T18:41:45.191Z [INFO]  server-059e6ad3-8829-80ad-b6c5-f0e801338502: added peer, starting replication: peer=server-d871ecae-c7ee-1fa7-87eb-4c97ec841084
2025-02-23T18:41:45.191Z [INFO]  server-059e6ad3-8829-80ad-b6c5-f0e801338502: pipelining replication: peer="{Voter server-d871ecae-c7ee-1fa7-87eb-4c97ec841084 d871ecae-c7ee-1fa7-87eb-4c97ec841084}"
2025-02-23T18:41:45.191Z [INFO]  server-059e6ad3-8829-80ad-b6c5-f0e801338502: pipelining replication: peer="{Voter server-9758470f-afbf-4e49-f873-a2d274645cd3 9758470f-afbf-4e49-f873-a2d274645cd3}"
2025-02-23T18:41:45.297Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:45.191570432 +0000 UTC m=+9.489832023 (1 nodes), 62.53148ms from start of poll, 62.995571ms from cluster start. Timeout at 2025-02-23 18:41:45.296577585 +0000 UTC m=+9.594838258, 105.006235ms after stability
2025-02-23T18:41:45.303Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:41:45.408Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:41:45.303138189 +0000 UTC m=+9.601399172 (2 nodes), 2.386µs from start of poll, 174.56272ms from cluster start. Timeout at 2025-02-23 18:41:45.408149787 +0000 UTC m=+9.706410672, 105.0115ms after stability
2025-02-23T18:41:45.408Z [INFO]  server-059e6ad3-8829-80ad-b6c5-f0e801338502: updating configuration: command=RemoveServer server-id=server-9758470f-afbf-4e49-f873-a2d274645cd3 server-addr= servers="[{Suffrage:Voter ID:server-d871ecae-c7ee-1fa7-87eb-4c97ec841084 Address:d871ecae-c7ee-1fa7-87eb-4c97ec841084} {Suffrage:Voter ID:server-059e6ad3-8829-80ad-b6c5-f0e801338502 Address:059e6ad3-8829-80ad-b6c5-f0e801338502}]"
2025-02-23T18:41:45.408Z [INFO]  server-059e6ad3-8829-80ad-b6c5-f0e801338502: removed peer, stopping replication: peer=server-9758470f-afbf-4e49-f873-a2d274645cd3 last-index=3
2025-02-23T18:41:45.408Z [INFO]  server-059e6ad3-8829-80ad-b6c5-f0e801338502: aborting pipeline replication: peer="{Voter server-9758470f-afbf-4e49-f873-a2d274645cd3 9758470f-afbf-4e49-f873-a2d274645cd3}"
2025-02-23T18:41:45.466Z [WARN]  server-9758470f-afbf-4e49-f873-a2d274645cd3: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-02-23T18:41:45.514Z [INFO]  server-059e6ad3-8829-80ad-b6c5-f0e801338502: aborting pipeline replication: peer="{Voter server-d871ecae-c7ee-1fa7-87eb-4c97ec841084 d871ecae-c7ee-1fa7-87eb-4c97ec841084}"
--- PASS: TestRaft_RemoveFollower (0.39s)
=== RUN   TestRaft_RemoveLeader
2025-02-23T18:41:45.516Z [INFO]  server-ebe6fa62-d609-e7f5-5a1d-62ed74e328a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ebe6fa62-d609-e7f5-5a1d-62ed74e328a4 Address:ebe6fa62-d609-e7f5-5a1d-62ed74e328a4} {Suffrage:Voter ID:server-35fa3db2-ec8f-17eb-9621-a0d81230896b Address:35fa3db2-ec8f-17eb-9621-a0d81230896b} {Suffrage:Voter ID:server-d137e9d2-3492-ab35-9897-62cf72ebae82 Address:d137e9d2-3492-ab35-9897-62cf72ebae82}]"
2025-02-23T18:41:45.516Z [INFO]  server-ebe6fa62-d609-e7f5-5a1d-62ed74e328a4: entering follower state: follower="Node at ebe6fa62-d609-e7f5-5a1d-62ed74e328a4 [Follower]" leader-address= leader-id=
2025-02-23T18:41:45.516Z [INFO]  server-35fa3db2-ec8f-17eb-9621-a0d81230896b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ebe6fa62-d609-e7f5-5a1d-62ed74e328a4 Address:ebe6fa62-d609-e7f5-5a1d-62ed74e328a4} {Suffrage:Voter ID:server-35fa3db2-ec8f-17eb-9621-a0d81230896b Address:35fa3db2-ec8f-17eb-9621-a0d81230896b} {Suffrage:Voter ID:server-d137e9d2-3492-ab35-9897-62cf72ebae82 Address:d137e9d2-3492-ab35-9897-62cf72ebae82}]"
2025-02-23T18:41:45.516Z [INFO]  server-35fa3db2-ec8f-17eb-9621-a0d81230896b: entering follower state: follower="Node at 35fa3db2-ec8f-17eb-9621-a0d81230896b [Follower]" leader-address= leader-id=
2025-02-23T18:41:45.516Z [INFO]  server-d137e9d2-3492-ab35-9897-62cf72ebae82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ebe6fa62-d609-e7f5-5a1d-62ed74e328a4 Address:ebe6fa62-d609-e7f5-5a1d-62ed74e328a4} {Suffrage:Voter ID:server-35fa3db2-ec8f-17eb-9621-a0d81230896b Address:35fa3db2-ec8f-17eb-9621-a0d81230896b} {Suffrage:Voter ID:server-d137e9d2-3492-ab35-9897-62cf72ebae82 Address:d137e9d2-3492-ab35-9897-62cf72ebae82}]"
2025-02-23T18:41:45.516Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:45.516Z [INFO]  server-d137e9d2-3492-ab35-9897-62cf72ebae82: entering follower state: follower="Node at d137e9d2-3492-ab35-9897-62cf72ebae82 [Follower]" leader-address= leader-id=
2025-02-23T18:41:45.582Z [WARN]  server-d137e9d2-3492-ab35-9897-62cf72ebae82: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:45.583Z [INFO]  server-d137e9d2-3492-ab35-9897-62cf72ebae82: entering candidate state: node="Node at d137e9d2-3492-ab35-9897-62cf72ebae82 [Candidate]" term=2
2025-02-23T18:41:45.583Z [INFO]  server-d137e9d2-3492-ab35-9897-62cf72ebae82: election won: term=2 tally=2
2025-02-23T18:41:45.583Z [INFO]  server-d137e9d2-3492-ab35-9897-62cf72ebae82: entering leader state: leader="Node at d137e9d2-3492-ab35-9897-62cf72ebae82 [Leader]"
2025-02-23T18:41:45.583Z [INFO]  server-d137e9d2-3492-ab35-9897-62cf72ebae82: added peer, starting replication: peer=server-ebe6fa62-d609-e7f5-5a1d-62ed74e328a4
2025-02-23T18:41:45.583Z [INFO]  server-d137e9d2-3492-ab35-9897-62cf72ebae82: added peer, starting replication: peer=server-35fa3db2-ec8f-17eb-9621-a0d81230896b
2025-02-23T18:41:45.583Z [INFO]  server-d137e9d2-3492-ab35-9897-62cf72ebae82: pipelining replication: peer="{Voter server-35fa3db2-ec8f-17eb-9621-a0d81230896b 35fa3db2-ec8f-17eb-9621-a0d81230896b}"
2025-02-23T18:41:45.583Z [INFO]  server-d137e9d2-3492-ab35-9897-62cf72ebae82: pipelining replication: peer="{Voter server-ebe6fa62-d609-e7f5-5a1d-62ed74e328a4 ebe6fa62-d609-e7f5-5a1d-62ed74e328a4}"
2025-02-23T18:41:45.688Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:45.58328936 +0000 UTC m=+9.881549490 (1 nodes), 66.680175ms from start of poll, 67.136251ms from cluster start. Timeout at 2025-02-23 18:41:45.688298495 +0000 UTC m=+9.986559195, 105.009705ms after stability
2025-02-23T18:41:45.694Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:41:45.800Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:41:45.694257341 +0000 UTC m=+9.992517966 (2 nodes), 2.132µs from start of poll, 178.104727ms from cluster start. Timeout at 2025-02-23 18:41:45.799274445 +0000 UTC m=+10.097535088, 105.017122ms after stability
2025-02-23T18:41:45.800Z [INFO]  server-d137e9d2-3492-ab35-9897-62cf72ebae82: updating configuration: command=RemoveServer server-id=server-d137e9d2-3492-ab35-9897-62cf72ebae82 server-addr= servers="[{Suffrage:Voter ID:server-ebe6fa62-d609-e7f5-5a1d-62ed74e328a4 Address:ebe6fa62-d609-e7f5-5a1d-62ed74e328a4} {Suffrage:Voter ID:server-35fa3db2-ec8f-17eb-9621-a0d81230896b Address:35fa3db2-ec8f-17eb-9621-a0d81230896b}]"
2025-02-23T18:41:45.800Z [INFO]  server-d137e9d2-3492-ab35-9897-62cf72ebae82: removed ourself, shutting down
2025-02-23T18:41:45.800Z [INFO]  server-d137e9d2-3492-ab35-9897-62cf72ebae82: aborting pipeline replication: peer="{Voter server-35fa3db2-ec8f-17eb-9621-a0d81230896b 35fa3db2-ec8f-17eb-9621-a0d81230896b}"
2025-02-23T18:41:45.800Z [INFO]  server-d137e9d2-3492-ab35-9897-62cf72ebae82: aborting pipeline replication: peer="{Voter server-ebe6fa62-d609-e7f5-5a1d-62ed74e328a4 ebe6fa62-d609-e7f5-5a1d-62ed74e328a4}"
2025-02-23T18:41:45.852Z [WARN]  server-35fa3db2-ec8f-17eb-9621-a0d81230896b: heartbeat timeout reached, starting election: last-leader-addr=d137e9d2-3492-ab35-9897-62cf72ebae82 last-leader-id=server-d137e9d2-3492-ab35-9897-62cf72ebae82
2025-02-23T18:41:45.852Z [INFO]  server-35fa3db2-ec8f-17eb-9621-a0d81230896b: entering candidate state: node="Node at 35fa3db2-ec8f-17eb-9621-a0d81230896b [Candidate]" term=3
2025-02-23T18:41:45.853Z [WARN]  server-ebe6fa62-d609-e7f5-5a1d-62ed74e328a4: rejecting vote request since we have a leader: from=35fa3db2-ec8f-17eb-9621-a0d81230896b leader=d137e9d2-3492-ab35-9897-62cf72ebae82 leader-id=server-d137e9d2-3492-ab35-9897-62cf72ebae82
2025-02-23T18:41:45.907Z [WARN]  server-ebe6fa62-d609-e7f5-5a1d-62ed74e328a4: heartbeat timeout reached, starting election: last-leader-addr=d137e9d2-3492-ab35-9897-62cf72ebae82 last-leader-id=server-d137e9d2-3492-ab35-9897-62cf72ebae82
2025-02-23T18:41:45.907Z [INFO]  server-ebe6fa62-d609-e7f5-5a1d-62ed74e328a4: entering candidate state: node="Node at ebe6fa62-d609-e7f5-5a1d-62ed74e328a4 [Candidate]" term=3
2025-02-23T18:41:45.907Z [INFO]  server-35fa3db2-ec8f-17eb-9621-a0d81230896b: duplicate requestVote for same term: term=3
2025-02-23T18:41:45.921Z [WARN]  server-35fa3db2-ec8f-17eb-9621-a0d81230896b: Election timeout reached, restarting election
2025-02-23T18:41:45.921Z [INFO]  server-35fa3db2-ec8f-17eb-9621-a0d81230896b: entering candidate state: node="Node at 35fa3db2-ec8f-17eb-9621-a0d81230896b [Candidate]" term=4
2025-02-23T18:41:45.921Z [INFO]  server-ebe6fa62-d609-e7f5-5a1d-62ed74e328a4: entering follower state: follower="Node at ebe6fa62-d609-e7f5-5a1d-62ed74e328a4 [Follower]" leader-address= leader-id=
2025-02-23T18:41:45.921Z [INFO]  server-35fa3db2-ec8f-17eb-9621-a0d81230896b: election won: term=4 tally=2
2025-02-23T18:41:45.921Z [INFO]  server-35fa3db2-ec8f-17eb-9621-a0d81230896b: entering leader state: leader="Node at 35fa3db2-ec8f-17eb-9621-a0d81230896b [Leader]"
2025-02-23T18:41:45.922Z [INFO]  server-35fa3db2-ec8f-17eb-9621-a0d81230896b: added peer, starting replication: peer=server-ebe6fa62-d609-e7f5-5a1d-62ed74e328a4
2025-02-23T18:41:45.922Z [INFO]  server-35fa3db2-ec8f-17eb-9621-a0d81230896b: pipelining replication: peer="{Voter server-ebe6fa62-d609-e7f5-5a1d-62ed74e328a4 ebe6fa62-d609-e7f5-5a1d-62ed74e328a4}"
2025-02-23T18:41:46.012Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:46.118Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:46.012310532 +0000 UTC m=+10.310570662 (1 nodes), 181.041µs from start of poll, 496.157423ms from cluster start. Timeout at 2025-02-23 18:41:46.11731861 +0000 UTC m=+10.415579229, 105.008567ms after stability
2025-02-23T18:41:46.118Z [INFO]  server-35fa3db2-ec8f-17eb-9621-a0d81230896b: aborting pipeline replication: peer="{Voter server-ebe6fa62-d609-e7f5-5a1d-62ed74e328a4 ebe6fa62-d609-e7f5-5a1d-62ed74e328a4}"
--- PASS: TestRaft_RemoveLeader (0.61s)
=== RUN   TestRaft_RemoveLeader_NoShutdown
2025-02-23T18:41:46.122Z [INFO]  server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb Address:35e7f9c8-731d-0c75-2bc3-846bc7fdacbb} {Suffrage:Voter ID:server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba Address:bd2b70f6-7e24-b37a-9c93-e4f27c5965ba} {Suffrage:Voter ID:server-b8bed032-1d63-b3c0-e8b6-29395154c6e4 Address:b8bed032-1d63-b3c0-e8b6-29395154c6e4}]"
2025-02-23T18:41:46.122Z [INFO]  server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb: entering follower state: follower="Node at 35e7f9c8-731d-0c75-2bc3-846bc7fdacbb [Follower]" leader-address= leader-id=
2025-02-23T18:41:46.122Z [INFO]  server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb Address:35e7f9c8-731d-0c75-2bc3-846bc7fdacbb} {Suffrage:Voter ID:server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba Address:bd2b70f6-7e24-b37a-9c93-e4f27c5965ba} {Suffrage:Voter ID:server-b8bed032-1d63-b3c0-e8b6-29395154c6e4 Address:b8bed032-1d63-b3c0-e8b6-29395154c6e4}]"
2025-02-23T18:41:46.123Z [INFO]  server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba: entering follower state: follower="Node at bd2b70f6-7e24-b37a-9c93-e4f27c5965ba [Follower]" leader-address= leader-id=
2025-02-23T18:41:46.123Z [INFO]  server-b8bed032-1d63-b3c0-e8b6-29395154c6e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb Address:35e7f9c8-731d-0c75-2bc3-846bc7fdacbb} {Suffrage:Voter ID:server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba Address:bd2b70f6-7e24-b37a-9c93-e4f27c5965ba} {Suffrage:Voter ID:server-b8bed032-1d63-b3c0-e8b6-29395154c6e4 Address:b8bed032-1d63-b3c0-e8b6-29395154c6e4}]"
2025-02-23T18:41:46.123Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:41:46.123Z [INFO]  server-b8bed032-1d63-b3c0-e8b6-29395154c6e4: entering follower state: follower="Node at b8bed032-1d63-b3c0-e8b6-29395154c6e4 [Follower]" leader-address= leader-id=
2025-02-23T18:41:46.174Z [WARN]  server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:46.174Z [INFO]  server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba: entering candidate state: node="Node at bd2b70f6-7e24-b37a-9c93-e4f27c5965ba [Candidate]" term=2
2025-02-23T18:41:46.174Z [INFO]  server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba: election won: term=2 tally=2
2025-02-23T18:41:46.174Z [INFO]  server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba: entering leader state: leader="Node at bd2b70f6-7e24-b37a-9c93-e4f27c5965ba [Leader]"
2025-02-23T18:41:46.174Z [INFO]  server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba: added peer, starting replication: peer=server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb
2025-02-23T18:41:46.174Z [INFO]  server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba: added peer, starting replication: peer=server-b8bed032-1d63-b3c0-e8b6-29395154c6e4
2025-02-23T18:41:46.174Z [INFO]  server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba: pipelining replication: peer="{Voter server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb 35e7f9c8-731d-0c75-2bc3-846bc7fdacbb}"
2025-02-23T18:41:46.174Z [INFO]  server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba: pipelining replication: peer="{Voter server-b8bed032-1d63-b3c0-e8b6-29395154c6e4 b8bed032-1d63-b3c0-e8b6-29395154c6e4}"
2025-02-23T18:41:46.279Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:41:46.17458745 +0000 UTC m=+10.472848175 (2 nodes), 51.426709ms from start of poll, 52.002739ms from cluster start. Timeout at 2025-02-23 18:41:46.279593253 +0000 UTC m=+10.577853913, 105.005738ms after stability
2025-02-23T18:41:46.280Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:46.386Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:46.280077051 +0000 UTC m=+10.578337181 (1 nodes), 0s from start of poll, 157.491745ms from cluster start. Timeout at 2025-02-23 18:41:46.385091506 +0000 UTC m=+10.683352158, 105.014977ms after stability
2025-02-23T18:41:46.395Z [INFO]  server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba: updating configuration: command=RemoveServer server-id=server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba server-addr= servers="[{Suffrage:Voter ID:server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb Address:35e7f9c8-731d-0c75-2bc3-846bc7fdacbb} {Suffrage:Voter ID:server-b8bed032-1d63-b3c0-e8b6-29395154c6e4 Address:b8bed032-1d63-b3c0-e8b6-29395154c6e4}]"
2025-02-23T18:41:46.397Z [INFO]  server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba: removed ourself, transitioning to follower
2025-02-23T18:41:46.397Z [INFO]  server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba: entering follower state: follower="Node at bd2b70f6-7e24-b37a-9c93-e4f27c5965ba [Follower]" leader-address= leader-id=
2025-02-23T18:41:46.397Z [INFO]  server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba: aborting pipeline replication: peer="{Voter server-b8bed032-1d63-b3c0-e8b6-29395154c6e4 b8bed032-1d63-b3c0-e8b6-29395154c6e4}"
2025-02-23T18:41:46.397Z [INFO]  server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba: aborting pipeline replication: peer="{Voter server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb 35e7f9c8-731d-0c75-2bc3-846bc7fdacbb}"
2025-02-23T18:41:46.449Z [WARN]  server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba: not part of stable configuration, aborting election
2025-02-23T18:41:46.476Z [WARN]  server-b8bed032-1d63-b3c0-e8b6-29395154c6e4: heartbeat timeout reached, starting election: last-leader-addr=bd2b70f6-7e24-b37a-9c93-e4f27c5965ba last-leader-id=server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba
2025-02-23T18:41:46.476Z [INFO]  server-b8bed032-1d63-b3c0-e8b6-29395154c6e4: entering candidate state: node="Node at b8bed032-1d63-b3c0-e8b6-29395154c6e4 [Candidate]" term=3
2025-02-23T18:41:46.476Z [WARN]  server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb: heartbeat timeout reached, starting election: last-leader-addr=bd2b70f6-7e24-b37a-9c93-e4f27c5965ba last-leader-id=server-bd2b70f6-7e24-b37a-9c93-e4f27c5965ba
2025-02-23T18:41:46.476Z [INFO]  server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb: entering candidate state: node="Node at 35e7f9c8-731d-0c75-2bc3-846bc7fdacbb [Candidate]" term=3
2025-02-23T18:41:46.476Z [INFO]  server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb: duplicate requestVote for same term: term=3
2025-02-23T18:41:46.476Z [INFO]  server-b8bed032-1d63-b3c0-e8b6-29395154c6e4: duplicate requestVote for same term: term=3
2025-02-23T18:41:46.502Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:46.541Z [WARN]  server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb: Election timeout reached, restarting election
2025-02-23T18:41:46.541Z [INFO]  server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb: entering candidate state: node="Node at 35e7f9c8-731d-0c75-2bc3-846bc7fdacbb [Candidate]" term=4
2025-02-23T18:41:46.541Z [INFO]  server-b8bed032-1d63-b3c0-e8b6-29395154c6e4: entering follower state: follower="Node at b8bed032-1d63-b3c0-e8b6-29395154c6e4 [Follower]" leader-address= leader-id=
2025-02-23T18:41:46.541Z [INFO]  server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb: election won: term=4 tally=2
2025-02-23T18:41:46.541Z [INFO]  server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb: entering leader state: leader="Node at 35e7f9c8-731d-0c75-2bc3-846bc7fdacbb [Leader]"
2025-02-23T18:41:46.541Z [INFO]  server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb: added peer, starting replication: peer=server-b8bed032-1d63-b3c0-e8b6-29395154c6e4
2025-02-23T18:41:46.541Z [INFO]  server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb: pipelining replication: peer="{Voter server-b8bed032-1d63-b3c0-e8b6-29395154c6e4 b8bed032-1d63-b3c0-e8b6-29395154c6e4}"
2025-02-23T18:41:46.647Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:46.541508429 +0000 UTC m=+10.839769387 (1 nodes), 38.490459ms from start of poll, 418.923951ms from cluster start. Timeout at 2025-02-23 18:41:46.646515028 +0000 UTC m=+10.944775754, 105.006367ms after stability
2025-02-23T18:41:46.752Z [INFO]  server-35e7f9c8-731d-0c75-2bc3-846bc7fdacbb: aborting pipeline replication: peer="{Voter server-b8bed032-1d63-b3c0-e8b6-29395154c6e4 b8bed032-1d63-b3c0-e8b6-29395154c6e4}"
--- PASS: TestRaft_RemoveLeader_NoShutdown (0.63s)
=== RUN   TestRaft_RemoveFollower_SplitCluster
2025-02-23T18:41:46.754Z [INFO]  server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb Address:d736d378-6cc4-4fdd-730a-6c52d9c86cdb} {Suffrage:Voter ID:server-6048ccc5-0cda-122e-22f1-4012d0fbceec Address:6048ccc5-0cda-122e-22f1-4012d0fbceec} {Suffrage:Voter ID:server-395898f0-ba48-a5e9-1619-8ef908966f79 Address:395898f0-ba48-a5e9-1619-8ef908966f79} {Suffrage:Voter ID:server-d8222eac-7591-fa09-814b-9234bc3cb55b Address:d8222eac-7591-fa09-814b-9234bc3cb55b}]"
2025-02-23T18:41:46.754Z [INFO]  server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb: entering follower state: follower="Node at d736d378-6cc4-4fdd-730a-6c52d9c86cdb [Follower]" leader-address= leader-id=
2025-02-23T18:41:46.754Z [INFO]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb Address:d736d378-6cc4-4fdd-730a-6c52d9c86cdb} {Suffrage:Voter ID:server-6048ccc5-0cda-122e-22f1-4012d0fbceec Address:6048ccc5-0cda-122e-22f1-4012d0fbceec} {Suffrage:Voter ID:server-395898f0-ba48-a5e9-1619-8ef908966f79 Address:395898f0-ba48-a5e9-1619-8ef908966f79} {Suffrage:Voter ID:server-d8222eac-7591-fa09-814b-9234bc3cb55b Address:d8222eac-7591-fa09-814b-9234bc3cb55b}]"
2025-02-23T18:41:46.754Z [INFO]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: entering follower state: follower="Node at 6048ccc5-0cda-122e-22f1-4012d0fbceec [Follower]" leader-address= leader-id=
2025-02-23T18:41:46.755Z [INFO]  server-395898f0-ba48-a5e9-1619-8ef908966f79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb Address:d736d378-6cc4-4fdd-730a-6c52d9c86cdb} {Suffrage:Voter ID:server-6048ccc5-0cda-122e-22f1-4012d0fbceec Address:6048ccc5-0cda-122e-22f1-4012d0fbceec} {Suffrage:Voter ID:server-395898f0-ba48-a5e9-1619-8ef908966f79 Address:395898f0-ba48-a5e9-1619-8ef908966f79} {Suffrage:Voter ID:server-d8222eac-7591-fa09-814b-9234bc3cb55b Address:d8222eac-7591-fa09-814b-9234bc3cb55b}]"
2025-02-23T18:41:46.755Z [INFO]  server-395898f0-ba48-a5e9-1619-8ef908966f79: entering follower state: follower="Node at 395898f0-ba48-a5e9-1619-8ef908966f79 [Follower]" leader-address= leader-id=
2025-02-23T18:41:46.755Z [INFO]  server-d8222eac-7591-fa09-814b-9234bc3cb55b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb Address:d736d378-6cc4-4fdd-730a-6c52d9c86cdb} {Suffrage:Voter ID:server-6048ccc5-0cda-122e-22f1-4012d0fbceec Address:6048ccc5-0cda-122e-22f1-4012d0fbceec} {Suffrage:Voter ID:server-395898f0-ba48-a5e9-1619-8ef908966f79 Address:395898f0-ba48-a5e9-1619-8ef908966f79} {Suffrage:Voter ID:server-d8222eac-7591-fa09-814b-9234bc3cb55b Address:d8222eac-7591-fa09-814b-9234bc3cb55b}]"
2025-02-23T18:41:46.755Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:46.755Z [INFO]  server-d8222eac-7591-fa09-814b-9234bc3cb55b: entering follower state: follower="Node at d8222eac-7591-fa09-814b-9234bc3cb55b [Follower]" leader-address= leader-id=
2025-02-23T18:41:46.816Z [WARN]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:46.816Z [INFO]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: entering candidate state: node="Node at 6048ccc5-0cda-122e-22f1-4012d0fbceec [Candidate]" term=2
2025-02-23T18:41:46.816Z [INFO]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: election won: term=2 tally=3
2025-02-23T18:41:46.816Z [INFO]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: entering leader state: leader="Node at 6048ccc5-0cda-122e-22f1-4012d0fbceec [Leader]"
2025-02-23T18:41:46.816Z [INFO]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: added peer, starting replication: peer=server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb
2025-02-23T18:41:46.816Z [INFO]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: added peer, starting replication: peer=server-395898f0-ba48-a5e9-1619-8ef908966f79
2025-02-23T18:41:46.816Z [INFO]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: added peer, starting replication: peer=server-d8222eac-7591-fa09-814b-9234bc3cb55b
2025-02-23T18:41:46.816Z [INFO]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: pipelining replication: peer="{Voter server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb d736d378-6cc4-4fdd-730a-6c52d9c86cdb}"
2025-02-23T18:41:46.816Z [INFO]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: pipelining replication: peer="{Voter server-d8222eac-7591-fa09-814b-9234bc3cb55b d8222eac-7591-fa09-814b-9234bc3cb55b}"
2025-02-23T18:41:46.816Z [INFO]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: pipelining replication: peer="{Voter server-395898f0-ba48-a5e9-1619-8ef908966f79 395898f0-ba48-a5e9-1619-8ef908966f79}"
2025-02-23T18:41:46.922Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:46.816669954 +0000 UTC m=+11.114930752 (1 nodes), 61.36204ms from start of poll, 62.123283ms from cluster start. Timeout at 2025-02-23 18:41:46.921675893 +0000 UTC m=+11.219936588, 105.005836ms after stability
2025-02-23T18:41:47.028Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:41:47.134Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:41:47.028924829 +0000 UTC m=+11.327185468 (3 nodes), 2.246µs from start of poll, 274.377999ms from cluster start. Timeout at 2025-02-23 18:41:47.133935168 +0000 UTC m=+11.432195813, 105.010345ms after stability
2025-02-23T18:41:47.134Z [INFO]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: updating configuration: command=RemoveServer server-id=server-d8222eac-7591-fa09-814b-9234bc3cb55b server-addr= servers="[{Suffrage:Voter ID:server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb Address:d736d378-6cc4-4fdd-730a-6c52d9c86cdb} {Suffrage:Voter ID:server-6048ccc5-0cda-122e-22f1-4012d0fbceec Address:6048ccc5-0cda-122e-22f1-4012d0fbceec} {Suffrage:Voter ID:server-395898f0-ba48-a5e9-1619-8ef908966f79 Address:395898f0-ba48-a5e9-1619-8ef908966f79}]"
2025-02-23T18:41:47.134Z [INFO]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: removed peer, stopping replication: peer=server-d8222eac-7591-fa09-814b-9234bc3cb55b last-index=3
2025-02-23T18:41:47.134Z [ERROR] server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to pipeline appendEntries: peer="{Voter server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb d736d378-6cc4-4fdd-730a-6c52d9c86cdb}" error="append pipeline closed"
2025-02-23T18:41:47.134Z [INFO]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: aborting pipeline replication: peer="{Voter server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb d736d378-6cc4-4fdd-730a-6c52d9c86cdb}"
2025-02-23T18:41:47.134Z [ERROR] server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to pipeline appendEntries: peer="{Voter server-395898f0-ba48-a5e9-1619-8ef908966f79 395898f0-ba48-a5e9-1619-8ef908966f79}" error="append pipeline closed"
2025-02-23T18:41:47.134Z [INFO]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: aborting pipeline replication: peer="{Voter server-395898f0-ba48-a5e9-1619-8ef908966f79 395898f0-ba48-a5e9-1619-8ef908966f79}"
2025-02-23T18:41:47.134Z [INFO]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: aborting pipeline replication: peer="{Voter server-d8222eac-7591-fa09-814b-9234bc3cb55b d8222eac-7591-fa09-814b-9234bc3cb55b}"
2025-02-23T18:41:47.138Z [ERROR] server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to heartbeat to: peer=395898f0-ba48-a5e9-1619-8ef908966f79 backoff time=10ms error="failed to connect to peer: 395898f0-ba48-a5e9-1619-8ef908966f79"
2025-02-23T18:41:47.140Z [ERROR] server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to heartbeat to: peer=d736d378-6cc4-4fdd-730a-6c52d9c86cdb backoff time=10ms error="failed to connect to peer: d736d378-6cc4-4fdd-730a-6c52d9c86cdb"
2025-02-23T18:41:47.143Z [ERROR] server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to appendEntries to: peer="{Voter server-395898f0-ba48-a5e9-1619-8ef908966f79 395898f0-ba48-a5e9-1619-8ef908966f79}" error="failed to connect to peer: 395898f0-ba48-a5e9-1619-8ef908966f79"
2025-02-23T18:41:47.144Z [ERROR] server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to appendEntries to: peer="{Voter server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb d736d378-6cc4-4fdd-730a-6c52d9c86cdb}" error="failed to connect to peer: d736d378-6cc4-4fdd-730a-6c52d9c86cdb"
2025-02-23T18:41:47.156Z [ERROR] server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to heartbeat to: peer=395898f0-ba48-a5e9-1619-8ef908966f79 backoff time=10ms error="failed to connect to peer: 395898f0-ba48-a5e9-1619-8ef908966f79"
2025-02-23T18:41:47.157Z [ERROR] server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to heartbeat to: peer=d736d378-6cc4-4fdd-730a-6c52d9c86cdb backoff time=10ms error="failed to connect to peer: d736d378-6cc4-4fdd-730a-6c52d9c86cdb"
2025-02-23T18:41:47.162Z [ERROR] server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to appendEntries to: peer="{Voter server-395898f0-ba48-a5e9-1619-8ef908966f79 395898f0-ba48-a5e9-1619-8ef908966f79}" error="failed to connect to peer: 395898f0-ba48-a5e9-1619-8ef908966f79"
2025-02-23T18:41:47.164Z [ERROR] server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to appendEntries to: peer="{Voter server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb d736d378-6cc4-4fdd-730a-6c52d9c86cdb}" error="failed to connect to peer: d736d378-6cc4-4fdd-730a-6c52d9c86cdb"
2025-02-23T18:41:47.177Z [ERROR] server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to heartbeat to: peer=395898f0-ba48-a5e9-1619-8ef908966f79 backoff time=10ms error="failed to connect to peer: 395898f0-ba48-a5e9-1619-8ef908966f79"
2025-02-23T18:41:47.179Z [ERROR] server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to heartbeat to: peer=d736d378-6cc4-4fdd-730a-6c52d9c86cdb backoff time=10ms error="failed to connect to peer: d736d378-6cc4-4fdd-730a-6c52d9c86cdb"
2025-02-23T18:41:47.182Z [WARN]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to contact: server-id=server-395898f0-ba48-a5e9-1619-8ef908966f79 time=50.424087ms
2025-02-23T18:41:47.182Z [ERROR] server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to appendEntries to: peer="{Voter server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb d736d378-6cc4-4fdd-730a-6c52d9c86cdb}" error="failed to connect to peer: d736d378-6cc4-4fdd-730a-6c52d9c86cdb"
2025-02-23T18:41:47.182Z [ERROR] server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to appendEntries to: peer="{Voter server-395898f0-ba48-a5e9-1619-8ef908966f79 395898f0-ba48-a5e9-1619-8ef908966f79}" error="failed to connect to peer: 395898f0-ba48-a5e9-1619-8ef908966f79"
2025-02-23T18:41:47.192Z [WARN]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to contact: server-id=server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb time=58.173918ms
2025-02-23T18:41:47.192Z [WARN]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to contact: server-id=server-395898f0-ba48-a5e9-1619-8ef908966f79 time=60.540713ms
2025-02-23T18:41:47.192Z [WARN]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to contact quorum of nodes, stepping down
2025-02-23T18:41:47.192Z [WARN]  server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb: heartbeat timeout reached, starting election: last-leader-addr=6048ccc5-0cda-122e-22f1-4012d0fbceec last-leader-id=server-6048ccc5-0cda-122e-22f1-4012d0fbceec
2025-02-23T18:41:47.192Z [INFO]  server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb: entering candidate state: node="Node at d736d378-6cc4-4fdd-730a-6c52d9c86cdb [Candidate]" term=3
2025-02-23T18:41:47.193Z [INFO]  server-6048ccc5-0cda-122e-22f1-4012d0fbceec: entering follower state: follower="Node at 6048ccc5-0cda-122e-22f1-4012d0fbceec [Follower]" leader-address= leader-id=
2025-02-23T18:41:47.193Z [ERROR] server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb: failed to make requestVote RPC: target="{Voter server-d8222eac-7591-fa09-814b-9234bc3cb55b d8222eac-7591-fa09-814b-9234bc3cb55b}" error="failed to connect to peer: d8222eac-7591-fa09-814b-9234bc3cb55b" term=3
2025-02-23T18:41:47.193Z [ERROR] server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to appendEntries to: peer="{Voter server-395898f0-ba48-a5e9-1619-8ef908966f79 395898f0-ba48-a5e9-1619-8ef908966f79}" error="failed to connect to peer: 395898f0-ba48-a5e9-1619-8ef908966f79"
2025-02-23T18:41:47.193Z [ERROR] server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb: failed to make requestVote RPC: target="{Voter server-6048ccc5-0cda-122e-22f1-4012d0fbceec 6048ccc5-0cda-122e-22f1-4012d0fbceec}" error="failed to connect to peer: 6048ccc5-0cda-122e-22f1-4012d0fbceec" term=3
2025-02-23T18:41:47.193Z [ERROR] server-6048ccc5-0cda-122e-22f1-4012d0fbceec: failed to appendEntries to: peer="{Voter server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb d736d378-6cc4-4fdd-730a-6c52d9c86cdb}" error="failed to connect to peer: d736d378-6cc4-4fdd-730a-6c52d9c86cdb"
2025-02-23T18:41:47.693Z [ERROR] server-d736d378-6cc4-4fdd-730a-6c52d9c86cdb: failed to make requestVote RPC: target="{Voter server-395898f0-ba48-a5e9-1619-8ef908966f79 395898f0-ba48-a5e9-1619-8ef908966f79}" error="command timed out" term=3
--- PASS: TestRaft_RemoveFollower_SplitCluster (0.94s)
=== RUN   TestRaft_AddKnownPeer
2025-02-23T18:41:47.695Z [INFO]  server-4bcd08b2-2f91-6c02-602d-39f62939b33d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4bcd08b2-2f91-6c02-602d-39f62939b33d Address:4bcd08b2-2f91-6c02-602d-39f62939b33d} {Suffrage:Voter ID:server-5a5a5137-23fa-aebb-34d3-46152f6373ff Address:5a5a5137-23fa-aebb-34d3-46152f6373ff} {Suffrage:Voter ID:server-cb407929-68ad-8ae6-5c97-d00c44c9f8f9 Address:cb407929-68ad-8ae6-5c97-d00c44c9f8f9}]"
2025-02-23T18:41:47.695Z [INFO]  server-5a5a5137-23fa-aebb-34d3-46152f6373ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4bcd08b2-2f91-6c02-602d-39f62939b33d Address:4bcd08b2-2f91-6c02-602d-39f62939b33d} {Suffrage:Voter ID:server-5a5a5137-23fa-aebb-34d3-46152f6373ff Address:5a5a5137-23fa-aebb-34d3-46152f6373ff} {Suffrage:Voter ID:server-cb407929-68ad-8ae6-5c97-d00c44c9f8f9 Address:cb407929-68ad-8ae6-5c97-d00c44c9f8f9}]"
2025-02-23T18:41:47.695Z [INFO]  server-5a5a5137-23fa-aebb-34d3-46152f6373ff: entering follower state: follower="Node at 5a5a5137-23fa-aebb-34d3-46152f6373ff [Follower]" leader-address= leader-id=
2025-02-23T18:41:47.695Z [INFO]  server-cb407929-68ad-8ae6-5c97-d00c44c9f8f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4bcd08b2-2f91-6c02-602d-39f62939b33d Address:4bcd08b2-2f91-6c02-602d-39f62939b33d} {Suffrage:Voter ID:server-5a5a5137-23fa-aebb-34d3-46152f6373ff Address:5a5a5137-23fa-aebb-34d3-46152f6373ff} {Suffrage:Voter ID:server-cb407929-68ad-8ae6-5c97-d00c44c9f8f9 Address:cb407929-68ad-8ae6-5c97-d00c44c9f8f9}]"
2025-02-23T18:41:47.695Z [INFO]  server-4bcd08b2-2f91-6c02-602d-39f62939b33d: entering follower state: follower="Node at 4bcd08b2-2f91-6c02-602d-39f62939b33d [Follower]" leader-address= leader-id=
2025-02-23T18:41:47.695Z [INFO]  server-cb407929-68ad-8ae6-5c97-d00c44c9f8f9: entering follower state: follower="Node at cb407929-68ad-8ae6-5c97-d00c44c9f8f9 [Follower]" leader-address= leader-id=
2025-02-23T18:41:47.695Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:47.747Z [WARN]  server-5a5a5137-23fa-aebb-34d3-46152f6373ff: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:47.747Z [INFO]  server-5a5a5137-23fa-aebb-34d3-46152f6373ff: entering candidate state: node="Node at 5a5a5137-23fa-aebb-34d3-46152f6373ff [Candidate]" term=2
2025-02-23T18:41:47.747Z [INFO]  server-5a5a5137-23fa-aebb-34d3-46152f6373ff: election won: term=2 tally=2
2025-02-23T18:41:47.747Z [INFO]  server-5a5a5137-23fa-aebb-34d3-46152f6373ff: entering leader state: leader="Node at 5a5a5137-23fa-aebb-34d3-46152f6373ff [Leader]"
2025-02-23T18:41:47.747Z [INFO]  server-5a5a5137-23fa-aebb-34d3-46152f6373ff: added peer, starting replication: peer=server-4bcd08b2-2f91-6c02-602d-39f62939b33d
2025-02-23T18:41:47.747Z [INFO]  server-5a5a5137-23fa-aebb-34d3-46152f6373ff: added peer, starting replication: peer=server-cb407929-68ad-8ae6-5c97-d00c44c9f8f9
2025-02-23T18:41:47.747Z [INFO]  server-5a5a5137-23fa-aebb-34d3-46152f6373ff: pipelining replication: peer="{Voter server-4bcd08b2-2f91-6c02-602d-39f62939b33d 4bcd08b2-2f91-6c02-602d-39f62939b33d}"
2025-02-23T18:41:47.747Z [INFO]  server-5a5a5137-23fa-aebb-34d3-46152f6373ff: pipelining replication: peer="{Voter server-cb407929-68ad-8ae6-5c97-d00c44c9f8f9 cb407929-68ad-8ae6-5c97-d00c44c9f8f9}"
2025-02-23T18:41:47.852Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:47.747268476 +0000 UTC m=+12.045529291 (1 nodes), 51.464609ms from start of poll, 52.013205ms from cluster start. Timeout at 2025-02-23 18:41:47.852274642 +0000 UTC m=+12.150535319, 105.006028ms after stability
2025-02-23T18:41:47.853Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:41:47.958Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:41:47.853026089 +0000 UTC m=+12.151286728 (2 nodes), 2.188µs from start of poll, 157.770642ms from cluster start. Timeout at 2025-02-23 18:41:47.958035904 +0000 UTC m=+12.256296552, 105.009824ms after stability
2025-02-23T18:41:47.958Z [INFO]  server-5a5a5137-23fa-aebb-34d3-46152f6373ff: updating configuration: command=AddVoter server-id=server-4bcd08b2-2f91-6c02-602d-39f62939b33d server-addr=4bcd08b2-2f91-6c02-602d-39f62939b33d servers="[{Suffrage:Voter ID:server-4bcd08b2-2f91-6c02-602d-39f62939b33d Address:4bcd08b2-2f91-6c02-602d-39f62939b33d} {Suffrage:Voter ID:server-5a5a5137-23fa-aebb-34d3-46152f6373ff Address:5a5a5137-23fa-aebb-34d3-46152f6373ff} {Suffrage:Voter ID:server-cb407929-68ad-8ae6-5c97-d00c44c9f8f9 Address:cb407929-68ad-8ae6-5c97-d00c44c9f8f9}]"
2025-02-23T18:41:47.959Z [INFO]  server-5a5a5137-23fa-aebb-34d3-46152f6373ff: aborting pipeline replication: peer="{Voter server-cb407929-68ad-8ae6-5c97-d00c44c9f8f9 cb407929-68ad-8ae6-5c97-d00c44c9f8f9}"
2025-02-23T18:41:47.959Z [INFO]  server-5a5a5137-23fa-aebb-34d3-46152f6373ff: aborting pipeline replication: peer="{Voter server-4bcd08b2-2f91-6c02-602d-39f62939b33d 4bcd08b2-2f91-6c02-602d-39f62939b33d}"
--- PASS: TestRaft_AddKnownPeer (0.27s)
=== RUN   TestRaft_RemoveUnknownPeer
2025-02-23T18:41:47.961Z [INFO]  server-11a54d02-247a-5998-d097-5eeb37d1746b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11a54d02-247a-5998-d097-5eeb37d1746b Address:11a54d02-247a-5998-d097-5eeb37d1746b} {Suffrage:Voter ID:server-70d9e902-ee98-63f4-f28f-676112943f15 Address:70d9e902-ee98-63f4-f28f-676112943f15} {Suffrage:Voter ID:server-59d18cf9-bce4-1006-9fea-ea6f0b339f80 Address:59d18cf9-bce4-1006-9fea-ea6f0b339f80}]"
2025-02-23T18:41:47.961Z [INFO]  server-70d9e902-ee98-63f4-f28f-676112943f15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11a54d02-247a-5998-d097-5eeb37d1746b Address:11a54d02-247a-5998-d097-5eeb37d1746b} {Suffrage:Voter ID:server-70d9e902-ee98-63f4-f28f-676112943f15 Address:70d9e902-ee98-63f4-f28f-676112943f15} {Suffrage:Voter ID:server-59d18cf9-bce4-1006-9fea-ea6f0b339f80 Address:59d18cf9-bce4-1006-9fea-ea6f0b339f80}]"
2025-02-23T18:41:47.961Z [INFO]  server-11a54d02-247a-5998-d097-5eeb37d1746b: entering follower state: follower="Node at 11a54d02-247a-5998-d097-5eeb37d1746b [Follower]" leader-address= leader-id=
2025-02-23T18:41:47.961Z [INFO]  server-70d9e902-ee98-63f4-f28f-676112943f15: entering follower state: follower="Node at 70d9e902-ee98-63f4-f28f-676112943f15 [Follower]" leader-address= leader-id=
2025-02-23T18:41:47.961Z [INFO]  server-59d18cf9-bce4-1006-9fea-ea6f0b339f80: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11a54d02-247a-5998-d097-5eeb37d1746b Address:11a54d02-247a-5998-d097-5eeb37d1746b} {Suffrage:Voter ID:server-70d9e902-ee98-63f4-f28f-676112943f15 Address:70d9e902-ee98-63f4-f28f-676112943f15} {Suffrage:Voter ID:server-59d18cf9-bce4-1006-9fea-ea6f0b339f80 Address:59d18cf9-bce4-1006-9fea-ea6f0b339f80}]"
2025-02-23T18:41:47.961Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:47.961Z [INFO]  server-59d18cf9-bce4-1006-9fea-ea6f0b339f80: entering follower state: follower="Node at 59d18cf9-bce4-1006-9fea-ea6f0b339f80 [Follower]" leader-address= leader-id=
2025-02-23T18:41:48.020Z [WARN]  server-70d9e902-ee98-63f4-f28f-676112943f15: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:48.020Z [INFO]  server-70d9e902-ee98-63f4-f28f-676112943f15: entering candidate state: node="Node at 70d9e902-ee98-63f4-f28f-676112943f15 [Candidate]" term=2
2025-02-23T18:41:48.021Z [INFO]  server-70d9e902-ee98-63f4-f28f-676112943f15: election won: term=2 tally=2
2025-02-23T18:41:48.021Z [INFO]  server-70d9e902-ee98-63f4-f28f-676112943f15: entering leader state: leader="Node at 70d9e902-ee98-63f4-f28f-676112943f15 [Leader]"
2025-02-23T18:41:48.021Z [INFO]  server-70d9e902-ee98-63f4-f28f-676112943f15: added peer, starting replication: peer=server-11a54d02-247a-5998-d097-5eeb37d1746b
2025-02-23T18:41:48.021Z [INFO]  server-70d9e902-ee98-63f4-f28f-676112943f15: added peer, starting replication: peer=server-59d18cf9-bce4-1006-9fea-ea6f0b339f80
2025-02-23T18:41:48.021Z [INFO]  server-70d9e902-ee98-63f4-f28f-676112943f15: pipelining replication: peer="{Voter server-11a54d02-247a-5998-d097-5eeb37d1746b 11a54d02-247a-5998-d097-5eeb37d1746b}"
2025-02-23T18:41:48.022Z [INFO]  server-70d9e902-ee98-63f4-f28f-676112943f15: pipelining replication: peer="{Voter server-59d18cf9-bce4-1006-9fea-ea6f0b339f80 59d18cf9-bce4-1006-9fea-ea6f0b339f80}"
2025-02-23T18:41:48.127Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:48.021603946 +0000 UTC m=+12.319864602 (1 nodes), 60.067719ms from start of poll, 60.596614ms from cluster start. Timeout at 2025-02-23 18:41:48.126610081 +0000 UTC m=+12.424870752, 105.00615ms after stability
2025-02-23T18:41:48.127Z [INFO]  server-70d9e902-ee98-63f4-f28f-676112943f15: updating configuration: command=RemoveServer server-id=a5dccf31-548c-92c8-8e37-b7299451b40d server-addr= servers="[{Suffrage:Voter ID:server-11a54d02-247a-5998-d097-5eeb37d1746b Address:11a54d02-247a-5998-d097-5eeb37d1746b} {Suffrage:Voter ID:server-70d9e902-ee98-63f4-f28f-676112943f15 Address:70d9e902-ee98-63f4-f28f-676112943f15} {Suffrage:Voter ID:server-59d18cf9-bce4-1006-9fea-ea6f0b339f80 Address:59d18cf9-bce4-1006-9fea-ea6f0b339f80}]"
2025-02-23T18:41:48.127Z [INFO]  server-70d9e902-ee98-63f4-f28f-676112943f15: aborting pipeline replication: peer="{Voter server-59d18cf9-bce4-1006-9fea-ea6f0b339f80 59d18cf9-bce4-1006-9fea-ea6f0b339f80}"
2025-02-23T18:41:48.127Z [INFO]  server-70d9e902-ee98-63f4-f28f-676112943f15: aborting pipeline replication: peer="{Voter server-11a54d02-247a-5998-d097-5eeb37d1746b 11a54d02-247a-5998-d097-5eeb37d1746b}"
--- PASS: TestRaft_RemoveUnknownPeer (0.17s)
=== RUN   TestRaft_SnapshotRestore
2025-02-23T18:41:48.128Z [INFO]  server-5cd026d9-9b69-8913-e577-c7a0e3b77273: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5cd026d9-9b69-8913-e577-c7a0e3b77273 Address:5cd026d9-9b69-8913-e577-c7a0e3b77273}]"
2025-02-23T18:41:48.128Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:48.128Z [INFO]  server-5cd026d9-9b69-8913-e577-c7a0e3b77273: entering follower state: follower="Node at 5cd026d9-9b69-8913-e577-c7a0e3b77273 [Follower]" leader-address= leader-id=
2025-02-23T18:41:48.199Z [WARN]  server-5cd026d9-9b69-8913-e577-c7a0e3b77273: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:48.199Z [INFO]  server-5cd026d9-9b69-8913-e577-c7a0e3b77273: entering candidate state: node="Node at 5cd026d9-9b69-8913-e577-c7a0e3b77273 [Candidate]" term=2
2025-02-23T18:41:48.200Z [INFO]  server-5cd026d9-9b69-8913-e577-c7a0e3b77273: election won: term=2 tally=1
2025-02-23T18:41:48.200Z [INFO]  server-5cd026d9-9b69-8913-e577-c7a0e3b77273: entering leader state: leader="Node at 5cd026d9-9b69-8913-e577-c7a0e3b77273 [Leader]"
2025-02-23T18:41:48.305Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:48.200157963 +0000 UTC m=+12.498418769 (1 nodes), 71.245794ms from start of poll, 71.408431ms from cluster start. Timeout at 2025-02-23 18:41:48.305159418 +0000 UTC m=+12.603419548, 105.000779ms after stability
2025-02-23T18:41:48.311Z [INFO]  server-5cd026d9-9b69-8913-e577-c7a0e3b77273: starting snapshot up to: index=102
2025-02-23T18:41:48.311Z [INFO]  creating new snapshot: path=/tmp/raft1866108037/snapshots/2-102-1740336108311.tmp
2025-02-23T18:41:48.312Z [INFO]  server-5cd026d9-9b69-8913-e577-c7a0e3b77273: compacting logs: from=1 to=92
2025-02-23T18:41:48.312Z [INFO]  server-5cd026d9-9b69-8913-e577-c7a0e3b77273: snapshot complete up to: index=102
2025-02-23T18:41:48.313Z [INFO]  server-5cd026d9-9b69-8913-e577-c7a0e3b77273: starting restore from snapshot: id=2-102-1740336108311 last-index=102 last-term=2 size-in-bytes=693
2025-02-23T18:41:48.313Z [INFO]  server-5cd026d9-9b69-8913-e577-c7a0e3b77273: snapshot restore progress: id=2-102-1740336108311 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-02-23T18:41:48.313Z [INFO]  server-5cd026d9-9b69-8913-e577-c7a0e3b77273: restored from snapshot: id=2-102-1740336108311 last-index=102 last-term=2 size-in-bytes=693
2025-02-23T18:41:48.313Z [INFO]  server-5cd026d9-9b69-8913-e577-c7a0e3b77273: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5cd026d9-9b69-8913-e577-c7a0e3b77273 Address:5cd026d9-9b69-8913-e577-c7a0e3b77273}]"
--- PASS: TestRaft_SnapshotRestore (0.19s)
=== RUN   TestRaft_RestoreSnapshotOnStartup_Monotonic
2025-02-23T18:41:48.314Z [INFO]  server-45624d19-6335-a690-0b24-29883e2d7f09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45624d19-6335-a690-0b24-29883e2d7f09 Address:45624d19-6335-a690-0b24-29883e2d7f09}]"
2025-02-23T18:41:48.314Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:48.314Z [INFO]  server-45624d19-6335-a690-0b24-29883e2d7f09: entering follower state: follower="Node at 45624d19-6335-a690-0b24-29883e2d7f09 [Follower]" leader-address= leader-id=
2025-02-23T18:41:48.413Z [WARN]  server-45624d19-6335-a690-0b24-29883e2d7f09: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:48.413Z [INFO]  server-45624d19-6335-a690-0b24-29883e2d7f09: entering candidate state: node="Node at 45624d19-6335-a690-0b24-29883e2d7f09 [Candidate]" term=2
2025-02-23T18:41:48.413Z [INFO]  server-45624d19-6335-a690-0b24-29883e2d7f09: election won: term=2 tally=1
2025-02-23T18:41:48.413Z [INFO]  server-45624d19-6335-a690-0b24-29883e2d7f09: entering leader state: leader="Node at 45624d19-6335-a690-0b24-29883e2d7f09 [Leader]"
2025-02-23T18:41:48.518Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:48.413344556 +0000 UTC m=+12.711605719 (1 nodes), 98.786585ms from start of poll, 98.94213ms from cluster start. Timeout at 2025-02-23 18:41:48.518352069 +0000 UTC m=+12.816612719, 105.007ms after stability
2025-02-23T18:41:48.525Z [INFO]  server-45624d19-6335-a690-0b24-29883e2d7f09: starting snapshot up to: index=102
2025-02-23T18:41:48.525Z [INFO]  creating new snapshot: path=/tmp/raft1287451606/snapshots/2-102-1740336108525.tmp
2025-02-23T18:41:48.525Z [INFO]  server-45624d19-6335-a690-0b24-29883e2d7f09: compacting logs: from=1 to=92
2025-02-23T18:41:48.526Z [INFO]  server-45624d19-6335-a690-0b24-29883e2d7f09: snapshot complete up to: index=102
2025-02-23T18:41:48.526Z [INFO]  server-45624d19-6335-a690-0b24-29883e2d7f09: starting restore from snapshot: id=2-102-1740336108525 last-index=102 last-term=2 size-in-bytes=693
2025-02-23T18:41:48.526Z [INFO]  server-45624d19-6335-a690-0b24-29883e2d7f09: snapshot restore progress: id=2-102-1740336108525 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-02-23T18:41:48.526Z [INFO]  server-45624d19-6335-a690-0b24-29883e2d7f09: restored from snapshot: id=2-102-1740336108525 last-index=102 last-term=2 size-in-bytes=693
2025-02-23T18:41:48.526Z [INFO]  server-45624d19-6335-a690-0b24-29883e2d7f09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45624d19-6335-a690-0b24-29883e2d7f09 Address:45624d19-6335-a690-0b24-29883e2d7f09}]"
--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.21s)
=== RUN   TestRaft_SnapshotRestore_Progress
2025-02-23T18:41:48.527Z [INFO]  server-4a69cd66-e46b-5862-680f-6d55121b18da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a69cd66-e46b-5862-680f-6d55121b18da Address:4a69cd66-e46b-5862-680f-6d55121b18da}]"
2025-02-23T18:41:48.527Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:48.527Z [INFO]  server-4a69cd66-e46b-5862-680f-6d55121b18da: entering follower state: follower="Node at 4a69cd66-e46b-5862-680f-6d55121b18da [Follower]" leader-address= leader-id=
2025-02-23T18:41:48.582Z [WARN]  server-4a69cd66-e46b-5862-680f-6d55121b18da: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:48.582Z [INFO]  server-4a69cd66-e46b-5862-680f-6d55121b18da: entering candidate state: node="Node at 4a69cd66-e46b-5862-680f-6d55121b18da [Candidate]" term=2
2025-02-23T18:41:48.582Z [INFO]  server-4a69cd66-e46b-5862-680f-6d55121b18da: election won: term=2 tally=1
2025-02-23T18:41:48.582Z [INFO]  server-4a69cd66-e46b-5862-680f-6d55121b18da: entering leader state: leader="Node at 4a69cd66-e46b-5862-680f-6d55121b18da [Leader]"
2025-02-23T18:41:48.709Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:48.582375548 +0000 UTC m=+12.880635678 (1 nodes), 54.649349ms from start of poll, 54.815551ms from cluster start. Timeout at 2025-02-23 18:41:48.687378123 +0000 UTC m=+12.985638781, 105.003103ms after stability
2025-02-23T18:41:48.715Z [INFO]  server-4a69cd66-e46b-5862-680f-6d55121b18da: starting snapshot up to: index=102
2025-02-23T18:41:48.715Z [INFO]  creating new snapshot: path=/tmp/raft1710131516/snapshots/2-102-1740336108715.tmp
2025-02-23T18:41:48.715Z [INFO]  server-4a69cd66-e46b-5862-680f-6d55121b18da: compacting logs: from=1 to=92
2025-02-23T18:41:48.715Z [INFO]  server-4a69cd66-e46b-5862-680f-6d55121b18da: snapshot complete up to: index=102
--- PASS: TestRaft_SnapshotRestore_Progress (0.19s)
=== RUN   TestRaft_NoRestoreOnStart
2025-02-23T18:41:48.717Z [INFO]  server-b2d418fb-d2a7-293c-2cc0-ec46d8d70e65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2d418fb-d2a7-293c-2cc0-ec46d8d70e65 Address:b2d418fb-d2a7-293c-2cc0-ec46d8d70e65}]"
2025-02-23T18:41:48.717Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:48.717Z [INFO]  server-b2d418fb-d2a7-293c-2cc0-ec46d8d70e65: entering follower state: follower="Node at b2d418fb-d2a7-293c-2cc0-ec46d8d70e65 [Follower]" leader-address= leader-id=
2025-02-23T18:41:48.792Z [WARN]  server-b2d418fb-d2a7-293c-2cc0-ec46d8d70e65: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:48.792Z [INFO]  server-b2d418fb-d2a7-293c-2cc0-ec46d8d70e65: entering candidate state: node="Node at b2d418fb-d2a7-293c-2cc0-ec46d8d70e65 [Candidate]" term=2
2025-02-23T18:41:48.792Z [INFO]  server-b2d418fb-d2a7-293c-2cc0-ec46d8d70e65: election won: term=2 tally=1
2025-02-23T18:41:48.792Z [INFO]  server-b2d418fb-d2a7-293c-2cc0-ec46d8d70e65: entering leader state: leader="Node at b2d418fb-d2a7-293c-2cc0-ec46d8d70e65 [Leader]"
2025-02-23T18:41:48.897Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:48.792550712 +0000 UTC m=+13.090811454 (1 nodes), 75.220568ms from start of poll, 75.354458ms from cluster start. Timeout at 2025-02-23 18:41:48.897557231 +0000 UTC m=+13.195817918, 105.006464ms after stability
2025-02-23T18:41:48.903Z [INFO]  server-b2d418fb-d2a7-293c-2cc0-ec46d8d70e65: starting snapshot up to: index=102
2025-02-23T18:41:48.903Z [INFO]  creating new snapshot: path=/tmp/raft3408558991/snapshots/2-102-1740336108903.tmp
2025-02-23T18:41:48.903Z [INFO]  server-b2d418fb-d2a7-293c-2cc0-ec46d8d70e65: compacting logs: from=1 to=92
2025-02-23T18:41:48.903Z [INFO]  server-b2d418fb-d2a7-293c-2cc0-ec46d8d70e65: snapshot complete up to: index=102
2025-02-23T18:41:48.904Z [INFO]  server-b2d418fb-d2a7-293c-2cc0-ec46d8d70e65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2d418fb-d2a7-293c-2cc0-ec46d8d70e65 Address:b2d418fb-d2a7-293c-2cc0-ec46d8d70e65}]"
--- PASS: TestRaft_NoRestoreOnStart (0.19s)
=== RUN   TestRaft_SnapshotRestore_PeerChange
    raft_test.go:1266: Skipping unreliable test
--- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
=== RUN   TestRaft_AutoSnapshot
2025-02-23T18:41:48.904Z [INFO]  server-4bbd5db5-1684-598b-a1fc-bdafa56e0f9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4bbd5db5-1684-598b-a1fc-bdafa56e0f9d Address:4bbd5db5-1684-598b-a1fc-bdafa56e0f9d}]"
2025-02-23T18:41:48.904Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:48.904Z [INFO]  server-b2d418fb-d2a7-293c-2cc0-ec46d8d70e65: entering follower state: follower="Node at b2d418fb-d2a7-293c-2cc0-ec46d8d70e65 [Follower]" leader-address= leader-id=
2025-02-23T18:41:48.904Z [INFO]  server-4bbd5db5-1684-598b-a1fc-bdafa56e0f9d: entering follower state: follower="Node at 4bbd5db5-1684-598b-a1fc-bdafa56e0f9d [Follower]" leader-address= leader-id=
2025-02-23T18:41:48.999Z [WARN]  server-b2d418fb-d2a7-293c-2cc0-ec46d8d70e65: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:48.999Z [INFO]  server-b2d418fb-d2a7-293c-2cc0-ec46d8d70e65: entering candidate state: node="Node at b2d418fb-d2a7-293c-2cc0-ec46d8d70e65 [Candidate]" term=3
2025-02-23T18:41:48.999Z [INFO]  server-b2d418fb-d2a7-293c-2cc0-ec46d8d70e65: election won: term=3 tally=1
2025-02-23T18:41:48.999Z [INFO]  server-b2d418fb-d2a7-293c-2cc0-ec46d8d70e65: entering leader state: leader="Node at b2d418fb-d2a7-293c-2cc0-ec46d8d70e65 [Leader]"
2025-02-23T18:41:49.002Z [WARN]  server-4bbd5db5-1684-598b-a1fc-bdafa56e0f9d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:49.003Z [INFO]  server-4bbd5db5-1684-598b-a1fc-bdafa56e0f9d: entering candidate state: node="Node at 4bbd5db5-1684-598b-a1fc-bdafa56e0f9d [Candidate]" term=2
2025-02-23T18:41:49.003Z [INFO]  server-4bbd5db5-1684-598b-a1fc-bdafa56e0f9d: election won: term=2 tally=1
2025-02-23T18:41:49.003Z [INFO]  server-4bbd5db5-1684-598b-a1fc-bdafa56e0f9d: entering leader state: leader="Node at 4bbd5db5-1684-598b-a1fc-bdafa56e0f9d [Leader]"
2025-02-23T18:41:49.109Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:49.003293097 +0000 UTC m=+13.301553734 (1 nodes), 98.592696ms from start of poll, 98.736869ms from cluster start. Timeout at 2025-02-23 18:41:49.108298637 +0000 UTC m=+13.406559328, 105.005594ms after stability
2025-02-23T18:41:49.121Z [INFO]  server-4bbd5db5-1684-598b-a1fc-bdafa56e0f9d: starting snapshot up to: index=102
2025-02-23T18:41:49.121Z [INFO]  creating new snapshot: path=/tmp/raft1764666621/snapshots/2-102-1740336109121.tmp
2025-02-23T18:41:49.122Z [INFO]  server-4bbd5db5-1684-598b-a1fc-bdafa56e0f9d: compacting logs: from=1 to=92
2025-02-23T18:41:49.122Z [INFO]  server-4bbd5db5-1684-598b-a1fc-bdafa56e0f9d: snapshot complete up to: index=102
--- PASS: TestRaft_AutoSnapshot (0.32s)
=== RUN   TestRaft_UserSnapshot
2025-02-23T18:41:49.222Z [INFO]  server-adab9239-aefa-156c-dbe1-13b1291d8a0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-adab9239-aefa-156c-dbe1-13b1291d8a0a Address:adab9239-aefa-156c-dbe1-13b1291d8a0a}]"
2025-02-23T18:41:49.222Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:49.223Z [INFO]  server-adab9239-aefa-156c-dbe1-13b1291d8a0a: entering follower state: follower="Node at adab9239-aefa-156c-dbe1-13b1291d8a0a [Follower]" leader-address= leader-id=
2025-02-23T18:41:49.304Z [WARN]  server-adab9239-aefa-156c-dbe1-13b1291d8a0a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:49.304Z [INFO]  server-adab9239-aefa-156c-dbe1-13b1291d8a0a: entering candidate state: node="Node at adab9239-aefa-156c-dbe1-13b1291d8a0a [Candidate]" term=2
2025-02-23T18:41:49.304Z [INFO]  server-adab9239-aefa-156c-dbe1-13b1291d8a0a: election won: term=2 tally=1
2025-02-23T18:41:49.304Z [INFO]  server-adab9239-aefa-156c-dbe1-13b1291d8a0a: entering leader state: leader="Node at adab9239-aefa-156c-dbe1-13b1291d8a0a [Leader]"
2025-02-23T18:41:49.410Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:49.304521585 +0000 UTC m=+13.602782443 (1 nodes), 81.512045ms from start of poll, 81.698672ms from cluster start. Timeout at 2025-02-23 18:41:49.409525239 +0000 UTC m=+13.707785874, 105.003431ms after stability
2025-02-23T18:41:49.410Z [ERROR] server-adab9239-aefa-156c-dbe1-13b1291d8a0a: failed to take snapshot: error="nothing new to snapshot"
2025-02-23T18:41:49.411Z [INFO]  server-adab9239-aefa-156c-dbe1-13b1291d8a0a: starting snapshot up to: index=12
2025-02-23T18:41:49.411Z [INFO]  creating new snapshot: path=/tmp/raft1060213345/snapshots/2-12-1740336109411.tmp
2025-02-23T18:41:49.412Z [INFO]  server-adab9239-aefa-156c-dbe1-13b1291d8a0a: compacting logs: from=1 to=2
2025-02-23T18:41:49.412Z [INFO]  server-adab9239-aefa-156c-dbe1-13b1291d8a0a: snapshot complete up to: index=12
--- PASS: TestRaft_UserSnapshot (0.19s)
=== RUN   TestRaft_UserRestore
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false
2025-02-23T18:41:49.413Z [INFO]  server-4ee97ee1-69c3-b16a-4ea1-5585297fa464: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ee97ee1-69c3-b16a-4ea1-5585297fa464 Address:4ee97ee1-69c3-b16a-4ea1-5585297fa464} {Suffrage:Voter ID:server-869e74be-5014-d20b-9a4b-d1578de48a7c Address:869e74be-5014-d20b-9a4b-d1578de48a7c} {Suffrage:Voter ID:server-48e2f3c2-2f12-5fb5-515d-7ab37012283d Address:48e2f3c2-2f12-5fb5-515d-7ab37012283d}]"
2025-02-23T18:41:49.413Z [INFO]  server-869e74be-5014-d20b-9a4b-d1578de48a7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ee97ee1-69c3-b16a-4ea1-5585297fa464 Address:4ee97ee1-69c3-b16a-4ea1-5585297fa464} {Suffrage:Voter ID:server-869e74be-5014-d20b-9a4b-d1578de48a7c Address:869e74be-5014-d20b-9a4b-d1578de48a7c} {Suffrage:Voter ID:server-48e2f3c2-2f12-5fb5-515d-7ab37012283d Address:48e2f3c2-2f12-5fb5-515d-7ab37012283d}]"
2025-02-23T18:41:49.414Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ee97ee1-69c3-b16a-4ea1-5585297fa464 Address:4ee97ee1-69c3-b16a-4ea1-5585297fa464} {Suffrage:Voter ID:server-869e74be-5014-d20b-9a4b-d1578de48a7c Address:869e74be-5014-d20b-9a4b-d1578de48a7c} {Suffrage:Voter ID:server-48e2f3c2-2f12-5fb5-515d-7ab37012283d Address:48e2f3c2-2f12-5fb5-515d-7ab37012283d}]"
2025-02-23T18:41:49.414Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:49.414Z [INFO]  server-4ee97ee1-69c3-b16a-4ea1-5585297fa464: entering follower state: follower="Node at 4ee97ee1-69c3-b16a-4ea1-5585297fa464 [Follower]" leader-address= leader-id=
2025-02-23T18:41:49.414Z [INFO]  server-869e74be-5014-d20b-9a4b-d1578de48a7c: entering follower state: follower="Node at 869e74be-5014-d20b-9a4b-d1578de48a7c [Follower]" leader-address= leader-id=
2025-02-23T18:41:49.414Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: entering follower state: follower="Node at 48e2f3c2-2f12-5fb5-515d-7ab37012283d [Follower]" leader-address= leader-id=
2025-02-23T18:41:50.040Z [WARN]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:50.040Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: entering candidate state: node="Node at 48e2f3c2-2f12-5fb5-515d-7ab37012283d [Candidate]" term=2
2025-02-23T18:41:50.040Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: election won: term=2 tally=2
2025-02-23T18:41:50.040Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: entering leader state: leader="Node at 48e2f3c2-2f12-5fb5-515d-7ab37012283d [Leader]"
2025-02-23T18:41:50.040Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: added peer, starting replication: peer=server-4ee97ee1-69c3-b16a-4ea1-5585297fa464
2025-02-23T18:41:50.040Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: added peer, starting replication: peer=server-869e74be-5014-d20b-9a4b-d1578de48a7c
2025-02-23T18:41:50.041Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: pipelining replication: peer="{Voter server-869e74be-5014-d20b-9a4b-d1578de48a7c 869e74be-5014-d20b-9a4b-d1578de48a7c}"
2025-02-23T18:41:50.041Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: pipelining replication: peer="{Voter server-4ee97ee1-69c3-b16a-4ea1-5585297fa464 4ee97ee1-69c3-b16a-4ea1-5585297fa464}"
2025-02-23T18:41:51.047Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:50.041544011 +0000 UTC m=+14.339804790 (1 nodes), 627.426299ms from start of poll, 627.938149ms from cluster start. Timeout at 2025-02-23 18:41:51.046553804 +0000 UTC m=+15.344814824, 1.005010034s after stability
2025-02-23T18:41:51.048Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: starting snapshot up to: index=12
2025-02-23T18:41:51.048Z [INFO]  creating new snapshot: path=/tmp/raft1343889158/snapshots/2-12-1740336111048.tmp
2025-02-23T18:41:51.049Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: snapshot complete up to: index=12
2025-02-23T18:41:51.051Z [INFO]  creating new snapshot: path=/tmp/raft1343889158/snapshots/2-23-1740336111051.tmp
2025-02-23T18:41:51.062Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: copied to local snapshot: bytes=71
2025-02-23T18:41:51.062Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: snapshot restore progress: id=2-23-1740336111051 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:41:51.062Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: restored user snapshot: index=23
2025-02-23T18:41:51.062Z [ERROR] server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: failed to get log: index=23 error="log not found"
2025-02-23T18:41:51.062Z [ERROR] server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: failed to get log: index=23 error="log not found"
2025-02-23T18:41:51.062Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: aborting pipeline replication: peer="{Voter server-4ee97ee1-69c3-b16a-4ea1-5585297fa464 4ee97ee1-69c3-b16a-4ea1-5585297fa464}"
2025-02-23T18:41:51.062Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: aborting pipeline replication: peer="{Voter server-869e74be-5014-d20b-9a4b-d1578de48a7c 869e74be-5014-d20b-9a4b-d1578de48a7c}"
2025-02-23T18:41:51.070Z [ERROR] server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: failed to get log: index=23 error="log not found"
2025-02-23T18:41:51.070Z [INFO]  creating new snapshot: path=/tmp/raft2747742493/snapshots/2-23-1740336111070.tmp
2025-02-23T18:41:51.071Z [ERROR] server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: failed to get log: index=23 error="log not found"
2025-02-23T18:41:51.072Z [INFO]  server-869e74be-5014-d20b-9a4b-d1578de48a7c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:41:51.072Z [INFO]  creating new snapshot: path=/tmp/raft2603432061/snapshots/2-23-1740336111072.tmp
2025-02-23T18:41:51.072Z [INFO]  server-869e74be-5014-d20b-9a4b-d1578de48a7c: copied to local snapshot: bytes=71
2025-02-23T18:41:51.073Z [INFO]  server-4ee97ee1-69c3-b16a-4ea1-5585297fa464: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:41:51.073Z [INFO]  server-869e74be-5014-d20b-9a4b-d1578de48a7c: snapshot restore progress: id=2-23-1740336111070 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:41:51.073Z [INFO]  server-869e74be-5014-d20b-9a4b-d1578de48a7c: Installed remote snapshot
2025-02-23T18:41:51.073Z [WARN]  server-869e74be-5014-d20b-9a4b-d1578de48a7c: skipping application of old log: index=23
2025-02-23T18:41:51.073Z [INFO]  server-4ee97ee1-69c3-b16a-4ea1-5585297fa464: copied to local snapshot: bytes=71
2025-02-23T18:41:51.073Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: pipelining replication: peer="{Voter server-869e74be-5014-d20b-9a4b-d1578de48a7c 869e74be-5014-d20b-9a4b-d1578de48a7c}"
2025-02-23T18:41:51.074Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:41:51.074Z [INFO]  server-4ee97ee1-69c3-b16a-4ea1-5585297fa464: snapshot restore progress: id=2-23-1740336111072 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:41:51.074Z [INFO]  server-4ee97ee1-69c3-b16a-4ea1-5585297fa464: Installed remote snapshot
2025-02-23T18:41:51.074Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: pipelining replication: peer="{Voter server-4ee97ee1-69c3-b16a-4ea1-5585297fa464 4ee97ee1-69c3-b16a-4ea1-5585297fa464}"
2025-02-23T18:41:52.079Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:41:51.074244156 +0000 UTC m=+15.372504286 (2 nodes), 0s from start of poll, 1.660637645s from cluster start. Timeout at 2025-02-23 18:41:52.079259325 +0000 UTC m=+16.377520193, 1.005015907s after stability
2025-02-23T18:41:52.093Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: aborting pipeline replication: peer="{Voter server-869e74be-5014-d20b-9a4b-d1578de48a7c 869e74be-5014-d20b-9a4b-d1578de48a7c}"
2025-02-23T18:41:52.093Z [INFO]  server-48e2f3c2-2f12-5fb5-515d-7ab37012283d: aborting pipeline replication: peer="{Voter server-4ee97ee1-69c3-b16a-4ea1-5585297fa464 4ee97ee1-69c3-b16a-4ea1-5585297fa464}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false
2025-02-23T18:41:52.095Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1600295b-1c7f-7711-41f6-eb0fa70d10ff Address:1600295b-1c7f-7711-41f6-eb0fa70d10ff} {Suffrage:Voter ID:server-0f4a0764-7c3e-1bd0-62fa-246afec1ab8c Address:0f4a0764-7c3e-1bd0-62fa-246afec1ab8c} {Suffrage:Voter ID:server-a8ec6b27-ee89-adbb-a399-8ed6115ebb1e Address:a8ec6b27-ee89-adbb-a399-8ed6115ebb1e}]"
2025-02-23T18:41:52.096Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: entering follower state: follower="Node at 1600295b-1c7f-7711-41f6-eb0fa70d10ff [Follower]" leader-address= leader-id=
2025-02-23T18:41:52.096Z [INFO]  server-0f4a0764-7c3e-1bd0-62fa-246afec1ab8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1600295b-1c7f-7711-41f6-eb0fa70d10ff Address:1600295b-1c7f-7711-41f6-eb0fa70d10ff} {Suffrage:Voter ID:server-0f4a0764-7c3e-1bd0-62fa-246afec1ab8c Address:0f4a0764-7c3e-1bd0-62fa-246afec1ab8c} {Suffrage:Voter ID:server-a8ec6b27-ee89-adbb-a399-8ed6115ebb1e Address:a8ec6b27-ee89-adbb-a399-8ed6115ebb1e}]"
2025-02-23T18:41:52.096Z [INFO]  server-0f4a0764-7c3e-1bd0-62fa-246afec1ab8c: entering follower state: follower="Node at 0f4a0764-7c3e-1bd0-62fa-246afec1ab8c [Follower]" leader-address= leader-id=
2025-02-23T18:41:52.096Z [INFO]  server-a8ec6b27-ee89-adbb-a399-8ed6115ebb1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1600295b-1c7f-7711-41f6-eb0fa70d10ff Address:1600295b-1c7f-7711-41f6-eb0fa70d10ff} {Suffrage:Voter ID:server-0f4a0764-7c3e-1bd0-62fa-246afec1ab8c Address:0f4a0764-7c3e-1bd0-62fa-246afec1ab8c} {Suffrage:Voter ID:server-a8ec6b27-ee89-adbb-a399-8ed6115ebb1e Address:a8ec6b27-ee89-adbb-a399-8ed6115ebb1e}]"
2025-02-23T18:41:52.097Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:52.097Z [INFO]  server-a8ec6b27-ee89-adbb-a399-8ed6115ebb1e: entering follower state: follower="Node at a8ec6b27-ee89-adbb-a399-8ed6115ebb1e [Follower]" leader-address= leader-id=
2025-02-23T18:41:52.976Z [WARN]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:52.976Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: entering candidate state: node="Node at 1600295b-1c7f-7711-41f6-eb0fa70d10ff [Candidate]" term=2
2025-02-23T18:41:52.976Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: election won: term=2 tally=2
2025-02-23T18:41:52.976Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: entering leader state: leader="Node at 1600295b-1c7f-7711-41f6-eb0fa70d10ff [Leader]"
2025-02-23T18:41:52.976Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: added peer, starting replication: peer=server-0f4a0764-7c3e-1bd0-62fa-246afec1ab8c
2025-02-23T18:41:52.976Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: added peer, starting replication: peer=server-a8ec6b27-ee89-adbb-a399-8ed6115ebb1e
2025-02-23T18:41:52.976Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: pipelining replication: peer="{Voter server-0f4a0764-7c3e-1bd0-62fa-246afec1ab8c 0f4a0764-7c3e-1bd0-62fa-246afec1ab8c}"
2025-02-23T18:41:52.976Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: pipelining replication: peer="{Voter server-a8ec6b27-ee89-adbb-a399-8ed6115ebb1e a8ec6b27-ee89-adbb-a399-8ed6115ebb1e}"
2025-02-23T18:41:53.982Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:52.976670752 +0000 UTC m=+17.274930882 (1 nodes), 879.633174ms from start of poll, 881.406687ms from cluster start. Timeout at 2025-02-23 18:41:53.98168101 +0000 UTC m=+18.279941760, 1.005010878s after stability
2025-02-23T18:41:53.983Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: starting snapshot up to: index=12
2025-02-23T18:41:53.983Z [INFO]  creating new snapshot: path=/tmp/raft627737788/snapshots/2-12-1740336113983.tmp
2025-02-23T18:41:53.984Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: snapshot complete up to: index=12
2025-02-23T18:41:53.986Z [INFO]  creating new snapshot: path=/tmp/raft627737788/snapshots/2-23-1740336113986.tmp
2025-02-23T18:41:53.989Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: copied to local snapshot: bytes=71
2025-02-23T18:41:53.989Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: snapshot restore progress: id=2-23-1740336113986 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:41:53.989Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: removing all old logs from log store
2025-02-23T18:41:53.989Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: compacting logs: from=1 to=22
2025-02-23T18:41:53.989Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: restored user snapshot: index=23
2025-02-23T18:41:53.989Z [ERROR] server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: failed to get log: index=22 error="log not found"
2025-02-23T18:41:53.989Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: aborting pipeline replication: peer="{Voter server-0f4a0764-7c3e-1bd0-62fa-246afec1ab8c 0f4a0764-7c3e-1bd0-62fa-246afec1ab8c}"
2025-02-23T18:41:53.989Z [ERROR] server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: failed to get log: index=22 error="log not found"
2025-02-23T18:41:53.990Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: aborting pipeline replication: peer="{Voter server-a8ec6b27-ee89-adbb-a399-8ed6115ebb1e a8ec6b27-ee89-adbb-a399-8ed6115ebb1e}"
2025-02-23T18:41:53.996Z [ERROR] server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: failed to get log: index=22 error="log not found"
2025-02-23T18:41:53.997Z [INFO]  creating new snapshot: path=/tmp/raft1903227800/snapshots/2-23-1740336113997.tmp
2025-02-23T18:41:53.998Z [INFO]  server-0f4a0764-7c3e-1bd0-62fa-246afec1ab8c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:41:53.998Z [ERROR] server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: failed to get log: index=22 error="log not found"
2025-02-23T18:41:53.998Z [INFO]  server-0f4a0764-7c3e-1bd0-62fa-246afec1ab8c: copied to local snapshot: bytes=71
2025-02-23T18:41:53.998Z [INFO]  creating new snapshot: path=/tmp/raft2088598491/snapshots/2-23-1740336113998.tmp
2025-02-23T18:41:53.998Z [INFO]  server-0f4a0764-7c3e-1bd0-62fa-246afec1ab8c: snapshot restore progress: id=2-23-1740336113997 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:41:53.998Z [INFO]  server-0f4a0764-7c3e-1bd0-62fa-246afec1ab8c: removing all old logs from log store
2025-02-23T18:41:53.998Z [INFO]  server-0f4a0764-7c3e-1bd0-62fa-246afec1ab8c: compacting logs: from=1 to=22
2025-02-23T18:41:53.998Z [INFO]  server-0f4a0764-7c3e-1bd0-62fa-246afec1ab8c: Installed remote snapshot
2025-02-23T18:41:53.998Z [WARN]  server-0f4a0764-7c3e-1bd0-62fa-246afec1ab8c: skipping application of old log: index=22
2025-02-23T18:41:53.998Z [INFO]  server-a8ec6b27-ee89-adbb-a399-8ed6115ebb1e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:41:53.999Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: pipelining replication: peer="{Voter server-0f4a0764-7c3e-1bd0-62fa-246afec1ab8c 0f4a0764-7c3e-1bd0-62fa-246afec1ab8c}"
2025-02-23T18:41:53.999Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:41:53.999Z [INFO]  server-a8ec6b27-ee89-adbb-a399-8ed6115ebb1e: copied to local snapshot: bytes=71
2025-02-23T18:41:53.999Z [INFO]  server-a8ec6b27-ee89-adbb-a399-8ed6115ebb1e: snapshot restore progress: id=2-23-1740336113998 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:41:53.999Z [INFO]  server-a8ec6b27-ee89-adbb-a399-8ed6115ebb1e: removing all old logs from log store
2025-02-23T18:41:53.999Z [INFO]  server-a8ec6b27-ee89-adbb-a399-8ed6115ebb1e: compacting logs: from=1 to=22
2025-02-23T18:41:53.999Z [INFO]  server-a8ec6b27-ee89-adbb-a399-8ed6115ebb1e: Installed remote snapshot
2025-02-23T18:41:53.999Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: pipelining replication: peer="{Voter server-a8ec6b27-ee89-adbb-a399-8ed6115ebb1e a8ec6b27-ee89-adbb-a399-8ed6115ebb1e}"
2025-02-23T18:41:55.005Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:41:53.999208967 +0000 UTC m=+18.297469097 (2 nodes), 0s from start of poll, 1.903944902s from cluster start. Timeout at 2025-02-23 18:41:55.004211979 +0000 UTC m=+19.302472109, 1.005003012s after stability
2025-02-23T18:41:55.019Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: aborting pipeline replication: peer="{Voter server-a8ec6b27-ee89-adbb-a399-8ed6115ebb1e a8ec6b27-ee89-adbb-a399-8ed6115ebb1e}"
2025-02-23T18:41:55.019Z [INFO]  server-1600295b-1c7f-7711-41f6-eb0fa70d10ff: aborting pipeline replication: peer="{Voter server-0f4a0764-7c3e-1bd0-62fa-246afec1ab8c 0f4a0764-7c3e-1bd0-62fa-246afec1ab8c}"
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true
2025-02-23T18:41:55.021Z [INFO]  server-1b70885f-536a-58bb-f394-dc0142351c9a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b70885f-536a-58bb-f394-dc0142351c9a Address:1b70885f-536a-58bb-f394-dc0142351c9a} {Suffrage:Voter ID:server-f5aa5e19-edf4-6c19-4459-a9bd0cd0e9e9 Address:f5aa5e19-edf4-6c19-4459-a9bd0cd0e9e9} {Suffrage:Voter ID:server-56797196-272f-3d6d-9efe-651c7492a21a Address:56797196-272f-3d6d-9efe-651c7492a21a}]"
2025-02-23T18:41:55.021Z [INFO]  server-f5aa5e19-edf4-6c19-4459-a9bd0cd0e9e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b70885f-536a-58bb-f394-dc0142351c9a Address:1b70885f-536a-58bb-f394-dc0142351c9a} {Suffrage:Voter ID:server-f5aa5e19-edf4-6c19-4459-a9bd0cd0e9e9 Address:f5aa5e19-edf4-6c19-4459-a9bd0cd0e9e9} {Suffrage:Voter ID:server-56797196-272f-3d6d-9efe-651c7492a21a Address:56797196-272f-3d6d-9efe-651c7492a21a}]"
2025-02-23T18:41:55.022Z [INFO]  server-56797196-272f-3d6d-9efe-651c7492a21a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b70885f-536a-58bb-f394-dc0142351c9a Address:1b70885f-536a-58bb-f394-dc0142351c9a} {Suffrage:Voter ID:server-f5aa5e19-edf4-6c19-4459-a9bd0cd0e9e9 Address:f5aa5e19-edf4-6c19-4459-a9bd0cd0e9e9} {Suffrage:Voter ID:server-56797196-272f-3d6d-9efe-651c7492a21a Address:56797196-272f-3d6d-9efe-651c7492a21a}]"
2025-02-23T18:41:55.022Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:55.022Z [INFO]  server-1b70885f-536a-58bb-f394-dc0142351c9a: entering follower state: follower="Node at 1b70885f-536a-58bb-f394-dc0142351c9a [Follower]" leader-address= leader-id=
2025-02-23T18:41:55.022Z [INFO]  server-f5aa5e19-edf4-6c19-4459-a9bd0cd0e9e9: entering follower state: follower="Node at f5aa5e19-edf4-6c19-4459-a9bd0cd0e9e9 [Follower]" leader-address= leader-id=
2025-02-23T18:41:55.022Z [INFO]  server-56797196-272f-3d6d-9efe-651c7492a21a: entering follower state: follower="Node at 56797196-272f-3d6d-9efe-651c7492a21a [Follower]" leader-address= leader-id=
2025-02-23T18:41:55.571Z [WARN]  server-56797196-272f-3d6d-9efe-651c7492a21a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:55.571Z [INFO]  server-56797196-272f-3d6d-9efe-651c7492a21a: entering candidate state: node="Node at 56797196-272f-3d6d-9efe-651c7492a21a [Candidate]" term=2
2025-02-23T18:41:55.571Z [INFO]  server-56797196-272f-3d6d-9efe-651c7492a21a: election won: term=2 tally=2
2025-02-23T18:41:55.571Z [INFO]  server-56797196-272f-3d6d-9efe-651c7492a21a: entering leader state: leader="Node at 56797196-272f-3d6d-9efe-651c7492a21a [Leader]"
2025-02-23T18:41:55.571Z [INFO]  server-56797196-272f-3d6d-9efe-651c7492a21a: added peer, starting replication: peer=server-1b70885f-536a-58bb-f394-dc0142351c9a
2025-02-23T18:41:55.571Z [INFO]  server-56797196-272f-3d6d-9efe-651c7492a21a: added peer, starting replication: peer=server-f5aa5e19-edf4-6c19-4459-a9bd0cd0e9e9
2025-02-23T18:41:55.572Z [INFO]  server-56797196-272f-3d6d-9efe-651c7492a21a: pipelining replication: peer="{Voter server-f5aa5e19-edf4-6c19-4459-a9bd0cd0e9e9 f5aa5e19-edf4-6c19-4459-a9bd0cd0e9e9}"
2025-02-23T18:41:55.572Z [INFO]  server-56797196-272f-3d6d-9efe-651c7492a21a: pipelining replication: peer="{Voter server-1b70885f-536a-58bb-f394-dc0142351c9a 1b70885f-536a-58bb-f394-dc0142351c9a}"
2025-02-23T18:41:56.577Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:55.572053359 +0000 UTC m=+19.870314151 (1 nodes), 549.945222ms from start of poll, 550.461626ms from cluster start. Timeout at 2025-02-23 18:41:56.577056625 +0000 UTC m=+20.875317346, 1.005003195s after stability
2025-02-23T18:41:56.579Z [INFO]  server-56797196-272f-3d6d-9efe-651c7492a21a: starting snapshot up to: index=12
2025-02-23T18:41:56.579Z [INFO]  creating new snapshot: path=/tmp/raft1964361322/snapshots/2-12-1740336116579.tmp
2025-02-23T18:41:56.580Z [INFO]  server-56797196-272f-3d6d-9efe-651c7492a21a: snapshot complete up to: index=12
2025-02-23T18:41:56.585Z [INFO]  server-2c2d0c4b-9dbe-85c8-d539-791e02103b51: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c2d0c4b-9dbe-85c8-d539-791e02103b51 Address:2c2d0c4b-9dbe-85c8-d539-791e02103b51} {Suffrage:Voter ID:server-8c9f016b-80cc-3c42-a69a-81ff1ca808c2 Address:8c9f016b-80cc-3c42-a69a-81ff1ca808c2} {Suffrage:Voter ID:server-880f119f-8e53-996d-b554-c6db1e4ffa04 Address:880f119f-8e53-996d-b554-c6db1e4ffa04}]"
2025-02-23T18:41:56.585Z [INFO]  server-8c9f016b-80cc-3c42-a69a-81ff1ca808c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c2d0c4b-9dbe-85c8-d539-791e02103b51 Address:2c2d0c4b-9dbe-85c8-d539-791e02103b51} {Suffrage:Voter ID:server-8c9f016b-80cc-3c42-a69a-81ff1ca808c2 Address:8c9f016b-80cc-3c42-a69a-81ff1ca808c2} {Suffrage:Voter ID:server-880f119f-8e53-996d-b554-c6db1e4ffa04 Address:880f119f-8e53-996d-b554-c6db1e4ffa04}]"
2025-02-23T18:41:56.585Z [INFO]  server-880f119f-8e53-996d-b554-c6db1e4ffa04: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c2d0c4b-9dbe-85c8-d539-791e02103b51 Address:2c2d0c4b-9dbe-85c8-d539-791e02103b51} {Suffrage:Voter ID:server-8c9f016b-80cc-3c42-a69a-81ff1ca808c2 Address:8c9f016b-80cc-3c42-a69a-81ff1ca808c2} {Suffrage:Voter ID:server-880f119f-8e53-996d-b554-c6db1e4ffa04 Address:880f119f-8e53-996d-b554-c6db1e4ffa04}]"
2025-02-23T18:41:56.585Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:56.585Z [INFO]  server-2c2d0c4b-9dbe-85c8-d539-791e02103b51: entering follower state: follower="Node at 2c2d0c4b-9dbe-85c8-d539-791e02103b51 [Follower]" leader-address= leader-id=
2025-02-23T18:41:56.585Z [INFO]  server-8c9f016b-80cc-3c42-a69a-81ff1ca808c2: entering follower state: follower="Node at 8c9f016b-80cc-3c42-a69a-81ff1ca808c2 [Follower]" leader-address= leader-id=
2025-02-23T18:41:56.585Z [INFO]  server-880f119f-8e53-996d-b554-c6db1e4ffa04: entering follower state: follower="Node at 880f119f-8e53-996d-b554-c6db1e4ffa04 [Follower]" leader-address= leader-id=
2025-02-23T18:41:57.113Z [WARN]  server-880f119f-8e53-996d-b554-c6db1e4ffa04: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:57.113Z [INFO]  server-880f119f-8e53-996d-b554-c6db1e4ffa04: entering candidate state: node="Node at 880f119f-8e53-996d-b554-c6db1e4ffa04 [Candidate]" term=2
2025-02-23T18:41:57.113Z [INFO]  server-880f119f-8e53-996d-b554-c6db1e4ffa04: election won: term=2 tally=2
2025-02-23T18:41:57.113Z [INFO]  server-880f119f-8e53-996d-b554-c6db1e4ffa04: entering leader state: leader="Node at 880f119f-8e53-996d-b554-c6db1e4ffa04 [Leader]"
2025-02-23T18:41:57.113Z [INFO]  server-880f119f-8e53-996d-b554-c6db1e4ffa04: added peer, starting replication: peer=server-2c2d0c4b-9dbe-85c8-d539-791e02103b51
2025-02-23T18:41:57.113Z [INFO]  server-880f119f-8e53-996d-b554-c6db1e4ffa04: added peer, starting replication: peer=server-8c9f016b-80cc-3c42-a69a-81ff1ca808c2
2025-02-23T18:41:57.113Z [INFO]  server-880f119f-8e53-996d-b554-c6db1e4ffa04: pipelining replication: peer="{Voter server-8c9f016b-80cc-3c42-a69a-81ff1ca808c2 8c9f016b-80cc-3c42-a69a-81ff1ca808c2}"
2025-02-23T18:41:57.114Z [INFO]  server-880f119f-8e53-996d-b554-c6db1e4ffa04: pipelining replication: peer="{Voter server-2c2d0c4b-9dbe-85c8-d539-791e02103b51 2c2d0c4b-9dbe-85c8-d539-791e02103b51}"
2025-02-23T18:41:58.119Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:57.11424644 +0000 UTC m=+21.412507139 (1 nodes), 528.548423ms from start of poll, 529.04762ms from cluster start. Timeout at 2025-02-23 18:41:58.119257591 +0000 UTC m=+22.417518272, 1.005011133s after stability
2025-02-23T18:41:58.120Z [INFO]  creating new snapshot: path=/tmp/raft728443306/snapshots/2-13-1740336118120.tmp
2025-02-23T18:41:58.120Z [INFO]  server-880f119f-8e53-996d-b554-c6db1e4ffa04: copied to local snapshot: bytes=71
2025-02-23T18:41:58.121Z [INFO]  server-880f119f-8e53-996d-b554-c6db1e4ffa04: snapshot restore progress: id=2-13-1740336118120 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:41:58.121Z [INFO]  server-880f119f-8e53-996d-b554-c6db1e4ffa04: restored user snapshot: index=13
2025-02-23T18:41:58.121Z [ERROR] server-880f119f-8e53-996d-b554-c6db1e4ffa04: failed to get log: index=3 error="log not found"
2025-02-23T18:41:58.121Z [INFO]  server-880f119f-8e53-996d-b554-c6db1e4ffa04: aborting pipeline replication: peer="{Voter server-8c9f016b-80cc-3c42-a69a-81ff1ca808c2 8c9f016b-80cc-3c42-a69a-81ff1ca808c2}"
2025-02-23T18:41:58.121Z [ERROR] server-880f119f-8e53-996d-b554-c6db1e4ffa04: failed to get log: index=3 error="log not found"
2025-02-23T18:41:58.121Z [INFO]  server-880f119f-8e53-996d-b554-c6db1e4ffa04: aborting pipeline replication: peer="{Voter server-2c2d0c4b-9dbe-85c8-d539-791e02103b51 2c2d0c4b-9dbe-85c8-d539-791e02103b51}"
2025-02-23T18:41:58.128Z [ERROR] server-880f119f-8e53-996d-b554-c6db1e4ffa04: failed to get log: index=3 error="log not found"
2025-02-23T18:41:58.128Z [INFO]  creating new snapshot: path=/tmp/raft1666517806/snapshots/2-13-1740336118128.tmp
2025-02-23T18:41:58.132Z [ERROR] server-880f119f-8e53-996d-b554-c6db1e4ffa04: failed to get log: index=3 error="log not found"
2025-02-23T18:41:58.132Z [INFO]  server-2c2d0c4b-9dbe-85c8-d539-791e02103b51: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:41:58.132Z [INFO]  creating new snapshot: path=/tmp/raft1725169597/snapshots/2-13-1740336118132.tmp
2025-02-23T18:41:58.132Z [INFO]  server-2c2d0c4b-9dbe-85c8-d539-791e02103b51: copied to local snapshot: bytes=71
2025-02-23T18:41:58.133Z [INFO]  server-8c9f016b-80cc-3c42-a69a-81ff1ca808c2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:41:58.133Z [INFO]  server-2c2d0c4b-9dbe-85c8-d539-791e02103b51: snapshot restore progress: id=2-13-1740336118128 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:41:58.133Z [INFO]  server-2c2d0c4b-9dbe-85c8-d539-791e02103b51: Installed remote snapshot
2025-02-23T18:41:58.133Z [INFO]  server-8c9f016b-80cc-3c42-a69a-81ff1ca808c2: copied to local snapshot: bytes=71
2025-02-23T18:41:58.133Z [INFO]  server-880f119f-8e53-996d-b554-c6db1e4ffa04: pipelining replication: peer="{Voter server-2c2d0c4b-9dbe-85c8-d539-791e02103b51 2c2d0c4b-9dbe-85c8-d539-791e02103b51}"
2025-02-23T18:41:58.133Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:41:58.134Z [INFO]  server-8c9f016b-80cc-3c42-a69a-81ff1ca808c2: snapshot restore progress: id=2-13-1740336118132 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:41:58.134Z [INFO]  server-8c9f016b-80cc-3c42-a69a-81ff1ca808c2: Installed remote snapshot
2025-02-23T18:41:58.134Z [INFO]  server-880f119f-8e53-996d-b554-c6db1e4ffa04: pipelining replication: peer="{Voter server-8c9f016b-80cc-3c42-a69a-81ff1ca808c2 8c9f016b-80cc-3c42-a69a-81ff1ca808c2}"
2025-02-23T18:41:59.138Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:41:58.133848647 +0000 UTC m=+22.432109304 (2 nodes), 2.152µs from start of poll, 1.548649785s from cluster start. Timeout at 2025-02-23 18:41:59.138868634 +0000 UTC m=+23.437129514, 1.00502021s after stability
2025-02-23T18:41:59.152Z [INFO]  server-56797196-272f-3d6d-9efe-651c7492a21a: aborting pipeline replication: peer="{Voter server-f5aa5e19-edf4-6c19-4459-a9bd0cd0e9e9 f5aa5e19-edf4-6c19-4459-a9bd0cd0e9e9}"
2025-02-23T18:41:59.152Z [INFO]  server-56797196-272f-3d6d-9efe-651c7492a21a: aborting pipeline replication: peer="{Voter server-1b70885f-536a-58bb-f394-dc0142351c9a 1b70885f-536a-58bb-f394-dc0142351c9a}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true
2025-02-23T18:41:59.154Z [INFO]  server-a24b8bca-4cde-053e-4639-229f2b89e5e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a24b8bca-4cde-053e-4639-229f2b89e5e5 Address:a24b8bca-4cde-053e-4639-229f2b89e5e5} {Suffrage:Voter ID:server-92e3934c-4d5b-84ea-72a3-241937a6cdc1 Address:92e3934c-4d5b-84ea-72a3-241937a6cdc1} {Suffrage:Voter ID:server-6897b583-56fd-b89c-8d02-b7814f6d64bb Address:6897b583-56fd-b89c-8d02-b7814f6d64bb}]"
2025-02-23T18:41:59.154Z [INFO]  server-92e3934c-4d5b-84ea-72a3-241937a6cdc1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a24b8bca-4cde-053e-4639-229f2b89e5e5 Address:a24b8bca-4cde-053e-4639-229f2b89e5e5} {Suffrage:Voter ID:server-92e3934c-4d5b-84ea-72a3-241937a6cdc1 Address:92e3934c-4d5b-84ea-72a3-241937a6cdc1} {Suffrage:Voter ID:server-6897b583-56fd-b89c-8d02-b7814f6d64bb Address:6897b583-56fd-b89c-8d02-b7814f6d64bb}]"
2025-02-23T18:41:59.154Z [INFO]  server-6897b583-56fd-b89c-8d02-b7814f6d64bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a24b8bca-4cde-053e-4639-229f2b89e5e5 Address:a24b8bca-4cde-053e-4639-229f2b89e5e5} {Suffrage:Voter ID:server-92e3934c-4d5b-84ea-72a3-241937a6cdc1 Address:92e3934c-4d5b-84ea-72a3-241937a6cdc1} {Suffrage:Voter ID:server-6897b583-56fd-b89c-8d02-b7814f6d64bb Address:6897b583-56fd-b89c-8d02-b7814f6d64bb}]"
2025-02-23T18:41:59.154Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:41:59.154Z [INFO]  server-a24b8bca-4cde-053e-4639-229f2b89e5e5: entering follower state: follower="Node at a24b8bca-4cde-053e-4639-229f2b89e5e5 [Follower]" leader-address= leader-id=
2025-02-23T18:41:59.154Z [INFO]  server-92e3934c-4d5b-84ea-72a3-241937a6cdc1: entering follower state: follower="Node at 92e3934c-4d5b-84ea-72a3-241937a6cdc1 [Follower]" leader-address= leader-id=
2025-02-23T18:41:59.154Z [INFO]  server-6897b583-56fd-b89c-8d02-b7814f6d64bb: entering follower state: follower="Node at 6897b583-56fd-b89c-8d02-b7814f6d64bb [Follower]" leader-address= leader-id=
2025-02-23T18:41:59.732Z [WARN]  server-6897b583-56fd-b89c-8d02-b7814f6d64bb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:41:59.732Z [INFO]  server-6897b583-56fd-b89c-8d02-b7814f6d64bb: entering candidate state: node="Node at 6897b583-56fd-b89c-8d02-b7814f6d64bb [Candidate]" term=2
2025-02-23T18:41:59.732Z [INFO]  server-6897b583-56fd-b89c-8d02-b7814f6d64bb: election won: term=2 tally=2
2025-02-23T18:41:59.732Z [INFO]  server-6897b583-56fd-b89c-8d02-b7814f6d64bb: entering leader state: leader="Node at 6897b583-56fd-b89c-8d02-b7814f6d64bb [Leader]"
2025-02-23T18:41:59.732Z [INFO]  server-6897b583-56fd-b89c-8d02-b7814f6d64bb: added peer, starting replication: peer=server-a24b8bca-4cde-053e-4639-229f2b89e5e5
2025-02-23T18:41:59.732Z [INFO]  server-6897b583-56fd-b89c-8d02-b7814f6d64bb: added peer, starting replication: peer=server-92e3934c-4d5b-84ea-72a3-241937a6cdc1
2025-02-23T18:41:59.732Z [INFO]  server-6897b583-56fd-b89c-8d02-b7814f6d64bb: pipelining replication: peer="{Voter server-92e3934c-4d5b-84ea-72a3-241937a6cdc1 92e3934c-4d5b-84ea-72a3-241937a6cdc1}"
2025-02-23T18:41:59.732Z [INFO]  server-6897b583-56fd-b89c-8d02-b7814f6d64bb: pipelining replication: peer="{Voter server-a24b8bca-4cde-053e-4639-229f2b89e5e5 a24b8bca-4cde-053e-4639-229f2b89e5e5}"
2025-02-23T18:42:00.738Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:41:59.732851725 +0000 UTC m=+24.031112376 (1 nodes), 578.278861ms from start of poll, 578.750331ms from cluster start. Timeout at 2025-02-23 18:42:00.737854879 +0000 UTC m=+25.036115547, 1.005003171s after stability
2025-02-23T18:42:00.739Z [INFO]  server-6897b583-56fd-b89c-8d02-b7814f6d64bb: starting snapshot up to: index=12
2025-02-23T18:42:00.739Z [INFO]  creating new snapshot: path=/tmp/raft2095836447/snapshots/2-12-1740336120739.tmp
2025-02-23T18:42:00.740Z [INFO]  server-6897b583-56fd-b89c-8d02-b7814f6d64bb: snapshot complete up to: index=12
2025-02-23T18:42:00.746Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fc714a6-10a1-f1dd-425f-52d79f21233d Address:5fc714a6-10a1-f1dd-425f-52d79f21233d} {Suffrage:Voter ID:server-e254be9b-46a6-61dc-8ede-e86e33c0f380 Address:e254be9b-46a6-61dc-8ede-e86e33c0f380} {Suffrage:Voter ID:server-d8fd34b5-c1d6-69cb-62be-c95280398b13 Address:d8fd34b5-c1d6-69cb-62be-c95280398b13}]"
2025-02-23T18:42:00.746Z [INFO]  server-e254be9b-46a6-61dc-8ede-e86e33c0f380: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fc714a6-10a1-f1dd-425f-52d79f21233d Address:5fc714a6-10a1-f1dd-425f-52d79f21233d} {Suffrage:Voter ID:server-e254be9b-46a6-61dc-8ede-e86e33c0f380 Address:e254be9b-46a6-61dc-8ede-e86e33c0f380} {Suffrage:Voter ID:server-d8fd34b5-c1d6-69cb-62be-c95280398b13 Address:d8fd34b5-c1d6-69cb-62be-c95280398b13}]"
2025-02-23T18:42:00.746Z [INFO]  server-d8fd34b5-c1d6-69cb-62be-c95280398b13: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fc714a6-10a1-f1dd-425f-52d79f21233d Address:5fc714a6-10a1-f1dd-425f-52d79f21233d} {Suffrage:Voter ID:server-e254be9b-46a6-61dc-8ede-e86e33c0f380 Address:e254be9b-46a6-61dc-8ede-e86e33c0f380} {Suffrage:Voter ID:server-d8fd34b5-c1d6-69cb-62be-c95280398b13 Address:d8fd34b5-c1d6-69cb-62be-c95280398b13}]"
2025-02-23T18:42:00.746Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:00.746Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: entering follower state: follower="Node at 5fc714a6-10a1-f1dd-425f-52d79f21233d [Follower]" leader-address= leader-id=
2025-02-23T18:42:00.746Z [INFO]  server-d8fd34b5-c1d6-69cb-62be-c95280398b13: entering follower state: follower="Node at d8fd34b5-c1d6-69cb-62be-c95280398b13 [Follower]" leader-address= leader-id=
2025-02-23T18:42:00.746Z [INFO]  server-e254be9b-46a6-61dc-8ede-e86e33c0f380: entering follower state: follower="Node at e254be9b-46a6-61dc-8ede-e86e33c0f380 [Follower]" leader-address= leader-id=
2025-02-23T18:42:01.387Z [WARN]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:01.387Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: entering candidate state: node="Node at 5fc714a6-10a1-f1dd-425f-52d79f21233d [Candidate]" term=2
2025-02-23T18:42:01.387Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: election won: term=2 tally=2
2025-02-23T18:42:01.387Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: entering leader state: leader="Node at 5fc714a6-10a1-f1dd-425f-52d79f21233d [Leader]"
2025-02-23T18:42:01.387Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: added peer, starting replication: peer=server-e254be9b-46a6-61dc-8ede-e86e33c0f380
2025-02-23T18:42:01.387Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: added peer, starting replication: peer=server-d8fd34b5-c1d6-69cb-62be-c95280398b13
2025-02-23T18:42:01.387Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: pipelining replication: peer="{Voter server-d8fd34b5-c1d6-69cb-62be-c95280398b13 d8fd34b5-c1d6-69cb-62be-c95280398b13}"
2025-02-23T18:42:01.387Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: pipelining replication: peer="{Voter server-e254be9b-46a6-61dc-8ede-e86e33c0f380 e254be9b-46a6-61dc-8ede-e86e33c0f380}"
2025-02-23T18:42:02.392Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:01.387823073 +0000 UTC m=+25.686083723 (1 nodes), 641.265377ms from start of poll, 641.686633ms from cluster start. Timeout at 2025-02-23 18:42:02.392825496 +0000 UTC m=+26.691086110, 1.005002387s after stability
2025-02-23T18:42:02.393Z [INFO]  creating new snapshot: path=/tmp/raft434895033/snapshots/2-13-1740336122393.tmp
2025-02-23T18:42:02.393Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: copied to local snapshot: bytes=71
2025-02-23T18:42:02.394Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: snapshot restore progress: id=2-13-1740336122393 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:02.394Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: removing all old logs from log store
2025-02-23T18:42:02.394Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: compacting logs: from=1 to=2
2025-02-23T18:42:02.394Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: restored user snapshot: index=13
2025-02-23T18:42:02.394Z [ERROR] server-5fc714a6-10a1-f1dd-425f-52d79f21233d: failed to get log: index=2 error="log not found"
2025-02-23T18:42:02.394Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: aborting pipeline replication: peer="{Voter server-d8fd34b5-c1d6-69cb-62be-c95280398b13 d8fd34b5-c1d6-69cb-62be-c95280398b13}"
2025-02-23T18:42:02.394Z [ERROR] server-5fc714a6-10a1-f1dd-425f-52d79f21233d: failed to get log: index=2 error="log not found"
2025-02-23T18:42:02.394Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: aborting pipeline replication: peer="{Voter server-e254be9b-46a6-61dc-8ede-e86e33c0f380 e254be9b-46a6-61dc-8ede-e86e33c0f380}"
2025-02-23T18:42:02.400Z [ERROR] server-5fc714a6-10a1-f1dd-425f-52d79f21233d: failed to get log: index=2 error="log not found"
2025-02-23T18:42:02.400Z [INFO]  creating new snapshot: path=/tmp/raft3442879516/snapshots/2-13-1740336122400.tmp
2025-02-23T18:42:02.401Z [INFO]  server-d8fd34b5-c1d6-69cb-62be-c95280398b13: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:02.401Z [INFO]  server-d8fd34b5-c1d6-69cb-62be-c95280398b13: copied to local snapshot: bytes=71
2025-02-23T18:42:02.401Z [INFO]  server-d8fd34b5-c1d6-69cb-62be-c95280398b13: snapshot restore progress: id=2-13-1740336122400 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:02.401Z [INFO]  server-d8fd34b5-c1d6-69cb-62be-c95280398b13: removing all old logs from log store
2025-02-23T18:42:02.401Z [INFO]  server-d8fd34b5-c1d6-69cb-62be-c95280398b13: compacting logs: from=1 to=2
2025-02-23T18:42:02.401Z [INFO]  server-d8fd34b5-c1d6-69cb-62be-c95280398b13: Installed remote snapshot
2025-02-23T18:42:02.402Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: pipelining replication: peer="{Voter server-d8fd34b5-c1d6-69cb-62be-c95280398b13 d8fd34b5-c1d6-69cb-62be-c95280398b13}"
2025-02-23T18:42:02.402Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:42:02.404Z [ERROR] server-5fc714a6-10a1-f1dd-425f-52d79f21233d: failed to get log: index=2 error="log not found"
2025-02-23T18:42:02.405Z [INFO]  creating new snapshot: path=/tmp/raft3599411767/snapshots/2-13-1740336122405.tmp
2025-02-23T18:42:02.405Z [INFO]  server-e254be9b-46a6-61dc-8ede-e86e33c0f380: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:02.405Z [INFO]  server-e254be9b-46a6-61dc-8ede-e86e33c0f380: copied to local snapshot: bytes=71
2025-02-23T18:42:02.406Z [INFO]  server-e254be9b-46a6-61dc-8ede-e86e33c0f380: snapshot restore progress: id=2-13-1740336122405 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:02.406Z [INFO]  server-e254be9b-46a6-61dc-8ede-e86e33c0f380: removing all old logs from log store
2025-02-23T18:42:02.406Z [INFO]  server-e254be9b-46a6-61dc-8ede-e86e33c0f380: compacting logs: from=1 to=2
2025-02-23T18:42:02.406Z [INFO]  server-e254be9b-46a6-61dc-8ede-e86e33c0f380: Installed remote snapshot
2025-02-23T18:42:02.406Z [INFO]  server-5fc714a6-10a1-f1dd-425f-52d79f21233d: pipelining replication: peer="{Voter server-e254be9b-46a6-61dc-8ede-e86e33c0f380 e254be9b-46a6-61dc-8ede-e86e33c0f380}"
2025-02-23T18:42:03.407Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:42:02.402276809 +0000 UTC m=+26.700537435 (2 nodes), 1.89µs from start of poll, 1.656140345s from cluster start. Timeout at 2025-02-23 18:42:03.407299665 +0000 UTC m=+27.705560548, 1.005023113s after stability
2025-02-23T18:42:03.421Z [INFO]  server-6897b583-56fd-b89c-8d02-b7814f6d64bb: aborting pipeline replication: peer="{Voter server-92e3934c-4d5b-84ea-72a3-241937a6cdc1 92e3934c-4d5b-84ea-72a3-241937a6cdc1}"
2025-02-23T18:42:03.421Z [INFO]  server-6897b583-56fd-b89c-8d02-b7814f6d64bb: aborting pipeline replication: peer="{Voter server-a24b8bca-4cde-053e-4639-229f2b89e5e5 a24b8bca-4cde-053e-4639-229f2b89e5e5}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false
2025-02-23T18:42:03.423Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762 Address:2a8d45d1-15ce-6bb9-52f6-6fb50e15c762} {Suffrage:Voter ID:server-03f14044-607f-119d-8a65-239847d6d0bf Address:03f14044-607f-119d-8a65-239847d6d0bf} {Suffrage:Voter ID:server-2b012cfd-feb0-3519-ce9c-c791763cde0f Address:2b012cfd-feb0-3519-ce9c-c791763cde0f}]"
2025-02-23T18:42:03.423Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: entering follower state: follower="Node at 2a8d45d1-15ce-6bb9-52f6-6fb50e15c762 [Follower]" leader-address= leader-id=
2025-02-23T18:42:03.423Z [INFO]  server-03f14044-607f-119d-8a65-239847d6d0bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762 Address:2a8d45d1-15ce-6bb9-52f6-6fb50e15c762} {Suffrage:Voter ID:server-03f14044-607f-119d-8a65-239847d6d0bf Address:03f14044-607f-119d-8a65-239847d6d0bf} {Suffrage:Voter ID:server-2b012cfd-feb0-3519-ce9c-c791763cde0f Address:2b012cfd-feb0-3519-ce9c-c791763cde0f}]"
2025-02-23T18:42:03.423Z [INFO]  server-03f14044-607f-119d-8a65-239847d6d0bf: entering follower state: follower="Node at 03f14044-607f-119d-8a65-239847d6d0bf [Follower]" leader-address= leader-id=
2025-02-23T18:42:03.423Z [INFO]  server-2b012cfd-feb0-3519-ce9c-c791763cde0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762 Address:2a8d45d1-15ce-6bb9-52f6-6fb50e15c762} {Suffrage:Voter ID:server-03f14044-607f-119d-8a65-239847d6d0bf Address:03f14044-607f-119d-8a65-239847d6d0bf} {Suffrage:Voter ID:server-2b012cfd-feb0-3519-ce9c-c791763cde0f Address:2b012cfd-feb0-3519-ce9c-c791763cde0f}]"
2025-02-23T18:42:03.423Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:03.423Z [INFO]  server-2b012cfd-feb0-3519-ce9c-c791763cde0f: entering follower state: follower="Node at 2b012cfd-feb0-3519-ce9c-c791763cde0f [Follower]" leader-address= leader-id=
2025-02-23T18:42:03.993Z [WARN]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:03.993Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: entering candidate state: node="Node at 2a8d45d1-15ce-6bb9-52f6-6fb50e15c762 [Candidate]" term=2
2025-02-23T18:42:03.993Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: election won: term=2 tally=2
2025-02-23T18:42:03.993Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: entering leader state: leader="Node at 2a8d45d1-15ce-6bb9-52f6-6fb50e15c762 [Leader]"
2025-02-23T18:42:03.993Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: added peer, starting replication: peer=server-03f14044-607f-119d-8a65-239847d6d0bf
2025-02-23T18:42:03.993Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: added peer, starting replication: peer=server-2b012cfd-feb0-3519-ce9c-c791763cde0f
2025-02-23T18:42:03.993Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: pipelining replication: peer="{Voter server-2b012cfd-feb0-3519-ce9c-c791763cde0f 2b012cfd-feb0-3519-ce9c-c791763cde0f}"
2025-02-23T18:42:03.994Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: pipelining replication: peer="{Voter server-03f14044-607f-119d-8a65-239847d6d0bf 03f14044-607f-119d-8a65-239847d6d0bf}"
2025-02-23T18:42:04.999Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:03.994189755 +0000 UTC m=+28.292450487 (1 nodes), 570.38205ms from start of poll, 570.839159ms from cluster start. Timeout at 2025-02-23 18:42:04.999193132 +0000 UTC m=+29.297453818, 1.005003331s after stability
2025-02-23T18:42:05.001Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: starting snapshot up to: index=12
2025-02-23T18:42:05.001Z [INFO]  creating new snapshot: path=/tmp/raft3659840133/snapshots/2-12-1740336125001.tmp
2025-02-23T18:42:05.018Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: snapshot complete up to: index=12
2025-02-23T18:42:05.020Z [INFO]  creating new snapshot: path=/tmp/raft3659840133/snapshots/2-23-1740336125020.tmp
2025-02-23T18:42:05.020Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: copied to local snapshot: bytes=71
2025-02-23T18:42:05.021Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: snapshot restore progress: id=2-23-1740336125020 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:05.021Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: restored user snapshot: index=23
2025-02-23T18:42:05.021Z [ERROR] server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: failed to get log: index=23 error="log not found"
2025-02-23T18:42:05.021Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: aborting pipeline replication: peer="{Voter server-2b012cfd-feb0-3519-ce9c-c791763cde0f 2b012cfd-feb0-3519-ce9c-c791763cde0f}"
2025-02-23T18:42:05.021Z [ERROR] server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: failed to get log: index=23 error="log not found"
2025-02-23T18:42:05.021Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: aborting pipeline replication: peer="{Voter server-03f14044-607f-119d-8a65-239847d6d0bf 03f14044-607f-119d-8a65-239847d6d0bf}"
2025-02-23T18:42:05.028Z [ERROR] server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: failed to get log: index=23 error="log not found"
2025-02-23T18:42:05.028Z [INFO]  creating new snapshot: path=/tmp/raft1704722915/snapshots/2-23-1740336125028.tmp
2025-02-23T18:42:05.029Z [INFO]  server-03f14044-607f-119d-8a65-239847d6d0bf: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:05.029Z [INFO]  server-03f14044-607f-119d-8a65-239847d6d0bf: copied to local snapshot: bytes=71
2025-02-23T18:42:05.033Z [ERROR] server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: failed to get log: index=23 error="log not found"
2025-02-23T18:42:05.034Z [INFO]  server-03f14044-607f-119d-8a65-239847d6d0bf: snapshot restore progress: id=2-23-1740336125028 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:05.034Z [INFO]  server-03f14044-607f-119d-8a65-239847d6d0bf: Installed remote snapshot
2025-02-23T18:42:05.034Z [INFO]  creating new snapshot: path=/tmp/raft2402578931/snapshots/2-23-1740336125034.tmp
2025-02-23T18:42:05.035Z [INFO]  server-2b012cfd-feb0-3519-ce9c-c791763cde0f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:05.037Z [INFO]  server-2b012cfd-feb0-3519-ce9c-c791763cde0f: copied to local snapshot: bytes=71
2025-02-23T18:42:05.037Z [INFO]  server-2b012cfd-feb0-3519-ce9c-c791763cde0f: snapshot restore progress: id=2-23-1740336125034 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:05.037Z [INFO]  server-2b012cfd-feb0-3519-ce9c-c791763cde0f: Installed remote snapshot
2025-02-23T18:42:05.038Z [WARN]  server-2b012cfd-feb0-3519-ce9c-c791763cde0f: skipping application of old log: index=22
2025-02-23T18:42:05.038Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: pipelining replication: peer="{Voter server-2b012cfd-feb0-3519-ce9c-c791763cde0f 2b012cfd-feb0-3519-ce9c-c791763cde0f}"
2025-02-23T18:42:05.038Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:42:05.041Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: pipelining replication: peer="{Voter server-03f14044-607f-119d-8a65-239847d6d0bf 03f14044-607f-119d-8a65-239847d6d0bf}"
2025-02-23T18:42:06.045Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:42:05.038191701 +0000 UTC m=+29.336452373 (2 nodes), 2.163µs from start of poll, 1.614841045s from cluster start. Timeout at 2025-02-23 18:42:06.043210629 +0000 UTC m=+30.341471279, 1.005018906s after stability
2025-02-23T18:42:06.059Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: aborting pipeline replication: peer="{Voter server-03f14044-607f-119d-8a65-239847d6d0bf 03f14044-607f-119d-8a65-239847d6d0bf}"
2025-02-23T18:42:06.059Z [INFO]  server-2a8d45d1-15ce-6bb9-52f6-6fb50e15c762: aborting pipeline replication: peer="{Voter server-2b012cfd-feb0-3519-ce9c-c791763cde0f 2b012cfd-feb0-3519-ce9c-c791763cde0f}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false
2025-02-23T18:42:06.061Z [INFO]  server-04fb88bb-6f85-7ff2-d159-fbb089a46244: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04fb88bb-6f85-7ff2-d159-fbb089a46244 Address:04fb88bb-6f85-7ff2-d159-fbb089a46244} {Suffrage:Voter ID:server-57ce8a54-481d-b635-6409-a7e64d364863 Address:57ce8a54-481d-b635-6409-a7e64d364863} {Suffrage:Voter ID:server-fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56 Address:fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56}]"
2025-02-23T18:42:06.061Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04fb88bb-6f85-7ff2-d159-fbb089a46244 Address:04fb88bb-6f85-7ff2-d159-fbb089a46244} {Suffrage:Voter ID:server-57ce8a54-481d-b635-6409-a7e64d364863 Address:57ce8a54-481d-b635-6409-a7e64d364863} {Suffrage:Voter ID:server-fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56 Address:fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56}]"
2025-02-23T18:42:06.061Z [INFO]  server-04fb88bb-6f85-7ff2-d159-fbb089a46244: entering follower state: follower="Node at 04fb88bb-6f85-7ff2-d159-fbb089a46244 [Follower]" leader-address= leader-id=
2025-02-23T18:42:06.061Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: entering follower state: follower="Node at 57ce8a54-481d-b635-6409-a7e64d364863 [Follower]" leader-address= leader-id=
2025-02-23T18:42:06.061Z [INFO]  server-fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04fb88bb-6f85-7ff2-d159-fbb089a46244 Address:04fb88bb-6f85-7ff2-d159-fbb089a46244} {Suffrage:Voter ID:server-57ce8a54-481d-b635-6409-a7e64d364863 Address:57ce8a54-481d-b635-6409-a7e64d364863} {Suffrage:Voter ID:server-fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56 Address:fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56}]"
2025-02-23T18:42:06.061Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:06.061Z [INFO]  server-fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56: entering follower state: follower="Node at fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56 [Follower]" leader-address= leader-id=
2025-02-23T18:42:06.565Z [WARN]  server-57ce8a54-481d-b635-6409-a7e64d364863: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:06.566Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: entering candidate state: node="Node at 57ce8a54-481d-b635-6409-a7e64d364863 [Candidate]" term=2
2025-02-23T18:42:06.566Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: election won: term=2 tally=2
2025-02-23T18:42:06.566Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: entering leader state: leader="Node at 57ce8a54-481d-b635-6409-a7e64d364863 [Leader]"
2025-02-23T18:42:06.566Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: added peer, starting replication: peer=server-04fb88bb-6f85-7ff2-d159-fbb089a46244
2025-02-23T18:42:06.566Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: added peer, starting replication: peer=server-fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56
2025-02-23T18:42:06.566Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: pipelining replication: peer="{Voter server-04fb88bb-6f85-7ff2-d159-fbb089a46244 04fb88bb-6f85-7ff2-d159-fbb089a46244}"
2025-02-23T18:42:06.566Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: pipelining replication: peer="{Voter server-fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56 fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56}"
2025-02-23T18:42:07.571Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:06.566278462 +0000 UTC m=+30.864539346 (1 nodes), 504.34932ms from start of poll, 504.781245ms from cluster start. Timeout at 2025-02-23 18:42:07.571281836 +0000 UTC m=+31.869542525, 1.005003179s after stability
2025-02-23T18:42:07.573Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: starting snapshot up to: index=12
2025-02-23T18:42:07.573Z [INFO]  creating new snapshot: path=/tmp/raft169546359/snapshots/2-12-1740336127573.tmp
2025-02-23T18:42:07.573Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: snapshot complete up to: index=12
2025-02-23T18:42:07.575Z [INFO]  creating new snapshot: path=/tmp/raft169546359/snapshots/2-23-1740336127575.tmp
2025-02-23T18:42:07.576Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: copied to local snapshot: bytes=71
2025-02-23T18:42:07.576Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: snapshot restore progress: id=2-23-1740336127575 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:07.576Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: removing all old logs from log store
2025-02-23T18:42:07.576Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: compacting logs: from=1 to=22
2025-02-23T18:42:07.576Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: restored user snapshot: index=23
2025-02-23T18:42:07.576Z [ERROR] server-57ce8a54-481d-b635-6409-a7e64d364863: failed to get log: index=22 error="log not found"
2025-02-23T18:42:07.576Z [ERROR] server-57ce8a54-481d-b635-6409-a7e64d364863: failed to get log: index=22 error="log not found"
2025-02-23T18:42:07.576Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: aborting pipeline replication: peer="{Voter server-fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56 fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56}"
2025-02-23T18:42:07.576Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: aborting pipeline replication: peer="{Voter server-04fb88bb-6f85-7ff2-d159-fbb089a46244 04fb88bb-6f85-7ff2-d159-fbb089a46244}"
2025-02-23T18:42:07.586Z [ERROR] server-57ce8a54-481d-b635-6409-a7e64d364863: failed to get log: index=22 error="log not found"
2025-02-23T18:42:07.586Z [ERROR] server-57ce8a54-481d-b635-6409-a7e64d364863: failed to get log: index=22 error="log not found"
2025-02-23T18:42:07.586Z [INFO]  creating new snapshot: path=/tmp/raft2442619074/snapshots/2-23-1740336127586.tmp
2025-02-23T18:42:07.586Z [INFO]  creating new snapshot: path=/tmp/raft2245858688/snapshots/2-23-1740336127586.tmp
2025-02-23T18:42:07.586Z [INFO]  server-fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:07.587Z [INFO]  server-04fb88bb-6f85-7ff2-d159-fbb089a46244: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:07.587Z [INFO]  server-fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56: copied to local snapshot: bytes=71
2025-02-23T18:42:07.587Z [INFO]  server-04fb88bb-6f85-7ff2-d159-fbb089a46244: copied to local snapshot: bytes=71
2025-02-23T18:42:07.587Z [INFO]  server-fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56: snapshot restore progress: id=2-23-1740336127586 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:07.590Z [INFO]  server-fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56: removing all old logs from log store
2025-02-23T18:42:07.590Z [INFO]  server-fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56: compacting logs: from=1 to=22
2025-02-23T18:42:07.590Z [INFO]  server-fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56: Installed remote snapshot
2025-02-23T18:42:07.590Z [WARN]  server-fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56: skipping application of old log: index=22
2025-02-23T18:42:07.590Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: pipelining replication: peer="{Voter server-fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56 fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56}"
2025-02-23T18:42:07.593Z [INFO]  server-04fb88bb-6f85-7ff2-d159-fbb089a46244: snapshot restore progress: id=2-23-1740336127586 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:07.593Z [INFO]  server-04fb88bb-6f85-7ff2-d159-fbb089a46244: removing all old logs from log store
2025-02-23T18:42:07.593Z [INFO]  server-04fb88bb-6f85-7ff2-d159-fbb089a46244: compacting logs: from=1 to=22
2025-02-23T18:42:07.593Z [INFO]  server-04fb88bb-6f85-7ff2-d159-fbb089a46244: Installed remote snapshot
2025-02-23T18:42:07.593Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: pipelining replication: peer="{Voter server-04fb88bb-6f85-7ff2-d159-fbb089a46244 04fb88bb-6f85-7ff2-d159-fbb089a46244}"
2025-02-23T18:42:07.597Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:42:08.603Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:42:07.597722721 +0000 UTC m=+31.895983564 (2 nodes), 3.95µs from start of poll, 1.536225463s from cluster start. Timeout at 2025-02-23 18:42:08.602751265 +0000 UTC m=+32.901011917, 1.005028353s after stability
2025-02-23T18:42:08.616Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: aborting pipeline replication: peer="{Voter server-fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56 fc009fe3-cfe4-eea9-48e7-08ac3d4a0d56}"
2025-02-23T18:42:08.617Z [INFO]  server-57ce8a54-481d-b635-6409-a7e64d364863: aborting pipeline replication: peer="{Voter server-04fb88bb-6f85-7ff2-d159-fbb089a46244 04fb88bb-6f85-7ff2-d159-fbb089a46244}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true
2025-02-23T18:42:08.619Z [INFO]  server-93c0ffed-8236-36fe-5d24-754aa2f023fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93c0ffed-8236-36fe-5d24-754aa2f023fa Address:93c0ffed-8236-36fe-5d24-754aa2f023fa} {Suffrage:Voter ID:server-09178b67-b07e-1af5-160d-bd7aaa1cc4df Address:09178b67-b07e-1af5-160d-bd7aaa1cc4df} {Suffrage:Voter ID:server-2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e Address:2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e}]"
2025-02-23T18:42:08.619Z [INFO]  server-09178b67-b07e-1af5-160d-bd7aaa1cc4df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93c0ffed-8236-36fe-5d24-754aa2f023fa Address:93c0ffed-8236-36fe-5d24-754aa2f023fa} {Suffrage:Voter ID:server-09178b67-b07e-1af5-160d-bd7aaa1cc4df Address:09178b67-b07e-1af5-160d-bd7aaa1cc4df} {Suffrage:Voter ID:server-2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e Address:2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e}]"
2025-02-23T18:42:08.619Z [INFO]  server-2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93c0ffed-8236-36fe-5d24-754aa2f023fa Address:93c0ffed-8236-36fe-5d24-754aa2f023fa} {Suffrage:Voter ID:server-09178b67-b07e-1af5-160d-bd7aaa1cc4df Address:09178b67-b07e-1af5-160d-bd7aaa1cc4df} {Suffrage:Voter ID:server-2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e Address:2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e}]"
2025-02-23T18:42:08.619Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:08.619Z [INFO]  server-93c0ffed-8236-36fe-5d24-754aa2f023fa: entering follower state: follower="Node at 93c0ffed-8236-36fe-5d24-754aa2f023fa [Follower]" leader-address= leader-id=
2025-02-23T18:42:08.619Z [INFO]  server-09178b67-b07e-1af5-160d-bd7aaa1cc4df: entering follower state: follower="Node at 09178b67-b07e-1af5-160d-bd7aaa1cc4df [Follower]" leader-address= leader-id=
2025-02-23T18:42:08.619Z [INFO]  server-2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e: entering follower state: follower="Node at 2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e [Follower]" leader-address= leader-id=
2025-02-23T18:42:09.172Z [WARN]  server-2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:09.172Z [INFO]  server-2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e: entering candidate state: node="Node at 2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e [Candidate]" term=2
2025-02-23T18:42:09.172Z [INFO]  server-2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e: election won: term=2 tally=2
2025-02-23T18:42:09.172Z [INFO]  server-2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e: entering leader state: leader="Node at 2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e [Leader]"
2025-02-23T18:42:09.172Z [INFO]  server-2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e: added peer, starting replication: peer=server-93c0ffed-8236-36fe-5d24-754aa2f023fa
2025-02-23T18:42:09.172Z [INFO]  server-2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e: added peer, starting replication: peer=server-09178b67-b07e-1af5-160d-bd7aaa1cc4df
2025-02-23T18:42:09.173Z [INFO]  server-2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e: pipelining replication: peer="{Voter server-09178b67-b07e-1af5-160d-bd7aaa1cc4df 09178b67-b07e-1af5-160d-bd7aaa1cc4df}"
2025-02-23T18:42:09.173Z [INFO]  server-2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e: pipelining replication: peer="{Voter server-93c0ffed-8236-36fe-5d24-754aa2f023fa 93c0ffed-8236-36fe-5d24-754aa2f023fa}"
2025-02-23T18:42:10.179Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:09.173532584 +0000 UTC m=+33.471793237 (1 nodes), 554.172108ms from start of poll, 554.558837ms from cluster start. Timeout at 2025-02-23 18:42:10.178536048 +0000 UTC m=+34.476796865, 1.005003628s after stability
2025-02-23T18:42:10.180Z [INFO]  server-2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e: starting snapshot up to: index=12
2025-02-23T18:42:10.180Z [INFO]  creating new snapshot: path=/tmp/raft1238489810/snapshots/2-12-1740336130180.tmp
2025-02-23T18:42:10.230Z [INFO]  server-2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e: snapshot complete up to: index=12
2025-02-23T18:42:10.232Z [INFO]  server-a52887cc-6440-696b-cce4-07786fd59db5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a52887cc-6440-696b-cce4-07786fd59db5 Address:a52887cc-6440-696b-cce4-07786fd59db5} {Suffrage:Voter ID:server-24021a42-2ec0-a481-76dd-fa6c893a57ce Address:24021a42-2ec0-a481-76dd-fa6c893a57ce} {Suffrage:Voter ID:server-647713b5-08ae-777d-76c8-4ff71e32cb62 Address:647713b5-08ae-777d-76c8-4ff71e32cb62}]"
2025-02-23T18:42:10.232Z [INFO]  server-24021a42-2ec0-a481-76dd-fa6c893a57ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a52887cc-6440-696b-cce4-07786fd59db5 Address:a52887cc-6440-696b-cce4-07786fd59db5} {Suffrage:Voter ID:server-24021a42-2ec0-a481-76dd-fa6c893a57ce Address:24021a42-2ec0-a481-76dd-fa6c893a57ce} {Suffrage:Voter ID:server-647713b5-08ae-777d-76c8-4ff71e32cb62 Address:647713b5-08ae-777d-76c8-4ff71e32cb62}]"
2025-02-23T18:42:10.232Z [INFO]  server-a52887cc-6440-696b-cce4-07786fd59db5: entering follower state: follower="Node at a52887cc-6440-696b-cce4-07786fd59db5 [Follower]" leader-address= leader-id=
2025-02-23T18:42:10.232Z [INFO]  server-24021a42-2ec0-a481-76dd-fa6c893a57ce: entering follower state: follower="Node at 24021a42-2ec0-a481-76dd-fa6c893a57ce [Follower]" leader-address= leader-id=
2025-02-23T18:42:10.232Z [INFO]  server-647713b5-08ae-777d-76c8-4ff71e32cb62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a52887cc-6440-696b-cce4-07786fd59db5 Address:a52887cc-6440-696b-cce4-07786fd59db5} {Suffrage:Voter ID:server-24021a42-2ec0-a481-76dd-fa6c893a57ce Address:24021a42-2ec0-a481-76dd-fa6c893a57ce} {Suffrage:Voter ID:server-647713b5-08ae-777d-76c8-4ff71e32cb62 Address:647713b5-08ae-777d-76c8-4ff71e32cb62}]"
2025-02-23T18:42:10.232Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:10.232Z [INFO]  server-647713b5-08ae-777d-76c8-4ff71e32cb62: entering follower state: follower="Node at 647713b5-08ae-777d-76c8-4ff71e32cb62 [Follower]" leader-address= leader-id=
2025-02-23T18:42:10.869Z [WARN]  server-24021a42-2ec0-a481-76dd-fa6c893a57ce: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:10.870Z [INFO]  server-24021a42-2ec0-a481-76dd-fa6c893a57ce: entering candidate state: node="Node at 24021a42-2ec0-a481-76dd-fa6c893a57ce [Candidate]" term=2
2025-02-23T18:42:10.870Z [INFO]  server-24021a42-2ec0-a481-76dd-fa6c893a57ce: election won: term=2 tally=2
2025-02-23T18:42:10.870Z [INFO]  server-24021a42-2ec0-a481-76dd-fa6c893a57ce: entering leader state: leader="Node at 24021a42-2ec0-a481-76dd-fa6c893a57ce [Leader]"
2025-02-23T18:42:10.870Z [INFO]  server-24021a42-2ec0-a481-76dd-fa6c893a57ce: added peer, starting replication: peer=server-a52887cc-6440-696b-cce4-07786fd59db5
2025-02-23T18:42:10.870Z [INFO]  server-24021a42-2ec0-a481-76dd-fa6c893a57ce: added peer, starting replication: peer=server-647713b5-08ae-777d-76c8-4ff71e32cb62
2025-02-23T18:42:10.870Z [INFO]  server-24021a42-2ec0-a481-76dd-fa6c893a57ce: pipelining replication: peer="{Voter server-a52887cc-6440-696b-cce4-07786fd59db5 a52887cc-6440-696b-cce4-07786fd59db5}"
2025-02-23T18:42:10.870Z [INFO]  server-24021a42-2ec0-a481-76dd-fa6c893a57ce: pipelining replication: peer="{Voter server-647713b5-08ae-777d-76c8-4ff71e32cb62 647713b5-08ae-777d-76c8-4ff71e32cb62}"
2025-02-23T18:42:11.876Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:10.870284103 +0000 UTC m=+35.168545136 (1 nodes), 637.42923ms from start of poll, 637.921094ms from cluster start. Timeout at 2025-02-23 18:42:11.875287749 +0000 UTC m=+36.173548514, 1.005003378s after stability
2025-02-23T18:42:11.879Z [INFO]  creating new snapshot: path=/tmp/raft3964543154/snapshots/2-14-1740336131879.tmp
2025-02-23T18:42:11.880Z [INFO]  server-24021a42-2ec0-a481-76dd-fa6c893a57ce: copied to local snapshot: bytes=71
2025-02-23T18:42:11.881Z [INFO]  server-24021a42-2ec0-a481-76dd-fa6c893a57ce: snapshot restore progress: id=2-14-1740336131879 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:11.881Z [INFO]  server-24021a42-2ec0-a481-76dd-fa6c893a57ce: restored user snapshot: index=14
2025-02-23T18:42:11.881Z [ERROR] server-24021a42-2ec0-a481-76dd-fa6c893a57ce: failed to get log: index=3 error="log not found"
2025-02-23T18:42:11.881Z [INFO]  server-24021a42-2ec0-a481-76dd-fa6c893a57ce: aborting pipeline replication: peer="{Voter server-647713b5-08ae-777d-76c8-4ff71e32cb62 647713b5-08ae-777d-76c8-4ff71e32cb62}"
2025-02-23T18:42:11.881Z [ERROR] server-24021a42-2ec0-a481-76dd-fa6c893a57ce: failed to get log: index=3 error="log not found"
2025-02-23T18:42:11.881Z [INFO]  server-24021a42-2ec0-a481-76dd-fa6c893a57ce: aborting pipeline replication: peer="{Voter server-a52887cc-6440-696b-cce4-07786fd59db5 a52887cc-6440-696b-cce4-07786fd59db5}"
2025-02-23T18:42:11.889Z [ERROR] server-24021a42-2ec0-a481-76dd-fa6c893a57ce: failed to get log: index=3 error="log not found"
2025-02-23T18:42:11.889Z [INFO]  creating new snapshot: path=/tmp/raft146151262/snapshots/2-14-1740336131889.tmp
2025-02-23T18:42:11.890Z [INFO]  server-647713b5-08ae-777d-76c8-4ff71e32cb62: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:11.890Z [INFO]  server-647713b5-08ae-777d-76c8-4ff71e32cb62: copied to local snapshot: bytes=71
2025-02-23T18:42:11.890Z [INFO]  server-647713b5-08ae-777d-76c8-4ff71e32cb62: snapshot restore progress: id=2-14-1740336131889 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:11.890Z [INFO]  server-647713b5-08ae-777d-76c8-4ff71e32cb62: Installed remote snapshot
2025-02-23T18:42:11.891Z [INFO]  server-24021a42-2ec0-a481-76dd-fa6c893a57ce: pipelining replication: peer="{Voter server-647713b5-08ae-777d-76c8-4ff71e32cb62 647713b5-08ae-777d-76c8-4ff71e32cb62}"
2025-02-23T18:42:11.891Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:42:11.891Z [ERROR] server-24021a42-2ec0-a481-76dd-fa6c893a57ce: failed to get log: index=3 error="log not found"
2025-02-23T18:42:11.891Z [INFO]  creating new snapshot: path=/tmp/raft2189223582/snapshots/2-14-1740336131891.tmp
2025-02-23T18:42:11.892Z [INFO]  server-a52887cc-6440-696b-cce4-07786fd59db5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:11.892Z [INFO]  server-a52887cc-6440-696b-cce4-07786fd59db5: copied to local snapshot: bytes=71
2025-02-23T18:42:11.892Z [INFO]  server-a52887cc-6440-696b-cce4-07786fd59db5: snapshot restore progress: id=2-14-1740336131891 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:11.893Z [INFO]  server-a52887cc-6440-696b-cce4-07786fd59db5: Installed remote snapshot
2025-02-23T18:42:11.893Z [INFO]  server-24021a42-2ec0-a481-76dd-fa6c893a57ce: pipelining replication: peer="{Voter server-a52887cc-6440-696b-cce4-07786fd59db5 a52887cc-6440-696b-cce4-07786fd59db5}"
2025-02-23T18:42:12.896Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:42:11.891299966 +0000 UTC m=+36.189560598 (2 nodes), 2.088µs from start of poll, 1.658936556s from cluster start. Timeout at 2025-02-23 18:42:12.896317003 +0000 UTC m=+37.194577730, 1.005017132s after stability
2025-02-23T18:42:12.909Z [INFO]  server-2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e: aborting pipeline replication: peer="{Voter server-09178b67-b07e-1af5-160d-bd7aaa1cc4df 09178b67-b07e-1af5-160d-bd7aaa1cc4df}"
2025-02-23T18:42:12.909Z [INFO]  server-2b5f37b0-a6d0-dfe2-c2b8-56b5e76efb5e: aborting pipeline replication: peer="{Voter server-93c0ffed-8236-36fe-5d24-754aa2f023fa 93c0ffed-8236-36fe-5d24-754aa2f023fa}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true
2025-02-23T18:42:12.915Z [INFO]  server-ee237e21-5258-4281-4f7f-1df8d0b96a1c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee237e21-5258-4281-4f7f-1df8d0b96a1c Address:ee237e21-5258-4281-4f7f-1df8d0b96a1c} {Suffrage:Voter ID:server-3cca2d57-0d09-3155-0319-33f6fe200d85 Address:3cca2d57-0d09-3155-0319-33f6fe200d85} {Suffrage:Voter ID:server-dd906e14-9889-54f1-aab6-afeb25fea148 Address:dd906e14-9889-54f1-aab6-afeb25fea148}]"
2025-02-23T18:42:12.915Z [INFO]  server-ee237e21-5258-4281-4f7f-1df8d0b96a1c: entering follower state: follower="Node at ee237e21-5258-4281-4f7f-1df8d0b96a1c [Follower]" leader-address= leader-id=
2025-02-23T18:42:12.915Z [INFO]  server-3cca2d57-0d09-3155-0319-33f6fe200d85: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee237e21-5258-4281-4f7f-1df8d0b96a1c Address:ee237e21-5258-4281-4f7f-1df8d0b96a1c} {Suffrage:Voter ID:server-3cca2d57-0d09-3155-0319-33f6fe200d85 Address:3cca2d57-0d09-3155-0319-33f6fe200d85} {Suffrage:Voter ID:server-dd906e14-9889-54f1-aab6-afeb25fea148 Address:dd906e14-9889-54f1-aab6-afeb25fea148}]"
2025-02-23T18:42:12.915Z [INFO]  server-3cca2d57-0d09-3155-0319-33f6fe200d85: entering follower state: follower="Node at 3cca2d57-0d09-3155-0319-33f6fe200d85 [Follower]" leader-address= leader-id=
2025-02-23T18:42:12.915Z [INFO]  server-dd906e14-9889-54f1-aab6-afeb25fea148: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee237e21-5258-4281-4f7f-1df8d0b96a1c Address:ee237e21-5258-4281-4f7f-1df8d0b96a1c} {Suffrage:Voter ID:server-3cca2d57-0d09-3155-0319-33f6fe200d85 Address:3cca2d57-0d09-3155-0319-33f6fe200d85} {Suffrage:Voter ID:server-dd906e14-9889-54f1-aab6-afeb25fea148 Address:dd906e14-9889-54f1-aab6-afeb25fea148}]"
2025-02-23T18:42:12.915Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:12.915Z [INFO]  server-dd906e14-9889-54f1-aab6-afeb25fea148: entering follower state: follower="Node at dd906e14-9889-54f1-aab6-afeb25fea148 [Follower]" leader-address= leader-id=
2025-02-23T18:42:13.493Z [WARN]  server-dd906e14-9889-54f1-aab6-afeb25fea148: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:13.493Z [INFO]  server-dd906e14-9889-54f1-aab6-afeb25fea148: entering candidate state: node="Node at dd906e14-9889-54f1-aab6-afeb25fea148 [Candidate]" term=2
2025-02-23T18:42:13.493Z [INFO]  server-dd906e14-9889-54f1-aab6-afeb25fea148: election won: term=2 tally=2
2025-02-23T18:42:13.493Z [INFO]  server-dd906e14-9889-54f1-aab6-afeb25fea148: entering leader state: leader="Node at dd906e14-9889-54f1-aab6-afeb25fea148 [Leader]"
2025-02-23T18:42:13.493Z [INFO]  server-dd906e14-9889-54f1-aab6-afeb25fea148: added peer, starting replication: peer=server-ee237e21-5258-4281-4f7f-1df8d0b96a1c
2025-02-23T18:42:13.493Z [INFO]  server-dd906e14-9889-54f1-aab6-afeb25fea148: added peer, starting replication: peer=server-3cca2d57-0d09-3155-0319-33f6fe200d85
2025-02-23T18:42:13.494Z [INFO]  server-dd906e14-9889-54f1-aab6-afeb25fea148: pipelining replication: peer="{Voter server-ee237e21-5258-4281-4f7f-1df8d0b96a1c ee237e21-5258-4281-4f7f-1df8d0b96a1c}"
2025-02-23T18:42:13.494Z [INFO]  server-dd906e14-9889-54f1-aab6-afeb25fea148: pipelining replication: peer="{Voter server-3cca2d57-0d09-3155-0319-33f6fe200d85 3cca2d57-0d09-3155-0319-33f6fe200d85}"
2025-02-23T18:42:14.500Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:13.494563295 +0000 UTC m=+37.792823931 (1 nodes), 579.014567ms from start of poll, 579.771395ms from cluster start. Timeout at 2025-02-23 18:42:14.499568934 +0000 UTC m=+38.797829667, 1.005005736s after stability
2025-02-23T18:42:14.501Z [INFO]  server-dd906e14-9889-54f1-aab6-afeb25fea148: starting snapshot up to: index=12
2025-02-23T18:42:14.501Z [INFO]  creating new snapshot: path=/tmp/raft3453120667/snapshots/2-12-1740336134501.tmp
2025-02-23T18:42:14.502Z [INFO]  server-dd906e14-9889-54f1-aab6-afeb25fea148: snapshot complete up to: index=12
2025-02-23T18:42:14.504Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67ff1859-728e-11a0-f959-c64c5b187cc0 Address:67ff1859-728e-11a0-f959-c64c5b187cc0} {Suffrage:Voter ID:server-6e3c02e2-6018-6ff2-e36c-b1ec0afd9974 Address:6e3c02e2-6018-6ff2-e36c-b1ec0afd9974} {Suffrage:Voter ID:server-27760def-e4ae-ad0e-25a0-e32b65644ce8 Address:27760def-e4ae-ad0e-25a0-e32b65644ce8}]"
2025-02-23T18:42:14.504Z [INFO]  server-6e3c02e2-6018-6ff2-e36c-b1ec0afd9974: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67ff1859-728e-11a0-f959-c64c5b187cc0 Address:67ff1859-728e-11a0-f959-c64c5b187cc0} {Suffrage:Voter ID:server-6e3c02e2-6018-6ff2-e36c-b1ec0afd9974 Address:6e3c02e2-6018-6ff2-e36c-b1ec0afd9974} {Suffrage:Voter ID:server-27760def-e4ae-ad0e-25a0-e32b65644ce8 Address:27760def-e4ae-ad0e-25a0-e32b65644ce8}]"
2025-02-23T18:42:14.504Z [INFO]  server-27760def-e4ae-ad0e-25a0-e32b65644ce8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67ff1859-728e-11a0-f959-c64c5b187cc0 Address:67ff1859-728e-11a0-f959-c64c5b187cc0} {Suffrage:Voter ID:server-6e3c02e2-6018-6ff2-e36c-b1ec0afd9974 Address:6e3c02e2-6018-6ff2-e36c-b1ec0afd9974} {Suffrage:Voter ID:server-27760def-e4ae-ad0e-25a0-e32b65644ce8 Address:27760def-e4ae-ad0e-25a0-e32b65644ce8}]"
2025-02-23T18:42:14.504Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: entering follower state: follower="Node at 67ff1859-728e-11a0-f959-c64c5b187cc0 [Follower]" leader-address= leader-id=
2025-02-23T18:42:14.504Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:14.504Z [INFO]  server-27760def-e4ae-ad0e-25a0-e32b65644ce8: entering follower state: follower="Node at 27760def-e4ae-ad0e-25a0-e32b65644ce8 [Follower]" leader-address= leader-id=
2025-02-23T18:42:14.504Z [INFO]  server-6e3c02e2-6018-6ff2-e36c-b1ec0afd9974: entering follower state: follower="Node at 6e3c02e2-6018-6ff2-e36c-b1ec0afd9974 [Follower]" leader-address= leader-id=
2025-02-23T18:42:15.070Z [WARN]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:15.070Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: entering candidate state: node="Node at 67ff1859-728e-11a0-f959-c64c5b187cc0 [Candidate]" term=2
2025-02-23T18:42:15.076Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: election won: term=2 tally=2
2025-02-23T18:42:15.076Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: entering leader state: leader="Node at 67ff1859-728e-11a0-f959-c64c5b187cc0 [Leader]"
2025-02-23T18:42:15.076Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: added peer, starting replication: peer=server-6e3c02e2-6018-6ff2-e36c-b1ec0afd9974
2025-02-23T18:42:15.076Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: added peer, starting replication: peer=server-27760def-e4ae-ad0e-25a0-e32b65644ce8
2025-02-23T18:42:15.076Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: pipelining replication: peer="{Voter server-27760def-e4ae-ad0e-25a0-e32b65644ce8 27760def-e4ae-ad0e-25a0-e32b65644ce8}"
2025-02-23T18:42:15.076Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: pipelining replication: peer="{Voter server-6e3c02e2-6018-6ff2-e36c-b1ec0afd9974 6e3c02e2-6018-6ff2-e36c-b1ec0afd9974}"
2025-02-23T18:42:16.081Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:15.076599072 +0000 UTC m=+39.374859982 (1 nodes), 571.743246ms from start of poll, 572.149677ms from cluster start. Timeout at 2025-02-23 18:42:16.081604932 +0000 UTC m=+40.379865656, 1.005005674s after stability
2025-02-23T18:42:16.082Z [INFO]  creating new snapshot: path=/tmp/raft2354861164/snapshots/2-14-1740336136082.tmp
2025-02-23T18:42:16.083Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: copied to local snapshot: bytes=71
2025-02-23T18:42:16.083Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: snapshot restore progress: id=2-14-1740336136082 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:16.083Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: removing all old logs from log store
2025-02-23T18:42:16.083Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: compacting logs: from=1 to=2
2025-02-23T18:42:16.083Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: restored user snapshot: index=14
2025-02-23T18:42:16.083Z [ERROR] server-67ff1859-728e-11a0-f959-c64c5b187cc0: failed to get log: index=2 error="log not found"
2025-02-23T18:42:16.083Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: aborting pipeline replication: peer="{Voter server-6e3c02e2-6018-6ff2-e36c-b1ec0afd9974 6e3c02e2-6018-6ff2-e36c-b1ec0afd9974}"
2025-02-23T18:42:16.083Z [ERROR] server-67ff1859-728e-11a0-f959-c64c5b187cc0: failed to get log: index=2 error="log not found"
2025-02-23T18:42:16.083Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: aborting pipeline replication: peer="{Voter server-27760def-e4ae-ad0e-25a0-e32b65644ce8 27760def-e4ae-ad0e-25a0-e32b65644ce8}"
2025-02-23T18:42:16.083Z [ERROR] server-67ff1859-728e-11a0-f959-c64c5b187cc0: failed to get log: index=2 error="log not found"
2025-02-23T18:42:16.084Z [INFO]  creating new snapshot: path=/tmp/raft1347929256/snapshots/2-14-1740336136084.tmp
2025-02-23T18:42:16.084Z [INFO]  server-6e3c02e2-6018-6ff2-e36c-b1ec0afd9974: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:16.084Z [INFO]  server-6e3c02e2-6018-6ff2-e36c-b1ec0afd9974: copied to local snapshot: bytes=71
2025-02-23T18:42:16.085Z [INFO]  server-6e3c02e2-6018-6ff2-e36c-b1ec0afd9974: snapshot restore progress: id=2-14-1740336136084 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:16.085Z [INFO]  server-6e3c02e2-6018-6ff2-e36c-b1ec0afd9974: removing all old logs from log store
2025-02-23T18:42:16.085Z [INFO]  server-6e3c02e2-6018-6ff2-e36c-b1ec0afd9974: compacting logs: from=1 to=2
2025-02-23T18:42:16.085Z [INFO]  server-6e3c02e2-6018-6ff2-e36c-b1ec0afd9974: Installed remote snapshot
2025-02-23T18:42:16.085Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: pipelining replication: peer="{Voter server-6e3c02e2-6018-6ff2-e36c-b1ec0afd9974 6e3c02e2-6018-6ff2-e36c-b1ec0afd9974}"
2025-02-23T18:42:16.086Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:42:16.090Z [ERROR] server-67ff1859-728e-11a0-f959-c64c5b187cc0: failed to get log: index=2 error="log not found"
2025-02-23T18:42:16.091Z [INFO]  creating new snapshot: path=/tmp/raft3377689112/snapshots/2-14-1740336136091.tmp
2025-02-23T18:42:16.091Z [INFO]  server-27760def-e4ae-ad0e-25a0-e32b65644ce8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:16.092Z [INFO]  server-27760def-e4ae-ad0e-25a0-e32b65644ce8: copied to local snapshot: bytes=71
2025-02-23T18:42:16.092Z [INFO]  server-27760def-e4ae-ad0e-25a0-e32b65644ce8: snapshot restore progress: id=2-14-1740336136091 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:16.092Z [INFO]  server-27760def-e4ae-ad0e-25a0-e32b65644ce8: removing all old logs from log store
2025-02-23T18:42:16.092Z [INFO]  server-27760def-e4ae-ad0e-25a0-e32b65644ce8: compacting logs: from=1 to=2
2025-02-23T18:42:16.092Z [INFO]  server-27760def-e4ae-ad0e-25a0-e32b65644ce8: Installed remote snapshot
2025-02-23T18:42:16.092Z [INFO]  server-67ff1859-728e-11a0-f959-c64c5b187cc0: pipelining replication: peer="{Voter server-27760def-e4ae-ad0e-25a0-e32b65644ce8 27760def-e4ae-ad0e-25a0-e32b65644ce8}"
2025-02-23T18:42:17.091Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:42:16.086074133 +0000 UTC m=+40.384334967 (2 nodes), 2.126µs from start of poll, 1.581624662s from cluster start. Timeout at 2025-02-23 18:42:17.09108822 +0000 UTC m=+41.389348976, 1.005014009s after stability
2025-02-23T18:42:17.104Z [INFO]  server-dd906e14-9889-54f1-aab6-afeb25fea148: aborting pipeline replication: peer="{Voter server-3cca2d57-0d09-3155-0319-33f6fe200d85 3cca2d57-0d09-3155-0319-33f6fe200d85}"
2025-02-23T18:42:17.104Z [INFO]  server-dd906e14-9889-54f1-aab6-afeb25fea148: aborting pipeline replication: peer="{Voter server-ee237e21-5258-4281-4f7f-1df8d0b96a1c ee237e21-5258-4281-4f7f-1df8d0b96a1c}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false
2025-02-23T18:42:17.106Z [INFO]  server-abb58518-5c42-7d37-e933-0404bb734b75: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abb58518-5c42-7d37-e933-0404bb734b75 Address:abb58518-5c42-7d37-e933-0404bb734b75} {Suffrage:Voter ID:server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf Address:e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf} {Suffrage:Voter ID:server-cdd4c47c-a078-55c7-6a97-d239b78c95fc Address:cdd4c47c-a078-55c7-6a97-d239b78c95fc}]"
2025-02-23T18:42:17.106Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abb58518-5c42-7d37-e933-0404bb734b75 Address:abb58518-5c42-7d37-e933-0404bb734b75} {Suffrage:Voter ID:server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf Address:e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf} {Suffrage:Voter ID:server-cdd4c47c-a078-55c7-6a97-d239b78c95fc Address:cdd4c47c-a078-55c7-6a97-d239b78c95fc}]"
2025-02-23T18:42:17.106Z [INFO]  server-cdd4c47c-a078-55c7-6a97-d239b78c95fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abb58518-5c42-7d37-e933-0404bb734b75 Address:abb58518-5c42-7d37-e933-0404bb734b75} {Suffrage:Voter ID:server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf Address:e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf} {Suffrage:Voter ID:server-cdd4c47c-a078-55c7-6a97-d239b78c95fc Address:cdd4c47c-a078-55c7-6a97-d239b78c95fc}]"
2025-02-23T18:42:17.106Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:17.106Z [INFO]  server-abb58518-5c42-7d37-e933-0404bb734b75: entering follower state: follower="Node at abb58518-5c42-7d37-e933-0404bb734b75 [Follower]" leader-address= leader-id=
2025-02-23T18:42:17.106Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: entering follower state: follower="Node at e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf [Follower]" leader-address= leader-id=
2025-02-23T18:42:17.106Z [INFO]  server-cdd4c47c-a078-55c7-6a97-d239b78c95fc: entering follower state: follower="Node at cdd4c47c-a078-55c7-6a97-d239b78c95fc [Follower]" leader-address= leader-id=
2025-02-23T18:42:17.675Z [WARN]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:17.675Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: entering candidate state: node="Node at e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf [Candidate]" term=2
2025-02-23T18:42:17.675Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: election won: term=2 tally=2
2025-02-23T18:42:17.675Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: entering leader state: leader="Node at e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf [Leader]"
2025-02-23T18:42:17.676Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: added peer, starting replication: peer=server-abb58518-5c42-7d37-e933-0404bb734b75
2025-02-23T18:42:17.676Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: added peer, starting replication: peer=server-cdd4c47c-a078-55c7-6a97-d239b78c95fc
2025-02-23T18:42:17.676Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: pipelining replication: peer="{Voter server-abb58518-5c42-7d37-e933-0404bb734b75 abb58518-5c42-7d37-e933-0404bb734b75}"
2025-02-23T18:42:17.676Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: pipelining replication: peer="{Voter server-cdd4c47c-a078-55c7-6a97-d239b78c95fc cdd4c47c-a078-55c7-6a97-d239b78c95fc}"
2025-02-23T18:42:18.681Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:17.676600988 +0000 UTC m=+41.974861681 (1 nodes), 570.037548ms from start of poll, 570.491976ms from cluster start. Timeout at 2025-02-23 18:42:18.68160761 +0000 UTC m=+42.979868336, 1.005006655s after stability
2025-02-23T18:42:18.682Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: starting snapshot up to: index=12
2025-02-23T18:42:18.682Z [INFO]  creating new snapshot: path=/tmp/raft2350980485/snapshots/2-12-1740336138682.tmp
2025-02-23T18:42:18.744Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: snapshot complete up to: index=12
2025-02-23T18:42:18.746Z [INFO]  creating new snapshot: path=/tmp/raft2350980485/snapshots/2-23-1740336138746.tmp
2025-02-23T18:42:18.747Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: copied to local snapshot: bytes=71
2025-02-23T18:42:18.747Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: snapshot restore progress: id=2-23-1740336138746 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:18.747Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: restored user snapshot: index=23
2025-02-23T18:42:18.747Z [ERROR] server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: failed to get log: index=23 error="log not found"
2025-02-23T18:42:18.747Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: aborting pipeline replication: peer="{Voter server-cdd4c47c-a078-55c7-6a97-d239b78c95fc cdd4c47c-a078-55c7-6a97-d239b78c95fc}"
2025-02-23T18:42:18.748Z [ERROR] server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: failed to get log: index=23 error="log not found"
2025-02-23T18:42:18.748Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: aborting pipeline replication: peer="{Voter server-abb58518-5c42-7d37-e933-0404bb734b75 abb58518-5c42-7d37-e933-0404bb734b75}"
2025-02-23T18:42:18.755Z [ERROR] server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: failed to get log: index=23 error="log not found"
2025-02-23T18:42:18.756Z [ERROR] server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: failed to get log: index=23 error="log not found"
2025-02-23T18:42:18.756Z [INFO]  creating new snapshot: path=/tmp/raft1956588589/snapshots/2-23-1740336138756.tmp
2025-02-23T18:42:18.757Z [INFO]  server-cdd4c47c-a078-55c7-6a97-d239b78c95fc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:18.757Z [INFO]  creating new snapshot: path=/tmp/raft3022278667/snapshots/2-23-1740336138757.tmp
2025-02-23T18:42:18.757Z [INFO]  server-abb58518-5c42-7d37-e933-0404bb734b75: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:18.764Z [INFO]  server-cdd4c47c-a078-55c7-6a97-d239b78c95fc: copied to local snapshot: bytes=71
2025-02-23T18:42:18.764Z [INFO]  server-cdd4c47c-a078-55c7-6a97-d239b78c95fc: snapshot restore progress: id=2-23-1740336138756 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:18.764Z [INFO]  server-cdd4c47c-a078-55c7-6a97-d239b78c95fc: Installed remote snapshot
2025-02-23T18:42:18.764Z [WARN]  server-cdd4c47c-a078-55c7-6a97-d239b78c95fc: skipping application of old log: index=22
2025-02-23T18:42:18.764Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: pipelining replication: peer="{Voter server-cdd4c47c-a078-55c7-6a97-d239b78c95fc cdd4c47c-a078-55c7-6a97-d239b78c95fc}"
2025-02-23T18:42:18.765Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:42:18.766Z [INFO]  server-abb58518-5c42-7d37-e933-0404bb734b75: copied to local snapshot: bytes=71
2025-02-23T18:42:18.767Z [INFO]  server-abb58518-5c42-7d37-e933-0404bb734b75: snapshot restore progress: id=2-23-1740336138757 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:18.767Z [INFO]  server-abb58518-5c42-7d37-e933-0404bb734b75: Installed remote snapshot
2025-02-23T18:42:18.767Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: pipelining replication: peer="{Voter server-abb58518-5c42-7d37-e933-0404bb734b75 abb58518-5c42-7d37-e933-0404bb734b75}"
2025-02-23T18:42:19.771Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:42:18.765049593 +0000 UTC m=+43.063310640 (2 nodes), 3.533µs from start of poll, 1.658940935s from cluster start. Timeout at 2025-02-23 18:42:19.770068095 +0000 UTC m=+44.068328876, 1.005018236s after stability
2025-02-23T18:42:19.785Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: aborting pipeline replication: peer="{Voter server-abb58518-5c42-7d37-e933-0404bb734b75 abb58518-5c42-7d37-e933-0404bb734b75}"
2025-02-23T18:42:19.785Z [INFO]  server-e939c79b-ab4b-35c2-bd15-4cd5eeca6fcf: aborting pipeline replication: peer="{Voter server-cdd4c47c-a078-55c7-6a97-d239b78c95fc cdd4c47c-a078-55c7-6a97-d239b78c95fc}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false
2025-02-23T18:42:19.787Z [INFO]  server-44b51aa9-f6ac-b1c3-1476-19b9702fecb7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44b51aa9-f6ac-b1c3-1476-19b9702fecb7 Address:44b51aa9-f6ac-b1c3-1476-19b9702fecb7} {Suffrage:Voter ID:server-f340c71c-c7e7-4026-152d-d7e54f666700 Address:f340c71c-c7e7-4026-152d-d7e54f666700} {Suffrage:Voter ID:server-cb7c2141-9398-edd7-27fd-0a47d18d3ffd Address:cb7c2141-9398-edd7-27fd-0a47d18d3ffd}]"
2025-02-23T18:42:19.788Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44b51aa9-f6ac-b1c3-1476-19b9702fecb7 Address:44b51aa9-f6ac-b1c3-1476-19b9702fecb7} {Suffrage:Voter ID:server-f340c71c-c7e7-4026-152d-d7e54f666700 Address:f340c71c-c7e7-4026-152d-d7e54f666700} {Suffrage:Voter ID:server-cb7c2141-9398-edd7-27fd-0a47d18d3ffd Address:cb7c2141-9398-edd7-27fd-0a47d18d3ffd}]"
2025-02-23T18:42:19.788Z [INFO]  server-cb7c2141-9398-edd7-27fd-0a47d18d3ffd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44b51aa9-f6ac-b1c3-1476-19b9702fecb7 Address:44b51aa9-f6ac-b1c3-1476-19b9702fecb7} {Suffrage:Voter ID:server-f340c71c-c7e7-4026-152d-d7e54f666700 Address:f340c71c-c7e7-4026-152d-d7e54f666700} {Suffrage:Voter ID:server-cb7c2141-9398-edd7-27fd-0a47d18d3ffd Address:cb7c2141-9398-edd7-27fd-0a47d18d3ffd}]"
2025-02-23T18:42:19.788Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:19.788Z [INFO]  server-44b51aa9-f6ac-b1c3-1476-19b9702fecb7: entering follower state: follower="Node at 44b51aa9-f6ac-b1c3-1476-19b9702fecb7 [Follower]" leader-address= leader-id=
2025-02-23T18:42:19.788Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: entering follower state: follower="Node at f340c71c-c7e7-4026-152d-d7e54f666700 [Follower]" leader-address= leader-id=
2025-02-23T18:42:19.788Z [INFO]  server-cb7c2141-9398-edd7-27fd-0a47d18d3ffd: entering follower state: follower="Node at cb7c2141-9398-edd7-27fd-0a47d18d3ffd [Follower]" leader-address= leader-id=
2025-02-23T18:42:20.304Z [WARN]  server-f340c71c-c7e7-4026-152d-d7e54f666700: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:20.304Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: entering candidate state: node="Node at f340c71c-c7e7-4026-152d-d7e54f666700 [Candidate]" term=2
2025-02-23T18:42:20.304Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: election won: term=2 tally=2
2025-02-23T18:42:20.304Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: entering leader state: leader="Node at f340c71c-c7e7-4026-152d-d7e54f666700 [Leader]"
2025-02-23T18:42:20.304Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: added peer, starting replication: peer=server-44b51aa9-f6ac-b1c3-1476-19b9702fecb7
2025-02-23T18:42:20.304Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: added peer, starting replication: peer=server-cb7c2141-9398-edd7-27fd-0a47d18d3ffd
2025-02-23T18:42:20.304Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: pipelining replication: peer="{Voter server-cb7c2141-9398-edd7-27fd-0a47d18d3ffd cb7c2141-9398-edd7-27fd-0a47d18d3ffd}"
2025-02-23T18:42:20.304Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: pipelining replication: peer="{Voter server-44b51aa9-f6ac-b1c3-1476-19b9702fecb7 44b51aa9-f6ac-b1c3-1476-19b9702fecb7}"
2025-02-23T18:42:21.310Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:20.30510456 +0000 UTC m=+44.603365215 (1 nodes), 516.829087ms from start of poll, 517.232487ms from cluster start. Timeout at 2025-02-23 18:42:21.310140565 +0000 UTC m=+45.608401200, 1.005035985s after stability
2025-02-23T18:42:21.311Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: starting snapshot up to: index=12
2025-02-23T18:42:21.312Z [INFO]  creating new snapshot: path=/tmp/raft2611856249/snapshots/2-12-1740336141312.tmp
2025-02-23T18:42:21.312Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: snapshot complete up to: index=12
2025-02-23T18:42:21.314Z [INFO]  creating new snapshot: path=/tmp/raft2611856249/snapshots/2-23-1740336141314.tmp
2025-02-23T18:42:21.315Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: copied to local snapshot: bytes=71
2025-02-23T18:42:21.315Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: snapshot restore progress: id=2-23-1740336141314 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:21.315Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: removing all old logs from log store
2025-02-23T18:42:21.315Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: compacting logs: from=1 to=22
2025-02-23T18:42:21.315Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: restored user snapshot: index=23
2025-02-23T18:42:21.316Z [ERROR] server-f340c71c-c7e7-4026-152d-d7e54f666700: failed to get log: index=22 error="log not found"
2025-02-23T18:42:21.316Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: aborting pipeline replication: peer="{Voter server-cb7c2141-9398-edd7-27fd-0a47d18d3ffd cb7c2141-9398-edd7-27fd-0a47d18d3ffd}"
2025-02-23T18:42:21.316Z [ERROR] server-f340c71c-c7e7-4026-152d-d7e54f666700: failed to get log: index=22 error="log not found"
2025-02-23T18:42:21.316Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: aborting pipeline replication: peer="{Voter server-44b51aa9-f6ac-b1c3-1476-19b9702fecb7 44b51aa9-f6ac-b1c3-1476-19b9702fecb7}"
2025-02-23T18:42:21.322Z [ERROR] server-f340c71c-c7e7-4026-152d-d7e54f666700: failed to get log: index=22 error="log not found"
2025-02-23T18:42:21.324Z [INFO]  creating new snapshot: path=/tmp/raft2732093771/snapshots/2-23-1740336141324.tmp
2025-02-23T18:42:21.325Z [ERROR] server-f340c71c-c7e7-4026-152d-d7e54f666700: failed to get log: index=22 error="log not found"
2025-02-23T18:42:21.327Z [INFO]  server-cb7c2141-9398-edd7-27fd-0a47d18d3ffd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:21.327Z [INFO]  creating new snapshot: path=/tmp/raft4238702045/snapshots/2-23-1740336141327.tmp
2025-02-23T18:42:21.328Z [INFO]  server-cb7c2141-9398-edd7-27fd-0a47d18d3ffd: copied to local snapshot: bytes=71
2025-02-23T18:42:21.328Z [INFO]  server-44b51aa9-f6ac-b1c3-1476-19b9702fecb7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:21.328Z [INFO]  server-cb7c2141-9398-edd7-27fd-0a47d18d3ffd: snapshot restore progress: id=2-23-1740336141324 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:21.328Z [INFO]  server-cb7c2141-9398-edd7-27fd-0a47d18d3ffd: removing all old logs from log store
2025-02-23T18:42:21.328Z [INFO]  server-cb7c2141-9398-edd7-27fd-0a47d18d3ffd: compacting logs: from=1 to=22
2025-02-23T18:42:21.328Z [INFO]  server-cb7c2141-9398-edd7-27fd-0a47d18d3ffd: Installed remote snapshot
2025-02-23T18:42:21.328Z [WARN]  server-cb7c2141-9398-edd7-27fd-0a47d18d3ffd: skipping application of old log: index=22
2025-02-23T18:42:21.329Z [INFO]  server-44b51aa9-f6ac-b1c3-1476-19b9702fecb7: copied to local snapshot: bytes=71
2025-02-23T18:42:21.329Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: pipelining replication: peer="{Voter server-cb7c2141-9398-edd7-27fd-0a47d18d3ffd cb7c2141-9398-edd7-27fd-0a47d18d3ffd}"
2025-02-23T18:42:21.329Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:42:21.329Z [INFO]  server-44b51aa9-f6ac-b1c3-1476-19b9702fecb7: snapshot restore progress: id=2-23-1740336141327 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:21.329Z [INFO]  server-44b51aa9-f6ac-b1c3-1476-19b9702fecb7: removing all old logs from log store
2025-02-23T18:42:21.329Z [INFO]  server-44b51aa9-f6ac-b1c3-1476-19b9702fecb7: compacting logs: from=1 to=22
2025-02-23T18:42:21.329Z [INFO]  server-44b51aa9-f6ac-b1c3-1476-19b9702fecb7: Installed remote snapshot
2025-02-23T18:42:21.329Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: pipelining replication: peer="{Voter server-44b51aa9-f6ac-b1c3-1476-19b9702fecb7 44b51aa9-f6ac-b1c3-1476-19b9702fecb7}"
2025-02-23T18:42:22.334Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:42:21.329487936 +0000 UTC m=+45.627748618 (2 nodes), 2.588µs from start of poll, 1.54161589s from cluster start. Timeout at 2025-02-23 18:42:22.334518642 +0000 UTC m=+46.632779290, 1.005030672s after stability
2025-02-23T18:42:22.351Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: aborting pipeline replication: peer="{Voter server-cb7c2141-9398-edd7-27fd-0a47d18d3ffd cb7c2141-9398-edd7-27fd-0a47d18d3ffd}"
2025-02-23T18:42:22.351Z [INFO]  server-f340c71c-c7e7-4026-152d-d7e54f666700: aborting pipeline replication: peer="{Voter server-44b51aa9-f6ac-b1c3-1476-19b9702fecb7 44b51aa9-f6ac-b1c3-1476-19b9702fecb7}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true
2025-02-23T18:42:22.354Z [INFO]  server-17931278-f050-a9e4-8387-3b227117477e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17931278-f050-a9e4-8387-3b227117477e Address:17931278-f050-a9e4-8387-3b227117477e} {Suffrage:Voter ID:server-92972803-5cd1-6e44-ac13-735c5e1e49f1 Address:92972803-5cd1-6e44-ac13-735c5e1e49f1} {Suffrage:Voter ID:server-7ab3b9d7-c7f3-c972-fa40-c3728654dfb5 Address:7ab3b9d7-c7f3-c972-fa40-c3728654dfb5}]"
2025-02-23T18:42:22.354Z [INFO]  server-92972803-5cd1-6e44-ac13-735c5e1e49f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17931278-f050-a9e4-8387-3b227117477e Address:17931278-f050-a9e4-8387-3b227117477e} {Suffrage:Voter ID:server-92972803-5cd1-6e44-ac13-735c5e1e49f1 Address:92972803-5cd1-6e44-ac13-735c5e1e49f1} {Suffrage:Voter ID:server-7ab3b9d7-c7f3-c972-fa40-c3728654dfb5 Address:7ab3b9d7-c7f3-c972-fa40-c3728654dfb5}]"
2025-02-23T18:42:22.354Z [INFO]  server-7ab3b9d7-c7f3-c972-fa40-c3728654dfb5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17931278-f050-a9e4-8387-3b227117477e Address:17931278-f050-a9e4-8387-3b227117477e} {Suffrage:Voter ID:server-92972803-5cd1-6e44-ac13-735c5e1e49f1 Address:92972803-5cd1-6e44-ac13-735c5e1e49f1} {Suffrage:Voter ID:server-7ab3b9d7-c7f3-c972-fa40-c3728654dfb5 Address:7ab3b9d7-c7f3-c972-fa40-c3728654dfb5}]"
2025-02-23T18:42:22.354Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:22.354Z [INFO]  server-17931278-f050-a9e4-8387-3b227117477e: entering follower state: follower="Node at 17931278-f050-a9e4-8387-3b227117477e [Follower]" leader-address= leader-id=
2025-02-23T18:42:22.354Z [INFO]  server-92972803-5cd1-6e44-ac13-735c5e1e49f1: entering follower state: follower="Node at 92972803-5cd1-6e44-ac13-735c5e1e49f1 [Follower]" leader-address= leader-id=
2025-02-23T18:42:22.354Z [INFO]  server-7ab3b9d7-c7f3-c972-fa40-c3728654dfb5: entering follower state: follower="Node at 7ab3b9d7-c7f3-c972-fa40-c3728654dfb5 [Follower]" leader-address= leader-id=
2025-02-23T18:42:22.869Z [WARN]  server-92972803-5cd1-6e44-ac13-735c5e1e49f1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:22.869Z [INFO]  server-92972803-5cd1-6e44-ac13-735c5e1e49f1: entering candidate state: node="Node at 92972803-5cd1-6e44-ac13-735c5e1e49f1 [Candidate]" term=2
2025-02-23T18:42:22.869Z [INFO]  server-92972803-5cd1-6e44-ac13-735c5e1e49f1: election won: term=2 tally=2
2025-02-23T18:42:22.869Z [INFO]  server-92972803-5cd1-6e44-ac13-735c5e1e49f1: entering leader state: leader="Node at 92972803-5cd1-6e44-ac13-735c5e1e49f1 [Leader]"
2025-02-23T18:42:22.869Z [INFO]  server-92972803-5cd1-6e44-ac13-735c5e1e49f1: added peer, starting replication: peer=server-17931278-f050-a9e4-8387-3b227117477e
2025-02-23T18:42:22.869Z [INFO]  server-92972803-5cd1-6e44-ac13-735c5e1e49f1: added peer, starting replication: peer=server-7ab3b9d7-c7f3-c972-fa40-c3728654dfb5
2025-02-23T18:42:22.869Z [INFO]  server-92972803-5cd1-6e44-ac13-735c5e1e49f1: pipelining replication: peer="{Voter server-7ab3b9d7-c7f3-c972-fa40-c3728654dfb5 7ab3b9d7-c7f3-c972-fa40-c3728654dfb5}"
2025-02-23T18:42:22.870Z [INFO]  server-92972803-5cd1-6e44-ac13-735c5e1e49f1: pipelining replication: peer="{Voter server-17931278-f050-a9e4-8387-3b227117477e 17931278-f050-a9e4-8387-3b227117477e}"
2025-02-23T18:42:23.875Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:22.86967174 +0000 UTC m=+47.167932385 (1 nodes), 515.100477ms from start of poll, 515.71083ms from cluster start. Timeout at 2025-02-23 18:42:23.874679382 +0000 UTC m=+48.172940313, 1.005007928s after stability
2025-02-23T18:42:23.877Z [INFO]  server-92972803-5cd1-6e44-ac13-735c5e1e49f1: starting snapshot up to: index=12
2025-02-23T18:42:23.877Z [INFO]  creating new snapshot: path=/tmp/raft1161454596/snapshots/2-12-1740336143877.tmp
2025-02-23T18:42:23.880Z [INFO]  server-92972803-5cd1-6e44-ac13-735c5e1e49f1: snapshot complete up to: index=12
2025-02-23T18:42:23.886Z [INFO]  server-359972dd-18e5-d8ee-7803-fb25de3d8c0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-359972dd-18e5-d8ee-7803-fb25de3d8c0e Address:359972dd-18e5-d8ee-7803-fb25de3d8c0e} {Suffrage:Voter ID:server-7a34b6db-a52d-6db3-6349-1da93df61501 Address:7a34b6db-a52d-6db3-6349-1da93df61501} {Suffrage:Voter ID:server-2388cf36-2113-18c4-7108-7e6f68cea958 Address:2388cf36-2113-18c4-7108-7e6f68cea958}]"
2025-02-23T18:42:23.886Z [INFO]  server-7a34b6db-a52d-6db3-6349-1da93df61501: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-359972dd-18e5-d8ee-7803-fb25de3d8c0e Address:359972dd-18e5-d8ee-7803-fb25de3d8c0e} {Suffrage:Voter ID:server-7a34b6db-a52d-6db3-6349-1da93df61501 Address:7a34b6db-a52d-6db3-6349-1da93df61501} {Suffrage:Voter ID:server-2388cf36-2113-18c4-7108-7e6f68cea958 Address:2388cf36-2113-18c4-7108-7e6f68cea958}]"
2025-02-23T18:42:23.886Z [INFO]  server-2388cf36-2113-18c4-7108-7e6f68cea958: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-359972dd-18e5-d8ee-7803-fb25de3d8c0e Address:359972dd-18e5-d8ee-7803-fb25de3d8c0e} {Suffrage:Voter ID:server-7a34b6db-a52d-6db3-6349-1da93df61501 Address:7a34b6db-a52d-6db3-6349-1da93df61501} {Suffrage:Voter ID:server-2388cf36-2113-18c4-7108-7e6f68cea958 Address:2388cf36-2113-18c4-7108-7e6f68cea958}]"
2025-02-23T18:42:23.886Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:23.886Z [INFO]  server-359972dd-18e5-d8ee-7803-fb25de3d8c0e: entering follower state: follower="Node at 359972dd-18e5-d8ee-7803-fb25de3d8c0e [Follower]" leader-address= leader-id=
2025-02-23T18:42:23.886Z [INFO]  server-7a34b6db-a52d-6db3-6349-1da93df61501: entering follower state: follower="Node at 7a34b6db-a52d-6db3-6349-1da93df61501 [Follower]" leader-address= leader-id=
2025-02-23T18:42:23.886Z [INFO]  server-2388cf36-2113-18c4-7108-7e6f68cea958: entering follower state: follower="Node at 2388cf36-2113-18c4-7108-7e6f68cea958 [Follower]" leader-address= leader-id=
2025-02-23T18:42:24.457Z [WARN]  server-2388cf36-2113-18c4-7108-7e6f68cea958: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:24.458Z [INFO]  server-2388cf36-2113-18c4-7108-7e6f68cea958: entering candidate state: node="Node at 2388cf36-2113-18c4-7108-7e6f68cea958 [Candidate]" term=2
2025-02-23T18:42:24.458Z [INFO]  server-2388cf36-2113-18c4-7108-7e6f68cea958: election won: term=2 tally=2
2025-02-23T18:42:24.458Z [INFO]  server-2388cf36-2113-18c4-7108-7e6f68cea958: entering leader state: leader="Node at 2388cf36-2113-18c4-7108-7e6f68cea958 [Leader]"
2025-02-23T18:42:24.458Z [INFO]  server-2388cf36-2113-18c4-7108-7e6f68cea958: added peer, starting replication: peer=server-359972dd-18e5-d8ee-7803-fb25de3d8c0e
2025-02-23T18:42:24.458Z [INFO]  server-2388cf36-2113-18c4-7108-7e6f68cea958: added peer, starting replication: peer=server-7a34b6db-a52d-6db3-6349-1da93df61501
2025-02-23T18:42:24.463Z [INFO]  server-2388cf36-2113-18c4-7108-7e6f68cea958: pipelining replication: peer="{Voter server-7a34b6db-a52d-6db3-6349-1da93df61501 7a34b6db-a52d-6db3-6349-1da93df61501}"
2025-02-23T18:42:24.464Z [INFO]  server-2388cf36-2113-18c4-7108-7e6f68cea958: pipelining replication: peer="{Voter server-359972dd-18e5-d8ee-7803-fb25de3d8c0e 359972dd-18e5-d8ee-7803-fb25de3d8c0e}"
2025-02-23T18:42:25.469Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:24.464266967 +0000 UTC m=+48.762527613 (1 nodes), 577.620726ms from start of poll, 578.072739ms from cluster start. Timeout at 2025-02-23 18:42:25.469278584 +0000 UTC m=+49.767539245, 1.005011632s after stability
2025-02-23T18:42:25.469Z [INFO]  creating new snapshot: path=/tmp/raft2524075947/snapshots/2-15-1740336145469.tmp
2025-02-23T18:42:25.501Z [INFO]  server-2388cf36-2113-18c4-7108-7e6f68cea958: copied to local snapshot: bytes=71
2025-02-23T18:42:25.502Z [INFO]  server-2388cf36-2113-18c4-7108-7e6f68cea958: snapshot restore progress: id=2-15-1740336145469 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:25.502Z [INFO]  server-2388cf36-2113-18c4-7108-7e6f68cea958: restored user snapshot: index=15
2025-02-23T18:42:25.502Z [ERROR] server-2388cf36-2113-18c4-7108-7e6f68cea958: failed to get log: index=3 error="log not found"
2025-02-23T18:42:25.502Z [INFO]  server-2388cf36-2113-18c4-7108-7e6f68cea958: aborting pipeline replication: peer="{Voter server-7a34b6db-a52d-6db3-6349-1da93df61501 7a34b6db-a52d-6db3-6349-1da93df61501}"
2025-02-23T18:42:25.502Z [ERROR] server-2388cf36-2113-18c4-7108-7e6f68cea958: failed to get log: index=3 error="log not found"
2025-02-23T18:42:25.502Z [INFO]  server-2388cf36-2113-18c4-7108-7e6f68cea958: aborting pipeline replication: peer="{Voter server-359972dd-18e5-d8ee-7803-fb25de3d8c0e 359972dd-18e5-d8ee-7803-fb25de3d8c0e}"
2025-02-23T18:42:25.510Z [ERROR] server-2388cf36-2113-18c4-7108-7e6f68cea958: failed to get log: index=3 error="log not found"
2025-02-23T18:42:25.510Z [ERROR] server-2388cf36-2113-18c4-7108-7e6f68cea958: failed to get log: index=3 error="log not found"
2025-02-23T18:42:25.510Z [INFO]  creating new snapshot: path=/tmp/raft1903340584/snapshots/2-15-1740336145510.tmp
2025-02-23T18:42:25.510Z [INFO]  server-359972dd-18e5-d8ee-7803-fb25de3d8c0e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:25.511Z [INFO]  creating new snapshot: path=/tmp/raft764574407/snapshots/2-15-1740336145511.tmp
2025-02-23T18:42:25.512Z [INFO]  server-7a34b6db-a52d-6db3-6349-1da93df61501: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:25.538Z [INFO]  server-7a34b6db-a52d-6db3-6349-1da93df61501: copied to local snapshot: bytes=71
2025-02-23T18:42:25.539Z [INFO]  server-7a34b6db-a52d-6db3-6349-1da93df61501: snapshot restore progress: id=2-15-1740336145511 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:25.539Z [INFO]  server-7a34b6db-a52d-6db3-6349-1da93df61501: Installed remote snapshot
2025-02-23T18:42:25.539Z [INFO]  server-2388cf36-2113-18c4-7108-7e6f68cea958: pipelining replication: peer="{Voter server-7a34b6db-a52d-6db3-6349-1da93df61501 7a34b6db-a52d-6db3-6349-1da93df61501}"
2025-02-23T18:42:25.539Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:42:25.543Z [INFO]  server-359972dd-18e5-d8ee-7803-fb25de3d8c0e: copied to local snapshot: bytes=71
2025-02-23T18:42:25.543Z [INFO]  server-359972dd-18e5-d8ee-7803-fb25de3d8c0e: snapshot restore progress: id=2-15-1740336145510 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:25.543Z [INFO]  server-359972dd-18e5-d8ee-7803-fb25de3d8c0e: Installed remote snapshot
2025-02-23T18:42:25.543Z [INFO]  server-2388cf36-2113-18c4-7108-7e6f68cea958: pipelining replication: peer="{Voter server-359972dd-18e5-d8ee-7803-fb25de3d8c0e 359972dd-18e5-d8ee-7803-fb25de3d8c0e}"
2025-02-23T18:42:26.545Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:42:25.53971906 +0000 UTC m=+49.837979684 (2 nodes), 2.258µs from start of poll, 1.65352481s from cluster start. Timeout at 2025-02-23 18:42:26.544720156 +0000 UTC m=+50.842980286, 1.005000602s after stability
2025-02-23T18:42:26.557Z [INFO]  server-92972803-5cd1-6e44-ac13-735c5e1e49f1: aborting pipeline replication: peer="{Voter server-17931278-f050-a9e4-8387-3b227117477e 17931278-f050-a9e4-8387-3b227117477e}"
2025-02-23T18:42:26.557Z [INFO]  server-92972803-5cd1-6e44-ac13-735c5e1e49f1: aborting pipeline replication: peer="{Voter server-7ab3b9d7-c7f3-c972-fa40-c3728654dfb5 7ab3b9d7-c7f3-c972-fa40-c3728654dfb5}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true
2025-02-23T18:42:26.559Z [INFO]  server-8d1f9e68-27ef-ff63-71b4-36f09e178555: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d1f9e68-27ef-ff63-71b4-36f09e178555 Address:8d1f9e68-27ef-ff63-71b4-36f09e178555} {Suffrage:Voter ID:server-8e976c5f-4fee-e623-b87d-301dd9aff9cb Address:8e976c5f-4fee-e623-b87d-301dd9aff9cb} {Suffrage:Voter ID:server-fedce316-9dba-59b4-ee67-989312268fac Address:fedce316-9dba-59b4-ee67-989312268fac}]"
2025-02-23T18:42:26.559Z [INFO]  server-8e976c5f-4fee-e623-b87d-301dd9aff9cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d1f9e68-27ef-ff63-71b4-36f09e178555 Address:8d1f9e68-27ef-ff63-71b4-36f09e178555} {Suffrage:Voter ID:server-8e976c5f-4fee-e623-b87d-301dd9aff9cb Address:8e976c5f-4fee-e623-b87d-301dd9aff9cb} {Suffrage:Voter ID:server-fedce316-9dba-59b4-ee67-989312268fac Address:fedce316-9dba-59b4-ee67-989312268fac}]"
2025-02-23T18:42:26.560Z [INFO]  server-fedce316-9dba-59b4-ee67-989312268fac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d1f9e68-27ef-ff63-71b4-36f09e178555 Address:8d1f9e68-27ef-ff63-71b4-36f09e178555} {Suffrage:Voter ID:server-8e976c5f-4fee-e623-b87d-301dd9aff9cb Address:8e976c5f-4fee-e623-b87d-301dd9aff9cb} {Suffrage:Voter ID:server-fedce316-9dba-59b4-ee67-989312268fac Address:fedce316-9dba-59b4-ee67-989312268fac}]"
2025-02-23T18:42:26.560Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:26.560Z [INFO]  server-8d1f9e68-27ef-ff63-71b4-36f09e178555: entering follower state: follower="Node at 8d1f9e68-27ef-ff63-71b4-36f09e178555 [Follower]" leader-address= leader-id=
2025-02-23T18:42:26.560Z [INFO]  server-8e976c5f-4fee-e623-b87d-301dd9aff9cb: entering follower state: follower="Node at 8e976c5f-4fee-e623-b87d-301dd9aff9cb [Follower]" leader-address= leader-id=
2025-02-23T18:42:26.560Z [INFO]  server-fedce316-9dba-59b4-ee67-989312268fac: entering follower state: follower="Node at fedce316-9dba-59b4-ee67-989312268fac [Follower]" leader-address= leader-id=
2025-02-23T18:42:27.337Z [WARN]  server-8d1f9e68-27ef-ff63-71b4-36f09e178555: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:27.337Z [INFO]  server-8d1f9e68-27ef-ff63-71b4-36f09e178555: entering candidate state: node="Node at 8d1f9e68-27ef-ff63-71b4-36f09e178555 [Candidate]" term=2
2025-02-23T18:42:27.337Z [INFO]  server-8d1f9e68-27ef-ff63-71b4-36f09e178555: election won: term=2 tally=2
2025-02-23T18:42:27.337Z [INFO]  server-8d1f9e68-27ef-ff63-71b4-36f09e178555: entering leader state: leader="Node at 8d1f9e68-27ef-ff63-71b4-36f09e178555 [Leader]"
2025-02-23T18:42:27.338Z [INFO]  server-8d1f9e68-27ef-ff63-71b4-36f09e178555: added peer, starting replication: peer=server-8e976c5f-4fee-e623-b87d-301dd9aff9cb
2025-02-23T18:42:27.338Z [INFO]  server-8d1f9e68-27ef-ff63-71b4-36f09e178555: added peer, starting replication: peer=server-fedce316-9dba-59b4-ee67-989312268fac
2025-02-23T18:42:27.338Z [INFO]  server-8d1f9e68-27ef-ff63-71b4-36f09e178555: pipelining replication: peer="{Voter server-8e976c5f-4fee-e623-b87d-301dd9aff9cb 8e976c5f-4fee-e623-b87d-301dd9aff9cb}"
2025-02-23T18:42:27.338Z [INFO]  server-8d1f9e68-27ef-ff63-71b4-36f09e178555: pipelining replication: peer="{Voter server-fedce316-9dba-59b4-ee67-989312268fac fedce316-9dba-59b4-ee67-989312268fac}"
2025-02-23T18:42:28.344Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:27.338576262 +0000 UTC m=+51.636836958 (1 nodes), 778.428469ms from start of poll, 778.875344ms from cluster start. Timeout at 2025-02-23 18:42:28.343579008 +0000 UTC m=+52.641839676, 1.005002718s after stability
2025-02-23T18:42:28.346Z [INFO]  server-8d1f9e68-27ef-ff63-71b4-36f09e178555: starting snapshot up to: index=12
2025-02-23T18:42:28.346Z [INFO]  creating new snapshot: path=/tmp/raft3124026176/snapshots/2-12-1740336148346.tmp
2025-02-23T18:42:28.347Z [INFO]  server-8d1f9e68-27ef-ff63-71b4-36f09e178555: snapshot complete up to: index=12
2025-02-23T18:42:28.350Z [INFO]  server-11ac978b-edbc-4080-cf3e-b61af7cf2508: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11ac978b-edbc-4080-cf3e-b61af7cf2508 Address:11ac978b-edbc-4080-cf3e-b61af7cf2508} {Suffrage:Voter ID:server-71f56385-bda7-ec08-4434-d6568c8a73cc Address:71f56385-bda7-ec08-4434-d6568c8a73cc} {Suffrage:Voter ID:server-603565fb-e04f-d1c1-577b-31ce59e4ca60 Address:603565fb-e04f-d1c1-577b-31ce59e4ca60}]"
2025-02-23T18:42:28.350Z [INFO]  server-71f56385-bda7-ec08-4434-d6568c8a73cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11ac978b-edbc-4080-cf3e-b61af7cf2508 Address:11ac978b-edbc-4080-cf3e-b61af7cf2508} {Suffrage:Voter ID:server-71f56385-bda7-ec08-4434-d6568c8a73cc Address:71f56385-bda7-ec08-4434-d6568c8a73cc} {Suffrage:Voter ID:server-603565fb-e04f-d1c1-577b-31ce59e4ca60 Address:603565fb-e04f-d1c1-577b-31ce59e4ca60}]"
2025-02-23T18:42:28.350Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11ac978b-edbc-4080-cf3e-b61af7cf2508 Address:11ac978b-edbc-4080-cf3e-b61af7cf2508} {Suffrage:Voter ID:server-71f56385-bda7-ec08-4434-d6568c8a73cc Address:71f56385-bda7-ec08-4434-d6568c8a73cc} {Suffrage:Voter ID:server-603565fb-e04f-d1c1-577b-31ce59e4ca60 Address:603565fb-e04f-d1c1-577b-31ce59e4ca60}]"
2025-02-23T18:42:28.350Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:28.350Z [INFO]  server-11ac978b-edbc-4080-cf3e-b61af7cf2508: entering follower state: follower="Node at 11ac978b-edbc-4080-cf3e-b61af7cf2508 [Follower]" leader-address= leader-id=
2025-02-23T18:42:28.350Z [INFO]  server-71f56385-bda7-ec08-4434-d6568c8a73cc: entering follower state: follower="Node at 71f56385-bda7-ec08-4434-d6568c8a73cc [Follower]" leader-address= leader-id=
2025-02-23T18:42:28.350Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: entering follower state: follower="Node at 603565fb-e04f-d1c1-577b-31ce59e4ca60 [Follower]" leader-address= leader-id=
2025-02-23T18:42:29.032Z [WARN]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:29.032Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: entering candidate state: node="Node at 603565fb-e04f-d1c1-577b-31ce59e4ca60 [Candidate]" term=2
2025-02-23T18:42:29.032Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: election won: term=2 tally=2
2025-02-23T18:42:29.032Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: entering leader state: leader="Node at 603565fb-e04f-d1c1-577b-31ce59e4ca60 [Leader]"
2025-02-23T18:42:29.032Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: added peer, starting replication: peer=server-11ac978b-edbc-4080-cf3e-b61af7cf2508
2025-02-23T18:42:29.033Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: added peer, starting replication: peer=server-71f56385-bda7-ec08-4434-d6568c8a73cc
2025-02-23T18:42:29.039Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: pipelining replication: peer="{Voter server-11ac978b-edbc-4080-cf3e-b61af7cf2508 11ac978b-edbc-4080-cf3e-b61af7cf2508}"
2025-02-23T18:42:29.040Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: pipelining replication: peer="{Voter server-71f56385-bda7-ec08-4434-d6568c8a73cc 71f56385-bda7-ec08-4434-d6568c8a73cc}"
2025-02-23T18:42:30.045Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:29.039824461 +0000 UTC m=+53.338085409 (1 nodes), 689.460559ms from start of poll, 689.904936ms from cluster start. Timeout at 2025-02-23 18:42:30.044826188 +0000 UTC m=+54.343086318, 1.005000909s after stability
2025-02-23T18:42:30.045Z [INFO]  creating new snapshot: path=/tmp/raft2552287615/snapshots/2-15-1740336150045.tmp
2025-02-23T18:42:30.053Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: copied to local snapshot: bytes=71
2025-02-23T18:42:30.053Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: snapshot restore progress: id=2-15-1740336150045 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:30.053Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: removing all old logs from log store
2025-02-23T18:42:30.053Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: compacting logs: from=1 to=2
2025-02-23T18:42:30.053Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: restored user snapshot: index=15
2025-02-23T18:42:30.053Z [ERROR] server-603565fb-e04f-d1c1-577b-31ce59e4ca60: failed to get log: index=2 error="log not found"
2025-02-23T18:42:30.053Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: aborting pipeline replication: peer="{Voter server-11ac978b-edbc-4080-cf3e-b61af7cf2508 11ac978b-edbc-4080-cf3e-b61af7cf2508}"
2025-02-23T18:42:30.053Z [ERROR] server-603565fb-e04f-d1c1-577b-31ce59e4ca60: failed to get log: index=2 error="log not found"
2025-02-23T18:42:30.053Z [ERROR] server-603565fb-e04f-d1c1-577b-31ce59e4ca60: failed to get log: index=2 error="log not found"
2025-02-23T18:42:30.053Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: aborting pipeline replication: peer="{Voter server-71f56385-bda7-ec08-4434-d6568c8a73cc 71f56385-bda7-ec08-4434-d6568c8a73cc}"
2025-02-23T18:42:30.053Z [ERROR] server-603565fb-e04f-d1c1-577b-31ce59e4ca60: failed to get log: index=2 error="log not found"
2025-02-23T18:42:30.054Z [INFO]  creating new snapshot: path=/tmp/raft674751349/snapshots/2-15-1740336150054.tmp
2025-02-23T18:42:30.054Z [INFO]  server-11ac978b-edbc-4080-cf3e-b61af7cf2508: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:30.054Z [INFO]  creating new snapshot: path=/tmp/raft1286452656/snapshots/2-15-1740336150054.tmp
2025-02-23T18:42:30.054Z [INFO]  server-11ac978b-edbc-4080-cf3e-b61af7cf2508: copied to local snapshot: bytes=71
2025-02-23T18:42:30.055Z [INFO]  server-11ac978b-edbc-4080-cf3e-b61af7cf2508: snapshot restore progress: id=2-15-1740336150054 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:30.055Z [INFO]  server-11ac978b-edbc-4080-cf3e-b61af7cf2508: removing all old logs from log store
2025-02-23T18:42:30.055Z [INFO]  server-11ac978b-edbc-4080-cf3e-b61af7cf2508: compacting logs: from=1 to=2
2025-02-23T18:42:30.055Z [INFO]  server-11ac978b-edbc-4080-cf3e-b61af7cf2508: Installed remote snapshot
2025-02-23T18:42:30.055Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: pipelining replication: peer="{Voter server-11ac978b-edbc-4080-cf3e-b61af7cf2508 11ac978b-edbc-4080-cf3e-b61af7cf2508}"
2025-02-23T18:42:30.056Z [INFO]  server-71f56385-bda7-ec08-4434-d6568c8a73cc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:30.056Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:42:30.056Z [INFO]  server-71f56385-bda7-ec08-4434-d6568c8a73cc: copied to local snapshot: bytes=71
2025-02-23T18:42:30.057Z [INFO]  server-71f56385-bda7-ec08-4434-d6568c8a73cc: snapshot restore progress: id=2-15-1740336150054 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:30.057Z [INFO]  server-71f56385-bda7-ec08-4434-d6568c8a73cc: removing all old logs from log store
2025-02-23T18:42:30.057Z [INFO]  server-71f56385-bda7-ec08-4434-d6568c8a73cc: compacting logs: from=1 to=2
2025-02-23T18:42:30.057Z [INFO]  server-71f56385-bda7-ec08-4434-d6568c8a73cc: Installed remote snapshot
2025-02-23T18:42:30.057Z [INFO]  server-603565fb-e04f-d1c1-577b-31ce59e4ca60: pipelining replication: peer="{Voter server-71f56385-bda7-ec08-4434-d6568c8a73cc 71f56385-bda7-ec08-4434-d6568c8a73cc}"
2025-02-23T18:42:31.061Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:42:30.056362429 +0000 UTC m=+54.354622559 (2 nodes), 0s from start of poll, 1.706442086s from cluster start. Timeout at 2025-02-23 18:42:31.061425116 +0000 UTC m=+55.359685749, 1.00506319s after stability
2025-02-23T18:42:31.074Z [INFO]  server-8d1f9e68-27ef-ff63-71b4-36f09e178555: aborting pipeline replication: peer="{Voter server-fedce316-9dba-59b4-ee67-989312268fac fedce316-9dba-59b4-ee67-989312268fac}"
2025-02-23T18:42:31.075Z [INFO]  server-8d1f9e68-27ef-ff63-71b4-36f09e178555: aborting pipeline replication: peer="{Voter server-8e976c5f-4fee-e623-b87d-301dd9aff9cb 8e976c5f-4fee-e623-b87d-301dd9aff9cb}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false
2025-02-23T18:42:31.076Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b4f127c-28de-7917-f45c-bc4eeb760cfb Address:4b4f127c-28de-7917-f45c-bc4eeb760cfb} {Suffrage:Voter ID:server-934389ce-87b9-b1d7-0fa1-d56c88b6a9fe Address:934389ce-87b9-b1d7-0fa1-d56c88b6a9fe} {Suffrage:Voter ID:server-1894c342-5c6f-165a-05f8-33f6d74ceaa1 Address:1894c342-5c6f-165a-05f8-33f6d74ceaa1}]"
2025-02-23T18:42:31.076Z [INFO]  server-934389ce-87b9-b1d7-0fa1-d56c88b6a9fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b4f127c-28de-7917-f45c-bc4eeb760cfb Address:4b4f127c-28de-7917-f45c-bc4eeb760cfb} {Suffrage:Voter ID:server-934389ce-87b9-b1d7-0fa1-d56c88b6a9fe Address:934389ce-87b9-b1d7-0fa1-d56c88b6a9fe} {Suffrage:Voter ID:server-1894c342-5c6f-165a-05f8-33f6d74ceaa1 Address:1894c342-5c6f-165a-05f8-33f6d74ceaa1}]"
2025-02-23T18:42:31.077Z [INFO]  server-1894c342-5c6f-165a-05f8-33f6d74ceaa1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b4f127c-28de-7917-f45c-bc4eeb760cfb Address:4b4f127c-28de-7917-f45c-bc4eeb760cfb} {Suffrage:Voter ID:server-934389ce-87b9-b1d7-0fa1-d56c88b6a9fe Address:934389ce-87b9-b1d7-0fa1-d56c88b6a9fe} {Suffrage:Voter ID:server-1894c342-5c6f-165a-05f8-33f6d74ceaa1 Address:1894c342-5c6f-165a-05f8-33f6d74ceaa1}]"
2025-02-23T18:42:31.077Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:31.077Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: entering follower state: follower="Node at 4b4f127c-28de-7917-f45c-bc4eeb760cfb [Follower]" leader-address= leader-id=
2025-02-23T18:42:31.077Z [INFO]  server-934389ce-87b9-b1d7-0fa1-d56c88b6a9fe: entering follower state: follower="Node at 934389ce-87b9-b1d7-0fa1-d56c88b6a9fe [Follower]" leader-address= leader-id=
2025-02-23T18:42:31.077Z [INFO]  server-1894c342-5c6f-165a-05f8-33f6d74ceaa1: entering follower state: follower="Node at 1894c342-5c6f-165a-05f8-33f6d74ceaa1 [Follower]" leader-address= leader-id=
2025-02-23T18:42:31.800Z [WARN]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:31.800Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: entering candidate state: node="Node at 4b4f127c-28de-7917-f45c-bc4eeb760cfb [Candidate]" term=2
2025-02-23T18:42:31.800Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: election won: term=2 tally=2
2025-02-23T18:42:31.800Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: entering leader state: leader="Node at 4b4f127c-28de-7917-f45c-bc4eeb760cfb [Leader]"
2025-02-23T18:42:31.800Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: added peer, starting replication: peer=server-934389ce-87b9-b1d7-0fa1-d56c88b6a9fe
2025-02-23T18:42:31.800Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: added peer, starting replication: peer=server-1894c342-5c6f-165a-05f8-33f6d74ceaa1
2025-02-23T18:42:31.801Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: pipelining replication: peer="{Voter server-1894c342-5c6f-165a-05f8-33f6d74ceaa1 1894c342-5c6f-165a-05f8-33f6d74ceaa1}"
2025-02-23T18:42:31.801Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: pipelining replication: peer="{Voter server-934389ce-87b9-b1d7-0fa1-d56c88b6a9fe 934389ce-87b9-b1d7-0fa1-d56c88b6a9fe}"
2025-02-23T18:42:32.807Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:31.801692232 +0000 UTC m=+56.099952901 (1 nodes), 724.593965ms from start of poll, 724.99671ms from cluster start. Timeout at 2025-02-23 18:42:32.806713454 +0000 UTC m=+57.104974131, 1.00502123s after stability
2025-02-23T18:42:32.809Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: starting snapshot up to: index=12
2025-02-23T18:42:32.809Z [INFO]  creating new snapshot: path=/tmp/raft2248841435/snapshots/2-12-1740336152809.tmp
2025-02-23T18:42:32.810Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: snapshot complete up to: index=12
2025-02-23T18:42:32.812Z [INFO]  creating new snapshot: path=/tmp/raft2248841435/snapshots/2-113-1740336152812.tmp
2025-02-23T18:42:32.815Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: copied to local snapshot: bytes=71
2025-02-23T18:42:32.815Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: snapshot restore progress: id=2-113-1740336152812 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:32.815Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: restored user snapshot: index=113
2025-02-23T18:42:32.815Z [ERROR] server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: failed to get log: index=23 error="log not found"
2025-02-23T18:42:32.815Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: aborting pipeline replication: peer="{Voter server-934389ce-87b9-b1d7-0fa1-d56c88b6a9fe 934389ce-87b9-b1d7-0fa1-d56c88b6a9fe}"
2025-02-23T18:42:32.815Z [ERROR] server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: failed to get log: index=23 error="log not found"
2025-02-23T18:42:32.815Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: aborting pipeline replication: peer="{Voter server-1894c342-5c6f-165a-05f8-33f6d74ceaa1 1894c342-5c6f-165a-05f8-33f6d74ceaa1}"
2025-02-23T18:42:32.822Z [ERROR] server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: failed to get log: index=23 error="log not found"
2025-02-23T18:42:32.823Z [INFO]  creating new snapshot: path=/tmp/raft2363612789/snapshots/2-113-1740336152823.tmp
2025-02-23T18:42:32.823Z [INFO]  server-934389ce-87b9-b1d7-0fa1-d56c88b6a9fe: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:32.824Z [INFO]  server-934389ce-87b9-b1d7-0fa1-d56c88b6a9fe: copied to local snapshot: bytes=71
2025-02-23T18:42:32.824Z [INFO]  server-934389ce-87b9-b1d7-0fa1-d56c88b6a9fe: snapshot restore progress: id=2-113-1740336152823 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:32.824Z [INFO]  server-934389ce-87b9-b1d7-0fa1-d56c88b6a9fe: Installed remote snapshot
2025-02-23T18:42:32.824Z [WARN]  server-934389ce-87b9-b1d7-0fa1-d56c88b6a9fe: skipping application of old log: index=22
2025-02-23T18:42:32.824Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: pipelining replication: peer="{Voter server-934389ce-87b9-b1d7-0fa1-d56c88b6a9fe 934389ce-87b9-b1d7-0fa1-d56c88b6a9fe}"
2025-02-23T18:42:32.824Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:42:32.824Z [ERROR] server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: failed to get log: index=23 error="log not found"
2025-02-23T18:42:32.825Z [INFO]  creating new snapshot: path=/tmp/raft3475358006/snapshots/2-113-1740336152825.tmp
2025-02-23T18:42:32.825Z [INFO]  server-1894c342-5c6f-165a-05f8-33f6d74ceaa1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:32.826Z [INFO]  server-1894c342-5c6f-165a-05f8-33f6d74ceaa1: copied to local snapshot: bytes=71
2025-02-23T18:42:32.826Z [INFO]  server-1894c342-5c6f-165a-05f8-33f6d74ceaa1: snapshot restore progress: id=2-113-1740336152825 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:32.826Z [INFO]  server-1894c342-5c6f-165a-05f8-33f6d74ceaa1: Installed remote snapshot
2025-02-23T18:42:32.826Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: pipelining replication: peer="{Voter server-1894c342-5c6f-165a-05f8-33f6d74ceaa1 1894c342-5c6f-165a-05f8-33f6d74ceaa1}"
2025-02-23T18:42:33.830Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:42:32.824658197 +0000 UTC m=+57.122918851 (2 nodes), 1.968µs from start of poll, 1.74796266s from cluster start. Timeout at 2025-02-23 18:42:33.829672096 +0000 UTC m=+58.127932935, 1.005014084s after stability
2025-02-23T18:42:33.843Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: aborting pipeline replication: peer="{Voter server-1894c342-5c6f-165a-05f8-33f6d74ceaa1 1894c342-5c6f-165a-05f8-33f6d74ceaa1}"
2025-02-23T18:42:33.843Z [INFO]  server-4b4f127c-28de-7917-f45c-bc4eeb760cfb: aborting pipeline replication: peer="{Voter server-934389ce-87b9-b1d7-0fa1-d56c88b6a9fe 934389ce-87b9-b1d7-0fa1-d56c88b6a9fe}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false
2025-02-23T18:42:33.846Z [INFO]  server-26dcf5e4-8bed-9058-9d39-ada5318f63a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26dcf5e4-8bed-9058-9d39-ada5318f63a4 Address:26dcf5e4-8bed-9058-9d39-ada5318f63a4} {Suffrage:Voter ID:server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926 Address:6ccdb76e-4e6c-66e6-9861-4f01a5b8f926} {Suffrage:Voter ID:server-e1078696-acef-3ea3-5133-a43c58237d12 Address:e1078696-acef-3ea3-5133-a43c58237d12}]"
2025-02-23T18:42:33.847Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26dcf5e4-8bed-9058-9d39-ada5318f63a4 Address:26dcf5e4-8bed-9058-9d39-ada5318f63a4} {Suffrage:Voter ID:server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926 Address:6ccdb76e-4e6c-66e6-9861-4f01a5b8f926} {Suffrage:Voter ID:server-e1078696-acef-3ea3-5133-a43c58237d12 Address:e1078696-acef-3ea3-5133-a43c58237d12}]"
2025-02-23T18:42:33.847Z [INFO]  server-e1078696-acef-3ea3-5133-a43c58237d12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26dcf5e4-8bed-9058-9d39-ada5318f63a4 Address:26dcf5e4-8bed-9058-9d39-ada5318f63a4} {Suffrage:Voter ID:server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926 Address:6ccdb76e-4e6c-66e6-9861-4f01a5b8f926} {Suffrage:Voter ID:server-e1078696-acef-3ea3-5133-a43c58237d12 Address:e1078696-acef-3ea3-5133-a43c58237d12}]"
2025-02-23T18:42:33.847Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:33.847Z [INFO]  server-26dcf5e4-8bed-9058-9d39-ada5318f63a4: entering follower state: follower="Node at 26dcf5e4-8bed-9058-9d39-ada5318f63a4 [Follower]" leader-address= leader-id=
2025-02-23T18:42:33.847Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: entering follower state: follower="Node at 6ccdb76e-4e6c-66e6-9861-4f01a5b8f926 [Follower]" leader-address= leader-id=
2025-02-23T18:42:33.847Z [INFO]  server-e1078696-acef-3ea3-5133-a43c58237d12: entering follower state: follower="Node at e1078696-acef-3ea3-5133-a43c58237d12 [Follower]" leader-address= leader-id=
2025-02-23T18:42:34.446Z [WARN]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:34.446Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: entering candidate state: node="Node at 6ccdb76e-4e6c-66e6-9861-4f01a5b8f926 [Candidate]" term=2
2025-02-23T18:42:34.446Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: election won: term=2 tally=2
2025-02-23T18:42:34.446Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: entering leader state: leader="Node at 6ccdb76e-4e6c-66e6-9861-4f01a5b8f926 [Leader]"
2025-02-23T18:42:34.446Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: added peer, starting replication: peer=server-26dcf5e4-8bed-9058-9d39-ada5318f63a4
2025-02-23T18:42:34.446Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: added peer, starting replication: peer=server-e1078696-acef-3ea3-5133-a43c58237d12
2025-02-23T18:42:34.446Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: pipelining replication: peer="{Voter server-e1078696-acef-3ea3-5133-a43c58237d12 e1078696-acef-3ea3-5133-a43c58237d12}"
2025-02-23T18:42:34.447Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: pipelining replication: peer="{Voter server-26dcf5e4-8bed-9058-9d39-ada5318f63a4 26dcf5e4-8bed-9058-9d39-ada5318f63a4}"
2025-02-23T18:42:35.452Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:34.447300741 +0000 UTC m=+58.745561466 (1 nodes), 599.934376ms from start of poll, 600.797234ms from cluster start. Timeout at 2025-02-23 18:42:35.45231619 +0000 UTC m=+59.750576857, 1.005015391s after stability
2025-02-23T18:42:35.454Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: starting snapshot up to: index=12
2025-02-23T18:42:35.454Z [INFO]  creating new snapshot: path=/tmp/raft938965388/snapshots/2-12-1740336155454.tmp
2025-02-23T18:42:35.455Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: snapshot complete up to: index=12
2025-02-23T18:42:35.456Z [INFO]  creating new snapshot: path=/tmp/raft938965388/snapshots/2-113-1740336155456.tmp
2025-02-23T18:42:35.458Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: copied to local snapshot: bytes=71
2025-02-23T18:42:35.458Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: snapshot restore progress: id=2-113-1740336155456 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:35.459Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: removing all old logs from log store
2025-02-23T18:42:35.459Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: compacting logs: from=1 to=22
2025-02-23T18:42:35.459Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: restored user snapshot: index=113
2025-02-23T18:42:35.459Z [ERROR] server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: failed to get log: index=22 error="log not found"
2025-02-23T18:42:35.459Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: aborting pipeline replication: peer="{Voter server-e1078696-acef-3ea3-5133-a43c58237d12 e1078696-acef-3ea3-5133-a43c58237d12}"
2025-02-23T18:42:35.459Z [ERROR] server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: failed to get log: index=22 error="log not found"
2025-02-23T18:42:35.459Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: aborting pipeline replication: peer="{Voter server-26dcf5e4-8bed-9058-9d39-ada5318f63a4 26dcf5e4-8bed-9058-9d39-ada5318f63a4}"
2025-02-23T18:42:35.465Z [ERROR] server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: failed to get log: index=22 error="log not found"
2025-02-23T18:42:35.466Z [INFO]  creating new snapshot: path=/tmp/raft4082745223/snapshots/2-113-1740336155466.tmp
2025-02-23T18:42:35.466Z [INFO]  server-e1078696-acef-3ea3-5133-a43c58237d12: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:35.467Z [INFO]  server-e1078696-acef-3ea3-5133-a43c58237d12: copied to local snapshot: bytes=71
2025-02-23T18:42:35.467Z [INFO]  server-e1078696-acef-3ea3-5133-a43c58237d12: snapshot restore progress: id=2-113-1740336155466 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:35.467Z [INFO]  server-e1078696-acef-3ea3-5133-a43c58237d12: removing all old logs from log store
2025-02-23T18:42:35.467Z [INFO]  server-e1078696-acef-3ea3-5133-a43c58237d12: compacting logs: from=1 to=22
2025-02-23T18:42:35.467Z [INFO]  server-e1078696-acef-3ea3-5133-a43c58237d12: Installed remote snapshot
2025-02-23T18:42:35.467Z [WARN]  server-e1078696-acef-3ea3-5133-a43c58237d12: skipping application of old log: index=22
2025-02-23T18:42:35.467Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: pipelining replication: peer="{Voter server-e1078696-acef-3ea3-5133-a43c58237d12 e1078696-acef-3ea3-5133-a43c58237d12}"
2025-02-23T18:42:35.467Z [ERROR] server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: failed to get log: index=22 error="log not found"
2025-02-23T18:42:35.468Z [INFO]  creating new snapshot: path=/tmp/raft3197900403/snapshots/2-113-1740336155468.tmp
2025-02-23T18:42:35.468Z [INFO]  server-26dcf5e4-8bed-9058-9d39-ada5318f63a4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:35.468Z [INFO]  server-26dcf5e4-8bed-9058-9d39-ada5318f63a4: copied to local snapshot: bytes=71
2025-02-23T18:42:35.469Z [INFO]  server-26dcf5e4-8bed-9058-9d39-ada5318f63a4: snapshot restore progress: id=2-113-1740336155468 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:35.469Z [INFO]  server-26dcf5e4-8bed-9058-9d39-ada5318f63a4: removing all old logs from log store
2025-02-23T18:42:35.469Z [INFO]  server-26dcf5e4-8bed-9058-9d39-ada5318f63a4: compacting logs: from=1 to=22
2025-02-23T18:42:35.469Z [INFO]  server-26dcf5e4-8bed-9058-9d39-ada5318f63a4: Installed remote snapshot
2025-02-23T18:42:35.469Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:42:35.470Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: pipelining replication: peer="{Voter server-26dcf5e4-8bed-9058-9d39-ada5318f63a4 26dcf5e4-8bed-9058-9d39-ada5318f63a4}"
2025-02-23T18:42:36.479Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:42:35.46969051 +0000 UTC m=+59.767951139 (2 nodes), 1.845µs from start of poll, 1.623186907s from cluster start. Timeout at 2025-02-23 18:42:36.474720804 +0000 UTC m=+60.772981475, 1.005030336s after stability
2025-02-23T18:42:36.492Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: aborting pipeline replication: peer="{Voter server-e1078696-acef-3ea3-5133-a43c58237d12 e1078696-acef-3ea3-5133-a43c58237d12}"
2025-02-23T18:42:36.492Z [INFO]  server-6ccdb76e-4e6c-66e6-9861-4f01a5b8f926: aborting pipeline replication: peer="{Voter server-26dcf5e4-8bed-9058-9d39-ada5318f63a4 26dcf5e4-8bed-9058-9d39-ada5318f63a4}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true
2025-02-23T18:42:36.494Z [INFO]  server-e5394cc5-2e6b-35e1-29b6-730ebdd901c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e5394cc5-2e6b-35e1-29b6-730ebdd901c5 Address:e5394cc5-2e6b-35e1-29b6-730ebdd901c5} {Suffrage:Voter ID:server-dd3ab79c-f485-eb58-21d4-e071d90ea52b Address:dd3ab79c-f485-eb58-21d4-e071d90ea52b} {Suffrage:Voter ID:server-3d7c64e3-34b5-a8f0-057a-9d177fded1aa Address:3d7c64e3-34b5-a8f0-057a-9d177fded1aa}]"
2025-02-23T18:42:36.494Z [INFO]  server-dd3ab79c-f485-eb58-21d4-e071d90ea52b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e5394cc5-2e6b-35e1-29b6-730ebdd901c5 Address:e5394cc5-2e6b-35e1-29b6-730ebdd901c5} {Suffrage:Voter ID:server-dd3ab79c-f485-eb58-21d4-e071d90ea52b Address:dd3ab79c-f485-eb58-21d4-e071d90ea52b} {Suffrage:Voter ID:server-3d7c64e3-34b5-a8f0-057a-9d177fded1aa Address:3d7c64e3-34b5-a8f0-057a-9d177fded1aa}]"
2025-02-23T18:42:36.494Z [INFO]  server-e5394cc5-2e6b-35e1-29b6-730ebdd901c5: entering follower state: follower="Node at e5394cc5-2e6b-35e1-29b6-730ebdd901c5 [Follower]" leader-address= leader-id=
2025-02-23T18:42:36.495Z [INFO]  server-3d7c64e3-34b5-a8f0-057a-9d177fded1aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e5394cc5-2e6b-35e1-29b6-730ebdd901c5 Address:e5394cc5-2e6b-35e1-29b6-730ebdd901c5} {Suffrage:Voter ID:server-dd3ab79c-f485-eb58-21d4-e071d90ea52b Address:dd3ab79c-f485-eb58-21d4-e071d90ea52b} {Suffrage:Voter ID:server-3d7c64e3-34b5-a8f0-057a-9d177fded1aa Address:3d7c64e3-34b5-a8f0-057a-9d177fded1aa}]"
2025-02-23T18:42:36.495Z [INFO]  server-dd3ab79c-f485-eb58-21d4-e071d90ea52b: entering follower state: follower="Node at dd3ab79c-f485-eb58-21d4-e071d90ea52b [Follower]" leader-address= leader-id=
2025-02-23T18:42:36.495Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:36.495Z [INFO]  server-3d7c64e3-34b5-a8f0-057a-9d177fded1aa: entering follower state: follower="Node at 3d7c64e3-34b5-a8f0-057a-9d177fded1aa [Follower]" leader-address= leader-id=
2025-02-23T18:42:37.002Z [WARN]  server-e5394cc5-2e6b-35e1-29b6-730ebdd901c5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:37.002Z [INFO]  server-e5394cc5-2e6b-35e1-29b6-730ebdd901c5: entering candidate state: node="Node at e5394cc5-2e6b-35e1-29b6-730ebdd901c5 [Candidate]" term=2
2025-02-23T18:42:37.002Z [INFO]  server-e5394cc5-2e6b-35e1-29b6-730ebdd901c5: election won: term=2 tally=2
2025-02-23T18:42:37.002Z [INFO]  server-e5394cc5-2e6b-35e1-29b6-730ebdd901c5: entering leader state: leader="Node at e5394cc5-2e6b-35e1-29b6-730ebdd901c5 [Leader]"
2025-02-23T18:42:37.002Z [INFO]  server-e5394cc5-2e6b-35e1-29b6-730ebdd901c5: added peer, starting replication: peer=server-dd3ab79c-f485-eb58-21d4-e071d90ea52b
2025-02-23T18:42:37.002Z [INFO]  server-e5394cc5-2e6b-35e1-29b6-730ebdd901c5: added peer, starting replication: peer=server-3d7c64e3-34b5-a8f0-057a-9d177fded1aa
2025-02-23T18:42:37.002Z [INFO]  server-e5394cc5-2e6b-35e1-29b6-730ebdd901c5: pipelining replication: peer="{Voter server-dd3ab79c-f485-eb58-21d4-e071d90ea52b dd3ab79c-f485-eb58-21d4-e071d90ea52b}"
2025-02-23T18:42:37.002Z [INFO]  server-e5394cc5-2e6b-35e1-29b6-730ebdd901c5: pipelining replication: peer="{Voter server-3d7c64e3-34b5-a8f0-057a-9d177fded1aa 3d7c64e3-34b5-a8f0-057a-9d177fded1aa}"
2025-02-23T18:42:38.009Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:37.003001561 +0000 UTC m=+61.301262201 (1 nodes), 507.876682ms from start of poll, 508.404337ms from cluster start. Timeout at 2025-02-23 18:42:38.008005845 +0000 UTC m=+62.306266575, 1.005004374s after stability
2025-02-23T18:42:38.010Z [INFO]  server-e5394cc5-2e6b-35e1-29b6-730ebdd901c5: starting snapshot up to: index=12
2025-02-23T18:42:38.010Z [INFO]  creating new snapshot: path=/tmp/raft629178062/snapshots/2-12-1740336158010.tmp
2025-02-23T18:42:38.012Z [INFO]  server-e5394cc5-2e6b-35e1-29b6-730ebdd901c5: snapshot complete up to: index=12
2025-02-23T18:42:38.018Z [INFO]  server-1ca31044-a314-3fef-4b40-29d111a3bff2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ca31044-a314-3fef-4b40-29d111a3bff2 Address:1ca31044-a314-3fef-4b40-29d111a3bff2} {Suffrage:Voter ID:server-f654dea4-ae5b-cd31-4fa6-679df922e6dd Address:f654dea4-ae5b-cd31-4fa6-679df922e6dd} {Suffrage:Voter ID:server-efd9cc6c-c55a-66fc-2520-4a60a589ba56 Address:efd9cc6c-c55a-66fc-2520-4a60a589ba56}]"
2025-02-23T18:42:38.018Z [INFO]  server-1ca31044-a314-3fef-4b40-29d111a3bff2: entering follower state: follower="Node at 1ca31044-a314-3fef-4b40-29d111a3bff2 [Follower]" leader-address= leader-id=
2025-02-23T18:42:38.018Z [INFO]  server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ca31044-a314-3fef-4b40-29d111a3bff2 Address:1ca31044-a314-3fef-4b40-29d111a3bff2} {Suffrage:Voter ID:server-f654dea4-ae5b-cd31-4fa6-679df922e6dd Address:f654dea4-ae5b-cd31-4fa6-679df922e6dd} {Suffrage:Voter ID:server-efd9cc6c-c55a-66fc-2520-4a60a589ba56 Address:efd9cc6c-c55a-66fc-2520-4a60a589ba56}]"
2025-02-23T18:42:38.018Z [INFO]  server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: entering follower state: follower="Node at f654dea4-ae5b-cd31-4fa6-679df922e6dd [Follower]" leader-address= leader-id=
2025-02-23T18:42:38.018Z [INFO]  server-efd9cc6c-c55a-66fc-2520-4a60a589ba56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ca31044-a314-3fef-4b40-29d111a3bff2 Address:1ca31044-a314-3fef-4b40-29d111a3bff2} {Suffrage:Voter ID:server-f654dea4-ae5b-cd31-4fa6-679df922e6dd Address:f654dea4-ae5b-cd31-4fa6-679df922e6dd} {Suffrage:Voter ID:server-efd9cc6c-c55a-66fc-2520-4a60a589ba56 Address:efd9cc6c-c55a-66fc-2520-4a60a589ba56}]"
2025-02-23T18:42:38.018Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:38.018Z [INFO]  server-efd9cc6c-c55a-66fc-2520-4a60a589ba56: entering follower state: follower="Node at efd9cc6c-c55a-66fc-2520-4a60a589ba56 [Follower]" leader-address= leader-id=
2025-02-23T18:42:38.719Z [WARN]  server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:38.719Z [INFO]  server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: entering candidate state: node="Node at f654dea4-ae5b-cd31-4fa6-679df922e6dd [Candidate]" term=2
2025-02-23T18:42:38.719Z [INFO]  server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: election won: term=2 tally=2
2025-02-23T18:42:38.719Z [INFO]  server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: entering leader state: leader="Node at f654dea4-ae5b-cd31-4fa6-679df922e6dd [Leader]"
2025-02-23T18:42:38.719Z [INFO]  server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: added peer, starting replication: peer=server-1ca31044-a314-3fef-4b40-29d111a3bff2
2025-02-23T18:42:38.719Z [INFO]  server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: added peer, starting replication: peer=server-efd9cc6c-c55a-66fc-2520-4a60a589ba56
2025-02-23T18:42:38.719Z [INFO]  server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: pipelining replication: peer="{Voter server-efd9cc6c-c55a-66fc-2520-4a60a589ba56 efd9cc6c-c55a-66fc-2520-4a60a589ba56}"
2025-02-23T18:42:38.720Z [INFO]  server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: pipelining replication: peer="{Voter server-1ca31044-a314-3fef-4b40-29d111a3bff2 1ca31044-a314-3fef-4b40-29d111a3bff2}"
2025-02-23T18:42:39.727Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:38.720240664 +0000 UTC m=+63.018501309 (1 nodes), 701.535453ms from start of poll, 702.118424ms from cluster start. Timeout at 2025-02-23 18:42:39.725244636 +0000 UTC m=+64.023505423, 1.005004114s after stability
2025-02-23T18:42:39.727Z [INFO]  creating new snapshot: path=/tmp/raft1901965605/snapshots/2-113-1740336159727.tmp
2025-02-23T18:42:39.728Z [INFO]  server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: copied to local snapshot: bytes=71
2025-02-23T18:42:39.734Z [INFO]  server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: snapshot restore progress: id=2-113-1740336159727 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:39.736Z [INFO]  server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: restored user snapshot: index=113
2025-02-23T18:42:39.736Z [ERROR] server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: failed to get log: index=3 error="log not found"
2025-02-23T18:42:39.736Z [INFO]  server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: aborting pipeline replication: peer="{Voter server-efd9cc6c-c55a-66fc-2520-4a60a589ba56 efd9cc6c-c55a-66fc-2520-4a60a589ba56}"
2025-02-23T18:42:39.736Z [ERROR] server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: failed to get log: index=3 error="log not found"
2025-02-23T18:42:39.736Z [INFO]  server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: aborting pipeline replication: peer="{Voter server-1ca31044-a314-3fef-4b40-29d111a3bff2 1ca31044-a314-3fef-4b40-29d111a3bff2}"
2025-02-23T18:42:39.745Z [ERROR] server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: failed to get log: index=3 error="log not found"
2025-02-23T18:42:39.745Z [ERROR] server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: failed to get log: index=3 error="log not found"
2025-02-23T18:42:39.746Z [INFO]  creating new snapshot: path=/tmp/raft3155028045/snapshots/2-113-1740336159746.tmp
2025-02-23T18:42:39.746Z [INFO]  creating new snapshot: path=/tmp/raft763582525/snapshots/2-113-1740336159746.tmp
2025-02-23T18:42:39.746Z [INFO]  server-efd9cc6c-c55a-66fc-2520-4a60a589ba56: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:39.746Z [INFO]  server-1ca31044-a314-3fef-4b40-29d111a3bff2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:39.747Z [INFO]  server-efd9cc6c-c55a-66fc-2520-4a60a589ba56: copied to local snapshot: bytes=71
2025-02-23T18:42:39.747Z [INFO]  server-efd9cc6c-c55a-66fc-2520-4a60a589ba56: snapshot restore progress: id=2-113-1740336159746 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:39.747Z [INFO]  server-1ca31044-a314-3fef-4b40-29d111a3bff2: copied to local snapshot: bytes=71
2025-02-23T18:42:39.747Z [INFO]  server-efd9cc6c-c55a-66fc-2520-4a60a589ba56: Installed remote snapshot
2025-02-23T18:42:39.747Z [INFO]  server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: pipelining replication: peer="{Voter server-efd9cc6c-c55a-66fc-2520-4a60a589ba56 efd9cc6c-c55a-66fc-2520-4a60a589ba56}"
2025-02-23T18:42:39.747Z [INFO]  server-1ca31044-a314-3fef-4b40-29d111a3bff2: snapshot restore progress: id=2-113-1740336159746 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:39.747Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:42:39.747Z [INFO]  server-1ca31044-a314-3fef-4b40-29d111a3bff2: Installed remote snapshot
2025-02-23T18:42:39.747Z [INFO]  server-f654dea4-ae5b-cd31-4fa6-679df922e6dd: pipelining replication: peer="{Voter server-1ca31044-a314-3fef-4b40-29d111a3bff2 1ca31044-a314-3fef-4b40-29d111a3bff2}"
2025-02-23T18:42:40.753Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:42:39.747700793 +0000 UTC m=+64.045961465 (2 nodes), 2.041µs from start of poll, 1.72957858s from cluster start. Timeout at 2025-02-23 18:42:40.752706939 +0000 UTC m=+65.050967603, 1.005006138s after stability
2025-02-23T18:42:40.767Z [INFO]  server-e5394cc5-2e6b-35e1-29b6-730ebdd901c5: aborting pipeline replication: peer="{Voter server-dd3ab79c-f485-eb58-21d4-e071d90ea52b dd3ab79c-f485-eb58-21d4-e071d90ea52b}"
2025-02-23T18:42:40.767Z [INFO]  server-e5394cc5-2e6b-35e1-29b6-730ebdd901c5: aborting pipeline replication: peer="{Voter server-3d7c64e3-34b5-a8f0-057a-9d177fded1aa 3d7c64e3-34b5-a8f0-057a-9d177fded1aa}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true
2025-02-23T18:42:40.769Z [INFO]  server-744f309e-14f5-51ac-31d2-0595121cc595: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-744f309e-14f5-51ac-31d2-0595121cc595 Address:744f309e-14f5-51ac-31d2-0595121cc595} {Suffrage:Voter ID:server-65c9732a-93bb-0885-a3b4-f9ccb1799098 Address:65c9732a-93bb-0885-a3b4-f9ccb1799098} {Suffrage:Voter ID:server-38ff05e3-cb76-59ca-a258-3b4d9db2c8a1 Address:38ff05e3-cb76-59ca-a258-3b4d9db2c8a1}]"
2025-02-23T18:42:40.769Z [INFO]  server-65c9732a-93bb-0885-a3b4-f9ccb1799098: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-744f309e-14f5-51ac-31d2-0595121cc595 Address:744f309e-14f5-51ac-31d2-0595121cc595} {Suffrage:Voter ID:server-65c9732a-93bb-0885-a3b4-f9ccb1799098 Address:65c9732a-93bb-0885-a3b4-f9ccb1799098} {Suffrage:Voter ID:server-38ff05e3-cb76-59ca-a258-3b4d9db2c8a1 Address:38ff05e3-cb76-59ca-a258-3b4d9db2c8a1}]"
2025-02-23T18:42:40.769Z [INFO]  server-38ff05e3-cb76-59ca-a258-3b4d9db2c8a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-744f309e-14f5-51ac-31d2-0595121cc595 Address:744f309e-14f5-51ac-31d2-0595121cc595} {Suffrage:Voter ID:server-65c9732a-93bb-0885-a3b4-f9ccb1799098 Address:65c9732a-93bb-0885-a3b4-f9ccb1799098} {Suffrage:Voter ID:server-38ff05e3-cb76-59ca-a258-3b4d9db2c8a1 Address:38ff05e3-cb76-59ca-a258-3b4d9db2c8a1}]"
2025-02-23T18:42:40.769Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:40.769Z [INFO]  server-744f309e-14f5-51ac-31d2-0595121cc595: entering follower state: follower="Node at 744f309e-14f5-51ac-31d2-0595121cc595 [Follower]" leader-address= leader-id=
2025-02-23T18:42:40.769Z [INFO]  server-65c9732a-93bb-0885-a3b4-f9ccb1799098: entering follower state: follower="Node at 65c9732a-93bb-0885-a3b4-f9ccb1799098 [Follower]" leader-address= leader-id=
2025-02-23T18:42:40.769Z [INFO]  server-38ff05e3-cb76-59ca-a258-3b4d9db2c8a1: entering follower state: follower="Node at 38ff05e3-cb76-59ca-a258-3b4d9db2c8a1 [Follower]" leader-address= leader-id=
2025-02-23T18:42:41.305Z [WARN]  server-744f309e-14f5-51ac-31d2-0595121cc595: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:41.305Z [INFO]  server-744f309e-14f5-51ac-31d2-0595121cc595: entering candidate state: node="Node at 744f309e-14f5-51ac-31d2-0595121cc595 [Candidate]" term=2
2025-02-23T18:42:41.305Z [INFO]  server-744f309e-14f5-51ac-31d2-0595121cc595: election won: term=2 tally=2
2025-02-23T18:42:41.305Z [INFO]  server-744f309e-14f5-51ac-31d2-0595121cc595: entering leader state: leader="Node at 744f309e-14f5-51ac-31d2-0595121cc595 [Leader]"
2025-02-23T18:42:41.305Z [INFO]  server-744f309e-14f5-51ac-31d2-0595121cc595: added peer, starting replication: peer=server-65c9732a-93bb-0885-a3b4-f9ccb1799098
2025-02-23T18:42:41.305Z [INFO]  server-744f309e-14f5-51ac-31d2-0595121cc595: added peer, starting replication: peer=server-38ff05e3-cb76-59ca-a258-3b4d9db2c8a1
2025-02-23T18:42:41.305Z [INFO]  server-744f309e-14f5-51ac-31d2-0595121cc595: pipelining replication: peer="{Voter server-38ff05e3-cb76-59ca-a258-3b4d9db2c8a1 38ff05e3-cb76-59ca-a258-3b4d9db2c8a1}"
2025-02-23T18:42:41.305Z [INFO]  server-744f309e-14f5-51ac-31d2-0595121cc595: pipelining replication: peer="{Voter server-65c9732a-93bb-0885-a3b4-f9ccb1799098 65c9732a-93bb-0885-a3b4-f9ccb1799098}"
2025-02-23T18:42:42.314Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:41.305977825 +0000 UTC m=+65.604238549 (1 nodes), 536.298565ms from start of poll, 536.752969ms from cluster start. Timeout at 2025-02-23 18:42:42.31099571 +0000 UTC m=+66.609256351, 1.005017802s after stability
2025-02-23T18:42:42.315Z [INFO]  server-744f309e-14f5-51ac-31d2-0595121cc595: starting snapshot up to: index=12
2025-02-23T18:42:42.315Z [INFO]  creating new snapshot: path=/tmp/raft3835190978/snapshots/2-12-1740336162315.tmp
2025-02-23T18:42:42.316Z [INFO]  server-744f309e-14f5-51ac-31d2-0595121cc595: snapshot complete up to: index=12
2025-02-23T18:42:42.318Z [INFO]  server-dcae6e78-2e0c-4d74-7612-888b943a69de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dcae6e78-2e0c-4d74-7612-888b943a69de Address:dcae6e78-2e0c-4d74-7612-888b943a69de} {Suffrage:Voter ID:server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34 Address:e70929b7-0d15-f6c3-09be-4ffa54cb1e34} {Suffrage:Voter ID:server-8070a8f4-5ebc-bd54-e3e6-4de71b1186d4 Address:8070a8f4-5ebc-bd54-e3e6-4de71b1186d4}]"
2025-02-23T18:42:42.318Z [INFO]  server-dcae6e78-2e0c-4d74-7612-888b943a69de: entering follower state: follower="Node at dcae6e78-2e0c-4d74-7612-888b943a69de [Follower]" leader-address= leader-id=
2025-02-23T18:42:42.318Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dcae6e78-2e0c-4d74-7612-888b943a69de Address:dcae6e78-2e0c-4d74-7612-888b943a69de} {Suffrage:Voter ID:server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34 Address:e70929b7-0d15-f6c3-09be-4ffa54cb1e34} {Suffrage:Voter ID:server-8070a8f4-5ebc-bd54-e3e6-4de71b1186d4 Address:8070a8f4-5ebc-bd54-e3e6-4de71b1186d4}]"
2025-02-23T18:42:42.318Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: entering follower state: follower="Node at e70929b7-0d15-f6c3-09be-4ffa54cb1e34 [Follower]" leader-address= leader-id=
2025-02-23T18:42:42.318Z [INFO]  server-8070a8f4-5ebc-bd54-e3e6-4de71b1186d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dcae6e78-2e0c-4d74-7612-888b943a69de Address:dcae6e78-2e0c-4d74-7612-888b943a69de} {Suffrage:Voter ID:server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34 Address:e70929b7-0d15-f6c3-09be-4ffa54cb1e34} {Suffrage:Voter ID:server-8070a8f4-5ebc-bd54-e3e6-4de71b1186d4 Address:8070a8f4-5ebc-bd54-e3e6-4de71b1186d4}]"
2025-02-23T18:42:42.318Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:42.318Z [INFO]  server-8070a8f4-5ebc-bd54-e3e6-4de71b1186d4: entering follower state: follower="Node at 8070a8f4-5ebc-bd54-e3e6-4de71b1186d4 [Follower]" leader-address= leader-id=
2025-02-23T18:42:42.850Z [WARN]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:42.850Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: entering candidate state: node="Node at e70929b7-0d15-f6c3-09be-4ffa54cb1e34 [Candidate]" term=2
2025-02-23T18:42:42.850Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: election won: term=2 tally=2
2025-02-23T18:42:42.850Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: entering leader state: leader="Node at e70929b7-0d15-f6c3-09be-4ffa54cb1e34 [Leader]"
2025-02-23T18:42:42.850Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: added peer, starting replication: peer=server-dcae6e78-2e0c-4d74-7612-888b943a69de
2025-02-23T18:42:42.850Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: added peer, starting replication: peer=server-8070a8f4-5ebc-bd54-e3e6-4de71b1186d4
2025-02-23T18:42:42.850Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: pipelining replication: peer="{Voter server-8070a8f4-5ebc-bd54-e3e6-4de71b1186d4 8070a8f4-5ebc-bd54-e3e6-4de71b1186d4}"
2025-02-23T18:42:42.850Z [WARN]  server-8070a8f4-5ebc-bd54-e3e6-4de71b1186d4: rejecting vote request since our last term is greater: candidate=e70929b7-0d15-f6c3-09be-4ffa54cb1e34 last-term=2 last-candidate-term=1
2025-02-23T18:42:42.851Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: pipelining replication: peer="{Voter server-dcae6e78-2e0c-4d74-7612-888b943a69de dcae6e78-2e0c-4d74-7612-888b943a69de}"
2025-02-23T18:42:43.857Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:42.85099382 +0000 UTC m=+67.149254781 (1 nodes), 532.410023ms from start of poll, 532.839705ms from cluster start. Timeout at 2025-02-23 18:42:43.856000192 +0000 UTC m=+68.154260856, 1.005006075s after stability
2025-02-23T18:42:43.857Z [INFO]  creating new snapshot: path=/tmp/raft453182338/snapshots/2-113-1740336163857.tmp
2025-02-23T18:42:43.858Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: copied to local snapshot: bytes=71
2025-02-23T18:42:43.858Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: snapshot restore progress: id=2-113-1740336163857 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:43.858Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: removing all old logs from log store
2025-02-23T18:42:43.858Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: compacting logs: from=1 to=2
2025-02-23T18:42:43.858Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: restored user snapshot: index=113
2025-02-23T18:42:43.858Z [ERROR] server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: failed to get log: index=2 error="log not found"
2025-02-23T18:42:43.858Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: aborting pipeline replication: peer="{Voter server-8070a8f4-5ebc-bd54-e3e6-4de71b1186d4 8070a8f4-5ebc-bd54-e3e6-4de71b1186d4}"
2025-02-23T18:42:43.858Z [ERROR] server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: failed to get log: index=2 error="log not found"
2025-02-23T18:42:43.858Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: aborting pipeline replication: peer="{Voter server-dcae6e78-2e0c-4d74-7612-888b943a69de dcae6e78-2e0c-4d74-7612-888b943a69de}"
2025-02-23T18:42:43.868Z [ERROR] server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: failed to get log: index=2 error="log not found"
2025-02-23T18:42:43.868Z [ERROR] server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: failed to get log: index=2 error="log not found"
2025-02-23T18:42:43.869Z [INFO]  creating new snapshot: path=/tmp/raft2683976803/snapshots/2-113-1740336163869.tmp
2025-02-23T18:42:43.869Z [INFO]  server-dcae6e78-2e0c-4d74-7612-888b943a69de: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:43.869Z [INFO]  server-dcae6e78-2e0c-4d74-7612-888b943a69de: copied to local snapshot: bytes=71
2025-02-23T18:42:43.870Z [INFO]  creating new snapshot: path=/tmp/raft1544958621/snapshots/2-113-1740336163870.tmp
2025-02-23T18:42:43.870Z [INFO]  server-8070a8f4-5ebc-bd54-e3e6-4de71b1186d4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:43.870Z [INFO]  server-8070a8f4-5ebc-bd54-e3e6-4de71b1186d4: copied to local snapshot: bytes=71
2025-02-23T18:42:43.870Z [INFO]  server-dcae6e78-2e0c-4d74-7612-888b943a69de: snapshot restore progress: id=2-113-1740336163869 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:43.871Z [INFO]  server-dcae6e78-2e0c-4d74-7612-888b943a69de: removing all old logs from log store
2025-02-23T18:42:43.871Z [INFO]  server-dcae6e78-2e0c-4d74-7612-888b943a69de: compacting logs: from=1 to=2
2025-02-23T18:42:43.871Z [INFO]  server-dcae6e78-2e0c-4d74-7612-888b943a69de: Installed remote snapshot
2025-02-23T18:42:43.871Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: pipelining replication: peer="{Voter server-dcae6e78-2e0c-4d74-7612-888b943a69de dcae6e78-2e0c-4d74-7612-888b943a69de}"
2025-02-23T18:42:43.871Z [INFO]  server-8070a8f4-5ebc-bd54-e3e6-4de71b1186d4: snapshot restore progress: id=2-113-1740336163870 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:43.871Z [INFO]  server-8070a8f4-5ebc-bd54-e3e6-4de71b1186d4: removing all old logs from log store
2025-02-23T18:42:43.871Z [INFO]  server-8070a8f4-5ebc-bd54-e3e6-4de71b1186d4: compacting logs: from=1 to=2
2025-02-23T18:42:43.871Z [INFO]  server-8070a8f4-5ebc-bd54-e3e6-4de71b1186d4: Installed remote snapshot
2025-02-23T18:42:43.871Z [INFO]  server-e70929b7-0d15-f6c3-09be-4ffa54cb1e34: pipelining replication: peer="{Voter server-8070a8f4-5ebc-bd54-e3e6-4de71b1186d4 8070a8f4-5ebc-bd54-e3e6-4de71b1186d4}"
2025-02-23T18:42:43.871Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:42:44.877Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:42:43.872020664 +0000 UTC m=+68.170281385 (2 nodes), 1.988µs from start of poll, 1.553866309s from cluster start. Timeout at 2025-02-23 18:42:44.877063011 +0000 UTC m=+69.175323644, 1.005042259s after stability
2025-02-23T18:42:44.893Z [INFO]  server-744f309e-14f5-51ac-31d2-0595121cc595: aborting pipeline replication: peer="{Voter server-38ff05e3-cb76-59ca-a258-3b4d9db2c8a1 38ff05e3-cb76-59ca-a258-3b4d9db2c8a1}"
2025-02-23T18:42:44.893Z [INFO]  server-744f309e-14f5-51ac-31d2-0595121cc595: aborting pipeline replication: peer="{Voter server-65c9732a-93bb-0885-a3b4-f9ccb1799098 65c9732a-93bb-0885-a3b4-f9ccb1799098}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false
2025-02-23T18:42:44.895Z [INFO]  server-e9d482d3-0039-c69d-4712-c87d1a244242: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9d482d3-0039-c69d-4712-c87d1a244242 Address:e9d482d3-0039-c69d-4712-c87d1a244242} {Suffrage:Voter ID:server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f Address:ebe086dd-a544-3b08-5c7a-b4df6a5b382f} {Suffrage:Voter ID:server-60123652-6790-1193-3f07-a789d32b98ff Address:60123652-6790-1193-3f07-a789d32b98ff}]"
2025-02-23T18:42:44.895Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9d482d3-0039-c69d-4712-c87d1a244242 Address:e9d482d3-0039-c69d-4712-c87d1a244242} {Suffrage:Voter ID:server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f Address:ebe086dd-a544-3b08-5c7a-b4df6a5b382f} {Suffrage:Voter ID:server-60123652-6790-1193-3f07-a789d32b98ff Address:60123652-6790-1193-3f07-a789d32b98ff}]"
2025-02-23T18:42:44.895Z [INFO]  server-60123652-6790-1193-3f07-a789d32b98ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9d482d3-0039-c69d-4712-c87d1a244242 Address:e9d482d3-0039-c69d-4712-c87d1a244242} {Suffrage:Voter ID:server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f Address:ebe086dd-a544-3b08-5c7a-b4df6a5b382f} {Suffrage:Voter ID:server-60123652-6790-1193-3f07-a789d32b98ff Address:60123652-6790-1193-3f07-a789d32b98ff}]"
2025-02-23T18:42:44.895Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:44.896Z [INFO]  server-e9d482d3-0039-c69d-4712-c87d1a244242: entering follower state: follower="Node at e9d482d3-0039-c69d-4712-c87d1a244242 [Follower]" leader-address= leader-id=
2025-02-23T18:42:44.896Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: entering follower state: follower="Node at ebe086dd-a544-3b08-5c7a-b4df6a5b382f [Follower]" leader-address= leader-id=
2025-02-23T18:42:44.896Z [INFO]  server-60123652-6790-1193-3f07-a789d32b98ff: entering follower state: follower="Node at 60123652-6790-1193-3f07-a789d32b98ff [Follower]" leader-address= leader-id=
2025-02-23T18:42:45.464Z [WARN]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:45.464Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: entering candidate state: node="Node at ebe086dd-a544-3b08-5c7a-b4df6a5b382f [Candidate]" term=2
2025-02-23T18:42:45.464Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: election won: term=2 tally=2
2025-02-23T18:42:45.464Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: entering leader state: leader="Node at ebe086dd-a544-3b08-5c7a-b4df6a5b382f [Leader]"
2025-02-23T18:42:45.464Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: added peer, starting replication: peer=server-e9d482d3-0039-c69d-4712-c87d1a244242
2025-02-23T18:42:45.464Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: added peer, starting replication: peer=server-60123652-6790-1193-3f07-a789d32b98ff
2025-02-23T18:42:45.464Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: pipelining replication: peer="{Voter server-60123652-6790-1193-3f07-a789d32b98ff 60123652-6790-1193-3f07-a789d32b98ff}"
2025-02-23T18:42:45.465Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: pipelining replication: peer="{Voter server-e9d482d3-0039-c69d-4712-c87d1a244242 e9d482d3-0039-c69d-4712-c87d1a244242}"
2025-02-23T18:42:46.470Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:45.465107855 +0000 UTC m=+69.763368566 (1 nodes), 569.133349ms from start of poll, 569.540168ms from cluster start. Timeout at 2025-02-23 18:42:46.470131682 +0000 UTC m=+70.768392399, 1.005023833s after stability
2025-02-23T18:42:46.471Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: starting snapshot up to: index=12
2025-02-23T18:42:46.471Z [INFO]  creating new snapshot: path=/tmp/raft2180162490/snapshots/2-12-1740336166471.tmp
2025-02-23T18:42:46.472Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: snapshot complete up to: index=12
2025-02-23T18:42:46.474Z [INFO]  creating new snapshot: path=/tmp/raft2180162490/snapshots/2-1013-1740336166474.tmp
2025-02-23T18:42:46.474Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: copied to local snapshot: bytes=71
2025-02-23T18:42:46.475Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: snapshot restore progress: id=2-1013-1740336166474 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:46.475Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: restored user snapshot: index=1013
2025-02-23T18:42:46.475Z [ERROR] server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: failed to get log: index=23 error="log not found"
2025-02-23T18:42:46.475Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: aborting pipeline replication: peer="{Voter server-60123652-6790-1193-3f07-a789d32b98ff 60123652-6790-1193-3f07-a789d32b98ff}"
2025-02-23T18:42:46.475Z [ERROR] server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: failed to get log: index=23 error="log not found"
2025-02-23T18:42:46.475Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: aborting pipeline replication: peer="{Voter server-e9d482d3-0039-c69d-4712-c87d1a244242 e9d482d3-0039-c69d-4712-c87d1a244242}"
2025-02-23T18:42:46.481Z [ERROR] server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: failed to get log: index=23 error="log not found"
2025-02-23T18:42:46.482Z [INFO]  creating new snapshot: path=/tmp/raft421020727/snapshots/2-1013-1740336166482.tmp
2025-02-23T18:42:46.482Z [INFO]  server-60123652-6790-1193-3f07-a789d32b98ff: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:46.483Z [INFO]  server-60123652-6790-1193-3f07-a789d32b98ff: copied to local snapshot: bytes=71
2025-02-23T18:42:46.483Z [ERROR] server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: failed to get log: index=23 error="log not found"
2025-02-23T18:42:46.488Z [INFO]  server-60123652-6790-1193-3f07-a789d32b98ff: snapshot restore progress: id=2-1013-1740336166482 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:46.488Z [INFO]  server-60123652-6790-1193-3f07-a789d32b98ff: Installed remote snapshot
2025-02-23T18:42:46.489Z [WARN]  server-60123652-6790-1193-3f07-a789d32b98ff: skipping application of old log: index=22
2025-02-23T18:42:46.489Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: pipelining replication: peer="{Voter server-60123652-6790-1193-3f07-a789d32b98ff 60123652-6790-1193-3f07-a789d32b98ff}"
2025-02-23T18:42:46.489Z [INFO]  creating new snapshot: path=/tmp/raft1202099422/snapshots/2-1013-1740336166489.tmp
2025-02-23T18:42:46.489Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:42:46.490Z [INFO]  server-e9d482d3-0039-c69d-4712-c87d1a244242: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:46.490Z [INFO]  server-e9d482d3-0039-c69d-4712-c87d1a244242: copied to local snapshot: bytes=71
2025-02-23T18:42:46.491Z [INFO]  server-e9d482d3-0039-c69d-4712-c87d1a244242: snapshot restore progress: id=2-1013-1740336166489 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:46.491Z [INFO]  server-e9d482d3-0039-c69d-4712-c87d1a244242: Installed remote snapshot
2025-02-23T18:42:46.491Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: pipelining replication: peer="{Voter server-e9d482d3-0039-c69d-4712-c87d1a244242 e9d482d3-0039-c69d-4712-c87d1a244242}"
2025-02-23T18:42:47.495Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:42:46.489879106 +0000 UTC m=+70.788139737 (2 nodes), 2.135µs from start of poll, 1.594311339s from cluster start. Timeout at 2025-02-23 18:42:47.494949848 +0000 UTC m=+71.793210504, 1.005070767s after stability
2025-02-23T18:42:47.507Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: aborting pipeline replication: peer="{Voter server-60123652-6790-1193-3f07-a789d32b98ff 60123652-6790-1193-3f07-a789d32b98ff}"
2025-02-23T18:42:47.507Z [INFO]  server-ebe086dd-a544-3b08-5c7a-b4df6a5b382f: aborting pipeline replication: peer="{Voter server-e9d482d3-0039-c69d-4712-c87d1a244242 e9d482d3-0039-c69d-4712-c87d1a244242}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false
2025-02-23T18:42:47.509Z [INFO]  server-778880ae-61a4-6722-0512-dccf94e4a51b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-778880ae-61a4-6722-0512-dccf94e4a51b Address:778880ae-61a4-6722-0512-dccf94e4a51b} {Suffrage:Voter ID:server-b36f0813-cead-e040-fe8f-7ea945858638 Address:b36f0813-cead-e040-fe8f-7ea945858638} {Suffrage:Voter ID:server-7f2c2602-ae21-17bb-b1bd-500db4cbc931 Address:7f2c2602-ae21-17bb-b1bd-500db4cbc931}]"
2025-02-23T18:42:47.509Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-778880ae-61a4-6722-0512-dccf94e4a51b Address:778880ae-61a4-6722-0512-dccf94e4a51b} {Suffrage:Voter ID:server-b36f0813-cead-e040-fe8f-7ea945858638 Address:b36f0813-cead-e040-fe8f-7ea945858638} {Suffrage:Voter ID:server-7f2c2602-ae21-17bb-b1bd-500db4cbc931 Address:7f2c2602-ae21-17bb-b1bd-500db4cbc931}]"
2025-02-23T18:42:47.509Z [INFO]  server-7f2c2602-ae21-17bb-b1bd-500db4cbc931: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-778880ae-61a4-6722-0512-dccf94e4a51b Address:778880ae-61a4-6722-0512-dccf94e4a51b} {Suffrage:Voter ID:server-b36f0813-cead-e040-fe8f-7ea945858638 Address:b36f0813-cead-e040-fe8f-7ea945858638} {Suffrage:Voter ID:server-7f2c2602-ae21-17bb-b1bd-500db4cbc931 Address:7f2c2602-ae21-17bb-b1bd-500db4cbc931}]"
2025-02-23T18:42:47.509Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:47.509Z [INFO]  server-778880ae-61a4-6722-0512-dccf94e4a51b: entering follower state: follower="Node at 778880ae-61a4-6722-0512-dccf94e4a51b [Follower]" leader-address= leader-id=
2025-02-23T18:42:47.510Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: entering follower state: follower="Node at b36f0813-cead-e040-fe8f-7ea945858638 [Follower]" leader-address= leader-id=
2025-02-23T18:42:47.510Z [INFO]  server-7f2c2602-ae21-17bb-b1bd-500db4cbc931: entering follower state: follower="Node at 7f2c2602-ae21-17bb-b1bd-500db4cbc931 [Follower]" leader-address= leader-id=
2025-02-23T18:42:48.072Z [WARN]  server-b36f0813-cead-e040-fe8f-7ea945858638: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:48.072Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: entering candidate state: node="Node at b36f0813-cead-e040-fe8f-7ea945858638 [Candidate]" term=2
2025-02-23T18:42:48.073Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: election won: term=2 tally=2
2025-02-23T18:42:48.073Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: entering leader state: leader="Node at b36f0813-cead-e040-fe8f-7ea945858638 [Leader]"
2025-02-23T18:42:48.073Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: added peer, starting replication: peer=server-778880ae-61a4-6722-0512-dccf94e4a51b
2025-02-23T18:42:48.073Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: added peer, starting replication: peer=server-7f2c2602-ae21-17bb-b1bd-500db4cbc931
2025-02-23T18:42:48.073Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: pipelining replication: peer="{Voter server-778880ae-61a4-6722-0512-dccf94e4a51b 778880ae-61a4-6722-0512-dccf94e4a51b}"
2025-02-23T18:42:48.073Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: pipelining replication: peer="{Voter server-7f2c2602-ae21-17bb-b1bd-500db4cbc931 7f2c2602-ae21-17bb-b1bd-500db4cbc931}"
2025-02-23T18:42:49.079Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:48.073583195 +0000 UTC m=+72.371843958 (1 nodes), 563.694935ms from start of poll, 564.104805ms from cluster start. Timeout at 2025-02-23 18:42:49.078589278 +0000 UTC m=+73.376849932, 1.005005974s after stability
2025-02-23T18:42:49.081Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: starting snapshot up to: index=12
2025-02-23T18:42:49.081Z [INFO]  creating new snapshot: path=/tmp/raft2667655634/snapshots/2-12-1740336169081.tmp
2025-02-23T18:42:49.082Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: snapshot complete up to: index=12
2025-02-23T18:42:49.091Z [INFO]  creating new snapshot: path=/tmp/raft2667655634/snapshots/2-1013-1740336169091.tmp
2025-02-23T18:42:49.092Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: copied to local snapshot: bytes=71
2025-02-23T18:42:49.092Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: snapshot restore progress: id=2-1013-1740336169091 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:49.092Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: removing all old logs from log store
2025-02-23T18:42:49.092Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: compacting logs: from=1 to=22
2025-02-23T18:42:49.092Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: restored user snapshot: index=1013
2025-02-23T18:42:49.092Z [ERROR] server-b36f0813-cead-e040-fe8f-7ea945858638: failed to get log: index=22 error="log not found"
2025-02-23T18:42:49.092Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: aborting pipeline replication: peer="{Voter server-7f2c2602-ae21-17bb-b1bd-500db4cbc931 7f2c2602-ae21-17bb-b1bd-500db4cbc931}"
2025-02-23T18:42:49.093Z [ERROR] server-b36f0813-cead-e040-fe8f-7ea945858638: failed to get log: index=22 error="log not found"
2025-02-23T18:42:49.093Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: aborting pipeline replication: peer="{Voter server-778880ae-61a4-6722-0512-dccf94e4a51b 778880ae-61a4-6722-0512-dccf94e4a51b}"
2025-02-23T18:42:49.101Z [ERROR] server-b36f0813-cead-e040-fe8f-7ea945858638: failed to get log: index=22 error="log not found"
2025-02-23T18:42:49.101Z [INFO]  creating new snapshot: path=/tmp/raft3183203545/snapshots/2-1013-1740336169101.tmp
2025-02-23T18:42:49.102Z [INFO]  server-778880ae-61a4-6722-0512-dccf94e4a51b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:49.102Z [INFO]  server-778880ae-61a4-6722-0512-dccf94e4a51b: copied to local snapshot: bytes=71
2025-02-23T18:42:49.102Z [INFO]  server-778880ae-61a4-6722-0512-dccf94e4a51b: snapshot restore progress: id=2-1013-1740336169101 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:49.102Z [INFO]  server-778880ae-61a4-6722-0512-dccf94e4a51b: removing all old logs from log store
2025-02-23T18:42:49.102Z [INFO]  server-778880ae-61a4-6722-0512-dccf94e4a51b: compacting logs: from=1 to=22
2025-02-23T18:42:49.102Z [INFO]  server-778880ae-61a4-6722-0512-dccf94e4a51b: Installed remote snapshot
2025-02-23T18:42:49.103Z [WARN]  server-778880ae-61a4-6722-0512-dccf94e4a51b: skipping application of old log: index=22
2025-02-23T18:42:49.103Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: pipelining replication: peer="{Voter server-778880ae-61a4-6722-0512-dccf94e4a51b 778880ae-61a4-6722-0512-dccf94e4a51b}"
2025-02-23T18:42:49.103Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:42:49.103Z [ERROR] server-b36f0813-cead-e040-fe8f-7ea945858638: failed to get log: index=12 error="log not found"
2025-02-23T18:42:49.104Z [INFO]  creating new snapshot: path=/tmp/raft3228428517/snapshots/2-1013-1740336169104.tmp
2025-02-23T18:42:49.104Z [INFO]  server-7f2c2602-ae21-17bb-b1bd-500db4cbc931: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:49.104Z [INFO]  server-7f2c2602-ae21-17bb-b1bd-500db4cbc931: copied to local snapshot: bytes=71
2025-02-23T18:42:49.104Z [INFO]  server-7f2c2602-ae21-17bb-b1bd-500db4cbc931: snapshot restore progress: id=2-1013-1740336169104 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:49.104Z [INFO]  server-7f2c2602-ae21-17bb-b1bd-500db4cbc931: removing all old logs from log store
2025-02-23T18:42:49.104Z [INFO]  server-7f2c2602-ae21-17bb-b1bd-500db4cbc931: compacting logs: from=1 to=22
2025-02-23T18:42:49.104Z [INFO]  server-7f2c2602-ae21-17bb-b1bd-500db4cbc931: Installed remote snapshot
2025-02-23T18:42:49.105Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: pipelining replication: peer="{Voter server-7f2c2602-ae21-17bb-b1bd-500db4cbc931 7f2c2602-ae21-17bb-b1bd-500db4cbc931}"
2025-02-23T18:42:50.108Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:42:49.103264082 +0000 UTC m=+73.401524720 (2 nodes), 1.938µs from start of poll, 1.593785567s from cluster start. Timeout at 2025-02-23 18:42:50.108293857 +0000 UTC m=+74.406554491, 1.005029771s after stability
2025-02-23T18:42:50.121Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: aborting pipeline replication: peer="{Voter server-7f2c2602-ae21-17bb-b1bd-500db4cbc931 7f2c2602-ae21-17bb-b1bd-500db4cbc931}"
2025-02-23T18:42:50.121Z [INFO]  server-b36f0813-cead-e040-fe8f-7ea945858638: aborting pipeline replication: peer="{Voter server-778880ae-61a4-6722-0512-dccf94e4a51b 778880ae-61a4-6722-0512-dccf94e4a51b}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true
2025-02-23T18:42:50.123Z [INFO]  server-4ea37167-a85a-d289-3b5a-ea9b099b353c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ea37167-a85a-d289-3b5a-ea9b099b353c Address:4ea37167-a85a-d289-3b5a-ea9b099b353c} {Suffrage:Voter ID:server-9138ef29-0b9d-0985-86f0-e10226309a2d Address:9138ef29-0b9d-0985-86f0-e10226309a2d} {Suffrage:Voter ID:server-7f5c28de-aa05-1762-fc3b-e66dc6d59a06 Address:7f5c28de-aa05-1762-fc3b-e66dc6d59a06}]"
2025-02-23T18:42:50.123Z [INFO]  server-9138ef29-0b9d-0985-86f0-e10226309a2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ea37167-a85a-d289-3b5a-ea9b099b353c Address:4ea37167-a85a-d289-3b5a-ea9b099b353c} {Suffrage:Voter ID:server-9138ef29-0b9d-0985-86f0-e10226309a2d Address:9138ef29-0b9d-0985-86f0-e10226309a2d} {Suffrage:Voter ID:server-7f5c28de-aa05-1762-fc3b-e66dc6d59a06 Address:7f5c28de-aa05-1762-fc3b-e66dc6d59a06}]"
2025-02-23T18:42:50.124Z [INFO]  server-7f5c28de-aa05-1762-fc3b-e66dc6d59a06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ea37167-a85a-d289-3b5a-ea9b099b353c Address:4ea37167-a85a-d289-3b5a-ea9b099b353c} {Suffrage:Voter ID:server-9138ef29-0b9d-0985-86f0-e10226309a2d Address:9138ef29-0b9d-0985-86f0-e10226309a2d} {Suffrage:Voter ID:server-7f5c28de-aa05-1762-fc3b-e66dc6d59a06 Address:7f5c28de-aa05-1762-fc3b-e66dc6d59a06}]"
2025-02-23T18:42:50.124Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:50.124Z [INFO]  server-4ea37167-a85a-d289-3b5a-ea9b099b353c: entering follower state: follower="Node at 4ea37167-a85a-d289-3b5a-ea9b099b353c [Follower]" leader-address= leader-id=
2025-02-23T18:42:50.124Z [INFO]  server-9138ef29-0b9d-0985-86f0-e10226309a2d: entering follower state: follower="Node at 9138ef29-0b9d-0985-86f0-e10226309a2d [Follower]" leader-address= leader-id=
2025-02-23T18:42:50.124Z [INFO]  server-7f5c28de-aa05-1762-fc3b-e66dc6d59a06: entering follower state: follower="Node at 7f5c28de-aa05-1762-fc3b-e66dc6d59a06 [Follower]" leader-address= leader-id=
2025-02-23T18:42:50.631Z [WARN]  server-7f5c28de-aa05-1762-fc3b-e66dc6d59a06: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:50.631Z [INFO]  server-7f5c28de-aa05-1762-fc3b-e66dc6d59a06: entering candidate state: node="Node at 7f5c28de-aa05-1762-fc3b-e66dc6d59a06 [Candidate]" term=2
2025-02-23T18:42:50.631Z [INFO]  server-7f5c28de-aa05-1762-fc3b-e66dc6d59a06: election won: term=2 tally=2
2025-02-23T18:42:50.631Z [INFO]  server-7f5c28de-aa05-1762-fc3b-e66dc6d59a06: entering leader state: leader="Node at 7f5c28de-aa05-1762-fc3b-e66dc6d59a06 [Leader]"
2025-02-23T18:42:50.631Z [INFO]  server-7f5c28de-aa05-1762-fc3b-e66dc6d59a06: added peer, starting replication: peer=server-4ea37167-a85a-d289-3b5a-ea9b099b353c
2025-02-23T18:42:50.631Z [INFO]  server-7f5c28de-aa05-1762-fc3b-e66dc6d59a06: added peer, starting replication: peer=server-9138ef29-0b9d-0985-86f0-e10226309a2d
2025-02-23T18:42:50.632Z [INFO]  server-7f5c28de-aa05-1762-fc3b-e66dc6d59a06: pipelining replication: peer="{Voter server-9138ef29-0b9d-0985-86f0-e10226309a2d 9138ef29-0b9d-0985-86f0-e10226309a2d}"
2025-02-23T18:42:50.632Z [INFO]  server-7f5c28de-aa05-1762-fc3b-e66dc6d59a06: pipelining replication: peer="{Voter server-4ea37167-a85a-d289-3b5a-ea9b099b353c 4ea37167-a85a-d289-3b5a-ea9b099b353c}"
2025-02-23T18:42:51.637Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:50.631804726 +0000 UTC m=+74.930065674 (1 nodes), 507.659356ms from start of poll, 508.040782ms from cluster start. Timeout at 2025-02-23 18:42:51.636811611 +0000 UTC m=+75.935072361, 1.005006687s after stability
2025-02-23T18:42:51.639Z [INFO]  server-7f5c28de-aa05-1762-fc3b-e66dc6d59a06: starting snapshot up to: index=12
2025-02-23T18:42:51.640Z [INFO]  creating new snapshot: path=/tmp/raft3002829855/snapshots/2-12-1740336171640.tmp
2025-02-23T18:42:51.710Z [INFO]  server-7f5c28de-aa05-1762-fc3b-e66dc6d59a06: snapshot complete up to: index=12
2025-02-23T18:42:51.712Z [INFO]  server-841431e2-8fc5-031e-3577-8c8f5fd46916: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-841431e2-8fc5-031e-3577-8c8f5fd46916 Address:841431e2-8fc5-031e-3577-8c8f5fd46916} {Suffrage:Voter ID:server-81102df5-89af-e95e-4940-6f730046a08b Address:81102df5-89af-e95e-4940-6f730046a08b} {Suffrage:Voter ID:server-8a31acd8-92a0-30aa-a645-4954420e61be Address:8a31acd8-92a0-30aa-a645-4954420e61be}]"
2025-02-23T18:42:51.712Z [INFO]  server-81102df5-89af-e95e-4940-6f730046a08b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-841431e2-8fc5-031e-3577-8c8f5fd46916 Address:841431e2-8fc5-031e-3577-8c8f5fd46916} {Suffrage:Voter ID:server-81102df5-89af-e95e-4940-6f730046a08b Address:81102df5-89af-e95e-4940-6f730046a08b} {Suffrage:Voter ID:server-8a31acd8-92a0-30aa-a645-4954420e61be Address:8a31acd8-92a0-30aa-a645-4954420e61be}]"
2025-02-23T18:42:51.713Z [INFO]  server-8a31acd8-92a0-30aa-a645-4954420e61be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-841431e2-8fc5-031e-3577-8c8f5fd46916 Address:841431e2-8fc5-031e-3577-8c8f5fd46916} {Suffrage:Voter ID:server-81102df5-89af-e95e-4940-6f730046a08b Address:81102df5-89af-e95e-4940-6f730046a08b} {Suffrage:Voter ID:server-8a31acd8-92a0-30aa-a645-4954420e61be Address:8a31acd8-92a0-30aa-a645-4954420e61be}]"
2025-02-23T18:42:51.713Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:51.713Z [INFO]  server-841431e2-8fc5-031e-3577-8c8f5fd46916: entering follower state: follower="Node at 841431e2-8fc5-031e-3577-8c8f5fd46916 [Follower]" leader-address= leader-id=
2025-02-23T18:42:51.713Z [INFO]  server-81102df5-89af-e95e-4940-6f730046a08b: entering follower state: follower="Node at 81102df5-89af-e95e-4940-6f730046a08b [Follower]" leader-address= leader-id=
2025-02-23T18:42:51.713Z [INFO]  server-8a31acd8-92a0-30aa-a645-4954420e61be: entering follower state: follower="Node at 8a31acd8-92a0-30aa-a645-4954420e61be [Follower]" leader-address= leader-id=
2025-02-23T18:42:52.267Z [WARN]  server-841431e2-8fc5-031e-3577-8c8f5fd46916: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:52.267Z [INFO]  server-841431e2-8fc5-031e-3577-8c8f5fd46916: entering candidate state: node="Node at 841431e2-8fc5-031e-3577-8c8f5fd46916 [Candidate]" term=2
2025-02-23T18:42:52.267Z [INFO]  server-841431e2-8fc5-031e-3577-8c8f5fd46916: election won: term=2 tally=2
2025-02-23T18:42:52.267Z [INFO]  server-841431e2-8fc5-031e-3577-8c8f5fd46916: entering leader state: leader="Node at 841431e2-8fc5-031e-3577-8c8f5fd46916 [Leader]"
2025-02-23T18:42:52.267Z [INFO]  server-841431e2-8fc5-031e-3577-8c8f5fd46916: added peer, starting replication: peer=server-81102df5-89af-e95e-4940-6f730046a08b
2025-02-23T18:42:52.267Z [INFO]  server-841431e2-8fc5-031e-3577-8c8f5fd46916: added peer, starting replication: peer=server-8a31acd8-92a0-30aa-a645-4954420e61be
2025-02-23T18:42:52.267Z [INFO]  server-841431e2-8fc5-031e-3577-8c8f5fd46916: pipelining replication: peer="{Voter server-8a31acd8-92a0-30aa-a645-4954420e61be 8a31acd8-92a0-30aa-a645-4954420e61be}"
2025-02-23T18:42:52.267Z [INFO]  server-841431e2-8fc5-031e-3577-8c8f5fd46916: pipelining replication: peer="{Voter server-81102df5-89af-e95e-4940-6f730046a08b 81102df5-89af-e95e-4940-6f730046a08b}"
2025-02-23T18:42:53.273Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:52.26785143 +0000 UTC m=+76.566112138 (1 nodes), 554.754288ms from start of poll, 555.147043ms from cluster start. Timeout at 2025-02-23 18:42:53.272865546 +0000 UTC m=+77.571126179, 1.005014041s after stability
2025-02-23T18:42:53.274Z [INFO]  creating new snapshot: path=/tmp/raft3938427182/snapshots/2-1013-1740336173274.tmp
2025-02-23T18:42:53.274Z [INFO]  server-841431e2-8fc5-031e-3577-8c8f5fd46916: copied to local snapshot: bytes=71
2025-02-23T18:42:53.275Z [INFO]  server-841431e2-8fc5-031e-3577-8c8f5fd46916: snapshot restore progress: id=2-1013-1740336173274 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:53.275Z [INFO]  server-841431e2-8fc5-031e-3577-8c8f5fd46916: restored user snapshot: index=1013
2025-02-23T18:42:53.275Z [ERROR] server-841431e2-8fc5-031e-3577-8c8f5fd46916: failed to get log: index=3 error="log not found"
2025-02-23T18:42:53.275Z [INFO]  server-841431e2-8fc5-031e-3577-8c8f5fd46916: aborting pipeline replication: peer="{Voter server-8a31acd8-92a0-30aa-a645-4954420e61be 8a31acd8-92a0-30aa-a645-4954420e61be}"
2025-02-23T18:42:53.275Z [ERROR] server-841431e2-8fc5-031e-3577-8c8f5fd46916: failed to get log: index=3 error="log not found"
2025-02-23T18:42:53.275Z [INFO]  server-841431e2-8fc5-031e-3577-8c8f5fd46916: aborting pipeline replication: peer="{Voter server-81102df5-89af-e95e-4940-6f730046a08b 81102df5-89af-e95e-4940-6f730046a08b}"
2025-02-23T18:42:53.283Z [ERROR] server-841431e2-8fc5-031e-3577-8c8f5fd46916: failed to get log: index=3 error="log not found"
2025-02-23T18:42:53.283Z [INFO]  creating new snapshot: path=/tmp/raft2019916028/snapshots/2-1013-1740336173283.tmp
2025-02-23T18:42:53.284Z [ERROR] server-841431e2-8fc5-031e-3577-8c8f5fd46916: failed to get log: index=3 error="log not found"
2025-02-23T18:42:53.284Z [INFO]  creating new snapshot: path=/tmp/raft4074006470/snapshots/2-1013-1740336173284.tmp
2025-02-23T18:42:53.284Z [INFO]  server-81102df5-89af-e95e-4940-6f730046a08b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:53.285Z [INFO]  server-81102df5-89af-e95e-4940-6f730046a08b: copied to local snapshot: bytes=71
2025-02-23T18:42:53.285Z [INFO]  server-81102df5-89af-e95e-4940-6f730046a08b: snapshot restore progress: id=2-1013-1740336173283 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:53.285Z [INFO]  server-81102df5-89af-e95e-4940-6f730046a08b: Installed remote snapshot
2025-02-23T18:42:53.285Z [INFO]  server-841431e2-8fc5-031e-3577-8c8f5fd46916: pipelining replication: peer="{Voter server-81102df5-89af-e95e-4940-6f730046a08b 81102df5-89af-e95e-4940-6f730046a08b}"
2025-02-23T18:42:53.285Z [INFO]  server-8a31acd8-92a0-30aa-a645-4954420e61be: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:53.286Z [INFO]  server-8a31acd8-92a0-30aa-a645-4954420e61be: copied to local snapshot: bytes=71
2025-02-23T18:42:53.286Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:42:53.286Z [INFO]  server-8a31acd8-92a0-30aa-a645-4954420e61be: snapshot restore progress: id=2-1013-1740336173284 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:53.286Z [INFO]  server-8a31acd8-92a0-30aa-a645-4954420e61be: Installed remote snapshot
2025-02-23T18:42:53.287Z [INFO]  server-841431e2-8fc5-031e-3577-8c8f5fd46916: pipelining replication: peer="{Voter server-8a31acd8-92a0-30aa-a645-4954420e61be 8a31acd8-92a0-30aa-a645-4954420e61be}"
2025-02-23T18:42:54.292Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:42:53.286398693 +0000 UTC m=+77.584659678 (2 nodes), 2.357µs from start of poll, 1.573694583s from cluster start. Timeout at 2025-02-23 18:42:54.291415989 +0000 UTC m=+78.589676644, 1.005016966s after stability
2025-02-23T18:42:54.313Z [INFO]  server-7f5c28de-aa05-1762-fc3b-e66dc6d59a06: aborting pipeline replication: peer="{Voter server-9138ef29-0b9d-0985-86f0-e10226309a2d 9138ef29-0b9d-0985-86f0-e10226309a2d}"
2025-02-23T18:42:54.313Z [INFO]  server-7f5c28de-aa05-1762-fc3b-e66dc6d59a06: aborting pipeline replication: peer="{Voter server-4ea37167-a85a-d289-3b5a-ea9b099b353c 4ea37167-a85a-d289-3b5a-ea9b099b353c}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true
2025-02-23T18:42:54.315Z [INFO]  server-8ad00926-3275-4832-6324-18347db800fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ad00926-3275-4832-6324-18347db800fa Address:8ad00926-3275-4832-6324-18347db800fa} {Suffrage:Voter ID:server-51ef0a13-ab61-3fef-0f8b-b699673d0f60 Address:51ef0a13-ab61-3fef-0f8b-b699673d0f60} {Suffrage:Voter ID:server-74a3ce00-2a87-c918-60e3-7addb9eb861e Address:74a3ce00-2a87-c918-60e3-7addb9eb861e}]"
2025-02-23T18:42:54.315Z [INFO]  server-51ef0a13-ab61-3fef-0f8b-b699673d0f60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ad00926-3275-4832-6324-18347db800fa Address:8ad00926-3275-4832-6324-18347db800fa} {Suffrage:Voter ID:server-51ef0a13-ab61-3fef-0f8b-b699673d0f60 Address:51ef0a13-ab61-3fef-0f8b-b699673d0f60} {Suffrage:Voter ID:server-74a3ce00-2a87-c918-60e3-7addb9eb861e Address:74a3ce00-2a87-c918-60e3-7addb9eb861e}]"
2025-02-23T18:42:54.315Z [INFO]  server-74a3ce00-2a87-c918-60e3-7addb9eb861e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ad00926-3275-4832-6324-18347db800fa Address:8ad00926-3275-4832-6324-18347db800fa} {Suffrage:Voter ID:server-51ef0a13-ab61-3fef-0f8b-b699673d0f60 Address:51ef0a13-ab61-3fef-0f8b-b699673d0f60} {Suffrage:Voter ID:server-74a3ce00-2a87-c918-60e3-7addb9eb861e Address:74a3ce00-2a87-c918-60e3-7addb9eb861e}]"
2025-02-23T18:42:54.315Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:54.316Z [INFO]  server-8ad00926-3275-4832-6324-18347db800fa: entering follower state: follower="Node at 8ad00926-3275-4832-6324-18347db800fa [Follower]" leader-address= leader-id=
2025-02-23T18:42:54.316Z [INFO]  server-51ef0a13-ab61-3fef-0f8b-b699673d0f60: entering follower state: follower="Node at 51ef0a13-ab61-3fef-0f8b-b699673d0f60 [Follower]" leader-address= leader-id=
2025-02-23T18:42:54.316Z [INFO]  server-74a3ce00-2a87-c918-60e3-7addb9eb861e: entering follower state: follower="Node at 74a3ce00-2a87-c918-60e3-7addb9eb861e [Follower]" leader-address= leader-id=
2025-02-23T18:42:54.845Z [WARN]  server-51ef0a13-ab61-3fef-0f8b-b699673d0f60: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:54.845Z [INFO]  server-51ef0a13-ab61-3fef-0f8b-b699673d0f60: entering candidate state: node="Node at 51ef0a13-ab61-3fef-0f8b-b699673d0f60 [Candidate]" term=2
2025-02-23T18:42:54.845Z [INFO]  server-51ef0a13-ab61-3fef-0f8b-b699673d0f60: election won: term=2 tally=2
2025-02-23T18:42:54.845Z [INFO]  server-51ef0a13-ab61-3fef-0f8b-b699673d0f60: entering leader state: leader="Node at 51ef0a13-ab61-3fef-0f8b-b699673d0f60 [Leader]"
2025-02-23T18:42:54.845Z [INFO]  server-51ef0a13-ab61-3fef-0f8b-b699673d0f60: added peer, starting replication: peer=server-8ad00926-3275-4832-6324-18347db800fa
2025-02-23T18:42:54.845Z [INFO]  server-51ef0a13-ab61-3fef-0f8b-b699673d0f60: added peer, starting replication: peer=server-74a3ce00-2a87-c918-60e3-7addb9eb861e
2025-02-23T18:42:54.845Z [INFO]  server-51ef0a13-ab61-3fef-0f8b-b699673d0f60: pipelining replication: peer="{Voter server-74a3ce00-2a87-c918-60e3-7addb9eb861e 74a3ce00-2a87-c918-60e3-7addb9eb861e}"
2025-02-23T18:42:54.845Z [INFO]  server-51ef0a13-ab61-3fef-0f8b-b699673d0f60: pipelining replication: peer="{Voter server-8ad00926-3275-4832-6324-18347db800fa 8ad00926-3275-4832-6324-18347db800fa}"
2025-02-23T18:42:55.850Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:54.845766532 +0000 UTC m=+79.144027172 (1 nodes), 529.824056ms from start of poll, 530.237555ms from cluster start. Timeout at 2025-02-23 18:42:55.85076921 +0000 UTC m=+80.149030038, 1.005002866s after stability
2025-02-23T18:42:55.852Z [INFO]  server-51ef0a13-ab61-3fef-0f8b-b699673d0f60: starting snapshot up to: index=12
2025-02-23T18:42:55.852Z [INFO]  creating new snapshot: path=/tmp/raft3240624285/snapshots/2-12-1740336175852.tmp
2025-02-23T18:42:55.853Z [INFO]  server-51ef0a13-ab61-3fef-0f8b-b699673d0f60: snapshot complete up to: index=12
2025-02-23T18:42:55.855Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed4acb96-a1ea-b559-44fe-042262a984a9 Address:ed4acb96-a1ea-b559-44fe-042262a984a9} {Suffrage:Voter ID:server-65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6 Address:65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6} {Suffrage:Voter ID:server-d1dcb53e-77db-0843-2be4-58eaa05920e8 Address:d1dcb53e-77db-0843-2be4-58eaa05920e8}]"
2025-02-23T18:42:55.855Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: entering follower state: follower="Node at ed4acb96-a1ea-b559-44fe-042262a984a9 [Follower]" leader-address= leader-id=
2025-02-23T18:42:55.855Z [INFO]  server-65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed4acb96-a1ea-b559-44fe-042262a984a9 Address:ed4acb96-a1ea-b559-44fe-042262a984a9} {Suffrage:Voter ID:server-65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6 Address:65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6} {Suffrage:Voter ID:server-d1dcb53e-77db-0843-2be4-58eaa05920e8 Address:d1dcb53e-77db-0843-2be4-58eaa05920e8}]"
2025-02-23T18:42:55.855Z [INFO]  server-65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6: entering follower state: follower="Node at 65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6 [Follower]" leader-address= leader-id=
2025-02-23T18:42:55.855Z [INFO]  server-d1dcb53e-77db-0843-2be4-58eaa05920e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed4acb96-a1ea-b559-44fe-042262a984a9 Address:ed4acb96-a1ea-b559-44fe-042262a984a9} {Suffrage:Voter ID:server-65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6 Address:65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6} {Suffrage:Voter ID:server-d1dcb53e-77db-0843-2be4-58eaa05920e8 Address:d1dcb53e-77db-0843-2be4-58eaa05920e8}]"
2025-02-23T18:42:55.855Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:55.855Z [INFO]  server-d1dcb53e-77db-0843-2be4-58eaa05920e8: entering follower state: follower="Node at d1dcb53e-77db-0843-2be4-58eaa05920e8 [Follower]" leader-address= leader-id=
2025-02-23T18:42:56.416Z [WARN]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:56.416Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: entering candidate state: node="Node at ed4acb96-a1ea-b559-44fe-042262a984a9 [Candidate]" term=2
2025-02-23T18:42:56.416Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: election won: term=2 tally=2
2025-02-23T18:42:56.416Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: entering leader state: leader="Node at ed4acb96-a1ea-b559-44fe-042262a984a9 [Leader]"
2025-02-23T18:42:56.416Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: added peer, starting replication: peer=server-65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6
2025-02-23T18:42:56.416Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: added peer, starting replication: peer=server-d1dcb53e-77db-0843-2be4-58eaa05920e8
2025-02-23T18:42:56.416Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: pipelining replication: peer="{Voter server-d1dcb53e-77db-0843-2be4-58eaa05920e8 d1dcb53e-77db-0843-2be4-58eaa05920e8}"
2025-02-23T18:42:56.417Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: pipelining replication: peer="{Voter server-65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6 65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6}"
2025-02-23T18:42:57.422Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:56.417199699 +0000 UTC m=+80.715460329 (1 nodes), 561.439586ms from start of poll, 561.855854ms from cluster start. Timeout at 2025-02-23 18:42:57.422202988 +0000 UTC m=+81.720463625, 1.005003296s after stability
2025-02-23T18:42:57.422Z [INFO]  creating new snapshot: path=/tmp/raft4226098421/snapshots/2-1013-1740336177422.tmp
2025-02-23T18:42:57.423Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: copied to local snapshot: bytes=71
2025-02-23T18:42:57.430Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: snapshot restore progress: id=2-1013-1740336177422 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:57.430Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: removing all old logs from log store
2025-02-23T18:42:57.430Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: compacting logs: from=1 to=2
2025-02-23T18:42:57.430Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: restored user snapshot: index=1013
2025-02-23T18:42:57.430Z [ERROR] server-ed4acb96-a1ea-b559-44fe-042262a984a9: failed to get log: index=2 error="log not found"
2025-02-23T18:42:57.430Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: aborting pipeline replication: peer="{Voter server-d1dcb53e-77db-0843-2be4-58eaa05920e8 d1dcb53e-77db-0843-2be4-58eaa05920e8}"
2025-02-23T18:42:57.430Z [ERROR] server-ed4acb96-a1ea-b559-44fe-042262a984a9: failed to get log: index=2 error="log not found"
2025-02-23T18:42:57.430Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: aborting pipeline replication: peer="{Voter server-65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6 65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6}"
2025-02-23T18:42:57.437Z [ERROR] server-ed4acb96-a1ea-b559-44fe-042262a984a9: failed to get log: index=2 error="log not found"
2025-02-23T18:42:57.437Z [INFO]  creating new snapshot: path=/tmp/raft2113474734/snapshots/2-1013-1740336177437.tmp
2025-02-23T18:42:57.438Z [INFO]  server-65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:57.438Z [ERROR] server-ed4acb96-a1ea-b559-44fe-042262a984a9: failed to get log: index=2 error="log not found"
2025-02-23T18:42:57.438Z [INFO]  server-65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6: copied to local snapshot: bytes=71
2025-02-23T18:42:57.438Z [INFO]  creating new snapshot: path=/tmp/raft3526757872/snapshots/2-1013-1740336177438.tmp
2025-02-23T18:42:57.438Z [INFO]  server-65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6: snapshot restore progress: id=2-1013-1740336177437 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:57.438Z [INFO]  server-65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6: removing all old logs from log store
2025-02-23T18:42:57.438Z [INFO]  server-65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6: compacting logs: from=1 to=2
2025-02-23T18:42:57.439Z [INFO]  server-65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6: Installed remote snapshot
2025-02-23T18:42:57.439Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: pipelining replication: peer="{Voter server-65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6 65ec838e-3d7a-95c2-c66d-4da7fdb6d8a6}"
2025-02-23T18:42:57.439Z [INFO]  server-d1dcb53e-77db-0843-2be4-58eaa05920e8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:57.439Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:42:57.439Z [INFO]  server-d1dcb53e-77db-0843-2be4-58eaa05920e8: copied to local snapshot: bytes=71
2025-02-23T18:42:57.440Z [INFO]  server-d1dcb53e-77db-0843-2be4-58eaa05920e8: snapshot restore progress: id=2-1013-1740336177438 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:57.440Z [INFO]  server-d1dcb53e-77db-0843-2be4-58eaa05920e8: removing all old logs from log store
2025-02-23T18:42:57.440Z [INFO]  server-d1dcb53e-77db-0843-2be4-58eaa05920e8: compacting logs: from=1 to=2
2025-02-23T18:42:57.440Z [INFO]  server-d1dcb53e-77db-0843-2be4-58eaa05920e8: Installed remote snapshot
2025-02-23T18:42:57.440Z [INFO]  server-ed4acb96-a1ea-b559-44fe-042262a984a9: pipelining replication: peer="{Voter server-d1dcb53e-77db-0843-2be4-58eaa05920e8 d1dcb53e-77db-0843-2be4-58eaa05920e8}"
2025-02-23T18:42:58.445Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:42:57.439366441 +0000 UTC m=+81.737627075 (2 nodes), 1.862µs from start of poll, 1.5840226s from cluster start. Timeout at 2025-02-23 18:42:58.444404215 +0000 UTC m=+82.742664939, 1.005037864s after stability
2025-02-23T18:42:58.458Z [INFO]  server-51ef0a13-ab61-3fef-0f8b-b699673d0f60: aborting pipeline replication: peer="{Voter server-74a3ce00-2a87-c918-60e3-7addb9eb861e 74a3ce00-2a87-c918-60e3-7addb9eb861e}"
2025-02-23T18:42:58.458Z [INFO]  server-51ef0a13-ab61-3fef-0f8b-b699673d0f60: aborting pipeline replication: peer="{Voter server-8ad00926-3275-4832-6324-18347db800fa 8ad00926-3275-4832-6324-18347db800fa}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false
2025-02-23T18:42:58.459Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1 Address:8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1} {Suffrage:Voter ID:server-86cc2b2c-8c3f-d45f-d389-181f26b3bf5b Address:86cc2b2c-8c3f-d45f-d389-181f26b3bf5b} {Suffrage:Voter ID:server-0fa0b3f3-781d-697c-d0fa-a7dbf22bf297 Address:0fa0b3f3-781d-697c-d0fa-a7dbf22bf297}]"
2025-02-23T18:42:58.460Z [INFO]  server-86cc2b2c-8c3f-d45f-d389-181f26b3bf5b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1 Address:8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1} {Suffrage:Voter ID:server-86cc2b2c-8c3f-d45f-d389-181f26b3bf5b Address:86cc2b2c-8c3f-d45f-d389-181f26b3bf5b} {Suffrage:Voter ID:server-0fa0b3f3-781d-697c-d0fa-a7dbf22bf297 Address:0fa0b3f3-781d-697c-d0fa-a7dbf22bf297}]"
2025-02-23T18:42:58.460Z [INFO]  server-0fa0b3f3-781d-697c-d0fa-a7dbf22bf297: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1 Address:8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1} {Suffrage:Voter ID:server-86cc2b2c-8c3f-d45f-d389-181f26b3bf5b Address:86cc2b2c-8c3f-d45f-d389-181f26b3bf5b} {Suffrage:Voter ID:server-0fa0b3f3-781d-697c-d0fa-a7dbf22bf297 Address:0fa0b3f3-781d-697c-d0fa-a7dbf22bf297}]"
2025-02-23T18:42:58.460Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:42:58.460Z [INFO]  server-86cc2b2c-8c3f-d45f-d389-181f26b3bf5b: entering follower state: follower="Node at 86cc2b2c-8c3f-d45f-d389-181f26b3bf5b [Follower]" leader-address= leader-id=
2025-02-23T18:42:58.460Z [INFO]  server-0fa0b3f3-781d-697c-d0fa-a7dbf22bf297: entering follower state: follower="Node at 0fa0b3f3-781d-697c-d0fa-a7dbf22bf297 [Follower]" leader-address= leader-id=
2025-02-23T18:42:58.460Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: entering follower state: follower="Node at 8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1 [Follower]" leader-address= leader-id=
2025-02-23T18:42:58.975Z [WARN]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:42:58.975Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: entering candidate state: node="Node at 8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1 [Candidate]" term=2
2025-02-23T18:42:58.975Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: election won: term=2 tally=2
2025-02-23T18:42:58.975Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: entering leader state: leader="Node at 8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1 [Leader]"
2025-02-23T18:42:58.975Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: added peer, starting replication: peer=server-86cc2b2c-8c3f-d45f-d389-181f26b3bf5b
2025-02-23T18:42:58.975Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: added peer, starting replication: peer=server-0fa0b3f3-781d-697c-d0fa-a7dbf22bf297
2025-02-23T18:42:58.975Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: pipelining replication: peer="{Voter server-86cc2b2c-8c3f-d45f-d389-181f26b3bf5b 86cc2b2c-8c3f-d45f-d389-181f26b3bf5b}"
2025-02-23T18:42:58.976Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: pipelining replication: peer="{Voter server-0fa0b3f3-781d-697c-d0fa-a7dbf22bf297 0fa0b3f3-781d-697c-d0fa-a7dbf22bf297}"
2025-02-23T18:42:59.981Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:42:58.975799129 +0000 UTC m=+83.274059819 (1 nodes), 515.561276ms from start of poll, 515.975975ms from cluster start. Timeout at 2025-02-23 18:42:59.98080597 +0000 UTC m=+84.279066803, 1.005006984s after stability
2025-02-23T18:42:59.983Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: starting snapshot up to: index=12
2025-02-23T18:42:59.983Z [INFO]  creating new snapshot: path=/tmp/raft1511117462/snapshots/2-12-1740336179983.tmp
2025-02-23T18:42:59.984Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: snapshot complete up to: index=12
2025-02-23T18:42:59.990Z [INFO]  creating new snapshot: path=/tmp/raft1511117462/snapshots/2-10013-1740336179990.tmp
2025-02-23T18:42:59.991Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: copied to local snapshot: bytes=71
2025-02-23T18:42:59.991Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: snapshot restore progress: id=2-10013-1740336179990 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:42:59.991Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: restored user snapshot: index=10013
2025-02-23T18:42:59.991Z [ERROR] server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: failed to get log: index=23 error="log not found"
2025-02-23T18:42:59.991Z [ERROR] server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: failed to get log: index=23 error="log not found"
2025-02-23T18:42:59.991Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: aborting pipeline replication: peer="{Voter server-86cc2b2c-8c3f-d45f-d389-181f26b3bf5b 86cc2b2c-8c3f-d45f-d389-181f26b3bf5b}"
2025-02-23T18:42:59.991Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: aborting pipeline replication: peer="{Voter server-0fa0b3f3-781d-697c-d0fa-a7dbf22bf297 0fa0b3f3-781d-697c-d0fa-a7dbf22bf297}"
2025-02-23T18:42:59.998Z [ERROR] server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: failed to get log: index=23 error="log not found"
2025-02-23T18:42:59.999Z [INFO]  creating new snapshot: path=/tmp/raft2504882281/snapshots/2-10013-1740336179999.tmp
2025-02-23T18:42:59.999Z [INFO]  server-86cc2b2c-8c3f-d45f-d389-181f26b3bf5b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:00.000Z [INFO]  server-86cc2b2c-8c3f-d45f-d389-181f26b3bf5b: copied to local snapshot: bytes=71
2025-02-23T18:43:00.000Z [INFO]  server-86cc2b2c-8c3f-d45f-d389-181f26b3bf5b: snapshot restore progress: id=2-10013-1740336179999 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:00.000Z [ERROR] server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: failed to get log: index=23 error="log not found"
2025-02-23T18:43:00.000Z [INFO]  server-86cc2b2c-8c3f-d45f-d389-181f26b3bf5b: Installed remote snapshot
2025-02-23T18:43:00.000Z [WARN]  server-86cc2b2c-8c3f-d45f-d389-181f26b3bf5b: skipping application of old log: index=22
2025-02-23T18:43:00.000Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: pipelining replication: peer="{Voter server-86cc2b2c-8c3f-d45f-d389-181f26b3bf5b 86cc2b2c-8c3f-d45f-d389-181f26b3bf5b}"
2025-02-23T18:43:00.000Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:00.001Z [INFO]  creating new snapshot: path=/tmp/raft144889056/snapshots/2-10013-1740336180001.tmp
2025-02-23T18:43:00.001Z [INFO]  server-0fa0b3f3-781d-697c-d0fa-a7dbf22bf297: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:00.001Z [INFO]  server-0fa0b3f3-781d-697c-d0fa-a7dbf22bf297: copied to local snapshot: bytes=71
2025-02-23T18:43:00.002Z [INFO]  server-0fa0b3f3-781d-697c-d0fa-a7dbf22bf297: snapshot restore progress: id=2-10013-1740336180001 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:00.002Z [INFO]  server-0fa0b3f3-781d-697c-d0fa-a7dbf22bf297: Installed remote snapshot
2025-02-23T18:43:00.002Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: pipelining replication: peer="{Voter server-0fa0b3f3-781d-697c-d0fa-a7dbf22bf297 0fa0b3f3-781d-697c-d0fa-a7dbf22bf297}"
2025-02-23T18:43:01.006Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:00.000856854 +0000 UTC m=+84.299117491 (2 nodes), 1.874µs from start of poll, 1.541033647s from cluster start. Timeout at 2025-02-23 18:43:01.005945845 +0000 UTC m=+85.304206545, 1.005089054s after stability
2025-02-23T18:43:01.019Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: aborting pipeline replication: peer="{Voter server-86cc2b2c-8c3f-d45f-d389-181f26b3bf5b 86cc2b2c-8c3f-d45f-d389-181f26b3bf5b}"
2025-02-23T18:43:01.019Z [INFO]  server-8b1dfbbc-bcaf-65f3-8b7f-05eec6d264e1: aborting pipeline replication: peer="{Voter server-0fa0b3f3-781d-697c-d0fa-a7dbf22bf297 0fa0b3f3-781d-697c-d0fa-a7dbf22bf297}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false
2025-02-23T18:43:01.021Z [INFO]  server-8af2fe00-e0d6-70cd-bd16-59dc9f96acf4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8af2fe00-e0d6-70cd-bd16-59dc9f96acf4 Address:8af2fe00-e0d6-70cd-bd16-59dc9f96acf4} {Suffrage:Voter ID:server-9b7b0048-fc26-0f67-a4cc-0bb35b994cd5 Address:9b7b0048-fc26-0f67-a4cc-0bb35b994cd5} {Suffrage:Voter ID:server-6397878f-6348-c773-190c-0fa7e2ed5aa4 Address:6397878f-6348-c773-190c-0fa7e2ed5aa4}]"
2025-02-23T18:43:01.021Z [INFO]  server-8af2fe00-e0d6-70cd-bd16-59dc9f96acf4: entering follower state: follower="Node at 8af2fe00-e0d6-70cd-bd16-59dc9f96acf4 [Follower]" leader-address= leader-id=
2025-02-23T18:43:01.021Z [INFO]  server-9b7b0048-fc26-0f67-a4cc-0bb35b994cd5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8af2fe00-e0d6-70cd-bd16-59dc9f96acf4 Address:8af2fe00-e0d6-70cd-bd16-59dc9f96acf4} {Suffrage:Voter ID:server-9b7b0048-fc26-0f67-a4cc-0bb35b994cd5 Address:9b7b0048-fc26-0f67-a4cc-0bb35b994cd5} {Suffrage:Voter ID:server-6397878f-6348-c773-190c-0fa7e2ed5aa4 Address:6397878f-6348-c773-190c-0fa7e2ed5aa4}]"
2025-02-23T18:43:01.021Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8af2fe00-e0d6-70cd-bd16-59dc9f96acf4 Address:8af2fe00-e0d6-70cd-bd16-59dc9f96acf4} {Suffrage:Voter ID:server-9b7b0048-fc26-0f67-a4cc-0bb35b994cd5 Address:9b7b0048-fc26-0f67-a4cc-0bb35b994cd5} {Suffrage:Voter ID:server-6397878f-6348-c773-190c-0fa7e2ed5aa4 Address:6397878f-6348-c773-190c-0fa7e2ed5aa4}]"
2025-02-23T18:43:01.021Z [INFO]  server-9b7b0048-fc26-0f67-a4cc-0bb35b994cd5: entering follower state: follower="Node at 9b7b0048-fc26-0f67-a4cc-0bb35b994cd5 [Follower]" leader-address= leader-id=
2025-02-23T18:43:01.021Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:01.021Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: entering follower state: follower="Node at 6397878f-6348-c773-190c-0fa7e2ed5aa4 [Follower]" leader-address= leader-id=
2025-02-23T18:43:01.576Z [WARN]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:01.576Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: entering candidate state: node="Node at 6397878f-6348-c773-190c-0fa7e2ed5aa4 [Candidate]" term=2
2025-02-23T18:43:01.576Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: election won: term=2 tally=2
2025-02-23T18:43:01.576Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: entering leader state: leader="Node at 6397878f-6348-c773-190c-0fa7e2ed5aa4 [Leader]"
2025-02-23T18:43:01.576Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: added peer, starting replication: peer=server-8af2fe00-e0d6-70cd-bd16-59dc9f96acf4
2025-02-23T18:43:01.576Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: added peer, starting replication: peer=server-9b7b0048-fc26-0f67-a4cc-0bb35b994cd5
2025-02-23T18:43:01.577Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: pipelining replication: peer="{Voter server-8af2fe00-e0d6-70cd-bd16-59dc9f96acf4 8af2fe00-e0d6-70cd-bd16-59dc9f96acf4}"
2025-02-23T18:43:01.577Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: pipelining replication: peer="{Voter server-9b7b0048-fc26-0f67-a4cc-0bb35b994cd5 9b7b0048-fc26-0f67-a4cc-0bb35b994cd5}"
2025-02-23T18:43:02.582Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:01.577219458 +0000 UTC m=+85.875480111 (1 nodes), 555.56482ms from start of poll, 556.034841ms from cluster start. Timeout at 2025-02-23 18:43:02.582224566 +0000 UTC m=+86.880485268, 1.005005157s after stability
2025-02-23T18:43:02.584Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: starting snapshot up to: index=12
2025-02-23T18:43:02.584Z [INFO]  creating new snapshot: path=/tmp/raft4257845749/snapshots/2-12-1740336182584.tmp
2025-02-23T18:43:02.585Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: snapshot complete up to: index=12
2025-02-23T18:43:02.587Z [INFO]  creating new snapshot: path=/tmp/raft4257845749/snapshots/2-10013-1740336182587.tmp
2025-02-23T18:43:02.588Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: copied to local snapshot: bytes=71
2025-02-23T18:43:02.588Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: snapshot restore progress: id=2-10013-1740336182587 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:02.588Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: removing all old logs from log store
2025-02-23T18:43:02.588Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: compacting logs: from=1 to=22
2025-02-23T18:43:02.588Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: restored user snapshot: index=10013
2025-02-23T18:43:02.588Z [ERROR] server-6397878f-6348-c773-190c-0fa7e2ed5aa4: failed to get log: index=22 error="log not found"
2025-02-23T18:43:02.588Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: aborting pipeline replication: peer="{Voter server-9b7b0048-fc26-0f67-a4cc-0bb35b994cd5 9b7b0048-fc26-0f67-a4cc-0bb35b994cd5}"
2025-02-23T18:43:02.588Z [ERROR] server-6397878f-6348-c773-190c-0fa7e2ed5aa4: failed to get log: index=22 error="log not found"
2025-02-23T18:43:02.588Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: aborting pipeline replication: peer="{Voter server-8af2fe00-e0d6-70cd-bd16-59dc9f96acf4 8af2fe00-e0d6-70cd-bd16-59dc9f96acf4}"
2025-02-23T18:43:02.596Z [ERROR] server-6397878f-6348-c773-190c-0fa7e2ed5aa4: failed to get log: index=22 error="log not found"
2025-02-23T18:43:02.596Z [INFO]  creating new snapshot: path=/tmp/raft225768936/snapshots/2-10013-1740336182596.tmp
2025-02-23T18:43:02.596Z [ERROR] server-6397878f-6348-c773-190c-0fa7e2ed5aa4: failed to get log: index=22 error="log not found"
2025-02-23T18:43:02.597Z [INFO]  server-9b7b0048-fc26-0f67-a4cc-0bb35b994cd5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:02.597Z [INFO]  server-9b7b0048-fc26-0f67-a4cc-0bb35b994cd5: copied to local snapshot: bytes=71
2025-02-23T18:43:02.597Z [INFO]  creating new snapshot: path=/tmp/raft3436228050/snapshots/2-10013-1740336182597.tmp
2025-02-23T18:43:02.598Z [INFO]  server-9b7b0048-fc26-0f67-a4cc-0bb35b994cd5: snapshot restore progress: id=2-10013-1740336182596 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:02.598Z [INFO]  server-9b7b0048-fc26-0f67-a4cc-0bb35b994cd5: removing all old logs from log store
2025-02-23T18:43:02.598Z [INFO]  server-9b7b0048-fc26-0f67-a4cc-0bb35b994cd5: compacting logs: from=1 to=22
2025-02-23T18:43:02.598Z [INFO]  server-9b7b0048-fc26-0f67-a4cc-0bb35b994cd5: Installed remote snapshot
2025-02-23T18:43:02.598Z [WARN]  server-9b7b0048-fc26-0f67-a4cc-0bb35b994cd5: skipping application of old log: index=22
2025-02-23T18:43:02.598Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: pipelining replication: peer="{Voter server-9b7b0048-fc26-0f67-a4cc-0bb35b994cd5 9b7b0048-fc26-0f67-a4cc-0bb35b994cd5}"
2025-02-23T18:43:02.598Z [INFO]  server-8af2fe00-e0d6-70cd-bd16-59dc9f96acf4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:02.598Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:02.598Z [INFO]  server-8af2fe00-e0d6-70cd-bd16-59dc9f96acf4: copied to local snapshot: bytes=71
2025-02-23T18:43:02.598Z [INFO]  server-8af2fe00-e0d6-70cd-bd16-59dc9f96acf4: snapshot restore progress: id=2-10013-1740336182597 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:02.599Z [INFO]  server-8af2fe00-e0d6-70cd-bd16-59dc9f96acf4: removing all old logs from log store
2025-02-23T18:43:02.599Z [INFO]  server-8af2fe00-e0d6-70cd-bd16-59dc9f96acf4: compacting logs: from=1 to=22
2025-02-23T18:43:02.599Z [INFO]  server-8af2fe00-e0d6-70cd-bd16-59dc9f96acf4: Installed remote snapshot
2025-02-23T18:43:02.599Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: pipelining replication: peer="{Voter server-8af2fe00-e0d6-70cd-bd16-59dc9f96acf4 8af2fe00-e0d6-70cd-bd16-59dc9f96acf4}"
2025-02-23T18:43:03.604Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:02.598485229 +0000 UTC m=+86.896745892 (2 nodes), 2.03µs from start of poll, 1.577300622s from cluster start. Timeout at 2025-02-23 18:43:03.603500448 +0000 UTC m=+87.901761115, 1.005015223s after stability
2025-02-23T18:43:03.617Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: aborting pipeline replication: peer="{Voter server-9b7b0048-fc26-0f67-a4cc-0bb35b994cd5 9b7b0048-fc26-0f67-a4cc-0bb35b994cd5}"
2025-02-23T18:43:03.617Z [INFO]  server-6397878f-6348-c773-190c-0fa7e2ed5aa4: aborting pipeline replication: peer="{Voter server-8af2fe00-e0d6-70cd-bd16-59dc9f96acf4 8af2fe00-e0d6-70cd-bd16-59dc9f96acf4}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true
2025-02-23T18:43:03.619Z [INFO]  server-9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7 Address:9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7} {Suffrage:Voter ID:server-6e01aed7-64bc-c266-003c-2d5214dc37e0 Address:6e01aed7-64bc-c266-003c-2d5214dc37e0} {Suffrage:Voter ID:server-1d6d69a9-38fb-9935-9be4-99cd81ba9f11 Address:1d6d69a9-38fb-9935-9be4-99cd81ba9f11}]"
2025-02-23T18:43:03.619Z [INFO]  server-9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7: entering follower state: follower="Node at 9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7 [Follower]" leader-address= leader-id=
2025-02-23T18:43:03.619Z [INFO]  server-6e01aed7-64bc-c266-003c-2d5214dc37e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7 Address:9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7} {Suffrage:Voter ID:server-6e01aed7-64bc-c266-003c-2d5214dc37e0 Address:6e01aed7-64bc-c266-003c-2d5214dc37e0} {Suffrage:Voter ID:server-1d6d69a9-38fb-9935-9be4-99cd81ba9f11 Address:1d6d69a9-38fb-9935-9be4-99cd81ba9f11}]"
2025-02-23T18:43:03.619Z [INFO]  server-6e01aed7-64bc-c266-003c-2d5214dc37e0: entering follower state: follower="Node at 6e01aed7-64bc-c266-003c-2d5214dc37e0 [Follower]" leader-address= leader-id=
2025-02-23T18:43:03.619Z [INFO]  server-1d6d69a9-38fb-9935-9be4-99cd81ba9f11: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7 Address:9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7} {Suffrage:Voter ID:server-6e01aed7-64bc-c266-003c-2d5214dc37e0 Address:6e01aed7-64bc-c266-003c-2d5214dc37e0} {Suffrage:Voter ID:server-1d6d69a9-38fb-9935-9be4-99cd81ba9f11 Address:1d6d69a9-38fb-9935-9be4-99cd81ba9f11}]"
2025-02-23T18:43:03.619Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:03.619Z [INFO]  server-1d6d69a9-38fb-9935-9be4-99cd81ba9f11: entering follower state: follower="Node at 1d6d69a9-38fb-9935-9be4-99cd81ba9f11 [Follower]" leader-address= leader-id=
2025-02-23T18:43:04.414Z [WARN]  server-9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:04.414Z [INFO]  server-9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7: entering candidate state: node="Node at 9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7 [Candidate]" term=2
2025-02-23T18:43:04.414Z [INFO]  server-9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7: election won: term=2 tally=2
2025-02-23T18:43:04.414Z [INFO]  server-9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7: entering leader state: leader="Node at 9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7 [Leader]"
2025-02-23T18:43:04.415Z [INFO]  server-9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7: added peer, starting replication: peer=server-6e01aed7-64bc-c266-003c-2d5214dc37e0
2025-02-23T18:43:04.415Z [INFO]  server-9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7: added peer, starting replication: peer=server-1d6d69a9-38fb-9935-9be4-99cd81ba9f11
2025-02-23T18:43:04.415Z [INFO]  server-9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7: pipelining replication: peer="{Voter server-6e01aed7-64bc-c266-003c-2d5214dc37e0 6e01aed7-64bc-c266-003c-2d5214dc37e0}"
2025-02-23T18:43:04.415Z [INFO]  server-9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7: pipelining replication: peer="{Voter server-1d6d69a9-38fb-9935-9be4-99cd81ba9f11 1d6d69a9-38fb-9935-9be4-99cd81ba9f11}"
2025-02-23T18:43:05.421Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:04.415059219 +0000 UTC m=+88.713319891 (1 nodes), 795.089905ms from start of poll, 795.542097ms from cluster start. Timeout at 2025-02-23 18:43:05.420069581 +0000 UTC m=+89.718330217, 1.005010326s after stability
2025-02-23T18:43:05.423Z [INFO]  server-9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7: starting snapshot up to: index=12
2025-02-23T18:43:05.423Z [INFO]  creating new snapshot: path=/tmp/raft378573031/snapshots/2-12-1740336185423.tmp
2025-02-23T18:43:05.424Z [INFO]  server-9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7: snapshot complete up to: index=12
2025-02-23T18:43:05.427Z [INFO]  server-b6c8bf40-a58a-869a-902d-ca94af5bd823: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6c8bf40-a58a-869a-902d-ca94af5bd823 Address:b6c8bf40-a58a-869a-902d-ca94af5bd823} {Suffrage:Voter ID:server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c Address:c6b2a1d1-c3a0-76df-ab91-f8c6423db95c} {Suffrage:Voter ID:server-66435036-5fa3-ab43-4722-cc275efc43dc Address:66435036-5fa3-ab43-4722-cc275efc43dc}]"
2025-02-23T18:43:05.427Z [INFO]  server-b6c8bf40-a58a-869a-902d-ca94af5bd823: entering follower state: follower="Node at b6c8bf40-a58a-869a-902d-ca94af5bd823 [Follower]" leader-address= leader-id=
2025-02-23T18:43:05.427Z [INFO]  server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6c8bf40-a58a-869a-902d-ca94af5bd823 Address:b6c8bf40-a58a-869a-902d-ca94af5bd823} {Suffrage:Voter ID:server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c Address:c6b2a1d1-c3a0-76df-ab91-f8c6423db95c} {Suffrage:Voter ID:server-66435036-5fa3-ab43-4722-cc275efc43dc Address:66435036-5fa3-ab43-4722-cc275efc43dc}]"
2025-02-23T18:43:05.427Z [INFO]  server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: entering follower state: follower="Node at c6b2a1d1-c3a0-76df-ab91-f8c6423db95c [Follower]" leader-address= leader-id=
2025-02-23T18:43:05.427Z [INFO]  server-66435036-5fa3-ab43-4722-cc275efc43dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6c8bf40-a58a-869a-902d-ca94af5bd823 Address:b6c8bf40-a58a-869a-902d-ca94af5bd823} {Suffrage:Voter ID:server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c Address:c6b2a1d1-c3a0-76df-ab91-f8c6423db95c} {Suffrage:Voter ID:server-66435036-5fa3-ab43-4722-cc275efc43dc Address:66435036-5fa3-ab43-4722-cc275efc43dc}]"
2025-02-23T18:43:05.427Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:05.427Z [INFO]  server-66435036-5fa3-ab43-4722-cc275efc43dc: entering follower state: follower="Node at 66435036-5fa3-ab43-4722-cc275efc43dc [Follower]" leader-address= leader-id=
2025-02-23T18:43:06.124Z [WARN]  server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:06.124Z [INFO]  server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: entering candidate state: node="Node at c6b2a1d1-c3a0-76df-ab91-f8c6423db95c [Candidate]" term=2
2025-02-23T18:43:06.124Z [INFO]  server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: election won: term=2 tally=2
2025-02-23T18:43:06.124Z [INFO]  server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: entering leader state: leader="Node at c6b2a1d1-c3a0-76df-ab91-f8c6423db95c [Leader]"
2025-02-23T18:43:06.124Z [INFO]  server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: added peer, starting replication: peer=server-b6c8bf40-a58a-869a-902d-ca94af5bd823
2025-02-23T18:43:06.124Z [INFO]  server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: added peer, starting replication: peer=server-66435036-5fa3-ab43-4722-cc275efc43dc
2025-02-23T18:43:06.124Z [INFO]  server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: pipelining replication: peer="{Voter server-66435036-5fa3-ab43-4722-cc275efc43dc 66435036-5fa3-ab43-4722-cc275efc43dc}"
2025-02-23T18:43:06.124Z [INFO]  server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: pipelining replication: peer="{Voter server-b6c8bf40-a58a-869a-902d-ca94af5bd823 b6c8bf40-a58a-869a-902d-ca94af5bd823}"
2025-02-23T18:43:07.129Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:06.124561552 +0000 UTC m=+90.422822220 (1 nodes), 696.667134ms from start of poll, 697.214187ms from cluster start. Timeout at 2025-02-23 18:43:07.129572056 +0000 UTC m=+91.427832709, 1.005010489s after stability
2025-02-23T18:43:07.130Z [INFO]  creating new snapshot: path=/tmp/raft1666356559/snapshots/2-10013-1740336187129.tmp
2025-02-23T18:43:07.130Z [INFO]  server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: copied to local snapshot: bytes=71
2025-02-23T18:43:07.131Z [INFO]  server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: snapshot restore progress: id=2-10013-1740336187129 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:07.131Z [INFO]  server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: restored user snapshot: index=10013
2025-02-23T18:43:07.131Z [ERROR] server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: failed to get log: index=3 error="log not found"
2025-02-23T18:43:07.131Z [INFO]  server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: aborting pipeline replication: peer="{Voter server-66435036-5fa3-ab43-4722-cc275efc43dc 66435036-5fa3-ab43-4722-cc275efc43dc}"
2025-02-23T18:43:07.131Z [ERROR] server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: failed to get log: index=3 error="log not found"
2025-02-23T18:43:07.131Z [INFO]  server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: aborting pipeline replication: peer="{Voter server-b6c8bf40-a58a-869a-902d-ca94af5bd823 b6c8bf40-a58a-869a-902d-ca94af5bd823}"
2025-02-23T18:43:07.136Z [ERROR] server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: failed to get log: index=3 error="log not found"
2025-02-23T18:43:07.137Z [INFO]  creating new snapshot: path=/tmp/raft1169517746/snapshots/2-10013-1740336187137.tmp
2025-02-23T18:43:07.137Z [INFO]  server-66435036-5fa3-ab43-4722-cc275efc43dc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:07.137Z [ERROR] server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: failed to get log: index=3 error="log not found"
2025-02-23T18:43:07.138Z [INFO]  server-66435036-5fa3-ab43-4722-cc275efc43dc: copied to local snapshot: bytes=71
2025-02-23T18:43:07.138Z [INFO]  server-66435036-5fa3-ab43-4722-cc275efc43dc: snapshot restore progress: id=2-10013-1740336187137 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:07.138Z [INFO]  server-66435036-5fa3-ab43-4722-cc275efc43dc: Installed remote snapshot
2025-02-23T18:43:07.138Z [INFO]  creating new snapshot: path=/tmp/raft3765898612/snapshots/2-10013-1740336187138.tmp
2025-02-23T18:43:07.139Z [INFO]  server-b6c8bf40-a58a-869a-902d-ca94af5bd823: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:07.139Z [INFO]  server-b6c8bf40-a58a-869a-902d-ca94af5bd823: copied to local snapshot: bytes=71
2025-02-23T18:43:07.139Z [INFO]  server-b6c8bf40-a58a-869a-902d-ca94af5bd823: snapshot restore progress: id=2-10013-1740336187138 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:07.139Z [INFO]  server-b6c8bf40-a58a-869a-902d-ca94af5bd823: Installed remote snapshot
2025-02-23T18:43:07.140Z [INFO]  server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: pipelining replication: peer="{Voter server-66435036-5fa3-ab43-4722-cc275efc43dc 66435036-5fa3-ab43-4722-cc275efc43dc}"
2025-02-23T18:43:07.140Z [INFO]  server-c6b2a1d1-c3a0-76df-ab91-f8c6423db95c: pipelining replication: peer="{Voter server-b6c8bf40-a58a-869a-902d-ca94af5bd823 b6c8bf40-a58a-869a-902d-ca94af5bd823}"
2025-02-23T18:43:07.140Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:08.146Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:07.140601002 +0000 UTC m=+91.438861697 (2 nodes), 2.316µs from start of poll, 1.713253664s from cluster start. Timeout at 2025-02-23 18:43:08.145656982 +0000 UTC m=+92.443917630, 1.005055933s after stability
2025-02-23T18:43:08.158Z [INFO]  server-9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7: aborting pipeline replication: peer="{Voter server-1d6d69a9-38fb-9935-9be4-99cd81ba9f11 1d6d69a9-38fb-9935-9be4-99cd81ba9f11}"
2025-02-23T18:43:08.158Z [INFO]  server-9d9975f3-9fe0-ed5d-2225-b7dd4ac60ae7: aborting pipeline replication: peer="{Voter server-6e01aed7-64bc-c266-003c-2d5214dc37e0 6e01aed7-64bc-c266-003c-2d5214dc37e0}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true
2025-02-23T18:43:08.160Z [INFO]  server-5d15d0fe-05fa-51d1-16c5-79f7258eafb6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5d15d0fe-05fa-51d1-16c5-79f7258eafb6 Address:5d15d0fe-05fa-51d1-16c5-79f7258eafb6} {Suffrage:Voter ID:server-cefdbd8b-c511-f2dc-5700-1e795b09406f Address:cefdbd8b-c511-f2dc-5700-1e795b09406f} {Suffrage:Voter ID:server-4cd90e73-42f4-b886-b4ce-242639c9a0a5 Address:4cd90e73-42f4-b886-b4ce-242639c9a0a5}]"
2025-02-23T18:43:08.160Z [INFO]  server-cefdbd8b-c511-f2dc-5700-1e795b09406f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5d15d0fe-05fa-51d1-16c5-79f7258eafb6 Address:5d15d0fe-05fa-51d1-16c5-79f7258eafb6} {Suffrage:Voter ID:server-cefdbd8b-c511-f2dc-5700-1e795b09406f Address:cefdbd8b-c511-f2dc-5700-1e795b09406f} {Suffrage:Voter ID:server-4cd90e73-42f4-b886-b4ce-242639c9a0a5 Address:4cd90e73-42f4-b886-b4ce-242639c9a0a5}]"
2025-02-23T18:43:08.160Z [INFO]  server-5d15d0fe-05fa-51d1-16c5-79f7258eafb6: entering follower state: follower="Node at 5d15d0fe-05fa-51d1-16c5-79f7258eafb6 [Follower]" leader-address= leader-id=
2025-02-23T18:43:08.160Z [INFO]  server-4cd90e73-42f4-b886-b4ce-242639c9a0a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5d15d0fe-05fa-51d1-16c5-79f7258eafb6 Address:5d15d0fe-05fa-51d1-16c5-79f7258eafb6} {Suffrage:Voter ID:server-cefdbd8b-c511-f2dc-5700-1e795b09406f Address:cefdbd8b-c511-f2dc-5700-1e795b09406f} {Suffrage:Voter ID:server-4cd90e73-42f4-b886-b4ce-242639c9a0a5 Address:4cd90e73-42f4-b886-b4ce-242639c9a0a5}]"
2025-02-23T18:43:08.160Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:08.160Z [INFO]  server-cefdbd8b-c511-f2dc-5700-1e795b09406f: entering follower state: follower="Node at cefdbd8b-c511-f2dc-5700-1e795b09406f [Follower]" leader-address= leader-id=
2025-02-23T18:43:08.160Z [INFO]  server-4cd90e73-42f4-b886-b4ce-242639c9a0a5: entering follower state: follower="Node at 4cd90e73-42f4-b886-b4ce-242639c9a0a5 [Follower]" leader-address= leader-id=
2025-02-23T18:43:08.791Z [WARN]  server-cefdbd8b-c511-f2dc-5700-1e795b09406f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:08.791Z [INFO]  server-cefdbd8b-c511-f2dc-5700-1e795b09406f: entering candidate state: node="Node at cefdbd8b-c511-f2dc-5700-1e795b09406f [Candidate]" term=2
2025-02-23T18:43:08.791Z [INFO]  server-cefdbd8b-c511-f2dc-5700-1e795b09406f: election won: term=2 tally=2
2025-02-23T18:43:08.791Z [INFO]  server-cefdbd8b-c511-f2dc-5700-1e795b09406f: entering leader state: leader="Node at cefdbd8b-c511-f2dc-5700-1e795b09406f [Leader]"
2025-02-23T18:43:08.791Z [INFO]  server-cefdbd8b-c511-f2dc-5700-1e795b09406f: added peer, starting replication: peer=server-5d15d0fe-05fa-51d1-16c5-79f7258eafb6
2025-02-23T18:43:08.791Z [INFO]  server-cefdbd8b-c511-f2dc-5700-1e795b09406f: added peer, starting replication: peer=server-4cd90e73-42f4-b886-b4ce-242639c9a0a5
2025-02-23T18:43:08.791Z [INFO]  server-cefdbd8b-c511-f2dc-5700-1e795b09406f: pipelining replication: peer="{Voter server-5d15d0fe-05fa-51d1-16c5-79f7258eafb6 5d15d0fe-05fa-51d1-16c5-79f7258eafb6}"
2025-02-23T18:43:08.791Z [INFO]  server-cefdbd8b-c511-f2dc-5700-1e795b09406f: pipelining replication: peer="{Voter server-4cd90e73-42f4-b886-b4ce-242639c9a0a5 4cd90e73-42f4-b886-b4ce-242639c9a0a5}"
2025-02-23T18:43:09.798Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:08.79202084 +0000 UTC m=+93.090281492 (1 nodes), 631.18053ms from start of poll, 631.596665ms from cluster start. Timeout at 2025-02-23 18:43:09.797049115 +0000 UTC m=+94.095309903, 1.005028411s after stability
2025-02-23T18:43:09.799Z [INFO]  server-cefdbd8b-c511-f2dc-5700-1e795b09406f: starting snapshot up to: index=12
2025-02-23T18:43:09.799Z [INFO]  creating new snapshot: path=/tmp/raft2627092135/snapshots/2-12-1740336189799.tmp
2025-02-23T18:43:09.800Z [INFO]  server-cefdbd8b-c511-f2dc-5700-1e795b09406f: snapshot complete up to: index=12
2025-02-23T18:43:09.802Z [INFO]  server-12c3b779-07b2-4d56-ee25-0fda5b2306bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12c3b779-07b2-4d56-ee25-0fda5b2306bb Address:12c3b779-07b2-4d56-ee25-0fda5b2306bb} {Suffrage:Voter ID:server-27af36f1-c2b3-9d62-8792-a86ba921fd10 Address:27af36f1-c2b3-9d62-8792-a86ba921fd10} {Suffrage:Voter ID:server-4cc0666e-20c6-ba95-5c8d-15739ca72845 Address:4cc0666e-20c6-ba95-5c8d-15739ca72845}]"
2025-02-23T18:43:09.802Z [INFO]  server-27af36f1-c2b3-9d62-8792-a86ba921fd10: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12c3b779-07b2-4d56-ee25-0fda5b2306bb Address:12c3b779-07b2-4d56-ee25-0fda5b2306bb} {Suffrage:Voter ID:server-27af36f1-c2b3-9d62-8792-a86ba921fd10 Address:27af36f1-c2b3-9d62-8792-a86ba921fd10} {Suffrage:Voter ID:server-4cc0666e-20c6-ba95-5c8d-15739ca72845 Address:4cc0666e-20c6-ba95-5c8d-15739ca72845}]"
2025-02-23T18:43:09.803Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12c3b779-07b2-4d56-ee25-0fda5b2306bb Address:12c3b779-07b2-4d56-ee25-0fda5b2306bb} {Suffrage:Voter ID:server-27af36f1-c2b3-9d62-8792-a86ba921fd10 Address:27af36f1-c2b3-9d62-8792-a86ba921fd10} {Suffrage:Voter ID:server-4cc0666e-20c6-ba95-5c8d-15739ca72845 Address:4cc0666e-20c6-ba95-5c8d-15739ca72845}]"
2025-02-23T18:43:09.803Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:09.803Z [INFO]  server-12c3b779-07b2-4d56-ee25-0fda5b2306bb: entering follower state: follower="Node at 12c3b779-07b2-4d56-ee25-0fda5b2306bb [Follower]" leader-address= leader-id=
2025-02-23T18:43:09.803Z [INFO]  server-27af36f1-c2b3-9d62-8792-a86ba921fd10: entering follower state: follower="Node at 27af36f1-c2b3-9d62-8792-a86ba921fd10 [Follower]" leader-address= leader-id=
2025-02-23T18:43:09.803Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: entering follower state: follower="Node at 4cc0666e-20c6-ba95-5c8d-15739ca72845 [Follower]" leader-address= leader-id=
2025-02-23T18:43:10.355Z [WARN]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:10.355Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: entering candidate state: node="Node at 4cc0666e-20c6-ba95-5c8d-15739ca72845 [Candidate]" term=2
2025-02-23T18:43:10.356Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: election won: term=2 tally=2
2025-02-23T18:43:10.356Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: entering leader state: leader="Node at 4cc0666e-20c6-ba95-5c8d-15739ca72845 [Leader]"
2025-02-23T18:43:10.356Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: added peer, starting replication: peer=server-12c3b779-07b2-4d56-ee25-0fda5b2306bb
2025-02-23T18:43:10.356Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: added peer, starting replication: peer=server-27af36f1-c2b3-9d62-8792-a86ba921fd10
2025-02-23T18:43:10.356Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: pipelining replication: peer="{Voter server-27af36f1-c2b3-9d62-8792-a86ba921fd10 27af36f1-c2b3-9d62-8792-a86ba921fd10}"
2025-02-23T18:43:10.356Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: pipelining replication: peer="{Voter server-12c3b779-07b2-4d56-ee25-0fda5b2306bb 12c3b779-07b2-4d56-ee25-0fda5b2306bb}"
2025-02-23T18:43:11.361Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:10.356513796 +0000 UTC m=+94.654774420 (1 nodes), 553.414898ms from start of poll, 553.855464ms from cluster start. Timeout at 2025-02-23 18:43:11.361519881 +0000 UTC m=+95.659780579, 1.005006159s after stability
2025-02-23T18:43:11.362Z [INFO]  creating new snapshot: path=/tmp/raft1981041260/snapshots/2-10013-1740336191362.tmp
2025-02-23T18:43:11.362Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: copied to local snapshot: bytes=71
2025-02-23T18:43:11.363Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: snapshot restore progress: id=2-10013-1740336191362 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:11.363Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: removing all old logs from log store
2025-02-23T18:43:11.363Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: compacting logs: from=1 to=2
2025-02-23T18:43:11.363Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: restored user snapshot: index=10013
2025-02-23T18:43:11.363Z [ERROR] server-4cc0666e-20c6-ba95-5c8d-15739ca72845: failed to get log: index=2 error="log not found"
2025-02-23T18:43:11.363Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: aborting pipeline replication: peer="{Voter server-27af36f1-c2b3-9d62-8792-a86ba921fd10 27af36f1-c2b3-9d62-8792-a86ba921fd10}"
2025-02-23T18:43:11.363Z [ERROR] server-4cc0666e-20c6-ba95-5c8d-15739ca72845: failed to get log: index=2 error="log not found"
2025-02-23T18:43:11.363Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: aborting pipeline replication: peer="{Voter server-12c3b779-07b2-4d56-ee25-0fda5b2306bb 12c3b779-07b2-4d56-ee25-0fda5b2306bb}"
2025-02-23T18:43:11.370Z [ERROR] server-4cc0666e-20c6-ba95-5c8d-15739ca72845: failed to get log: index=2 error="log not found"
2025-02-23T18:43:11.371Z [INFO]  creating new snapshot: path=/tmp/raft2425610615/snapshots/2-10013-1740336191370.tmp
2025-02-23T18:43:11.371Z [INFO]  server-27af36f1-c2b3-9d62-8792-a86ba921fd10: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:11.371Z [INFO]  server-27af36f1-c2b3-9d62-8792-a86ba921fd10: copied to local snapshot: bytes=71
2025-02-23T18:43:11.371Z [INFO]  server-27af36f1-c2b3-9d62-8792-a86ba921fd10: snapshot restore progress: id=2-10013-1740336191370 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:11.371Z [INFO]  server-27af36f1-c2b3-9d62-8792-a86ba921fd10: removing all old logs from log store
2025-02-23T18:43:11.371Z [INFO]  server-27af36f1-c2b3-9d62-8792-a86ba921fd10: compacting logs: from=1 to=2
2025-02-23T18:43:11.372Z [INFO]  server-27af36f1-c2b3-9d62-8792-a86ba921fd10: Installed remote snapshot
2025-02-23T18:43:11.372Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: pipelining replication: peer="{Voter server-27af36f1-c2b3-9d62-8792-a86ba921fd10 27af36f1-c2b3-9d62-8792-a86ba921fd10}"
2025-02-23T18:43:11.372Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:11.373Z [ERROR] server-4cc0666e-20c6-ba95-5c8d-15739ca72845: failed to get log: index=2 error="log not found"
2025-02-23T18:43:11.373Z [INFO]  creating new snapshot: path=/tmp/raft3414966827/snapshots/2-10013-1740336191373.tmp
2025-02-23T18:43:11.374Z [INFO]  server-12c3b779-07b2-4d56-ee25-0fda5b2306bb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:11.374Z [INFO]  server-12c3b779-07b2-4d56-ee25-0fda5b2306bb: copied to local snapshot: bytes=71
2025-02-23T18:43:11.374Z [INFO]  server-12c3b779-07b2-4d56-ee25-0fda5b2306bb: snapshot restore progress: id=2-10013-1740336191373 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-02-23T18:43:11.374Z [INFO]  server-12c3b779-07b2-4d56-ee25-0fda5b2306bb: removing all old logs from log store
2025-02-23T18:43:11.374Z [INFO]  server-12c3b779-07b2-4d56-ee25-0fda5b2306bb: compacting logs: from=1 to=2
2025-02-23T18:43:11.374Z [INFO]  server-12c3b779-07b2-4d56-ee25-0fda5b2306bb: Installed remote snapshot
2025-02-23T18:43:11.375Z [INFO]  server-4cc0666e-20c6-ba95-5c8d-15739ca72845: pipelining replication: peer="{Voter server-12c3b779-07b2-4d56-ee25-0fda5b2306bb 12c3b779-07b2-4d56-ee25-0fda5b2306bb}"
2025-02-23T18:43:12.378Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:11.372345491 +0000 UTC m=+95.670606416 (2 nodes), 2.266µs from start of poll, 1.56968746s from cluster start. Timeout at 2025-02-23 18:43:12.377392458 +0000 UTC m=+96.675653254, 1.005046838s after stability
2025-02-23T18:43:12.391Z [INFO]  server-cefdbd8b-c511-f2dc-5700-1e795b09406f: aborting pipeline replication: peer="{Voter server-4cd90e73-42f4-b886-b4ce-242639c9a0a5 4cd90e73-42f4-b886-b4ce-242639c9a0a5}"
2025-02-23T18:43:12.391Z [INFO]  server-cefdbd8b-c511-f2dc-5700-1e795b09406f: aborting pipeline replication: peer="{Voter server-5d15d0fe-05fa-51d1-16c5-79f7258eafb6 5d15d0fe-05fa-51d1-16c5-79f7258eafb6}"
--- PASS: TestRaft_UserRestore (82.98s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.68s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.93s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.13s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.27s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.64s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.56s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.29s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.19s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.68s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.57s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.21s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.52s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.77s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.65s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.27s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.13s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.19s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.14s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.56s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.60s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.54s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.23s)
=== RUN   TestRaft_SendSnapshotFollower
2025-02-23T18:43:12.395Z [INFO]  server-e67f8485-5460-fb40-d4a8-3de912e6f0df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e67f8485-5460-fb40-d4a8-3de912e6f0df Address:e67f8485-5460-fb40-d4a8-3de912e6f0df} {Suffrage:Voter ID:server-5d058d9c-3fc7-86e5-273f-72b2608eaa79 Address:5d058d9c-3fc7-86e5-273f-72b2608eaa79} {Suffrage:Voter ID:server-d8d61fc5-f874-6ade-0e9a-e16a420b3ed6 Address:d8d61fc5-f874-6ade-0e9a-e16a420b3ed6}]"
2025-02-23T18:43:12.395Z [INFO]  server-e67f8485-5460-fb40-d4a8-3de912e6f0df: entering follower state: follower="Node at e67f8485-5460-fb40-d4a8-3de912e6f0df [Follower]" leader-address= leader-id=
2025-02-23T18:43:12.395Z [INFO]  server-5d058d9c-3fc7-86e5-273f-72b2608eaa79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e67f8485-5460-fb40-d4a8-3de912e6f0df Address:e67f8485-5460-fb40-d4a8-3de912e6f0df} {Suffrage:Voter ID:server-5d058d9c-3fc7-86e5-273f-72b2608eaa79 Address:5d058d9c-3fc7-86e5-273f-72b2608eaa79} {Suffrage:Voter ID:server-d8d61fc5-f874-6ade-0e9a-e16a420b3ed6 Address:d8d61fc5-f874-6ade-0e9a-e16a420b3ed6}]"
2025-02-23T18:43:12.395Z [INFO]  server-d8d61fc5-f874-6ade-0e9a-e16a420b3ed6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e67f8485-5460-fb40-d4a8-3de912e6f0df Address:e67f8485-5460-fb40-d4a8-3de912e6f0df} {Suffrage:Voter ID:server-5d058d9c-3fc7-86e5-273f-72b2608eaa79 Address:5d058d9c-3fc7-86e5-273f-72b2608eaa79} {Suffrage:Voter ID:server-d8d61fc5-f874-6ade-0e9a-e16a420b3ed6 Address:d8d61fc5-f874-6ade-0e9a-e16a420b3ed6}]"
2025-02-23T18:43:12.395Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:12.395Z [INFO]  server-5d058d9c-3fc7-86e5-273f-72b2608eaa79: entering follower state: follower="Node at 5d058d9c-3fc7-86e5-273f-72b2608eaa79 [Follower]" leader-address= leader-id=
2025-02-23T18:43:12.395Z [INFO]  server-d8d61fc5-f874-6ade-0e9a-e16a420b3ed6: entering follower state: follower="Node at d8d61fc5-f874-6ade-0e9a-e16a420b3ed6 [Follower]" leader-address= leader-id=
2025-02-23T18:43:12.450Z [WARN]  server-e67f8485-5460-fb40-d4a8-3de912e6f0df: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:12.450Z [INFO]  server-e67f8485-5460-fb40-d4a8-3de912e6f0df: entering candidate state: node="Node at e67f8485-5460-fb40-d4a8-3de912e6f0df [Candidate]" term=2
2025-02-23T18:43:12.450Z [INFO]  server-e67f8485-5460-fb40-d4a8-3de912e6f0df: election won: term=2 tally=2
2025-02-23T18:43:12.450Z [INFO]  server-e67f8485-5460-fb40-d4a8-3de912e6f0df: entering leader state: leader="Node at e67f8485-5460-fb40-d4a8-3de912e6f0df [Leader]"
2025-02-23T18:43:12.450Z [INFO]  server-e67f8485-5460-fb40-d4a8-3de912e6f0df: added peer, starting replication: peer=server-5d058d9c-3fc7-86e5-273f-72b2608eaa79
2025-02-23T18:43:12.450Z [INFO]  server-e67f8485-5460-fb40-d4a8-3de912e6f0df: added peer, starting replication: peer=server-d8d61fc5-f874-6ade-0e9a-e16a420b3ed6
2025-02-23T18:43:12.450Z [INFO]  server-e67f8485-5460-fb40-d4a8-3de912e6f0df: pipelining replication: peer="{Voter server-d8d61fc5-f874-6ade-0e9a-e16a420b3ed6 d8d61fc5-f874-6ade-0e9a-e16a420b3ed6}"
2025-02-23T18:43:12.450Z [INFO]  server-e67f8485-5460-fb40-d4a8-3de912e6f0df: pipelining replication: peer="{Voter server-5d058d9c-3fc7-86e5-273f-72b2608eaa79 5d058d9c-3fc7-86e5-273f-72b2608eaa79}"
2025-02-23T18:43:12.555Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:12.450590737 +0000 UTC m=+96.748851820 (2 nodes), 55.023161ms from start of poll, 55.48964ms from cluster start. Timeout at 2025-02-23 18:43:12.555603626 +0000 UTC m=+96.853864318, 105.012498ms after stability
2025-02-23T18:43:12.555Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:12.661Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:12.555859548 +0000 UTC m=+96.854120196 (1 nodes), 2.002µs from start of poll, 160.758016ms from cluster start. Timeout at 2025-02-23 18:43:12.660949632 +0000 UTC m=+96.959210245, 105.090049ms after stability
2025-02-23T18:43:12.661Z [ERROR] server-e67f8485-5460-fb40-d4a8-3de912e6f0df: failed to pipeline appendEntries: peer="{Voter server-5d058d9c-3fc7-86e5-273f-72b2608eaa79 5d058d9c-3fc7-86e5-273f-72b2608eaa79}" error="append pipeline closed"
2025-02-23T18:43:12.661Z [INFO]  server-e67f8485-5460-fb40-d4a8-3de912e6f0df: aborting pipeline replication: peer="{Voter server-5d058d9c-3fc7-86e5-273f-72b2608eaa79 5d058d9c-3fc7-86e5-273f-72b2608eaa79}"
2025-02-23T18:43:12.662Z [ERROR] server-e67f8485-5460-fb40-d4a8-3de912e6f0df: failed to appendEntries to: peer="{Voter server-5d058d9c-3fc7-86e5-273f-72b2608eaa79 5d058d9c-3fc7-86e5-273f-72b2608eaa79}" error="failed to connect to peer: 5d058d9c-3fc7-86e5-273f-72b2608eaa79"
2025-02-23T18:43:12.669Z [ERROR] server-e67f8485-5460-fb40-d4a8-3de912e6f0df: failed to heartbeat to: peer=5d058d9c-3fc7-86e5-273f-72b2608eaa79 backoff time=10ms error="failed to connect to peer: 5d058d9c-3fc7-86e5-273f-72b2608eaa79"
    raft_test.go:1627: [INFO] Finished apply without behind follower
2025-02-23T18:43:12.670Z [INFO]  server-e67f8485-5460-fb40-d4a8-3de912e6f0df: starting snapshot up to: index=102
2025-02-23T18:43:12.670Z [INFO]  creating new snapshot: path=/tmp/raft4217238271/snapshots/2-102-1740336192670.tmp
2025-02-23T18:43:12.671Z [INFO]  server-e67f8485-5460-fb40-d4a8-3de912e6f0df: compacting logs: from=1 to=92
2025-02-23T18:43:12.671Z [INFO]  server-e67f8485-5460-fb40-d4a8-3de912e6f0df: snapshot complete up to: index=102
2025-02-23T18:43:12.671Z [ERROR] server-5d058d9c-3fc7-86e5-273f-72b2608eaa79: failed to take snapshot: error="nothing new to snapshot"
2025-02-23T18:43:12.671Z [INFO]  server-d8d61fc5-f874-6ade-0e9a-e16a420b3ed6: starting snapshot up to: index=99
2025-02-23T18:43:12.671Z [INFO]  creating new snapshot: path=/tmp/raft3938084153/snapshots/2-99-1740336192671.tmp
2025-02-23T18:43:12.671Z [INFO]  server-d8d61fc5-f874-6ade-0e9a-e16a420b3ed6: compacting logs: from=1 to=92
2025-02-23T18:43:12.671Z [INFO]  server-d8d61fc5-f874-6ade-0e9a-e16a420b3ed6: snapshot complete up to: index=99
2025-02-23T18:43:12.672Z [ERROR] server-e67f8485-5460-fb40-d4a8-3de912e6f0df: failed to get log: index=2 error="log not found"
2025-02-23T18:43:12.672Z [INFO]  creating new snapshot: path=/tmp/raft2367442595/snapshots/2-102-1740336192672.tmp
2025-02-23T18:43:12.673Z [INFO]  server-5d058d9c-3fc7-86e5-273f-72b2608eaa79: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-02-23T18:43:12.673Z [INFO]  server-5d058d9c-3fc7-86e5-273f-72b2608eaa79: copied to local snapshot: bytes=693
2025-02-23T18:43:12.673Z [INFO]  server-5d058d9c-3fc7-86e5-273f-72b2608eaa79: snapshot restore progress: id=2-102-1740336192672 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-02-23T18:43:12.673Z [INFO]  server-5d058d9c-3fc7-86e5-273f-72b2608eaa79: Installed remote snapshot
2025-02-23T18:43:12.674Z [WARN]  server-5d058d9c-3fc7-86e5-273f-72b2608eaa79: skipping application of old log: index=102
2025-02-23T18:43:12.674Z [INFO]  server-e67f8485-5460-fb40-d4a8-3de912e6f0df: pipelining replication: peer="{Voter server-5d058d9c-3fc7-86e5-273f-72b2608eaa79 5d058d9c-3fc7-86e5-273f-72b2608eaa79}"
2025-02-23T18:43:12.677Z [INFO]  server-e67f8485-5460-fb40-d4a8-3de912e6f0df: aborting pipeline replication: peer="{Voter server-5d058d9c-3fc7-86e5-273f-72b2608eaa79 5d058d9c-3fc7-86e5-273f-72b2608eaa79}"
2025-02-23T18:43:12.677Z [INFO]  server-e67f8485-5460-fb40-d4a8-3de912e6f0df: aborting pipeline replication: peer="{Voter server-d8d61fc5-f874-6ade-0e9a-e16a420b3ed6 d8d61fc5-f874-6ade-0e9a-e16a420b3ed6}"
--- PASS: TestRaft_SendSnapshotFollower (0.28s)
=== RUN   TestRaft_SendSnapshotAndLogsFollower
2025-02-23T18:43:12.679Z [INFO]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e Address:2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e} {Suffrage:Voter ID:server-56748e26-3354-e2d8-bc2e-6546c348c33a Address:56748e26-3354-e2d8-bc2e-6546c348c33a} {Suffrage:Voter ID:server-64151baa-ef44-3b34-02aa-e05d86a5d75a Address:64151baa-ef44-3b34-02aa-e05d86a5d75a}]"
2025-02-23T18:43:12.679Z [INFO]  server-56748e26-3354-e2d8-bc2e-6546c348c33a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e Address:2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e} {Suffrage:Voter ID:server-56748e26-3354-e2d8-bc2e-6546c348c33a Address:56748e26-3354-e2d8-bc2e-6546c348c33a} {Suffrage:Voter ID:server-64151baa-ef44-3b34-02aa-e05d86a5d75a Address:64151baa-ef44-3b34-02aa-e05d86a5d75a}]"
2025-02-23T18:43:12.680Z [INFO]  server-64151baa-ef44-3b34-02aa-e05d86a5d75a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e Address:2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e} {Suffrage:Voter ID:server-56748e26-3354-e2d8-bc2e-6546c348c33a Address:56748e26-3354-e2d8-bc2e-6546c348c33a} {Suffrage:Voter ID:server-64151baa-ef44-3b34-02aa-e05d86a5d75a Address:64151baa-ef44-3b34-02aa-e05d86a5d75a}]"
2025-02-23T18:43:12.680Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:12.680Z [INFO]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: entering follower state: follower="Node at 2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e [Follower]" leader-address= leader-id=
2025-02-23T18:43:12.680Z [INFO]  server-56748e26-3354-e2d8-bc2e-6546c348c33a: entering follower state: follower="Node at 56748e26-3354-e2d8-bc2e-6546c348c33a [Follower]" leader-address= leader-id=
2025-02-23T18:43:12.680Z [INFO]  server-64151baa-ef44-3b34-02aa-e05d86a5d75a: entering follower state: follower="Node at 64151baa-ef44-3b34-02aa-e05d86a5d75a [Follower]" leader-address= leader-id=
2025-02-23T18:43:12.746Z [WARN]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:12.746Z [INFO]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: entering candidate state: node="Node at 2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e [Candidate]" term=2
2025-02-23T18:43:12.747Z [INFO]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: election won: term=2 tally=2
2025-02-23T18:43:12.747Z [INFO]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: entering leader state: leader="Node at 2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e [Leader]"
2025-02-23T18:43:12.747Z [INFO]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: added peer, starting replication: peer=server-56748e26-3354-e2d8-bc2e-6546c348c33a
2025-02-23T18:43:12.747Z [INFO]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: added peer, starting replication: peer=server-64151baa-ef44-3b34-02aa-e05d86a5d75a
2025-02-23T18:43:12.747Z [INFO]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: pipelining replication: peer="{Voter server-64151baa-ef44-3b34-02aa-e05d86a5d75a 64151baa-ef44-3b34-02aa-e05d86a5d75a}"
2025-02-23T18:43:12.747Z [INFO]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: pipelining replication: peer="{Voter server-56748e26-3354-e2d8-bc2e-6546c348c33a 56748e26-3354-e2d8-bc2e-6546c348c33a}"
2025-02-23T18:43:12.853Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:12.747445007 +0000 UTC m=+97.045705848 (2 nodes), 67.346296ms from start of poll, 67.777309ms from cluster start. Timeout at 2025-02-23 18:43:12.85245176 +0000 UTC m=+97.150712638, 105.00679ms after stability
2025-02-23T18:43:12.853Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:12.959Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:12.853916289 +0000 UTC m=+97.152176961 (1 nodes), 1.661µs from start of poll, 174.248422ms from cluster start. Timeout at 2025-02-23 18:43:12.958965648 +0000 UTC m=+97.257226306, 105.049345ms after stability
2025-02-23T18:43:12.960Z [ERROR] server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: failed to pipeline appendEntries: peer="{Voter server-56748e26-3354-e2d8-bc2e-6546c348c33a 56748e26-3354-e2d8-bc2e-6546c348c33a}" error="append pipeline closed"
2025-02-23T18:43:12.960Z [INFO]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: aborting pipeline replication: peer="{Voter server-56748e26-3354-e2d8-bc2e-6546c348c33a 56748e26-3354-e2d8-bc2e-6546c348c33a}"
2025-02-23T18:43:12.960Z [ERROR] server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: failed to appendEntries to: peer="{Voter server-56748e26-3354-e2d8-bc2e-6546c348c33a 56748e26-3354-e2d8-bc2e-6546c348c33a}" error="failed to connect to peer: 56748e26-3354-e2d8-bc2e-6546c348c33a"
2025-02-23T18:43:12.967Z [ERROR] server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: failed to heartbeat to: peer=56748e26-3354-e2d8-bc2e-6546c348c33a backoff time=10ms error="failed to connect to peer: 56748e26-3354-e2d8-bc2e-6546c348c33a"
2025-02-23T18:43:12.972Z [ERROR] server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: failed to appendEntries to: peer="{Voter server-56748e26-3354-e2d8-bc2e-6546c348c33a 56748e26-3354-e2d8-bc2e-6546c348c33a}" error="failed to connect to peer: 56748e26-3354-e2d8-bc2e-6546c348c33a"
    raft_test.go:1669: [INFO] Finished apply without behind follower
2025-02-23T18:43:12.977Z [INFO]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: starting snapshot up to: index=102
2025-02-23T18:43:12.977Z [INFO]  creating new snapshot: path=/tmp/raft546614032/snapshots/2-102-1740336192977.tmp
2025-02-23T18:43:12.977Z [INFO]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: compacting logs: from=1 to=92
2025-02-23T18:43:12.977Z [INFO]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: snapshot complete up to: index=102
2025-02-23T18:43:12.978Z [ERROR] server-56748e26-3354-e2d8-bc2e-6546c348c33a: failed to take snapshot: error="nothing new to snapshot"
2025-02-23T18:43:12.978Z [INFO]  server-64151baa-ef44-3b34-02aa-e05d86a5d75a: starting snapshot up to: index=93
2025-02-23T18:43:12.978Z [INFO]  creating new snapshot: path=/tmp/raft4204631327/snapshots/2-93-1740336192978.tmp
2025-02-23T18:43:12.978Z [INFO]  server-64151baa-ef44-3b34-02aa-e05d86a5d75a: compacting logs: from=1 to=92
2025-02-23T18:43:12.978Z [INFO]  server-64151baa-ef44-3b34-02aa-e05d86a5d75a: snapshot complete up to: index=93
2025-02-23T18:43:12.984Z [ERROR] server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: failed to get log: index=2 error="log not found"
2025-02-23T18:43:12.985Z [ERROR] server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: failed to install snapshot: id=2-102-1740336192977 error="failed to connect to peer: 56748e26-3354-e2d8-bc2e-6546c348c33a"
2025-02-23T18:43:12.985Z [ERROR] server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: failed to send snapshot to: peer="{Voter server-56748e26-3354-e2d8-bc2e-6546c348c33a 56748e26-3354-e2d8-bc2e-6546c348c33a}" error="failed to connect to peer: 56748e26-3354-e2d8-bc2e-6546c348c33a"
2025-02-23T18:43:12.985Z [ERROR] server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: failed to heartbeat to: peer=56748e26-3354-e2d8-bc2e-6546c348c33a backoff time=10ms error="failed to connect to peer: 56748e26-3354-e2d8-bc2e-6546c348c33a"
    raft_test.go:1690: [INFO] Finished apply without behind follower
2025-02-23T18:43:13.002Z [WARN]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: failed to contact: server-id=server-56748e26-3354-e2d8-bc2e-6546c348c33a time=50.363367ms
2025-02-23T18:43:13.005Z [ERROR] server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: failed to get log: index=2 error="log not found"
2025-02-23T18:43:13.006Z [INFO]  creating new snapshot: path=/tmp/raft1520943831/snapshots/2-102-1740336193006.tmp
2025-02-23T18:43:13.006Z [INFO]  server-56748e26-3354-e2d8-bc2e-6546c348c33a: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-02-23T18:43:13.006Z [INFO]  server-56748e26-3354-e2d8-bc2e-6546c348c33a: copied to local snapshot: bytes=693
2025-02-23T18:43:13.007Z [INFO]  server-56748e26-3354-e2d8-bc2e-6546c348c33a: snapshot restore progress: id=2-102-1740336193006 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-02-23T18:43:13.007Z [INFO]  server-56748e26-3354-e2d8-bc2e-6546c348c33a: Installed remote snapshot
2025-02-23T18:43:13.007Z [INFO]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: pipelining replication: peer="{Voter server-56748e26-3354-e2d8-bc2e-6546c348c33a 56748e26-3354-e2d8-bc2e-6546c348c33a}"
2025-02-23T18:43:13.009Z [INFO]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: aborting pipeline replication: peer="{Voter server-64151baa-ef44-3b34-02aa-e05d86a5d75a 64151baa-ef44-3b34-02aa-e05d86a5d75a}"
2025-02-23T18:43:13.009Z [INFO]  server-2cea1792-ebdf-51f8-fd57-6c74ca3c9c0e: aborting pipeline replication: peer="{Voter server-56748e26-3354-e2d8-bc2e-6546c348c33a 56748e26-3354-e2d8-bc2e-6546c348c33a}"
--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.33s)
=== RUN   TestRaft_ReJoinFollower
2025-02-23T18:43:13.011Z [INFO]  server-b01fb004-13e9-ebcd-0bc4-d3d67776f904: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b01fb004-13e9-ebcd-0bc4-d3d67776f904 Address:b01fb004-13e9-ebcd-0bc4-d3d67776f904} {Suffrage:Voter ID:server-8e4c006e-c442-e841-f431-719f1fb37503 Address:8e4c006e-c442-e841-f431-719f1fb37503} {Suffrage:Voter ID:server-53701aa3-d7f1-6785-5f4b-826633d5885d Address:53701aa3-d7f1-6785-5f4b-826633d5885d}]"
2025-02-23T18:43:13.011Z [INFO]  server-8e4c006e-c442-e841-f431-719f1fb37503: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b01fb004-13e9-ebcd-0bc4-d3d67776f904 Address:b01fb004-13e9-ebcd-0bc4-d3d67776f904} {Suffrage:Voter ID:server-8e4c006e-c442-e841-f431-719f1fb37503 Address:8e4c006e-c442-e841-f431-719f1fb37503} {Suffrage:Voter ID:server-53701aa3-d7f1-6785-5f4b-826633d5885d Address:53701aa3-d7f1-6785-5f4b-826633d5885d}]"
2025-02-23T18:43:13.011Z [INFO]  server-53701aa3-d7f1-6785-5f4b-826633d5885d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b01fb004-13e9-ebcd-0bc4-d3d67776f904 Address:b01fb004-13e9-ebcd-0bc4-d3d67776f904} {Suffrage:Voter ID:server-8e4c006e-c442-e841-f431-719f1fb37503 Address:8e4c006e-c442-e841-f431-719f1fb37503} {Suffrage:Voter ID:server-53701aa3-d7f1-6785-5f4b-826633d5885d Address:53701aa3-d7f1-6785-5f4b-826633d5885d}]"
2025-02-23T18:43:13.011Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:13.011Z [INFO]  server-b01fb004-13e9-ebcd-0bc4-d3d67776f904: entering follower state: follower="Node at b01fb004-13e9-ebcd-0bc4-d3d67776f904 [Follower]" leader-address= leader-id=
2025-02-23T18:43:13.011Z [INFO]  server-8e4c006e-c442-e841-f431-719f1fb37503: entering follower state: follower="Node at 8e4c006e-c442-e841-f431-719f1fb37503 [Follower]" leader-address= leader-id=
2025-02-23T18:43:13.011Z [INFO]  server-53701aa3-d7f1-6785-5f4b-826633d5885d: entering follower state: follower="Node at 53701aa3-d7f1-6785-5f4b-826633d5885d [Follower]" leader-address= leader-id=
2025-02-23T18:43:13.073Z [WARN]  server-8e4c006e-c442-e841-f431-719f1fb37503: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:13.073Z [INFO]  server-8e4c006e-c442-e841-f431-719f1fb37503: entering candidate state: node="Node at 8e4c006e-c442-e841-f431-719f1fb37503 [Candidate]" term=2
2025-02-23T18:43:13.073Z [INFO]  server-8e4c006e-c442-e841-f431-719f1fb37503: election won: term=2 tally=2
2025-02-23T18:43:13.073Z [INFO]  server-8e4c006e-c442-e841-f431-719f1fb37503: entering leader state: leader="Node at 8e4c006e-c442-e841-f431-719f1fb37503 [Leader]"
2025-02-23T18:43:13.073Z [INFO]  server-8e4c006e-c442-e841-f431-719f1fb37503: added peer, starting replication: peer=server-b01fb004-13e9-ebcd-0bc4-d3d67776f904
2025-02-23T18:43:13.073Z [INFO]  server-8e4c006e-c442-e841-f431-719f1fb37503: added peer, starting replication: peer=server-53701aa3-d7f1-6785-5f4b-826633d5885d
2025-02-23T18:43:13.073Z [INFO]  server-8e4c006e-c442-e841-f431-719f1fb37503: pipelining replication: peer="{Voter server-53701aa3-d7f1-6785-5f4b-826633d5885d 53701aa3-d7f1-6785-5f4b-826633d5885d}"
2025-02-23T18:43:13.073Z [INFO]  server-8e4c006e-c442-e841-f431-719f1fb37503: pipelining replication: peer="{Voter server-b01fb004-13e9-ebcd-0bc4-d3d67776f904 b01fb004-13e9-ebcd-0bc4-d3d67776f904}"
2025-02-23T18:43:13.178Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:13.073829823 +0000 UTC m=+97.372090448 (1 nodes), 62.264405ms from start of poll, 62.660815ms from cluster start. Timeout at 2025-02-23 18:43:13.178838427 +0000 UTC m=+97.477099057, 105.008609ms after stability
2025-02-23T18:43:13.184Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:13.291Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:13.184991466 +0000 UTC m=+97.483252237 (2 nodes), 2.467µs from start of poll, 173.822604ms from cluster start. Timeout at 2025-02-23 18:43:13.290032115 +0000 UTC m=+97.588292748, 105.040511ms after stability
2025-02-23T18:43:13.291Z [INFO]  server-8e4c006e-c442-e841-f431-719f1fb37503: updating configuration: command=RemoveServer server-id=server-b01fb004-13e9-ebcd-0bc4-d3d67776f904 server-addr= servers="[{Suffrage:Voter ID:server-8e4c006e-c442-e841-f431-719f1fb37503 Address:8e4c006e-c442-e841-f431-719f1fb37503} {Suffrage:Voter ID:server-53701aa3-d7f1-6785-5f4b-826633d5885d Address:53701aa3-d7f1-6785-5f4b-826633d5885d}]"
2025-02-23T18:43:13.291Z [INFO]  server-8e4c006e-c442-e841-f431-719f1fb37503: removed peer, stopping replication: peer=server-b01fb004-13e9-ebcd-0bc4-d3d67776f904 last-index=3
2025-02-23T18:43:13.291Z [INFO]  server-8e4c006e-c442-e841-f431-719f1fb37503: aborting pipeline replication: peer="{Voter server-b01fb004-13e9-ebcd-0bc4-d3d67776f904 b01fb004-13e9-ebcd-0bc4-d3d67776f904}"
2025-02-23T18:43:13.371Z [WARN]  server-b01fb004-13e9-ebcd-0bc4-d3d67776f904: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-02-23T18:43:13.396Z [INFO]  server-8e4c006e-c442-e841-f431-719f1fb37503: updating configuration: command=AddVoter server-id=server-b01fb004-13e9-ebcd-0bc4-d3d67776f904 server-addr=b01fb004-13e9-ebcd-0bc4-d3d67776f904 servers="[{Suffrage:Voter ID:server-8e4c006e-c442-e841-f431-719f1fb37503 Address:8e4c006e-c442-e841-f431-719f1fb37503} {Suffrage:Voter ID:server-53701aa3-d7f1-6785-5f4b-826633d5885d Address:53701aa3-d7f1-6785-5f4b-826633d5885d} {Suffrage:Voter ID:server-b01fb004-13e9-ebcd-0bc4-d3d67776f904 Address:b01fb004-13e9-ebcd-0bc4-d3d67776f904}]"
2025-02-23T18:43:13.396Z [INFO]  server-8e4c006e-c442-e841-f431-719f1fb37503: added peer, starting replication: peer=server-b01fb004-13e9-ebcd-0bc4-d3d67776f904
2025-02-23T18:43:13.397Z [WARN]  server-b01fb004-13e9-ebcd-0bc4-d3d67776f904: failed to get previous log: previous-index=4 last-index=3 error="log not found"
2025-02-23T18:43:13.397Z [WARN]  server-8e4c006e-c442-e841-f431-719f1fb37503: appendEntries rejected, sending older logs: peer="{Voter server-b01fb004-13e9-ebcd-0bc4-d3d67776f904 b01fb004-13e9-ebcd-0bc4-d3d67776f904}" next=4
2025-02-23T18:43:13.397Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:13.397Z [INFO]  server-8e4c006e-c442-e841-f431-719f1fb37503: pipelining replication: peer="{Voter server-b01fb004-13e9-ebcd-0bc4-d3d67776f904 b01fb004-13e9-ebcd-0bc4-d3d67776f904}"
2025-02-23T18:43:13.502Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:13.397326236 +0000 UTC m=+97.695586906 (1 nodes), 1.734µs from start of poll, 386.157273ms from cluster start. Timeout at 2025-02-23 18:43:13.502410213 +0000 UTC m=+97.800670833, 105.083927ms after stability
2025-02-23T18:43:13.502Z [INFO]  server-8e4c006e-c442-e841-f431-719f1fb37503: aborting pipeline replication: peer="{Voter server-53701aa3-d7f1-6785-5f4b-826633d5885d 53701aa3-d7f1-6785-5f4b-826633d5885d}"
2025-02-23T18:43:13.502Z [INFO]  server-8e4c006e-c442-e841-f431-719f1fb37503: aborting pipeline replication: peer="{Voter server-b01fb004-13e9-ebcd-0bc4-d3d67776f904 b01fb004-13e9-ebcd-0bc4-d3d67776f904}"
2025-02-23T18:43:14.003Z [ERROR] server-8e4c006e-c442-e841-f431-719f1fb37503: failed to heartbeat to: peer=b01fb004-13e9-ebcd-0bc4-d3d67776f904 backoff time=10ms error="command timed out"
--- PASS: TestRaft_ReJoinFollower (0.99s)
=== RUN   TestRaft_LeaderLeaseExpire
2025-02-23T18:43:14.005Z [INFO]  server-e7141697-017d-a360-f263-6717693cf498: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7141697-017d-a360-f263-6717693cf498 Address:e7141697-017d-a360-f263-6717693cf498} {Suffrage:Voter ID:server-82dfe88c-5ad3-3b6e-2e15-c267eace3975 Address:82dfe88c-5ad3-3b6e-2e15-c267eace3975}]"
2025-02-23T18:43:14.005Z [INFO]  server-82dfe88c-5ad3-3b6e-2e15-c267eace3975: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7141697-017d-a360-f263-6717693cf498 Address:e7141697-017d-a360-f263-6717693cf498} {Suffrage:Voter ID:server-82dfe88c-5ad3-3b6e-2e15-c267eace3975 Address:82dfe88c-5ad3-3b6e-2e15-c267eace3975}]"
2025-02-23T18:43:14.005Z [INFO]  server-e7141697-017d-a360-f263-6717693cf498: entering follower state: follower="Node at e7141697-017d-a360-f263-6717693cf498 [Follower]" leader-address= leader-id=
2025-02-23T18:43:14.005Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:14.005Z [INFO]  server-82dfe88c-5ad3-3b6e-2e15-c267eace3975: entering follower state: follower="Node at 82dfe88c-5ad3-3b6e-2e15-c267eace3975 [Follower]" leader-address= leader-id=
2025-02-23T18:43:14.086Z [WARN]  server-e7141697-017d-a360-f263-6717693cf498: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:14.086Z [INFO]  server-e7141697-017d-a360-f263-6717693cf498: entering candidate state: node="Node at e7141697-017d-a360-f263-6717693cf498 [Candidate]" term=2
2025-02-23T18:43:14.087Z [INFO]  server-e7141697-017d-a360-f263-6717693cf498: election won: term=2 tally=2
2025-02-23T18:43:14.087Z [INFO]  server-e7141697-017d-a360-f263-6717693cf498: entering leader state: leader="Node at e7141697-017d-a360-f263-6717693cf498 [Leader]"
2025-02-23T18:43:14.087Z [INFO]  server-e7141697-017d-a360-f263-6717693cf498: added peer, starting replication: peer=server-82dfe88c-5ad3-3b6e-2e15-c267eace3975
2025-02-23T18:43:14.089Z [INFO]  server-e7141697-017d-a360-f263-6717693cf498: pipelining replication: peer="{Voter server-82dfe88c-5ad3-3b6e-2e15-c267eace3975 82dfe88c-5ad3-3b6e-2e15-c267eace3975}"
2025-02-23T18:43:14.192Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:14.087864065 +0000 UTC m=+98.386124909 (1 nodes), 82.177976ms from start of poll, 82.585219ms from cluster start. Timeout at 2025-02-23 18:43:14.192882138 +0000 UTC m=+98.491142816, 105.017907ms after stability
2025-02-23T18:43:14.199Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:14.305Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:14.199134683 +0000 UTC m=+98.497395448 (1 nodes), 2.094µs from start of poll, 193.855758ms from cluster start. Timeout at 2025-02-23 18:43:14.304179834 +0000 UTC m=+98.602440464, 105.045016ms after stability
    raft_test.go:1799: [INFO] Disconnecting Node at 82dfe88c-5ad3-3b6e-2e15-c267eace3975 [Follower]
2025-02-23T18:43:14.305Z [ERROR] server-e7141697-017d-a360-f263-6717693cf498: failed to pipeline appendEntries: peer="{Voter server-82dfe88c-5ad3-3b6e-2e15-c267eace3975 82dfe88c-5ad3-3b6e-2e15-c267eace3975}" error="append pipeline closed"
2025-02-23T18:43:14.305Z [INFO]  server-e7141697-017d-a360-f263-6717693cf498: aborting pipeline replication: peer="{Voter server-82dfe88c-5ad3-3b6e-2e15-c267eace3975 82dfe88c-5ad3-3b6e-2e15-c267eace3975}"
2025-02-23T18:43:14.312Z [ERROR] server-e7141697-017d-a360-f263-6717693cf498: failed to appendEntries to: peer="{Voter server-82dfe88c-5ad3-3b6e-2e15-c267eace3975 82dfe88c-5ad3-3b6e-2e15-c267eace3975}" error="failed to connect to peer: 82dfe88c-5ad3-3b6e-2e15-c267eace3975"
2025-02-23T18:43:14.314Z [ERROR] server-e7141697-017d-a360-f263-6717693cf498: failed to heartbeat to: peer=82dfe88c-5ad3-3b6e-2e15-c267eace3975 backoff time=10ms error="failed to connect to peer: 82dfe88c-5ad3-3b6e-2e15-c267eace3975"
2025-02-23T18:43:14.332Z [ERROR] server-e7141697-017d-a360-f263-6717693cf498: failed to appendEntries to: peer="{Voter server-82dfe88c-5ad3-3b6e-2e15-c267eace3975 82dfe88c-5ad3-3b6e-2e15-c267eace3975}" error="failed to connect to peer: 82dfe88c-5ad3-3b6e-2e15-c267eace3975"
2025-02-23T18:43:14.333Z [ERROR] server-e7141697-017d-a360-f263-6717693cf498: failed to heartbeat to: peer=82dfe88c-5ad3-3b6e-2e15-c267eace3975 backoff time=10ms error="failed to connect to peer: 82dfe88c-5ad3-3b6e-2e15-c267eace3975"
2025-02-23T18:43:14.352Z [ERROR] server-e7141697-017d-a360-f263-6717693cf498: failed to appendEntries to: peer="{Voter server-82dfe88c-5ad3-3b6e-2e15-c267eace3975 82dfe88c-5ad3-3b6e-2e15-c267eace3975}" error="failed to connect to peer: 82dfe88c-5ad3-3b6e-2e15-c267eace3975"
2025-02-23T18:43:14.353Z [ERROR] server-e7141697-017d-a360-f263-6717693cf498: failed to heartbeat to: peer=82dfe88c-5ad3-3b6e-2e15-c267eace3975 backoff time=10ms error="failed to connect to peer: 82dfe88c-5ad3-3b6e-2e15-c267eace3975"
2025-02-23T18:43:14.355Z [WARN]  server-e7141697-017d-a360-f263-6717693cf498: failed to contact: server-id=server-82dfe88c-5ad3-3b6e-2e15-c267eace3975 time=50.403993ms
2025-02-23T18:43:14.355Z [WARN]  server-e7141697-017d-a360-f263-6717693cf498: failed to contact quorum of nodes, stepping down
2025-02-23T18:43:14.355Z [INFO]  server-e7141697-017d-a360-f263-6717693cf498: entering follower state: follower="Node at e7141697-017d-a360-f263-6717693cf498 [Follower]" leader-address= leader-id=
2025-02-23T18:43:14.355Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:14.405Z [WARN]  server-82dfe88c-5ad3-3b6e-2e15-c267eace3975: heartbeat timeout reached, starting election: last-leader-addr=e7141697-017d-a360-f263-6717693cf498 last-leader-id=server-e7141697-017d-a360-f263-6717693cf498
2025-02-23T18:43:14.406Z [INFO]  server-82dfe88c-5ad3-3b6e-2e15-c267eace3975: entering candidate state: node="Node at 82dfe88c-5ad3-3b6e-2e15-c267eace3975 [Candidate]" term=3
2025-02-23T18:43:14.406Z [ERROR] server-82dfe88c-5ad3-3b6e-2e15-c267eace3975: failed to make requestVote RPC: target="{Voter server-e7141697-017d-a360-f263-6717693cf498 e7141697-017d-a360-f263-6717693cf498}" error="failed to connect to peer: e7141697-017d-a360-f263-6717693cf498" term=3
2025-02-23T18:43:14.427Z [WARN]  server-e7141697-017d-a360-f263-6717693cf498: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:14.427Z [INFO]  server-e7141697-017d-a360-f263-6717693cf498: entering candidate state: node="Node at e7141697-017d-a360-f263-6717693cf498 [Candidate]" term=3
2025-02-23T18:43:14.427Z [ERROR] server-e7141697-017d-a360-f263-6717693cf498: failed to make requestVote RPC: target="{Voter server-82dfe88c-5ad3-3b6e-2e15-c267eace3975 82dfe88c-5ad3-3b6e-2e15-c267eace3975}" error="failed to connect to peer: 82dfe88c-5ad3-3b6e-2e15-c267eace3975" term=3
2025-02-23T18:43:14.494Z [WARN]  server-82dfe88c-5ad3-3b6e-2e15-c267eace3975: Election timeout reached, restarting election
2025-02-23T18:43:14.494Z [INFO]  server-82dfe88c-5ad3-3b6e-2e15-c267eace3975: entering candidate state: node="Node at 82dfe88c-5ad3-3b6e-2e15-c267eace3975 [Candidate]" term=4
2025-02-23T18:43:14.494Z [ERROR] server-82dfe88c-5ad3-3b6e-2e15-c267eace3975: failed to make requestVote RPC: target="{Voter server-e7141697-017d-a360-f263-6717693cf498 e7141697-017d-a360-f263-6717693cf498}" error="failed to connect to peer: e7141697-017d-a360-f263-6717693cf498" term=4
2025-02-23T18:43:14.516Z [WARN]  server-e7141697-017d-a360-f263-6717693cf498: Election timeout reached, restarting election
2025-02-23T18:43:14.516Z [INFO]  server-e7141697-017d-a360-f263-6717693cf498: entering candidate state: node="Node at e7141697-017d-a360-f263-6717693cf498 [Candidate]" term=4
2025-02-23T18:43:14.516Z [ERROR] server-e7141697-017d-a360-f263-6717693cf498: failed to make requestVote RPC: target="{Voter server-82dfe88c-5ad3-3b6e-2e15-c267eace3975 82dfe88c-5ad3-3b6e-2e15-c267eace3975}" error="failed to connect to peer: 82dfe88c-5ad3-3b6e-2e15-c267eace3975" term=4
2025-02-23T18:43:14.533Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:14.427157309 +0000 UTC m=+98.725417955 (0 nodes), 71.661515ms from start of poll, 421.878265ms from cluster start. Timeout at 2025-02-23 18:43:14.53216832 +0000 UTC m=+98.830428989, 105.011034ms after stability
2025-02-23T18:43:14.554Z [WARN]  server-82dfe88c-5ad3-3b6e-2e15-c267eace3975: Election timeout reached, restarting election
2025-02-23T18:43:14.554Z [INFO]  server-82dfe88c-5ad3-3b6e-2e15-c267eace3975: entering candidate state: node="Node at 82dfe88c-5ad3-3b6e-2e15-c267eace3975 [Candidate]" term=5
2025-02-23T18:43:14.554Z [ERROR] server-82dfe88c-5ad3-3b6e-2e15-c267eace3975: failed to make requestVote RPC: target="{Voter server-e7141697-017d-a360-f263-6717693cf498 e7141697-017d-a360-f263-6717693cf498}" error="failed to connect to peer: e7141697-017d-a360-f263-6717693cf498" term=5
2025-02-23T18:43:14.595Z [WARN]  server-e7141697-017d-a360-f263-6717693cf498: Election timeout reached, restarting election
2025-02-23T18:43:14.595Z [INFO]  server-e7141697-017d-a360-f263-6717693cf498: entering candidate state: node="Node at e7141697-017d-a360-f263-6717693cf498 [Candidate]" term=5
2025-02-23T18:43:14.595Z [ERROR] server-e7141697-017d-a360-f263-6717693cf498: failed to make requestVote RPC: target="{Voter server-82dfe88c-5ad3-3b6e-2e15-c267eace3975 82dfe88c-5ad3-3b6e-2e15-c267eace3975}" error="failed to connect to peer: 82dfe88c-5ad3-3b6e-2e15-c267eace3975" term=5
--- PASS: TestRaft_LeaderLeaseExpire (0.63s)
=== RUN   TestRaft_Barrier
2025-02-23T18:43:14.640Z [INFO]  server-580fe3cc-2ea6-a8a9-5c10-6f680ce39b68: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-580fe3cc-2ea6-a8a9-5c10-6f680ce39b68 Address:580fe3cc-2ea6-a8a9-5c10-6f680ce39b68} {Suffrage:Voter ID:server-74ddafe0-9073-edf8-7560-1ba9955a92b4 Address:74ddafe0-9073-edf8-7560-1ba9955a92b4} {Suffrage:Voter ID:server-bef6d169-bec6-f762-10fd-228712b8221e Address:bef6d169-bec6-f762-10fd-228712b8221e}]"
2025-02-23T18:43:14.640Z [INFO]  server-74ddafe0-9073-edf8-7560-1ba9955a92b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-580fe3cc-2ea6-a8a9-5c10-6f680ce39b68 Address:580fe3cc-2ea6-a8a9-5c10-6f680ce39b68} {Suffrage:Voter ID:server-74ddafe0-9073-edf8-7560-1ba9955a92b4 Address:74ddafe0-9073-edf8-7560-1ba9955a92b4} {Suffrage:Voter ID:server-bef6d169-bec6-f762-10fd-228712b8221e Address:bef6d169-bec6-f762-10fd-228712b8221e}]"
2025-02-23T18:43:14.640Z [INFO]  server-bef6d169-bec6-f762-10fd-228712b8221e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-580fe3cc-2ea6-a8a9-5c10-6f680ce39b68 Address:580fe3cc-2ea6-a8a9-5c10-6f680ce39b68} {Suffrage:Voter ID:server-74ddafe0-9073-edf8-7560-1ba9955a92b4 Address:74ddafe0-9073-edf8-7560-1ba9955a92b4} {Suffrage:Voter ID:server-bef6d169-bec6-f762-10fd-228712b8221e Address:bef6d169-bec6-f762-10fd-228712b8221e}]"
2025-02-23T18:43:14.640Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:14.640Z [INFO]  server-580fe3cc-2ea6-a8a9-5c10-6f680ce39b68: entering follower state: follower="Node at 580fe3cc-2ea6-a8a9-5c10-6f680ce39b68 [Follower]" leader-address= leader-id=
2025-02-23T18:43:14.640Z [INFO]  server-74ddafe0-9073-edf8-7560-1ba9955a92b4: entering follower state: follower="Node at 74ddafe0-9073-edf8-7560-1ba9955a92b4 [Follower]" leader-address= leader-id=
2025-02-23T18:43:14.640Z [INFO]  server-bef6d169-bec6-f762-10fd-228712b8221e: entering follower state: follower="Node at bef6d169-bec6-f762-10fd-228712b8221e [Follower]" leader-address= leader-id=
2025-02-23T18:43:14.705Z [WARN]  server-74ddafe0-9073-edf8-7560-1ba9955a92b4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:14.705Z [INFO]  server-74ddafe0-9073-edf8-7560-1ba9955a92b4: entering candidate state: node="Node at 74ddafe0-9073-edf8-7560-1ba9955a92b4 [Candidate]" term=2
2025-02-23T18:43:14.705Z [INFO]  server-74ddafe0-9073-edf8-7560-1ba9955a92b4: election won: term=2 tally=2
2025-02-23T18:43:14.706Z [INFO]  server-74ddafe0-9073-edf8-7560-1ba9955a92b4: entering leader state: leader="Node at 74ddafe0-9073-edf8-7560-1ba9955a92b4 [Leader]"
2025-02-23T18:43:14.706Z [INFO]  server-74ddafe0-9073-edf8-7560-1ba9955a92b4: added peer, starting replication: peer=server-580fe3cc-2ea6-a8a9-5c10-6f680ce39b68
2025-02-23T18:43:14.706Z [INFO]  server-74ddafe0-9073-edf8-7560-1ba9955a92b4: added peer, starting replication: peer=server-bef6d169-bec6-f762-10fd-228712b8221e
2025-02-23T18:43:14.706Z [INFO]  server-74ddafe0-9073-edf8-7560-1ba9955a92b4: pipelining replication: peer="{Voter server-bef6d169-bec6-f762-10fd-228712b8221e bef6d169-bec6-f762-10fd-228712b8221e}"
2025-02-23T18:43:14.706Z [INFO]  server-74ddafe0-9073-edf8-7560-1ba9955a92b4: pipelining replication: peer="{Voter server-580fe3cc-2ea6-a8a9-5c10-6f680ce39b68 580fe3cc-2ea6-a8a9-5c10-6f680ce39b68}"
2025-02-23T18:43:14.811Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:14.706052448 +0000 UTC m=+99.004313156 (1 nodes), 65.422498ms from start of poll, 65.864772ms from cluster start. Timeout at 2025-02-23 18:43:14.811079678 +0000 UTC m=+99.109340311, 105.027155ms after stability
2025-02-23T18:43:14.842Z [INFO]  server-74ddafe0-9073-edf8-7560-1ba9955a92b4: aborting pipeline replication: peer="{Voter server-bef6d169-bec6-f762-10fd-228712b8221e bef6d169-bec6-f762-10fd-228712b8221e}"
2025-02-23T18:43:14.842Z [INFO]  server-74ddafe0-9073-edf8-7560-1ba9955a92b4: aborting pipeline replication: peer="{Voter server-580fe3cc-2ea6-a8a9-5c10-6f680ce39b68 580fe3cc-2ea6-a8a9-5c10-6f680ce39b68}"
--- PASS: TestRaft_Barrier (0.20s)
=== RUN   TestRaft_VerifyLeader
2025-02-23T18:43:14.844Z [INFO]  server-22c4f391-4e75-af9f-a57d-b2020e94e5f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22c4f391-4e75-af9f-a57d-b2020e94e5f9 Address:22c4f391-4e75-af9f-a57d-b2020e94e5f9} {Suffrage:Voter ID:server-6e4ef050-6349-e176-d6ce-8ac0b6e1e45c Address:6e4ef050-6349-e176-d6ce-8ac0b6e1e45c} {Suffrage:Voter ID:server-dabf2580-c7a1-a922-4da0-89deaf0e3d3a Address:dabf2580-c7a1-a922-4da0-89deaf0e3d3a}]"
2025-02-23T18:43:14.844Z [INFO]  server-22c4f391-4e75-af9f-a57d-b2020e94e5f9: entering follower state: follower="Node at 22c4f391-4e75-af9f-a57d-b2020e94e5f9 [Follower]" leader-address= leader-id=
2025-02-23T18:43:14.844Z [INFO]  server-6e4ef050-6349-e176-d6ce-8ac0b6e1e45c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22c4f391-4e75-af9f-a57d-b2020e94e5f9 Address:22c4f391-4e75-af9f-a57d-b2020e94e5f9} {Suffrage:Voter ID:server-6e4ef050-6349-e176-d6ce-8ac0b6e1e45c Address:6e4ef050-6349-e176-d6ce-8ac0b6e1e45c} {Suffrage:Voter ID:server-dabf2580-c7a1-a922-4da0-89deaf0e3d3a Address:dabf2580-c7a1-a922-4da0-89deaf0e3d3a}]"
2025-02-23T18:43:14.844Z [INFO]  server-6e4ef050-6349-e176-d6ce-8ac0b6e1e45c: entering follower state: follower="Node at 6e4ef050-6349-e176-d6ce-8ac0b6e1e45c [Follower]" leader-address= leader-id=
2025-02-23T18:43:14.844Z [INFO]  server-dabf2580-c7a1-a922-4da0-89deaf0e3d3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22c4f391-4e75-af9f-a57d-b2020e94e5f9 Address:22c4f391-4e75-af9f-a57d-b2020e94e5f9} {Suffrage:Voter ID:server-6e4ef050-6349-e176-d6ce-8ac0b6e1e45c Address:6e4ef050-6349-e176-d6ce-8ac0b6e1e45c} {Suffrage:Voter ID:server-dabf2580-c7a1-a922-4da0-89deaf0e3d3a Address:dabf2580-c7a1-a922-4da0-89deaf0e3d3a}]"
2025-02-23T18:43:14.844Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:14.844Z [INFO]  server-dabf2580-c7a1-a922-4da0-89deaf0e3d3a: entering follower state: follower="Node at dabf2580-c7a1-a922-4da0-89deaf0e3d3a [Follower]" leader-address= leader-id=
2025-02-23T18:43:14.909Z [WARN]  server-22c4f391-4e75-af9f-a57d-b2020e94e5f9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:14.910Z [INFO]  server-22c4f391-4e75-af9f-a57d-b2020e94e5f9: entering candidate state: node="Node at 22c4f391-4e75-af9f-a57d-b2020e94e5f9 [Candidate]" term=2
2025-02-23T18:43:14.910Z [INFO]  server-22c4f391-4e75-af9f-a57d-b2020e94e5f9: election won: term=2 tally=2
2025-02-23T18:43:14.910Z [INFO]  server-22c4f391-4e75-af9f-a57d-b2020e94e5f9: entering leader state: leader="Node at 22c4f391-4e75-af9f-a57d-b2020e94e5f9 [Leader]"
2025-02-23T18:43:14.910Z [INFO]  server-22c4f391-4e75-af9f-a57d-b2020e94e5f9: added peer, starting replication: peer=server-6e4ef050-6349-e176-d6ce-8ac0b6e1e45c
2025-02-23T18:43:14.910Z [INFO]  server-22c4f391-4e75-af9f-a57d-b2020e94e5f9: added peer, starting replication: peer=server-dabf2580-c7a1-a922-4da0-89deaf0e3d3a
2025-02-23T18:43:14.910Z [INFO]  server-22c4f391-4e75-af9f-a57d-b2020e94e5f9: pipelining replication: peer="{Voter server-dabf2580-c7a1-a922-4da0-89deaf0e3d3a dabf2580-c7a1-a922-4da0-89deaf0e3d3a}"
2025-02-23T18:43:14.910Z [INFO]  server-22c4f391-4e75-af9f-a57d-b2020e94e5f9: pipelining replication: peer="{Voter server-6e4ef050-6349-e176-d6ce-8ac0b6e1e45c 6e4ef050-6349-e176-d6ce-8ac0b6e1e45c}"
2025-02-23T18:43:15.016Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:14.910279836 +0000 UTC m=+99.208540610 (1 nodes), 65.435695ms from start of poll, 66.081634ms from cluster start. Timeout at 2025-02-23 18:43:15.015285391 +0000 UTC m=+99.313546295, 105.005685ms after stability
2025-02-23T18:43:15.017Z [INFO]  server-22c4f391-4e75-af9f-a57d-b2020e94e5f9: aborting pipeline replication: peer="{Voter server-dabf2580-c7a1-a922-4da0-89deaf0e3d3a dabf2580-c7a1-a922-4da0-89deaf0e3d3a}"
2025-02-23T18:43:15.017Z [INFO]  server-22c4f391-4e75-af9f-a57d-b2020e94e5f9: aborting pipeline replication: peer="{Voter server-6e4ef050-6349-e176-d6ce-8ac0b6e1e45c 6e4ef050-6349-e176-d6ce-8ac0b6e1e45c}"
2025-02-23T18:43:15.517Z [ERROR] server-22c4f391-4e75-af9f-a57d-b2020e94e5f9: failed to heartbeat to: peer=6e4ef050-6349-e176-d6ce-8ac0b6e1e45c backoff time=10ms error="command timed out"
2025-02-23T18:43:15.517Z [ERROR] server-22c4f391-4e75-af9f-a57d-b2020e94e5f9: failed to heartbeat to: peer=dabf2580-c7a1-a922-4da0-89deaf0e3d3a backoff time=10ms error="command timed out"
--- PASS: TestRaft_VerifyLeader (0.68s)
=== RUN   TestRaft_VerifyLeader_Single
2025-02-23T18:43:15.519Z [INFO]  server-abd603cc-1ca6-3bff-23ce-0fb250121045: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abd603cc-1ca6-3bff-23ce-0fb250121045 Address:abd603cc-1ca6-3bff-23ce-0fb250121045}]"
2025-02-23T18:43:15.519Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:15.519Z [INFO]  server-abd603cc-1ca6-3bff-23ce-0fb250121045: entering follower state: follower="Node at abd603cc-1ca6-3bff-23ce-0fb250121045 [Follower]" leader-address= leader-id=
2025-02-23T18:43:15.600Z [WARN]  server-abd603cc-1ca6-3bff-23ce-0fb250121045: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:15.600Z [INFO]  server-abd603cc-1ca6-3bff-23ce-0fb250121045: entering candidate state: node="Node at abd603cc-1ca6-3bff-23ce-0fb250121045 [Candidate]" term=2
2025-02-23T18:43:15.600Z [INFO]  server-abd603cc-1ca6-3bff-23ce-0fb250121045: election won: term=2 tally=1
2025-02-23T18:43:15.600Z [INFO]  server-abd603cc-1ca6-3bff-23ce-0fb250121045: entering leader state: leader="Node at abd603cc-1ca6-3bff-23ce-0fb250121045 [Leader]"
2025-02-23T18:43:15.706Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:15.600666395 +0000 UTC m=+99.898927029 (1 nodes), 81.476549ms from start of poll, 81.647643ms from cluster start. Timeout at 2025-02-23 18:43:15.705676016 +0000 UTC m=+100.003936824, 105.009795ms after stability
--- PASS: TestRaft_VerifyLeader_Single (0.19s)
=== RUN   TestRaft_VerifyLeader_Fail
2025-02-23T18:43:15.707Z [INFO]  server-ae158d48-2616-2bcf-e6c9-e1a9036d753b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae158d48-2616-2bcf-e6c9-e1a9036d753b Address:ae158d48-2616-2bcf-e6c9-e1a9036d753b} {Suffrage:Voter ID:server-2d1fa3a4-0c69-c171-2588-24e21aa17057 Address:2d1fa3a4-0c69-c171-2588-24e21aa17057}]"
2025-02-23T18:43:15.707Z [INFO]  server-2d1fa3a4-0c69-c171-2588-24e21aa17057: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae158d48-2616-2bcf-e6c9-e1a9036d753b Address:ae158d48-2616-2bcf-e6c9-e1a9036d753b} {Suffrage:Voter ID:server-2d1fa3a4-0c69-c171-2588-24e21aa17057 Address:2d1fa3a4-0c69-c171-2588-24e21aa17057}]"
2025-02-23T18:43:15.707Z [INFO]  server-ae158d48-2616-2bcf-e6c9-e1a9036d753b: entering follower state: follower="Node at ae158d48-2616-2bcf-e6c9-e1a9036d753b [Follower]" leader-address= leader-id=
2025-02-23T18:43:15.707Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:15.707Z [INFO]  server-2d1fa3a4-0c69-c171-2588-24e21aa17057: entering follower state: follower="Node at 2d1fa3a4-0c69-c171-2588-24e21aa17057 [Follower]" leader-address= leader-id=
2025-02-23T18:43:15.759Z [WARN]  server-ae158d48-2616-2bcf-e6c9-e1a9036d753b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:15.759Z [INFO]  server-ae158d48-2616-2bcf-e6c9-e1a9036d753b: entering candidate state: node="Node at ae158d48-2616-2bcf-e6c9-e1a9036d753b [Candidate]" term=2
2025-02-23T18:43:15.759Z [INFO]  server-ae158d48-2616-2bcf-e6c9-e1a9036d753b: election won: term=2 tally=2
2025-02-23T18:43:15.759Z [INFO]  server-ae158d48-2616-2bcf-e6c9-e1a9036d753b: entering leader state: leader="Node at ae158d48-2616-2bcf-e6c9-e1a9036d753b [Leader]"
2025-02-23T18:43:15.759Z [INFO]  server-ae158d48-2616-2bcf-e6c9-e1a9036d753b: added peer, starting replication: peer=server-2d1fa3a4-0c69-c171-2588-24e21aa17057
2025-02-23T18:43:15.759Z [INFO]  server-ae158d48-2616-2bcf-e6c9-e1a9036d753b: pipelining replication: peer="{Voter server-2d1fa3a4-0c69-c171-2588-24e21aa17057 2d1fa3a4-0c69-c171-2588-24e21aa17057}"
2025-02-23T18:43:15.866Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:15.759650292 +0000 UTC m=+100.057911114 (1 nodes), 52.164449ms from start of poll, 52.496537ms from cluster start. Timeout at 2025-02-23 18:43:15.86466584 +0000 UTC m=+100.162926472, 105.015358ms after stability
2025-02-23T18:43:15.866Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:15.971Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:15.866152496 +0000 UTC m=+100.164413126 (1 nodes), 1.707µs from start of poll, 158.998549ms from cluster start. Timeout at 2025-02-23 18:43:15.971197699 +0000 UTC m=+100.269458369, 105.045243ms after stability
2025-02-23T18:43:15.974Z [ERROR] server-ae158d48-2616-2bcf-e6c9-e1a9036d753b: peer has newer term, stopping replication: peer="{Voter server-2d1fa3a4-0c69-c171-2588-24e21aa17057 2d1fa3a4-0c69-c171-2588-24e21aa17057}"
2025-02-23T18:43:15.974Z [INFO]  server-ae158d48-2616-2bcf-e6c9-e1a9036d753b: aborting pipeline replication: peer="{Voter server-2d1fa3a4-0c69-c171-2588-24e21aa17057 2d1fa3a4-0c69-c171-2588-24e21aa17057}"
2025-02-23T18:43:15.974Z [INFO]  server-ae158d48-2616-2bcf-e6c9-e1a9036d753b: entering follower state: follower="Node at ae158d48-2616-2bcf-e6c9-e1a9036d753b [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VerifyLeader_Fail (0.27s)
=== RUN   TestRaft_VerifyLeader_PartialConnect
2025-02-23T18:43:15.976Z [INFO]  server-e92db850-23d1-f0a6-7868-a2a4f527596f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e92db850-23d1-f0a6-7868-a2a4f527596f Address:e92db850-23d1-f0a6-7868-a2a4f527596f} {Suffrage:Voter ID:server-a617c573-f31d-ea47-5e4e-ddfb23e86808 Address:a617c573-f31d-ea47-5e4e-ddfb23e86808} {Suffrage:Voter ID:server-bd3af005-df20-fd6f-d485-5ee79962eca8 Address:bd3af005-df20-fd6f-d485-5ee79962eca8}]"
2025-02-23T18:43:15.976Z [INFO]  server-a617c573-f31d-ea47-5e4e-ddfb23e86808: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e92db850-23d1-f0a6-7868-a2a4f527596f Address:e92db850-23d1-f0a6-7868-a2a4f527596f} {Suffrage:Voter ID:server-a617c573-f31d-ea47-5e4e-ddfb23e86808 Address:a617c573-f31d-ea47-5e4e-ddfb23e86808} {Suffrage:Voter ID:server-bd3af005-df20-fd6f-d485-5ee79962eca8 Address:bd3af005-df20-fd6f-d485-5ee79962eca8}]"
2025-02-23T18:43:15.976Z [INFO]  server-bd3af005-df20-fd6f-d485-5ee79962eca8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e92db850-23d1-f0a6-7868-a2a4f527596f Address:e92db850-23d1-f0a6-7868-a2a4f527596f} {Suffrage:Voter ID:server-a617c573-f31d-ea47-5e4e-ddfb23e86808 Address:a617c573-f31d-ea47-5e4e-ddfb23e86808} {Suffrage:Voter ID:server-bd3af005-df20-fd6f-d485-5ee79962eca8 Address:bd3af005-df20-fd6f-d485-5ee79962eca8}]"
2025-02-23T18:43:15.976Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:15.976Z [INFO]  server-e92db850-23d1-f0a6-7868-a2a4f527596f: entering follower state: follower="Node at e92db850-23d1-f0a6-7868-a2a4f527596f [Follower]" leader-address= leader-id=
2025-02-23T18:43:15.976Z [INFO]  server-a617c573-f31d-ea47-5e4e-ddfb23e86808: entering follower state: follower="Node at a617c573-f31d-ea47-5e4e-ddfb23e86808 [Follower]" leader-address= leader-id=
2025-02-23T18:43:15.976Z [INFO]  server-bd3af005-df20-fd6f-d485-5ee79962eca8: entering follower state: follower="Node at bd3af005-df20-fd6f-d485-5ee79962eca8 [Follower]" leader-address= leader-id=
2025-02-23T18:43:16.051Z [WARN]  server-bd3af005-df20-fd6f-d485-5ee79962eca8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:16.051Z [INFO]  server-bd3af005-df20-fd6f-d485-5ee79962eca8: entering candidate state: node="Node at bd3af005-df20-fd6f-d485-5ee79962eca8 [Candidate]" term=2
2025-02-23T18:43:16.051Z [INFO]  server-bd3af005-df20-fd6f-d485-5ee79962eca8: election won: term=2 tally=2
2025-02-23T18:43:16.051Z [INFO]  server-bd3af005-df20-fd6f-d485-5ee79962eca8: entering leader state: leader="Node at bd3af005-df20-fd6f-d485-5ee79962eca8 [Leader]"
2025-02-23T18:43:16.051Z [INFO]  server-bd3af005-df20-fd6f-d485-5ee79962eca8: added peer, starting replication: peer=server-e92db850-23d1-f0a6-7868-a2a4f527596f
2025-02-23T18:43:16.051Z [INFO]  server-bd3af005-df20-fd6f-d485-5ee79962eca8: added peer, starting replication: peer=server-a617c573-f31d-ea47-5e4e-ddfb23e86808
2025-02-23T18:43:16.051Z [INFO]  server-bd3af005-df20-fd6f-d485-5ee79962eca8: pipelining replication: peer="{Voter server-a617c573-f31d-ea47-5e4e-ddfb23e86808 a617c573-f31d-ea47-5e4e-ddfb23e86808}"
2025-02-23T18:43:16.051Z [INFO]  server-bd3af005-df20-fd6f-d485-5ee79962eca8: pipelining replication: peer="{Voter server-e92db850-23d1-f0a6-7868-a2a4f527596f e92db850-23d1-f0a6-7868-a2a4f527596f}"
2025-02-23T18:43:16.156Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:16.051533372 +0000 UTC m=+100.349794248 (1 nodes), 74.792298ms from start of poll, 75.19429ms from cluster start. Timeout at 2025-02-23 18:43:16.156536225 +0000 UTC m=+100.454796849, 105.002601ms after stability
2025-02-23T18:43:16.162Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:16.267Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:16.162377023 +0000 UTC m=+100.460637670 (2 nodes), 2.024µs from start of poll, 186.037712ms from cluster start. Timeout at 2025-02-23 18:43:16.267458521 +0000 UTC m=+100.565719161, 105.081491ms after stability
    raft_test.go:1968: [INFO] Disconnecting Node at e92db850-23d1-f0a6-7868-a2a4f527596f [Follower]
2025-02-23T18:43:16.267Z [ERROR] server-bd3af005-df20-fd6f-d485-5ee79962eca8: failed to heartbeat to: peer=e92db850-23d1-f0a6-7868-a2a4f527596f backoff time=10ms error="failed to connect to peer: e92db850-23d1-f0a6-7868-a2a4f527596f"
2025-02-23T18:43:16.267Z [INFO]  server-bd3af005-df20-fd6f-d485-5ee79962eca8: aborting pipeline replication: peer="{Voter server-e92db850-23d1-f0a6-7868-a2a4f527596f e92db850-23d1-f0a6-7868-a2a4f527596f}"
2025-02-23T18:43:16.268Z [INFO]  server-bd3af005-df20-fd6f-d485-5ee79962eca8: aborting pipeline replication: peer="{Voter server-a617c573-f31d-ea47-5e4e-ddfb23e86808 a617c573-f31d-ea47-5e4e-ddfb23e86808}"
--- PASS: TestRaft_VerifyLeader_PartialConnect (0.29s)
=== RUN   TestRaft_NotifyCh
2025-02-23T18:43:16.269Z [INFO]  server-71c2e753-3383-6b63-3b80-7e882974f6d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-71c2e753-3383-6b63-3b80-7e882974f6d6 Address:71c2e753-3383-6b63-3b80-7e882974f6d6}]"
2025-02-23T18:43:16.269Z [INFO]  server-71c2e753-3383-6b63-3b80-7e882974f6d6: entering follower state: follower="Node at 71c2e753-3383-6b63-3b80-7e882974f6d6 [Follower]" leader-address= leader-id=
2025-02-23T18:43:16.345Z [WARN]  server-71c2e753-3383-6b63-3b80-7e882974f6d6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:16.345Z [INFO]  server-71c2e753-3383-6b63-3b80-7e882974f6d6: entering candidate state: node="Node at 71c2e753-3383-6b63-3b80-7e882974f6d6 [Candidate]" term=2
2025-02-23T18:43:16.345Z [INFO]  server-71c2e753-3383-6b63-3b80-7e882974f6d6: election won: term=2 tally=1
2025-02-23T18:43:16.345Z [INFO]  server-71c2e753-3383-6b63-3b80-7e882974f6d6: entering leader state: leader="Node at 71c2e753-3383-6b63-3b80-7e882974f6d6 [Leader]"
--- PASS: TestRaft_NotifyCh (0.08s)
=== RUN   TestRaft_AppendEntry
2025-02-23T18:43:16.347Z [INFO]  server-2550ede5-4e23-25c7-ce7e-4d2748c25660: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2550ede5-4e23-25c7-ce7e-4d2748c25660 Address:2550ede5-4e23-25c7-ce7e-4d2748c25660} {Suffrage:Voter ID:server-a97cc956-60a0-0fb8-0c06-97db4fd63e68 Address:a97cc956-60a0-0fb8-0c06-97db4fd63e68} {Suffrage:Voter ID:server-41e810dd-9483-34fa-17e1-402ea8cdaf46 Address:41e810dd-9483-34fa-17e1-402ea8cdaf46}]"
2025-02-23T18:43:16.347Z [INFO]  server-a97cc956-60a0-0fb8-0c06-97db4fd63e68: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2550ede5-4e23-25c7-ce7e-4d2748c25660 Address:2550ede5-4e23-25c7-ce7e-4d2748c25660} {Suffrage:Voter ID:server-a97cc956-60a0-0fb8-0c06-97db4fd63e68 Address:a97cc956-60a0-0fb8-0c06-97db4fd63e68} {Suffrage:Voter ID:server-41e810dd-9483-34fa-17e1-402ea8cdaf46 Address:41e810dd-9483-34fa-17e1-402ea8cdaf46}]"
2025-02-23T18:43:16.347Z [INFO]  server-41e810dd-9483-34fa-17e1-402ea8cdaf46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2550ede5-4e23-25c7-ce7e-4d2748c25660 Address:2550ede5-4e23-25c7-ce7e-4d2748c25660} {Suffrage:Voter ID:server-a97cc956-60a0-0fb8-0c06-97db4fd63e68 Address:a97cc956-60a0-0fb8-0c06-97db4fd63e68} {Suffrage:Voter ID:server-41e810dd-9483-34fa-17e1-402ea8cdaf46 Address:41e810dd-9483-34fa-17e1-402ea8cdaf46}]"
2025-02-23T18:43:16.347Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:16.347Z [INFO]  server-2550ede5-4e23-25c7-ce7e-4d2748c25660: entering follower state: follower="Node at 2550ede5-4e23-25c7-ce7e-4d2748c25660 [Follower]" leader-address= leader-id=
2025-02-23T18:43:16.347Z [INFO]  server-a97cc956-60a0-0fb8-0c06-97db4fd63e68: entering follower state: follower="Node at a97cc956-60a0-0fb8-0c06-97db4fd63e68 [Follower]" leader-address= leader-id=
2025-02-23T18:43:16.347Z [INFO]  server-41e810dd-9483-34fa-17e1-402ea8cdaf46: entering follower state: follower="Node at 41e810dd-9483-34fa-17e1-402ea8cdaf46 [Follower]" leader-address= leader-id=
2025-02-23T18:43:16.414Z [WARN]  server-41e810dd-9483-34fa-17e1-402ea8cdaf46: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:16.414Z [INFO]  server-41e810dd-9483-34fa-17e1-402ea8cdaf46: entering candidate state: node="Node at 41e810dd-9483-34fa-17e1-402ea8cdaf46 [Candidate]" term=2
2025-02-23T18:43:16.414Z [INFO]  server-41e810dd-9483-34fa-17e1-402ea8cdaf46: election won: term=2 tally=2
2025-02-23T18:43:16.414Z [INFO]  server-41e810dd-9483-34fa-17e1-402ea8cdaf46: entering leader state: leader="Node at 41e810dd-9483-34fa-17e1-402ea8cdaf46 [Leader]"
2025-02-23T18:43:16.414Z [INFO]  server-41e810dd-9483-34fa-17e1-402ea8cdaf46: added peer, starting replication: peer=server-2550ede5-4e23-25c7-ce7e-4d2748c25660
2025-02-23T18:43:16.414Z [INFO]  server-41e810dd-9483-34fa-17e1-402ea8cdaf46: added peer, starting replication: peer=server-a97cc956-60a0-0fb8-0c06-97db4fd63e68
2025-02-23T18:43:16.414Z [INFO]  server-41e810dd-9483-34fa-17e1-402ea8cdaf46: pipelining replication: peer="{Voter server-a97cc956-60a0-0fb8-0c06-97db4fd63e68 a97cc956-60a0-0fb8-0c06-97db4fd63e68}"
2025-02-23T18:43:16.414Z [INFO]  server-41e810dd-9483-34fa-17e1-402ea8cdaf46: pipelining replication: peer="{Voter server-2550ede5-4e23-25c7-ce7e-4d2748c25660 2550ede5-4e23-25c7-ce7e-4d2748c25660}"
2025-02-23T18:43:16.520Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:16.414823241 +0000 UTC m=+100.713083902 (2 nodes), 67.222408ms from start of poll, 67.610441ms from cluster start. Timeout at 2025-02-23 18:43:16.519833337 +0000 UTC m=+100.818093953, 105.010051ms after stability
2025-02-23T18:43:16.520Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:16.626Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:16.520063464 +0000 UTC m=+100.818324125 (1 nodes), 2.073µs from start of poll, 172.850664ms from cluster start. Timeout at 2025-02-23 18:43:16.625136401 +0000 UTC m=+100.923397020, 105.072895ms after stability
2025-02-23T18:43:16.626Z [WARN]  server-2550ede5-4e23-25c7-ce7e-4d2748c25660: clearing log suffix: from=1 to=2
2025-02-23T18:43:16.626Z [WARN]  server-2550ede5-4e23-25c7-ce7e-4d2748c25660: skipping application of old log: index=1
2025-02-23T18:43:16.626Z [WARN]  server-2550ede5-4e23-25c7-ce7e-4d2748c25660: skipping application of old log: index=1
2025-02-23T18:43:16.626Z [INFO]  server-41e810dd-9483-34fa-17e1-402ea8cdaf46: aborting pipeline replication: peer="{Voter server-a97cc956-60a0-0fb8-0c06-97db4fd63e68 a97cc956-60a0-0fb8-0c06-97db4fd63e68}"
2025-02-23T18:43:16.626Z [INFO]  server-41e810dd-9483-34fa-17e1-402ea8cdaf46: aborting pipeline replication: peer="{Voter server-2550ede5-4e23-25c7-ce7e-4d2748c25660 2550ede5-4e23-25c7-ce7e-4d2748c25660}"
--- PASS: TestRaft_AppendEntry (0.28s)
=== RUN   TestRaft_VotingGrant_WhenLeaderAvailable
2025-02-23T18:43:16.628Z [INFO]  server-48dfceef-d6fc-62df-97b3-eb7108323b00: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48dfceef-d6fc-62df-97b3-eb7108323b00 Address:48dfceef-d6fc-62df-97b3-eb7108323b00} {Suffrage:Voter ID:server-5de1eedf-cb7d-c47f-7faa-ea7af8339c71 Address:5de1eedf-cb7d-c47f-7faa-ea7af8339c71} {Suffrage:Voter ID:server-b64cdb00-12a3-c568-487a-283a3fefb26e Address:b64cdb00-12a3-c568-487a-283a3fefb26e}]"
2025-02-23T18:43:16.628Z [INFO]  server-5de1eedf-cb7d-c47f-7faa-ea7af8339c71: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48dfceef-d6fc-62df-97b3-eb7108323b00 Address:48dfceef-d6fc-62df-97b3-eb7108323b00} {Suffrage:Voter ID:server-5de1eedf-cb7d-c47f-7faa-ea7af8339c71 Address:5de1eedf-cb7d-c47f-7faa-ea7af8339c71} {Suffrage:Voter ID:server-b64cdb00-12a3-c568-487a-283a3fefb26e Address:b64cdb00-12a3-c568-487a-283a3fefb26e}]"
2025-02-23T18:43:16.628Z [INFO]  server-b64cdb00-12a3-c568-487a-283a3fefb26e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48dfceef-d6fc-62df-97b3-eb7108323b00 Address:48dfceef-d6fc-62df-97b3-eb7108323b00} {Suffrage:Voter ID:server-5de1eedf-cb7d-c47f-7faa-ea7af8339c71 Address:5de1eedf-cb7d-c47f-7faa-ea7af8339c71} {Suffrage:Voter ID:server-b64cdb00-12a3-c568-487a-283a3fefb26e Address:b64cdb00-12a3-c568-487a-283a3fefb26e}]"
2025-02-23T18:43:16.628Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:16.628Z [INFO]  server-48dfceef-d6fc-62df-97b3-eb7108323b00: entering follower state: follower="Node at 48dfceef-d6fc-62df-97b3-eb7108323b00 [Follower]" leader-address= leader-id=
2025-02-23T18:43:16.628Z [INFO]  server-5de1eedf-cb7d-c47f-7faa-ea7af8339c71: entering follower state: follower="Node at 5de1eedf-cb7d-c47f-7faa-ea7af8339c71 [Follower]" leader-address= leader-id=
2025-02-23T18:43:16.628Z [INFO]  server-b64cdb00-12a3-c568-487a-283a3fefb26e: entering follower state: follower="Node at b64cdb00-12a3-c568-487a-283a3fefb26e [Follower]" leader-address= leader-id=
2025-02-23T18:43:16.692Z [WARN]  server-b64cdb00-12a3-c568-487a-283a3fefb26e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:16.692Z [INFO]  server-b64cdb00-12a3-c568-487a-283a3fefb26e: entering candidate state: node="Node at b64cdb00-12a3-c568-487a-283a3fefb26e [Candidate]" term=2
2025-02-23T18:43:16.692Z [INFO]  server-b64cdb00-12a3-c568-487a-283a3fefb26e: election won: term=2 tally=2
2025-02-23T18:43:16.692Z [INFO]  server-b64cdb00-12a3-c568-487a-283a3fefb26e: entering leader state: leader="Node at b64cdb00-12a3-c568-487a-283a3fefb26e [Leader]"
2025-02-23T18:43:16.692Z [INFO]  server-b64cdb00-12a3-c568-487a-283a3fefb26e: added peer, starting replication: peer=server-48dfceef-d6fc-62df-97b3-eb7108323b00
2025-02-23T18:43:16.692Z [INFO]  server-b64cdb00-12a3-c568-487a-283a3fefb26e: added peer, starting replication: peer=server-5de1eedf-cb7d-c47f-7faa-ea7af8339c71
2025-02-23T18:43:16.692Z [INFO]  server-b64cdb00-12a3-c568-487a-283a3fefb26e: pipelining replication: peer="{Voter server-48dfceef-d6fc-62df-97b3-eb7108323b00 48dfceef-d6fc-62df-97b3-eb7108323b00}"
2025-02-23T18:43:16.692Z [INFO]  server-b64cdb00-12a3-c568-487a-283a3fefb26e: pipelining replication: peer="{Voter server-5de1eedf-cb7d-c47f-7faa-ea7af8339c71 5de1eedf-cb7d-c47f-7faa-ea7af8339c71}"
2025-02-23T18:43:16.797Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:16.692551166 +0000 UTC m=+100.990811988 (2 nodes), 64.058458ms from start of poll, 64.478115ms from cluster start. Timeout at 2025-02-23 18:43:16.797556109 +0000 UTC m=+101.095816768, 105.00478ms after stability
2025-02-23T18:43:16.797Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:16.904Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:16.797756247 +0000 UTC m=+101.096016898 (1 nodes), 1.95µs from start of poll, 169.683025ms from cluster start. Timeout at 2025-02-23 18:43:16.902842752 +0000 UTC m=+101.201103410, 105.086512ms after stability
2025-02-23T18:43:16.904Z [WARN]  server-5de1eedf-cb7d-c47f-7faa-ea7af8339c71: rejecting vote request since we have a leader: from=48dfceef-d6fc-62df-97b3-eb7108323b00 leader=b64cdb00-12a3-c568-487a-283a3fefb26e leader-id=server-b64cdb00-12a3-c568-487a-283a3fefb26e
2025-02-23T18:43:16.904Z [INFO]  server-b64cdb00-12a3-c568-487a-283a3fefb26e: aborting pipeline replication: peer="{Voter server-5de1eedf-cb7d-c47f-7faa-ea7af8339c71 5de1eedf-cb7d-c47f-7faa-ea7af8339c71}"
2025-02-23T18:43:16.904Z [INFO]  server-b64cdb00-12a3-c568-487a-283a3fefb26e: aborting pipeline replication: peer="{Voter server-48dfceef-d6fc-62df-97b3-eb7108323b00 48dfceef-d6fc-62df-97b3-eb7108323b00}"
--- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.28s)
=== RUN   TestRaft_ProtocolVersion_RejectRPC
2025-02-23T18:43:16.906Z [INFO]  server-d1efa352-486e-cd30-43dd-289cb5e0a00c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d1efa352-486e-cd30-43dd-289cb5e0a00c Address:d1efa352-486e-cd30-43dd-289cb5e0a00c} {Suffrage:Voter ID:server-72ecf6cc-4dcc-5abd-c414-2848b92f34b2 Address:72ecf6cc-4dcc-5abd-c414-2848b92f34b2} {Suffrage:Voter ID:server-816f0ef4-3e3a-e8ac-2ca2-b0391a3a0414 Address:816f0ef4-3e3a-e8ac-2ca2-b0391a3a0414}]"
2025-02-23T18:43:16.906Z [INFO]  server-72ecf6cc-4dcc-5abd-c414-2848b92f34b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d1efa352-486e-cd30-43dd-289cb5e0a00c Address:d1efa352-486e-cd30-43dd-289cb5e0a00c} {Suffrage:Voter ID:server-72ecf6cc-4dcc-5abd-c414-2848b92f34b2 Address:72ecf6cc-4dcc-5abd-c414-2848b92f34b2} {Suffrage:Voter ID:server-816f0ef4-3e3a-e8ac-2ca2-b0391a3a0414 Address:816f0ef4-3e3a-e8ac-2ca2-b0391a3a0414}]"
2025-02-23T18:43:16.906Z [INFO]  server-816f0ef4-3e3a-e8ac-2ca2-b0391a3a0414: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d1efa352-486e-cd30-43dd-289cb5e0a00c Address:d1efa352-486e-cd30-43dd-289cb5e0a00c} {Suffrage:Voter ID:server-72ecf6cc-4dcc-5abd-c414-2848b92f34b2 Address:72ecf6cc-4dcc-5abd-c414-2848b92f34b2} {Suffrage:Voter ID:server-816f0ef4-3e3a-e8ac-2ca2-b0391a3a0414 Address:816f0ef4-3e3a-e8ac-2ca2-b0391a3a0414}]"
2025-02-23T18:43:16.906Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:16.906Z [INFO]  server-d1efa352-486e-cd30-43dd-289cb5e0a00c: entering follower state: follower="Node at d1efa352-486e-cd30-43dd-289cb5e0a00c [Follower]" leader-address= leader-id=
2025-02-23T18:43:16.906Z [INFO]  server-72ecf6cc-4dcc-5abd-c414-2848b92f34b2: entering follower state: follower="Node at 72ecf6cc-4dcc-5abd-c414-2848b92f34b2 [Follower]" leader-address= leader-id=
2025-02-23T18:43:16.906Z [INFO]  server-816f0ef4-3e3a-e8ac-2ca2-b0391a3a0414: entering follower state: follower="Node at 816f0ef4-3e3a-e8ac-2ca2-b0391a3a0414 [Follower]" leader-address= leader-id=
2025-02-23T18:43:16.987Z [WARN]  server-72ecf6cc-4dcc-5abd-c414-2848b92f34b2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:16.987Z [INFO]  server-72ecf6cc-4dcc-5abd-c414-2848b92f34b2: entering candidate state: node="Node at 72ecf6cc-4dcc-5abd-c414-2848b92f34b2 [Candidate]" term=2
2025-02-23T18:43:16.987Z [INFO]  server-72ecf6cc-4dcc-5abd-c414-2848b92f34b2: election won: term=2 tally=2
2025-02-23T18:43:16.987Z [INFO]  server-72ecf6cc-4dcc-5abd-c414-2848b92f34b2: entering leader state: leader="Node at 72ecf6cc-4dcc-5abd-c414-2848b92f34b2 [Leader]"
2025-02-23T18:43:16.987Z [INFO]  server-72ecf6cc-4dcc-5abd-c414-2848b92f34b2: added peer, starting replication: peer=server-d1efa352-486e-cd30-43dd-289cb5e0a00c
2025-02-23T18:43:16.987Z [INFO]  server-72ecf6cc-4dcc-5abd-c414-2848b92f34b2: added peer, starting replication: peer=server-816f0ef4-3e3a-e8ac-2ca2-b0391a3a0414
2025-02-23T18:43:16.987Z [INFO]  server-72ecf6cc-4dcc-5abd-c414-2848b92f34b2: pipelining replication: peer="{Voter server-816f0ef4-3e3a-e8ac-2ca2-b0391a3a0414 816f0ef4-3e3a-e8ac-2ca2-b0391a3a0414}"
2025-02-23T18:43:16.987Z [INFO]  server-72ecf6cc-4dcc-5abd-c414-2848b92f34b2: pipelining replication: peer="{Voter server-d1efa352-486e-cd30-43dd-289cb5e0a00c d1efa352-486e-cd30-43dd-289cb5e0a00c}"
2025-02-23T18:43:17.092Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:16.987535763 +0000 UTC m=+101.285795893 (2 nodes), 81.106692ms from start of poll, 81.490041ms from cluster start. Timeout at 2025-02-23 18:43:17.092548918 +0000 UTC m=+101.390809048, 105.013155ms after stability
2025-02-23T18:43:17.092Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:17.199Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:17.09265497 +0000 UTC m=+101.390915100 (1 nodes), 3.438µs from start of poll, 186.609248ms from cluster start. Timeout at 2025-02-23 18:43:17.197729441 +0000 UTC m=+101.495990074, 105.074974ms after stability
2025-02-23T18:43:17.199Z [INFO]  server-72ecf6cc-4dcc-5abd-c414-2848b92f34b2: aborting pipeline replication: peer="{Voter server-816f0ef4-3e3a-e8ac-2ca2-b0391a3a0414 816f0ef4-3e3a-e8ac-2ca2-b0391a3a0414}"
2025-02-23T18:43:17.199Z [INFO]  server-72ecf6cc-4dcc-5abd-c414-2848b92f34b2: aborting pipeline replication: peer="{Voter server-d1efa352-486e-cd30-43dd-289cb5e0a00c d1efa352-486e-cd30-43dd-289cb5e0a00c}"
--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.29s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_1_2
    raft_test.go:2151: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_2_3
    raft_test.go:2196: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s)
=== RUN   TestRaft_LeaderID_Propagated
2025-02-23T18:43:17.200Z [INFO]  server-507f8fe6-4823-50ab-45d1-a46de5f2fd09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-507f8fe6-4823-50ab-45d1-a46de5f2fd09 Address:507f8fe6-4823-50ab-45d1-a46de5f2fd09} {Suffrage:Voter ID:server-1c6438a5-a796-e162-459e-8227df278634 Address:1c6438a5-a796-e162-459e-8227df278634} {Suffrage:Voter ID:server-757038de-0ef0-4569-8323-25e3b1a243bb Address:757038de-0ef0-4569-8323-25e3b1a243bb}]"
2025-02-23T18:43:17.200Z [INFO]  server-1c6438a5-a796-e162-459e-8227df278634: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-507f8fe6-4823-50ab-45d1-a46de5f2fd09 Address:507f8fe6-4823-50ab-45d1-a46de5f2fd09} {Suffrage:Voter ID:server-1c6438a5-a796-e162-459e-8227df278634 Address:1c6438a5-a796-e162-459e-8227df278634} {Suffrage:Voter ID:server-757038de-0ef0-4569-8323-25e3b1a243bb Address:757038de-0ef0-4569-8323-25e3b1a243bb}]"
2025-02-23T18:43:17.201Z [INFO]  server-757038de-0ef0-4569-8323-25e3b1a243bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-507f8fe6-4823-50ab-45d1-a46de5f2fd09 Address:507f8fe6-4823-50ab-45d1-a46de5f2fd09} {Suffrage:Voter ID:server-1c6438a5-a796-e162-459e-8227df278634 Address:1c6438a5-a796-e162-459e-8227df278634} {Suffrage:Voter ID:server-757038de-0ef0-4569-8323-25e3b1a243bb Address:757038de-0ef0-4569-8323-25e3b1a243bb}]"
2025-02-23T18:43:17.201Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:17.201Z [INFO]  server-507f8fe6-4823-50ab-45d1-a46de5f2fd09: entering follower state: follower="Node at 507f8fe6-4823-50ab-45d1-a46de5f2fd09 [Follower]" leader-address= leader-id=
2025-02-23T18:43:17.201Z [INFO]  server-757038de-0ef0-4569-8323-25e3b1a243bb: entering follower state: follower="Node at 757038de-0ef0-4569-8323-25e3b1a243bb [Follower]" leader-address= leader-id=
2025-02-23T18:43:17.201Z [INFO]  server-1c6438a5-a796-e162-459e-8227df278634: entering follower state: follower="Node at 1c6438a5-a796-e162-459e-8227df278634 [Follower]" leader-address= leader-id=
2025-02-23T18:43:17.253Z [WARN]  server-1c6438a5-a796-e162-459e-8227df278634: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:17.253Z [INFO]  server-1c6438a5-a796-e162-459e-8227df278634: entering candidate state: node="Node at 1c6438a5-a796-e162-459e-8227df278634 [Candidate]" term=2
2025-02-23T18:43:17.254Z [INFO]  server-1c6438a5-a796-e162-459e-8227df278634: election won: term=2 tally=2
2025-02-23T18:43:17.254Z [INFO]  server-1c6438a5-a796-e162-459e-8227df278634: entering leader state: leader="Node at 1c6438a5-a796-e162-459e-8227df278634 [Leader]"
2025-02-23T18:43:17.254Z [INFO]  server-1c6438a5-a796-e162-459e-8227df278634: added peer, starting replication: peer=server-507f8fe6-4823-50ab-45d1-a46de5f2fd09
2025-02-23T18:43:17.254Z [INFO]  server-1c6438a5-a796-e162-459e-8227df278634: added peer, starting replication: peer=server-757038de-0ef0-4569-8323-25e3b1a243bb
2025-02-23T18:43:17.254Z [INFO]  server-1c6438a5-a796-e162-459e-8227df278634: pipelining replication: peer="{Voter server-757038de-0ef0-4569-8323-25e3b1a243bb 757038de-0ef0-4569-8323-25e3b1a243bb}"
2025-02-23T18:43:17.254Z [INFO]  server-1c6438a5-a796-e162-459e-8227df278634: pipelining replication: peer="{Voter server-507f8fe6-4823-50ab-45d1-a46de5f2fd09 507f8fe6-4823-50ab-45d1-a46de5f2fd09}"
2025-02-23T18:43:17.359Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:17.25419732 +0000 UTC m=+101.552457964 (1 nodes), 52.986552ms from start of poll, 53.46328ms from cluster start. Timeout at 2025-02-23 18:43:17.359209276 +0000 UTC m=+101.657470483, 105.012519ms after stability
2025-02-23T18:43:17.359Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:17.464Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:17.359371046 +0000 UTC m=+101.657631681 (1 nodes), 1.757µs from start of poll, 158.636997ms from cluster start. Timeout at 2025-02-23 18:43:17.464472532 +0000 UTC m=+101.762733161, 105.10148ms after stability
2025-02-23T18:43:17.465Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:17.570Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:17.465425869 +0000 UTC m=+101.763686484 (1 nodes), 1.676µs from start of poll, 264.6918ms from cluster start. Timeout at 2025-02-23 18:43:17.570486243 +0000 UTC m=+101.868746883, 105.060399ms after stability
2025-02-23T18:43:17.571Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:17.676Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:17.57122017 +0000 UTC m=+101.869480803 (1 nodes), 1.723µs from start of poll, 370.486119ms from cluster start. Timeout at 2025-02-23 18:43:17.676273375 +0000 UTC m=+101.974534002, 105.053199ms after stability
2025-02-23T18:43:17.677Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:17.782Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:17.677187341 +0000 UTC m=+101.975447471 (1 nodes), 0s from start of poll, 476.452787ms from cluster start. Timeout at 2025-02-23 18:43:17.782214744 +0000 UTC m=+102.080474874, 105.027403ms after stability
2025-02-23T18:43:17.782Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:17.888Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:17.782891914 +0000 UTC m=+102.081152044 (1 nodes), 10.315µs from start of poll, 582.15736ms from cluster start. Timeout at 2025-02-23 18:43:17.887944606 +0000 UTC m=+102.186205275, 105.053231ms after stability
2025-02-23T18:43:17.993Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:18.099Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:17.993977459 +0000 UTC m=+102.292238305 (1 nodes), 2.13µs from start of poll, 793.243621ms from cluster start. Timeout at 2025-02-23 18:43:18.099014198 +0000 UTC m=+102.397274979, 105.036674ms after stability
2025-02-23T18:43:18.100Z [INFO]  server-1c6438a5-a796-e162-459e-8227df278634: aborting pipeline replication: peer="{Voter server-757038de-0ef0-4569-8323-25e3b1a243bb 757038de-0ef0-4569-8323-25e3b1a243bb}"
2025-02-23T18:43:18.100Z [INFO]  server-1c6438a5-a796-e162-459e-8227df278634: aborting pipeline replication: peer="{Voter server-507f8fe6-4823-50ab-45d1-a46de5f2fd09 507f8fe6-4823-50ab-45d1-a46de5f2fd09}"
--- PASS: TestRaft_LeaderID_Propagated (0.90s)
=== RUN   TestRaft_LeadershipTransferInProgress
--- PASS: TestRaft_LeadershipTransferInProgress (0.00s)
=== RUN   TestRaft_LeadershipTransferPickServer
--- PASS: TestRaft_LeadershipTransferPickServer (0.00s)
=== RUN   TestRaft_LeadershipTransfer
2025-02-23T18:43:18.101Z [INFO]  server-d695d146-9be5-983e-fe27-7afe60d47d8a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d695d146-9be5-983e-fe27-7afe60d47d8a Address:d695d146-9be5-983e-fe27-7afe60d47d8a} {Suffrage:Voter ID:server-a46a0310-3bbd-b357-51b7-7c180b291526 Address:a46a0310-3bbd-b357-51b7-7c180b291526} {Suffrage:Voter ID:server-918ed159-6d50-0831-9f84-d0ac8cd4ff71 Address:918ed159-6d50-0831-9f84-d0ac8cd4ff71}]"
2025-02-23T18:43:18.101Z [INFO]  server-d695d146-9be5-983e-fe27-7afe60d47d8a: entering follower state: follower="Node at d695d146-9be5-983e-fe27-7afe60d47d8a [Follower]" leader-address= leader-id=
2025-02-23T18:43:18.101Z [INFO]  server-a46a0310-3bbd-b357-51b7-7c180b291526: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d695d146-9be5-983e-fe27-7afe60d47d8a Address:d695d146-9be5-983e-fe27-7afe60d47d8a} {Suffrage:Voter ID:server-a46a0310-3bbd-b357-51b7-7c180b291526 Address:a46a0310-3bbd-b357-51b7-7c180b291526} {Suffrage:Voter ID:server-918ed159-6d50-0831-9f84-d0ac8cd4ff71 Address:918ed159-6d50-0831-9f84-d0ac8cd4ff71}]"
2025-02-23T18:43:18.102Z [INFO]  server-a46a0310-3bbd-b357-51b7-7c180b291526: entering follower state: follower="Node at a46a0310-3bbd-b357-51b7-7c180b291526 [Follower]" leader-address= leader-id=
2025-02-23T18:43:18.102Z [INFO]  server-918ed159-6d50-0831-9f84-d0ac8cd4ff71: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d695d146-9be5-983e-fe27-7afe60d47d8a Address:d695d146-9be5-983e-fe27-7afe60d47d8a} {Suffrage:Voter ID:server-a46a0310-3bbd-b357-51b7-7c180b291526 Address:a46a0310-3bbd-b357-51b7-7c180b291526} {Suffrage:Voter ID:server-918ed159-6d50-0831-9f84-d0ac8cd4ff71 Address:918ed159-6d50-0831-9f84-d0ac8cd4ff71}]"
2025-02-23T18:43:18.102Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:18.102Z [INFO]  server-918ed159-6d50-0831-9f84-d0ac8cd4ff71: entering follower state: follower="Node at 918ed159-6d50-0831-9f84-d0ac8cd4ff71 [Follower]" leader-address= leader-id=
2025-02-23T18:43:18.155Z [WARN]  server-918ed159-6d50-0831-9f84-d0ac8cd4ff71: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:18.155Z [INFO]  server-918ed159-6d50-0831-9f84-d0ac8cd4ff71: entering candidate state: node="Node at 918ed159-6d50-0831-9f84-d0ac8cd4ff71 [Candidate]" term=2
2025-02-23T18:43:18.155Z [INFO]  server-918ed159-6d50-0831-9f84-d0ac8cd4ff71: election won: term=2 tally=2
2025-02-23T18:43:18.155Z [INFO]  server-918ed159-6d50-0831-9f84-d0ac8cd4ff71: entering leader state: leader="Node at 918ed159-6d50-0831-9f84-d0ac8cd4ff71 [Leader]"
2025-02-23T18:43:18.155Z [INFO]  server-918ed159-6d50-0831-9f84-d0ac8cd4ff71: added peer, starting replication: peer=server-d695d146-9be5-983e-fe27-7afe60d47d8a
2025-02-23T18:43:18.155Z [INFO]  server-918ed159-6d50-0831-9f84-d0ac8cd4ff71: added peer, starting replication: peer=server-a46a0310-3bbd-b357-51b7-7c180b291526
2025-02-23T18:43:18.156Z [INFO]  server-918ed159-6d50-0831-9f84-d0ac8cd4ff71: pipelining replication: peer="{Voter server-a46a0310-3bbd-b357-51b7-7c180b291526 a46a0310-3bbd-b357-51b7-7c180b291526}"
2025-02-23T18:43:18.156Z [WARN]  server-a46a0310-3bbd-b357-51b7-7c180b291526: rejecting vote request since our last term is greater: candidate=918ed159-6d50-0831-9f84-d0ac8cd4ff71 last-term=2 last-candidate-term=1
2025-02-23T18:43:18.156Z [INFO]  server-918ed159-6d50-0831-9f84-d0ac8cd4ff71: pipelining replication: peer="{Voter server-d695d146-9be5-983e-fe27-7afe60d47d8a d695d146-9be5-983e-fe27-7afe60d47d8a}"
2025-02-23T18:43:18.261Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:18.156328189 +0000 UTC m=+102.454588824 (1 nodes), 54.025067ms from start of poll, 54.581094ms from cluster start. Timeout at 2025-02-23 18:43:18.261330489 +0000 UTC m=+102.559591132, 105.002308ms after stability
2025-02-23T18:43:18.261Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:18.366Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:18.261416727 +0000 UTC m=+102.559677814 (1 nodes), 2.084µs from start of poll, 159.670084ms from cluster start. Timeout at 2025-02-23 18:43:18.366459643 +0000 UTC m=+102.664720537, 105.042723ms after stability
2025-02-23T18:43:18.366Z [INFO]  server-d695d146-9be5-983e-fe27-7afe60d47d8a: entering candidate state: node="Node at d695d146-9be5-983e-fe27-7afe60d47d8a [Candidate]" term=3
2025-02-23T18:43:18.366Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:18.366Z [INFO]  server-918ed159-6d50-0831-9f84-d0ac8cd4ff71: entering follower state: follower="Node at 918ed159-6d50-0831-9f84-d0ac8cd4ff71 [Follower]" leader-address= leader-id=
2025-02-23T18:43:18.366Z [INFO]  server-918ed159-6d50-0831-9f84-d0ac8cd4ff71: aborting pipeline replication: peer="{Voter server-a46a0310-3bbd-b357-51b7-7c180b291526 a46a0310-3bbd-b357-51b7-7c180b291526}"
2025-02-23T18:43:18.366Z [INFO]  server-d695d146-9be5-983e-fe27-7afe60d47d8a: election won: term=3 tally=2
2025-02-23T18:43:18.366Z [INFO]  server-918ed159-6d50-0831-9f84-d0ac8cd4ff71: aborting pipeline replication: peer="{Voter server-d695d146-9be5-983e-fe27-7afe60d47d8a d695d146-9be5-983e-fe27-7afe60d47d8a}"
2025-02-23T18:43:18.366Z [INFO]  server-d695d146-9be5-983e-fe27-7afe60d47d8a: entering leader state: leader="Node at d695d146-9be5-983e-fe27-7afe60d47d8a [Leader]"
2025-02-23T18:43:18.366Z [INFO]  server-d695d146-9be5-983e-fe27-7afe60d47d8a: added peer, starting replication: peer=server-a46a0310-3bbd-b357-51b7-7c180b291526
2025-02-23T18:43:18.366Z [INFO]  server-d695d146-9be5-983e-fe27-7afe60d47d8a: added peer, starting replication: peer=server-918ed159-6d50-0831-9f84-d0ac8cd4ff71
2025-02-23T18:43:18.367Z [INFO]  server-d695d146-9be5-983e-fe27-7afe60d47d8a: pipelining replication: peer="{Voter server-a46a0310-3bbd-b357-51b7-7c180b291526 a46a0310-3bbd-b357-51b7-7c180b291526}"
2025-02-23T18:43:18.367Z [INFO]  server-d695d146-9be5-983e-fe27-7afe60d47d8a: pipelining replication: peer="{Voter server-918ed159-6d50-0831-9f84-d0ac8cd4ff71 918ed159-6d50-0831-9f84-d0ac8cd4ff71}"
2025-02-23T18:43:18.472Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:18.366977375 +0000 UTC m=+102.665238055 (1 nodes), 200.824µs from start of poll, 265.230325ms from cluster start. Timeout at 2025-02-23 18:43:18.472020434 +0000 UTC m=+102.770281270, 105.043215ms after stability
2025-02-23T18:43:18.473Z [INFO]  server-d695d146-9be5-983e-fe27-7afe60d47d8a: aborting pipeline replication: peer="{Voter server-918ed159-6d50-0831-9f84-d0ac8cd4ff71 918ed159-6d50-0831-9f84-d0ac8cd4ff71}"
2025-02-23T18:43:18.473Z [INFO]  server-d695d146-9be5-983e-fe27-7afe60d47d8a: aborting pipeline replication: peer="{Voter server-a46a0310-3bbd-b357-51b7-7c180b291526 a46a0310-3bbd-b357-51b7-7c180b291526}"
--- PASS: TestRaft_LeadershipTransfer (0.37s)
=== RUN   TestRaft_LeadershipTransferWithOneNode
2025-02-23T18:43:18.474Z [INFO]  server-5a527a26-6d3d-f70c-7760-fc956563799d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a527a26-6d3d-f70c-7760-fc956563799d Address:5a527a26-6d3d-f70c-7760-fc956563799d}]"
2025-02-23T18:43:18.474Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:18.474Z [INFO]  server-5a527a26-6d3d-f70c-7760-fc956563799d: entering follower state: follower="Node at 5a527a26-6d3d-f70c-7760-fc956563799d [Follower]" leader-address= leader-id=
2025-02-23T18:43:18.543Z [WARN]  server-5a527a26-6d3d-f70c-7760-fc956563799d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:18.543Z [INFO]  server-5a527a26-6d3d-f70c-7760-fc956563799d: entering candidate state: node="Node at 5a527a26-6d3d-f70c-7760-fc956563799d [Candidate]" term=2
2025-02-23T18:43:18.543Z [INFO]  server-5a527a26-6d3d-f70c-7760-fc956563799d: election won: term=2 tally=1
2025-02-23T18:43:18.543Z [INFO]  server-5a527a26-6d3d-f70c-7760-fc956563799d: entering leader state: leader="Node at 5a527a26-6d3d-f70c-7760-fc956563799d [Leader]"
2025-02-23T18:43:18.650Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:18.543574946 +0000 UTC m=+102.841836907 (1 nodes), 69.054214ms from start of poll, 69.235616ms from cluster start. Timeout at 2025-02-23 18:43:18.648597693 +0000 UTC m=+102.946858442, 105.021535ms after stability
--- PASS: TestRaft_LeadershipTransferWithOneNode (0.18s)
=== RUN   TestRaft_LeadershipTransferWithSevenNodes
2025-02-23T18:43:18.656Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce0b2325-2791-90a9-e086-e32f5d68eb5f Address:ce0b2325-2791-90a9-e086-e32f5d68eb5f} {Suffrage:Voter ID:server-212e764c-acde-6bde-90ba-f83e08d58dcd Address:212e764c-acde-6bde-90ba-f83e08d58dcd} {Suffrage:Voter ID:server-46bc41e7-e0b1-b6e6-7490-65ee7087c46b Address:46bc41e7-e0b1-b6e6-7490-65ee7087c46b} {Suffrage:Voter ID:server-6580279d-1385-cc46-1f2f-38f3d844649a Address:6580279d-1385-cc46-1f2f-38f3d844649a} {Suffrage:Voter ID:server-06041430-2866-ffc1-bd86-715625c09f5e Address:06041430-2866-ffc1-bd86-715625c09f5e} {Suffrage:Voter ID:server-d09c3bf3-3ffc-2e3c-344c-efcc9116214d Address:d09c3bf3-3ffc-2e3c-344c-efcc9116214d} {Suffrage:Voter ID:server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0 Address:c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0}]"
2025-02-23T18:43:18.656Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: entering follower state: follower="Node at ce0b2325-2791-90a9-e086-e32f5d68eb5f [Follower]" leader-address= leader-id=
2025-02-23T18:43:18.656Z [INFO]  server-212e764c-acde-6bde-90ba-f83e08d58dcd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce0b2325-2791-90a9-e086-e32f5d68eb5f Address:ce0b2325-2791-90a9-e086-e32f5d68eb5f} {Suffrage:Voter ID:server-212e764c-acde-6bde-90ba-f83e08d58dcd Address:212e764c-acde-6bde-90ba-f83e08d58dcd} {Suffrage:Voter ID:server-46bc41e7-e0b1-b6e6-7490-65ee7087c46b Address:46bc41e7-e0b1-b6e6-7490-65ee7087c46b} {Suffrage:Voter ID:server-6580279d-1385-cc46-1f2f-38f3d844649a Address:6580279d-1385-cc46-1f2f-38f3d844649a} {Suffrage:Voter ID:server-06041430-2866-ffc1-bd86-715625c09f5e Address:06041430-2866-ffc1-bd86-715625c09f5e} {Suffrage:Voter ID:server-d09c3bf3-3ffc-2e3c-344c-efcc9116214d Address:d09c3bf3-3ffc-2e3c-344c-efcc9116214d} {Suffrage:Voter ID:server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0 Address:c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0}]"
2025-02-23T18:43:18.656Z [INFO]  server-212e764c-acde-6bde-90ba-f83e08d58dcd: entering follower state: follower="Node at 212e764c-acde-6bde-90ba-f83e08d58dcd [Follower]" leader-address= leader-id=
2025-02-23T18:43:18.656Z [INFO]  server-46bc41e7-e0b1-b6e6-7490-65ee7087c46b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce0b2325-2791-90a9-e086-e32f5d68eb5f Address:ce0b2325-2791-90a9-e086-e32f5d68eb5f} {Suffrage:Voter ID:server-212e764c-acde-6bde-90ba-f83e08d58dcd Address:212e764c-acde-6bde-90ba-f83e08d58dcd} {Suffrage:Voter ID:server-46bc41e7-e0b1-b6e6-7490-65ee7087c46b Address:46bc41e7-e0b1-b6e6-7490-65ee7087c46b} {Suffrage:Voter ID:server-6580279d-1385-cc46-1f2f-38f3d844649a Address:6580279d-1385-cc46-1f2f-38f3d844649a} {Suffrage:Voter ID:server-06041430-2866-ffc1-bd86-715625c09f5e Address:06041430-2866-ffc1-bd86-715625c09f5e} {Suffrage:Voter ID:server-d09c3bf3-3ffc-2e3c-344c-efcc9116214d Address:d09c3bf3-3ffc-2e3c-344c-efcc9116214d} {Suffrage:Voter ID:server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0 Address:c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0}]"
2025-02-23T18:43:18.656Z [INFO]  server-46bc41e7-e0b1-b6e6-7490-65ee7087c46b: entering follower state: follower="Node at 46bc41e7-e0b1-b6e6-7490-65ee7087c46b [Follower]" leader-address= leader-id=
2025-02-23T18:43:18.656Z [INFO]  server-6580279d-1385-cc46-1f2f-38f3d844649a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce0b2325-2791-90a9-e086-e32f5d68eb5f Address:ce0b2325-2791-90a9-e086-e32f5d68eb5f} {Suffrage:Voter ID:server-212e764c-acde-6bde-90ba-f83e08d58dcd Address:212e764c-acde-6bde-90ba-f83e08d58dcd} {Suffrage:Voter ID:server-46bc41e7-e0b1-b6e6-7490-65ee7087c46b Address:46bc41e7-e0b1-b6e6-7490-65ee7087c46b} {Suffrage:Voter ID:server-6580279d-1385-cc46-1f2f-38f3d844649a Address:6580279d-1385-cc46-1f2f-38f3d844649a} {Suffrage:Voter ID:server-06041430-2866-ffc1-bd86-715625c09f5e Address:06041430-2866-ffc1-bd86-715625c09f5e} {Suffrage:Voter ID:server-d09c3bf3-3ffc-2e3c-344c-efcc9116214d Address:d09c3bf3-3ffc-2e3c-344c-efcc9116214d} {Suffrage:Voter ID:server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0 Address:c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0}]"
2025-02-23T18:43:18.656Z [INFO]  server-6580279d-1385-cc46-1f2f-38f3d844649a: entering follower state: follower="Node at 6580279d-1385-cc46-1f2f-38f3d844649a [Follower]" leader-address= leader-id=
2025-02-23T18:43:18.656Z [INFO]  server-06041430-2866-ffc1-bd86-715625c09f5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce0b2325-2791-90a9-e086-e32f5d68eb5f Address:ce0b2325-2791-90a9-e086-e32f5d68eb5f} {Suffrage:Voter ID:server-212e764c-acde-6bde-90ba-f83e08d58dcd Address:212e764c-acde-6bde-90ba-f83e08d58dcd} {Suffrage:Voter ID:server-46bc41e7-e0b1-b6e6-7490-65ee7087c46b Address:46bc41e7-e0b1-b6e6-7490-65ee7087c46b} {Suffrage:Voter ID:server-6580279d-1385-cc46-1f2f-38f3d844649a Address:6580279d-1385-cc46-1f2f-38f3d844649a} {Suffrage:Voter ID:server-06041430-2866-ffc1-bd86-715625c09f5e Address:06041430-2866-ffc1-bd86-715625c09f5e} {Suffrage:Voter ID:server-d09c3bf3-3ffc-2e3c-344c-efcc9116214d Address:d09c3bf3-3ffc-2e3c-344c-efcc9116214d} {Suffrage:Voter ID:server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0 Address:c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0}]"
2025-02-23T18:43:18.656Z [INFO]  server-d09c3bf3-3ffc-2e3c-344c-efcc9116214d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce0b2325-2791-90a9-e086-e32f5d68eb5f Address:ce0b2325-2791-90a9-e086-e32f5d68eb5f} {Suffrage:Voter ID:server-212e764c-acde-6bde-90ba-f83e08d58dcd Address:212e764c-acde-6bde-90ba-f83e08d58dcd} {Suffrage:Voter ID:server-46bc41e7-e0b1-b6e6-7490-65ee7087c46b Address:46bc41e7-e0b1-b6e6-7490-65ee7087c46b} {Suffrage:Voter ID:server-6580279d-1385-cc46-1f2f-38f3d844649a Address:6580279d-1385-cc46-1f2f-38f3d844649a} {Suffrage:Voter ID:server-06041430-2866-ffc1-bd86-715625c09f5e Address:06041430-2866-ffc1-bd86-715625c09f5e} {Suffrage:Voter ID:server-d09c3bf3-3ffc-2e3c-344c-efcc9116214d Address:d09c3bf3-3ffc-2e3c-344c-efcc9116214d} {Suffrage:Voter ID:server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0 Address:c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0}]"
2025-02-23T18:43:18.656Z [INFO]  server-06041430-2866-ffc1-bd86-715625c09f5e: entering follower state: follower="Node at 06041430-2866-ffc1-bd86-715625c09f5e [Follower]" leader-address= leader-id=
2025-02-23T18:43:18.656Z [INFO]  server-d09c3bf3-3ffc-2e3c-344c-efcc9116214d: entering follower state: follower="Node at d09c3bf3-3ffc-2e3c-344c-efcc9116214d [Follower]" leader-address= leader-id=
2025-02-23T18:43:18.657Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce0b2325-2791-90a9-e086-e32f5d68eb5f Address:ce0b2325-2791-90a9-e086-e32f5d68eb5f} {Suffrage:Voter ID:server-212e764c-acde-6bde-90ba-f83e08d58dcd Address:212e764c-acde-6bde-90ba-f83e08d58dcd} {Suffrage:Voter ID:server-46bc41e7-e0b1-b6e6-7490-65ee7087c46b Address:46bc41e7-e0b1-b6e6-7490-65ee7087c46b} {Suffrage:Voter ID:server-6580279d-1385-cc46-1f2f-38f3d844649a Address:6580279d-1385-cc46-1f2f-38f3d844649a} {Suffrage:Voter ID:server-06041430-2866-ffc1-bd86-715625c09f5e Address:06041430-2866-ffc1-bd86-715625c09f5e} {Suffrage:Voter ID:server-d09c3bf3-3ffc-2e3c-344c-efcc9116214d Address:d09c3bf3-3ffc-2e3c-344c-efcc9116214d} {Suffrage:Voter ID:server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0 Address:c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0}]"
2025-02-23T18:43:18.657Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:18.657Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: entering follower state: follower="Node at c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0 [Follower]" leader-address= leader-id=
2025-02-23T18:43:18.713Z [WARN]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:18.713Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: entering candidate state: node="Node at c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0 [Candidate]" term=2
2025-02-23T18:43:18.713Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: election won: term=2 tally=4
2025-02-23T18:43:18.713Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: entering leader state: leader="Node at c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0 [Leader]"
2025-02-23T18:43:18.713Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: added peer, starting replication: peer=server-ce0b2325-2791-90a9-e086-e32f5d68eb5f
2025-02-23T18:43:18.713Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: added peer, starting replication: peer=server-212e764c-acde-6bde-90ba-f83e08d58dcd
2025-02-23T18:43:18.713Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: added peer, starting replication: peer=server-46bc41e7-e0b1-b6e6-7490-65ee7087c46b
2025-02-23T18:43:18.713Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: added peer, starting replication: peer=server-6580279d-1385-cc46-1f2f-38f3d844649a
2025-02-23T18:43:18.713Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: added peer, starting replication: peer=server-06041430-2866-ffc1-bd86-715625c09f5e
2025-02-23T18:43:18.713Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: added peer, starting replication: peer=server-d09c3bf3-3ffc-2e3c-344c-efcc9116214d
2025-02-23T18:43:18.713Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: pipelining replication: peer="{Voter server-d09c3bf3-3ffc-2e3c-344c-efcc9116214d d09c3bf3-3ffc-2e3c-344c-efcc9116214d}"
2025-02-23T18:43:18.713Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: pipelining replication: peer="{Voter server-ce0b2325-2791-90a9-e086-e32f5d68eb5f ce0b2325-2791-90a9-e086-e32f5d68eb5f}"
2025-02-23T18:43:18.713Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: pipelining replication: peer="{Voter server-46bc41e7-e0b1-b6e6-7490-65ee7087c46b 46bc41e7-e0b1-b6e6-7490-65ee7087c46b}"
2025-02-23T18:43:18.714Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: pipelining replication: peer="{Voter server-212e764c-acde-6bde-90ba-f83e08d58dcd 212e764c-acde-6bde-90ba-f83e08d58dcd}"
2025-02-23T18:43:18.714Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: pipelining replication: peer="{Voter server-06041430-2866-ffc1-bd86-715625c09f5e 06041430-2866-ffc1-bd86-715625c09f5e}"
2025-02-23T18:43:18.714Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: pipelining replication: peer="{Voter server-6580279d-1385-cc46-1f2f-38f3d844649a 6580279d-1385-cc46-1f2f-38f3d844649a}"
2025-02-23T18:43:18.818Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:18.713445346 +0000 UTC m=+103.011706001 (1 nodes), 56.365392ms from start of poll, 57.650332ms from cluster start. Timeout at 2025-02-23 18:43:18.818449057 +0000 UTC m=+103.116709871, 105.00387ms after stability
2025-02-23T18:43:18.818Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:18.923Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:18.818591131 +0000 UTC m=+103.116852063 (6 nodes), 3.373µs from start of poll, 162.796394ms from cluster start. Timeout at 2025-02-23 18:43:18.923634739 +0000 UTC m=+103.221895475, 105.043412ms after stability
2025-02-23T18:43:18.923Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:19.030Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:18.923989012 +0000 UTC m=+103.222249706 (1 nodes), 2.015µs from start of poll, 268.194037ms from cluster start. Timeout at 2025-02-23 18:43:19.0290268 +0000 UTC m=+103.327287449, 105.037743ms after stability
2025-02-23T18:43:19.030Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: entering candidate state: node="Node at ce0b2325-2791-90a9-e086-e32f5d68eb5f [Candidate]" term=3
2025-02-23T18:43:19.030Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: entering follower state: follower="Node at c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0 [Follower]" leader-address= leader-id=
2025-02-23T18:43:19.030Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: aborting pipeline replication: peer="{Voter server-06041430-2866-ffc1-bd86-715625c09f5e 06041430-2866-ffc1-bd86-715625c09f5e}"
2025-02-23T18:43:19.030Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: aborting pipeline replication: peer="{Voter server-d09c3bf3-3ffc-2e3c-344c-efcc9116214d d09c3bf3-3ffc-2e3c-344c-efcc9116214d}"
2025-02-23T18:43:19.030Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: aborting pipeline replication: peer="{Voter server-ce0b2325-2791-90a9-e086-e32f5d68eb5f ce0b2325-2791-90a9-e086-e32f5d68eb5f}"
2025-02-23T18:43:19.030Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: aborting pipeline replication: peer="{Voter server-212e764c-acde-6bde-90ba-f83e08d58dcd 212e764c-acde-6bde-90ba-f83e08d58dcd}"
2025-02-23T18:43:19.030Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: aborting pipeline replication: peer="{Voter server-46bc41e7-e0b1-b6e6-7490-65ee7087c46b 46bc41e7-e0b1-b6e6-7490-65ee7087c46b}"
2025-02-23T18:43:19.030Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: election won: term=3 tally=4
2025-02-23T18:43:19.030Z [INFO]  server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0: aborting pipeline replication: peer="{Voter server-6580279d-1385-cc46-1f2f-38f3d844649a 6580279d-1385-cc46-1f2f-38f3d844649a}"
2025-02-23T18:43:19.030Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: entering leader state: leader="Node at ce0b2325-2791-90a9-e086-e32f5d68eb5f [Leader]"
2025-02-23T18:43:19.030Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: added peer, starting replication: peer=server-212e764c-acde-6bde-90ba-f83e08d58dcd
2025-02-23T18:43:19.030Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: added peer, starting replication: peer=server-46bc41e7-e0b1-b6e6-7490-65ee7087c46b
2025-02-23T18:43:19.030Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: added peer, starting replication: peer=server-6580279d-1385-cc46-1f2f-38f3d844649a
2025-02-23T18:43:19.030Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: added peer, starting replication: peer=server-06041430-2866-ffc1-bd86-715625c09f5e
2025-02-23T18:43:19.030Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: added peer, starting replication: peer=server-d09c3bf3-3ffc-2e3c-344c-efcc9116214d
2025-02-23T18:43:19.030Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: added peer, starting replication: peer=server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0
2025-02-23T18:43:19.030Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: pipelining replication: peer="{Voter server-212e764c-acde-6bde-90ba-f83e08d58dcd 212e764c-acde-6bde-90ba-f83e08d58dcd}"
2025-02-23T18:43:19.030Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: pipelining replication: peer="{Voter server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0 c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0}"
2025-02-23T18:43:19.031Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:19.031Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: pipelining replication: peer="{Voter server-46bc41e7-e0b1-b6e6-7490-65ee7087c46b 46bc41e7-e0b1-b6e6-7490-65ee7087c46b}"
2025-02-23T18:43:19.031Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: pipelining replication: peer="{Voter server-6580279d-1385-cc46-1f2f-38f3d844649a 6580279d-1385-cc46-1f2f-38f3d844649a}"
2025-02-23T18:43:19.031Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: pipelining replication: peer="{Voter server-06041430-2866-ffc1-bd86-715625c09f5e 06041430-2866-ffc1-bd86-715625c09f5e}"
2025-02-23T18:43:19.031Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: pipelining replication: peer="{Voter server-d09c3bf3-3ffc-2e3c-344c-efcc9116214d d09c3bf3-3ffc-2e3c-344c-efcc9116214d}"
2025-02-23T18:43:19.136Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:19.031158136 +0000 UTC m=+103.329418266 (1 nodes), 86.76µs from start of poll, 375.362597ms from cluster start. Timeout at 2025-02-23 18:43:19.136235227 +0000 UTC m=+103.434495838, 105.077572ms after stability
2025-02-23T18:43:19.136Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: aborting pipeline replication: peer="{Voter server-212e764c-acde-6bde-90ba-f83e08d58dcd 212e764c-acde-6bde-90ba-f83e08d58dcd}"
2025-02-23T18:43:19.136Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: aborting pipeline replication: peer="{Voter server-46bc41e7-e0b1-b6e6-7490-65ee7087c46b 46bc41e7-e0b1-b6e6-7490-65ee7087c46b}"
2025-02-23T18:43:19.136Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: aborting pipeline replication: peer="{Voter server-6580279d-1385-cc46-1f2f-38f3d844649a 6580279d-1385-cc46-1f2f-38f3d844649a}"
2025-02-23T18:43:19.136Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: aborting pipeline replication: peer="{Voter server-06041430-2866-ffc1-bd86-715625c09f5e 06041430-2866-ffc1-bd86-715625c09f5e}"
2025-02-23T18:43:19.136Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: aborting pipeline replication: peer="{Voter server-d09c3bf3-3ffc-2e3c-344c-efcc9116214d d09c3bf3-3ffc-2e3c-344c-efcc9116214d}"
2025-02-23T18:43:19.136Z [INFO]  server-ce0b2325-2791-90a9-e086-e32f5d68eb5f: aborting pipeline replication: peer="{Voter server-c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0 c6d1ee0c-c9d2-81b7-d483-334f36f1e5b0}"
--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.48s)
=== RUN   TestRaft_LeadershipTransferToInvalidID
2025-02-23T18:43:19.139Z [INFO]  server-a76ebf11-e24b-b3be-1949-01cb1b43f796: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a76ebf11-e24b-b3be-1949-01cb1b43f796 Address:a76ebf11-e24b-b3be-1949-01cb1b43f796} {Suffrage:Voter ID:server-f1856b7a-8277-d86a-3065-4583945f5c5b Address:f1856b7a-8277-d86a-3065-4583945f5c5b} {Suffrage:Voter ID:server-af2ce189-65b1-926c-3b63-31155df423c7 Address:af2ce189-65b1-926c-3b63-31155df423c7}]"
2025-02-23T18:43:19.139Z [INFO]  server-a76ebf11-e24b-b3be-1949-01cb1b43f796: entering follower state: follower="Node at a76ebf11-e24b-b3be-1949-01cb1b43f796 [Follower]" leader-address= leader-id=
2025-02-23T18:43:19.139Z [INFO]  server-f1856b7a-8277-d86a-3065-4583945f5c5b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a76ebf11-e24b-b3be-1949-01cb1b43f796 Address:a76ebf11-e24b-b3be-1949-01cb1b43f796} {Suffrage:Voter ID:server-f1856b7a-8277-d86a-3065-4583945f5c5b Address:f1856b7a-8277-d86a-3065-4583945f5c5b} {Suffrage:Voter ID:server-af2ce189-65b1-926c-3b63-31155df423c7 Address:af2ce189-65b1-926c-3b63-31155df423c7}]"
2025-02-23T18:43:19.139Z [INFO]  server-af2ce189-65b1-926c-3b63-31155df423c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a76ebf11-e24b-b3be-1949-01cb1b43f796 Address:a76ebf11-e24b-b3be-1949-01cb1b43f796} {Suffrage:Voter ID:server-f1856b7a-8277-d86a-3065-4583945f5c5b Address:f1856b7a-8277-d86a-3065-4583945f5c5b} {Suffrage:Voter ID:server-af2ce189-65b1-926c-3b63-31155df423c7 Address:af2ce189-65b1-926c-3b63-31155df423c7}]"
2025-02-23T18:43:19.139Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:19.139Z [INFO]  server-f1856b7a-8277-d86a-3065-4583945f5c5b: entering follower state: follower="Node at f1856b7a-8277-d86a-3065-4583945f5c5b [Follower]" leader-address= leader-id=
2025-02-23T18:43:19.139Z [INFO]  server-af2ce189-65b1-926c-3b63-31155df423c7: entering follower state: follower="Node at af2ce189-65b1-926c-3b63-31155df423c7 [Follower]" leader-address= leader-id=
2025-02-23T18:43:19.199Z [WARN]  server-f1856b7a-8277-d86a-3065-4583945f5c5b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:19.199Z [INFO]  server-f1856b7a-8277-d86a-3065-4583945f5c5b: entering candidate state: node="Node at f1856b7a-8277-d86a-3065-4583945f5c5b [Candidate]" term=2
2025-02-23T18:43:19.199Z [INFO]  server-f1856b7a-8277-d86a-3065-4583945f5c5b: election won: term=2 tally=2
2025-02-23T18:43:19.199Z [INFO]  server-f1856b7a-8277-d86a-3065-4583945f5c5b: entering leader state: leader="Node at f1856b7a-8277-d86a-3065-4583945f5c5b [Leader]"
2025-02-23T18:43:19.199Z [INFO]  server-f1856b7a-8277-d86a-3065-4583945f5c5b: added peer, starting replication: peer=server-a76ebf11-e24b-b3be-1949-01cb1b43f796
2025-02-23T18:43:19.199Z [INFO]  server-f1856b7a-8277-d86a-3065-4583945f5c5b: added peer, starting replication: peer=server-af2ce189-65b1-926c-3b63-31155df423c7
2025-02-23T18:43:19.199Z [INFO]  server-f1856b7a-8277-d86a-3065-4583945f5c5b: pipelining replication: peer="{Voter server-a76ebf11-e24b-b3be-1949-01cb1b43f796 a76ebf11-e24b-b3be-1949-01cb1b43f796}"
2025-02-23T18:43:19.200Z [INFO]  server-f1856b7a-8277-d86a-3065-4583945f5c5b: pipelining replication: peer="{Voter server-af2ce189-65b1-926c-3b63-31155df423c7 af2ce189-65b1-926c-3b63-31155df423c7}"
2025-02-23T18:43:19.305Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:19.199781354 +0000 UTC m=+103.498042189 (1 nodes), 60.034301ms from start of poll, 60.663188ms from cluster start. Timeout at 2025-02-23 18:43:19.304794591 +0000 UTC m=+103.603055394, 105.013205ms after stability
2025-02-23T18:43:19.305Z [INFO]  server-f1856b7a-8277-d86a-3065-4583945f5c5b: aborting pipeline replication: peer="{Voter server-af2ce189-65b1-926c-3b63-31155df423c7 af2ce189-65b1-926c-3b63-31155df423c7}"
2025-02-23T18:43:19.305Z [INFO]  server-f1856b7a-8277-d86a-3065-4583945f5c5b: aborting pipeline replication: peer="{Voter server-a76ebf11-e24b-b3be-1949-01cb1b43f796 a76ebf11-e24b-b3be-1949-01cb1b43f796}"
--- PASS: TestRaft_LeadershipTransferToInvalidID (0.17s)
=== RUN   TestRaft_LeadershipTransferToInvalidAddress
2025-02-23T18:43:19.307Z [INFO]  server-a99a60a6-50b8-e164-6777-3d3b41c61b50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a99a60a6-50b8-e164-6777-3d3b41c61b50 Address:a99a60a6-50b8-e164-6777-3d3b41c61b50} {Suffrage:Voter ID:server-b67afa67-6fe9-c8f4-be80-761e201cdd01 Address:b67afa67-6fe9-c8f4-be80-761e201cdd01} {Suffrage:Voter ID:server-2d3e90e5-7e21-cae0-0087-ce1fea496393 Address:2d3e90e5-7e21-cae0-0087-ce1fea496393}]"
2025-02-23T18:43:19.307Z [INFO]  server-b67afa67-6fe9-c8f4-be80-761e201cdd01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a99a60a6-50b8-e164-6777-3d3b41c61b50 Address:a99a60a6-50b8-e164-6777-3d3b41c61b50} {Suffrage:Voter ID:server-b67afa67-6fe9-c8f4-be80-761e201cdd01 Address:b67afa67-6fe9-c8f4-be80-761e201cdd01} {Suffrage:Voter ID:server-2d3e90e5-7e21-cae0-0087-ce1fea496393 Address:2d3e90e5-7e21-cae0-0087-ce1fea496393}]"
2025-02-23T18:43:19.307Z [INFO]  server-2d3e90e5-7e21-cae0-0087-ce1fea496393: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a99a60a6-50b8-e164-6777-3d3b41c61b50 Address:a99a60a6-50b8-e164-6777-3d3b41c61b50} {Suffrage:Voter ID:server-b67afa67-6fe9-c8f4-be80-761e201cdd01 Address:b67afa67-6fe9-c8f4-be80-761e201cdd01} {Suffrage:Voter ID:server-2d3e90e5-7e21-cae0-0087-ce1fea496393 Address:2d3e90e5-7e21-cae0-0087-ce1fea496393}]"
2025-02-23T18:43:19.307Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:19.307Z [INFO]  server-a99a60a6-50b8-e164-6777-3d3b41c61b50: entering follower state: follower="Node at a99a60a6-50b8-e164-6777-3d3b41c61b50 [Follower]" leader-address= leader-id=
2025-02-23T18:43:19.307Z [INFO]  server-b67afa67-6fe9-c8f4-be80-761e201cdd01: entering follower state: follower="Node at b67afa67-6fe9-c8f4-be80-761e201cdd01 [Follower]" leader-address= leader-id=
2025-02-23T18:43:19.307Z [INFO]  server-2d3e90e5-7e21-cae0-0087-ce1fea496393: entering follower state: follower="Node at 2d3e90e5-7e21-cae0-0087-ce1fea496393 [Follower]" leader-address= leader-id=
2025-02-23T18:43:19.367Z [WARN]  server-a99a60a6-50b8-e164-6777-3d3b41c61b50: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:19.367Z [INFO]  server-a99a60a6-50b8-e164-6777-3d3b41c61b50: entering candidate state: node="Node at a99a60a6-50b8-e164-6777-3d3b41c61b50 [Candidate]" term=2
2025-02-23T18:43:19.367Z [INFO]  server-a99a60a6-50b8-e164-6777-3d3b41c61b50: election won: term=2 tally=2
2025-02-23T18:43:19.367Z [INFO]  server-a99a60a6-50b8-e164-6777-3d3b41c61b50: entering leader state: leader="Node at a99a60a6-50b8-e164-6777-3d3b41c61b50 [Leader]"
2025-02-23T18:43:19.367Z [INFO]  server-a99a60a6-50b8-e164-6777-3d3b41c61b50: added peer, starting replication: peer=server-b67afa67-6fe9-c8f4-be80-761e201cdd01
2025-02-23T18:43:19.367Z [INFO]  server-a99a60a6-50b8-e164-6777-3d3b41c61b50: added peer, starting replication: peer=server-2d3e90e5-7e21-cae0-0087-ce1fea496393
2025-02-23T18:43:19.368Z [INFO]  server-a99a60a6-50b8-e164-6777-3d3b41c61b50: pipelining replication: peer="{Voter server-b67afa67-6fe9-c8f4-be80-761e201cdd01 b67afa67-6fe9-c8f4-be80-761e201cdd01}"
2025-02-23T18:43:19.368Z [INFO]  server-a99a60a6-50b8-e164-6777-3d3b41c61b50: pipelining replication: peer="{Voter server-2d3e90e5-7e21-cae0-0087-ce1fea496393 2d3e90e5-7e21-cae0-0087-ce1fea496393}"
2025-02-23T18:43:19.473Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:19.367949807 +0000 UTC m=+103.666210468 (2 nodes), 60.550184ms from start of poll, 61.024728ms from cluster start. Timeout at 2025-02-23 18:43:19.472957984 +0000 UTC m=+103.771218653, 105.008185ms after stability
2025-02-23T18:43:19.473Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:19.579Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:19.47314743 +0000 UTC m=+103.771408249 (1 nodes), 2.238µs from start of poll, 166.222509ms from cluster start. Timeout at 2025-02-23 18:43:19.57823142 +0000 UTC m=+103.876492052, 105.083803ms after stability
2025-02-23T18:43:19.579Z [INFO]  server-a99a60a6-50b8-e164-6777-3d3b41c61b50: aborting pipeline replication: peer="{Voter server-2d3e90e5-7e21-cae0-0087-ce1fea496393 2d3e90e5-7e21-cae0-0087-ce1fea496393}"
2025-02-23T18:43:19.579Z [INFO]  server-a99a60a6-50b8-e164-6777-3d3b41c61b50: aborting pipeline replication: peer="{Voter server-b67afa67-6fe9-c8f4-be80-761e201cdd01 b67afa67-6fe9-c8f4-be80-761e201cdd01}"
--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.27s)
=== RUN   TestRaft_LeadershipTransferToBehindServer
2025-02-23T18:43:19.581Z [INFO]  server-02686d67-3522-4b9f-6392-29ff55f5e6c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02686d67-3522-4b9f-6392-29ff55f5e6c6 Address:02686d67-3522-4b9f-6392-29ff55f5e6c6} {Suffrage:Voter ID:server-9606346c-5bcb-fdee-b272-dbac49dbe259 Address:9606346c-5bcb-fdee-b272-dbac49dbe259} {Suffrage:Voter ID:server-aea4fdc9-3d0f-f777-96ac-80d46a038afb Address:aea4fdc9-3d0f-f777-96ac-80d46a038afb}]"
2025-02-23T18:43:19.581Z [INFO]  server-9606346c-5bcb-fdee-b272-dbac49dbe259: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02686d67-3522-4b9f-6392-29ff55f5e6c6 Address:02686d67-3522-4b9f-6392-29ff55f5e6c6} {Suffrage:Voter ID:server-9606346c-5bcb-fdee-b272-dbac49dbe259 Address:9606346c-5bcb-fdee-b272-dbac49dbe259} {Suffrage:Voter ID:server-aea4fdc9-3d0f-f777-96ac-80d46a038afb Address:aea4fdc9-3d0f-f777-96ac-80d46a038afb}]"
2025-02-23T18:43:19.581Z [INFO]  server-aea4fdc9-3d0f-f777-96ac-80d46a038afb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02686d67-3522-4b9f-6392-29ff55f5e6c6 Address:02686d67-3522-4b9f-6392-29ff55f5e6c6} {Suffrage:Voter ID:server-9606346c-5bcb-fdee-b272-dbac49dbe259 Address:9606346c-5bcb-fdee-b272-dbac49dbe259} {Suffrage:Voter ID:server-aea4fdc9-3d0f-f777-96ac-80d46a038afb Address:aea4fdc9-3d0f-f777-96ac-80d46a038afb}]"
2025-02-23T18:43:19.581Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:19.581Z [INFO]  server-aea4fdc9-3d0f-f777-96ac-80d46a038afb: entering follower state: follower="Node at aea4fdc9-3d0f-f777-96ac-80d46a038afb [Follower]" leader-address= leader-id=
2025-02-23T18:43:19.581Z [INFO]  server-9606346c-5bcb-fdee-b272-dbac49dbe259: entering follower state: follower="Node at 9606346c-5bcb-fdee-b272-dbac49dbe259 [Follower]" leader-address= leader-id=
2025-02-23T18:43:19.581Z [INFO]  server-02686d67-3522-4b9f-6392-29ff55f5e6c6: entering follower state: follower="Node at 02686d67-3522-4b9f-6392-29ff55f5e6c6 [Follower]" leader-address= leader-id=
2025-02-23T18:43:19.662Z [WARN]  server-aea4fdc9-3d0f-f777-96ac-80d46a038afb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:19.662Z [INFO]  server-aea4fdc9-3d0f-f777-96ac-80d46a038afb: entering candidate state: node="Node at aea4fdc9-3d0f-f777-96ac-80d46a038afb [Candidate]" term=2
2025-02-23T18:43:19.662Z [INFO]  server-aea4fdc9-3d0f-f777-96ac-80d46a038afb: election won: term=2 tally=2
2025-02-23T18:43:19.662Z [INFO]  server-aea4fdc9-3d0f-f777-96ac-80d46a038afb: entering leader state: leader="Node at aea4fdc9-3d0f-f777-96ac-80d46a038afb [Leader]"
2025-02-23T18:43:19.662Z [INFO]  server-aea4fdc9-3d0f-f777-96ac-80d46a038afb: added peer, starting replication: peer=server-02686d67-3522-4b9f-6392-29ff55f5e6c6
2025-02-23T18:43:19.662Z [INFO]  server-aea4fdc9-3d0f-f777-96ac-80d46a038afb: added peer, starting replication: peer=server-9606346c-5bcb-fdee-b272-dbac49dbe259
2025-02-23T18:43:19.662Z [INFO]  server-aea4fdc9-3d0f-f777-96ac-80d46a038afb: pipelining replication: peer="{Voter server-02686d67-3522-4b9f-6392-29ff55f5e6c6 02686d67-3522-4b9f-6392-29ff55f5e6c6}"
2025-02-23T18:43:19.662Z [INFO]  server-aea4fdc9-3d0f-f777-96ac-80d46a038afb: pipelining replication: peer="{Voter server-9606346c-5bcb-fdee-b272-dbac49dbe259 9606346c-5bcb-fdee-b272-dbac49dbe259}"
2025-02-23T18:43:19.768Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:19.662798859 +0000 UTC m=+103.961059494 (1 nodes), 81.265362ms from start of poll, 81.741481ms from cluster start. Timeout at 2025-02-23 18:43:19.767818176 +0000 UTC m=+104.066079115, 105.019621ms after stability
2025-02-23T18:43:19.768Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:19.873Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:19.768582537 +0000 UTC m=+104.066843469 (2 nodes), 2.548µs from start of poll, 187.525456ms from cluster start. Timeout at 2025-02-23 18:43:19.873626542 +0000 UTC m=+104.171886672, 105.043203ms after stability
2025-02-23T18:43:20.014Z [INFO]  server-02686d67-3522-4b9f-6392-29ff55f5e6c6: entering candidate state: node="Node at 02686d67-3522-4b9f-6392-29ff55f5e6c6 [Candidate]" term=3
2025-02-23T18:43:20.015Z [INFO]  server-aea4fdc9-3d0f-f777-96ac-80d46a038afb: entering follower state: follower="Node at aea4fdc9-3d0f-f777-96ac-80d46a038afb [Follower]" leader-address= leader-id=
2025-02-23T18:43:20.015Z [INFO]  server-02686d67-3522-4b9f-6392-29ff55f5e6c6: election won: term=3 tally=2
2025-02-23T18:43:20.015Z [INFO]  server-02686d67-3522-4b9f-6392-29ff55f5e6c6: entering leader state: leader="Node at 02686d67-3522-4b9f-6392-29ff55f5e6c6 [Leader]"
2025-02-23T18:43:20.015Z [INFO]  server-02686d67-3522-4b9f-6392-29ff55f5e6c6: added peer, starting replication: peer=server-9606346c-5bcb-fdee-b272-dbac49dbe259
2025-02-23T18:43:20.015Z [INFO]  server-02686d67-3522-4b9f-6392-29ff55f5e6c6: added peer, starting replication: peer=server-aea4fdc9-3d0f-f777-96ac-80d46a038afb
2025-02-23T18:43:20.015Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:20.015Z [INFO]  server-aea4fdc9-3d0f-f777-96ac-80d46a038afb: aborting pipeline replication: peer="{Voter server-02686d67-3522-4b9f-6392-29ff55f5e6c6 02686d67-3522-4b9f-6392-29ff55f5e6c6}"
2025-02-23T18:43:20.015Z [INFO]  server-aea4fdc9-3d0f-f777-96ac-80d46a038afb: aborting pipeline replication: peer="{Voter server-9606346c-5bcb-fdee-b272-dbac49dbe259 9606346c-5bcb-fdee-b272-dbac49dbe259}"
2025-02-23T18:43:20.015Z [INFO]  server-02686d67-3522-4b9f-6392-29ff55f5e6c6: pipelining replication: peer="{Voter server-aea4fdc9-3d0f-f777-96ac-80d46a038afb aea4fdc9-3d0f-f777-96ac-80d46a038afb}"
2025-02-23T18:43:20.015Z [INFO]  server-02686d67-3522-4b9f-6392-29ff55f5e6c6: pipelining replication: peer="{Voter server-9606346c-5bcb-fdee-b272-dbac49dbe259 9606346c-5bcb-fdee-b272-dbac49dbe259}"
2025-02-23T18:43:20.120Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:20.015408076 +0000 UTC m=+104.313668206 (1 nodes), 52.025µs from start of poll, 434.350193ms from cluster start. Timeout at 2025-02-23 18:43:20.120408076 +0000 UTC m=+104.418668206, 105ms after stability
2025-02-23T18:43:20.120Z [INFO]  server-02686d67-3522-4b9f-6392-29ff55f5e6c6: aborting pipeline replication: peer="{Voter server-9606346c-5bcb-fdee-b272-dbac49dbe259 9606346c-5bcb-fdee-b272-dbac49dbe259}"
2025-02-23T18:43:20.120Z [INFO]  server-02686d67-3522-4b9f-6392-29ff55f5e6c6: aborting pipeline replication: peer="{Voter server-aea4fdc9-3d0f-f777-96ac-80d46a038afb aea4fdc9-3d0f-f777-96ac-80d46a038afb}"
--- PASS: TestRaft_LeadershipTransferToBehindServer (0.54s)
=== RUN   TestRaft_LeadershipTransferToItself
2025-02-23T18:43:20.122Z [INFO]  server-7afeff97-1b76-b33e-6ac8-1e286c469b8a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7afeff97-1b76-b33e-6ac8-1e286c469b8a Address:7afeff97-1b76-b33e-6ac8-1e286c469b8a} {Suffrage:Voter ID:server-f1d2dab2-11e7-371d-632f-0e174304f389 Address:f1d2dab2-11e7-371d-632f-0e174304f389} {Suffrage:Voter ID:server-f637fb8e-125c-4d2b-492c-e4c841254545 Address:f637fb8e-125c-4d2b-492c-e4c841254545}]"
2025-02-23T18:43:20.122Z [INFO]  server-f1d2dab2-11e7-371d-632f-0e174304f389: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7afeff97-1b76-b33e-6ac8-1e286c469b8a Address:7afeff97-1b76-b33e-6ac8-1e286c469b8a} {Suffrage:Voter ID:server-f1d2dab2-11e7-371d-632f-0e174304f389 Address:f1d2dab2-11e7-371d-632f-0e174304f389} {Suffrage:Voter ID:server-f637fb8e-125c-4d2b-492c-e4c841254545 Address:f637fb8e-125c-4d2b-492c-e4c841254545}]"
2025-02-23T18:43:20.122Z [INFO]  server-f637fb8e-125c-4d2b-492c-e4c841254545: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7afeff97-1b76-b33e-6ac8-1e286c469b8a Address:7afeff97-1b76-b33e-6ac8-1e286c469b8a} {Suffrage:Voter ID:server-f1d2dab2-11e7-371d-632f-0e174304f389 Address:f1d2dab2-11e7-371d-632f-0e174304f389} {Suffrage:Voter ID:server-f637fb8e-125c-4d2b-492c-e4c841254545 Address:f637fb8e-125c-4d2b-492c-e4c841254545}]"
2025-02-23T18:43:20.122Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:20.122Z [INFO]  server-7afeff97-1b76-b33e-6ac8-1e286c469b8a: entering follower state: follower="Node at 7afeff97-1b76-b33e-6ac8-1e286c469b8a [Follower]" leader-address= leader-id=
2025-02-23T18:43:20.122Z [INFO]  server-f1d2dab2-11e7-371d-632f-0e174304f389: entering follower state: follower="Node at f1d2dab2-11e7-371d-632f-0e174304f389 [Follower]" leader-address= leader-id=
2025-02-23T18:43:20.122Z [INFO]  server-f637fb8e-125c-4d2b-492c-e4c841254545: entering follower state: follower="Node at f637fb8e-125c-4d2b-492c-e4c841254545 [Follower]" leader-address= leader-id=
2025-02-23T18:43:20.188Z [WARN]  server-f1d2dab2-11e7-371d-632f-0e174304f389: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:20.188Z [INFO]  server-f1d2dab2-11e7-371d-632f-0e174304f389: entering candidate state: node="Node at f1d2dab2-11e7-371d-632f-0e174304f389 [Candidate]" term=2
2025-02-23T18:43:20.188Z [INFO]  server-f1d2dab2-11e7-371d-632f-0e174304f389: election won: term=2 tally=2
2025-02-23T18:43:20.188Z [INFO]  server-f1d2dab2-11e7-371d-632f-0e174304f389: entering leader state: leader="Node at f1d2dab2-11e7-371d-632f-0e174304f389 [Leader]"
2025-02-23T18:43:20.188Z [INFO]  server-f1d2dab2-11e7-371d-632f-0e174304f389: added peer, starting replication: peer=server-7afeff97-1b76-b33e-6ac8-1e286c469b8a
2025-02-23T18:43:20.188Z [INFO]  server-f1d2dab2-11e7-371d-632f-0e174304f389: added peer, starting replication: peer=server-f637fb8e-125c-4d2b-492c-e4c841254545
2025-02-23T18:43:20.189Z [INFO]  server-f1d2dab2-11e7-371d-632f-0e174304f389: pipelining replication: peer="{Voter server-f637fb8e-125c-4d2b-492c-e4c841254545 f637fb8e-125c-4d2b-492c-e4c841254545}"
2025-02-23T18:43:20.189Z [INFO]  server-f1d2dab2-11e7-371d-632f-0e174304f389: pipelining replication: peer="{Voter server-7afeff97-1b76-b33e-6ac8-1e286c469b8a 7afeff97-1b76-b33e-6ac8-1e286c469b8a}"
2025-02-23T18:43:20.295Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:20.189273522 +0000 UTC m=+104.487534152 (1 nodes), 66.712676ms from start of poll, 67.123477ms from cluster start. Timeout at 2025-02-23 18:43:20.294289376 +0000 UTC m=+104.592550005, 105.015853ms after stability
2025-02-23T18:43:20.295Z [INFO]  server-f1d2dab2-11e7-371d-632f-0e174304f389: cannot transfer leadership to itself
2025-02-23T18:43:20.295Z [INFO]  server-f1d2dab2-11e7-371d-632f-0e174304f389: aborting pipeline replication: peer="{Voter server-f637fb8e-125c-4d2b-492c-e4c841254545 f637fb8e-125c-4d2b-492c-e4c841254545}"
2025-02-23T18:43:20.295Z [INFO]  server-f1d2dab2-11e7-371d-632f-0e174304f389: aborting pipeline replication: peer="{Voter server-7afeff97-1b76-b33e-6ac8-1e286c469b8a 7afeff97-1b76-b33e-6ac8-1e286c469b8a}"
--- PASS: TestRaft_LeadershipTransferToItself (0.17s)
=== RUN   TestRaft_LeadershipTransferLeaderRejectsClientRequests
2025-02-23T18:43:20.297Z [INFO]  server-47191aea-f87b-908d-13d8-36fcbc7c7cd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47191aea-f87b-908d-13d8-36fcbc7c7cd1 Address:47191aea-f87b-908d-13d8-36fcbc7c7cd1} {Suffrage:Voter ID:server-c5e0d95c-7fc7-247d-ef76-974de46693c7 Address:c5e0d95c-7fc7-247d-ef76-974de46693c7} {Suffrage:Voter ID:server-b6bdba50-ec63-20da-2a3c-bd463159df1d Address:b6bdba50-ec63-20da-2a3c-bd463159df1d}]"
2025-02-23T18:43:20.297Z [INFO]  server-c5e0d95c-7fc7-247d-ef76-974de46693c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47191aea-f87b-908d-13d8-36fcbc7c7cd1 Address:47191aea-f87b-908d-13d8-36fcbc7c7cd1} {Suffrage:Voter ID:server-c5e0d95c-7fc7-247d-ef76-974de46693c7 Address:c5e0d95c-7fc7-247d-ef76-974de46693c7} {Suffrage:Voter ID:server-b6bdba50-ec63-20da-2a3c-bd463159df1d Address:b6bdba50-ec63-20da-2a3c-bd463159df1d}]"
2025-02-23T18:43:20.297Z [INFO]  server-47191aea-f87b-908d-13d8-36fcbc7c7cd1: entering follower state: follower="Node at 47191aea-f87b-908d-13d8-36fcbc7c7cd1 [Follower]" leader-address= leader-id=
2025-02-23T18:43:20.297Z [INFO]  server-c5e0d95c-7fc7-247d-ef76-974de46693c7: entering follower state: follower="Node at c5e0d95c-7fc7-247d-ef76-974de46693c7 [Follower]" leader-address= leader-id=
2025-02-23T18:43:20.298Z [INFO]  server-b6bdba50-ec63-20da-2a3c-bd463159df1d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47191aea-f87b-908d-13d8-36fcbc7c7cd1 Address:47191aea-f87b-908d-13d8-36fcbc7c7cd1} {Suffrage:Voter ID:server-c5e0d95c-7fc7-247d-ef76-974de46693c7 Address:c5e0d95c-7fc7-247d-ef76-974de46693c7} {Suffrage:Voter ID:server-b6bdba50-ec63-20da-2a3c-bd463159df1d Address:b6bdba50-ec63-20da-2a3c-bd463159df1d}]"
2025-02-23T18:43:20.298Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:20.298Z [INFO]  server-b6bdba50-ec63-20da-2a3c-bd463159df1d: entering follower state: follower="Node at b6bdba50-ec63-20da-2a3c-bd463159df1d [Follower]" leader-address= leader-id=
2025-02-23T18:43:20.352Z [WARN]  server-b6bdba50-ec63-20da-2a3c-bd463159df1d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:20.352Z [INFO]  server-b6bdba50-ec63-20da-2a3c-bd463159df1d: entering candidate state: node="Node at b6bdba50-ec63-20da-2a3c-bd463159df1d [Candidate]" term=2
2025-02-23T18:43:20.353Z [INFO]  server-b6bdba50-ec63-20da-2a3c-bd463159df1d: election won: term=2 tally=2
2025-02-23T18:43:20.353Z [INFO]  server-b6bdba50-ec63-20da-2a3c-bd463159df1d: entering leader state: leader="Node at b6bdba50-ec63-20da-2a3c-bd463159df1d [Leader]"
2025-02-23T18:43:20.353Z [INFO]  server-b6bdba50-ec63-20da-2a3c-bd463159df1d: added peer, starting replication: peer=server-47191aea-f87b-908d-13d8-36fcbc7c7cd1
2025-02-23T18:43:20.353Z [INFO]  server-b6bdba50-ec63-20da-2a3c-bd463159df1d: added peer, starting replication: peer=server-c5e0d95c-7fc7-247d-ef76-974de46693c7
2025-02-23T18:43:20.353Z [INFO]  server-b6bdba50-ec63-20da-2a3c-bd463159df1d: pipelining replication: peer="{Voter server-c5e0d95c-7fc7-247d-ef76-974de46693c7 c5e0d95c-7fc7-247d-ef76-974de46693c7}"
2025-02-23T18:43:20.353Z [INFO]  server-b6bdba50-ec63-20da-2a3c-bd463159df1d: pipelining replication: peer="{Voter server-47191aea-f87b-908d-13d8-36fcbc7c7cd1 47191aea-f87b-908d-13d8-36fcbc7c7cd1}"
2025-02-23T18:43:20.458Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:20.353053405 +0000 UTC m=+104.651313784 (1 nodes), 54.960183ms from start of poll, 55.90104ms from cluster start. Timeout at 2025-02-23 18:43:20.45806062 +0000 UTC m=+104.756321517, 105.007733ms after stability
    raft_test.go:2457: waiting on future 0
    raft_test.go:2457: waiting on future 1
    raft_test.go:2457: waiting on future 2
    raft_test.go:2457: waiting on future 3
    raft_test.go:2457: waiting on future 4
    raft_test.go:2457: waiting on future 5
    raft_test.go:2457: waiting on future 6
2025-02-23T18:43:20.458Z [INFO]  server-b6bdba50-ec63-20da-2a3c-bd463159df1d: aborting pipeline replication: peer="{Voter server-c5e0d95c-7fc7-247d-ef76-974de46693c7 c5e0d95c-7fc7-247d-ef76-974de46693c7}"
2025-02-23T18:43:20.458Z [INFO]  server-b6bdba50-ec63-20da-2a3c-bd463159df1d: aborting pipeline replication: peer="{Voter server-47191aea-f87b-908d-13d8-36fcbc7c7cd1 47191aea-f87b-908d-13d8-36fcbc7c7cd1}"
--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.16s)
=== RUN   TestRaft_LeadershipTransferLeaderReplicationTimeout
2025-02-23T18:43:20.460Z [INFO]  server-093483f7-e429-9ee4-bc63-f9445789cb12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-093483f7-e429-9ee4-bc63-f9445789cb12 Address:093483f7-e429-9ee4-bc63-f9445789cb12} {Suffrage:Voter ID:server-d16eefee-774d-1704-fac4-c6bd9bca555f Address:d16eefee-774d-1704-fac4-c6bd9bca555f} {Suffrage:Voter ID:server-7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a Address:7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a}]"
2025-02-23T18:43:20.460Z [INFO]  server-d16eefee-774d-1704-fac4-c6bd9bca555f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-093483f7-e429-9ee4-bc63-f9445789cb12 Address:093483f7-e429-9ee4-bc63-f9445789cb12} {Suffrage:Voter ID:server-d16eefee-774d-1704-fac4-c6bd9bca555f Address:d16eefee-774d-1704-fac4-c6bd9bca555f} {Suffrage:Voter ID:server-7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a Address:7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a}]"
2025-02-23T18:43:20.460Z [INFO]  server-7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-093483f7-e429-9ee4-bc63-f9445789cb12 Address:093483f7-e429-9ee4-bc63-f9445789cb12} {Suffrage:Voter ID:server-d16eefee-774d-1704-fac4-c6bd9bca555f Address:d16eefee-774d-1704-fac4-c6bd9bca555f} {Suffrage:Voter ID:server-7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a Address:7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a}]"
2025-02-23T18:43:20.460Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:20.460Z [INFO]  server-093483f7-e429-9ee4-bc63-f9445789cb12: entering follower state: follower="Node at 093483f7-e429-9ee4-bc63-f9445789cb12 [Follower]" leader-address= leader-id=
2025-02-23T18:43:20.460Z [INFO]  server-d16eefee-774d-1704-fac4-c6bd9bca555f: entering follower state: follower="Node at d16eefee-774d-1704-fac4-c6bd9bca555f [Follower]" leader-address= leader-id=
2025-02-23T18:43:20.460Z [INFO]  server-7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a: entering follower state: follower="Node at 7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a [Follower]" leader-address= leader-id=
2025-02-23T18:43:20.532Z [WARN]  server-7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:20.532Z [INFO]  server-7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a: entering candidate state: node="Node at 7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a [Candidate]" term=2
2025-02-23T18:43:20.532Z [WARN]  server-d16eefee-774d-1704-fac4-c6bd9bca555f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:20.532Z [INFO]  server-d16eefee-774d-1704-fac4-c6bd9bca555f: entering candidate state: node="Node at d16eefee-774d-1704-fac4-c6bd9bca555f [Candidate]" term=2
2025-02-23T18:43:20.533Z [INFO]  server-7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a: duplicate requestVote for same term: term=2
2025-02-23T18:43:20.533Z [INFO]  server-7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a: election won: term=2 tally=2
2025-02-23T18:43:20.533Z [INFO]  server-7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a: entering leader state: leader="Node at 7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a [Leader]"
2025-02-23T18:43:20.533Z [INFO]  server-7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a: added peer, starting replication: peer=server-093483f7-e429-9ee4-bc63-f9445789cb12
2025-02-23T18:43:20.533Z [INFO]  server-7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a: added peer, starting replication: peer=server-d16eefee-774d-1704-fac4-c6bd9bca555f
2025-02-23T18:43:20.533Z [INFO]  server-d16eefee-774d-1704-fac4-c6bd9bca555f: duplicate requestVote for same term: term=2
2025-02-23T18:43:20.533Z [INFO]  server-d16eefee-774d-1704-fac4-c6bd9bca555f: entering follower state: follower="Node at d16eefee-774d-1704-fac4-c6bd9bca555f [Follower]" leader-address=7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a leader-id=server-7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a
2025-02-23T18:43:20.533Z [INFO]  server-093483f7-e429-9ee4-bc63-f9445789cb12: duplicate requestVote for same term: term=2
2025-02-23T18:43:20.533Z [INFO]  server-7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a: pipelining replication: peer="{Voter server-d16eefee-774d-1704-fac4-c6bd9bca555f d16eefee-774d-1704-fac4-c6bd9bca555f}"
2025-02-23T18:43:20.533Z [INFO]  server-7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a: pipelining replication: peer="{Voter server-093483f7-e429-9ee4-bc63-f9445789cb12 093483f7-e429-9ee4-bc63-f9445789cb12}"
2025-02-23T18:43:20.638Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:20.533391628 +0000 UTC m=+104.831652685 (1 nodes), 72.79215ms from start of poll, 73.210503ms from cluster start. Timeout at 2025-02-23 18:43:20.63839472 +0000 UTC m=+104.936655332, 105.002647ms after stability
2025-02-23T18:43:20.638Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:20.743Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:20.638508674 +0000 UTC m=+104.936769322 (2 nodes), 2.009µs from start of poll, 178.32714ms from cluster start. Timeout at 2025-02-23 18:43:20.743558336 +0000 UTC m=+105.041818977, 105.049655ms after stability
2025-02-23T18:43:22.093Z [INFO]  server-7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a: aborting pipeline replication: peer="{Voter server-093483f7-e429-9ee4-bc63-f9445789cb12 093483f7-e429-9ee4-bc63-f9445789cb12}"
2025-02-23T18:43:22.093Z [INFO]  server-7b9098f8-5bd2-f6f6-0d13-a8a98deebf7a: aborting pipeline replication: peer="{Voter server-d16eefee-774d-1704-fac4-c6bd9bca555f d16eefee-774d-1704-fac4-c6bd9bca555f}"
--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (1.63s)
=== RUN   TestRaft_LeadershipTransferIgnoresNonvoters
2025-02-23T18:43:22.095Z [INFO]  server-341ec0f6-4799-a0c9-0168-f4015c827d37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-341ec0f6-4799-a0c9-0168-f4015c827d37 Address:341ec0f6-4799-a0c9-0168-f4015c827d37} {Suffrage:Voter ID:server-ec9413c0-1498-c5b3-ab94-f02ca0491b91 Address:ec9413c0-1498-c5b3-ab94-f02ca0491b91}]"
2025-02-23T18:43:22.095Z [INFO]  server-341ec0f6-4799-a0c9-0168-f4015c827d37: entering follower state: follower="Node at 341ec0f6-4799-a0c9-0168-f4015c827d37 [Follower]" leader-address= leader-id=
2025-02-23T18:43:22.095Z [INFO]  server-ec9413c0-1498-c5b3-ab94-f02ca0491b91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-341ec0f6-4799-a0c9-0168-f4015c827d37 Address:341ec0f6-4799-a0c9-0168-f4015c827d37} {Suffrage:Voter ID:server-ec9413c0-1498-c5b3-ab94-f02ca0491b91 Address:ec9413c0-1498-c5b3-ab94-f02ca0491b91}]"
2025-02-23T18:43:22.095Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:22.095Z [INFO]  server-ec9413c0-1498-c5b3-ab94-f02ca0491b91: entering follower state: follower="Node at ec9413c0-1498-c5b3-ab94-f02ca0491b91 [Follower]" leader-address= leader-id=
2025-02-23T18:43:22.150Z [WARN]  server-341ec0f6-4799-a0c9-0168-f4015c827d37: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:22.150Z [INFO]  server-341ec0f6-4799-a0c9-0168-f4015c827d37: entering candidate state: node="Node at 341ec0f6-4799-a0c9-0168-f4015c827d37 [Candidate]" term=2
2025-02-23T18:43:22.150Z [INFO]  server-341ec0f6-4799-a0c9-0168-f4015c827d37: election won: term=2 tally=2
2025-02-23T18:43:22.150Z [INFO]  server-341ec0f6-4799-a0c9-0168-f4015c827d37: entering leader state: leader="Node at 341ec0f6-4799-a0c9-0168-f4015c827d37 [Leader]"
2025-02-23T18:43:22.150Z [INFO]  server-341ec0f6-4799-a0c9-0168-f4015c827d37: added peer, starting replication: peer=server-ec9413c0-1498-c5b3-ab94-f02ca0491b91
2025-02-23T18:43:22.150Z [INFO]  server-341ec0f6-4799-a0c9-0168-f4015c827d37: pipelining replication: peer="{Voter server-ec9413c0-1498-c5b3-ab94-f02ca0491b91 ec9413c0-1498-c5b3-ab94-f02ca0491b91}"
2025-02-23T18:43:22.255Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:22.150430738 +0000 UTC m=+106.448692593 (1 nodes), 55.134168ms from start of poll, 55.47244ms from cluster start. Timeout at 2025-02-23 18:43:22.255436079 +0000 UTC m=+106.553696708, 105.004115ms after stability
2025-02-23T18:43:22.255Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:22.361Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:22.255621787 +0000 UTC m=+106.553881917 (1 nodes), 1.544µs from start of poll, 160.661764ms from cluster start. Timeout at 2025-02-23 18:43:22.360736333 +0000 UTC m=+106.658997102, 105.115185ms after stability
2025-02-23T18:43:22.361Z [INFO]  server-341ec0f6-4799-a0c9-0168-f4015c827d37: updating configuration: command=DemoteVoter server-id=server-ec9413c0-1498-c5b3-ab94-f02ca0491b91 server-addr= servers="[{Suffrage:Voter ID:server-341ec0f6-4799-a0c9-0168-f4015c827d37 Address:341ec0f6-4799-a0c9-0168-f4015c827d37} {Suffrage:Nonvoter ID:server-ec9413c0-1498-c5b3-ab94-f02ca0491b91 Address:ec9413c0-1498-c5b3-ab94-f02ca0491b91}]"
2025-02-23T18:43:22.362Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:22.467Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:22.36227072 +0000 UTC m=+106.660531624 (1 nodes), 2.419µs from start of poll, 267.311471ms from cluster start. Timeout at 2025-02-23 18:43:22.467376074 +0000 UTC m=+106.765636897, 105.105273ms after stability
2025-02-23T18:43:22.468Z [INFO]  server-341ec0f6-4799-a0c9-0168-f4015c827d37: aborting pipeline replication: peer="{Voter server-ec9413c0-1498-c5b3-ab94-f02ca0491b91 ec9413c0-1498-c5b3-ab94-f02ca0491b91}"
--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.37s)
=== RUN   TestRaft_LeadershipTransferStopRightAway
--- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s)
=== RUN   TestRaft_GetConfigurationNoBootstrap
2025-02-23T18:43:22.469Z [INFO]  server-4c1d5449-dcd0-652d-e218-68426f8d6b69: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c1d5449-dcd0-652d-e218-68426f8d6b69 Address:4c1d5449-dcd0-652d-e218-68426f8d6b69} {Suffrage:Voter ID:server-2b860e23-a096-7220-5f2e-8bafc51d30ed Address:2b860e23-a096-7220-5f2e-8bafc51d30ed}]"
2025-02-23T18:43:22.469Z [INFO]  server-4c1d5449-dcd0-652d-e218-68426f8d6b69: entering follower state: follower="Node at 4c1d5449-dcd0-652d-e218-68426f8d6b69 [Follower]" leader-address= leader-id=
2025-02-23T18:43:22.469Z [INFO]  server-2b860e23-a096-7220-5f2e-8bafc51d30ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c1d5449-dcd0-652d-e218-68426f8d6b69 Address:4c1d5449-dcd0-652d-e218-68426f8d6b69} {Suffrage:Voter ID:server-2b860e23-a096-7220-5f2e-8bafc51d30ed Address:2b860e23-a096-7220-5f2e-8bafc51d30ed}]"
2025-02-23T18:43:22.469Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:22.469Z [INFO]  server-2b860e23-a096-7220-5f2e-8bafc51d30ed: entering follower state: follower="Node at 2b860e23-a096-7220-5f2e-8bafc51d30ed [Follower]" leader-address= leader-id=
2025-02-23T18:43:22.540Z [WARN]  server-2b860e23-a096-7220-5f2e-8bafc51d30ed: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:22.540Z [INFO]  server-2b860e23-a096-7220-5f2e-8bafc51d30ed: entering candidate state: node="Node at 2b860e23-a096-7220-5f2e-8bafc51d30ed [Candidate]" term=2
2025-02-23T18:43:22.540Z [INFO]  server-2b860e23-a096-7220-5f2e-8bafc51d30ed: election won: term=2 tally=2
2025-02-23T18:43:22.540Z [INFO]  server-2b860e23-a096-7220-5f2e-8bafc51d30ed: entering leader state: leader="Node at 2b860e23-a096-7220-5f2e-8bafc51d30ed [Leader]"
2025-02-23T18:43:22.540Z [INFO]  server-2b860e23-a096-7220-5f2e-8bafc51d30ed: added peer, starting replication: peer=server-4c1d5449-dcd0-652d-e218-68426f8d6b69
2025-02-23T18:43:22.540Z [INFO]  server-2b860e23-a096-7220-5f2e-8bafc51d30ed: pipelining replication: peer="{Voter server-4c1d5449-dcd0-652d-e218-68426f8d6b69 4c1d5449-dcd0-652d-e218-68426f8d6b69}"
2025-02-23T18:43:22.646Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:22.540801362 +0000 UTC m=+106.839062618 (1 nodes), 71.152343ms from start of poll, 71.530973ms from cluster start. Timeout at 2025-02-23 18:43:22.645805571 +0000 UTC m=+106.944066196, 105.003578ms after stability
2025-02-23T18:43:22.646Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:22.752Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:22.646317811 +0000 UTC m=+106.944578599 (1 nodes), 1.888µs from start of poll, 177.046954ms from cluster start. Timeout at 2025-02-23 18:43:22.751366519 +0000 UTC m=+107.049627246, 105.048647ms after stability
2025-02-23T18:43:22.767Z [INFO]  server-2b860e23-a096-7220-5f2e-8bafc51d30ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c1d5449-dcd0-652d-e218-68426f8d6b69 Address:4c1d5449-dcd0-652d-e218-68426f8d6b69} {Suffrage:Voter ID:server-2b860e23-a096-7220-5f2e-8bafc51d30ed Address:2b860e23-a096-7220-5f2e-8bafc51d30ed}]"
2025-02-23T18:43:22.767Z [INFO]  server-2b860e23-a096-7220-5f2e-8bafc51d30ed: aborting pipeline replication: peer="{Voter server-4c1d5449-dcd0-652d-e218-68426f8d6b69 4c1d5449-dcd0-652d-e218-68426f8d6b69}"
--- PASS: TestRaft_GetConfigurationNoBootstrap (0.30s)
=== RUN   TestRaft_LogStoreIsMonotonic
2025-02-23T18:43:22.768Z [INFO]  server-3cb65d38-fdd5-3f9f-4381-72c17ef21e17: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3cb65d38-fdd5-3f9f-4381-72c17ef21e17 Address:3cb65d38-fdd5-3f9f-4381-72c17ef21e17}]"
2025-02-23T18:43:22.768Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:22.768Z [INFO]  server-3cb65d38-fdd5-3f9f-4381-72c17ef21e17: entering follower state: follower="Node at 3cb65d38-fdd5-3f9f-4381-72c17ef21e17 [Follower]" leader-address= leader-id=
2025-02-23T18:43:22.865Z [WARN]  server-3cb65d38-fdd5-3f9f-4381-72c17ef21e17: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:22.865Z [INFO]  server-3cb65d38-fdd5-3f9f-4381-72c17ef21e17: entering candidate state: node="Node at 3cb65d38-fdd5-3f9f-4381-72c17ef21e17 [Candidate]" term=2
2025-02-23T18:43:22.866Z [INFO]  server-3cb65d38-fdd5-3f9f-4381-72c17ef21e17: election won: term=2 tally=1
2025-02-23T18:43:22.866Z [INFO]  server-3cb65d38-fdd5-3f9f-4381-72c17ef21e17: entering leader state: leader="Node at 3cb65d38-fdd5-3f9f-4381-72c17ef21e17 [Leader]"
2025-02-23T18:43:22.971Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:22.86618388 +0000 UTC m=+107.164445150 (1 nodes), 97.250369ms from start of poll, 97.426809ms from cluster start. Timeout at 2025-02-23 18:43:22.971197956 +0000 UTC m=+107.269458086, 105.012936ms after stability
--- PASS: TestRaft_LogStoreIsMonotonic (0.20s)
=== RUN   TestRaft_CacheLogWithStoreError
2025-02-23T18:43:22.972Z [INFO]  server-caf74f25-d2d2-b1eb-4487-3fc624974e89: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-caf74f25-d2d2-b1eb-4487-3fc624974e89 Address:caf74f25-d2d2-b1eb-4487-3fc624974e89} {Suffrage:Voter ID:server-926bce57-52e3-9c15-9126-128e3136f728 Address:926bce57-52e3-9c15-9126-128e3136f728}]"
2025-02-23T18:43:22.972Z [INFO]  server-caf74f25-d2d2-b1eb-4487-3fc624974e89: entering follower state: follower="Node at caf74f25-d2d2-b1eb-4487-3fc624974e89 [Follower]" leader-address= leader-id=
2025-02-23T18:43:22.972Z [INFO]  server-926bce57-52e3-9c15-9126-128e3136f728: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-caf74f25-d2d2-b1eb-4487-3fc624974e89 Address:caf74f25-d2d2-b1eb-4487-3fc624974e89} {Suffrage:Voter ID:server-926bce57-52e3-9c15-9126-128e3136f728 Address:926bce57-52e3-9c15-9126-128e3136f728}]"
2025-02-23T18:43:22.972Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:22.972Z [INFO]  server-926bce57-52e3-9c15-9126-128e3136f728: entering follower state: follower="Node at 926bce57-52e3-9c15-9126-128e3136f728 [Follower]" leader-address= leader-id=
2025-02-23T18:43:23.028Z [WARN]  server-926bce57-52e3-9c15-9126-128e3136f728: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:23.028Z [INFO]  server-926bce57-52e3-9c15-9126-128e3136f728: entering candidate state: node="Node at 926bce57-52e3-9c15-9126-128e3136f728 [Candidate]" term=2
2025-02-23T18:43:23.028Z [INFO]  server-926bce57-52e3-9c15-9126-128e3136f728: election won: term=2 tally=2
2025-02-23T18:43:23.028Z [INFO]  server-926bce57-52e3-9c15-9126-128e3136f728: entering leader state: leader="Node at 926bce57-52e3-9c15-9126-128e3136f728 [Leader]"
2025-02-23T18:43:23.029Z [INFO]  server-926bce57-52e3-9c15-9126-128e3136f728: added peer, starting replication: peer=server-caf74f25-d2d2-b1eb-4487-3fc624974e89
2025-02-23T18:43:23.029Z [INFO]  server-926bce57-52e3-9c15-9126-128e3136f728: pipelining replication: peer="{Voter server-caf74f25-d2d2-b1eb-4487-3fc624974e89 caf74f25-d2d2-b1eb-4487-3fc624974e89}"
2025-02-23T18:43:23.134Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:23.029064657 +0000 UTC m=+107.327325535 (1 nodes), 56.284608ms from start of poll, 56.628163ms from cluster start. Timeout at 2025-02-23 18:43:23.134069711 +0000 UTC m=+107.432330577, 105.005042ms after stability
2025-02-23T18:43:23.134Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:23.240Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:23.134421758 +0000 UTC m=+107.432682584 (1 nodes), 2.008µs from start of poll, 161.985212ms from cluster start. Timeout at 2025-02-23 18:43:23.23946125 +0000 UTC m=+107.537721901, 105.039317ms after stability
2025-02-23T18:43:23.244Z [ERROR] server-caf74f25-d2d2-b1eb-4487-3fc624974e89: failed to append to logs: error="unable to store logs within log store, err: "some error""
2025-02-23T18:43:23.245Z [INFO]  server-926bce57-52e3-9c15-9126-128e3136f728: aborting pipeline replication: peer="{Voter server-caf74f25-d2d2-b1eb-4487-3fc624974e89 caf74f25-d2d2-b1eb-4487-3fc624974e89}"
2025-02-23T18:43:23.245Z [WARN]  server-caf74f25-d2d2-b1eb-4487-3fc624974e89: failed to get previous log: previous-index=8 last-index=7 error="log not found"
2025-02-23T18:43:23.255Z [INFO]  server-926bce57-52e3-9c15-9126-128e3136f728: pipelining replication: peer="{Voter server-caf74f25-d2d2-b1eb-4487-3fc624974e89 caf74f25-d2d2-b1eb-4487-3fc624974e89}"
2025-02-23T18:43:23.255Z [INFO]  server-caf74f25-d2d2-b1eb-4487-3fc624974e89: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-caf74f25-d2d2-b1eb-4487-3fc624974e89 Address:caf74f25-d2d2-b1eb-4487-3fc624974e89} {Suffrage:Voter ID:server-926bce57-52e3-9c15-9126-128e3136f728 Address:926bce57-52e3-9c15-9126-128e3136f728}]"
2025-02-23T18:43:23.255Z [INFO]  server-926bce57-52e3-9c15-9126-128e3136f728: aborting pipeline replication: peer="{Voter server-caf74f25-d2d2-b1eb-4487-3fc624974e89 caf74f25-d2d2-b1eb-4487-3fc624974e89}"
--- PASS: TestRaft_CacheLogWithStoreError (0.29s)
=== RUN   TestRaft_ReloadConfig
2025-02-23T18:43:23.258Z [INFO]  server-44a601b0-76a0-09f3-8215-7fab12613c61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44a601b0-76a0-09f3-8215-7fab12613c61 Address:44a601b0-76a0-09f3-8215-7fab12613c61}]"
--- PASS: TestRaft_ReloadConfig (0.00s)
=== RUN   TestRaft_ReloadConfigValidates
2025-02-23T18:43:23.258Z [INFO]  server-2de24c68-629f-e3a0-45b5-303ef42656b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2de24c68-629f-e3a0-45b5-303ef42656b0 Address:2de24c68-629f-e3a0-45b5-303ef42656b0}]"
2025-02-23T18:43:23.258Z [INFO]  server-2de24c68-629f-e3a0-45b5-303ef42656b0: entering follower state: follower="Node at 2de24c68-629f-e3a0-45b5-303ef42656b0 [Shutdown]" leader-address= leader-id=
--- PASS: TestRaft_ReloadConfigValidates (0.00s)
=== RUN   TestRaft_InstallSnapshot_InvalidPeers
2025-02-23T18:43:23.259Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)"
--- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s)
=== RUN   TestRaft_VoteNotGranted_WhenNodeNotInCluster
2025-02-23T18:43:23.260Z [INFO]  server-8659acf9-ead3-5153-4eeb-730651c1b628: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8659acf9-ead3-5153-4eeb-730651c1b628 Address:8659acf9-ead3-5153-4eeb-730651c1b628} {Suffrage:Voter ID:server-440ffb48-e3ea-e555-765e-e2cd8f114ed2 Address:440ffb48-e3ea-e555-765e-e2cd8f114ed2} {Suffrage:Voter ID:server-6d6d2902-c046-0242-d2e0-f94990ca71a8 Address:6d6d2902-c046-0242-d2e0-f94990ca71a8}]"
2025-02-23T18:43:23.260Z [INFO]  server-8659acf9-ead3-5153-4eeb-730651c1b628: entering follower state: follower="Node at 8659acf9-ead3-5153-4eeb-730651c1b628 [Follower]" leader-address= leader-id=
2025-02-23T18:43:23.260Z [INFO]  server-440ffb48-e3ea-e555-765e-e2cd8f114ed2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8659acf9-ead3-5153-4eeb-730651c1b628 Address:8659acf9-ead3-5153-4eeb-730651c1b628} {Suffrage:Voter ID:server-440ffb48-e3ea-e555-765e-e2cd8f114ed2 Address:440ffb48-e3ea-e555-765e-e2cd8f114ed2} {Suffrage:Voter ID:server-6d6d2902-c046-0242-d2e0-f94990ca71a8 Address:6d6d2902-c046-0242-d2e0-f94990ca71a8}]"
2025-02-23T18:43:23.260Z [INFO]  server-440ffb48-e3ea-e555-765e-e2cd8f114ed2: entering follower state: follower="Node at 440ffb48-e3ea-e555-765e-e2cd8f114ed2 [Follower]" leader-address= leader-id=
2025-02-23T18:43:23.260Z [INFO]  server-6d6d2902-c046-0242-d2e0-f94990ca71a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8659acf9-ead3-5153-4eeb-730651c1b628 Address:8659acf9-ead3-5153-4eeb-730651c1b628} {Suffrage:Voter ID:server-440ffb48-e3ea-e555-765e-e2cd8f114ed2 Address:440ffb48-e3ea-e555-765e-e2cd8f114ed2} {Suffrage:Voter ID:server-6d6d2902-c046-0242-d2e0-f94990ca71a8 Address:6d6d2902-c046-0242-d2e0-f94990ca71a8}]"
2025-02-23T18:43:23.260Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:23.260Z [INFO]  server-6d6d2902-c046-0242-d2e0-f94990ca71a8: entering follower state: follower="Node at 6d6d2902-c046-0242-d2e0-f94990ca71a8 [Follower]" leader-address= leader-id=
2025-02-23T18:43:23.324Z [WARN]  server-6d6d2902-c046-0242-d2e0-f94990ca71a8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:23.324Z [INFO]  server-6d6d2902-c046-0242-d2e0-f94990ca71a8: entering candidate state: node="Node at 6d6d2902-c046-0242-d2e0-f94990ca71a8 [Candidate]" term=2
2025-02-23T18:43:23.325Z [INFO]  server-6d6d2902-c046-0242-d2e0-f94990ca71a8: election won: term=2 tally=2
2025-02-23T18:43:23.325Z [INFO]  server-6d6d2902-c046-0242-d2e0-f94990ca71a8: entering leader state: leader="Node at 6d6d2902-c046-0242-d2e0-f94990ca71a8 [Leader]"
2025-02-23T18:43:23.325Z [INFO]  server-6d6d2902-c046-0242-d2e0-f94990ca71a8: added peer, starting replication: peer=server-8659acf9-ead3-5153-4eeb-730651c1b628
2025-02-23T18:43:23.325Z [INFO]  server-6d6d2902-c046-0242-d2e0-f94990ca71a8: added peer, starting replication: peer=server-440ffb48-e3ea-e555-765e-e2cd8f114ed2
2025-02-23T18:43:23.325Z [INFO]  server-6d6d2902-c046-0242-d2e0-f94990ca71a8: pipelining replication: peer="{Voter server-440ffb48-e3ea-e555-765e-e2cd8f114ed2 440ffb48-e3ea-e555-765e-e2cd8f114ed2}"
2025-02-23T18:43:23.325Z [INFO]  server-6d6d2902-c046-0242-d2e0-f94990ca71a8: pipelining replication: peer="{Voter server-8659acf9-ead3-5153-4eeb-730651c1b628 8659acf9-ead3-5153-4eeb-730651c1b628}"
2025-02-23T18:43:23.431Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:23.325051055 +0000 UTC m=+107.623311834 (1 nodes), 64.400929ms from start of poll, 64.82437ms from cluster start. Timeout at 2025-02-23 18:43:23.430054096 +0000 UTC m=+107.728314732, 105.002898ms after stability
2025-02-23T18:43:23.437Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:23.542Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:23.43724823 +0000 UTC m=+107.735509433 (2 nodes), 2.818µs from start of poll, 177.021969ms from cluster start. Timeout at 2025-02-23 18:43:23.542287495 +0000 UTC m=+107.840548257, 105.038824ms after stability
2025-02-23T18:43:23.542Z [INFO]  server-6d6d2902-c046-0242-d2e0-f94990ca71a8: updating configuration: command=RemoveServer server-id=server-8659acf9-ead3-5153-4eeb-730651c1b628 server-addr= servers="[{Suffrage:Voter ID:server-440ffb48-e3ea-e555-765e-e2cd8f114ed2 Address:440ffb48-e3ea-e555-765e-e2cd8f114ed2} {Suffrage:Voter ID:server-6d6d2902-c046-0242-d2e0-f94990ca71a8 Address:6d6d2902-c046-0242-d2e0-f94990ca71a8}]"
2025-02-23T18:43:23.542Z [INFO]  server-6d6d2902-c046-0242-d2e0-f94990ca71a8: removed peer, stopping replication: peer=server-8659acf9-ead3-5153-4eeb-730651c1b628 last-index=3
2025-02-23T18:43:23.542Z [INFO]  server-6d6d2902-c046-0242-d2e0-f94990ca71a8: aborting pipeline replication: peer="{Voter server-8659acf9-ead3-5153-4eeb-730651c1b628 8659acf9-ead3-5153-4eeb-730651c1b628}"
2025-02-23T18:43:23.647Z [WARN]  server-8659acf9-ead3-5153-4eeb-730651c1b628: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-02-23T18:43:23.650Z [ERROR] server-6d6d2902-c046-0242-d2e0-f94990ca71a8: failed to heartbeat to: peer=440ffb48-e3ea-e555-765e-e2cd8f114ed2 backoff time=10ms error="failed to connect to peer: 440ffb48-e3ea-e555-765e-e2cd8f114ed2"
2025-02-23T18:43:23.656Z [ERROR] server-6d6d2902-c046-0242-d2e0-f94990ca71a8: failed to pipeline appendEntries: peer="{Voter server-440ffb48-e3ea-e555-765e-e2cd8f114ed2 440ffb48-e3ea-e555-765e-e2cd8f114ed2}" error="append pipeline closed"
2025-02-23T18:43:23.656Z [INFO]  server-6d6d2902-c046-0242-d2e0-f94990ca71a8: aborting pipeline replication: peer="{Voter server-440ffb48-e3ea-e555-765e-e2cd8f114ed2 440ffb48-e3ea-e555-765e-e2cd8f114ed2}"
2025-02-23T18:43:23.663Z [ERROR] server-6d6d2902-c046-0242-d2e0-f94990ca71a8: failed to appendEntries to: peer="{Voter server-440ffb48-e3ea-e555-765e-e2cd8f114ed2 440ffb48-e3ea-e555-765e-e2cd8f114ed2}" error="failed to connect to peer: 440ffb48-e3ea-e555-765e-e2cd8f114ed2"
2025-02-23T18:43:23.666Z [ERROR] server-6d6d2902-c046-0242-d2e0-f94990ca71a8: failed to heartbeat to: peer=440ffb48-e3ea-e555-765e-e2cd8f114ed2 backoff time=10ms error="failed to connect to peer: 440ffb48-e3ea-e555-765e-e2cd8f114ed2"
2025-02-23T18:43:23.682Z [ERROR] server-6d6d2902-c046-0242-d2e0-f94990ca71a8: failed to appendEntries to: peer="{Voter server-440ffb48-e3ea-e555-765e-e2cd8f114ed2 440ffb48-e3ea-e555-765e-e2cd8f114ed2}" error="failed to connect to peer: 440ffb48-e3ea-e555-765e-e2cd8f114ed2"
2025-02-23T18:43:23.688Z [ERROR] server-6d6d2902-c046-0242-d2e0-f94990ca71a8: failed to heartbeat to: peer=440ffb48-e3ea-e555-765e-e2cd8f114ed2 backoff time=10ms error="failed to connect to peer: 440ffb48-e3ea-e555-765e-e2cd8f114ed2"
2025-02-23T18:43:23.698Z [WARN]  server-6d6d2902-c046-0242-d2e0-f94990ca71a8: failed to contact: server-id=server-440ffb48-e3ea-e555-765e-e2cd8f114ed2 time=51.197576ms
2025-02-23T18:43:23.698Z [WARN]  server-6d6d2902-c046-0242-d2e0-f94990ca71a8: failed to contact quorum of nodes, stepping down
2025-02-23T18:43:23.698Z [INFO]  server-6d6d2902-c046-0242-d2e0-f94990ca71a8: entering follower state: follower="Node at 6d6d2902-c046-0242-d2e0-f94990ca71a8 [Follower]" leader-address= leader-id=
2025-02-23T18:43:23.701Z [ERROR] server-6d6d2902-c046-0242-d2e0-f94990ca71a8: failed to appendEntries to: peer="{Voter server-440ffb48-e3ea-e555-765e-e2cd8f114ed2 440ffb48-e3ea-e555-765e-e2cd8f114ed2}" error="failed to connect to peer: 440ffb48-e3ea-e555-765e-e2cd8f114ed2"
2025-02-23T18:43:23.733Z [WARN]  server-440ffb48-e3ea-e555-765e-e2cd8f114ed2: heartbeat timeout reached, starting election: last-leader-addr=6d6d2902-c046-0242-d2e0-f94990ca71a8 last-leader-id=server-6d6d2902-c046-0242-d2e0-f94990ca71a8
2025-02-23T18:43:23.733Z [INFO]  server-440ffb48-e3ea-e555-765e-e2cd8f114ed2: entering candidate state: node="Node at 440ffb48-e3ea-e555-765e-e2cd8f114ed2 [Candidate]" term=3
2025-02-23T18:43:23.733Z [ERROR] server-440ffb48-e3ea-e555-765e-e2cd8f114ed2: failed to make requestVote RPC: target="{Voter server-6d6d2902-c046-0242-d2e0-f94990ca71a8 6d6d2902-c046-0242-d2e0-f94990ca71a8}" error="failed to connect to peer: 6d6d2902-c046-0242-d2e0-f94990ca71a8" term=3
2025-02-23T18:43:23.754Z [WARN]  server-440ffb48-e3ea-e555-765e-e2cd8f114ed2: rejecting vote request since node is not in configuration: from=8659acf9-ead3-5153-4eeb-730651c1b628
--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.50s)
=== RUN   TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin
2025-02-23T18:43:23.755Z [INFO]  server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 Address:1c8869ae-8f2f-fd4b-e937-b34ff812ddb1} {Suffrage:Voter ID:server-8283aa00-f1be-67bb-03b8-a47f991b6f4a Address:8283aa00-f1be-67bb-03b8-a47f991b6f4a} {Suffrage:Voter ID:server-fba4b812-0985-b582-387f-54e2e62d6f4d Address:fba4b812-0985-b582-387f-54e2e62d6f4d}]"
2025-02-23T18:43:23.756Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 Address:1c8869ae-8f2f-fd4b-e937-b34ff812ddb1} {Suffrage:Voter ID:server-8283aa00-f1be-67bb-03b8-a47f991b6f4a Address:8283aa00-f1be-67bb-03b8-a47f991b6f4a} {Suffrage:Voter ID:server-fba4b812-0985-b582-387f-54e2e62d6f4d Address:fba4b812-0985-b582-387f-54e2e62d6f4d}]"
2025-02-23T18:43:23.756Z [INFO]  server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1: entering follower state: follower="Node at 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 [Follower]" leader-address= leader-id=
2025-02-23T18:43:23.756Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: entering follower state: follower="Node at 8283aa00-f1be-67bb-03b8-a47f991b6f4a [Follower]" leader-address= leader-id=
2025-02-23T18:43:23.756Z [INFO]  server-fba4b812-0985-b582-387f-54e2e62d6f4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 Address:1c8869ae-8f2f-fd4b-e937-b34ff812ddb1} {Suffrage:Voter ID:server-8283aa00-f1be-67bb-03b8-a47f991b6f4a Address:8283aa00-f1be-67bb-03b8-a47f991b6f4a} {Suffrage:Voter ID:server-fba4b812-0985-b582-387f-54e2e62d6f4d Address:fba4b812-0985-b582-387f-54e2e62d6f4d}]"
2025-02-23T18:43:23.756Z [INFO]  server-fba4b812-0985-b582-387f-54e2e62d6f4d: entering follower state: follower="Node at fba4b812-0985-b582-387f-54e2e62d6f4d [Follower]" leader-address= leader-id=
2025-02-23T18:43:23.756Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:23.817Z [WARN]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:23.817Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: entering candidate state: node="Node at 8283aa00-f1be-67bb-03b8-a47f991b6f4a [Candidate]" term=2
2025-02-23T18:43:23.817Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: election won: term=2 tally=2
2025-02-23T18:43:23.817Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: entering leader state: leader="Node at 8283aa00-f1be-67bb-03b8-a47f991b6f4a [Leader]"
2025-02-23T18:43:23.817Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: added peer, starting replication: peer=server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1
2025-02-23T18:43:23.817Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: added peer, starting replication: peer=server-fba4b812-0985-b582-387f-54e2e62d6f4d
2025-02-23T18:43:23.817Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: pipelining replication: peer="{Voter server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}"
2025-02-23T18:43:23.817Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: pipelining replication: peer="{Voter server-fba4b812-0985-b582-387f-54e2e62d6f4d fba4b812-0985-b582-387f-54e2e62d6f4d}"
2025-02-23T18:43:23.923Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:23.817612672 +0000 UTC m=+108.115873553 (1 nodes), 61.236486ms from start of poll, 61.764332ms from cluster start. Timeout at 2025-02-23 18:43:23.922619933 +0000 UTC m=+108.220880063, 105.00651ms after stability
2025-02-23T18:43:23.928Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:24.034Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:23.928930666 +0000 UTC m=+108.227191816 (2 nodes), 9.113µs from start of poll, 173.082595ms from cluster start. Timeout at 2025-02-23 18:43:24.033977201 +0000 UTC m=+108.332237842, 105.046026ms after stability
2025-02-23T18:43:24.034Z [ERROR] server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: failed to pipeline appendEntries: peer="{Voter server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}" error="append pipeline closed"
2025-02-23T18:43:24.034Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: aborting pipeline replication: peer="{Voter server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}"
2025-02-23T18:43:24.040Z [ERROR] server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: failed to heartbeat to: peer=1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 backoff time=10ms error="failed to connect to peer: 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1"
2025-02-23T18:43:24.043Z [ERROR] server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: failed to appendEntries to: peer="{Voter server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}" error="failed to connect to peer: 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1"
2025-02-23T18:43:24.060Z [ERROR] server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: failed to heartbeat to: peer=1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 backoff time=10ms error="failed to connect to peer: 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1"
2025-02-23T18:43:24.061Z [ERROR] server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: failed to appendEntries to: peer="{Voter server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}" error="failed to connect to peer: 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1"
2025-02-23T18:43:24.078Z [ERROR] server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: failed to heartbeat to: peer=1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 backoff time=10ms error="failed to connect to peer: 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1"
2025-02-23T18:43:24.081Z [WARN]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: failed to contact: server-id=server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 time=50.0732ms
2025-02-23T18:43:24.082Z [ERROR] server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: failed to appendEntries to: peer="{Voter server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}" error="failed to connect to peer: 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1"
2025-02-23T18:43:24.098Z [ERROR] server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: failed to heartbeat to: peer=1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 backoff time=20ms error="failed to connect to peer: 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1"
2025-02-23T18:43:24.110Z [ERROR] server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: failed to appendEntries to: peer="{Voter server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}" error="failed to connect to peer: 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1"
2025-02-23T18:43:24.117Z [WARN]  server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1: heartbeat timeout reached, starting election: last-leader-addr=8283aa00-f1be-67bb-03b8-a47f991b6f4a last-leader-id=server-8283aa00-f1be-67bb-03b8-a47f991b6f4a
2025-02-23T18:43:24.117Z [INFO]  server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1: entering candidate state: node="Node at 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 [Candidate]" term=3
2025-02-23T18:43:24.117Z [ERROR] server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1: failed to make requestVote RPC: target="{Voter server-8283aa00-f1be-67bb-03b8-a47f991b6f4a 8283aa00-f1be-67bb-03b8-a47f991b6f4a}" error="failed to connect to peer: 8283aa00-f1be-67bb-03b8-a47f991b6f4a" term=3
2025-02-23T18:43:24.117Z [ERROR] server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1: failed to make requestVote RPC: target="{Voter server-fba4b812-0985-b582-387f-54e2e62d6f4d fba4b812-0985-b582-387f-54e2e62d6f4d}" error="failed to connect to peer: fba4b812-0985-b582-387f-54e2e62d6f4d" term=3
2025-02-23T18:43:24.132Z [ERROR] server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: failed to heartbeat to: peer=1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 backoff time=25ms error="failed to connect to peer: 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1"
2025-02-23T18:43:24.133Z [WARN]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: failed to contact: server-id=server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 time=101.65831ms
2025-02-23T18:43:24.139Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: updating configuration: command=RemoveServer server-id=server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 server-addr= servers="[{Suffrage:Voter ID:server-8283aa00-f1be-67bb-03b8-a47f991b6f4a Address:8283aa00-f1be-67bb-03b8-a47f991b6f4a} {Suffrage:Voter ID:server-fba4b812-0985-b582-387f-54e2e62d6f4d Address:fba4b812-0985-b582-387f-54e2e62d6f4d}]"
2025-02-23T18:43:24.140Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: removed peer, stopping replication: peer=server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 last-index=3
2025-02-23T18:43:24.140Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: updating configuration: command=AddNonvoter server-id=server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 server-addr=1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 servers="[{Suffrage:Voter ID:server-8283aa00-f1be-67bb-03b8-a47f991b6f4a Address:8283aa00-f1be-67bb-03b8-a47f991b6f4a} {Suffrage:Voter ID:server-fba4b812-0985-b582-387f-54e2e62d6f4d Address:fba4b812-0985-b582-387f-54e2e62d6f4d} {Suffrage:Nonvoter ID:server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 Address:1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}]"
2025-02-23T18:43:24.140Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: added peer, starting replication: peer=server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1
2025-02-23T18:43:24.140Z [ERROR] server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: failed to appendEntries to: peer="{Nonvoter server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}" error="failed to connect to peer: 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1"
2025-02-23T18:43:24.157Z [ERROR] server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: peer has newer term, stopping replication: peer="{Voter server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}"
2025-02-23T18:43:24.157Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: entering follower state: follower="Node at 8283aa00-f1be-67bb-03b8-a47f991b6f4a [Follower]" leader-address= leader-id=
2025-02-23T18:43:24.157Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: aborting pipeline replication: peer="{Voter server-fba4b812-0985-b582-387f-54e2e62d6f4d fba4b812-0985-b582-387f-54e2e62d6f4d}"
2025-02-23T18:43:24.158Z [ERROR] server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: peer has newer term, stopping replication: peer="{Nonvoter server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}"
2025-02-23T18:43:24.212Z [WARN]  server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1: Election timeout reached, restarting election
2025-02-23T18:43:24.212Z [INFO]  server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1: entering candidate state: node="Node at 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 [Candidate]" term=23
2025-02-23T18:43:24.212Z [WARN]  server-fba4b812-0985-b582-387f-54e2e62d6f4d: rejecting vote request since we have a leader: from=1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 leader=8283aa00-f1be-67bb-03b8-a47f991b6f4a leader-id=server-8283aa00-f1be-67bb-03b8-a47f991b6f4a
2025-02-23T18:43:24.212Z [WARN]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: rejecting vote request since node is not a voter: from=1c8869ae-8f2f-fd4b-e937-b34ff812ddb1
2025-02-23T18:43:24.227Z [WARN]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:24.227Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: entering candidate state: node="Node at 8283aa00-f1be-67bb-03b8-a47f991b6f4a [Candidate]" term=24
2025-02-23T18:43:24.227Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: election won: term=24 tally=2
2025-02-23T18:43:24.227Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: entering leader state: leader="Node at 8283aa00-f1be-67bb-03b8-a47f991b6f4a [Leader]"
2025-02-23T18:43:24.227Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: added peer, starting replication: peer=server-fba4b812-0985-b582-387f-54e2e62d6f4d
2025-02-23T18:43:24.227Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: added peer, starting replication: peer=server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1
2025-02-23T18:43:24.227Z [WARN]  server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1: failed to get previous log: previous-index=4 last-index=2 error="log not found"
2025-02-23T18:43:24.227Z [INFO]  server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1: entering follower state: follower="Node at 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 [Follower]" leader-address=8283aa00-f1be-67bb-03b8-a47f991b6f4a leader-id=server-8283aa00-f1be-67bb-03b8-a47f991b6f4a
2025-02-23T18:43:24.227Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: pipelining replication: peer="{Voter server-fba4b812-0985-b582-387f-54e2e62d6f4d fba4b812-0985-b582-387f-54e2e62d6f4d}"
2025-02-23T18:43:24.227Z [WARN]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: appendEntries rejected, sending older logs: peer="{Nonvoter server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}" next=3
2025-02-23T18:43:24.227Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: pipelining replication: peer="{Nonvoter server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}"
2025-02-23T18:43:24.246Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:24.352Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:24.2465535 +0000 UTC m=+108.544813630 (1 nodes), 106.776µs from start of poll, 490.704409ms from cluster start. Timeout at 2025-02-23 18:43:24.351599866 +0000 UTC m=+108.649860539, 105.046909ms after stability
2025-02-23T18:43:24.392Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: updating configuration: command=RemoveServer server-id=server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 server-addr= servers="[{Suffrage:Voter ID:server-8283aa00-f1be-67bb-03b8-a47f991b6f4a Address:8283aa00-f1be-67bb-03b8-a47f991b6f4a} {Suffrage:Voter ID:server-fba4b812-0985-b582-387f-54e2e62d6f4d Address:fba4b812-0985-b582-387f-54e2e62d6f4d}]"
2025-02-23T18:43:24.392Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: removed peer, stopping replication: peer=server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 last-index=106
2025-02-23T18:43:24.393Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: aborting pipeline replication: peer="{Nonvoter server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}"
2025-02-23T18:43:24.393Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: updating configuration: command=AddVoter server-id=server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 server-addr=1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 servers="[{Suffrage:Voter ID:server-8283aa00-f1be-67bb-03b8-a47f991b6f4a Address:8283aa00-f1be-67bb-03b8-a47f991b6f4a} {Suffrage:Voter ID:server-fba4b812-0985-b582-387f-54e2e62d6f4d Address:fba4b812-0985-b582-387f-54e2e62d6f4d} {Suffrage:Voter ID:server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 Address:1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}]"
2025-02-23T18:43:24.393Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: added peer, starting replication: peer=server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1
2025-02-23T18:43:24.393Z [WARN]  server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1: failed to get previous log: previous-index=107 last-index=106 error="log not found"
2025-02-23T18:43:24.393Z [WARN]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: appendEntries rejected, sending older logs: peer="{Voter server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}" next=107
2025-02-23T18:43:24.393Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: pipelining replication: peer="{Voter server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}"
2025-02-23T18:43:25.481Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:25.587Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:25.481105282 +0000 UTC m=+109.779366070 (1 nodes), 2.359µs from start of poll, 1.725256849s from cluster start. Timeout at 2025-02-23 18:43:25.586162869 +0000 UTC m=+109.884423632, 105.057562ms after stability
2025-02-23T18:43:25.587Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: aborting pipeline replication: peer="{Voter server-1c8869ae-8f2f-fd4b-e937-b34ff812ddb1 1c8869ae-8f2f-fd4b-e937-b34ff812ddb1}"
2025-02-23T18:43:25.587Z [INFO]  server-8283aa00-f1be-67bb-03b8-a47f991b6f4a: aborting pipeline replication: peer="{Voter server-fba4b812-0985-b582-387f-54e2e62d6f4d fba4b812-0985-b582-387f-54e2e62d6f4d}"
--- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.83s)
=== RUN   TestRaft_FollowerRemovalNoElection
2025-02-23T18:43:25.588Z [INFO]  server-2e833bfe-1877-00f5-5ac4-36d44ba7b4a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e833bfe-1877-00f5-5ac4-36d44ba7b4a6 Address:2e833bfe-1877-00f5-5ac4-36d44ba7b4a6} {Suffrage:Voter ID:server-10cb87da-e721-87f9-e289-3f099142888c Address:10cb87da-e721-87f9-e289-3f099142888c} {Suffrage:Voter ID:server-b35639ab-5931-bd8e-429e-0d2143204ea3 Address:b35639ab-5931-bd8e-429e-0d2143204ea3}]"
2025-02-23T18:43:25.589Z [INFO]  server-10cb87da-e721-87f9-e289-3f099142888c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e833bfe-1877-00f5-5ac4-36d44ba7b4a6 Address:2e833bfe-1877-00f5-5ac4-36d44ba7b4a6} {Suffrage:Voter ID:server-10cb87da-e721-87f9-e289-3f099142888c Address:10cb87da-e721-87f9-e289-3f099142888c} {Suffrage:Voter ID:server-b35639ab-5931-bd8e-429e-0d2143204ea3 Address:b35639ab-5931-bd8e-429e-0d2143204ea3}]"
2025-02-23T18:43:25.589Z [INFO]  server-2e833bfe-1877-00f5-5ac4-36d44ba7b4a6: entering follower state: follower="Node at 2e833bfe-1877-00f5-5ac4-36d44ba7b4a6 [Follower]" leader-address= leader-id=
2025-02-23T18:43:25.589Z [INFO]  server-b35639ab-5931-bd8e-429e-0d2143204ea3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e833bfe-1877-00f5-5ac4-36d44ba7b4a6 Address:2e833bfe-1877-00f5-5ac4-36d44ba7b4a6} {Suffrage:Voter ID:server-10cb87da-e721-87f9-e289-3f099142888c Address:10cb87da-e721-87f9-e289-3f099142888c} {Suffrage:Voter ID:server-b35639ab-5931-bd8e-429e-0d2143204ea3 Address:b35639ab-5931-bd8e-429e-0d2143204ea3}]"
2025-02-23T18:43:25.589Z [INFO]  server-10cb87da-e721-87f9-e289-3f099142888c: entering follower state: follower="Node at 10cb87da-e721-87f9-e289-3f099142888c [Follower]" leader-address= leader-id=
2025-02-23T18:43:25.589Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:25.589Z [INFO]  server-b35639ab-5931-bd8e-429e-0d2143204ea3: entering follower state: follower="Node at b35639ab-5931-bd8e-429e-0d2143204ea3 [Follower]" leader-address= leader-id=
2025-02-23T18:43:25.706Z [WARN]  server-10cb87da-e721-87f9-e289-3f099142888c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:25.706Z [INFO]  server-10cb87da-e721-87f9-e289-3f099142888c: entering candidate state: node="Node at 10cb87da-e721-87f9-e289-3f099142888c [Candidate]" term=2
2025-02-23T18:43:25.706Z [INFO]  server-10cb87da-e721-87f9-e289-3f099142888c: election won: term=2 tally=2
2025-02-23T18:43:25.706Z [INFO]  server-10cb87da-e721-87f9-e289-3f099142888c: entering leader state: leader="Node at 10cb87da-e721-87f9-e289-3f099142888c [Leader]"
2025-02-23T18:43:25.706Z [INFO]  server-10cb87da-e721-87f9-e289-3f099142888c: added peer, starting replication: peer=server-2e833bfe-1877-00f5-5ac4-36d44ba7b4a6
2025-02-23T18:43:25.706Z [INFO]  server-10cb87da-e721-87f9-e289-3f099142888c: added peer, starting replication: peer=server-b35639ab-5931-bd8e-429e-0d2143204ea3
2025-02-23T18:43:25.706Z [INFO]  server-10cb87da-e721-87f9-e289-3f099142888c: pipelining replication: peer="{Voter server-2e833bfe-1877-00f5-5ac4-36d44ba7b4a6 2e833bfe-1877-00f5-5ac4-36d44ba7b4a6}"
2025-02-23T18:43:25.706Z [INFO]  server-10cb87da-e721-87f9-e289-3f099142888c: pipelining replication: peer="{Voter server-b35639ab-5931-bd8e-429e-0d2143204ea3 b35639ab-5931-bd8e-429e-0d2143204ea3}"
2025-02-23T18:43:25.911Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:25.706343266 +0000 UTC m=+110.004604305 (1 nodes), 117.085843ms from start of poll, 117.601336ms from cluster start. Timeout at 2025-02-23 18:43:25.911444087 +0000 UTC m=+110.209704942, 205.100637ms after stability
2025-02-23T18:43:25.911Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:26.123Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:25.911611515 +0000 UTC m=+110.209872170 (1 nodes), 2.051µs from start of poll, 322.869201ms from cluster start. Timeout at 2025-02-23 18:43:26.116669484 +0000 UTC m=+110.414930131, 205.057961ms after stability
2025-02-23T18:43:26.129Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:26.335Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:26.129788505 +0000 UTC m=+110.428049955 (2 nodes), 3.353µs from start of poll, 541.046986ms from cluster start. Timeout at 2025-02-23 18:43:26.334878715 +0000 UTC m=+110.633139424, 205.089469ms after stability
2025-02-23T18:43:26.335Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:26.541Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:26.335999811 +0000 UTC m=+110.634260578 (1 nodes), 2.68µs from start of poll, 747.257609ms from cluster start. Timeout at 2025-02-23 18:43:26.541039409 +0000 UTC m=+110.839300158, 205.03958ms after stability
    raft_test.go:2961: [INFO] restarting Node at 2e833bfe-1877-00f5-5ac4-36d44ba7b4a6 [Follower]
2025-02-23T18:43:26.541Z [INFO]  server-2e833bfe-1877-00f5-5ac4-36d44ba7b4a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e833bfe-1877-00f5-5ac4-36d44ba7b4a6 Address:2e833bfe-1877-00f5-5ac4-36d44ba7b4a6} {Suffrage:Voter ID:server-10cb87da-e721-87f9-e289-3f099142888c Address:10cb87da-e721-87f9-e289-3f099142888c} {Suffrage:Voter ID:server-b35639ab-5931-bd8e-429e-0d2143204ea3 Address:b35639ab-5931-bd8e-429e-0d2143204ea3}]"
2025-02-23T18:43:26.541Z [INFO]  server-2e833bfe-1877-00f5-5ac4-36d44ba7b4a6: entering follower state: follower="Node at 2e833bfe-1877-00f5-5ac4-36d44ba7b4a6 [Follower]" leader-address= leader-id=
2025-02-23T18:43:26.792Z [INFO]  server-10cb87da-e721-87f9-e289-3f099142888c: aborting pipeline replication: peer="{Voter server-b35639ab-5931-bd8e-429e-0d2143204ea3 b35639ab-5931-bd8e-429e-0d2143204ea3}"
2025-02-23T18:43:27.181Z [ERROR] server-10cb87da-e721-87f9-e289-3f099142888c: failed to pipeline appendEntries: peer="{Voter server-2e833bfe-1877-00f5-5ac4-36d44ba7b4a6 2e833bfe-1877-00f5-5ac4-36d44ba7b4a6}" error="command enqueue timeout"
2025-02-23T18:43:27.181Z [INFO]  server-10cb87da-e721-87f9-e289-3f099142888c: aborting pipeline replication: peer="{Voter server-2e833bfe-1877-00f5-5ac4-36d44ba7b4a6 2e833bfe-1877-00f5-5ac4-36d44ba7b4a6}"
2025-02-23T18:43:27.306Z [ERROR] server-10cb87da-e721-87f9-e289-3f099142888c: failed to heartbeat to: peer=2e833bfe-1877-00f5-5ac4-36d44ba7b4a6 backoff time=10ms error="command timed out"
--- PASS: TestRaft_FollowerRemovalNoElection (1.72s)
=== RUN   TestRaft_VoteWithNoIDNoAddr
2025-02-23T18:43:27.308Z [INFO]  server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641 Address:7da6c7be-dcbc-8be5-5a2e-cac8793d7641} {Suffrage:Voter ID:server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763 Address:18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763} {Suffrage:Voter ID:server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372 Address:d3b6370e-f922-08e3-7b5f-a86e9e3f6372}]"
2025-02-23T18:43:27.308Z [INFO]  server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641 Address:7da6c7be-dcbc-8be5-5a2e-cac8793d7641} {Suffrage:Voter ID:server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763 Address:18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763} {Suffrage:Voter ID:server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372 Address:d3b6370e-f922-08e3-7b5f-a86e9e3f6372}]"
2025-02-23T18:43:27.308Z [INFO]  server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641 Address:7da6c7be-dcbc-8be5-5a2e-cac8793d7641} {Suffrage:Voter ID:server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763 Address:18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763} {Suffrage:Voter ID:server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372 Address:d3b6370e-f922-08e3-7b5f-a86e9e3f6372}]"
2025-02-23T18:43:27.308Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:27.308Z [INFO]  server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641: entering follower state: follower="Node at 7da6c7be-dcbc-8be5-5a2e-cac8793d7641 [Follower]" leader-address= leader-id=
2025-02-23T18:43:27.308Z [INFO]  server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: entering follower state: follower="Node at d3b6370e-f922-08e3-7b5f-a86e9e3f6372 [Follower]" leader-address= leader-id=
2025-02-23T18:43:27.308Z [INFO]  server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763: entering follower state: follower="Node at 18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763 [Follower]" leader-address= leader-id=
2025-02-23T18:43:27.361Z [WARN]  server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-02-23T18:43:27.361Z [INFO]  server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: entering candidate state: node="Node at d3b6370e-f922-08e3-7b5f-a86e9e3f6372 [Candidate]" term=2
2025-02-23T18:43:27.362Z [INFO]  server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: election won: term=2 tally=2
2025-02-23T18:43:27.362Z [INFO]  server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: entering leader state: leader="Node at d3b6370e-f922-08e3-7b5f-a86e9e3f6372 [Leader]"
2025-02-23T18:43:27.362Z [INFO]  server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: added peer, starting replication: peer=server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641
2025-02-23T18:43:27.362Z [INFO]  server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: added peer, starting replication: peer=server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763
2025-02-23T18:43:27.362Z [INFO]  server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: pipelining replication: peer="{Voter server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763 18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763}"
2025-02-23T18:43:27.362Z [INFO]  server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: pipelining replication: peer="{Voter server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641 7da6c7be-dcbc-8be5-5a2e-cac8793d7641}"
2025-02-23T18:43:27.467Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:27.362601192 +0000 UTC m=+111.660861976 (1 nodes), 53.687222ms from start of poll, 54.128843ms from cluster start. Timeout at 2025-02-23 18:43:27.467603681 +0000 UTC m=+111.765864299, 105.002323ms after stability
2025-02-23T18:43:27.467Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-02-23T18:43:27.573Z [INFO]  cluster: stable state for Leader reached at 2025-02-23 18:43:27.467721526 +0000 UTC m=+111.765982266 (1 nodes), 1.91µs from start of poll, 159.249133ms from cluster start. Timeout at 2025-02-23 18:43:27.572771682 +0000 UTC m=+111.871032347, 105.050081ms after stability
2025-02-23T18:43:27.579Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-02-23T18:43:27.689Z [INFO]  cluster: stable state for Follower reached at 2025-02-23 18:43:27.579634006 +0000 UTC m=+111.877894820 (2 nodes), 2.242µs from start of poll, 271.161687ms from cluster start. Timeout at 2025-02-23 18:43:27.684651797 +0000 UTC m=+111.982912751, 105.017931ms after stability
2025-02-23T18:43:27.691Z [ERROR] server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to pipeline appendEntries: peer="{Voter server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763 18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763}" error="append pipeline closed"
2025-02-23T18:43:27.691Z [INFO]  server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: aborting pipeline replication: peer="{Voter server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763 18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763}"
2025-02-23T18:43:27.696Z [ERROR] server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to pipeline appendEntries: peer="{Voter server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641 7da6c7be-dcbc-8be5-5a2e-cac8793d7641}" error="append pipeline closed"
2025-02-23T18:43:27.696Z [INFO]  server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: aborting pipeline replication: peer="{Voter server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641 7da6c7be-dcbc-8be5-5a2e-cac8793d7641}"
2025-02-23T18:43:27.697Z [ERROR] server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to appendEntries to: peer="{Voter server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763 18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763}" error="failed to connect to peer: 18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763"
2025-02-23T18:43:27.697Z [ERROR] server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to heartbeat to: peer=7da6c7be-dcbc-8be5-5a2e-cac8793d7641 backoff time=10ms error="failed to connect to peer: 7da6c7be-dcbc-8be5-5a2e-cac8793d7641"
2025-02-23T18:43:27.699Z [ERROR] server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to heartbeat to: peer=18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763 backoff time=10ms error="failed to connect to peer: 18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763"
2025-02-23T18:43:27.704Z [ERROR] server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to appendEntries to: peer="{Voter server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641 7da6c7be-dcbc-8be5-5a2e-cac8793d7641}" error="failed to connect to peer: 7da6c7be-dcbc-8be5-5a2e-cac8793d7641"
2025-02-23T18:43:27.714Z [ERROR] server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to appendEntries to: peer="{Voter server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763 18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763}" error="failed to connect to peer: 18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763"
2025-02-23T18:43:27.718Z [ERROR] server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to heartbeat to: peer=7da6c7be-dcbc-8be5-5a2e-cac8793d7641 backoff time=10ms error="failed to connect to peer: 7da6c7be-dcbc-8be5-5a2e-cac8793d7641"
2025-02-23T18:43:27.718Z [ERROR] server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to heartbeat to: peer=18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763 backoff time=10ms error="failed to connect to peer: 18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763"
2025-02-23T18:43:27.724Z [ERROR] server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to appendEntries to: peer="{Voter server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641 7da6c7be-dcbc-8be5-5a2e-cac8793d7641}" error="failed to connect to peer: 7da6c7be-dcbc-8be5-5a2e-cac8793d7641"
2025-02-23T18:43:27.731Z [ERROR] server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to appendEntries to: peer="{Voter server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763 18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763}" error="failed to connect to peer: 18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763"
2025-02-23T18:43:27.737Z [ERROR] server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to heartbeat to: peer=18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763 backoff time=10ms error="failed to connect to peer: 18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763"
2025-02-23T18:43:27.737Z [ERROR] server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to heartbeat to: peer=7da6c7be-dcbc-8be5-5a2e-cac8793d7641 backoff time=10ms error="failed to connect to peer: 7da6c7be-dcbc-8be5-5a2e-cac8793d7641"
2025-02-23T18:43:27.741Z [WARN]  server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to contact: server-id=server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641 time=51.124205ms
2025-02-23T18:43:27.741Z [WARN]  server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to contact: server-id=server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763 time=51.052826ms
2025-02-23T18:43:27.741Z [WARN]  server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to contact quorum of nodes, stepping down
2025-02-23T18:43:27.741Z [INFO]  server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: entering follower state: follower="Node at d3b6370e-f922-08e3-7b5f-a86e9e3f6372 [Follower]" leader-address= leader-id=
2025-02-23T18:43:27.742Z [ERROR] server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to appendEntries to: peer="{Voter server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641 7da6c7be-dcbc-8be5-5a2e-cac8793d7641}" error="failed to connect to peer: 7da6c7be-dcbc-8be5-5a2e-cac8793d7641"
2025-02-23T18:43:27.754Z [ERROR] server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to heartbeat to: peer=18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763 backoff time=20ms error="failed to connect to peer: 18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763"
2025-02-23T18:43:27.755Z [WARN]  server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641: heartbeat timeout reached, starting election: last-leader-addr=d3b6370e-f922-08e3-7b5f-a86e9e3f6372 last-leader-id=server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372
2025-02-23T18:43:27.755Z [INFO]  server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641: entering candidate state: node="Node at 7da6c7be-dcbc-8be5-5a2e-cac8793d7641 [Candidate]" term=3
2025-02-23T18:43:27.756Z [WARN]  server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763: rejecting vote request since we have a leader: from=7da6c7be-dcbc-8be5-5a2e-cac8793d7641 leader=d3b6370e-f922-08e3-7b5f-a86e9e3f6372 leader-id=server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372
2025-02-23T18:43:27.756Z [ERROR] server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641: failed to make requestVote RPC: target="{Voter server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372 d3b6370e-f922-08e3-7b5f-a86e9e3f6372}" error="failed to connect to peer: d3b6370e-f922-08e3-7b5f-a86e9e3f6372" term=3
2025-02-23T18:43:27.757Z [ERROR] server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372: failed to appendEntries to: peer="{Voter server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763 18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763}" error="failed to connect to peer: 18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763"
2025-02-23T18:43:27.763Z [WARN]  server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763: heartbeat timeout reached, starting election: last-leader-addr=d3b6370e-f922-08e3-7b5f-a86e9e3f6372 last-leader-id=server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372
2025-02-23T18:43:27.763Z [INFO]  server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763: entering candidate state: node="Node at 18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763 [Candidate]" term=3
2025-02-23T18:43:27.763Z [ERROR] server-18e0cfd4-7f4e-ecf9-f8ef-e04d3b914763: failed to make requestVote RPC: target="{Voter server-d3b6370e-f922-08e3-7b5f-a86e9e3f6372 d3b6370e-f922-08e3-7b5f-a86e9e3f6372}" error="failed to connect to peer: d3b6370e-f922-08e3-7b5f-a86e9e3f6372" term=3
2025-02-23T18:43:27.763Z [INFO]  server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641: duplicate requestVote for same term: term=3
2025-02-23T18:43:27.795Z [INFO]  server-7da6c7be-dcbc-8be5-5a2e-cac8793d7641: entering follower state: follower="Node at 7da6c7be-dcbc-8be5-5a2e-cac8793d7641 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VoteWithNoIDNoAddr (0.49s)
=== RUN   TestRaft_runFollower_State_Transition
=== RUN   TestRaft_runFollower_State_Transition/NonVoter
2025-02-23T18:43:27.796Z [INFO]  127.0.0.1:33585 :: starting node: addr=127.0.0.1:33585
2025-02-23T18:43:27.796Z [INFO]  127.0.0.1:33585 :: initial configuration: index=0 servers=[]
2025-02-23T18:43:27.796Z [INFO]  127.0.0.1:33585 :: entering follower state: follower="Node at 127.0.0.1:33585 [Follower]" leader-address= leader-id=
2025-02-23T18:43:27.858Z [WARN]  127.0.0.1:33585 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
=== RUN   TestRaft_runFollower_State_Transition/Voter
2025-02-23T18:43:27.947Z [INFO]  127.0.0.1:36237 :: starting node: addr=127.0.0.1:36237
2025-02-23T18:43:27.947Z [INFO]  127.0.0.1:36237 :: initial configuration: index=0 servers=[]
2025-02-23T18:43:27.947Z [INFO]  127.0.0.1:36237 :: entering follower state: follower="Node at 127.0.0.1:36237 [Follower]" leader-address= leader-id=
2025-02-23T18:43:28.020Z [WARN]  127.0.0.1:36237 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
=== RUN   TestRaft_runFollower_State_Transition/Not_in_Config
2025-02-23T18:43:28.098Z [INFO]  127.0.0.1:46341 :: starting node: addr=127.0.0.1:46341
2025-02-23T18:43:28.098Z [INFO]  127.0.0.1:46341 :: initial configuration: index=0 servers=[]
2025-02-23T18:43:28.098Z [INFO]  127.0.0.1:46341 :: entering follower state: follower="Node at 127.0.0.1:46341 [Follower]" leader-address= leader-id=
2025-02-23T18:43:28.192Z [WARN]  127.0.0.1:46341 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
--- PASS: TestRaft_runFollower_State_Transition (0.45s)
    --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s)
=== RUN   TestRaft_runFollower_ReloadTimeoutConfigs
2025-02-23T18:43:28.249Z [INFO]  127.0.0.1:34001 :: starting node: addr=127.0.0.1:34001
2025-02-23T18:43:28.249Z [INFO]  127.0.0.1:34001 :: initial configuration: index=0 servers=[]
2025-02-23T18:43:28.249Z [INFO]  127.0.0.1:34001 :: entering follower state: follower="Node at 127.0.0.1:34001 [Follower]" leader-address= leader-id=
2025-02-23T18:43:28.250Z [WARN]  127.0.0.1:34001 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
--- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s)
=== RUN   TestSaturationMetric
=== RUN   TestSaturationMetric/without_smoothing
--- PASS: TestSaturationMetric (0.00s)
    --- PASS: TestSaturationMetric/without_smoothing (0.00s)
=== RUN   TestSaturationMetric_IncorrectUsage
=== RUN   TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_first
--- PASS: TestSaturationMetric_IncorrectUsage (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s)
=== RUN   TestTCPTransport_BadAddr
--- PASS: TestTCPTransport_BadAddr (0.00s)
=== RUN   TestTCPTransport_EmptyAddr
--- PASS: TestTCPTransport_EmptyAddr (0.00s)
=== RUN   TestTCPTransport_WithAdvertise
--- PASS: TestTCPTransport_WithAdvertise (0.00s)
=== RUN   TestTransport_StartStop
--- PASS: TestTransport_StartStop (0.00s)
=== RUN   TestTransport_AppendEntries
--- PASS: TestTransport_AppendEntries (0.00s)
=== RUN   TestTransport_AppendEntriesPipeline
--- PASS: TestTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestTransport_RequestVote
--- PASS: TestTransport_RequestVote (0.00s)
=== RUN   TestTransport_InstallSnapshot
--- PASS: TestTransport_InstallSnapshot (0.00s)
=== RUN   TestTransport_EncodeDecode
--- PASS: TestTransport_EncodeDecode (0.00s)
=== RUN   TestRandomTimeout
--- PASS: TestRandomTimeout (0.00s)
=== RUN   TestNewSeed
--- PASS: TestNewSeed (0.00s)
=== RUN   TestRandomTimeout_NoTime
--- PASS: TestRandomTimeout_NoTime (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
=== RUN   TestMax
--- PASS: TestMax (0.00s)
=== RUN   TestGenerateUUID
--- PASS: TestGenerateUUID (0.00s)
=== RUN   TestBackoff
--- PASS: TestBackoff (0.00s)
=== RUN   TestOverrideNotifyBool
--- PASS: TestOverrideNotifyBool (0.00s)
PASS
ok  	github.com/hashicorp/raft	112.719s
?   	github.com/hashicorp/raft/bench	[no test files]
make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
   create-stamp debian/debhelper-build-stamp
   dh_testroot -O--buildsystem=golang
   dh_prep -O--buildsystem=golang
   dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang
   dh_installdocs -O--buildsystem=golang
   dh_installchangelogs -O--buildsystem=golang
   dh_installsystemduser -O--buildsystem=golang
   dh_perl -O--buildsystem=golang
   dh_link -O--buildsystem=golang
   dh_strip_nondeterminism -O--buildsystem=golang
   dh_compress -O--buildsystem=golang
   dh_fixperms -O--buildsystem=golang
   dh_missing -O--buildsystem=golang
   dh_installdeb -O--buildsystem=golang
   dh_golang -O--buildsystem=golang
   dh_gencontrol -O--buildsystem=golang
   dh_md5sums -O--buildsystem=golang
   dh_builddeb -O--buildsystem=golang
dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'.
 dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_i386.buildinfo
 dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_i386.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-buildpackage: info: binary-only upload (no source included)
dpkg-genchanges: info: including full source code in upload
I: copying local configuration
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /srv/workspace/pbuilder/30230 and its subdirectories
I: Current time: Sun Feb 23 06:43:35 -12 2025
I: pbuilder-time-stamp: 1740336215
Sun Feb 23 18:43:37 UTC 2025  I: 1st build successful. Starting 2nd build on remote node ionos16-i386.debian.net.
Sun Feb 23 18:43:37 UTC 2025  I: Preparing to do remote build '2' on ionos16-i386.debian.net.
Sun Feb 23 18:46:32 UTC 2025  I: Deleting $TMPDIR on ionos16-i386.debian.net.