Tue Feb 25 23:52:14 UTC 2025 I: starting to build golang-github-hashicorp-raft/trixie/arm64 on jenkins on '2025-02-25 23:51' Tue Feb 25 23:52:14 UTC 2025 I: The jenkins build log is/was available at https://jenkins.debian.net/userContent/reproducible/debian/build_service/arm64_4/58864/console.log Tue Feb 25 23:52:14 UTC 2025 I: Downloading source for trixie/golang-github-hashicorp-raft=1.5.0-1 --2025-02-25 23:52:15-- http://deb.debian.org/debian/pool/main/g/golang-github-hashicorp-raft/golang-github-hashicorp-raft_1.5.0-1.dsc Connecting to 46.16.76.132:3128... connected. Proxy request sent, awaiting response... 200 OK Length: 2480 (2.4K) [text/prs.lines.tag] Saving to: ‘golang-github-hashicorp-raft_1.5.0-1.dsc’ 0K .. 100% 404M=0s 2025-02-25 23:52:15 (404 MB/s) - ‘golang-github-hashicorp-raft_1.5.0-1.dsc’ saved [2480/2480] Tue Feb 25 23:52:15 UTC 2025 I: golang-github-hashicorp-raft_1.5.0-1.dsc -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Format: 3.0 (quilt) Source: golang-github-hashicorp-raft Binary: golang-github-hashicorp-raft-dev Architecture: all Version: 1.5.0-1 Maintainer: Debian Go Packaging Team Uploaders: Tim Potter , Dmitry Smirnov , Homepage: https://github.com/hashicorp/raft Standards-Version: 4.6.2 Vcs-Browser: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft Vcs-Git: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git Testsuite: autopkgtest-pkg-go Build-Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev Package-List: golang-github-hashicorp-raft-dev deb golang optional arch=all Checksums-Sha1: cea1c93d584c6f8dfbf53aef3438ab18ec7e7222 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz 015fac87182a135b38ce31d1eb99f68ca251ae23 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz Checksums-Sha256: f4dc9aabc7784a1b9b5da36b85a3797ac37911f2a6dc54a6fd6cb60a6fd643e9 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz 0d95ee03abc0b447cfb92e400d914ecee5113fd181a4ea1d51fa5b6c37288658 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz Files: 7ebc703c8f7e2d7fc02568216f2e3b0a 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz 1f0ee65ab76bca0e05a75b8f9ad32d8d 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz Go-Import-Path: github.com/hashicorp/raft -----BEGIN PGP SIGNATURE----- iQJKBAEBCgA0FiEEMD9oek78sa58GjWjtwAXP7uAWikFAmTCiiIWHGRzd2FyYnJp Y2tAZGViaWFuLm9yZwAKCRC3ABc/u4BaKYS6D/wN81mTx6pt43be/9H7Ovy6z+5p nv0+kgrOLtH5y4KUjNLZJuPljAp+uoELmotxp1wqkoLSYs1AwCZ11X4NVs8YoLXj sLd+Ub+QcWb0Gejqjw7Nk9VZqqXjwfFJW54n8NC2Zy8D6OelCl47sDPmiu03v3uz oRu4dp2DW/uKkC0vb4RczT81GBmJ19Tu2ZdDvtk7kwnvF48cqDhXMhUTQAEjoyVU Q8kfY13blza+uXZk6xhWyYsc+wpTh6RHZIWsbpgejKmEYnZCoCdaR0UqOoau33Bs T1OXVNM0P6vrmeXyoiEy66Q2XyHeei+HYRUgED16Ujg0qxPxSEVE5timSwcXHa/u xb5ZcKQwqEoJISjWTFDG8gww8MBPOKNGW5BKRdQSdtDhNEsnmn58jl1lsRs5L6Xg +DrufjU6KKqqYyOBS+H+s68KXleg2t/qP+K8GZjJl/eX3+Tw5xOdt3ZWDMWxB2qF gS+ASmxYV0Uu/pd0MLWXjAFcllnl93MzPVkF14A8eYrghujLjN3+oUOD0RQzxpAb LHc5b0iPnN4/YQtJX3fZi2y6c62BCZ5Slb5sB0bsgNPbk1T3qQrtMPWJXUfr32ys VOow6eNA+VDxNxHZ3aSUiVC0lcUm6dJDYqcfk/PgDdBR/E7iigkTi3O/2rgW0Y7Z 14FQMVvbK31veGXJjA== =okTI -----END PGP SIGNATURE----- Tue Feb 25 23:52:15 UTC 2025 I: Checking whether the package is not for us Tue Feb 25 23:52:15 UTC 2025 I: Starting 1st build on remote node codethink03-arm64.debian.net. Tue Feb 25 23:52:15 UTC 2025 I: Preparing to do remote build '1' on codethink03-arm64.debian.net. Tue Feb 25 23:55:12 UTC 2025 I: Deleting $TMPDIR on codethink03-arm64.debian.net. I: pbuilder: network access will be disabled during build I: Current time: Mon Mar 30 18:15:16 -12 2026 I: pbuilder-time-stamp: 1774937716 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/trixie-reproducible-base.tgz] I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: Copying source file I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc] I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz] I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz] I: Extracting source dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: unsupported subcommand dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying 0001-Skip-unreliable-tests.patch I: Not using root during the build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/674357/tmp/hooks/D02_print_environment starting I: set BUILDDIR='/build/reproducible-path' BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' BUILDUSERNAME='pbuilder1' BUILD_ARCH='arm64' DEBIAN_FRONTEND='noninteractive' DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=12 ' DISTRIBUTION='trixie' HOME='/root' HOST_ARCH='arm64' IFS=' ' INVOCATION_ID='c8d09372adf848c383950b010f0e8b77' LANG='C' LANGUAGE='en_US:en' LC_ALL='C' MAIL='/var/mail/root' OPTIND='1' PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' PBCURRENTCOMMANDLINEOPERATION='build' PBUILDER_OPERATION='build' PBUILDER_PKGDATADIR='/usr/share/pbuilder' PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' PBUILDER_SYSCONFDIR='/etc' PPID='674357' PS1='# ' PS2='> ' PS4='+ ' PWD='/' SHELL='/bin/bash' SHLVL='2' SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.xirTGdOv/pbuilderrc_sjg6 --distribution trixie --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/trixie-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.xirTGdOv/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc' SUDO_GID='109' SUDO_UID='104' SUDO_USER='jenkins' TERM='unknown' TZ='/usr/share/zoneinfo/Etc/GMT+12' USER='root' _='/usr/bin/systemd-run' http_proxy='http://192.168.101.4:3128' I: uname -a Linux codethink03-arm64 6.1.0-31-cloud-arm64 #1 SMP Debian 6.1.128-1 (2025-02-07) aarch64 GNU/Linux I: ls -l /bin lrwxrwxrwx 1 root root 7 Nov 22 2024 /bin -> usr/bin I: user script /srv/workspace/pbuilder/674357/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: arm64 Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 19916 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however: Package debhelper-compat is not installed. pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however: Package dh-sequence-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however: Package golang-github-stretchr-testify-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} binutils-gold{a} binutils-gold-aarch64-linux-gnu{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.24-go{a} golang-1.24-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-prometheus-sigv4-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libunistring5{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 0 packages upgraded, 109 newly installed, 0 to remove and 0 not upgraded. Need to get 128 MB of archives. After unpacking 715 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian trixie/main arm64 sensible-utils all 0.0.24 [24.8 kB] Get: 2 http://deb.debian.org/debian trixie/main arm64 libmagic-mgc arm64 1:5.45-3+b1 [314 kB] Get: 3 http://deb.debian.org/debian trixie/main arm64 libmagic1t64 arm64 1:5.45-3+b1 [102 kB] Get: 4 http://deb.debian.org/debian trixie/main arm64 file arm64 1:5.45-3+b1 [43.4 kB] Get: 5 http://deb.debian.org/debian trixie/main arm64 gettext-base arm64 0.23.1-1 [241 kB] Get: 6 http://deb.debian.org/debian trixie/main arm64 libuchardet0 arm64 0.0.8-1+b2 [69.2 kB] Get: 7 http://deb.debian.org/debian trixie/main arm64 groff-base arm64 1.23.0-7 [1129 kB] Get: 8 http://deb.debian.org/debian trixie/main arm64 bsdextrautils arm64 2.40.4-4 [92.1 kB] Get: 9 http://deb.debian.org/debian trixie/main arm64 libpipeline1 arm64 1.5.8-1 [40.2 kB] Get: 10 http://deb.debian.org/debian trixie/main arm64 man-db arm64 2.13.0-1 [1404 kB] Get: 11 http://deb.debian.org/debian trixie/main arm64 m4 arm64 1.4.19-5 [284 kB] Get: 12 http://deb.debian.org/debian trixie/main arm64 autoconf all 2.72-3 [493 kB] Get: 13 http://deb.debian.org/debian trixie/main arm64 autotools-dev all 20220109.1 [51.6 kB] Get: 14 http://deb.debian.org/debian trixie/main arm64 automake all 1:1.17-3 [862 kB] Get: 15 http://deb.debian.org/debian trixie/main arm64 autopoint all 0.23.1-1 [770 kB] Get: 16 http://deb.debian.org/debian trixie/main arm64 binutils-gold-aarch64-linux-gnu arm64 2.44-1 [1704 kB] Get: 17 http://deb.debian.org/debian trixie/main arm64 binutils-gold arm64 2.44-1 [202 kB] Get: 18 http://deb.debian.org/debian trixie/main arm64 libdebhelper-perl all 13.24.1 [90.9 kB] Get: 19 http://deb.debian.org/debian trixie/main arm64 libtool all 2.5.4-3 [539 kB] Get: 20 http://deb.debian.org/debian trixie/main arm64 dh-autoreconf all 20 [17.1 kB] Get: 21 http://deb.debian.org/debian trixie/main arm64 libarchive-zip-perl all 1.68-1 [104 kB] Get: 22 http://deb.debian.org/debian trixie/main arm64 libfile-stripnondeterminism-perl all 1.14.1-2 [19.7 kB] Get: 23 http://deb.debian.org/debian trixie/main arm64 dh-strip-nondeterminism all 1.14.1-2 [8620 B] Get: 24 http://deb.debian.org/debian trixie/main arm64 libelf1t64 arm64 0.192-4 [189 kB] Get: 25 http://deb.debian.org/debian trixie/main arm64 dwz arm64 0.15-1+b1 [102 kB] Get: 26 http://deb.debian.org/debian trixie/main arm64 libunistring5 arm64 1.3-1 [449 kB] Get: 27 http://deb.debian.org/debian trixie/main arm64 libicu72 arm64 72.1-6 [9239 kB] Get: 28 http://deb.debian.org/debian trixie/main arm64 libxml2 arm64 2.12.7+dfsg+really2.9.14-0.2+b1 [630 kB] Get: 29 http://deb.debian.org/debian trixie/main arm64 gettext arm64 0.23.1-1 [1610 kB] Get: 30 http://deb.debian.org/debian trixie/main arm64 intltool-debian all 0.35.0+20060710.6 [22.9 kB] Get: 31 http://deb.debian.org/debian trixie/main arm64 po-debconf all 1.0.21+nmu1 [248 kB] Get: 32 http://deb.debian.org/debian trixie/main arm64 debhelper all 13.24.1 [920 kB] Get: 33 http://deb.debian.org/debian trixie/main arm64 dh-golang all 1.63 [24.4 kB] Get: 34 http://deb.debian.org/debian trixie/main arm64 golang-1.24-src all 1.24.0-1 [20.3 MB] Get: 35 http://deb.debian.org/debian trixie/main arm64 golang-1.24-go arm64 1.24.0-1 [24.6 MB] Get: 36 http://deb.debian.org/debian trixie/main arm64 golang-src all 2:1.24~2 [5136 B] Get: 37 http://deb.debian.org/debian trixie/main arm64 golang-go arm64 2:1.24~2 [44.3 kB] Get: 38 http://deb.debian.org/debian trixie/main arm64 golang-any arm64 2:1.24~2 [5216 B] Get: 39 http://deb.debian.org/debian trixie/main arm64 golang-github-alecthomas-units-dev all 0.0~git20240626.61d1e34-1 [9032 B] Get: 40 http://deb.debian.org/debian trixie/main arm64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 41 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 42 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-sys-dev all 0.22.0-1 [431 kB] Get: 43 http://deb.debian.org/debian trixie/main arm64 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B] Get: 44 http://deb.debian.org/debian trixie/main arm64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB] Get: 45 http://deb.debian.org/debian trixie/main arm64 golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB] Get: 46 http://deb.debian.org/debian trixie/main arm64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB] Get: 47 http://deb.debian.org/debian trixie/main arm64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 48 http://deb.debian.org/debian trixie/main arm64 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB] Get: 49 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB] Get: 50 http://deb.debian.org/debian trixie/main arm64 golang-github-stretchr-testify-dev all 1.9.0-1 [74.7 kB] Get: 51 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 52 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB] Get: 53 http://deb.debian.org/debian trixie/main arm64 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB] Get: 54 http://deb.debian.org/debian trixie/main arm64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B] Get: 55 http://deb.debian.org/debian trixie/main arm64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB] Get: 56 http://deb.debian.org/debian trixie/main arm64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 57 http://deb.debian.org/debian trixie/main arm64 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB] Get: 58 http://deb.debian.org/debian trixie/main arm64 golang-google-protobuf-dev all 1.33.0-1 [625 kB] Get: 59 http://deb.debian.org/debian trixie/main arm64 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB] Get: 60 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B] Get: 61 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 62 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB] Get: 63 http://deb.debian.org/debian trixie/main arm64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB] Get: 64 http://deb.debian.org/debian trixie/main arm64 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB] Get: 65 http://deb.debian.org/debian trixie/main arm64 golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB] Get: 66 http://deb.debian.org/debian trixie/main arm64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 67 http://deb.debian.org/debian trixie/main arm64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB] Get: 68 http://deb.debian.org/debian trixie/main arm64 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB] Get: 69 http://deb.debian.org/debian trixie/main arm64 golang-github-klauspost-compress-dev all 1.17.11+ds1-1 [33.3 MB] Get: 70 http://deb.debian.org/debian trixie/main arm64 golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB] Get: 71 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB] Get: 72 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB] Get: 73 http://deb.debian.org/debian trixie/main arm64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB] Get: 74 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-term-dev all 0.22.0-1 [14.8 kB] Get: 75 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-text-dev all 0.22.0-1 [3995 kB] Get: 76 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-crypto-dev all 1:0.25.0-1 [1682 kB] Get: 77 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-net-dev all 1:0.27.0-1 [898 kB] Get: 78 http://deb.debian.org/debian trixie/main arm64 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB] Get: 79 http://deb.debian.org/debian trixie/main arm64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 80 http://deb.debian.org/debian trixie/main arm64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB] Get: 81 http://deb.debian.org/debian trixie/main arm64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB] Get: 82 http://deb.debian.org/debian trixie/main arm64 golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B] Get: 83 http://deb.debian.org/debian trixie/main arm64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 84 http://deb.debian.org/debian trixie/main arm64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB] Get: 85 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-sigv4-dev all 0.1.2-2 [8484 B] Get: 86 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-oauth2-dev all 0.26.0-1 [37.0 kB] Get: 87 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB] Get: 88 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-common-dev all 0.62.0-3 [118 kB] Get: 89 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB] Get: 90 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-procfs-dev all 0.15.1-1 [184 kB] Get: 91 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-client-golang-dev all 1.21.0-1 [195 kB] Get: 92 http://deb.debian.org/debian trixie/main arm64 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB] Get: 93 http://deb.debian.org/debian trixie/main arm64 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB] Get: 94 http://deb.debian.org/debian trixie/main arm64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B] Get: 95 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 96 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 97 http://deb.debian.org/debian trixie/main arm64 libsasl2-modules-db arm64 2.1.28+dfsg1-8+b1 [20.3 kB] Get: 98 http://deb.debian.org/debian trixie/main arm64 libsasl2-2 arm64 2.1.28+dfsg1-8+b1 [55.7 kB] Get: 99 http://deb.debian.org/debian trixie/main arm64 libssl-dev arm64 3.4.1-1 [3232 kB] Get: 100 http://deb.debian.org/debian trixie/main arm64 libsasl2-dev arm64 2.1.28+dfsg1-8+b1 [230 kB] Get: 101 http://deb.debian.org/debian trixie/main arm64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB] Get: 102 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-mod-dev all 0.19.0-1 [104 kB] Get: 103 http://deb.debian.org/debian trixie/main arm64 golang-github-yuin-goldmark-dev all 1.7.4-1 [113 kB] Get: 104 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-tools-dev all 1:0.23.0+ds-1 [1593 kB] Get: 105 http://deb.debian.org/debian trixie/main arm64 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB] Get: 106 http://deb.debian.org/debian trixie/main arm64 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB] Get: 107 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 108 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 109 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 128 MB in 1s (253 MB/s) Preconfiguring packages ... Selecting previously unselected package sensible-utils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19916 files and directories currently installed.) Preparing to unpack .../000-sensible-utils_0.0.24_all.deb ... Unpacking sensible-utils (0.0.24) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3+b1_arm64.deb ... Unpacking libmagic-mgc (1:5.45-3+b1) ... Selecting previously unselected package libmagic1t64:arm64. Preparing to unpack .../002-libmagic1t64_1%3a5.45-3+b1_arm64.deb ... Unpacking libmagic1t64:arm64 (1:5.45-3+b1) ... Selecting previously unselected package file. Preparing to unpack .../003-file_1%3a5.45-3+b1_arm64.deb ... Unpacking file (1:5.45-3+b1) ... Selecting previously unselected package gettext-base. Preparing to unpack .../004-gettext-base_0.23.1-1_arm64.deb ... Unpacking gettext-base (0.23.1-1) ... Selecting previously unselected package libuchardet0:arm64. Preparing to unpack .../005-libuchardet0_0.0.8-1+b2_arm64.deb ... Unpacking libuchardet0:arm64 (0.0.8-1+b2) ... Selecting previously unselected package groff-base. Preparing to unpack .../006-groff-base_1.23.0-7_arm64.deb ... Unpacking groff-base (1.23.0-7) ... Selecting previously unselected package bsdextrautils. Preparing to unpack .../007-bsdextrautils_2.40.4-4_arm64.deb ... Unpacking bsdextrautils (2.40.4-4) ... Selecting previously unselected package libpipeline1:arm64. Preparing to unpack .../008-libpipeline1_1.5.8-1_arm64.deb ... Unpacking libpipeline1:arm64 (1.5.8-1) ... Selecting previously unselected package man-db. Preparing to unpack .../009-man-db_2.13.0-1_arm64.deb ... Unpacking man-db (2.13.0-1) ... Selecting previously unselected package m4. Preparing to unpack .../010-m4_1.4.19-5_arm64.deb ... Unpacking m4 (1.4.19-5) ... Selecting previously unselected package autoconf. Preparing to unpack .../011-autoconf_2.72-3_all.deb ... Unpacking autoconf (2.72-3) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ... Unpacking autotools-dev (20220109.1) ... Selecting previously unselected package automake. Preparing to unpack .../013-automake_1%3a1.17-3_all.deb ... Unpacking automake (1:1.17-3) ... Selecting previously unselected package autopoint. Preparing to unpack .../014-autopoint_0.23.1-1_all.deb ... Unpacking autopoint (0.23.1-1) ... Selecting previously unselected package binutils-gold-aarch64-linux-gnu. Preparing to unpack .../015-binutils-gold-aarch64-linux-gnu_2.44-1_arm64.deb ... Unpacking binutils-gold-aarch64-linux-gnu (2.44-1) ... Selecting previously unselected package binutils-gold. Preparing to unpack .../016-binutils-gold_2.44-1_arm64.deb ... Unpacking binutils-gold (2.44-1) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../017-libdebhelper-perl_13.24.1_all.deb ... Unpacking libdebhelper-perl (13.24.1) ... Selecting previously unselected package libtool. Preparing to unpack .../018-libtool_2.5.4-3_all.deb ... Unpacking libtool (2.5.4-3) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../019-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../020-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../021-libfile-stripnondeterminism-perl_1.14.1-2_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.14.1-2) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../022-dh-strip-nondeterminism_1.14.1-2_all.deb ... Unpacking dh-strip-nondeterminism (1.14.1-2) ... Selecting previously unselected package libelf1t64:arm64. Preparing to unpack .../023-libelf1t64_0.192-4_arm64.deb ... Unpacking libelf1t64:arm64 (0.192-4) ... Selecting previously unselected package dwz. Preparing to unpack .../024-dwz_0.15-1+b1_arm64.deb ... Unpacking dwz (0.15-1+b1) ... Selecting previously unselected package libunistring5:arm64. Preparing to unpack .../025-libunistring5_1.3-1_arm64.deb ... Unpacking libunistring5:arm64 (1.3-1) ... Selecting previously unselected package libicu72:arm64. Preparing to unpack .../026-libicu72_72.1-6_arm64.deb ... Unpacking libicu72:arm64 (72.1-6) ... Selecting previously unselected package libxml2:arm64. Preparing to unpack .../027-libxml2_2.12.7+dfsg+really2.9.14-0.2+b1_arm64.deb ... Unpacking libxml2:arm64 (2.12.7+dfsg+really2.9.14-0.2+b1) ... Selecting previously unselected package gettext. Preparing to unpack .../028-gettext_0.23.1-1_arm64.deb ... Unpacking gettext (0.23.1-1) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../029-intltool-debian_0.35.0+20060710.6_all.deb ... Unpacking intltool-debian (0.35.0+20060710.6) ... Selecting previously unselected package po-debconf. Preparing to unpack .../030-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../031-debhelper_13.24.1_all.deb ... Unpacking debhelper (13.24.1) ... Selecting previously unselected package dh-golang. Preparing to unpack .../032-dh-golang_1.63_all.deb ... Unpacking dh-golang (1.63) ... Selecting previously unselected package golang-1.24-src. Preparing to unpack .../033-golang-1.24-src_1.24.0-1_all.deb ... Unpacking golang-1.24-src (1.24.0-1) ... Selecting previously unselected package golang-1.24-go. Preparing to unpack .../034-golang-1.24-go_1.24.0-1_arm64.deb ... Unpacking golang-1.24-go (1.24.0-1) ... Selecting previously unselected package golang-src. Preparing to unpack .../035-golang-src_2%3a1.24~2_all.deb ... Unpacking golang-src (2:1.24~2) ... Selecting previously unselected package golang-go:arm64. Preparing to unpack .../036-golang-go_2%3a1.24~2_arm64.deb ... Unpacking golang-go:arm64 (2:1.24~2) ... Selecting previously unselected package golang-any:arm64. Preparing to unpack .../037-golang-any_2%3a1.24~2_arm64.deb ... Unpacking golang-any:arm64 (2:1.24~2) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../038-golang-github-alecthomas-units-dev_0.0~git20240626.61d1e34-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../039-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../040-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../041-golang-golang-x-sys-dev_0.22.0-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.22.0-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../042-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../043-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../044-golang-github-fatih-color-dev_1.17.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.17.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../045-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../046-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../047-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../048-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../049-golang-github-stretchr-testify-dev_1.9.0-1_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.9.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../050-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../051-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../052-golang-github-pkg-errors-dev_0.9.1-3_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-3) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../053-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../054-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../055-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-google-go-cmp-dev. Preparing to unpack .../056-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ... Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ... Selecting previously unselected package golang-google-protobuf-dev. Preparing to unpack .../057-golang-google-protobuf-dev_1.33.0-1_all.deb ... Unpacking golang-google-protobuf-dev (1.33.0-1) ... Selecting previously unselected package golang-github-golang-protobuf-1-5-dev. Preparing to unpack .../058-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ... Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../059-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../060-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../061-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../062-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../063-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../064-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../065-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../066-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../067-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ... Selecting previously unselected package golang-github-klauspost-compress-dev. Preparing to unpack .../068-golang-github-klauspost-compress-dev_1.17.11+ds1-1_all.deb ... Unpacking golang-github-klauspost-compress-dev (1.17.11+ds1-1) ... Selecting previously unselected package golang-github-kylelemons-godebug-dev. Preparing to unpack .../069-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ... Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../070-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../071-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ... Selecting previously unselected package golang-github-jmespath-go-jmespath-dev. Preparing to unpack .../072-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ... Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Selecting previously unselected package golang-golang-x-term-dev. Preparing to unpack .../073-golang-golang-x-term-dev_0.22.0-1_all.deb ... Unpacking golang-golang-x-term-dev (0.22.0-1) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../074-golang-golang-x-text-dev_0.22.0-1_all.deb ... Unpacking golang-golang-x-text-dev (0.22.0-1) ... Selecting previously unselected package golang-golang-x-crypto-dev. Preparing to unpack .../075-golang-golang-x-crypto-dev_1%3a0.25.0-1_all.deb ... Unpacking golang-golang-x-crypto-dev (1:0.25.0-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../076-golang-golang-x-net-dev_1%3a0.27.0-1_all.deb ... Unpacking golang-golang-x-net-dev (1:0.27.0-1) ... Selecting previously unselected package golang-github-aws-aws-sdk-go-dev. Preparing to unpack .../077-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ... Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../078-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-kit-log-dev. Preparing to unpack .../079-golang-github-go-kit-log-dev_0.2.1-1_all.deb ... Unpacking golang-github-go-kit-log-dev (0.2.1-1) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../080-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Selecting previously unselected package golang-github-munnerz-goautoneg-dev. Preparing to unpack .../081-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ... Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../082-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../083-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Selecting previously unselected package golang-github-prometheus-sigv4-dev. Preparing to unpack .../084-golang-github-prometheus-sigv4-dev_0.1.2-2_all.deb ... Unpacking golang-github-prometheus-sigv4-dev (0.1.2-2) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../085-golang-golang-x-oauth2-dev_0.26.0-1_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.26.0-1) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../086-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../087-golang-github-prometheus-common-dev_0.62.0-3_all.deb ... Unpacking golang-github-prometheus-common-dev (0.62.0-3) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../088-golang-golang-x-sync-dev_0.7.0-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.7.0-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../089-golang-github-prometheus-procfs-dev_0.15.1-1_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.15.1-1) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../090-golang-github-prometheus-client-golang-dev_1.21.0-1_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.21.0-1) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../091-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../092-golang-github-creack-pty-dev_1.1.21-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.21-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../093-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../094-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../095-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:arm64. Preparing to unpack .../096-libsasl2-modules-db_2.1.28+dfsg1-8+b1_arm64.deb ... Unpacking libsasl2-modules-db:arm64 (2.1.28+dfsg1-8+b1) ... Selecting previously unselected package libsasl2-2:arm64. Preparing to unpack .../097-libsasl2-2_2.1.28+dfsg1-8+b1_arm64.deb ... Unpacking libsasl2-2:arm64 (2.1.28+dfsg1-8+b1) ... Selecting previously unselected package libssl-dev:arm64. Preparing to unpack .../098-libssl-dev_3.4.1-1_arm64.deb ... Unpacking libssl-dev:arm64 (3.4.1-1) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../099-libsasl2-dev_2.1.28+dfsg1-8+b1_arm64.deb ... Unpacking libsasl2-dev (2.1.28+dfsg1-8+b1) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../100-golang-github-kr-text-dev_0.2.0-2_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-2) ... Selecting previously unselected package golang-golang-x-mod-dev. Preparing to unpack .../101-golang-golang-x-mod-dev_0.19.0-1_all.deb ... Unpacking golang-golang-x-mod-dev (0.19.0-1) ... Selecting previously unselected package golang-github-yuin-goldmark-dev. Preparing to unpack .../102-golang-github-yuin-goldmark-dev_1.7.4-1_all.deb ... Unpacking golang-github-yuin-goldmark-dev (1.7.4-1) ... Selecting previously unselected package golang-golang-x-tools-dev. Preparing to unpack .../103-golang-golang-x-tools-dev_1%3a0.23.0+ds-1_all.deb ... Unpacking golang-golang-x-tools-dev (1:0.23.0+ds-1) ... Selecting previously unselected package golang-github-rogpeppe-go-internal-dev. Preparing to unpack .../104-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ... Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../105-golang-github-kr-pretty-dev_0.3.1-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.3.1-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../106-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../107-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../108-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up golang-1.24-src (1.24.0-1) ... Setting up libpipeline1:arm64 (1.5.8-1) ... Setting up golang-github-google-go-cmp-dev (0.6.0-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Setting up libicu72:arm64 (72.1-6) ... Setting up bsdextrautils (2.40.4-4) ... Setting up golang-github-creack-pty-dev (1.1.21-1) ... Setting up golang-github-stretchr-objx-dev (0.5.2-1) ... Setting up libmagic-mgc (1:5.45-3+b1) ... Setting up golang-github-pkg-errors-dev (0.9.1-3) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ... Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ... Setting up libdebhelper-perl (13.24.1) ... Setting up golang-golang-x-sys-dev (0.22.0-1) ... Setting up libmagic1t64:arm64 (1:5.45-3+b1) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.23.1-1) ... Setting up m4 (1.4.19-5) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up binutils-gold-aarch64-linux-gnu (2.44-1) ... Setting up golang-golang-x-term-dev (0.22.0-1) ... Setting up file (1:5.45-3+b1) ... Setting up golang-github-beorn7-perks-dev (1.0.1-1) ... Setting up libelf1t64:arm64 (0.192-4) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:arm64 (2.1.28+dfsg1-8+b1) ... Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ... Setting up autotools-dev (20220109.1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-google-protobuf-dev (1.33.0-1) ... Setting up golang-github-kr-text-dev (0.2.0-2) ... Setting up golang-github-stretchr-testify-dev (1.9.0-1) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Setting up golang-github-yuin-goldmark-dev (1.7.4-1) ... Setting up libunistring5:arm64 (1.3-1) ... Setting up libssl-dev:arm64 (3.4.1-1) ... Setting up golang-golang-x-sync-dev (0.7.0-1) ... Setting up golang-golang-x-mod-dev (0.19.0-1) ... Setting up autopoint (0.23.1-1) ... Setting up golang-github-klauspost-compress-dev (1.17.11+ds1-1) ... Setting up libsasl2-2:arm64 (2.1.28+dfsg1-8+b1) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Setting up autoconf (2.72-3) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Setting up dwz (0.15-1+b1) ... Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ... Setting up sensible-utils (0.0.24) ... Setting up golang-golang-x-text-dev (0.22.0-1) ... Setting up libuchardet0:arm64 (0.0.8-1+b2) ... Setting up golang-github-prometheus-procfs-dev (0.15.1-1) ... Setting up golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ... Setting up golang-golang-x-crypto-dev (1:0.25.0-1) ... Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-golang-x-oauth2-dev (0.26.0-1) ... Setting up golang-src (2:1.24~2) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Setting up libxml2:arm64 (2.12.7+dfsg+really2.9.14-0.2+b1) ... Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Setting up automake (1:1.17-3) ... update-alternatives: using /usr/bin/automake-1.17 to provide /usr/bin/automake (automake) in auto mode Setting up libfile-stripnondeterminism-perl (1.14.1-2) ... Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ... Setting up golang-github-go-kit-log-dev (0.2.1-1) ... Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ... Setting up gettext (0.23.1-1) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Setting up libtool (2.5.4-3) ... Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ... Setting up binutils-gold (2.44-1) ... Setting up golang-github-json-iterator-go-dev (1.1.12-2) ... Setting up intltool-debian (0.35.0+20060710.6) ... Setting up dh-autoreconf (20) ... Setting up libsasl2-dev (2.1.28+dfsg1-8+b1) ... Setting up golang-1.24-go (1.24.0-1) ... Setting up golang-golang-x-net-dev (1:0.27.0-1) ... Setting up dh-strip-nondeterminism (1.14.1-2) ... Setting up groff-base (1.23.0-7) ... Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ... Setting up golang-github-fatih-color-dev (1.17.0-1) ... Setting up golang-go:arm64 (2:1.24~2) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up golang-golang-x-tools-dev (1:0.23.0+ds-1) ... Setting up man-db (2.13.0-1) ... Not building database; man-db/auto-update is not 'true'. Setting up golang-any:arm64 (2:1.24~2) ... Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up golang-github-kr-pretty-dev (0.3.1-1) ... Setting up debhelper (13.24.1) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Setting up dh-golang (1.63) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Setting up golang-github-prometheus-sigv4-dev (0.1.2-2) ... Setting up golang-github-prometheus-common-dev (0.62.0-3) ... Setting up golang-github-prometheus-client-golang-dev (1.21.0-1) ... Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ... Processing triggers for libc-bin (2.40-7) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps I: Building the package I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.5.0-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.5.0-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Daniel Swarbrick dpkg-source --before-build . dpkg-buildpackage: info: host architecture arm64 debian/rules clean dh clean --buildsystem=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules binary dh binary --buildsystem=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-aarch64-linux-gnu && go install -trimpath -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/goarch internal/asan internal/unsafeheader internal/profilerecord internal/msan internal/coverage/rtcov internal/godebugs internal/goexperiment internal/byteorder internal/goos internal/runtime/syscall internal/runtime/math internal/abi sync/atomic internal/cpu math/bits unicode unicode/utf8 container/list internal/itoa internal/chacha8rand crypto/internal/fips140/alias crypto/internal/fips140deps/byteorder cmp crypto/internal/fips140/subtle crypto/internal/boring/sig encoding unicode/utf16 log/internal internal/nettrace vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/alias math internal/bytealg crypto/internal/fips140deps/cpu internal/runtime/sys internal/runtime/atomic internal/stringslite internal/runtime/exithook internal/race internal/sync internal/runtime/maps runtime iter internal/reflectlite crypto/subtle weak sync maps slices internal/bisect internal/singleflight internal/testlog unique runtime/cgo errors sort io path internal/oserror github.com/hashicorp/golang-lru/simplelru internal/godebug vendor/golang.org/x/net/dns/dnsmessage math/rand/v2 strconv hash bytes strings syscall crypto/internal/randutil internal/saferio hash/crc32 hash/crc64 crypto/internal/fips140deps/godebug math/rand crypto reflect net/netip crypto/internal/fips140 net/http/internal/ascii regexp/syntax crypto/internal/impl html bufio github.com/hashicorp/go-immutable-radix vendor/golang.org/x/text/transform crypto/internal/fips140/sha256 crypto/internal/fips140/sha3 crypto/internal/fips140/sha512 crypto/tls/internal/fips140tls crypto/sha3 crypto/internal/fips140hash crypto/internal/fips140/hmac crypto/internal/fips140/check crypto/internal/fips140/aes crypto/internal/fips140/nistec/fiat crypto/internal/fips140/tls12 crypto/internal/fips140/hkdf crypto/internal/fips140/edwards25519/field crypto/internal/fips140/bigmod regexp crypto/internal/fips140/tls13 crypto/internal/fips140/edwards25519 time internal/syscall/unix internal/syscall/execenv crypto/internal/fips140/nistec context io/fs internal/poll internal/filepathlite internal/fmtsort encoding/binary os encoding/base64 vendor/golang.org/x/crypto/internal/poly1305 golang.org/x/sys/unix encoding/pem crypto/internal/sysrand fmt io/ioutil internal/sysinfo os/signal path/filepath net crypto/internal/entropy crypto/internal/fips140/drbg crypto/internal/fips140only crypto/internal/fips140/aes/gcm crypto/internal/fips140/ecdh crypto/internal/fips140/ecdsa crypto/internal/fips140/ed25519 crypto/internal/fips140/rsa crypto/internal/fips140/mlkem crypto/md5 crypto/rc4 crypto/cipher math/big log vendor/golang.org/x/text/unicode/norm encoding/json encoding/hex mime crypto/internal/boring net/url vendor/golang.org/x/net/http2/hpack compress/flate crypto/aes crypto/des crypto/ecdh crypto/sha512 crypto/hmac vendor/golang.org/x/crypto/chacha20 crypto/sha1 crypto/sha256 vendor/golang.org/x/text/unicode/bidi mime/quotedprintable net/http/internal encoding/gob go/token text/template/parse compress/gzip vendor/golang.org/x/crypto/chacha20poly1305 flag runtime/debug runtime/trace vendor/golang.org/x/text/secure/bidirule vendor/golang.org/x/net/idna github.com/mattn/go-isatty testing github.com/mattn/go-colorable text/template github.com/fatih/color crypto/rand encoding/asn1 crypto/dsa crypto/elliptic crypto/internal/boring/bbig github.com/hashicorp/go-hclog crypto/ed25519 crypto/internal/hpke crypto/rsa vendor/golang.org/x/crypto/cryptobyte crypto/x509/pkix html/template crypto/ecdsa vendor/golang.org/x/net/http/httpproxy net/textproto crypto/x509 vendor/golang.org/x/net/http/httpguts mime/multipart crypto/tls net/http/httptrace net/http net/rpc github.com/armon/go-metrics github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench debian/rules override_dh_auto_test make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' dh_auto_test cd obj-aarch64-linux-gnu && go test -vet=off -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir 2026-03-31T06:16:11.730Z [INFO] creating new snapshot: path=/tmp/raft3140976653/raft2138142903/snapshots/3-10-1774937771730.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s) === RUN TestFileSS_CreateSnapshot 2026-03-31T06:16:11.731Z [INFO] creating new snapshot: path=/tmp/raft85908770/snapshots/3-10-1774937771731.tmp 2026-03-31T06:16:11.731Z [WARN] found temporary snapshot: name=3-10-1774937771731.tmp --- PASS: TestFileSS_CreateSnapshot (0.00s) === RUN TestFileSS_CancelSnapshot 2026-03-31T06:16:11.731Z [INFO] creating new snapshot: path=/tmp/raft4091427153/snapshots/3-10-1774937771731.tmp --- PASS: TestFileSS_CancelSnapshot (0.00s) === RUN TestFileSS_Retention 2026-03-31T06:16:11.732Z [INFO] creating new snapshot: path=/tmp/raft463726041/snapshots/3-10-1774937771732.tmp 2026-03-31T06:16:11.732Z [INFO] creating new snapshot: path=/tmp/raft463726041/snapshots/3-11-1774937771732.tmp 2026-03-31T06:16:11.732Z [INFO] creating new snapshot: path=/tmp/raft463726041/snapshots/3-12-1774937771732.tmp 2026-03-31T06:16:11.733Z [INFO] reaping snapshot: path=/tmp/raft463726041/snapshots/3-10-1774937771732 2026-03-31T06:16:11.733Z [INFO] creating new snapshot: path=/tmp/raft463726041/snapshots/3-13-1774937771733.tmp 2026-03-31T06:16:11.733Z [INFO] reaping snapshot: path=/tmp/raft463726041/snapshots/3-11-1774937771732 2026-03-31T06:16:11.733Z [INFO] creating new snapshot: path=/tmp/raft463726041/snapshots/3-14-1774937771733.tmp 2026-03-31T06:16:11.733Z [INFO] reaping snapshot: path=/tmp/raft463726041/snapshots/3-12-1774937771732 --- PASS: TestFileSS_Retention (0.00s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering 2026-03-31T06:16:11.734Z [INFO] creating new snapshot: path=/tmp/raft3556000333/snapshots/5-130350-1774937771734.tmp 2026-03-31T06:16:11.735Z [INFO] creating new snapshot: path=/tmp/raft3556000333/snapshots/36-204917-1774937771735.tmp --- PASS: TestFileSS_Ordering (0.00s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestRaft_RestartFollower_LongInitialHeartbeat --- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestLogCacheWithBackendStoreError --- PASS: TestLogCacheWithBackendStoreError (0.00s) === RUN TestOldestLog === RUN TestOldestLog/empty_logs === RUN TestOldestLog/simple_case --- PASS: TestOldestLog (0.00s) --- PASS: TestOldestLog/empty_logs (0.00s) --- PASS: TestOldestLog/simple_case (0.00s) === RUN TestEmitsLogStoreMetrics --- PASS: TestEmitsLogStoreMetrics (0.01s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.01s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams --- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.02s) === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 2026-03-31T06:16:11.836Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 2026-03-31T06:16:11.888Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 2026-03-31T06:16:11.940Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s) === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:839: countingWriter: 2026-03-31T06:16:11.948Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-03-31T06:16:11.953Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-03-31T06:16:11.964Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-03-31T06:16:11.985Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-03-31T06:16:12.025Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-03-31T06:16:12.106Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-03-31T06:16:12.266Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-03-31T06:16:12.587Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-03-31T06:16:13.228Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-03-31T06:16:14.231Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-03-31T06:16:15.233Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop 2026-03-31T06:16:15.949Z [INFO] server-5f839d57-c59f-7f1f-3c2d-7593330fd8a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f839d57-c59f-7f1f-3c2d-7593330fd8a1 Address:5f839d57-c59f-7f1f-3c2d-7593330fd8a1}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown 2026-03-31T06:16:15.950Z [INFO] server-b5e0161f-34a4-0e8d-bf56-a6652278082b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5e0161f-34a4-0e8d-bf56-a6652278082b Address:b5e0161f-34a4-0e8d-bf56-a6652278082b}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap 2026-03-31T06:16:15.950Z [INFO] server-5d49e3d8-4c1d-6c7b-5101-d234c85d6fb3: initial configuration: index=0 servers=[] 2026-03-31T06:16:15.950Z [INFO] server-a8e71736-85fc-d3fe-c4d3-3b4f10c9ee7f: initial configuration: index=0 servers=[] 2026-03-31T06:16:15.950Z [INFO] server-04907084-88b6-833a-049f-c616dac0f8d3: initial configuration: index=0 servers=[] 2026-03-31T06:16:15.950Z [INFO] server-5d49e3d8-4c1d-6c7b-5101-d234c85d6fb3: entering follower state: follower="Node at 5d49e3d8-4c1d-6c7b-5101-d234c85d6fb3 [Follower]" leader-address= leader-id= 2026-03-31T06:16:15.950Z [INFO] server-a8e71736-85fc-d3fe-c4d3-3b4f10c9ee7f: entering follower state: follower="Node at a8e71736-85fc-d3fe-c4d3-3b4f10c9ee7f [Follower]" leader-address= leader-id= 2026-03-31T06:16:15.950Z [INFO] server-04907084-88b6-833a-049f-c616dac0f8d3: entering follower state: follower="Node at 04907084-88b6-833a-049f-c616dac0f8d3 [Follower]" leader-address= leader-id= 2026-03-31T06:16:15.950Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:16.003Z [WARN] server-04907084-88b6-833a-049f-c616dac0f8d3: no known peers, aborting election 2026-03-31T06:16:16.008Z [WARN] server-5d49e3d8-4c1d-6c7b-5101-d234c85d6fb3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:16.008Z [INFO] server-5d49e3d8-4c1d-6c7b-5101-d234c85d6fb3: entering candidate state: node="Node at 5d49e3d8-4c1d-6c7b-5101-d234c85d6fb3 [Candidate]" term=2 2026-03-31T06:16:16.008Z [INFO] server-5d49e3d8-4c1d-6c7b-5101-d234c85d6fb3: election won: term=2 tally=2 2026-03-31T06:16:16.008Z [INFO] server-5d49e3d8-4c1d-6c7b-5101-d234c85d6fb3: entering leader state: leader="Node at 5d49e3d8-4c1d-6c7b-5101-d234c85d6fb3 [Leader]" 2026-03-31T06:16:16.008Z [INFO] server-5d49e3d8-4c1d-6c7b-5101-d234c85d6fb3: added peer, starting replication: peer=server-a8e71736-85fc-d3fe-c4d3-3b4f10c9ee7f 2026-03-31T06:16:16.008Z [INFO] server-5d49e3d8-4c1d-6c7b-5101-d234c85d6fb3: added peer, starting replication: peer=server-04907084-88b6-833a-049f-c616dac0f8d3 2026-03-31T06:16:16.008Z [WARN] server-04907084-88b6-833a-049f-c616dac0f8d3: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2026-03-31T06:16:16.008Z [WARN] server-5d49e3d8-4c1d-6c7b-5101-d234c85d6fb3: appendEntries rejected, sending older logs: peer="{Voter server-04907084-88b6-833a-049f-c616dac0f8d3 04907084-88b6-833a-049f-c616dac0f8d3}" next=1 2026-03-31T06:16:16.008Z [WARN] server-a8e71736-85fc-d3fe-c4d3-3b4f10c9ee7f: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2026-03-31T06:16:16.008Z [WARN] server-5d49e3d8-4c1d-6c7b-5101-d234c85d6fb3: appendEntries rejected, sending older logs: peer="{Voter server-a8e71736-85fc-d3fe-c4d3-3b4f10c9ee7f a8e71736-85fc-d3fe-c4d3-3b4f10c9ee7f}" next=1 2026-03-31T06:16:16.009Z [INFO] server-5d49e3d8-4c1d-6c7b-5101-d234c85d6fb3: pipelining replication: peer="{Voter server-04907084-88b6-833a-049f-c616dac0f8d3 04907084-88b6-833a-049f-c616dac0f8d3}" 2026-03-31T06:16:16.009Z [INFO] server-5d49e3d8-4c1d-6c7b-5101-d234c85d6fb3: pipelining replication: peer="{Voter server-a8e71736-85fc-d3fe-c4d3-3b4f10c9ee7f a8e71736-85fc-d3fe-c4d3-3b4f10c9ee7f}" 2026-03-31T06:16:16.113Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:16.00864786 +0000 UTC m=+4.286255241 (2 nodes), 57.80008ms from start of poll, 58.22624ms from cluster start. Timeout at 2026-03-31 06:16:16.1136497 +0000 UTC m=+4.391257121, 105.00188ms after stability 2026-03-31T06:16:16.114Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:16.219Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:16.11425886 +0000 UTC m=+4.391866241 (1 nodes), 920ns from start of poll, 163.83724ms from cluster start. Timeout at 2026-03-31 06:16:16.2192633 +0000 UTC m=+4.496870681, 105.00444ms after stability 2026-03-31T06:16:16.231Z [INFO] server-5d49e3d8-4c1d-6c7b-5101-d234c85d6fb3: aborting pipeline replication: peer="{Voter server-04907084-88b6-833a-049f-c616dac0f8d3 04907084-88b6-833a-049f-c616dac0f8d3}" 2026-03-31T06:16:16.231Z [INFO] server-5d49e3d8-4c1d-6c7b-5101-d234c85d6fb3: aborting pipeline replication: peer="{Voter server-a8e71736-85fc-d3fe-c4d3-3b4f10c9ee7f a8e71736-85fc-d3fe-c4d3-3b4f10c9ee7f}" --- PASS: TestRaft_LiveBootstrap (0.28s) === RUN TestRaft_LiveBootstrap_From_NonVoter 2026-03-31T06:16:16.231Z [INFO] server-f9e54233-cc69-420a-fc40-502fb2e6b344: initial configuration: index=0 servers=[] 2026-03-31T06:16:16.231Z [INFO] server-e13ba5b7-7243-8898-a04d-0f32510b4ab9: initial configuration: index=0 servers=[] 2026-03-31T06:16:16.231Z [INFO] server-f9e54233-cc69-420a-fc40-502fb2e6b344: entering follower state: follower="Node at f9e54233-cc69-420a-fc40-502fb2e6b344 [Follower]" leader-address= leader-id= 2026-03-31T06:16:16.232Z [INFO] server-e13ba5b7-7243-8898-a04d-0f32510b4ab9: entering follower state: follower="Node at e13ba5b7-7243-8898-a04d-0f32510b4ab9 [Follower]" leader-address= leader-id= --- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s) === RUN TestRaft_RecoverCluster_NoState 2026-03-31T06:16:16.232Z [INFO] server-dd601535-ef7b-40b8-61f3-adc489ce6c96: initial configuration: index=0 servers=[] --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster === RUN TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs 2026-03-31T06:16:16.233Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4 Address:0c7a4bd6-4359-4f7e-1acb-723e029d5df4} {Suffrage:Voter ID:server-06de34ac-9c9f-7038-9e62-39d784c0f039 Address:06de34ac-9c9f-7038-9e62-39d784c0f039} {Suffrage:Voter ID:server-55f8baf6-ef9b-9237-dbf4-58580ca5b245 Address:55f8baf6-ef9b-9237-dbf4-58580ca5b245}]" 2026-03-31T06:16:16.233Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: entering follower state: follower="Node at 0c7a4bd6-4359-4f7e-1acb-723e029d5df4 [Follower]" leader-address= leader-id= 2026-03-31T06:16:16.233Z [INFO] server-06de34ac-9c9f-7038-9e62-39d784c0f039: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4 Address:0c7a4bd6-4359-4f7e-1acb-723e029d5df4} {Suffrage:Voter ID:server-06de34ac-9c9f-7038-9e62-39d784c0f039 Address:06de34ac-9c9f-7038-9e62-39d784c0f039} {Suffrage:Voter ID:server-55f8baf6-ef9b-9237-dbf4-58580ca5b245 Address:55f8baf6-ef9b-9237-dbf4-58580ca5b245}]" 2026-03-31T06:16:16.233Z [INFO] server-06de34ac-9c9f-7038-9e62-39d784c0f039: entering follower state: follower="Node at 06de34ac-9c9f-7038-9e62-39d784c0f039 [Follower]" leader-address= leader-id= 2026-03-31T06:16:16.233Z [INFO] server-55f8baf6-ef9b-9237-dbf4-58580ca5b245: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4 Address:0c7a4bd6-4359-4f7e-1acb-723e029d5df4} {Suffrage:Voter ID:server-06de34ac-9c9f-7038-9e62-39d784c0f039 Address:06de34ac-9c9f-7038-9e62-39d784c0f039} {Suffrage:Voter ID:server-55f8baf6-ef9b-9237-dbf4-58580ca5b245 Address:55f8baf6-ef9b-9237-dbf4-58580ca5b245}]" 2026-03-31T06:16:16.233Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:16.233Z [INFO] server-55f8baf6-ef9b-9237-dbf4-58580ca5b245: entering follower state: follower="Node at 55f8baf6-ef9b-9237-dbf4-58580ca5b245 [Follower]" leader-address= leader-id= 2026-03-31T06:16:16.307Z [WARN] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:16.307Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: entering candidate state: node="Node at 0c7a4bd6-4359-4f7e-1acb-723e029d5df4 [Candidate]" term=2 2026-03-31T06:16:16.307Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: election won: term=2 tally=2 2026-03-31T06:16:16.307Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: entering leader state: leader="Node at 0c7a4bd6-4359-4f7e-1acb-723e029d5df4 [Leader]" 2026-03-31T06:16:16.307Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: added peer, starting replication: peer=server-06de34ac-9c9f-7038-9e62-39d784c0f039 2026-03-31T06:16:16.307Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: added peer, starting replication: peer=server-55f8baf6-ef9b-9237-dbf4-58580ca5b245 2026-03-31T06:16:16.307Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: pipelining replication: peer="{Voter server-55f8baf6-ef9b-9237-dbf4-58580ca5b245 55f8baf6-ef9b-9237-dbf4-58580ca5b245}" 2026-03-31T06:16:16.307Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: pipelining replication: peer="{Voter server-06de34ac-9c9f-7038-9e62-39d784c0f039 06de34ac-9c9f-7038-9e62-39d784c0f039}" 2026-03-31T06:16:16.413Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:16.30791278 +0000 UTC m=+4.585520161 (1 nodes), 74.3782ms from start of poll, 74.8186ms from cluster start. Timeout at 2026-03-31 06:16:16.41291862 +0000 UTC m=+4.690526041, 105.00588ms after stability 2026-03-31T06:16:16.413Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: aborting pipeline replication: peer="{Voter server-55f8baf6-ef9b-9237-dbf4-58580ca5b245 55f8baf6-ef9b-9237-dbf4-58580ca5b245}" 2026-03-31T06:16:16.413Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: aborting pipeline replication: peer="{Voter server-06de34ac-9c9f-7038-9e62-39d784c0f039 06de34ac-9c9f-7038-9e62-39d784c0f039}" 2026-03-31T06:16:16.413Z [INFO] creating new snapshot: path=/tmp/raft9000834/snapshots/2-2-1774937776413.tmp 2026-03-31T06:16:16.414Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: starting restore from snapshot: id=2-2-1774937776413 last-index=2 last-term=2 size-in-bytes=1 2026-03-31T06:16:16.414Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: snapshot restore progress: id=2-2-1774937776413 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2026-03-31T06:16:16.414Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: restored from snapshot: id=2-2-1774937776413 last-index=2 last-term=2 size-in-bytes=1 2026-03-31T06:16:16.414Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4 Address:0c7a4bd6-4359-4f7e-1acb-723e029d5df4} {Suffrage:Voter ID:server-06de34ac-9c9f-7038-9e62-39d784c0f039 Address:06de34ac-9c9f-7038-9e62-39d784c0f039} {Suffrage:Voter ID:server-55f8baf6-ef9b-9237-dbf4-58580ca5b245 Address:55f8baf6-ef9b-9237-dbf4-58580ca5b245}]" 2026-03-31T06:16:16.414Z [INFO] creating new snapshot: path=/tmp/raft2138275400/snapshots/2-2-1774937776414.tmp 2026-03-31T06:16:16.414Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: entering follower state: follower="Node at 0c7a4bd6-4359-4f7e-1acb-723e029d5df4 [Follower]" leader-address= leader-id= 2026-03-31T06:16:16.415Z [INFO] server-06de34ac-9c9f-7038-9e62-39d784c0f039: starting restore from snapshot: id=2-2-1774937776414 last-index=2 last-term=2 size-in-bytes=1 2026-03-31T06:16:16.415Z [INFO] server-06de34ac-9c9f-7038-9e62-39d784c0f039: snapshot restore progress: id=2-2-1774937776414 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2026-03-31T06:16:16.415Z [INFO] server-06de34ac-9c9f-7038-9e62-39d784c0f039: restored from snapshot: id=2-2-1774937776414 last-index=2 last-term=2 size-in-bytes=1 2026-03-31T06:16:16.415Z [INFO] server-06de34ac-9c9f-7038-9e62-39d784c0f039: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4 Address:0c7a4bd6-4359-4f7e-1acb-723e029d5df4} {Suffrage:Voter ID:server-06de34ac-9c9f-7038-9e62-39d784c0f039 Address:06de34ac-9c9f-7038-9e62-39d784c0f039} {Suffrage:Voter ID:server-55f8baf6-ef9b-9237-dbf4-58580ca5b245 Address:55f8baf6-ef9b-9237-dbf4-58580ca5b245}]" 2026-03-31T06:16:16.415Z [INFO] server-06de34ac-9c9f-7038-9e62-39d784c0f039: entering follower state: follower="Node at 06de34ac-9c9f-7038-9e62-39d784c0f039 [Follower]" leader-address= leader-id= 2026-03-31T06:16:16.415Z [INFO] creating new snapshot: path=/tmp/raft2579886401/snapshots/2-2-1774937776415.tmp 2026-03-31T06:16:16.415Z [INFO] server-55f8baf6-ef9b-9237-dbf4-58580ca5b245: starting restore from snapshot: id=2-2-1774937776415 last-index=2 last-term=2 size-in-bytes=1 2026-03-31T06:16:16.416Z [INFO] server-55f8baf6-ef9b-9237-dbf4-58580ca5b245: snapshot restore progress: id=2-2-1774937776415 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2026-03-31T06:16:16.416Z [INFO] server-55f8baf6-ef9b-9237-dbf4-58580ca5b245: restored from snapshot: id=2-2-1774937776415 last-index=2 last-term=2 size-in-bytes=1 2026-03-31T06:16:16.416Z [INFO] server-55f8baf6-ef9b-9237-dbf4-58580ca5b245: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4 Address:0c7a4bd6-4359-4f7e-1acb-723e029d5df4} {Suffrage:Voter ID:server-06de34ac-9c9f-7038-9e62-39d784c0f039 Address:06de34ac-9c9f-7038-9e62-39d784c0f039} {Suffrage:Voter ID:server-55f8baf6-ef9b-9237-dbf4-58580ca5b245 Address:55f8baf6-ef9b-9237-dbf4-58580ca5b245}]" 2026-03-31T06:16:16.416Z [INFO] server-55f8baf6-ef9b-9237-dbf4-58580ca5b245: entering follower state: follower="Node at 55f8baf6-ef9b-9237-dbf4-58580ca5b245 [Follower]" leader-address= leader-id= 2026-03-31T06:16:16.470Z [WARN] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:16.470Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: entering candidate state: node="Node at 0c7a4bd6-4359-4f7e-1acb-723e029d5df4 [Candidate]" term=3 2026-03-31T06:16:16.471Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: election won: term=3 tally=2 2026-03-31T06:16:16.471Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: entering leader state: leader="Node at 0c7a4bd6-4359-4f7e-1acb-723e029d5df4 [Leader]" 2026-03-31T06:16:16.471Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: added peer, starting replication: peer=server-06de34ac-9c9f-7038-9e62-39d784c0f039 2026-03-31T06:16:16.471Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: added peer, starting replication: peer=server-55f8baf6-ef9b-9237-dbf4-58580ca5b245 2026-03-31T06:16:16.471Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: pipelining replication: peer="{Voter server-55f8baf6-ef9b-9237-dbf4-58580ca5b245 55f8baf6-ef9b-9237-dbf4-58580ca5b245}" 2026-03-31T06:16:16.471Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: pipelining replication: peer="{Voter server-06de34ac-9c9f-7038-9e62-39d784c0f039 06de34ac-9c9f-7038-9e62-39d784c0f039}" 2026-03-31T06:16:16.732Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:16.837Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:16.73226362 +0000 UTC m=+5.009871001 (1 nodes), 38.92µs from start of poll, 499.16944ms from cluster start. Timeout at 2026-03-31 06:16:16.83727246 +0000 UTC m=+5.114879881, 105.00888ms after stability 2026-03-31T06:16:16.837Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: aborting pipeline replication: peer="{Voter server-55f8baf6-ef9b-9237-dbf4-58580ca5b245 55f8baf6-ef9b-9237-dbf4-58580ca5b245}" 2026-03-31T06:16:16.837Z [INFO] server-0c7a4bd6-4359-4f7e-1acb-723e029d5df4: aborting pipeline replication: peer="{Voter server-06de34ac-9c9f-7038-9e62-39d784c0f039 06de34ac-9c9f-7038-9e62-39d784c0f039}" === RUN TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs 2026-03-31T06:16:16.838Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff76cb51-d726-545c-4f69-56250cbe1509 Address:ff76cb51-d726-545c-4f69-56250cbe1509} {Suffrage:Voter ID:server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1 Address:5e99fa44-bf1f-a11a-e177-eed474f3a3d1} {Suffrage:Voter ID:server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7 Address:a4890974-a552-9dd5-5a2f-e5a2e18ac6d7}]" 2026-03-31T06:16:16.838Z [INFO] server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff76cb51-d726-545c-4f69-56250cbe1509 Address:ff76cb51-d726-545c-4f69-56250cbe1509} {Suffrage:Voter ID:server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1 Address:5e99fa44-bf1f-a11a-e177-eed474f3a3d1} {Suffrage:Voter ID:server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7 Address:a4890974-a552-9dd5-5a2f-e5a2e18ac6d7}]" 2026-03-31T06:16:16.838Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: entering follower state: follower="Node at ff76cb51-d726-545c-4f69-56250cbe1509 [Follower]" leader-address= leader-id= 2026-03-31T06:16:16.838Z [INFO] server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1: entering follower state: follower="Node at 5e99fa44-bf1f-a11a-e177-eed474f3a3d1 [Follower]" leader-address= leader-id= 2026-03-31T06:16:16.838Z [INFO] server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff76cb51-d726-545c-4f69-56250cbe1509 Address:ff76cb51-d726-545c-4f69-56250cbe1509} {Suffrage:Voter ID:server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1 Address:5e99fa44-bf1f-a11a-e177-eed474f3a3d1} {Suffrage:Voter ID:server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7 Address:a4890974-a552-9dd5-5a2f-e5a2e18ac6d7}]" 2026-03-31T06:16:16.838Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:16.838Z [INFO] server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7: entering follower state: follower="Node at a4890974-a552-9dd5-5a2f-e5a2e18ac6d7 [Follower]" leader-address= leader-id= 2026-03-31T06:16:16.898Z [WARN] server-ff76cb51-d726-545c-4f69-56250cbe1509: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:16.898Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: entering candidate state: node="Node at ff76cb51-d726-545c-4f69-56250cbe1509 [Candidate]" term=2 2026-03-31T06:16:16.898Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: election won: term=2 tally=2 2026-03-31T06:16:16.898Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: entering leader state: leader="Node at ff76cb51-d726-545c-4f69-56250cbe1509 [Leader]" 2026-03-31T06:16:16.898Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: added peer, starting replication: peer=server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1 2026-03-31T06:16:16.898Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: added peer, starting replication: peer=server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7 2026-03-31T06:16:16.898Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: pipelining replication: peer="{Voter server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7 a4890974-a552-9dd5-5a2f-e5a2e18ac6d7}" 2026-03-31T06:16:16.898Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: pipelining replication: peer="{Voter server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1 5e99fa44-bf1f-a11a-e177-eed474f3a3d1}" 2026-03-31T06:16:17.004Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:16.8987399 +0000 UTC m=+5.176347281 (1 nodes), 59.96684ms from start of poll, 60.3752ms from cluster start. Timeout at 2026-03-31 06:16:17.00374286 +0000 UTC m=+5.281350281, 105.003ms after stability 2026-03-31T06:16:17.005Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: aborting pipeline replication: peer="{Voter server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7 a4890974-a552-9dd5-5a2f-e5a2e18ac6d7}" 2026-03-31T06:16:17.005Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: aborting pipeline replication: peer="{Voter server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1 5e99fa44-bf1f-a11a-e177-eed474f3a3d1}" 2026-03-31T06:16:17.005Z [INFO] creating new snapshot: path=/tmp/raft1344959734/snapshots/2-6-1774937777005.tmp 2026-03-31T06:16:17.006Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: starting restore from snapshot: id=2-6-1774937777005 last-index=6 last-term=2 size-in-bytes=25 2026-03-31T06:16:17.006Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: snapshot restore progress: id=2-6-1774937777005 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2026-03-31T06:16:17.006Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: restored from snapshot: id=2-6-1774937777005 last-index=6 last-term=2 size-in-bytes=25 2026-03-31T06:16:17.006Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff76cb51-d726-545c-4f69-56250cbe1509 Address:ff76cb51-d726-545c-4f69-56250cbe1509} {Suffrage:Voter ID:server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1 Address:5e99fa44-bf1f-a11a-e177-eed474f3a3d1} {Suffrage:Voter ID:server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7 Address:a4890974-a552-9dd5-5a2f-e5a2e18ac6d7}]" 2026-03-31T06:16:17.006Z [INFO] creating new snapshot: path=/tmp/raft3239712273/snapshots/2-6-1774937777006.tmp 2026-03-31T06:16:17.006Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: entering follower state: follower="Node at ff76cb51-d726-545c-4f69-56250cbe1509 [Follower]" leader-address= leader-id= 2026-03-31T06:16:17.006Z [INFO] server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1: starting restore from snapshot: id=2-6-1774937777006 last-index=6 last-term=2 size-in-bytes=25 2026-03-31T06:16:17.007Z [INFO] server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1: snapshot restore progress: id=2-6-1774937777006 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2026-03-31T06:16:17.007Z [INFO] server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1: restored from snapshot: id=2-6-1774937777006 last-index=6 last-term=2 size-in-bytes=25 2026-03-31T06:16:17.007Z [INFO] server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff76cb51-d726-545c-4f69-56250cbe1509 Address:ff76cb51-d726-545c-4f69-56250cbe1509} {Suffrage:Voter ID:server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1 Address:5e99fa44-bf1f-a11a-e177-eed474f3a3d1} {Suffrage:Voter ID:server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7 Address:a4890974-a552-9dd5-5a2f-e5a2e18ac6d7}]" 2026-03-31T06:16:17.007Z [INFO] server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1: entering follower state: follower="Node at 5e99fa44-bf1f-a11a-e177-eed474f3a3d1 [Follower]" leader-address= leader-id= 2026-03-31T06:16:17.007Z [INFO] creating new snapshot: path=/tmp/raft3948468735/snapshots/2-6-1774937777007.tmp 2026-03-31T06:16:17.007Z [INFO] server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7: starting restore from snapshot: id=2-6-1774937777007 last-index=6 last-term=2 size-in-bytes=25 2026-03-31T06:16:17.007Z [INFO] server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7: snapshot restore progress: id=2-6-1774937777007 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2026-03-31T06:16:17.007Z [INFO] server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7: restored from snapshot: id=2-6-1774937777007 last-index=6 last-term=2 size-in-bytes=25 2026-03-31T06:16:17.007Z [INFO] server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff76cb51-d726-545c-4f69-56250cbe1509 Address:ff76cb51-d726-545c-4f69-56250cbe1509} {Suffrage:Voter ID:server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1 Address:5e99fa44-bf1f-a11a-e177-eed474f3a3d1} {Suffrage:Voter ID:server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7 Address:a4890974-a552-9dd5-5a2f-e5a2e18ac6d7}]" 2026-03-31T06:16:17.008Z [INFO] server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7: entering follower state: follower="Node at a4890974-a552-9dd5-5a2f-e5a2e18ac6d7 [Follower]" leader-address= leader-id= 2026-03-31T06:16:17.063Z [WARN] server-ff76cb51-d726-545c-4f69-56250cbe1509: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:17.063Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: entering candidate state: node="Node at ff76cb51-d726-545c-4f69-56250cbe1509 [Candidate]" term=3 2026-03-31T06:16:17.063Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: election won: term=3 tally=2 2026-03-31T06:16:17.063Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: entering leader state: leader="Node at ff76cb51-d726-545c-4f69-56250cbe1509 [Leader]" 2026-03-31T06:16:17.064Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: added peer, starting replication: peer=server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1 2026-03-31T06:16:17.064Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: added peer, starting replication: peer=server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7 2026-03-31T06:16:17.064Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: pipelining replication: peer="{Voter server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7 a4890974-a552-9dd5-5a2f-e5a2e18ac6d7}" 2026-03-31T06:16:17.064Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: pipelining replication: peer="{Voter server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1 5e99fa44-bf1f-a11a-e177-eed474f3a3d1}" 2026-03-31T06:16:17.323Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:17.429Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:17.32368618 +0000 UTC m=+5.601293561 (1 nodes), 34.36µs from start of poll, 485.32148ms from cluster start. Timeout at 2026-03-31 06:16:17.4286941 +0000 UTC m=+5.706301481, 105.00792ms after stability 2026-03-31T06:16:17.429Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: aborting pipeline replication: peer="{Voter server-a4890974-a552-9dd5-5a2f-e5a2e18ac6d7 a4890974-a552-9dd5-5a2f-e5a2e18ac6d7}" 2026-03-31T06:16:17.429Z [INFO] server-ff76cb51-d726-545c-4f69-56250cbe1509: aborting pipeline replication: peer="{Voter server-5e99fa44-bf1f-a11a-e177-eed474f3a3d1 5e99fa44-bf1f-a11a-e177-eed474f3a3d1}" === RUN TestRaft_RecoverCluster/snapshot,_with_trailing_logs 2026-03-31T06:16:17.430Z [INFO] server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c Address:ef1867ae-6e71-dd2e-b0ac-aa540f6df96c} {Suffrage:Voter ID:server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a Address:ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a} {Suffrage:Voter ID:server-137cbdce-bc9a-ea6b-bfda-a7be81de0367 Address:137cbdce-bc9a-ea6b-bfda-a7be81de0367}]" 2026-03-31T06:16:17.430Z [INFO] server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c Address:ef1867ae-6e71-dd2e-b0ac-aa540f6df96c} {Suffrage:Voter ID:server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a Address:ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a} {Suffrage:Voter ID:server-137cbdce-bc9a-ea6b-bfda-a7be81de0367 Address:137cbdce-bc9a-ea6b-bfda-a7be81de0367}]" 2026-03-31T06:16:17.430Z [INFO] server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c: entering follower state: follower="Node at ef1867ae-6e71-dd2e-b0ac-aa540f6df96c [Follower]" leader-address= leader-id= 2026-03-31T06:16:17.430Z [INFO] server-137cbdce-bc9a-ea6b-bfda-a7be81de0367: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c Address:ef1867ae-6e71-dd2e-b0ac-aa540f6df96c} {Suffrage:Voter ID:server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a Address:ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a} {Suffrage:Voter ID:server-137cbdce-bc9a-ea6b-bfda-a7be81de0367 Address:137cbdce-bc9a-ea6b-bfda-a7be81de0367}]" 2026-03-31T06:16:17.430Z [INFO] server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a: entering follower state: follower="Node at ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a [Follower]" leader-address= leader-id= 2026-03-31T06:16:17.430Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:17.430Z [INFO] server-137cbdce-bc9a-ea6b-bfda-a7be81de0367: entering follower state: follower="Node at 137cbdce-bc9a-ea6b-bfda-a7be81de0367 [Follower]" leader-address= leader-id= 2026-03-31T06:16:17.485Z [WARN] server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:17.486Z [INFO] server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a: entering candidate state: node="Node at ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a [Candidate]" term=2 2026-03-31T06:16:17.486Z [INFO] server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a: election won: term=2 tally=2 2026-03-31T06:16:17.486Z [INFO] server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a: entering leader state: leader="Node at ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a [Leader]" 2026-03-31T06:16:17.486Z [INFO] server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a: added peer, starting replication: peer=server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c 2026-03-31T06:16:17.486Z [INFO] server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a: added peer, starting replication: peer=server-137cbdce-bc9a-ea6b-bfda-a7be81de0367 2026-03-31T06:16:17.486Z [INFO] server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a: pipelining replication: peer="{Voter server-137cbdce-bc9a-ea6b-bfda-a7be81de0367 137cbdce-bc9a-ea6b-bfda-a7be81de0367}" 2026-03-31T06:16:17.486Z [INFO] server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a: pipelining replication: peer="{Voter server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c ef1867ae-6e71-dd2e-b0ac-aa540f6df96c}" 2026-03-31T06:16:17.591Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:17.48619346 +0000 UTC m=+5.763800841 (1 nodes), 55.43368ms from start of poll, 55.86948ms from cluster start. Timeout at 2026-03-31 06:16:17.59119518 +0000 UTC m=+5.868802561, 105.00172ms after stability 2026-03-31T06:16:17.594Z [INFO] server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a: aborting pipeline replication: peer="{Voter server-137cbdce-bc9a-ea6b-bfda-a7be81de0367 137cbdce-bc9a-ea6b-bfda-a7be81de0367}" 2026-03-31T06:16:17.594Z [INFO] server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a: aborting pipeline replication: peer="{Voter server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c ef1867ae-6e71-dd2e-b0ac-aa540f6df96c}" 2026-03-31T06:16:17.594Z [INFO] creating new snapshot: path=/tmp/raft3767806645/snapshots/2-27-1774937777594.tmp 2026-03-31T06:16:17.595Z [INFO] server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c: starting restore from snapshot: id=2-27-1774937777594 last-index=27 last-term=2 size-in-bytes=168 2026-03-31T06:16:17.595Z [INFO] server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c: snapshot restore progress: id=2-27-1774937777594 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2026-03-31T06:16:17.596Z [INFO] server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c: restored from snapshot: id=2-27-1774937777594 last-index=27 last-term=2 size-in-bytes=168 2026-03-31T06:16:17.596Z [INFO] server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c Address:ef1867ae-6e71-dd2e-b0ac-aa540f6df96c} {Suffrage:Voter ID:server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a Address:ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a} {Suffrage:Voter ID:server-137cbdce-bc9a-ea6b-bfda-a7be81de0367 Address:137cbdce-bc9a-ea6b-bfda-a7be81de0367}]" 2026-03-31T06:16:17.596Z [INFO] creating new snapshot: path=/tmp/raft2126451450/snapshots/2-27-1774937777596.tmp 2026-03-31T06:16:17.596Z [INFO] server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c: entering follower state: follower="Node at ef1867ae-6e71-dd2e-b0ac-aa540f6df96c [Follower]" leader-address= leader-id= 2026-03-31T06:16:17.596Z [INFO] server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a: starting restore from snapshot: id=2-27-1774937777596 last-index=27 last-term=2 size-in-bytes=168 2026-03-31T06:16:17.596Z [INFO] server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a: snapshot restore progress: id=2-27-1774937777596 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2026-03-31T06:16:17.596Z [INFO] server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a: restored from snapshot: id=2-27-1774937777596 last-index=27 last-term=2 size-in-bytes=168 2026-03-31T06:16:17.596Z [INFO] server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c Address:ef1867ae-6e71-dd2e-b0ac-aa540f6df96c} {Suffrage:Voter ID:server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a Address:ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a} {Suffrage:Voter ID:server-137cbdce-bc9a-ea6b-bfda-a7be81de0367 Address:137cbdce-bc9a-ea6b-bfda-a7be81de0367}]" 2026-03-31T06:16:17.596Z [INFO] creating new snapshot: path=/tmp/raft2729932002/snapshots/2-27-1774937777596.tmp 2026-03-31T06:16:17.596Z [INFO] server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a: entering follower state: follower="Node at ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a [Follower]" leader-address= leader-id= 2026-03-31T06:16:17.597Z [INFO] server-137cbdce-bc9a-ea6b-bfda-a7be81de0367: starting restore from snapshot: id=2-27-1774937777596 last-index=27 last-term=2 size-in-bytes=168 2026-03-31T06:16:17.598Z [INFO] server-137cbdce-bc9a-ea6b-bfda-a7be81de0367: snapshot restore progress: id=2-27-1774937777596 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2026-03-31T06:16:17.598Z [INFO] server-137cbdce-bc9a-ea6b-bfda-a7be81de0367: restored from snapshot: id=2-27-1774937777596 last-index=27 last-term=2 size-in-bytes=168 2026-03-31T06:16:17.598Z [INFO] server-137cbdce-bc9a-ea6b-bfda-a7be81de0367: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c Address:ef1867ae-6e71-dd2e-b0ac-aa540f6df96c} {Suffrage:Voter ID:server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a Address:ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a} {Suffrage:Voter ID:server-137cbdce-bc9a-ea6b-bfda-a7be81de0367 Address:137cbdce-bc9a-ea6b-bfda-a7be81de0367}]" 2026-03-31T06:16:17.598Z [INFO] server-137cbdce-bc9a-ea6b-bfda-a7be81de0367: entering follower state: follower="Node at 137cbdce-bc9a-ea6b-bfda-a7be81de0367 [Follower]" leader-address= leader-id= 2026-03-31T06:16:17.668Z [WARN] server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:17.668Z [INFO] server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c: entering candidate state: node="Node at ef1867ae-6e71-dd2e-b0ac-aa540f6df96c [Candidate]" term=3 2026-03-31T06:16:17.669Z [INFO] server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c: election won: term=3 tally=2 2026-03-31T06:16:17.669Z [INFO] server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c: entering leader state: leader="Node at ef1867ae-6e71-dd2e-b0ac-aa540f6df96c [Leader]" 2026-03-31T06:16:17.669Z [INFO] server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c: added peer, starting replication: peer=server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a 2026-03-31T06:16:17.669Z [INFO] server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c: added peer, starting replication: peer=server-137cbdce-bc9a-ea6b-bfda-a7be81de0367 2026-03-31T06:16:17.669Z [INFO] server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c: pipelining replication: peer="{Voter server-137cbdce-bc9a-ea6b-bfda-a7be81de0367 137cbdce-bc9a-ea6b-bfda-a7be81de0367}" 2026-03-31T06:16:17.669Z [INFO] server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c: pipelining replication: peer="{Voter server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a}" 2026-03-31T06:16:17.914Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:18.020Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:17.91450278 +0000 UTC m=+6.192110161 (1 nodes), 30.28µs from start of poll, 484.1788ms from cluster start. Timeout at 2026-03-31 06:16:18.01950978 +0000 UTC m=+6.297117161, 105.007ms after stability 2026-03-31T06:16:18.020Z [INFO] server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c: aborting pipeline replication: peer="{Voter server-ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a ecbb11df-9dd4-f0c4-fc3a-183b65e64d1a}" 2026-03-31T06:16:18.020Z [INFO] server-ef1867ae-6e71-dd2e-b0ac-aa540f6df96c: aborting pipeline replication: peer="{Voter server-137cbdce-bc9a-ea6b-bfda-a7be81de0367 137cbdce-bc9a-ea6b-bfda-a7be81de0367}" --- PASS: TestRaft_RecoverCluster (1.79s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.61s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.59s) --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.59s) === RUN TestRaft_HasExistingState raft_test.go:258: Skipping unreliable test --- SKIP: TestRaft_HasExistingState (0.00s) === RUN TestRaft_SingleNode 2026-03-31T06:16:18.021Z [INFO] server-a6f4bad6-772a-627f-495e-96474a1b896a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f4bad6-772a-627f-495e-96474a1b896a Address:a6f4bad6-772a-627f-495e-96474a1b896a}]" 2026-03-31T06:16:18.021Z [INFO] server-a6f4bad6-772a-627f-495e-96474a1b896a: entering follower state: follower="Node at a6f4bad6-772a-627f-495e-96474a1b896a [Follower]" leader-address= leader-id= 2026-03-31T06:16:18.118Z [WARN] server-a6f4bad6-772a-627f-495e-96474a1b896a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:18.118Z [INFO] server-a6f4bad6-772a-627f-495e-96474a1b896a: entering candidate state: node="Node at a6f4bad6-772a-627f-495e-96474a1b896a [Candidate]" term=2 2026-03-31T06:16:18.119Z [INFO] server-a6f4bad6-772a-627f-495e-96474a1b896a: election won: term=2 tally=1 2026-03-31T06:16:18.119Z [INFO] server-a6f4bad6-772a-627f-495e-96474a1b896a: entering leader state: leader="Node at a6f4bad6-772a-627f-495e-96474a1b896a [Leader]" --- PASS: TestRaft_SingleNode (0.10s) === RUN TestRaft_TripleNode 2026-03-31T06:16:18.119Z [INFO] server-0ab41155-c586-3942-8c55-ca0728f19689: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ab41155-c586-3942-8c55-ca0728f19689 Address:0ab41155-c586-3942-8c55-ca0728f19689} {Suffrage:Voter ID:server-c34f092d-6a17-8044-4298-3c065a931b0d Address:c34f092d-6a17-8044-4298-3c065a931b0d} {Suffrage:Voter ID:server-d2629d42-c97d-bf78-b1a3-90202d2397f2 Address:d2629d42-c97d-bf78-b1a3-90202d2397f2}]" 2026-03-31T06:16:18.120Z [INFO] server-c34f092d-6a17-8044-4298-3c065a931b0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ab41155-c586-3942-8c55-ca0728f19689 Address:0ab41155-c586-3942-8c55-ca0728f19689} {Suffrage:Voter ID:server-c34f092d-6a17-8044-4298-3c065a931b0d Address:c34f092d-6a17-8044-4298-3c065a931b0d} {Suffrage:Voter ID:server-d2629d42-c97d-bf78-b1a3-90202d2397f2 Address:d2629d42-c97d-bf78-b1a3-90202d2397f2}]" 2026-03-31T06:16:18.120Z [INFO] server-0ab41155-c586-3942-8c55-ca0728f19689: entering follower state: follower="Node at 0ab41155-c586-3942-8c55-ca0728f19689 [Follower]" leader-address= leader-id= 2026-03-31T06:16:18.120Z [INFO] server-d2629d42-c97d-bf78-b1a3-90202d2397f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ab41155-c586-3942-8c55-ca0728f19689 Address:0ab41155-c586-3942-8c55-ca0728f19689} {Suffrage:Voter ID:server-c34f092d-6a17-8044-4298-3c065a931b0d Address:c34f092d-6a17-8044-4298-3c065a931b0d} {Suffrage:Voter ID:server-d2629d42-c97d-bf78-b1a3-90202d2397f2 Address:d2629d42-c97d-bf78-b1a3-90202d2397f2}]" 2026-03-31T06:16:18.120Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:18.120Z [INFO] server-c34f092d-6a17-8044-4298-3c065a931b0d: entering follower state: follower="Node at c34f092d-6a17-8044-4298-3c065a931b0d [Follower]" leader-address= leader-id= 2026-03-31T06:16:18.120Z [INFO] server-d2629d42-c97d-bf78-b1a3-90202d2397f2: entering follower state: follower="Node at d2629d42-c97d-bf78-b1a3-90202d2397f2 [Follower]" leader-address= leader-id= 2026-03-31T06:16:18.189Z [WARN] server-0ab41155-c586-3942-8c55-ca0728f19689: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:18.189Z [INFO] server-0ab41155-c586-3942-8c55-ca0728f19689: entering candidate state: node="Node at 0ab41155-c586-3942-8c55-ca0728f19689 [Candidate]" term=2 2026-03-31T06:16:18.189Z [INFO] server-0ab41155-c586-3942-8c55-ca0728f19689: election won: term=2 tally=2 2026-03-31T06:16:18.189Z [INFO] server-0ab41155-c586-3942-8c55-ca0728f19689: entering leader state: leader="Node at 0ab41155-c586-3942-8c55-ca0728f19689 [Leader]" 2026-03-31T06:16:18.189Z [INFO] server-0ab41155-c586-3942-8c55-ca0728f19689: added peer, starting replication: peer=server-c34f092d-6a17-8044-4298-3c065a931b0d 2026-03-31T06:16:18.189Z [INFO] server-0ab41155-c586-3942-8c55-ca0728f19689: added peer, starting replication: peer=server-d2629d42-c97d-bf78-b1a3-90202d2397f2 2026-03-31T06:16:18.189Z [INFO] server-0ab41155-c586-3942-8c55-ca0728f19689: pipelining replication: peer="{Voter server-c34f092d-6a17-8044-4298-3c065a931b0d c34f092d-6a17-8044-4298-3c065a931b0d}" 2026-03-31T06:16:18.189Z [INFO] server-0ab41155-c586-3942-8c55-ca0728f19689: pipelining replication: peer="{Voter server-d2629d42-c97d-bf78-b1a3-90202d2397f2 d2629d42-c97d-bf78-b1a3-90202d2397f2}" 2026-03-31T06:16:18.294Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:18.1896047 +0000 UTC m=+6.467212081 (2 nodes), 69.4228ms from start of poll, 69.82196ms from cluster start. Timeout at 2026-03-31 06:16:18.2946061 +0000 UTC m=+6.572213481, 105.0014ms after stability 2026-03-31T06:16:18.294Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:18.400Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:18.29493138 +0000 UTC m=+6.572538761 (1 nodes), 560ns from start of poll, 175.14864ms from cluster start. Timeout at 2026-03-31 06:16:18.39993922 +0000 UTC m=+6.677546641, 105.00788ms after stability 2026-03-31T06:16:18.411Z [INFO] server-0ab41155-c586-3942-8c55-ca0728f19689: aborting pipeline replication: peer="{Voter server-d2629d42-c97d-bf78-b1a3-90202d2397f2 d2629d42-c97d-bf78-b1a3-90202d2397f2}" 2026-03-31T06:16:18.411Z [INFO] server-0ab41155-c586-3942-8c55-ca0728f19689: aborting pipeline replication: peer="{Voter server-c34f092d-6a17-8044-4298-3c065a931b0d c34f092d-6a17-8044-4298-3c065a931b0d}" 2026-03-31T06:16:18.912Z [ERROR] server-0ab41155-c586-3942-8c55-ca0728f19689: failed to heartbeat to: peer=d2629d42-c97d-bf78-b1a3-90202d2397f2 backoff time=10ms error="command timed out" --- PASS: TestRaft_TripleNode (0.79s) === RUN TestRaft_LeaderFail 2026-03-31T06:16:18.913Z [INFO] server-7b65268c-9b33-f65a-0ffa-685844701264: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b65268c-9b33-f65a-0ffa-685844701264 Address:7b65268c-9b33-f65a-0ffa-685844701264} {Suffrage:Voter ID:server-18df5993-7078-67ed-d008-2b4a05b6a459 Address:18df5993-7078-67ed-d008-2b4a05b6a459} {Suffrage:Voter ID:server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5 Address:896e7a91-d6dc-d4f3-7117-3c7d045b95a5}]" 2026-03-31T06:16:18.913Z [INFO] server-18df5993-7078-67ed-d008-2b4a05b6a459: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b65268c-9b33-f65a-0ffa-685844701264 Address:7b65268c-9b33-f65a-0ffa-685844701264} {Suffrage:Voter ID:server-18df5993-7078-67ed-d008-2b4a05b6a459 Address:18df5993-7078-67ed-d008-2b4a05b6a459} {Suffrage:Voter ID:server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5 Address:896e7a91-d6dc-d4f3-7117-3c7d045b95a5}]" 2026-03-31T06:16:18.913Z [INFO] server-7b65268c-9b33-f65a-0ffa-685844701264: entering follower state: follower="Node at 7b65268c-9b33-f65a-0ffa-685844701264 [Follower]" leader-address= leader-id= 2026-03-31T06:16:18.913Z [INFO] server-18df5993-7078-67ed-d008-2b4a05b6a459: entering follower state: follower="Node at 18df5993-7078-67ed-d008-2b4a05b6a459 [Follower]" leader-address= leader-id= 2026-03-31T06:16:18.913Z [INFO] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b65268c-9b33-f65a-0ffa-685844701264 Address:7b65268c-9b33-f65a-0ffa-685844701264} {Suffrage:Voter ID:server-18df5993-7078-67ed-d008-2b4a05b6a459 Address:18df5993-7078-67ed-d008-2b4a05b6a459} {Suffrage:Voter ID:server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5 Address:896e7a91-d6dc-d4f3-7117-3c7d045b95a5}]" 2026-03-31T06:16:18.913Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:18.913Z [INFO] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: entering follower state: follower="Node at 896e7a91-d6dc-d4f3-7117-3c7d045b95a5 [Follower]" leader-address= leader-id= 2026-03-31T06:16:18.980Z [WARN] server-7b65268c-9b33-f65a-0ffa-685844701264: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:18.981Z [INFO] server-7b65268c-9b33-f65a-0ffa-685844701264: entering candidate state: node="Node at 7b65268c-9b33-f65a-0ffa-685844701264 [Candidate]" term=2 2026-03-31T06:16:18.981Z [INFO] server-7b65268c-9b33-f65a-0ffa-685844701264: election won: term=2 tally=2 2026-03-31T06:16:18.981Z [INFO] server-7b65268c-9b33-f65a-0ffa-685844701264: entering leader state: leader="Node at 7b65268c-9b33-f65a-0ffa-685844701264 [Leader]" 2026-03-31T06:16:18.981Z [INFO] server-7b65268c-9b33-f65a-0ffa-685844701264: added peer, starting replication: peer=server-18df5993-7078-67ed-d008-2b4a05b6a459 2026-03-31T06:16:18.981Z [INFO] server-7b65268c-9b33-f65a-0ffa-685844701264: added peer, starting replication: peer=server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5 2026-03-31T06:16:18.981Z [INFO] server-7b65268c-9b33-f65a-0ffa-685844701264: pipelining replication: peer="{Voter server-18df5993-7078-67ed-d008-2b4a05b6a459 18df5993-7078-67ed-d008-2b4a05b6a459}" 2026-03-31T06:16:18.981Z [INFO] server-7b65268c-9b33-f65a-0ffa-685844701264: pipelining replication: peer="{Voter server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5 896e7a91-d6dc-d4f3-7117-3c7d045b95a5}" 2026-03-31T06:16:19.086Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:18.98124702 +0000 UTC m=+7.258854401 (2 nodes), 67.81ms from start of poll, 68.20088ms from cluster start. Timeout at 2026-03-31 06:16:19.0862491 +0000 UTC m=+7.363856481, 105.00208ms after stability 2026-03-31T06:16:19.086Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:19.192Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:19.0868199 +0000 UTC m=+7.364427281 (1 nodes), 2.76µs from start of poll, 173.77376ms from cluster start. Timeout at 2026-03-31 06:16:19.19182466 +0000 UTC m=+7.469432041, 105.00476ms after stability raft_test.go:378: [INFO] Disconnecting Node at 7b65268c-9b33-f65a-0ffa-685844701264 [Leader] 2026-03-31T06:16:19.207Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to heartbeat to: peer=18df5993-7078-67ed-d008-2b4a05b6a459 backoff time=10ms error="failed to connect to peer: 18df5993-7078-67ed-d008-2b4a05b6a459" 2026-03-31T06:16:19.207Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:19.208Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to pipeline appendEntries: peer="{Voter server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5 896e7a91-d6dc-d4f3-7117-3c7d045b95a5}" error="append pipeline closed" 2026-03-31T06:16:19.208Z [INFO] server-7b65268c-9b33-f65a-0ffa-685844701264: aborting pipeline replication: peer="{Voter server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5 896e7a91-d6dc-d4f3-7117-3c7d045b95a5}" 2026-03-31T06:16:19.208Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to heartbeat to: peer=896e7a91-d6dc-d4f3-7117-3c7d045b95a5 backoff time=10ms error="failed to connect to peer: 896e7a91-d6dc-d4f3-7117-3c7d045b95a5" 2026-03-31T06:16:19.209Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to pipeline appendEntries: peer="{Voter server-18df5993-7078-67ed-d008-2b4a05b6a459 18df5993-7078-67ed-d008-2b4a05b6a459}" error="append pipeline closed" 2026-03-31T06:16:19.209Z [INFO] server-7b65268c-9b33-f65a-0ffa-685844701264: aborting pipeline replication: peer="{Voter server-18df5993-7078-67ed-d008-2b4a05b6a459 18df5993-7078-67ed-d008-2b4a05b6a459}" 2026-03-31T06:16:19.218Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to appendEntries to: peer="{Voter server-18df5993-7078-67ed-d008-2b4a05b6a459 18df5993-7078-67ed-d008-2b4a05b6a459}" error="failed to connect to peer: 18df5993-7078-67ed-d008-2b4a05b6a459" 2026-03-31T06:16:19.218Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to appendEntries to: peer="{Voter server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5 896e7a91-d6dc-d4f3-7117-3c7d045b95a5}" error="failed to connect to peer: 896e7a91-d6dc-d4f3-7117-3c7d045b95a5" 2026-03-31T06:16:19.223Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to heartbeat to: peer=18df5993-7078-67ed-d008-2b4a05b6a459 backoff time=10ms error="failed to connect to peer: 18df5993-7078-67ed-d008-2b4a05b6a459" 2026-03-31T06:16:19.226Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to heartbeat to: peer=896e7a91-d6dc-d4f3-7117-3c7d045b95a5 backoff time=10ms error="failed to connect to peer: 896e7a91-d6dc-d4f3-7117-3c7d045b95a5" 2026-03-31T06:16:19.234Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to appendEntries to: peer="{Voter server-18df5993-7078-67ed-d008-2b4a05b6a459 18df5993-7078-67ed-d008-2b4a05b6a459}" error="failed to connect to peer: 18df5993-7078-67ed-d008-2b4a05b6a459" 2026-03-31T06:16:19.235Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to appendEntries to: peer="{Voter server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5 896e7a91-d6dc-d4f3-7117-3c7d045b95a5}" error="failed to connect to peer: 896e7a91-d6dc-d4f3-7117-3c7d045b95a5" 2026-03-31T06:16:19.242Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to heartbeat to: peer=18df5993-7078-67ed-d008-2b4a05b6a459 backoff time=10ms error="failed to connect to peer: 18df5993-7078-67ed-d008-2b4a05b6a459" 2026-03-31T06:16:19.244Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to heartbeat to: peer=896e7a91-d6dc-d4f3-7117-3c7d045b95a5 backoff time=10ms error="failed to connect to peer: 896e7a91-d6dc-d4f3-7117-3c7d045b95a5" 2026-03-31T06:16:19.250Z [WARN] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to contact: server-id=server-18df5993-7078-67ed-d008-2b4a05b6a459 time=50.37584ms 2026-03-31T06:16:19.251Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to appendEntries to: peer="{Voter server-18df5993-7078-67ed-d008-2b4a05b6a459 18df5993-7078-67ed-d008-2b4a05b6a459}" error="failed to connect to peer: 18df5993-7078-67ed-d008-2b4a05b6a459" 2026-03-31T06:16:19.255Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to appendEntries to: peer="{Voter server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5 896e7a91-d6dc-d4f3-7117-3c7d045b95a5}" error="failed to connect to peer: 896e7a91-d6dc-d4f3-7117-3c7d045b95a5" 2026-03-31T06:16:19.260Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to heartbeat to: peer=896e7a91-d6dc-d4f3-7117-3c7d045b95a5 backoff time=20ms error="failed to connect to peer: 896e7a91-d6dc-d4f3-7117-3c7d045b95a5" 2026-03-31T06:16:19.260Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to heartbeat to: peer=18df5993-7078-67ed-d008-2b4a05b6a459 backoff time=20ms error="failed to connect to peer: 18df5993-7078-67ed-d008-2b4a05b6a459" 2026-03-31T06:16:19.261Z [WARN] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to contact: server-id=server-18df5993-7078-67ed-d008-2b4a05b6a459 time=61.29244ms 2026-03-31T06:16:19.261Z [WARN] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to contact: server-id=server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5 time=59.07168ms 2026-03-31T06:16:19.261Z [WARN] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to contact quorum of nodes, stepping down 2026-03-31T06:16:19.261Z [INFO] server-7b65268c-9b33-f65a-0ffa-685844701264: entering follower state: follower="Node at 7b65268c-9b33-f65a-0ffa-685844701264 [Follower]" leader-address= leader-id= 2026-03-31T06:16:19.281Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to appendEntries to: peer="{Voter server-18df5993-7078-67ed-d008-2b4a05b6a459 18df5993-7078-67ed-d008-2b4a05b6a459}" error="failed to connect to peer: 18df5993-7078-67ed-d008-2b4a05b6a459" 2026-03-31T06:16:19.283Z [WARN] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: heartbeat timeout reached, starting election: last-leader-addr=7b65268c-9b33-f65a-0ffa-685844701264 last-leader-id=server-7b65268c-9b33-f65a-0ffa-685844701264 2026-03-31T06:16:19.283Z [INFO] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: entering candidate state: node="Node at 896e7a91-d6dc-d4f3-7117-3c7d045b95a5 [Candidate]" term=3 2026-03-31T06:16:19.283Z [WARN] server-18df5993-7078-67ed-d008-2b4a05b6a459: rejecting vote request since we have a leader: from=896e7a91-d6dc-d4f3-7117-3c7d045b95a5 leader=7b65268c-9b33-f65a-0ffa-685844701264 leader-id=server-7b65268c-9b33-f65a-0ffa-685844701264 2026-03-31T06:16:19.283Z [ERROR] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: failed to make requestVote RPC: target="{Voter server-7b65268c-9b33-f65a-0ffa-685844701264 7b65268c-9b33-f65a-0ffa-685844701264}" error="failed to connect to peer: 7b65268c-9b33-f65a-0ffa-685844701264" term=3 2026-03-31T06:16:19.322Z [WARN] server-18df5993-7078-67ed-d008-2b4a05b6a459: heartbeat timeout reached, starting election: last-leader-addr=7b65268c-9b33-f65a-0ffa-685844701264 last-leader-id=server-7b65268c-9b33-f65a-0ffa-685844701264 2026-03-31T06:16:19.322Z [INFO] server-18df5993-7078-67ed-d008-2b4a05b6a459: entering candidate state: node="Node at 18df5993-7078-67ed-d008-2b4a05b6a459 [Candidate]" term=3 2026-03-31T06:16:19.322Z [INFO] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: duplicate requestVote for same term: term=3 2026-03-31T06:16:19.322Z [ERROR] server-18df5993-7078-67ed-d008-2b4a05b6a459: failed to make requestVote RPC: target="{Voter server-7b65268c-9b33-f65a-0ffa-685844701264 7b65268c-9b33-f65a-0ffa-685844701264}" error="failed to connect to peer: 7b65268c-9b33-f65a-0ffa-685844701264" term=3 2026-03-31T06:16:19.338Z [WARN] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: Election timeout reached, restarting election 2026-03-31T06:16:19.338Z [INFO] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: entering candidate state: node="Node at 896e7a91-d6dc-d4f3-7117-3c7d045b95a5 [Candidate]" term=4 2026-03-31T06:16:19.338Z [INFO] server-18df5993-7078-67ed-d008-2b4a05b6a459: entering follower state: follower="Node at 18df5993-7078-67ed-d008-2b4a05b6a459 [Follower]" leader-address= leader-id= 2026-03-31T06:16:19.338Z [ERROR] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: failed to make requestVote RPC: target="{Voter server-7b65268c-9b33-f65a-0ffa-685844701264 7b65268c-9b33-f65a-0ffa-685844701264}" error="failed to connect to peer: 7b65268c-9b33-f65a-0ffa-685844701264" term=4 2026-03-31T06:16:19.338Z [INFO] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: election won: term=4 tally=2 2026-03-31T06:16:19.338Z [INFO] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: entering leader state: leader="Node at 896e7a91-d6dc-d4f3-7117-3c7d045b95a5 [Leader]" 2026-03-31T06:16:19.338Z [INFO] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: added peer, starting replication: peer=server-7b65268c-9b33-f65a-0ffa-685844701264 2026-03-31T06:16:19.338Z [INFO] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: added peer, starting replication: peer=server-18df5993-7078-67ed-d008-2b4a05b6a459 2026-03-31T06:16:19.338Z [ERROR] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: failed to appendEntries to: peer="{Voter server-7b65268c-9b33-f65a-0ffa-685844701264 7b65268c-9b33-f65a-0ffa-685844701264}" error="failed to connect to peer: 7b65268c-9b33-f65a-0ffa-685844701264" 2026-03-31T06:16:19.338Z [INFO] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: pipelining replication: peer="{Voter server-18df5993-7078-67ed-d008-2b4a05b6a459 18df5993-7078-67ed-d008-2b4a05b6a459}" 2026-03-31T06:16:19.344Z [ERROR] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: failed to heartbeat to: peer=7b65268c-9b33-f65a-0ffa-685844701264 backoff time=10ms error="failed to connect to peer: 7b65268c-9b33-f65a-0ffa-685844701264" 2026-03-31T06:16:19.353Z [WARN] server-7b65268c-9b33-f65a-0ffa-685844701264: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:19.353Z [INFO] server-7b65268c-9b33-f65a-0ffa-685844701264: entering candidate state: node="Node at 7b65268c-9b33-f65a-0ffa-685844701264 [Candidate]" term=3 2026-03-31T06:16:19.353Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to make requestVote RPC: target="{Voter server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5 896e7a91-d6dc-d4f3-7117-3c7d045b95a5}" error="failed to connect to peer: 896e7a91-d6dc-d4f3-7117-3c7d045b95a5" term=3 2026-03-31T06:16:19.353Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to make requestVote RPC: target="{Voter server-18df5993-7078-67ed-d008-2b4a05b6a459 18df5993-7078-67ed-d008-2b4a05b6a459}" error="failed to connect to peer: 18df5993-7078-67ed-d008-2b4a05b6a459" term=3 2026-03-31T06:16:19.358Z [ERROR] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: failed to appendEntries to: peer="{Voter server-7b65268c-9b33-f65a-0ffa-685844701264 7b65268c-9b33-f65a-0ffa-685844701264}" error="failed to connect to peer: 7b65268c-9b33-f65a-0ffa-685844701264" 2026-03-31T06:16:19.361Z [ERROR] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: failed to heartbeat to: peer=7b65268c-9b33-f65a-0ffa-685844701264 backoff time=10ms error="failed to connect to peer: 7b65268c-9b33-f65a-0ffa-685844701264" 2026-03-31T06:16:19.379Z [ERROR] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: failed to appendEntries to: peer="{Voter server-7b65268c-9b33-f65a-0ffa-685844701264 7b65268c-9b33-f65a-0ffa-685844701264}" error="failed to connect to peer: 7b65268c-9b33-f65a-0ffa-685844701264" 2026-03-31T06:16:19.381Z [ERROR] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: failed to heartbeat to: peer=7b65268c-9b33-f65a-0ffa-685844701264 backoff time=10ms error="failed to connect to peer: 7b65268c-9b33-f65a-0ffa-685844701264" 2026-03-31T06:16:19.389Z [WARN] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: failed to contact: server-id=server-7b65268c-9b33-f65a-0ffa-685844701264 time=50.46716ms 2026-03-31T06:16:19.397Z [ERROR] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: failed to heartbeat to: peer=7b65268c-9b33-f65a-0ffa-685844701264 backoff time=20ms error="failed to connect to peer: 7b65268c-9b33-f65a-0ffa-685844701264" 2026-03-31T06:16:19.406Z [ERROR] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: failed to appendEntries to: peer="{Voter server-7b65268c-9b33-f65a-0ffa-685844701264 7b65268c-9b33-f65a-0ffa-685844701264}" error="failed to connect to peer: 7b65268c-9b33-f65a-0ffa-685844701264" 2026-03-31T06:16:19.427Z [ERROR] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: failed to heartbeat to: peer=7b65268c-9b33-f65a-0ffa-685844701264 backoff time=25ms error="failed to connect to peer: 7b65268c-9b33-f65a-0ffa-685844701264" 2026-03-31T06:16:19.432Z [WARN] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: failed to contact: server-id=server-7b65268c-9b33-f65a-0ffa-685844701264 time=94.0816ms 2026-03-31T06:16:19.443Z [WARN] server-7b65268c-9b33-f65a-0ffa-685844701264: Election timeout reached, restarting election 2026-03-31T06:16:19.443Z [INFO] server-7b65268c-9b33-f65a-0ffa-685844701264: entering candidate state: node="Node at 7b65268c-9b33-f65a-0ffa-685844701264 [Candidate]" term=4 2026-03-31T06:16:19.443Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to make requestVote RPC: target="{Voter server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5 896e7a91-d6dc-d4f3-7117-3c7d045b95a5}" error="failed to connect to peer: 896e7a91-d6dc-d4f3-7117-3c7d045b95a5" term=4 2026-03-31T06:16:19.443Z [ERROR] server-7b65268c-9b33-f65a-0ffa-685844701264: failed to make requestVote RPC: target="{Voter server-18df5993-7078-67ed-d008-2b4a05b6a459 18df5993-7078-67ed-d008-2b4a05b6a459}" error="failed to connect to peer: 18df5993-7078-67ed-d008-2b4a05b6a459" term=4 2026-03-31T06:16:19.453Z [ERROR] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: failed to appendEntries to: peer="{Voter server-7b65268c-9b33-f65a-0ffa-685844701264 7b65268c-9b33-f65a-0ffa-685844701264}" error="failed to connect to peer: 7b65268c-9b33-f65a-0ffa-685844701264" 2026-03-31T06:16:19.458Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:19.3533147 +0000 UTC m=+7.630922081 (1 nodes), 146.19356ms from start of poll, 440.26856ms from cluster start. Timeout at 2026-03-31 06:16:19.45831746 +0000 UTC m=+7.735924841, 105.00276ms after stability raft_test.go:413: [INFO] Reconnecting Node at 7b65268c-9b33-f65a-0ffa-685844701264 [Candidate] 2026-03-31T06:16:19.462Z [INFO] server-7b65268c-9b33-f65a-0ffa-685844701264: entering follower state: follower="Node at 7b65268c-9b33-f65a-0ffa-685844701264 [Follower]" leader-address=896e7a91-d6dc-d4f3-7117-3c7d045b95a5 leader-id=server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5 2026-03-31T06:16:19.539Z [INFO] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: pipelining replication: peer="{Voter server-7b65268c-9b33-f65a-0ffa-685844701264 7b65268c-9b33-f65a-0ffa-685844701264}" 2026-03-31T06:16:19.540Z [INFO] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: aborting pipeline replication: peer="{Voter server-18df5993-7078-67ed-d008-2b4a05b6a459 18df5993-7078-67ed-d008-2b4a05b6a459}" 2026-03-31T06:16:19.540Z [INFO] server-896e7a91-d6dc-d4f3-7117-3c7d045b95a5: aborting pipeline replication: peer="{Voter server-7b65268c-9b33-f65a-0ffa-685844701264 7b65268c-9b33-f65a-0ffa-685844701264}" --- PASS: TestRaft_LeaderFail (0.63s) === RUN TestRaft_BehindFollower 2026-03-31T06:16:19.541Z [INFO] server-805abb5a-1de8-3992-f75d-6d8c19c77bf3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-805abb5a-1de8-3992-f75d-6d8c19c77bf3 Address:805abb5a-1de8-3992-f75d-6d8c19c77bf3} {Suffrage:Voter ID:server-8ceff989-0d1c-2139-eb40-50a62095d20d Address:8ceff989-0d1c-2139-eb40-50a62095d20d} {Suffrage:Voter ID:server-48d5acbd-b164-5174-fe61-9c7dab4485b0 Address:48d5acbd-b164-5174-fe61-9c7dab4485b0}]" 2026-03-31T06:16:19.541Z [INFO] server-805abb5a-1de8-3992-f75d-6d8c19c77bf3: entering follower state: follower="Node at 805abb5a-1de8-3992-f75d-6d8c19c77bf3 [Follower]" leader-address= leader-id= 2026-03-31T06:16:19.541Z [INFO] server-8ceff989-0d1c-2139-eb40-50a62095d20d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-805abb5a-1de8-3992-f75d-6d8c19c77bf3 Address:805abb5a-1de8-3992-f75d-6d8c19c77bf3} {Suffrage:Voter ID:server-8ceff989-0d1c-2139-eb40-50a62095d20d Address:8ceff989-0d1c-2139-eb40-50a62095d20d} {Suffrage:Voter ID:server-48d5acbd-b164-5174-fe61-9c7dab4485b0 Address:48d5acbd-b164-5174-fe61-9c7dab4485b0}]" 2026-03-31T06:16:19.541Z [INFO] server-8ceff989-0d1c-2139-eb40-50a62095d20d: entering follower state: follower="Node at 8ceff989-0d1c-2139-eb40-50a62095d20d [Follower]" leader-address= leader-id= 2026-03-31T06:16:19.541Z [INFO] server-48d5acbd-b164-5174-fe61-9c7dab4485b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-805abb5a-1de8-3992-f75d-6d8c19c77bf3 Address:805abb5a-1de8-3992-f75d-6d8c19c77bf3} {Suffrage:Voter ID:server-8ceff989-0d1c-2139-eb40-50a62095d20d Address:8ceff989-0d1c-2139-eb40-50a62095d20d} {Suffrage:Voter ID:server-48d5acbd-b164-5174-fe61-9c7dab4485b0 Address:48d5acbd-b164-5174-fe61-9c7dab4485b0}]" 2026-03-31T06:16:19.541Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:19.541Z [INFO] server-48d5acbd-b164-5174-fe61-9c7dab4485b0: entering follower state: follower="Node at 48d5acbd-b164-5174-fe61-9c7dab4485b0 [Follower]" leader-address= leader-id= 2026-03-31T06:16:19.607Z [WARN] server-48d5acbd-b164-5174-fe61-9c7dab4485b0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:19.607Z [INFO] server-48d5acbd-b164-5174-fe61-9c7dab4485b0: entering candidate state: node="Node at 48d5acbd-b164-5174-fe61-9c7dab4485b0 [Candidate]" term=2 2026-03-31T06:16:19.607Z [INFO] server-48d5acbd-b164-5174-fe61-9c7dab4485b0: election won: term=2 tally=2 2026-03-31T06:16:19.607Z [INFO] server-48d5acbd-b164-5174-fe61-9c7dab4485b0: entering leader state: leader="Node at 48d5acbd-b164-5174-fe61-9c7dab4485b0 [Leader]" 2026-03-31T06:16:19.607Z [INFO] server-48d5acbd-b164-5174-fe61-9c7dab4485b0: added peer, starting replication: peer=server-805abb5a-1de8-3992-f75d-6d8c19c77bf3 2026-03-31T06:16:19.607Z [INFO] server-48d5acbd-b164-5174-fe61-9c7dab4485b0: added peer, starting replication: peer=server-8ceff989-0d1c-2139-eb40-50a62095d20d 2026-03-31T06:16:19.607Z [INFO] server-48d5acbd-b164-5174-fe61-9c7dab4485b0: pipelining replication: peer="{Voter server-805abb5a-1de8-3992-f75d-6d8c19c77bf3 805abb5a-1de8-3992-f75d-6d8c19c77bf3}" 2026-03-31T06:16:19.607Z [INFO] server-48d5acbd-b164-5174-fe61-9c7dab4485b0: pipelining replication: peer="{Voter server-8ceff989-0d1c-2139-eb40-50a62095d20d 8ceff989-0d1c-2139-eb40-50a62095d20d}" 2026-03-31T06:16:19.712Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:19.60759606 +0000 UTC m=+7.885203441 (1 nodes), 65.7132ms from start of poll, 66.16972ms from cluster start. Timeout at 2026-03-31 06:16:19.7125991 +0000 UTC m=+7.990206481, 105.00304ms after stability 2026-03-31T06:16:19.712Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:19.818Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:19.71271698 +0000 UTC m=+7.990324361 (2 nodes), 1.04µs from start of poll, 171.29064ms from cluster start. Timeout at 2026-03-31 06:16:19.81772174 +0000 UTC m=+8.095329121, 105.00476ms after stability 2026-03-31T06:16:19.818Z [ERROR] server-48d5acbd-b164-5174-fe61-9c7dab4485b0: failed to pipeline appendEntries: peer="{Voter server-805abb5a-1de8-3992-f75d-6d8c19c77bf3 805abb5a-1de8-3992-f75d-6d8c19c77bf3}" error="append pipeline closed" 2026-03-31T06:16:19.818Z [INFO] server-48d5acbd-b164-5174-fe61-9c7dab4485b0: aborting pipeline replication: peer="{Voter server-805abb5a-1de8-3992-f75d-6d8c19c77bf3 805abb5a-1de8-3992-f75d-6d8c19c77bf3}" 2026-03-31T06:16:19.818Z [ERROR] server-48d5acbd-b164-5174-fe61-9c7dab4485b0: failed to appendEntries to: peer="{Voter server-805abb5a-1de8-3992-f75d-6d8c19c77bf3 805abb5a-1de8-3992-f75d-6d8c19c77bf3}" error="failed to connect to peer: 805abb5a-1de8-3992-f75d-6d8c19c77bf3" raft_test.go:459: [INFO] Finished apply without behind follower 2026-03-31T06:16:19.829Z [INFO] server-48d5acbd-b164-5174-fe61-9c7dab4485b0: pipelining replication: peer="{Voter server-805abb5a-1de8-3992-f75d-6d8c19c77bf3 805abb5a-1de8-3992-f75d-6d8c19c77bf3}" 2026-03-31T06:16:19.840Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:19.946Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:19.84047854 +0000 UTC m=+8.118085921 (1 nodes), 880ns from start of poll, 299.0522ms from cluster start. Timeout at 2026-03-31 06:16:19.94548218 +0000 UTC m=+8.223089561, 105.00364ms after stability 2026-03-31T06:16:19.946Z [INFO] server-48d5acbd-b164-5174-fe61-9c7dab4485b0: aborting pipeline replication: peer="{Voter server-8ceff989-0d1c-2139-eb40-50a62095d20d 8ceff989-0d1c-2139-eb40-50a62095d20d}" 2026-03-31T06:16:19.946Z [INFO] server-48d5acbd-b164-5174-fe61-9c7dab4485b0: aborting pipeline replication: peer="{Voter server-805abb5a-1de8-3992-f75d-6d8c19c77bf3 805abb5a-1de8-3992-f75d-6d8c19c77bf3}" --- PASS: TestRaft_BehindFollower (0.41s) === RUN TestRaft_ApplyNonLeader 2026-03-31T06:16:19.947Z [INFO] server-b519d033-d31b-60de-1b83-4925c20d146e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b519d033-d31b-60de-1b83-4925c20d146e Address:b519d033-d31b-60de-1b83-4925c20d146e} {Suffrage:Voter ID:server-b4e122ef-a4ae-7c9b-687f-72e54d6e3652 Address:b4e122ef-a4ae-7c9b-687f-72e54d6e3652} {Suffrage:Voter ID:server-04dc1cf0-b759-2d77-6f98-e070e55ec3b0 Address:04dc1cf0-b759-2d77-6f98-e070e55ec3b0}]" 2026-03-31T06:16:19.947Z [INFO] server-b519d033-d31b-60de-1b83-4925c20d146e: entering follower state: follower="Node at b519d033-d31b-60de-1b83-4925c20d146e [Follower]" leader-address= leader-id= 2026-03-31T06:16:19.947Z [INFO] server-b4e122ef-a4ae-7c9b-687f-72e54d6e3652: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b519d033-d31b-60de-1b83-4925c20d146e Address:b519d033-d31b-60de-1b83-4925c20d146e} {Suffrage:Voter ID:server-b4e122ef-a4ae-7c9b-687f-72e54d6e3652 Address:b4e122ef-a4ae-7c9b-687f-72e54d6e3652} {Suffrage:Voter ID:server-04dc1cf0-b759-2d77-6f98-e070e55ec3b0 Address:04dc1cf0-b759-2d77-6f98-e070e55ec3b0}]" 2026-03-31T06:16:19.947Z [INFO] server-b4e122ef-a4ae-7c9b-687f-72e54d6e3652: entering follower state: follower="Node at b4e122ef-a4ae-7c9b-687f-72e54d6e3652 [Follower]" leader-address= leader-id= 2026-03-31T06:16:19.947Z [INFO] server-04dc1cf0-b759-2d77-6f98-e070e55ec3b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b519d033-d31b-60de-1b83-4925c20d146e Address:b519d033-d31b-60de-1b83-4925c20d146e} {Suffrage:Voter ID:server-b4e122ef-a4ae-7c9b-687f-72e54d6e3652 Address:b4e122ef-a4ae-7c9b-687f-72e54d6e3652} {Suffrage:Voter ID:server-04dc1cf0-b759-2d77-6f98-e070e55ec3b0 Address:04dc1cf0-b759-2d77-6f98-e070e55ec3b0}]" 2026-03-31T06:16:19.947Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:19.947Z [INFO] server-04dc1cf0-b759-2d77-6f98-e070e55ec3b0: entering follower state: follower="Node at 04dc1cf0-b759-2d77-6f98-e070e55ec3b0 [Follower]" leader-address= leader-id= 2026-03-31T06:16:20.006Z [WARN] server-b519d033-d31b-60de-1b83-4925c20d146e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:20.006Z [INFO] server-b519d033-d31b-60de-1b83-4925c20d146e: entering candidate state: node="Node at b519d033-d31b-60de-1b83-4925c20d146e [Candidate]" term=2 2026-03-31T06:16:20.006Z [INFO] server-b519d033-d31b-60de-1b83-4925c20d146e: election won: term=2 tally=2 2026-03-31T06:16:20.006Z [INFO] server-b519d033-d31b-60de-1b83-4925c20d146e: entering leader state: leader="Node at b519d033-d31b-60de-1b83-4925c20d146e [Leader]" 2026-03-31T06:16:20.006Z [INFO] server-b519d033-d31b-60de-1b83-4925c20d146e: added peer, starting replication: peer=server-b4e122ef-a4ae-7c9b-687f-72e54d6e3652 2026-03-31T06:16:20.006Z [INFO] server-b519d033-d31b-60de-1b83-4925c20d146e: added peer, starting replication: peer=server-04dc1cf0-b759-2d77-6f98-e070e55ec3b0 2026-03-31T06:16:20.006Z [INFO] server-b519d033-d31b-60de-1b83-4925c20d146e: pipelining replication: peer="{Voter server-b4e122ef-a4ae-7c9b-687f-72e54d6e3652 b4e122ef-a4ae-7c9b-687f-72e54d6e3652}" 2026-03-31T06:16:20.006Z [INFO] server-b519d033-d31b-60de-1b83-4925c20d146e: pipelining replication: peer="{Voter server-04dc1cf0-b759-2d77-6f98-e070e55ec3b0 04dc1cf0-b759-2d77-6f98-e070e55ec3b0}" 2026-03-31T06:16:20.112Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:20.0069299 +0000 UTC m=+8.284537281 (1 nodes), 59.37ms from start of poll, 59.89936ms from cluster start. Timeout at 2026-03-31 06:16:20.11193394 +0000 UTC m=+8.389541321, 105.00404ms after stability 2026-03-31T06:16:20.112Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:20.217Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:20.11223634 +0000 UTC m=+8.389843761 (2 nodes), 1.16µs from start of poll, 165.20584ms from cluster start. Timeout at 2026-03-31 06:16:20.2172403 +0000 UTC m=+8.494847681, 105.00392ms after stability 2026-03-31T06:16:20.217Z [INFO] server-b519d033-d31b-60de-1b83-4925c20d146e: aborting pipeline replication: peer="{Voter server-04dc1cf0-b759-2d77-6f98-e070e55ec3b0 04dc1cf0-b759-2d77-6f98-e070e55ec3b0}" 2026-03-31T06:16:20.217Z [INFO] server-b519d033-d31b-60de-1b83-4925c20d146e: aborting pipeline replication: peer="{Voter server-b4e122ef-a4ae-7c9b-687f-72e54d6e3652 b4e122ef-a4ae-7c9b-687f-72e54d6e3652}" --- PASS: TestRaft_ApplyNonLeader (0.27s) === RUN TestRaft_ApplyConcurrent 2026-03-31T06:16:20.218Z [INFO] server-6a431bab-9d17-b7d7-84bb-fa7083f86350: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a431bab-9d17-b7d7-84bb-fa7083f86350 Address:6a431bab-9d17-b7d7-84bb-fa7083f86350} {Suffrage:Voter ID:server-6bce5666-e071-9fd6-6b49-96d1eefbf07e Address:6bce5666-e071-9fd6-6b49-96d1eefbf07e} {Suffrage:Voter ID:server-7910871e-c42a-1b27-1be7-f93678240163 Address:7910871e-c42a-1b27-1be7-f93678240163}]" 2026-03-31T06:16:20.218Z [INFO] server-6bce5666-e071-9fd6-6b49-96d1eefbf07e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a431bab-9d17-b7d7-84bb-fa7083f86350 Address:6a431bab-9d17-b7d7-84bb-fa7083f86350} {Suffrage:Voter ID:server-6bce5666-e071-9fd6-6b49-96d1eefbf07e Address:6bce5666-e071-9fd6-6b49-96d1eefbf07e} {Suffrage:Voter ID:server-7910871e-c42a-1b27-1be7-f93678240163 Address:7910871e-c42a-1b27-1be7-f93678240163}]" 2026-03-31T06:16:20.218Z [INFO] server-6a431bab-9d17-b7d7-84bb-fa7083f86350: entering follower state: follower="Node at 6a431bab-9d17-b7d7-84bb-fa7083f86350 [Follower]" leader-address= leader-id= 2026-03-31T06:16:20.218Z [INFO] server-6bce5666-e071-9fd6-6b49-96d1eefbf07e: entering follower state: follower="Node at 6bce5666-e071-9fd6-6b49-96d1eefbf07e [Follower]" leader-address= leader-id= 2026-03-31T06:16:20.218Z [INFO] server-7910871e-c42a-1b27-1be7-f93678240163: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a431bab-9d17-b7d7-84bb-fa7083f86350 Address:6a431bab-9d17-b7d7-84bb-fa7083f86350} {Suffrage:Voter ID:server-6bce5666-e071-9fd6-6b49-96d1eefbf07e Address:6bce5666-e071-9fd6-6b49-96d1eefbf07e} {Suffrage:Voter ID:server-7910871e-c42a-1b27-1be7-f93678240163 Address:7910871e-c42a-1b27-1be7-f93678240163}]" 2026-03-31T06:16:20.218Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:20.218Z [INFO] server-7910871e-c42a-1b27-1be7-f93678240163: entering follower state: follower="Node at 7910871e-c42a-1b27-1be7-f93678240163 [Follower]" leader-address= leader-id= 2026-03-31T06:16:20.357Z [WARN] server-7910871e-c42a-1b27-1be7-f93678240163: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:20.357Z [INFO] server-7910871e-c42a-1b27-1be7-f93678240163: entering candidate state: node="Node at 7910871e-c42a-1b27-1be7-f93678240163 [Candidate]" term=2 2026-03-31T06:16:20.357Z [INFO] server-7910871e-c42a-1b27-1be7-f93678240163: election won: term=2 tally=2 2026-03-31T06:16:20.357Z [INFO] server-7910871e-c42a-1b27-1be7-f93678240163: entering leader state: leader="Node at 7910871e-c42a-1b27-1be7-f93678240163 [Leader]" 2026-03-31T06:16:20.357Z [INFO] server-7910871e-c42a-1b27-1be7-f93678240163: added peer, starting replication: peer=server-6a431bab-9d17-b7d7-84bb-fa7083f86350 2026-03-31T06:16:20.357Z [INFO] server-7910871e-c42a-1b27-1be7-f93678240163: added peer, starting replication: peer=server-6bce5666-e071-9fd6-6b49-96d1eefbf07e 2026-03-31T06:16:20.357Z [INFO] server-7910871e-c42a-1b27-1be7-f93678240163: pipelining replication: peer="{Voter server-6a431bab-9d17-b7d7-84bb-fa7083f86350 6a431bab-9d17-b7d7-84bb-fa7083f86350}" 2026-03-31T06:16:20.357Z [INFO] server-7910871e-c42a-1b27-1be7-f93678240163: pipelining replication: peer="{Voter server-6bce5666-e071-9fd6-6b49-96d1eefbf07e 6bce5666-e071-9fd6-6b49-96d1eefbf07e}" 2026-03-31T06:16:20.563Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:20.35778102 +0000 UTC m=+8.635388401 (1 nodes), 139.25108ms from start of poll, 139.63992ms from cluster start. Timeout at 2026-03-31 06:16:20.56278706 +0000 UTC m=+8.840394441, 205.00604ms after stability 2026-03-31T06:16:20.578Z [INFO] server-7910871e-c42a-1b27-1be7-f93678240163: aborting pipeline replication: peer="{Voter server-6bce5666-e071-9fd6-6b49-96d1eefbf07e 6bce5666-e071-9fd6-6b49-96d1eefbf07e}" 2026-03-31T06:16:20.578Z [INFO] server-7910871e-c42a-1b27-1be7-f93678240163: aborting pipeline replication: peer="{Voter server-6a431bab-9d17-b7d7-84bb-fa7083f86350 6a431bab-9d17-b7d7-84bb-fa7083f86350}" --- PASS: TestRaft_ApplyConcurrent (0.36s) === RUN TestRaft_ApplyConcurrent_Timeout 2026-03-31T06:16:20.578Z [INFO] server-b51f4794-0c81-1d25-de20-6c2d54d1d61f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b51f4794-0c81-1d25-de20-6c2d54d1d61f Address:b51f4794-0c81-1d25-de20-6c2d54d1d61f}]" 2026-03-31T06:16:20.578Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:20.578Z [INFO] server-b51f4794-0c81-1d25-de20-6c2d54d1d61f: entering follower state: follower="Node at b51f4794-0c81-1d25-de20-6c2d54d1d61f [Follower]" leader-address= leader-id= 2026-03-31T06:16:20.739Z [WARN] server-b51f4794-0c81-1d25-de20-6c2d54d1d61f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:20.739Z [INFO] server-b51f4794-0c81-1d25-de20-6c2d54d1d61f: entering candidate state: node="Node at b51f4794-0c81-1d25-de20-6c2d54d1d61f [Candidate]" term=2 2026-03-31T06:16:20.739Z [INFO] server-b51f4794-0c81-1d25-de20-6c2d54d1d61f: election won: term=2 tally=1 2026-03-31T06:16:20.739Z [INFO] server-b51f4794-0c81-1d25-de20-6c2d54d1d61f: entering leader state: leader="Node at b51f4794-0c81-1d25-de20-6c2d54d1d61f [Leader]" 2026-03-31T06:16:20.941Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:20.73965426 +0000 UTC m=+9.017261641 (1 nodes), 160.75504ms from start of poll, 160.93788ms from cluster start. Timeout at 2026-03-31 06:16:20.94065734 +0000 UTC m=+9.218264721, 201.00308ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.36s) === RUN TestRaft_JoinNode 2026-03-31T06:16:20.942Z [INFO] server-9335ae3d-a47c-730f-dda1-bb94a4c94c3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9335ae3d-a47c-730f-dda1-bb94a4c94c3f Address:9335ae3d-a47c-730f-dda1-bb94a4c94c3f} {Suffrage:Voter ID:server-6163e78d-d2c6-0207-021c-580c19332fd3 Address:6163e78d-d2c6-0207-021c-580c19332fd3}]" 2026-03-31T06:16:20.942Z [INFO] server-6163e78d-d2c6-0207-021c-580c19332fd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9335ae3d-a47c-730f-dda1-bb94a4c94c3f Address:9335ae3d-a47c-730f-dda1-bb94a4c94c3f} {Suffrage:Voter ID:server-6163e78d-d2c6-0207-021c-580c19332fd3 Address:6163e78d-d2c6-0207-021c-580c19332fd3}]" 2026-03-31T06:16:20.942Z [INFO] server-9335ae3d-a47c-730f-dda1-bb94a4c94c3f: entering follower state: follower="Node at 9335ae3d-a47c-730f-dda1-bb94a4c94c3f [Follower]" leader-address= leader-id= 2026-03-31T06:16:20.942Z [INFO] server-6163e78d-d2c6-0207-021c-580c19332fd3: entering follower state: follower="Node at 6163e78d-d2c6-0207-021c-580c19332fd3 [Follower]" leader-address= leader-id= 2026-03-31T06:16:20.942Z [INFO] server-6e984533-1245-9731-be03-46cba4baba10: initial configuration: index=0 servers=[] 2026-03-31T06:16:20.942Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:20.942Z [INFO] server-6e984533-1245-9731-be03-46cba4baba10: entering follower state: follower="Node at 6e984533-1245-9731-be03-46cba4baba10 [Follower]" leader-address= leader-id= 2026-03-31T06:16:20.998Z [WARN] server-6163e78d-d2c6-0207-021c-580c19332fd3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:20.998Z [INFO] server-6163e78d-d2c6-0207-021c-580c19332fd3: entering candidate state: node="Node at 6163e78d-d2c6-0207-021c-580c19332fd3 [Candidate]" term=2 2026-03-31T06:16:20.998Z [INFO] server-6163e78d-d2c6-0207-021c-580c19332fd3: election won: term=2 tally=2 2026-03-31T06:16:20.998Z [INFO] server-6163e78d-d2c6-0207-021c-580c19332fd3: entering leader state: leader="Node at 6163e78d-d2c6-0207-021c-580c19332fd3 [Leader]" 2026-03-31T06:16:20.998Z [INFO] server-6163e78d-d2c6-0207-021c-580c19332fd3: added peer, starting replication: peer=server-9335ae3d-a47c-730f-dda1-bb94a4c94c3f 2026-03-31T06:16:20.998Z [INFO] server-6163e78d-d2c6-0207-021c-580c19332fd3: pipelining replication: peer="{Voter server-9335ae3d-a47c-730f-dda1-bb94a4c94c3f 9335ae3d-a47c-730f-dda1-bb94a4c94c3f}" 2026-03-31T06:16:21.034Z [WARN] server-6e984533-1245-9731-be03-46cba4baba10: no known peers, aborting election 2026-03-31T06:16:21.104Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:20.99881058 +0000 UTC m=+9.276417961 (1 nodes), 56.06148ms from start of poll, 56.52536ms from cluster start. Timeout at 2026-03-31 06:16:21.10381398 +0000 UTC m=+9.381421361, 105.0034ms after stability 2026-03-31T06:16:21.104Z [INFO] server-6163e78d-d2c6-0207-021c-580c19332fd3: updating configuration: command=AddVoter server-id=server-6e984533-1245-9731-be03-46cba4baba10 server-addr=6e984533-1245-9731-be03-46cba4baba10 servers="[{Suffrage:Voter ID:server-9335ae3d-a47c-730f-dda1-bb94a4c94c3f Address:9335ae3d-a47c-730f-dda1-bb94a4c94c3f} {Suffrage:Voter ID:server-6163e78d-d2c6-0207-021c-580c19332fd3 Address:6163e78d-d2c6-0207-021c-580c19332fd3} {Suffrage:Voter ID:server-6e984533-1245-9731-be03-46cba4baba10 Address:6e984533-1245-9731-be03-46cba4baba10}]" 2026-03-31T06:16:21.104Z [INFO] server-6163e78d-d2c6-0207-021c-580c19332fd3: added peer, starting replication: peer=server-6e984533-1245-9731-be03-46cba4baba10 2026-03-31T06:16:21.104Z [WARN] server-6e984533-1245-9731-be03-46cba4baba10: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2026-03-31T06:16:21.104Z [WARN] server-6163e78d-d2c6-0207-021c-580c19332fd3: appendEntries rejected, sending older logs: peer="{Voter server-6e984533-1245-9731-be03-46cba4baba10 6e984533-1245-9731-be03-46cba4baba10}" next=1 2026-03-31T06:16:21.105Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:21.105Z [INFO] server-6163e78d-d2c6-0207-021c-580c19332fd3: pipelining replication: peer="{Voter server-6e984533-1245-9731-be03-46cba4baba10 6e984533-1245-9731-be03-46cba4baba10}" 2026-03-31T06:16:21.210Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:21.10510234 +0000 UTC m=+9.382709721 (1 nodes), 1.08µs from start of poll, 162.81712ms from cluster start. Timeout at 2026-03-31 06:16:21.21010614 +0000 UTC m=+9.487713521, 105.0038ms after stability 2026-03-31T06:16:21.210Z [INFO] server-6163e78d-d2c6-0207-021c-580c19332fd3: aborting pipeline replication: peer="{Voter server-9335ae3d-a47c-730f-dda1-bb94a4c94c3f 9335ae3d-a47c-730f-dda1-bb94a4c94c3f}" 2026-03-31T06:16:21.210Z [INFO] server-6163e78d-d2c6-0207-021c-580c19332fd3: aborting pipeline replication: peer="{Voter server-6e984533-1245-9731-be03-46cba4baba10 6e984533-1245-9731-be03-46cba4baba10}" --- PASS: TestRaft_JoinNode (0.27s) === RUN TestRaft_JoinNode_ConfigStore 2026-03-31T06:16:21.211Z [INFO] server-58fdd232-3ed7-6d1d-fb99-4a69719cef94: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58fdd232-3ed7-6d1d-fb99-4a69719cef94 Address:58fdd232-3ed7-6d1d-fb99-4a69719cef94}]" 2026-03-31T06:16:21.211Z [INFO] server-58fdd232-3ed7-6d1d-fb99-4a69719cef94: entering follower state: follower="Node at 58fdd232-3ed7-6d1d-fb99-4a69719cef94 [Follower]" leader-address= leader-id= 2026-03-31T06:16:21.211Z [INFO] server-b88643c9-13dd-bac9-6875-4f9622c41007: initial configuration: index=0 servers=[] 2026-03-31T06:16:21.211Z [INFO] server-b88643c9-13dd-bac9-6875-4f9622c41007: entering follower state: follower="Node at b88643c9-13dd-bac9-6875-4f9622c41007 [Follower]" leader-address= leader-id= 2026-03-31T06:16:21.211Z [INFO] server-f27d6332-7234-8e36-704a-aae9f4b32b16: initial configuration: index=0 servers=[] 2026-03-31T06:16:21.211Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:21.211Z [INFO] server-f27d6332-7234-8e36-704a-aae9f4b32b16: entering follower state: follower="Node at f27d6332-7234-8e36-704a-aae9f4b32b16 [Follower]" leader-address= leader-id= 2026-03-31T06:16:21.274Z [WARN] server-b88643c9-13dd-bac9-6875-4f9622c41007: no known peers, aborting election 2026-03-31T06:16:21.295Z [WARN] server-f27d6332-7234-8e36-704a-aae9f4b32b16: no known peers, aborting election 2026-03-31T06:16:21.296Z [WARN] server-58fdd232-3ed7-6d1d-fb99-4a69719cef94: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:21.296Z [INFO] server-58fdd232-3ed7-6d1d-fb99-4a69719cef94: entering candidate state: node="Node at 58fdd232-3ed7-6d1d-fb99-4a69719cef94 [Candidate]" term=2 2026-03-31T06:16:21.296Z [INFO] server-58fdd232-3ed7-6d1d-fb99-4a69719cef94: election won: term=2 tally=1 2026-03-31T06:16:21.296Z [INFO] server-58fdd232-3ed7-6d1d-fb99-4a69719cef94: entering leader state: leader="Node at 58fdd232-3ed7-6d1d-fb99-4a69719cef94 [Leader]" 2026-03-31T06:16:21.402Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:21.29681234 +0000 UTC m=+9.574419721 (1 nodes), 85.0566ms from start of poll, 85.56656ms from cluster start. Timeout at 2026-03-31 06:16:21.4018137 +0000 UTC m=+9.679421081, 105.00136ms after stability 2026-03-31T06:16:21.402Z [INFO] server-58fdd232-3ed7-6d1d-fb99-4a69719cef94: updating configuration: command=AddVoter server-id=server-b88643c9-13dd-bac9-6875-4f9622c41007 server-addr=b88643c9-13dd-bac9-6875-4f9622c41007 servers="[{Suffrage:Voter ID:server-58fdd232-3ed7-6d1d-fb99-4a69719cef94 Address:58fdd232-3ed7-6d1d-fb99-4a69719cef94} {Suffrage:Voter ID:server-b88643c9-13dd-bac9-6875-4f9622c41007 Address:b88643c9-13dd-bac9-6875-4f9622c41007}]" 2026-03-31T06:16:21.402Z [INFO] server-58fdd232-3ed7-6d1d-fb99-4a69719cef94: added peer, starting replication: peer=server-b88643c9-13dd-bac9-6875-4f9622c41007 2026-03-31T06:16:21.402Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:21.402Z [WARN] server-b88643c9-13dd-bac9-6875-4f9622c41007: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2026-03-31T06:16:21.402Z [WARN] server-58fdd232-3ed7-6d1d-fb99-4a69719cef94: appendEntries rejected, sending older logs: peer="{Voter server-b88643c9-13dd-bac9-6875-4f9622c41007 b88643c9-13dd-bac9-6875-4f9622c41007}" next=1 2026-03-31T06:16:21.402Z [INFO] server-58fdd232-3ed7-6d1d-fb99-4a69719cef94: pipelining replication: peer="{Voter server-b88643c9-13dd-bac9-6875-4f9622c41007 b88643c9-13dd-bac9-6875-4f9622c41007}" 2026-03-31T06:16:21.508Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:21.4026179 +0000 UTC m=+9.680225281 (1 nodes), 520ns from start of poll, 191.37212ms from cluster start. Timeout at 2026-03-31 06:16:21.5076233 +0000 UTC m=+9.785230681, 105.0054ms after stability 2026-03-31T06:16:21.508Z [INFO] server-58fdd232-3ed7-6d1d-fb99-4a69719cef94: updating configuration: command=AddVoter server-id=server-f27d6332-7234-8e36-704a-aae9f4b32b16 server-addr=f27d6332-7234-8e36-704a-aae9f4b32b16 servers="[{Suffrage:Voter ID:server-58fdd232-3ed7-6d1d-fb99-4a69719cef94 Address:58fdd232-3ed7-6d1d-fb99-4a69719cef94} {Suffrage:Voter ID:server-b88643c9-13dd-bac9-6875-4f9622c41007 Address:b88643c9-13dd-bac9-6875-4f9622c41007} {Suffrage:Voter ID:server-f27d6332-7234-8e36-704a-aae9f4b32b16 Address:f27d6332-7234-8e36-704a-aae9f4b32b16}]" 2026-03-31T06:16:21.508Z [INFO] server-58fdd232-3ed7-6d1d-fb99-4a69719cef94: added peer, starting replication: peer=server-f27d6332-7234-8e36-704a-aae9f4b32b16 2026-03-31T06:16:21.508Z [WARN] server-f27d6332-7234-8e36-704a-aae9f4b32b16: failed to get previous log: previous-index=4 last-index=0 error="log not found" 2026-03-31T06:16:21.509Z [WARN] server-58fdd232-3ed7-6d1d-fb99-4a69719cef94: appendEntries rejected, sending older logs: peer="{Voter server-f27d6332-7234-8e36-704a-aae9f4b32b16 f27d6332-7234-8e36-704a-aae9f4b32b16}" next=1 2026-03-31T06:16:21.509Z [INFO] server-58fdd232-3ed7-6d1d-fb99-4a69719cef94: pipelining replication: peer="{Voter server-f27d6332-7234-8e36-704a-aae9f4b32b16 f27d6332-7234-8e36-704a-aae9f4b32b16}" 2026-03-31T06:16:21.509Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:21.614Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:21.50921238 +0000 UTC m=+9.786819761 (1 nodes), 720ns from start of poll, 297.9666ms from cluster start. Timeout at 2026-03-31 06:16:21.61421682 +0000 UTC m=+9.891824201, 105.00444ms after stability 2026-03-31T06:16:21.614Z [INFO] server-58fdd232-3ed7-6d1d-fb99-4a69719cef94: aborting pipeline replication: peer="{Voter server-f27d6332-7234-8e36-704a-aae9f4b32b16 f27d6332-7234-8e36-704a-aae9f4b32b16}" 2026-03-31T06:16:21.614Z [INFO] server-58fdd232-3ed7-6d1d-fb99-4a69719cef94: aborting pipeline replication: peer="{Voter server-b88643c9-13dd-bac9-6875-4f9622c41007 b88643c9-13dd-bac9-6875-4f9622c41007}" --- PASS: TestRaft_JoinNode_ConfigStore (0.40s) === RUN TestRaft_RemoveFollower 2026-03-31T06:16:21.615Z [INFO] server-80f53b23-acd0-3cc7-7ae2-87ba2a93a672: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-80f53b23-acd0-3cc7-7ae2-87ba2a93a672 Address:80f53b23-acd0-3cc7-7ae2-87ba2a93a672} {Suffrage:Voter ID:server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9 Address:ae99ad6e-813e-bb0a-abb8-e2b73d7148a9} {Suffrage:Voter ID:server-744f63e4-fb77-a40c-c79d-75a8ff36341e Address:744f63e4-fb77-a40c-c79d-75a8ff36341e}]" 2026-03-31T06:16:21.615Z [INFO] server-80f53b23-acd0-3cc7-7ae2-87ba2a93a672: entering follower state: follower="Node at 80f53b23-acd0-3cc7-7ae2-87ba2a93a672 [Follower]" leader-address= leader-id= 2026-03-31T06:16:21.615Z [INFO] server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-80f53b23-acd0-3cc7-7ae2-87ba2a93a672 Address:80f53b23-acd0-3cc7-7ae2-87ba2a93a672} {Suffrage:Voter ID:server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9 Address:ae99ad6e-813e-bb0a-abb8-e2b73d7148a9} {Suffrage:Voter ID:server-744f63e4-fb77-a40c-c79d-75a8ff36341e Address:744f63e4-fb77-a40c-c79d-75a8ff36341e}]" 2026-03-31T06:16:21.615Z [INFO] server-744f63e4-fb77-a40c-c79d-75a8ff36341e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-80f53b23-acd0-3cc7-7ae2-87ba2a93a672 Address:80f53b23-acd0-3cc7-7ae2-87ba2a93a672} {Suffrage:Voter ID:server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9 Address:ae99ad6e-813e-bb0a-abb8-e2b73d7148a9} {Suffrage:Voter ID:server-744f63e4-fb77-a40c-c79d-75a8ff36341e Address:744f63e4-fb77-a40c-c79d-75a8ff36341e}]" 2026-03-31T06:16:21.615Z [INFO] server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9: entering follower state: follower="Node at ae99ad6e-813e-bb0a-abb8-e2b73d7148a9 [Follower]" leader-address= leader-id= 2026-03-31T06:16:21.615Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:21.615Z [INFO] server-744f63e4-fb77-a40c-c79d-75a8ff36341e: entering follower state: follower="Node at 744f63e4-fb77-a40c-c79d-75a8ff36341e [Follower]" leader-address= leader-id= 2026-03-31T06:16:21.666Z [WARN] server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:21.666Z [INFO] server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9: entering candidate state: node="Node at ae99ad6e-813e-bb0a-abb8-e2b73d7148a9 [Candidate]" term=2 2026-03-31T06:16:21.666Z [INFO] server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9: election won: term=2 tally=2 2026-03-31T06:16:21.666Z [INFO] server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9: entering leader state: leader="Node at ae99ad6e-813e-bb0a-abb8-e2b73d7148a9 [Leader]" 2026-03-31T06:16:21.666Z [INFO] server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9: added peer, starting replication: peer=server-80f53b23-acd0-3cc7-7ae2-87ba2a93a672 2026-03-31T06:16:21.666Z [INFO] server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9: added peer, starting replication: peer=server-744f63e4-fb77-a40c-c79d-75a8ff36341e 2026-03-31T06:16:21.667Z [INFO] server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9: pipelining replication: peer="{Voter server-80f53b23-acd0-3cc7-7ae2-87ba2a93a672 80f53b23-acd0-3cc7-7ae2-87ba2a93a672}" 2026-03-31T06:16:21.667Z [INFO] server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9: pipelining replication: peer="{Voter server-744f63e4-fb77-a40c-c79d-75a8ff36341e 744f63e4-fb77-a40c-c79d-75a8ff36341e}" 2026-03-31T06:16:21.772Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:21.66741822 +0000 UTC m=+9.945025601 (1 nodes), 51.6744ms from start of poll, 52.12216ms from cluster start. Timeout at 2026-03-31 06:16:21.7724201 +0000 UTC m=+10.050027481, 105.00188ms after stability 2026-03-31T06:16:21.778Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:21.884Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:21.7786595 +0000 UTC m=+10.056266881 (2 nodes), 2.32µs from start of poll, 163.36344ms from cluster start. Timeout at 2026-03-31 06:16:21.88366522 +0000 UTC m=+10.161272601, 105.00572ms after stability 2026-03-31T06:16:21.884Z [INFO] server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9: updating configuration: command=RemoveServer server-id=server-80f53b23-acd0-3cc7-7ae2-87ba2a93a672 server-addr= servers="[{Suffrage:Voter ID:server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9 Address:ae99ad6e-813e-bb0a-abb8-e2b73d7148a9} {Suffrage:Voter ID:server-744f63e4-fb77-a40c-c79d-75a8ff36341e Address:744f63e4-fb77-a40c-c79d-75a8ff36341e}]" 2026-03-31T06:16:21.884Z [INFO] server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9: removed peer, stopping replication: peer=server-80f53b23-acd0-3cc7-7ae2-87ba2a93a672 last-index=3 2026-03-31T06:16:21.884Z [INFO] server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9: aborting pipeline replication: peer="{Voter server-80f53b23-acd0-3cc7-7ae2-87ba2a93a672 80f53b23-acd0-3cc7-7ae2-87ba2a93a672}" 2026-03-31T06:16:21.977Z [WARN] server-80f53b23-acd0-3cc7-7ae2-87ba2a93a672: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2026-03-31T06:16:21.990Z [INFO] server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9: aborting pipeline replication: peer="{Voter server-744f63e4-fb77-a40c-c79d-75a8ff36341e 744f63e4-fb77-a40c-c79d-75a8ff36341e}" 2026-03-31T06:16:22.491Z [ERROR] server-ae99ad6e-813e-bb0a-abb8-e2b73d7148a9: failed to heartbeat to: peer=744f63e4-fb77-a40c-c79d-75a8ff36341e backoff time=10ms error="command timed out" --- PASS: TestRaft_RemoveFollower (0.88s) === RUN TestRaft_RemoveLeader 2026-03-31T06:16:22.492Z [INFO] server-1eabc591-ab00-eea7-12bd-f3853f88487f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1eabc591-ab00-eea7-12bd-f3853f88487f Address:1eabc591-ab00-eea7-12bd-f3853f88487f} {Suffrage:Voter ID:server-817ad916-9f5d-19f3-3db9-d139a195b3f6 Address:817ad916-9f5d-19f3-3db9-d139a195b3f6} {Suffrage:Voter ID:server-151608f6-3f1c-c194-5bc1-77aac3d79581 Address:151608f6-3f1c-c194-5bc1-77aac3d79581}]" 2026-03-31T06:16:22.492Z [INFO] server-817ad916-9f5d-19f3-3db9-d139a195b3f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1eabc591-ab00-eea7-12bd-f3853f88487f Address:1eabc591-ab00-eea7-12bd-f3853f88487f} {Suffrage:Voter ID:server-817ad916-9f5d-19f3-3db9-d139a195b3f6 Address:817ad916-9f5d-19f3-3db9-d139a195b3f6} {Suffrage:Voter ID:server-151608f6-3f1c-c194-5bc1-77aac3d79581 Address:151608f6-3f1c-c194-5bc1-77aac3d79581}]" 2026-03-31T06:16:22.492Z [INFO] server-1eabc591-ab00-eea7-12bd-f3853f88487f: entering follower state: follower="Node at 1eabc591-ab00-eea7-12bd-f3853f88487f [Follower]" leader-address= leader-id= 2026-03-31T06:16:22.492Z [INFO] server-151608f6-3f1c-c194-5bc1-77aac3d79581: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1eabc591-ab00-eea7-12bd-f3853f88487f Address:1eabc591-ab00-eea7-12bd-f3853f88487f} {Suffrage:Voter ID:server-817ad916-9f5d-19f3-3db9-d139a195b3f6 Address:817ad916-9f5d-19f3-3db9-d139a195b3f6} {Suffrage:Voter ID:server-151608f6-3f1c-c194-5bc1-77aac3d79581 Address:151608f6-3f1c-c194-5bc1-77aac3d79581}]" 2026-03-31T06:16:22.492Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:22.492Z [INFO] server-817ad916-9f5d-19f3-3db9-d139a195b3f6: entering follower state: follower="Node at 817ad916-9f5d-19f3-3db9-d139a195b3f6 [Follower]" leader-address= leader-id= 2026-03-31T06:16:22.492Z [INFO] server-151608f6-3f1c-c194-5bc1-77aac3d79581: entering follower state: follower="Node at 151608f6-3f1c-c194-5bc1-77aac3d79581 [Follower]" leader-address= leader-id= 2026-03-31T06:16:22.558Z [WARN] server-817ad916-9f5d-19f3-3db9-d139a195b3f6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:22.558Z [INFO] server-817ad916-9f5d-19f3-3db9-d139a195b3f6: entering candidate state: node="Node at 817ad916-9f5d-19f3-3db9-d139a195b3f6 [Candidate]" term=2 2026-03-31T06:16:22.558Z [INFO] server-817ad916-9f5d-19f3-3db9-d139a195b3f6: election won: term=2 tally=2 2026-03-31T06:16:22.558Z [INFO] server-817ad916-9f5d-19f3-3db9-d139a195b3f6: entering leader state: leader="Node at 817ad916-9f5d-19f3-3db9-d139a195b3f6 [Leader]" 2026-03-31T06:16:22.558Z [INFO] server-817ad916-9f5d-19f3-3db9-d139a195b3f6: added peer, starting replication: peer=server-1eabc591-ab00-eea7-12bd-f3853f88487f 2026-03-31T06:16:22.558Z [INFO] server-817ad916-9f5d-19f3-3db9-d139a195b3f6: added peer, starting replication: peer=server-151608f6-3f1c-c194-5bc1-77aac3d79581 2026-03-31T06:16:22.558Z [INFO] server-817ad916-9f5d-19f3-3db9-d139a195b3f6: pipelining replication: peer="{Voter server-151608f6-3f1c-c194-5bc1-77aac3d79581 151608f6-3f1c-c194-5bc1-77aac3d79581}" 2026-03-31T06:16:22.558Z [INFO] server-817ad916-9f5d-19f3-3db9-d139a195b3f6: pipelining replication: peer="{Voter server-1eabc591-ab00-eea7-12bd-f3853f88487f 1eabc591-ab00-eea7-12bd-f3853f88487f}" 2026-03-31T06:16:22.663Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:22.55820138 +0000 UTC m=+10.835808801 (1 nodes), 65.68236ms from start of poll, 66.078ms from cluster start. Timeout at 2026-03-31 06:16:22.6632033 +0000 UTC m=+10.940810721, 105.00192ms after stability 2026-03-31T06:16:22.669Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:22.774Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:22.66937326 +0000 UTC m=+10.946980681 (2 nodes), 1.2µs from start of poll, 177.24988ms from cluster start. Timeout at 2026-03-31 06:16:22.77437758 +0000 UTC m=+11.051984961, 105.00428ms after stability 2026-03-31T06:16:22.774Z [INFO] server-817ad916-9f5d-19f3-3db9-d139a195b3f6: updating configuration: command=RemoveServer server-id=server-817ad916-9f5d-19f3-3db9-d139a195b3f6 server-addr= servers="[{Suffrage:Voter ID:server-1eabc591-ab00-eea7-12bd-f3853f88487f Address:1eabc591-ab00-eea7-12bd-f3853f88487f} {Suffrage:Voter ID:server-151608f6-3f1c-c194-5bc1-77aac3d79581 Address:151608f6-3f1c-c194-5bc1-77aac3d79581}]" 2026-03-31T06:16:22.774Z [INFO] server-817ad916-9f5d-19f3-3db9-d139a195b3f6: removed ourself, shutting down 2026-03-31T06:16:22.774Z [INFO] server-817ad916-9f5d-19f3-3db9-d139a195b3f6: aborting pipeline replication: peer="{Voter server-151608f6-3f1c-c194-5bc1-77aac3d79581 151608f6-3f1c-c194-5bc1-77aac3d79581}" 2026-03-31T06:16:22.775Z [INFO] server-817ad916-9f5d-19f3-3db9-d139a195b3f6: aborting pipeline replication: peer="{Voter server-1eabc591-ab00-eea7-12bd-f3853f88487f 1eabc591-ab00-eea7-12bd-f3853f88487f}" 2026-03-31T06:16:22.846Z [WARN] server-1eabc591-ab00-eea7-12bd-f3853f88487f: heartbeat timeout reached, starting election: last-leader-addr=817ad916-9f5d-19f3-3db9-d139a195b3f6 last-leader-id=server-817ad916-9f5d-19f3-3db9-d139a195b3f6 2026-03-31T06:16:22.846Z [INFO] server-1eabc591-ab00-eea7-12bd-f3853f88487f: entering candidate state: node="Node at 1eabc591-ab00-eea7-12bd-f3853f88487f [Candidate]" term=3 2026-03-31T06:16:22.846Z [WARN] server-151608f6-3f1c-c194-5bc1-77aac3d79581: rejecting vote request since we have a leader: from=1eabc591-ab00-eea7-12bd-f3853f88487f leader=817ad916-9f5d-19f3-3db9-d139a195b3f6 leader-id=server-817ad916-9f5d-19f3-3db9-d139a195b3f6 2026-03-31T06:16:22.847Z [WARN] server-151608f6-3f1c-c194-5bc1-77aac3d79581: heartbeat timeout reached, starting election: last-leader-addr=817ad916-9f5d-19f3-3db9-d139a195b3f6 last-leader-id=server-817ad916-9f5d-19f3-3db9-d139a195b3f6 2026-03-31T06:16:22.847Z [INFO] server-151608f6-3f1c-c194-5bc1-77aac3d79581: entering candidate state: node="Node at 151608f6-3f1c-c194-5bc1-77aac3d79581 [Candidate]" term=3 2026-03-31T06:16:22.847Z [INFO] server-1eabc591-ab00-eea7-12bd-f3853f88487f: duplicate requestVote for same term: term=3 2026-03-31T06:16:22.908Z [WARN] server-151608f6-3f1c-c194-5bc1-77aac3d79581: Election timeout reached, restarting election 2026-03-31T06:16:22.908Z [INFO] server-151608f6-3f1c-c194-5bc1-77aac3d79581: entering candidate state: node="Node at 151608f6-3f1c-c194-5bc1-77aac3d79581 [Candidate]" term=4 2026-03-31T06:16:22.909Z [INFO] server-1eabc591-ab00-eea7-12bd-f3853f88487f: entering follower state: follower="Node at 1eabc591-ab00-eea7-12bd-f3853f88487f [Follower]" leader-address= leader-id= 2026-03-31T06:16:22.909Z [INFO] server-151608f6-3f1c-c194-5bc1-77aac3d79581: election won: term=4 tally=2 2026-03-31T06:16:22.909Z [INFO] server-151608f6-3f1c-c194-5bc1-77aac3d79581: entering leader state: leader="Node at 151608f6-3f1c-c194-5bc1-77aac3d79581 [Leader]" 2026-03-31T06:16:22.909Z [INFO] server-151608f6-3f1c-c194-5bc1-77aac3d79581: added peer, starting replication: peer=server-1eabc591-ab00-eea7-12bd-f3853f88487f 2026-03-31T06:16:22.909Z [INFO] server-151608f6-3f1c-c194-5bc1-77aac3d79581: pipelining replication: peer="{Voter server-1eabc591-ab00-eea7-12bd-f3853f88487f 1eabc591-ab00-eea7-12bd-f3853f88487f}" 2026-03-31T06:16:22.986Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:23.091Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:22.98633582 +0000 UTC m=+11.263943201 (1 nodes), 71.44µs from start of poll, 494.2124ms from cluster start. Timeout at 2026-03-31 06:16:23.09134618 +0000 UTC m=+11.368953561, 105.01036ms after stability 2026-03-31T06:16:23.091Z [INFO] server-151608f6-3f1c-c194-5bc1-77aac3d79581: aborting pipeline replication: peer="{Voter server-1eabc591-ab00-eea7-12bd-f3853f88487f 1eabc591-ab00-eea7-12bd-f3853f88487f}" --- PASS: TestRaft_RemoveLeader (0.60s) === RUN TestRaft_RemoveLeader_NoShutdown 2026-03-31T06:16:23.092Z [INFO] server-206e8035-06fc-d2ab-83c8-804b52c0b508: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-206e8035-06fc-d2ab-83c8-804b52c0b508 Address:206e8035-06fc-d2ab-83c8-804b52c0b508} {Suffrage:Voter ID:server-733a912c-14fc-08d5-3e8e-a79353ae5d05 Address:733a912c-14fc-08d5-3e8e-a79353ae5d05} {Suffrage:Voter ID:server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1 Address:093a8f37-29dd-6357-7fc7-f9ef53ae6bc1}]" 2026-03-31T06:16:23.092Z [INFO] server-206e8035-06fc-d2ab-83c8-804b52c0b508: entering follower state: follower="Node at 206e8035-06fc-d2ab-83c8-804b52c0b508 [Follower]" leader-address= leader-id= 2026-03-31T06:16:23.092Z [INFO] server-733a912c-14fc-08d5-3e8e-a79353ae5d05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-206e8035-06fc-d2ab-83c8-804b52c0b508 Address:206e8035-06fc-d2ab-83c8-804b52c0b508} {Suffrage:Voter ID:server-733a912c-14fc-08d5-3e8e-a79353ae5d05 Address:733a912c-14fc-08d5-3e8e-a79353ae5d05} {Suffrage:Voter ID:server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1 Address:093a8f37-29dd-6357-7fc7-f9ef53ae6bc1}]" 2026-03-31T06:16:23.092Z [INFO] server-733a912c-14fc-08d5-3e8e-a79353ae5d05: entering follower state: follower="Node at 733a912c-14fc-08d5-3e8e-a79353ae5d05 [Follower]" leader-address= leader-id= 2026-03-31T06:16:23.092Z [INFO] server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-206e8035-06fc-d2ab-83c8-804b52c0b508 Address:206e8035-06fc-d2ab-83c8-804b52c0b508} {Suffrage:Voter ID:server-733a912c-14fc-08d5-3e8e-a79353ae5d05 Address:733a912c-14fc-08d5-3e8e-a79353ae5d05} {Suffrage:Voter ID:server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1 Address:093a8f37-29dd-6357-7fc7-f9ef53ae6bc1}]" 2026-03-31T06:16:23.092Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:23.092Z [INFO] server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1: entering follower state: follower="Node at 093a8f37-29dd-6357-7fc7-f9ef53ae6bc1 [Follower]" leader-address= leader-id= 2026-03-31T06:16:23.158Z [WARN] server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:23.158Z [INFO] server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1: entering candidate state: node="Node at 093a8f37-29dd-6357-7fc7-f9ef53ae6bc1 [Candidate]" term=2 2026-03-31T06:16:23.158Z [INFO] server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1: election won: term=2 tally=2 2026-03-31T06:16:23.158Z [INFO] server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1: entering leader state: leader="Node at 093a8f37-29dd-6357-7fc7-f9ef53ae6bc1 [Leader]" 2026-03-31T06:16:23.158Z [INFO] server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1: added peer, starting replication: peer=server-206e8035-06fc-d2ab-83c8-804b52c0b508 2026-03-31T06:16:23.158Z [INFO] server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1: added peer, starting replication: peer=server-733a912c-14fc-08d5-3e8e-a79353ae5d05 2026-03-31T06:16:23.158Z [INFO] server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1: pipelining replication: peer="{Voter server-733a912c-14fc-08d5-3e8e-a79353ae5d05 733a912c-14fc-08d5-3e8e-a79353ae5d05}" 2026-03-31T06:16:23.158Z [INFO] server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1: pipelining replication: peer="{Voter server-206e8035-06fc-d2ab-83c8-804b52c0b508 206e8035-06fc-d2ab-83c8-804b52c0b508}" 2026-03-31T06:16:23.264Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:23.15883102 +0000 UTC m=+11.436438401 (2 nodes), 65.88564ms from start of poll, 66.43092ms from cluster start. Timeout at 2026-03-31 06:16:23.26383362 +0000 UTC m=+11.541441001, 105.0026ms after stability 2026-03-31T06:16:23.264Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:23.371Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:23.26496006 +0000 UTC m=+11.542567441 (1 nodes), 1.92µs from start of poll, 172.55996ms from cluster start. Timeout at 2026-03-31 06:16:23.36996422 +0000 UTC m=+11.647571561, 105.00412ms after stability 2026-03-31T06:16:23.375Z [INFO] server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1: updating configuration: command=RemoveServer server-id=server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1 server-addr= servers="[{Suffrage:Voter ID:server-206e8035-06fc-d2ab-83c8-804b52c0b508 Address:206e8035-06fc-d2ab-83c8-804b52c0b508} {Suffrage:Voter ID:server-733a912c-14fc-08d5-3e8e-a79353ae5d05 Address:733a912c-14fc-08d5-3e8e-a79353ae5d05}]" 2026-03-31T06:16:23.376Z [INFO] server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1: removed ourself, transitioning to follower 2026-03-31T06:16:23.376Z [INFO] server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1: entering follower state: follower="Node at 093a8f37-29dd-6357-7fc7-f9ef53ae6bc1 [Follower]" leader-address= leader-id= 2026-03-31T06:16:23.376Z [INFO] server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1: aborting pipeline replication: peer="{Voter server-733a912c-14fc-08d5-3e8e-a79353ae5d05 733a912c-14fc-08d5-3e8e-a79353ae5d05}" 2026-03-31T06:16:23.376Z [INFO] server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1: aborting pipeline replication: peer="{Voter server-206e8035-06fc-d2ab-83c8-804b52c0b508 206e8035-06fc-d2ab-83c8-804b52c0b508}" 2026-03-31T06:16:23.441Z [WARN] server-733a912c-14fc-08d5-3e8e-a79353ae5d05: heartbeat timeout reached, starting election: last-leader-addr=093a8f37-29dd-6357-7fc7-f9ef53ae6bc1 last-leader-id=server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1 2026-03-31T06:16:23.441Z [INFO] server-733a912c-14fc-08d5-3e8e-a79353ae5d05: entering candidate state: node="Node at 733a912c-14fc-08d5-3e8e-a79353ae5d05 [Candidate]" term=3 2026-03-31T06:16:23.441Z [WARN] server-206e8035-06fc-d2ab-83c8-804b52c0b508: rejecting vote request since we have a leader: from=733a912c-14fc-08d5-3e8e-a79353ae5d05 leader=093a8f37-29dd-6357-7fc7-f9ef53ae6bc1 leader-id=server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1 2026-03-31T06:16:23.451Z [WARN] server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1: not part of stable configuration, aborting election 2026-03-31T06:16:23.482Z [WARN] server-206e8035-06fc-d2ab-83c8-804b52c0b508: heartbeat timeout reached, starting election: last-leader-addr=093a8f37-29dd-6357-7fc7-f9ef53ae6bc1 last-leader-id=server-093a8f37-29dd-6357-7fc7-f9ef53ae6bc1 2026-03-31T06:16:23.482Z [INFO] server-206e8035-06fc-d2ab-83c8-804b52c0b508: entering candidate state: node="Node at 206e8035-06fc-d2ab-83c8-804b52c0b508 [Candidate]" term=3 2026-03-31T06:16:23.482Z [INFO] server-733a912c-14fc-08d5-3e8e-a79353ae5d05: duplicate requestVote for same term: term=3 2026-03-31T06:16:23.482Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:23.530Z [WARN] server-733a912c-14fc-08d5-3e8e-a79353ae5d05: Election timeout reached, restarting election 2026-03-31T06:16:23.530Z [INFO] server-733a912c-14fc-08d5-3e8e-a79353ae5d05: entering candidate state: node="Node at 733a912c-14fc-08d5-3e8e-a79353ae5d05 [Candidate]" term=4 2026-03-31T06:16:23.530Z [INFO] server-206e8035-06fc-d2ab-83c8-804b52c0b508: entering follower state: follower="Node at 206e8035-06fc-d2ab-83c8-804b52c0b508 [Follower]" leader-address= leader-id= 2026-03-31T06:16:23.530Z [INFO] server-733a912c-14fc-08d5-3e8e-a79353ae5d05: election won: term=4 tally=2 2026-03-31T06:16:23.530Z [INFO] server-733a912c-14fc-08d5-3e8e-a79353ae5d05: entering leader state: leader="Node at 733a912c-14fc-08d5-3e8e-a79353ae5d05 [Leader]" 2026-03-31T06:16:23.530Z [INFO] server-733a912c-14fc-08d5-3e8e-a79353ae5d05: added peer, starting replication: peer=server-206e8035-06fc-d2ab-83c8-804b52c0b508 2026-03-31T06:16:23.530Z [INFO] server-733a912c-14fc-08d5-3e8e-a79353ae5d05: pipelining replication: peer="{Voter server-206e8035-06fc-d2ab-83c8-804b52c0b508 206e8035-06fc-d2ab-83c8-804b52c0b508}" 2026-03-31T06:16:23.636Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:23.53061138 +0000 UTC m=+11.808218801 (1 nodes), 48.19668ms from start of poll, 438.21132ms from cluster start. Timeout at 2026-03-31 06:16:23.63561314 +0000 UTC m=+11.913220521, 105.00172ms after stability 2026-03-31T06:16:23.742Z [INFO] server-733a912c-14fc-08d5-3e8e-a79353ae5d05: aborting pipeline replication: peer="{Voter server-206e8035-06fc-d2ab-83c8-804b52c0b508 206e8035-06fc-d2ab-83c8-804b52c0b508}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.65s) === RUN TestRaft_RemoveFollower_SplitCluster 2026-03-31T06:16:23.743Z [INFO] server-2d50918c-128e-a400-4c46-c2c4410f26c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d50918c-128e-a400-4c46-c2c4410f26c9 Address:2d50918c-128e-a400-4c46-c2c4410f26c9} {Suffrage:Voter ID:server-8ab3fd73-750c-5061-ffdc-b419fcbc9a81 Address:8ab3fd73-750c-5061-ffdc-b419fcbc9a81} {Suffrage:Voter ID:server-3d1b2d19-c112-cbdd-d59f-a1c10ba297e4 Address:3d1b2d19-c112-cbdd-d59f-a1c10ba297e4} {Suffrage:Voter ID:server-6fdca903-eef1-a757-3da1-a30a9fe3095e Address:6fdca903-eef1-a757-3da1-a30a9fe3095e}]" 2026-03-31T06:16:23.743Z [INFO] server-2d50918c-128e-a400-4c46-c2c4410f26c9: entering follower state: follower="Node at 2d50918c-128e-a400-4c46-c2c4410f26c9 [Follower]" leader-address= leader-id= 2026-03-31T06:16:23.743Z [INFO] server-8ab3fd73-750c-5061-ffdc-b419fcbc9a81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d50918c-128e-a400-4c46-c2c4410f26c9 Address:2d50918c-128e-a400-4c46-c2c4410f26c9} {Suffrage:Voter ID:server-8ab3fd73-750c-5061-ffdc-b419fcbc9a81 Address:8ab3fd73-750c-5061-ffdc-b419fcbc9a81} {Suffrage:Voter ID:server-3d1b2d19-c112-cbdd-d59f-a1c10ba297e4 Address:3d1b2d19-c112-cbdd-d59f-a1c10ba297e4} {Suffrage:Voter ID:server-6fdca903-eef1-a757-3da1-a30a9fe3095e Address:6fdca903-eef1-a757-3da1-a30a9fe3095e}]" 2026-03-31T06:16:23.743Z [INFO] server-3d1b2d19-c112-cbdd-d59f-a1c10ba297e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d50918c-128e-a400-4c46-c2c4410f26c9 Address:2d50918c-128e-a400-4c46-c2c4410f26c9} {Suffrage:Voter ID:server-8ab3fd73-750c-5061-ffdc-b419fcbc9a81 Address:8ab3fd73-750c-5061-ffdc-b419fcbc9a81} {Suffrage:Voter ID:server-3d1b2d19-c112-cbdd-d59f-a1c10ba297e4 Address:3d1b2d19-c112-cbdd-d59f-a1c10ba297e4} {Suffrage:Voter ID:server-6fdca903-eef1-a757-3da1-a30a9fe3095e Address:6fdca903-eef1-a757-3da1-a30a9fe3095e}]" 2026-03-31T06:16:23.743Z [INFO] server-8ab3fd73-750c-5061-ffdc-b419fcbc9a81: entering follower state: follower="Node at 8ab3fd73-750c-5061-ffdc-b419fcbc9a81 [Follower]" leader-address= leader-id= 2026-03-31T06:16:23.743Z [INFO] server-3d1b2d19-c112-cbdd-d59f-a1c10ba297e4: entering follower state: follower="Node at 3d1b2d19-c112-cbdd-d59f-a1c10ba297e4 [Follower]" leader-address= leader-id= 2026-03-31T06:16:23.743Z [INFO] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d50918c-128e-a400-4c46-c2c4410f26c9 Address:2d50918c-128e-a400-4c46-c2c4410f26c9} {Suffrage:Voter ID:server-8ab3fd73-750c-5061-ffdc-b419fcbc9a81 Address:8ab3fd73-750c-5061-ffdc-b419fcbc9a81} {Suffrage:Voter ID:server-3d1b2d19-c112-cbdd-d59f-a1c10ba297e4 Address:3d1b2d19-c112-cbdd-d59f-a1c10ba297e4} {Suffrage:Voter ID:server-6fdca903-eef1-a757-3da1-a30a9fe3095e Address:6fdca903-eef1-a757-3da1-a30a9fe3095e}]" 2026-03-31T06:16:23.743Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:23.743Z [INFO] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: entering follower state: follower="Node at 6fdca903-eef1-a757-3da1-a30a9fe3095e [Follower]" leader-address= leader-id= 2026-03-31T06:16:23.806Z [WARN] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:23.806Z [INFO] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: entering candidate state: node="Node at 6fdca903-eef1-a757-3da1-a30a9fe3095e [Candidate]" term=2 2026-03-31T06:16:23.806Z [INFO] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: election won: term=2 tally=3 2026-03-31T06:16:23.806Z [INFO] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: entering leader state: leader="Node at 6fdca903-eef1-a757-3da1-a30a9fe3095e [Leader]" 2026-03-31T06:16:23.807Z [INFO] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: added peer, starting replication: peer=server-2d50918c-128e-a400-4c46-c2c4410f26c9 2026-03-31T06:16:23.807Z [INFO] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: added peer, starting replication: peer=server-8ab3fd73-750c-5061-ffdc-b419fcbc9a81 2026-03-31T06:16:23.807Z [INFO] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: added peer, starting replication: peer=server-3d1b2d19-c112-cbdd-d59f-a1c10ba297e4 2026-03-31T06:16:23.807Z [INFO] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: pipelining replication: peer="{Voter server-2d50918c-128e-a400-4c46-c2c4410f26c9 2d50918c-128e-a400-4c46-c2c4410f26c9}" 2026-03-31T06:16:23.807Z [INFO] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: pipelining replication: peer="{Voter server-3d1b2d19-c112-cbdd-d59f-a1c10ba297e4 3d1b2d19-c112-cbdd-d59f-a1c10ba297e4}" 2026-03-31T06:16:23.807Z [INFO] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: pipelining replication: peer="{Voter server-8ab3fd73-750c-5061-ffdc-b419fcbc9a81 8ab3fd73-750c-5061-ffdc-b419fcbc9a81}" 2026-03-31T06:16:23.912Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:23.80703418 +0000 UTC m=+12.084641561 (1 nodes), 63.46944ms from start of poll, 64.04212ms from cluster start. Timeout at 2026-03-31 06:16:23.91211286 +0000 UTC m=+12.189720241, 105.07868ms after stability 2026-03-31T06:16:24.018Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:24.123Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:24.01814518 +0000 UTC m=+12.295752561 (3 nodes), 2.6µs from start of poll, 275.15312ms from cluster start. Timeout at 2026-03-31 06:16:24.12315246 +0000 UTC m=+12.400759841, 105.00728ms after stability 2026-03-31T06:16:24.123Z [INFO] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: updating configuration: command=RemoveServer server-id=server-3d1b2d19-c112-cbdd-d59f-a1c10ba297e4 server-addr= servers="[{Suffrage:Voter ID:server-2d50918c-128e-a400-4c46-c2c4410f26c9 Address:2d50918c-128e-a400-4c46-c2c4410f26c9} {Suffrage:Voter ID:server-8ab3fd73-750c-5061-ffdc-b419fcbc9a81 Address:8ab3fd73-750c-5061-ffdc-b419fcbc9a81} {Suffrage:Voter ID:server-6fdca903-eef1-a757-3da1-a30a9fe3095e Address:6fdca903-eef1-a757-3da1-a30a9fe3095e}]" 2026-03-31T06:16:24.123Z [INFO] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: removed peer, stopping replication: peer=server-3d1b2d19-c112-cbdd-d59f-a1c10ba297e4 last-index=3 2026-03-31T06:16:24.123Z [ERROR] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: failed to pipeline appendEntries: peer="{Voter server-2d50918c-128e-a400-4c46-c2c4410f26c9 2d50918c-128e-a400-4c46-c2c4410f26c9}" error="append pipeline closed" 2026-03-31T06:16:24.123Z [INFO] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: aborting pipeline replication: peer="{Voter server-2d50918c-128e-a400-4c46-c2c4410f26c9 2d50918c-128e-a400-4c46-c2c4410f26c9}" 2026-03-31T06:16:24.123Z [INFO] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: aborting pipeline replication: peer="{Voter server-3d1b2d19-c112-cbdd-d59f-a1c10ba297e4 3d1b2d19-c112-cbdd-d59f-a1c10ba297e4}" 2026-03-31T06:16:24.123Z [ERROR] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: failed to pipeline appendEntries: peer="{Voter server-8ab3fd73-750c-5061-ffdc-b419fcbc9a81 8ab3fd73-750c-5061-ffdc-b419fcbc9a81}" error="append pipeline closed" 2026-03-31T06:16:24.124Z [INFO] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: aborting pipeline replication: peer="{Voter server-8ab3fd73-750c-5061-ffdc-b419fcbc9a81 8ab3fd73-750c-5061-ffdc-b419fcbc9a81}" 2026-03-31T06:16:24.128Z [ERROR] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: failed to heartbeat to: peer=2d50918c-128e-a400-4c46-c2c4410f26c9 backoff time=10ms error="failed to connect to peer: 2d50918c-128e-a400-4c46-c2c4410f26c9" 2026-03-31T06:16:24.128Z [ERROR] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: failed to heartbeat to: peer=8ab3fd73-750c-5061-ffdc-b419fcbc9a81 backoff time=10ms error="failed to connect to peer: 8ab3fd73-750c-5061-ffdc-b419fcbc9a81" 2026-03-31T06:16:24.129Z [ERROR] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: failed to appendEntries to: peer="{Voter server-8ab3fd73-750c-5061-ffdc-b419fcbc9a81 8ab3fd73-750c-5061-ffdc-b419fcbc9a81}" error="failed to connect to peer: 8ab3fd73-750c-5061-ffdc-b419fcbc9a81" 2026-03-31T06:16:24.134Z [ERROR] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: failed to appendEntries to: peer="{Voter server-2d50918c-128e-a400-4c46-c2c4410f26c9 2d50918c-128e-a400-4c46-c2c4410f26c9}" error="failed to connect to peer: 2d50918c-128e-a400-4c46-c2c4410f26c9" 2026-03-31T06:16:24.146Z [ERROR] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: failed to heartbeat to: peer=2d50918c-128e-a400-4c46-c2c4410f26c9 backoff time=10ms error="failed to connect to peer: 2d50918c-128e-a400-4c46-c2c4410f26c9" 2026-03-31T06:16:24.149Z [ERROR] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: failed to heartbeat to: peer=8ab3fd73-750c-5061-ffdc-b419fcbc9a81 backoff time=10ms error="failed to connect to peer: 8ab3fd73-750c-5061-ffdc-b419fcbc9a81" 2026-03-31T06:16:24.149Z [ERROR] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: failed to appendEntries to: peer="{Voter server-8ab3fd73-750c-5061-ffdc-b419fcbc9a81 8ab3fd73-750c-5061-ffdc-b419fcbc9a81}" error="failed to connect to peer: 8ab3fd73-750c-5061-ffdc-b419fcbc9a81" 2026-03-31T06:16:24.154Z [ERROR] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: failed to appendEntries to: peer="{Voter server-2d50918c-128e-a400-4c46-c2c4410f26c9 2d50918c-128e-a400-4c46-c2c4410f26c9}" error="failed to connect to peer: 2d50918c-128e-a400-4c46-c2c4410f26c9" 2026-03-31T06:16:24.163Z [ERROR] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: failed to heartbeat to: peer=2d50918c-128e-a400-4c46-c2c4410f26c9 backoff time=10ms error="failed to connect to peer: 2d50918c-128e-a400-4c46-c2c4410f26c9" 2026-03-31T06:16:24.167Z [ERROR] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: failed to heartbeat to: peer=8ab3fd73-750c-5061-ffdc-b419fcbc9a81 backoff time=10ms error="failed to connect to peer: 8ab3fd73-750c-5061-ffdc-b419fcbc9a81" 2026-03-31T06:16:24.169Z [ERROR] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: failed to appendEntries to: peer="{Voter server-8ab3fd73-750c-5061-ffdc-b419fcbc9a81 8ab3fd73-750c-5061-ffdc-b419fcbc9a81}" error="failed to connect to peer: 8ab3fd73-750c-5061-ffdc-b419fcbc9a81" 2026-03-31T06:16:24.172Z [WARN] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: failed to contact: server-id=server-2d50918c-128e-a400-4c46-c2c4410f26c9 time=50.67476ms 2026-03-31T06:16:24.172Z [WARN] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: failed to contact: server-id=server-8ab3fd73-750c-5061-ffdc-b419fcbc9a81 time=50.61688ms 2026-03-31T06:16:24.172Z [WARN] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: failed to contact quorum of nodes, stepping down 2026-03-31T06:16:24.172Z [INFO] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: entering follower state: follower="Node at 6fdca903-eef1-a757-3da1-a30a9fe3095e [Follower]" leader-address= leader-id= 2026-03-31T06:16:24.172Z [ERROR] server-6fdca903-eef1-a757-3da1-a30a9fe3095e: failed to appendEntries to: peer="{Voter server-2d50918c-128e-a400-4c46-c2c4410f26c9 2d50918c-128e-a400-4c46-c2c4410f26c9}" error="failed to connect to peer: 2d50918c-128e-a400-4c46-c2c4410f26c9" --- PASS: TestRaft_RemoveFollower_SplitCluster (0.43s) === RUN TestRaft_AddKnownPeer 2026-03-31T06:16:24.173Z [INFO] server-281c9aa4-aeff-a825-cf72-0fed0ef320ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-281c9aa4-aeff-a825-cf72-0fed0ef320ca Address:281c9aa4-aeff-a825-cf72-0fed0ef320ca} {Suffrage:Voter ID:server-ae1c07e2-5520-78d7-ed95-96dbabebd5bd Address:ae1c07e2-5520-78d7-ed95-96dbabebd5bd} {Suffrage:Voter ID:server-bbd93742-ca21-d7d6-2c60-fd7d66c55b3c Address:bbd93742-ca21-d7d6-2c60-fd7d66c55b3c}]" 2026-03-31T06:16:24.173Z [INFO] server-281c9aa4-aeff-a825-cf72-0fed0ef320ca: entering follower state: follower="Node at 281c9aa4-aeff-a825-cf72-0fed0ef320ca [Follower]" leader-address= leader-id= 2026-03-31T06:16:24.173Z [INFO] server-ae1c07e2-5520-78d7-ed95-96dbabebd5bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-281c9aa4-aeff-a825-cf72-0fed0ef320ca Address:281c9aa4-aeff-a825-cf72-0fed0ef320ca} {Suffrage:Voter ID:server-ae1c07e2-5520-78d7-ed95-96dbabebd5bd Address:ae1c07e2-5520-78d7-ed95-96dbabebd5bd} {Suffrage:Voter ID:server-bbd93742-ca21-d7d6-2c60-fd7d66c55b3c Address:bbd93742-ca21-d7d6-2c60-fd7d66c55b3c}]" 2026-03-31T06:16:24.173Z [INFO] server-bbd93742-ca21-d7d6-2c60-fd7d66c55b3c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-281c9aa4-aeff-a825-cf72-0fed0ef320ca Address:281c9aa4-aeff-a825-cf72-0fed0ef320ca} {Suffrage:Voter ID:server-ae1c07e2-5520-78d7-ed95-96dbabebd5bd Address:ae1c07e2-5520-78d7-ed95-96dbabebd5bd} {Suffrage:Voter ID:server-bbd93742-ca21-d7d6-2c60-fd7d66c55b3c Address:bbd93742-ca21-d7d6-2c60-fd7d66c55b3c}]" 2026-03-31T06:16:24.173Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:24.173Z [INFO] server-ae1c07e2-5520-78d7-ed95-96dbabebd5bd: entering follower state: follower="Node at ae1c07e2-5520-78d7-ed95-96dbabebd5bd [Follower]" leader-address= leader-id= 2026-03-31T06:16:24.173Z [INFO] server-bbd93742-ca21-d7d6-2c60-fd7d66c55b3c: entering follower state: follower="Node at bbd93742-ca21-d7d6-2c60-fd7d66c55b3c [Follower]" leader-address= leader-id= 2026-03-31T06:16:24.226Z [WARN] server-281c9aa4-aeff-a825-cf72-0fed0ef320ca: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:24.226Z [INFO] server-281c9aa4-aeff-a825-cf72-0fed0ef320ca: entering candidate state: node="Node at 281c9aa4-aeff-a825-cf72-0fed0ef320ca [Candidate]" term=2 2026-03-31T06:16:24.226Z [INFO] server-281c9aa4-aeff-a825-cf72-0fed0ef320ca: election won: term=2 tally=2 2026-03-31T06:16:24.226Z [INFO] server-281c9aa4-aeff-a825-cf72-0fed0ef320ca: entering leader state: leader="Node at 281c9aa4-aeff-a825-cf72-0fed0ef320ca [Leader]" 2026-03-31T06:16:24.226Z [INFO] server-281c9aa4-aeff-a825-cf72-0fed0ef320ca: added peer, starting replication: peer=server-ae1c07e2-5520-78d7-ed95-96dbabebd5bd 2026-03-31T06:16:24.226Z [INFO] server-281c9aa4-aeff-a825-cf72-0fed0ef320ca: added peer, starting replication: peer=server-bbd93742-ca21-d7d6-2c60-fd7d66c55b3c 2026-03-31T06:16:24.226Z [INFO] server-281c9aa4-aeff-a825-cf72-0fed0ef320ca: pipelining replication: peer="{Voter server-ae1c07e2-5520-78d7-ed95-96dbabebd5bd ae1c07e2-5520-78d7-ed95-96dbabebd5bd}" 2026-03-31T06:16:24.226Z [INFO] server-281c9aa4-aeff-a825-cf72-0fed0ef320ca: pipelining replication: peer="{Voter server-bbd93742-ca21-d7d6-2c60-fd7d66c55b3c bbd93742-ca21-d7d6-2c60-fd7d66c55b3c}" 2026-03-31T06:16:24.332Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:24.22622198 +0000 UTC m=+12.503829401 (1 nodes), 52.79972ms from start of poll, 53.1958ms from cluster start. Timeout at 2026-03-31 06:16:24.33122378 +0000 UTC m=+12.608831201, 105.0018ms after stability 2026-03-31T06:16:24.332Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:24.437Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:24.33244118 +0000 UTC m=+12.610048601 (2 nodes), 1.4µs from start of poll, 159.415ms from cluster start. Timeout at 2026-03-31 06:16:24.43744918 +0000 UTC m=+12.715056561, 105.00796ms after stability 2026-03-31T06:16:24.438Z [INFO] server-281c9aa4-aeff-a825-cf72-0fed0ef320ca: updating configuration: command=AddVoter server-id=server-ae1c07e2-5520-78d7-ed95-96dbabebd5bd server-addr=ae1c07e2-5520-78d7-ed95-96dbabebd5bd servers="[{Suffrage:Voter ID:server-281c9aa4-aeff-a825-cf72-0fed0ef320ca Address:281c9aa4-aeff-a825-cf72-0fed0ef320ca} {Suffrage:Voter ID:server-ae1c07e2-5520-78d7-ed95-96dbabebd5bd Address:ae1c07e2-5520-78d7-ed95-96dbabebd5bd} {Suffrage:Voter ID:server-bbd93742-ca21-d7d6-2c60-fd7d66c55b3c Address:bbd93742-ca21-d7d6-2c60-fd7d66c55b3c}]" 2026-03-31T06:16:24.438Z [INFO] server-281c9aa4-aeff-a825-cf72-0fed0ef320ca: aborting pipeline replication: peer="{Voter server-bbd93742-ca21-d7d6-2c60-fd7d66c55b3c bbd93742-ca21-d7d6-2c60-fd7d66c55b3c}" 2026-03-31T06:16:24.438Z [INFO] server-281c9aa4-aeff-a825-cf72-0fed0ef320ca: aborting pipeline replication: peer="{Voter server-ae1c07e2-5520-78d7-ed95-96dbabebd5bd ae1c07e2-5520-78d7-ed95-96dbabebd5bd}" --- PASS: TestRaft_AddKnownPeer (0.27s) === RUN TestRaft_RemoveUnknownPeer 2026-03-31T06:16:24.439Z [INFO] server-863f4971-e881-9ecd-9f62-dcb87bccd606: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-863f4971-e881-9ecd-9f62-dcb87bccd606 Address:863f4971-e881-9ecd-9f62-dcb87bccd606} {Suffrage:Voter ID:server-4e45e5d6-cc5e-9090-8713-d0e61ffeff2a Address:4e45e5d6-cc5e-9090-8713-d0e61ffeff2a} {Suffrage:Voter ID:server-a06c8f26-5165-2a8e-d209-6efc70600249 Address:a06c8f26-5165-2a8e-d209-6efc70600249}]" 2026-03-31T06:16:24.439Z [INFO] server-4e45e5d6-cc5e-9090-8713-d0e61ffeff2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-863f4971-e881-9ecd-9f62-dcb87bccd606 Address:863f4971-e881-9ecd-9f62-dcb87bccd606} {Suffrage:Voter ID:server-4e45e5d6-cc5e-9090-8713-d0e61ffeff2a Address:4e45e5d6-cc5e-9090-8713-d0e61ffeff2a} {Suffrage:Voter ID:server-a06c8f26-5165-2a8e-d209-6efc70600249 Address:a06c8f26-5165-2a8e-d209-6efc70600249}]" 2026-03-31T06:16:24.439Z [INFO] server-863f4971-e881-9ecd-9f62-dcb87bccd606: entering follower state: follower="Node at 863f4971-e881-9ecd-9f62-dcb87bccd606 [Follower]" leader-address= leader-id= 2026-03-31T06:16:24.439Z [INFO] server-a06c8f26-5165-2a8e-d209-6efc70600249: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-863f4971-e881-9ecd-9f62-dcb87bccd606 Address:863f4971-e881-9ecd-9f62-dcb87bccd606} {Suffrage:Voter ID:server-4e45e5d6-cc5e-9090-8713-d0e61ffeff2a Address:4e45e5d6-cc5e-9090-8713-d0e61ffeff2a} {Suffrage:Voter ID:server-a06c8f26-5165-2a8e-d209-6efc70600249 Address:a06c8f26-5165-2a8e-d209-6efc70600249}]" 2026-03-31T06:16:24.439Z [INFO] server-4e45e5d6-cc5e-9090-8713-d0e61ffeff2a: entering follower state: follower="Node at 4e45e5d6-cc5e-9090-8713-d0e61ffeff2a [Follower]" leader-address= leader-id= 2026-03-31T06:16:24.439Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:24.439Z [INFO] server-a06c8f26-5165-2a8e-d209-6efc70600249: entering follower state: follower="Node at a06c8f26-5165-2a8e-d209-6efc70600249 [Follower]" leader-address= leader-id= 2026-03-31T06:16:24.495Z [WARN] server-a06c8f26-5165-2a8e-d209-6efc70600249: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:24.495Z [INFO] server-a06c8f26-5165-2a8e-d209-6efc70600249: entering candidate state: node="Node at a06c8f26-5165-2a8e-d209-6efc70600249 [Candidate]" term=2 2026-03-31T06:16:24.495Z [INFO] server-a06c8f26-5165-2a8e-d209-6efc70600249: election won: term=2 tally=2 2026-03-31T06:16:24.495Z [INFO] server-a06c8f26-5165-2a8e-d209-6efc70600249: entering leader state: leader="Node at a06c8f26-5165-2a8e-d209-6efc70600249 [Leader]" 2026-03-31T06:16:24.495Z [INFO] server-a06c8f26-5165-2a8e-d209-6efc70600249: added peer, starting replication: peer=server-863f4971-e881-9ecd-9f62-dcb87bccd606 2026-03-31T06:16:24.495Z [INFO] server-a06c8f26-5165-2a8e-d209-6efc70600249: added peer, starting replication: peer=server-4e45e5d6-cc5e-9090-8713-d0e61ffeff2a 2026-03-31T06:16:24.495Z [INFO] server-a06c8f26-5165-2a8e-d209-6efc70600249: pipelining replication: peer="{Voter server-863f4971-e881-9ecd-9f62-dcb87bccd606 863f4971-e881-9ecd-9f62-dcb87bccd606}" 2026-03-31T06:16:24.495Z [INFO] server-a06c8f26-5165-2a8e-d209-6efc70600249: pipelining replication: peer="{Voter server-4e45e5d6-cc5e-9090-8713-d0e61ffeff2a 4e45e5d6-cc5e-9090-8713-d0e61ffeff2a}" 2026-03-31T06:16:24.602Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:24.49579774 +0000 UTC m=+12.773405161 (1 nodes), 56.30644ms from start of poll, 56.75132ms from cluster start. Timeout at 2026-03-31 06:16:24.6008809 +0000 UTC m=+12.878488281, 105.08312ms after stability 2026-03-31T06:16:24.602Z [INFO] server-a06c8f26-5165-2a8e-d209-6efc70600249: updating configuration: command=RemoveServer server-id=bc9fff21-40a9-8492-124e-aabf4ffea9e9 server-addr= servers="[{Suffrage:Voter ID:server-863f4971-e881-9ecd-9f62-dcb87bccd606 Address:863f4971-e881-9ecd-9f62-dcb87bccd606} {Suffrage:Voter ID:server-4e45e5d6-cc5e-9090-8713-d0e61ffeff2a Address:4e45e5d6-cc5e-9090-8713-d0e61ffeff2a} {Suffrage:Voter ID:server-a06c8f26-5165-2a8e-d209-6efc70600249 Address:a06c8f26-5165-2a8e-d209-6efc70600249}]" 2026-03-31T06:16:24.602Z [INFO] server-a06c8f26-5165-2a8e-d209-6efc70600249: aborting pipeline replication: peer="{Voter server-4e45e5d6-cc5e-9090-8713-d0e61ffeff2a 4e45e5d6-cc5e-9090-8713-d0e61ffeff2a}" 2026-03-31T06:16:24.602Z [INFO] server-a06c8f26-5165-2a8e-d209-6efc70600249: aborting pipeline replication: peer="{Voter server-863f4971-e881-9ecd-9f62-dcb87bccd606 863f4971-e881-9ecd-9f62-dcb87bccd606}" --- PASS: TestRaft_RemoveUnknownPeer (0.16s) === RUN TestRaft_SnapshotRestore 2026-03-31T06:16:24.602Z [INFO] server-d3ae96dc-33f3-446c-dd49-cd98c916bb77: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3ae96dc-33f3-446c-dd49-cd98c916bb77 Address:d3ae96dc-33f3-446c-dd49-cd98c916bb77}]" 2026-03-31T06:16:24.602Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:24.602Z [INFO] server-d3ae96dc-33f3-446c-dd49-cd98c916bb77: entering follower state: follower="Node at d3ae96dc-33f3-446c-dd49-cd98c916bb77 [Follower]" leader-address= leader-id= 2026-03-31T06:16:24.671Z [WARN] server-d3ae96dc-33f3-446c-dd49-cd98c916bb77: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:24.671Z [INFO] server-d3ae96dc-33f3-446c-dd49-cd98c916bb77: entering candidate state: node="Node at d3ae96dc-33f3-446c-dd49-cd98c916bb77 [Candidate]" term=2 2026-03-31T06:16:24.671Z [INFO] server-d3ae96dc-33f3-446c-dd49-cd98c916bb77: election won: term=2 tally=1 2026-03-31T06:16:24.671Z [INFO] server-d3ae96dc-33f3-446c-dd49-cd98c916bb77: entering leader state: leader="Node at d3ae96dc-33f3-446c-dd49-cd98c916bb77 [Leader]" 2026-03-31T06:16:24.777Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:24.67143702 +0000 UTC m=+12.949044401 (1 nodes), 68.51212ms from start of poll, 68.6144ms from cluster start. Timeout at 2026-03-31 06:16:24.77643994 +0000 UTC m=+13.054047361, 105.00296ms after stability 2026-03-31T06:16:24.780Z [INFO] server-d3ae96dc-33f3-446c-dd49-cd98c916bb77: starting snapshot up to: index=102 2026-03-31T06:16:24.780Z [INFO] creating new snapshot: path=/tmp/raft135916015/snapshots/2-102-1774937784780.tmp 2026-03-31T06:16:24.780Z [INFO] server-d3ae96dc-33f3-446c-dd49-cd98c916bb77: compacting logs: from=1 to=92 2026-03-31T06:16:24.780Z [INFO] server-d3ae96dc-33f3-446c-dd49-cd98c916bb77: snapshot complete up to: index=102 2026-03-31T06:16:24.780Z [INFO] server-d3ae96dc-33f3-446c-dd49-cd98c916bb77: starting restore from snapshot: id=2-102-1774937784780 last-index=102 last-term=2 size-in-bytes=693 2026-03-31T06:16:24.781Z [INFO] server-d3ae96dc-33f3-446c-dd49-cd98c916bb77: snapshot restore progress: id=2-102-1774937784780 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2026-03-31T06:16:24.781Z [INFO] server-d3ae96dc-33f3-446c-dd49-cd98c916bb77: restored from snapshot: id=2-102-1774937784780 last-index=102 last-term=2 size-in-bytes=693 2026-03-31T06:16:24.781Z [INFO] server-d3ae96dc-33f3-446c-dd49-cd98c916bb77: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3ae96dc-33f3-446c-dd49-cd98c916bb77 Address:d3ae96dc-33f3-446c-dd49-cd98c916bb77}]" --- PASS: TestRaft_SnapshotRestore (0.18s) === RUN TestRaft_RestoreSnapshotOnStartup_Monotonic 2026-03-31T06:16:24.781Z [INFO] server-4312a246-f720-0798-e053-91a1e5c82917: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4312a246-f720-0798-e053-91a1e5c82917 Address:4312a246-f720-0798-e053-91a1e5c82917}]" 2026-03-31T06:16:24.781Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:24.781Z [INFO] server-4312a246-f720-0798-e053-91a1e5c82917: entering follower state: follower="Node at 4312a246-f720-0798-e053-91a1e5c82917 [Follower]" leader-address= leader-id= 2026-03-31T06:16:24.870Z [WARN] server-4312a246-f720-0798-e053-91a1e5c82917: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:24.870Z [INFO] server-4312a246-f720-0798-e053-91a1e5c82917: entering candidate state: node="Node at 4312a246-f720-0798-e053-91a1e5c82917 [Candidate]" term=2 2026-03-31T06:16:24.870Z [INFO] server-4312a246-f720-0798-e053-91a1e5c82917: election won: term=2 tally=1 2026-03-31T06:16:24.870Z [INFO] server-4312a246-f720-0798-e053-91a1e5c82917: entering leader state: leader="Node at 4312a246-f720-0798-e053-91a1e5c82917 [Leader]" 2026-03-31T06:16:24.975Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:24.87036626 +0000 UTC m=+13.147973641 (1 nodes), 88.46244ms from start of poll, 88.65188ms from cluster start. Timeout at 2026-03-31 06:16:24.97536866 +0000 UTC m=+13.252976081, 105.00244ms after stability 2026-03-31T06:16:24.979Z [INFO] server-4312a246-f720-0798-e053-91a1e5c82917: starting snapshot up to: index=102 2026-03-31T06:16:24.980Z [INFO] creating new snapshot: path=/tmp/raft2946128651/snapshots/2-102-1774937784980.tmp 2026-03-31T06:16:24.980Z [INFO] server-4312a246-f720-0798-e053-91a1e5c82917: compacting logs: from=1 to=92 2026-03-31T06:16:24.980Z [INFO] server-4312a246-f720-0798-e053-91a1e5c82917: snapshot complete up to: index=102 2026-03-31T06:16:24.980Z [INFO] server-4312a246-f720-0798-e053-91a1e5c82917: starting restore from snapshot: id=2-102-1774937784980 last-index=102 last-term=2 size-in-bytes=693 2026-03-31T06:16:24.981Z [INFO] server-4312a246-f720-0798-e053-91a1e5c82917: snapshot restore progress: id=2-102-1774937784980 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2026-03-31T06:16:24.981Z [INFO] server-4312a246-f720-0798-e053-91a1e5c82917: restored from snapshot: id=2-102-1774937784980 last-index=102 last-term=2 size-in-bytes=693 2026-03-31T06:16:24.981Z [INFO] server-4312a246-f720-0798-e053-91a1e5c82917: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4312a246-f720-0798-e053-91a1e5c82917 Address:4312a246-f720-0798-e053-91a1e5c82917}]" --- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.20s) === RUN TestRaft_SnapshotRestore_Progress 2026-03-31T06:16:24.981Z [INFO] server-ebbba981-e6f3-1e13-993f-c1cebd3fe2f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ebbba981-e6f3-1e13-993f-c1cebd3fe2f9 Address:ebbba981-e6f3-1e13-993f-c1cebd3fe2f9}]" 2026-03-31T06:16:24.981Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:24.981Z [INFO] server-ebbba981-e6f3-1e13-993f-c1cebd3fe2f9: entering follower state: follower="Node at ebbba981-e6f3-1e13-993f-c1cebd3fe2f9 [Follower]" leader-address= leader-id= 2026-03-31T06:16:25.034Z [WARN] server-ebbba981-e6f3-1e13-993f-c1cebd3fe2f9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:25.034Z [INFO] server-ebbba981-e6f3-1e13-993f-c1cebd3fe2f9: entering candidate state: node="Node at ebbba981-e6f3-1e13-993f-c1cebd3fe2f9 [Candidate]" term=2 2026-03-31T06:16:25.034Z [INFO] server-ebbba981-e6f3-1e13-993f-c1cebd3fe2f9: election won: term=2 tally=1 2026-03-31T06:16:25.034Z [INFO] server-ebbba981-e6f3-1e13-993f-c1cebd3fe2f9: entering leader state: leader="Node at ebbba981-e6f3-1e13-993f-c1cebd3fe2f9 [Leader]" 2026-03-31T06:16:25.140Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:25.03433718 +0000 UTC m=+13.311944561 (1 nodes), 52.5816ms from start of poll, 52.7604ms from cluster start. Timeout at 2026-03-31 06:16:25.1393385 +0000 UTC m=+13.416945921, 105.00136ms after stability 2026-03-31T06:16:25.143Z [INFO] server-ebbba981-e6f3-1e13-993f-c1cebd3fe2f9: starting snapshot up to: index=102 2026-03-31T06:16:25.143Z [INFO] creating new snapshot: path=/tmp/raft3753111462/snapshots/2-102-1774937785143.tmp 2026-03-31T06:16:25.143Z [INFO] server-ebbba981-e6f3-1e13-993f-c1cebd3fe2f9: compacting logs: from=1 to=92 2026-03-31T06:16:25.143Z [INFO] server-ebbba981-e6f3-1e13-993f-c1cebd3fe2f9: snapshot complete up to: index=102 --- PASS: TestRaft_SnapshotRestore_Progress (0.16s) === RUN TestRaft_NoRestoreOnStart 2026-03-31T06:16:25.144Z [INFO] server-f0491838-8406-a74f-5b62-2309b8312d19: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0491838-8406-a74f-5b62-2309b8312d19 Address:f0491838-8406-a74f-5b62-2309b8312d19}]" 2026-03-31T06:16:25.144Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:25.145Z [INFO] server-f0491838-8406-a74f-5b62-2309b8312d19: entering follower state: follower="Node at f0491838-8406-a74f-5b62-2309b8312d19 [Follower]" leader-address= leader-id= 2026-03-31T06:16:25.213Z [WARN] server-f0491838-8406-a74f-5b62-2309b8312d19: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:25.213Z [INFO] server-f0491838-8406-a74f-5b62-2309b8312d19: entering candidate state: node="Node at f0491838-8406-a74f-5b62-2309b8312d19 [Candidate]" term=2 2026-03-31T06:16:25.213Z [INFO] server-f0491838-8406-a74f-5b62-2309b8312d19: election won: term=2 tally=1 2026-03-31T06:16:25.213Z [INFO] server-f0491838-8406-a74f-5b62-2309b8312d19: entering leader state: leader="Node at f0491838-8406-a74f-5b62-2309b8312d19 [Leader]" 2026-03-31T06:16:25.318Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:25.21321686 +0000 UTC m=+13.490824241 (1 nodes), 68.20564ms from start of poll, 68.38264ms from cluster start. Timeout at 2026-03-31 06:16:25.31821918 +0000 UTC m=+13.595826561, 105.00232ms after stability 2026-03-31T06:16:25.321Z [INFO] server-f0491838-8406-a74f-5b62-2309b8312d19: starting snapshot up to: index=102 2026-03-31T06:16:25.321Z [INFO] creating new snapshot: path=/tmp/raft3388937895/snapshots/2-102-1774937785321.tmp 2026-03-31T06:16:25.321Z [INFO] server-f0491838-8406-a74f-5b62-2309b8312d19: compacting logs: from=1 to=92 2026-03-31T06:16:25.322Z [INFO] server-f0491838-8406-a74f-5b62-2309b8312d19: snapshot complete up to: index=102 2026-03-31T06:16:25.322Z [INFO] server-f0491838-8406-a74f-5b62-2309b8312d19: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0491838-8406-a74f-5b62-2309b8312d19 Address:f0491838-8406-a74f-5b62-2309b8312d19}]" --- PASS: TestRaft_NoRestoreOnStart (0.18s) === RUN TestRaft_SnapshotRestore_PeerChange 2026-03-31T06:16:25.322Z [INFO] server-f0491838-8406-a74f-5b62-2309b8312d19: entering follower state: follower="Node at f0491838-8406-a74f-5b62-2309b8312d19 [Follower]" leader-address= leader-id= raft_test.go:1266: Skipping unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot 2026-03-31T06:16:25.322Z [INFO] server-367ee835-7411-704f-23d2-e5c6dcc02ab4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-367ee835-7411-704f-23d2-e5c6dcc02ab4 Address:367ee835-7411-704f-23d2-e5c6dcc02ab4}]" 2026-03-31T06:16:25.322Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:25.322Z [INFO] server-367ee835-7411-704f-23d2-e5c6dcc02ab4: entering follower state: follower="Node at 367ee835-7411-704f-23d2-e5c6dcc02ab4 [Follower]" leader-address= leader-id= 2026-03-31T06:16:25.390Z [WARN] server-f0491838-8406-a74f-5b62-2309b8312d19: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:25.390Z [INFO] server-f0491838-8406-a74f-5b62-2309b8312d19: entering candidate state: node="Node at f0491838-8406-a74f-5b62-2309b8312d19 [Candidate]" term=3 2026-03-31T06:16:25.390Z [INFO] server-f0491838-8406-a74f-5b62-2309b8312d19: election won: term=3 tally=1 2026-03-31T06:16:25.390Z [INFO] server-f0491838-8406-a74f-5b62-2309b8312d19: entering leader state: leader="Node at f0491838-8406-a74f-5b62-2309b8312d19 [Leader]" 2026-03-31T06:16:25.414Z [WARN] server-367ee835-7411-704f-23d2-e5c6dcc02ab4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:25.414Z [INFO] server-367ee835-7411-704f-23d2-e5c6dcc02ab4: entering candidate state: node="Node at 367ee835-7411-704f-23d2-e5c6dcc02ab4 [Candidate]" term=2 2026-03-31T06:16:25.414Z [INFO] server-367ee835-7411-704f-23d2-e5c6dcc02ab4: election won: term=2 tally=1 2026-03-31T06:16:25.414Z [INFO] server-367ee835-7411-704f-23d2-e5c6dcc02ab4: entering leader state: leader="Node at 367ee835-7411-704f-23d2-e5c6dcc02ab4 [Leader]" 2026-03-31T06:16:25.520Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:25.41473554 +0000 UTC m=+13.692342921 (1 nodes), 92.07676ms from start of poll, 92.24068ms from cluster start. Timeout at 2026-03-31 06:16:25.51973674 +0000 UTC m=+13.797344121, 105.0012ms after stability 2026-03-31T06:16:25.524Z [INFO] server-367ee835-7411-704f-23d2-e5c6dcc02ab4: starting snapshot up to: index=92 2026-03-31T06:16:25.524Z [INFO] creating new snapshot: path=/tmp/raft1774242072/snapshots/2-92-1774937785524.tmp 2026-03-31T06:16:25.524Z [INFO] server-367ee835-7411-704f-23d2-e5c6dcc02ab4: compacting logs: from=1 to=92 2026-03-31T06:16:25.524Z [INFO] server-367ee835-7411-704f-23d2-e5c6dcc02ab4: snapshot complete up to: index=92 --- PASS: TestRaft_AutoSnapshot (0.31s) === RUN TestRaft_UserSnapshot 2026-03-31T06:16:25.631Z [INFO] server-1c0a7c0d-5f14-71a9-1b59-06b1b6572756: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c0a7c0d-5f14-71a9-1b59-06b1b6572756 Address:1c0a7c0d-5f14-71a9-1b59-06b1b6572756}]" 2026-03-31T06:16:25.631Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:25.631Z [INFO] server-1c0a7c0d-5f14-71a9-1b59-06b1b6572756: entering follower state: follower="Node at 1c0a7c0d-5f14-71a9-1b59-06b1b6572756 [Follower]" leader-address= leader-id= 2026-03-31T06:16:25.724Z [WARN] server-1c0a7c0d-5f14-71a9-1b59-06b1b6572756: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:25.724Z [INFO] server-1c0a7c0d-5f14-71a9-1b59-06b1b6572756: entering candidate state: node="Node at 1c0a7c0d-5f14-71a9-1b59-06b1b6572756 [Candidate]" term=2 2026-03-31T06:16:25.724Z [INFO] server-1c0a7c0d-5f14-71a9-1b59-06b1b6572756: election won: term=2 tally=1 2026-03-31T06:16:25.724Z [INFO] server-1c0a7c0d-5f14-71a9-1b59-06b1b6572756: entering leader state: leader="Node at 1c0a7c0d-5f14-71a9-1b59-06b1b6572756 [Leader]" 2026-03-31T06:16:25.831Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:25.7250139 +0000 UTC m=+14.002621281 (1 nodes), 93.61656ms from start of poll, 93.80816ms from cluster start. Timeout at 2026-03-31 06:16:25.8300161 +0000 UTC m=+14.107623481, 105.0022ms after stability 2026-03-31T06:16:25.831Z [ERROR] server-1c0a7c0d-5f14-71a9-1b59-06b1b6572756: failed to take snapshot: error="nothing new to snapshot" 2026-03-31T06:16:25.831Z [INFO] server-1c0a7c0d-5f14-71a9-1b59-06b1b6572756: starting snapshot up to: index=12 2026-03-31T06:16:25.831Z [INFO] creating new snapshot: path=/tmp/raft2787383667/snapshots/2-12-1774937785831.tmp 2026-03-31T06:16:25.831Z [INFO] server-1c0a7c0d-5f14-71a9-1b59-06b1b6572756: compacting logs: from=1 to=2 2026-03-31T06:16:25.832Z [INFO] server-1c0a7c0d-5f14-71a9-1b59-06b1b6572756: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.20s) === RUN TestRaft_UserRestore === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false 2026-03-31T06:16:25.832Z [INFO] server-38472f22-a93f-becd-8296-5595571dca61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38472f22-a93f-becd-8296-5595571dca61 Address:38472f22-a93f-becd-8296-5595571dca61} {Suffrage:Voter ID:server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11 Address:6ffcb4fd-b7d3-8fa2-db69-0a913145ac11} {Suffrage:Voter ID:server-134d61c4-dd66-ff95-9cb7-36b477acc453 Address:134d61c4-dd66-ff95-9cb7-36b477acc453}]" 2026-03-31T06:16:25.832Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38472f22-a93f-becd-8296-5595571dca61 Address:38472f22-a93f-becd-8296-5595571dca61} {Suffrage:Voter ID:server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11 Address:6ffcb4fd-b7d3-8fa2-db69-0a913145ac11} {Suffrage:Voter ID:server-134d61c4-dd66-ff95-9cb7-36b477acc453 Address:134d61c4-dd66-ff95-9cb7-36b477acc453}]" 2026-03-31T06:16:25.833Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: entering follower state: follower="Node at 6ffcb4fd-b7d3-8fa2-db69-0a913145ac11 [Follower]" leader-address= leader-id= 2026-03-31T06:16:25.832Z [INFO] server-38472f22-a93f-becd-8296-5595571dca61: entering follower state: follower="Node at 38472f22-a93f-becd-8296-5595571dca61 [Follower]" leader-address= leader-id= 2026-03-31T06:16:25.833Z [INFO] server-134d61c4-dd66-ff95-9cb7-36b477acc453: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38472f22-a93f-becd-8296-5595571dca61 Address:38472f22-a93f-becd-8296-5595571dca61} {Suffrage:Voter ID:server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11 Address:6ffcb4fd-b7d3-8fa2-db69-0a913145ac11} {Suffrage:Voter ID:server-134d61c4-dd66-ff95-9cb7-36b477acc453 Address:134d61c4-dd66-ff95-9cb7-36b477acc453}]" 2026-03-31T06:16:25.833Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:25.833Z [INFO] server-134d61c4-dd66-ff95-9cb7-36b477acc453: entering follower state: follower="Node at 134d61c4-dd66-ff95-9cb7-36b477acc453 [Follower]" leader-address= leader-id= 2026-03-31T06:16:26.348Z [WARN] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:26.348Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: entering candidate state: node="Node at 6ffcb4fd-b7d3-8fa2-db69-0a913145ac11 [Candidate]" term=2 2026-03-31T06:16:26.348Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: election won: term=2 tally=2 2026-03-31T06:16:26.348Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: entering leader state: leader="Node at 6ffcb4fd-b7d3-8fa2-db69-0a913145ac11 [Leader]" 2026-03-31T06:16:26.348Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: added peer, starting replication: peer=server-38472f22-a93f-becd-8296-5595571dca61 2026-03-31T06:16:26.348Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: added peer, starting replication: peer=server-134d61c4-dd66-ff95-9cb7-36b477acc453 2026-03-31T06:16:26.348Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: pipelining replication: peer="{Voter server-134d61c4-dd66-ff95-9cb7-36b477acc453 134d61c4-dd66-ff95-9cb7-36b477acc453}" 2026-03-31T06:16:26.348Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: pipelining replication: peer="{Voter server-38472f22-a93f-becd-8296-5595571dca61 38472f22-a93f-becd-8296-5595571dca61}" 2026-03-31T06:16:27.354Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:26.34824774 +0000 UTC m=+14.625855121 (1 nodes), 515.11136ms from start of poll, 515.56212ms from cluster start. Timeout at 2026-03-31 06:16:27.35325126 +0000 UTC m=+15.630858641, 1.00500352s after stability 2026-03-31T06:16:27.354Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: starting snapshot up to: index=12 2026-03-31T06:16:27.354Z [INFO] creating new snapshot: path=/tmp/raft3355499100/snapshots/2-12-1774937787354.tmp 2026-03-31T06:16:27.355Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: snapshot complete up to: index=12 2026-03-31T06:16:27.356Z [INFO] creating new snapshot: path=/tmp/raft3355499100/snapshots/2-23-1774937787356.tmp 2026-03-31T06:16:27.356Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: copied to local snapshot: bytes=71 2026-03-31T06:16:27.356Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: snapshot restore progress: id=2-23-1774937787356 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:27.357Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: restored user snapshot: index=23 2026-03-31T06:16:27.357Z [ERROR] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: failed to get log: index=23 error="log not found" 2026-03-31T06:16:27.357Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: aborting pipeline replication: peer="{Voter server-134d61c4-dd66-ff95-9cb7-36b477acc453 134d61c4-dd66-ff95-9cb7-36b477acc453}" 2026-03-31T06:16:27.357Z [ERROR] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: failed to get log: index=23 error="log not found" 2026-03-31T06:16:27.357Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: aborting pipeline replication: peer="{Voter server-38472f22-a93f-becd-8296-5595571dca61 38472f22-a93f-becd-8296-5595571dca61}" 2026-03-31T06:16:27.364Z [ERROR] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: failed to get log: index=23 error="log not found" 2026-03-31T06:16:27.364Z [INFO] creating new snapshot: path=/tmp/raft3383489412/snapshots/2-23-1774937787364.tmp 2026-03-31T06:16:27.364Z [INFO] server-134d61c4-dd66-ff95-9cb7-36b477acc453: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:27.365Z [INFO] server-134d61c4-dd66-ff95-9cb7-36b477acc453: copied to local snapshot: bytes=71 2026-03-31T06:16:27.365Z [INFO] server-134d61c4-dd66-ff95-9cb7-36b477acc453: snapshot restore progress: id=2-23-1774937787364 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:27.365Z [INFO] server-134d61c4-dd66-ff95-9cb7-36b477acc453: Installed remote snapshot 2026-03-31T06:16:27.366Z [WARN] server-134d61c4-dd66-ff95-9cb7-36b477acc453: skipping application of old log: index=22 2026-03-31T06:16:27.366Z [ERROR] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: failed to get log: index=23 error="log not found" 2026-03-31T06:16:27.366Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: pipelining replication: peer="{Voter server-134d61c4-dd66-ff95-9cb7-36b477acc453 134d61c4-dd66-ff95-9cb7-36b477acc453}" 2026-03-31T06:16:27.366Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:27.367Z [INFO] creating new snapshot: path=/tmp/raft1961861786/snapshots/2-23-1774937787367.tmp 2026-03-31T06:16:27.367Z [INFO] server-38472f22-a93f-becd-8296-5595571dca61: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:27.367Z [INFO] server-38472f22-a93f-becd-8296-5595571dca61: copied to local snapshot: bytes=71 2026-03-31T06:16:27.368Z [INFO] server-38472f22-a93f-becd-8296-5595571dca61: snapshot restore progress: id=2-23-1774937787367 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:27.368Z [INFO] server-38472f22-a93f-becd-8296-5595571dca61: Installed remote snapshot 2026-03-31T06:16:27.368Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: pipelining replication: peer="{Voter server-38472f22-a93f-becd-8296-5595571dca61 38472f22-a93f-becd-8296-5595571dca61}" 2026-03-31T06:16:28.371Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:27.36664662 +0000 UTC m=+15.644254041 (2 nodes), 13.96µs from start of poll, 1.53396104s from cluster start. Timeout at 2026-03-31 06:16:28.37165286 +0000 UTC m=+16.649260201, 1.00500616s after stability 2026-03-31T06:16:28.383Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: aborting pipeline replication: peer="{Voter server-134d61c4-dd66-ff95-9cb7-36b477acc453 134d61c4-dd66-ff95-9cb7-36b477acc453}" 2026-03-31T06:16:28.383Z [INFO] server-6ffcb4fd-b7d3-8fa2-db69-0a913145ac11: aborting pipeline replication: peer="{Voter server-38472f22-a93f-becd-8296-5595571dca61 38472f22-a93f-becd-8296-5595571dca61}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false 2026-03-31T06:16:28.384Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12 Address:f01d8e74-4617-1aef-3fc2-9dc5ae844d12} {Suffrage:Voter ID:server-91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb Address:91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb} {Suffrage:Voter ID:server-740bd9a0-3fba-a645-be76-72b5c9b5d0b1 Address:740bd9a0-3fba-a645-be76-72b5c9b5d0b1}]" 2026-03-31T06:16:28.385Z [INFO] server-91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12 Address:f01d8e74-4617-1aef-3fc2-9dc5ae844d12} {Suffrage:Voter ID:server-91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb Address:91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb} {Suffrage:Voter ID:server-740bd9a0-3fba-a645-be76-72b5c9b5d0b1 Address:740bd9a0-3fba-a645-be76-72b5c9b5d0b1}]" 2026-03-31T06:16:28.385Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: entering follower state: follower="Node at f01d8e74-4617-1aef-3fc2-9dc5ae844d12 [Follower]" leader-address= leader-id= 2026-03-31T06:16:28.385Z [INFO] server-91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb: entering follower state: follower="Node at 91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb [Follower]" leader-address= leader-id= 2026-03-31T06:16:28.385Z [INFO] server-740bd9a0-3fba-a645-be76-72b5c9b5d0b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12 Address:f01d8e74-4617-1aef-3fc2-9dc5ae844d12} {Suffrage:Voter ID:server-91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb Address:91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb} {Suffrage:Voter ID:server-740bd9a0-3fba-a645-be76-72b5c9b5d0b1 Address:740bd9a0-3fba-a645-be76-72b5c9b5d0b1}]" 2026-03-31T06:16:28.385Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:28.385Z [INFO] server-740bd9a0-3fba-a645-be76-72b5c9b5d0b1: entering follower state: follower="Node at 740bd9a0-3fba-a645-be76-72b5c9b5d0b1 [Follower]" leader-address= leader-id= 2026-03-31T06:16:28.905Z [WARN] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:28.905Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: entering candidate state: node="Node at f01d8e74-4617-1aef-3fc2-9dc5ae844d12 [Candidate]" term=2 2026-03-31T06:16:28.905Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: election won: term=2 tally=2 2026-03-31T06:16:28.905Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: entering leader state: leader="Node at f01d8e74-4617-1aef-3fc2-9dc5ae844d12 [Leader]" 2026-03-31T06:16:28.905Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: added peer, starting replication: peer=server-91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb 2026-03-31T06:16:28.905Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: added peer, starting replication: peer=server-740bd9a0-3fba-a645-be76-72b5c9b5d0b1 2026-03-31T06:16:28.906Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: pipelining replication: peer="{Voter server-740bd9a0-3fba-a645-be76-72b5c9b5d0b1 740bd9a0-3fba-a645-be76-72b5c9b5d0b1}" 2026-03-31T06:16:28.906Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: pipelining replication: peer="{Voter server-91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb 91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb}" 2026-03-31T06:16:29.910Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:28.90590426 +0000 UTC m=+17.183511641 (1 nodes), 520.65728ms from start of poll, 521.05852ms from cluster start. Timeout at 2026-03-31 06:16:29.91090626 +0000 UTC m=+18.188513561, 1.00500192s after stability 2026-03-31T06:16:29.911Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: starting snapshot up to: index=12 2026-03-31T06:16:29.911Z [INFO] creating new snapshot: path=/tmp/raft437249075/snapshots/2-12-1774937789911.tmp 2026-03-31T06:16:29.912Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: snapshot complete up to: index=12 2026-03-31T06:16:29.913Z [INFO] creating new snapshot: path=/tmp/raft437249075/snapshots/2-23-1774937789913.tmp 2026-03-31T06:16:29.914Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: copied to local snapshot: bytes=71 2026-03-31T06:16:29.915Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: snapshot restore progress: id=2-23-1774937789913 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:29.915Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: removing all old logs from log store 2026-03-31T06:16:29.915Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: compacting logs: from=1 to=22 2026-03-31T06:16:29.915Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: restored user snapshot: index=23 2026-03-31T06:16:29.915Z [ERROR] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: failed to get log: index=22 error="log not found" 2026-03-31T06:16:29.915Z [ERROR] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: failed to get log: index=22 error="log not found" 2026-03-31T06:16:29.915Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: aborting pipeline replication: peer="{Voter server-91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb 91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb}" 2026-03-31T06:16:29.915Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: aborting pipeline replication: peer="{Voter server-740bd9a0-3fba-a645-be76-72b5c9b5d0b1 740bd9a0-3fba-a645-be76-72b5c9b5d0b1}" 2026-03-31T06:16:29.924Z [ERROR] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: failed to get log: index=22 error="log not found" 2026-03-31T06:16:29.924Z [INFO] creating new snapshot: path=/tmp/raft3353438048/snapshots/2-23-1774937789924.tmp 2026-03-31T06:16:29.924Z [INFO] server-740bd9a0-3fba-a645-be76-72b5c9b5d0b1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:29.925Z [INFO] server-740bd9a0-3fba-a645-be76-72b5c9b5d0b1: copied to local snapshot: bytes=71 2026-03-31T06:16:29.925Z [ERROR] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: failed to get log: index=22 error="log not found" 2026-03-31T06:16:29.925Z [INFO] server-740bd9a0-3fba-a645-be76-72b5c9b5d0b1: snapshot restore progress: id=2-23-1774937789924 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:29.925Z [INFO] server-740bd9a0-3fba-a645-be76-72b5c9b5d0b1: removing all old logs from log store 2026-03-31T06:16:29.925Z [INFO] server-740bd9a0-3fba-a645-be76-72b5c9b5d0b1: compacting logs: from=1 to=22 2026-03-31T06:16:29.925Z [INFO] server-740bd9a0-3fba-a645-be76-72b5c9b5d0b1: Installed remote snapshot 2026-03-31T06:16:29.925Z [WARN] server-740bd9a0-3fba-a645-be76-72b5c9b5d0b1: skipping application of old log: index=22 2026-03-31T06:16:29.925Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: pipelining replication: peer="{Voter server-740bd9a0-3fba-a645-be76-72b5c9b5d0b1 740bd9a0-3fba-a645-be76-72b5c9b5d0b1}" 2026-03-31T06:16:29.925Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:29.925Z [INFO] creating new snapshot: path=/tmp/raft2910877366/snapshots/2-23-1774937789925.tmp 2026-03-31T06:16:29.926Z [INFO] server-91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:29.926Z [INFO] server-91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb: copied to local snapshot: bytes=71 2026-03-31T06:16:29.926Z [INFO] server-91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb: snapshot restore progress: id=2-23-1774937789925 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:29.926Z [INFO] server-91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb: removing all old logs from log store 2026-03-31T06:16:29.926Z [INFO] server-91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb: compacting logs: from=1 to=22 2026-03-31T06:16:29.926Z [INFO] server-91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb: Installed remote snapshot 2026-03-31T06:16:29.926Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: pipelining replication: peer="{Voter server-91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb 91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb}" 2026-03-31T06:16:30.931Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:29.92570502 +0000 UTC m=+18.203312401 (2 nodes), 2.16µs from start of poll, 1.54085928s from cluster start. Timeout at 2026-03-31 06:16:30.93071042 +0000 UTC m=+19.208317801, 1.0050054s after stability 2026-03-31T06:16:30.943Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: aborting pipeline replication: peer="{Voter server-740bd9a0-3fba-a645-be76-72b5c9b5d0b1 740bd9a0-3fba-a645-be76-72b5c9b5d0b1}" 2026-03-31T06:16:30.943Z [INFO] server-f01d8e74-4617-1aef-3fc2-9dc5ae844d12: aborting pipeline replication: peer="{Voter server-91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb 91a6f5ef-66b0-e68d-6cc8-2b0e5f3a1deb}" === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true 2026-03-31T06:16:30.945Z [INFO] server-81f2cd18-c224-6e16-c95a-72edfb8f7b43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81f2cd18-c224-6e16-c95a-72edfb8f7b43 Address:81f2cd18-c224-6e16-c95a-72edfb8f7b43} {Suffrage:Voter ID:server-6888f31a-0f01-ab3b-c81c-bc5b93c62a94 Address:6888f31a-0f01-ab3b-c81c-bc5b93c62a94} {Suffrage:Voter ID:server-1290471b-0fd0-ca61-9b8e-50c783c5f8a9 Address:1290471b-0fd0-ca61-9b8e-50c783c5f8a9}]" 2026-03-31T06:16:30.945Z [INFO] server-6888f31a-0f01-ab3b-c81c-bc5b93c62a94: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81f2cd18-c224-6e16-c95a-72edfb8f7b43 Address:81f2cd18-c224-6e16-c95a-72edfb8f7b43} {Suffrage:Voter ID:server-6888f31a-0f01-ab3b-c81c-bc5b93c62a94 Address:6888f31a-0f01-ab3b-c81c-bc5b93c62a94} {Suffrage:Voter ID:server-1290471b-0fd0-ca61-9b8e-50c783c5f8a9 Address:1290471b-0fd0-ca61-9b8e-50c783c5f8a9}]" 2026-03-31T06:16:30.945Z [INFO] server-81f2cd18-c224-6e16-c95a-72edfb8f7b43: entering follower state: follower="Node at 81f2cd18-c224-6e16-c95a-72edfb8f7b43 [Follower]" leader-address= leader-id= 2026-03-31T06:16:30.945Z [INFO] server-6888f31a-0f01-ab3b-c81c-bc5b93c62a94: entering follower state: follower="Node at 6888f31a-0f01-ab3b-c81c-bc5b93c62a94 [Follower]" leader-address= leader-id= 2026-03-31T06:16:30.945Z [INFO] server-1290471b-0fd0-ca61-9b8e-50c783c5f8a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81f2cd18-c224-6e16-c95a-72edfb8f7b43 Address:81f2cd18-c224-6e16-c95a-72edfb8f7b43} {Suffrage:Voter ID:server-6888f31a-0f01-ab3b-c81c-bc5b93c62a94 Address:6888f31a-0f01-ab3b-c81c-bc5b93c62a94} {Suffrage:Voter ID:server-1290471b-0fd0-ca61-9b8e-50c783c5f8a9 Address:1290471b-0fd0-ca61-9b8e-50c783c5f8a9}]" 2026-03-31T06:16:30.945Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:30.945Z [INFO] server-1290471b-0fd0-ca61-9b8e-50c783c5f8a9: entering follower state: follower="Node at 1290471b-0fd0-ca61-9b8e-50c783c5f8a9 [Follower]" leader-address= leader-id= 2026-03-31T06:16:31.559Z [WARN] server-6888f31a-0f01-ab3b-c81c-bc5b93c62a94: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:31.559Z [INFO] server-6888f31a-0f01-ab3b-c81c-bc5b93c62a94: entering candidate state: node="Node at 6888f31a-0f01-ab3b-c81c-bc5b93c62a94 [Candidate]" term=2 2026-03-31T06:16:31.559Z [INFO] server-6888f31a-0f01-ab3b-c81c-bc5b93c62a94: election won: term=2 tally=2 2026-03-31T06:16:31.559Z [INFO] server-6888f31a-0f01-ab3b-c81c-bc5b93c62a94: entering leader state: leader="Node at 6888f31a-0f01-ab3b-c81c-bc5b93c62a94 [Leader]" 2026-03-31T06:16:31.559Z [INFO] server-6888f31a-0f01-ab3b-c81c-bc5b93c62a94: added peer, starting replication: peer=server-81f2cd18-c224-6e16-c95a-72edfb8f7b43 2026-03-31T06:16:31.559Z [INFO] server-6888f31a-0f01-ab3b-c81c-bc5b93c62a94: added peer, starting replication: peer=server-1290471b-0fd0-ca61-9b8e-50c783c5f8a9 2026-03-31T06:16:31.559Z [INFO] server-6888f31a-0f01-ab3b-c81c-bc5b93c62a94: pipelining replication: peer="{Voter server-81f2cd18-c224-6e16-c95a-72edfb8f7b43 81f2cd18-c224-6e16-c95a-72edfb8f7b43}" 2026-03-31T06:16:31.559Z [INFO] server-6888f31a-0f01-ab3b-c81c-bc5b93c62a94: pipelining replication: peer="{Voter server-1290471b-0fd0-ca61-9b8e-50c783c5f8a9 1290471b-0fd0-ca61-9b8e-50c783c5f8a9}" 2026-03-31T06:16:32.565Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:31.5596427 +0000 UTC m=+19.837250081 (1 nodes), 614.33152ms from start of poll, 614.7414ms from cluster start. Timeout at 2026-03-31 06:16:32.5646489 +0000 UTC m=+20.842256281, 1.0050062s after stability 2026-03-31T06:16:32.566Z [INFO] server-6888f31a-0f01-ab3b-c81c-bc5b93c62a94: starting snapshot up to: index=12 2026-03-31T06:16:32.566Z [INFO] creating new snapshot: path=/tmp/raft3747166050/snapshots/2-12-1774937792566.tmp 2026-03-31T06:16:32.566Z [INFO] server-6888f31a-0f01-ab3b-c81c-bc5b93c62a94: snapshot complete up to: index=12 2026-03-31T06:16:32.569Z [INFO] server-dbbb8175-4c5d-7bb6-d372-646abd205236: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dbbb8175-4c5d-7bb6-d372-646abd205236 Address:dbbb8175-4c5d-7bb6-d372-646abd205236} {Suffrage:Voter ID:server-4029e8e5-63e2-6291-fb9d-1fd11ba6063a Address:4029e8e5-63e2-6291-fb9d-1fd11ba6063a} {Suffrage:Voter ID:server-2dbd3c89-4372-d301-5a7c-4d5eb25141fd Address:2dbd3c89-4372-d301-5a7c-4d5eb25141fd}]" 2026-03-31T06:16:32.569Z [INFO] server-dbbb8175-4c5d-7bb6-d372-646abd205236: entering follower state: follower="Node at dbbb8175-4c5d-7bb6-d372-646abd205236 [Follower]" leader-address= leader-id= 2026-03-31T06:16:32.569Z [INFO] server-4029e8e5-63e2-6291-fb9d-1fd11ba6063a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dbbb8175-4c5d-7bb6-d372-646abd205236 Address:dbbb8175-4c5d-7bb6-d372-646abd205236} {Suffrage:Voter ID:server-4029e8e5-63e2-6291-fb9d-1fd11ba6063a Address:4029e8e5-63e2-6291-fb9d-1fd11ba6063a} {Suffrage:Voter ID:server-2dbd3c89-4372-d301-5a7c-4d5eb25141fd Address:2dbd3c89-4372-d301-5a7c-4d5eb25141fd}]" 2026-03-31T06:16:32.569Z [INFO] server-4029e8e5-63e2-6291-fb9d-1fd11ba6063a: entering follower state: follower="Node at 4029e8e5-63e2-6291-fb9d-1fd11ba6063a [Follower]" leader-address= leader-id= 2026-03-31T06:16:32.570Z [INFO] server-2dbd3c89-4372-d301-5a7c-4d5eb25141fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dbbb8175-4c5d-7bb6-d372-646abd205236 Address:dbbb8175-4c5d-7bb6-d372-646abd205236} {Suffrage:Voter ID:server-4029e8e5-63e2-6291-fb9d-1fd11ba6063a Address:4029e8e5-63e2-6291-fb9d-1fd11ba6063a} {Suffrage:Voter ID:server-2dbd3c89-4372-d301-5a7c-4d5eb25141fd Address:2dbd3c89-4372-d301-5a7c-4d5eb25141fd}]" 2026-03-31T06:16:32.570Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:32.570Z [INFO] server-2dbd3c89-4372-d301-5a7c-4d5eb25141fd: entering follower state: follower="Node at 2dbd3c89-4372-d301-5a7c-4d5eb25141fd [Follower]" leader-address= leader-id= 2026-03-31T06:16:33.127Z [WARN] server-dbbb8175-4c5d-7bb6-d372-646abd205236: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:33.127Z [INFO] server-dbbb8175-4c5d-7bb6-d372-646abd205236: entering candidate state: node="Node at dbbb8175-4c5d-7bb6-d372-646abd205236 [Candidate]" term=2 2026-03-31T06:16:33.127Z [INFO] server-dbbb8175-4c5d-7bb6-d372-646abd205236: election won: term=2 tally=2 2026-03-31T06:16:33.127Z [INFO] server-dbbb8175-4c5d-7bb6-d372-646abd205236: entering leader state: leader="Node at dbbb8175-4c5d-7bb6-d372-646abd205236 [Leader]" 2026-03-31T06:16:33.127Z [INFO] server-dbbb8175-4c5d-7bb6-d372-646abd205236: added peer, starting replication: peer=server-4029e8e5-63e2-6291-fb9d-1fd11ba6063a 2026-03-31T06:16:33.127Z [INFO] server-dbbb8175-4c5d-7bb6-d372-646abd205236: added peer, starting replication: peer=server-2dbd3c89-4372-d301-5a7c-4d5eb25141fd 2026-03-31T06:16:33.127Z [INFO] server-dbbb8175-4c5d-7bb6-d372-646abd205236: pipelining replication: peer="{Voter server-2dbd3c89-4372-d301-5a7c-4d5eb25141fd 2dbd3c89-4372-d301-5a7c-4d5eb25141fd}" 2026-03-31T06:16:33.127Z [INFO] server-dbbb8175-4c5d-7bb6-d372-646abd205236: pipelining replication: peer="{Voter server-4029e8e5-63e2-6291-fb9d-1fd11ba6063a 4029e8e5-63e2-6291-fb9d-1fd11ba6063a}" 2026-03-31T06:16:34.132Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:33.12779546 +0000 UTC m=+21.405402841 (1 nodes), 557.629ms from start of poll, 558.58036ms from cluster start. Timeout at 2026-03-31 06:16:34.13280114 +0000 UTC m=+22.410408521, 1.00500568s after stability 2026-03-31T06:16:34.133Z [INFO] creating new snapshot: path=/tmp/raft2998966099/snapshots/2-13-1774937794133.tmp 2026-03-31T06:16:34.133Z [INFO] server-dbbb8175-4c5d-7bb6-d372-646abd205236: copied to local snapshot: bytes=71 2026-03-31T06:16:34.133Z [INFO] server-dbbb8175-4c5d-7bb6-d372-646abd205236: snapshot restore progress: id=2-13-1774937794133 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:34.133Z [INFO] server-dbbb8175-4c5d-7bb6-d372-646abd205236: restored user snapshot: index=13 2026-03-31T06:16:34.134Z [ERROR] server-dbbb8175-4c5d-7bb6-d372-646abd205236: failed to get log: index=3 error="log not found" 2026-03-31T06:16:34.134Z [INFO] server-dbbb8175-4c5d-7bb6-d372-646abd205236: aborting pipeline replication: peer="{Voter server-2dbd3c89-4372-d301-5a7c-4d5eb25141fd 2dbd3c89-4372-d301-5a7c-4d5eb25141fd}" 2026-03-31T06:16:34.134Z [ERROR] server-dbbb8175-4c5d-7bb6-d372-646abd205236: failed to get log: index=3 error="log not found" 2026-03-31T06:16:34.134Z [INFO] server-dbbb8175-4c5d-7bb6-d372-646abd205236: aborting pipeline replication: peer="{Voter server-4029e8e5-63e2-6291-fb9d-1fd11ba6063a 4029e8e5-63e2-6291-fb9d-1fd11ba6063a}" 2026-03-31T06:16:34.139Z [ERROR] server-dbbb8175-4c5d-7bb6-d372-646abd205236: failed to get log: index=3 error="log not found" 2026-03-31T06:16:34.140Z [INFO] creating new snapshot: path=/tmp/raft3811405727/snapshots/2-13-1774937794140.tmp 2026-03-31T06:16:34.140Z [INFO] server-4029e8e5-63e2-6291-fb9d-1fd11ba6063a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:34.140Z [INFO] server-4029e8e5-63e2-6291-fb9d-1fd11ba6063a: copied to local snapshot: bytes=71 2026-03-31T06:16:34.140Z [INFO] server-4029e8e5-63e2-6291-fb9d-1fd11ba6063a: snapshot restore progress: id=2-13-1774937794140 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:34.141Z [INFO] server-4029e8e5-63e2-6291-fb9d-1fd11ba6063a: Installed remote snapshot 2026-03-31T06:16:34.141Z [INFO] server-dbbb8175-4c5d-7bb6-d372-646abd205236: pipelining replication: peer="{Voter server-4029e8e5-63e2-6291-fb9d-1fd11ba6063a 4029e8e5-63e2-6291-fb9d-1fd11ba6063a}" 2026-03-31T06:16:34.141Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:34.141Z [ERROR] server-dbbb8175-4c5d-7bb6-d372-646abd205236: failed to get log: index=3 error="log not found" 2026-03-31T06:16:34.142Z [INFO] creating new snapshot: path=/tmp/raft237358390/snapshots/2-13-1774937794142.tmp 2026-03-31T06:16:34.142Z [INFO] server-2dbd3c89-4372-d301-5a7c-4d5eb25141fd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:34.142Z [INFO] server-2dbd3c89-4372-d301-5a7c-4d5eb25141fd: copied to local snapshot: bytes=71 2026-03-31T06:16:34.143Z [INFO] server-2dbd3c89-4372-d301-5a7c-4d5eb25141fd: snapshot restore progress: id=2-13-1774937794142 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:34.143Z [INFO] server-2dbd3c89-4372-d301-5a7c-4d5eb25141fd: Installed remote snapshot 2026-03-31T06:16:34.143Z [INFO] server-dbbb8175-4c5d-7bb6-d372-646abd205236: pipelining replication: peer="{Voter server-2dbd3c89-4372-d301-5a7c-4d5eb25141fd 2dbd3c89-4372-d301-5a7c-4d5eb25141fd}" 2026-03-31T06:16:35.146Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:34.14120318 +0000 UTC m=+22.418810561 (2 nodes), 1.28µs from start of poll, 1.57198808s from cluster start. Timeout at 2026-03-31 06:16:35.14620674 +0000 UTC m=+23.423814121, 1.00500356s after stability 2026-03-31T06:16:35.159Z [INFO] server-6888f31a-0f01-ab3b-c81c-bc5b93c62a94: aborting pipeline replication: peer="{Voter server-81f2cd18-c224-6e16-c95a-72edfb8f7b43 81f2cd18-c224-6e16-c95a-72edfb8f7b43}" 2026-03-31T06:16:35.159Z [INFO] server-6888f31a-0f01-ab3b-c81c-bc5b93c62a94: aborting pipeline replication: peer="{Voter server-1290471b-0fd0-ca61-9b8e-50c783c5f8a9 1290471b-0fd0-ca61-9b8e-50c783c5f8a9}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true 2026-03-31T06:16:35.160Z [INFO] server-e59c77f4-52fb-a5b4-e2ab-7614508a7ad7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e59c77f4-52fb-a5b4-e2ab-7614508a7ad7 Address:e59c77f4-52fb-a5b4-e2ab-7614508a7ad7} {Suffrage:Voter ID:server-d1bf814d-82db-7f77-9976-78a3241bcb93 Address:d1bf814d-82db-7f77-9976-78a3241bcb93} {Suffrage:Voter ID:server-4c12f43f-dd8d-f129-e09f-06cc29032f6e Address:4c12f43f-dd8d-f129-e09f-06cc29032f6e}]" 2026-03-31T06:16:35.160Z [INFO] server-d1bf814d-82db-7f77-9976-78a3241bcb93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e59c77f4-52fb-a5b4-e2ab-7614508a7ad7 Address:e59c77f4-52fb-a5b4-e2ab-7614508a7ad7} {Suffrage:Voter ID:server-d1bf814d-82db-7f77-9976-78a3241bcb93 Address:d1bf814d-82db-7f77-9976-78a3241bcb93} {Suffrage:Voter ID:server-4c12f43f-dd8d-f129-e09f-06cc29032f6e Address:4c12f43f-dd8d-f129-e09f-06cc29032f6e}]" 2026-03-31T06:16:35.160Z [INFO] server-e59c77f4-52fb-a5b4-e2ab-7614508a7ad7: entering follower state: follower="Node at e59c77f4-52fb-a5b4-e2ab-7614508a7ad7 [Follower]" leader-address= leader-id= 2026-03-31T06:16:35.160Z [INFO] server-d1bf814d-82db-7f77-9976-78a3241bcb93: entering follower state: follower="Node at d1bf814d-82db-7f77-9976-78a3241bcb93 [Follower]" leader-address= leader-id= 2026-03-31T06:16:35.160Z [INFO] server-4c12f43f-dd8d-f129-e09f-06cc29032f6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e59c77f4-52fb-a5b4-e2ab-7614508a7ad7 Address:e59c77f4-52fb-a5b4-e2ab-7614508a7ad7} {Suffrage:Voter ID:server-d1bf814d-82db-7f77-9976-78a3241bcb93 Address:d1bf814d-82db-7f77-9976-78a3241bcb93} {Suffrage:Voter ID:server-4c12f43f-dd8d-f129-e09f-06cc29032f6e Address:4c12f43f-dd8d-f129-e09f-06cc29032f6e}]" 2026-03-31T06:16:35.160Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:35.160Z [INFO] server-4c12f43f-dd8d-f129-e09f-06cc29032f6e: entering follower state: follower="Node at 4c12f43f-dd8d-f129-e09f-06cc29032f6e [Follower]" leader-address= leader-id= 2026-03-31T06:16:36.029Z [WARN] server-d1bf814d-82db-7f77-9976-78a3241bcb93: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:36.029Z [INFO] server-d1bf814d-82db-7f77-9976-78a3241bcb93: entering candidate state: node="Node at d1bf814d-82db-7f77-9976-78a3241bcb93 [Candidate]" term=2 2026-03-31T06:16:36.029Z [INFO] server-d1bf814d-82db-7f77-9976-78a3241bcb93: election won: term=2 tally=2 2026-03-31T06:16:36.029Z [INFO] server-d1bf814d-82db-7f77-9976-78a3241bcb93: entering leader state: leader="Node at d1bf814d-82db-7f77-9976-78a3241bcb93 [Leader]" 2026-03-31T06:16:36.029Z [INFO] server-d1bf814d-82db-7f77-9976-78a3241bcb93: added peer, starting replication: peer=server-e59c77f4-52fb-a5b4-e2ab-7614508a7ad7 2026-03-31T06:16:36.029Z [INFO] server-d1bf814d-82db-7f77-9976-78a3241bcb93: added peer, starting replication: peer=server-4c12f43f-dd8d-f129-e09f-06cc29032f6e 2026-03-31T06:16:36.029Z [INFO] server-d1bf814d-82db-7f77-9976-78a3241bcb93: pipelining replication: peer="{Voter server-e59c77f4-52fb-a5b4-e2ab-7614508a7ad7 e59c77f4-52fb-a5b4-e2ab-7614508a7ad7}" 2026-03-31T06:16:36.029Z [INFO] server-d1bf814d-82db-7f77-9976-78a3241bcb93: pipelining replication: peer="{Voter server-4c12f43f-dd8d-f129-e09f-06cc29032f6e 4c12f43f-dd8d-f129-e09f-06cc29032f6e}" 2026-03-31T06:16:37.035Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:36.02953062 +0000 UTC m=+24.307138001 (1 nodes), 868.5464ms from start of poll, 868.95944ms from cluster start. Timeout at 2026-03-31 06:16:37.03453626 +0000 UTC m=+25.312143641, 1.00500564s after stability 2026-03-31T06:16:37.035Z [INFO] server-d1bf814d-82db-7f77-9976-78a3241bcb93: starting snapshot up to: index=12 2026-03-31T06:16:37.035Z [INFO] creating new snapshot: path=/tmp/raft27049692/snapshots/2-12-1774937797035.tmp 2026-03-31T06:16:37.036Z [INFO] server-d1bf814d-82db-7f77-9976-78a3241bcb93: snapshot complete up to: index=12 2026-03-31T06:16:37.038Z [INFO] server-f990da9c-3bb4-1284-fe64-6b865bb760c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f990da9c-3bb4-1284-fe64-6b865bb760c8 Address:f990da9c-3bb4-1284-fe64-6b865bb760c8} {Suffrage:Voter ID:server-c5786428-a93a-d807-cfdf-81d45b7caf1e Address:c5786428-a93a-d807-cfdf-81d45b7caf1e} {Suffrage:Voter ID:server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5 Address:5cfa8c24-ae0c-7b71-bd67-d713e599aef5}]" 2026-03-31T06:16:37.038Z [INFO] server-c5786428-a93a-d807-cfdf-81d45b7caf1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f990da9c-3bb4-1284-fe64-6b865bb760c8 Address:f990da9c-3bb4-1284-fe64-6b865bb760c8} {Suffrage:Voter ID:server-c5786428-a93a-d807-cfdf-81d45b7caf1e Address:c5786428-a93a-d807-cfdf-81d45b7caf1e} {Suffrage:Voter ID:server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5 Address:5cfa8c24-ae0c-7b71-bd67-d713e599aef5}]" 2026-03-31T06:16:37.038Z [INFO] server-f990da9c-3bb4-1284-fe64-6b865bb760c8: entering follower state: follower="Node at f990da9c-3bb4-1284-fe64-6b865bb760c8 [Follower]" leader-address= leader-id= 2026-03-31T06:16:37.038Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f990da9c-3bb4-1284-fe64-6b865bb760c8 Address:f990da9c-3bb4-1284-fe64-6b865bb760c8} {Suffrage:Voter ID:server-c5786428-a93a-d807-cfdf-81d45b7caf1e Address:c5786428-a93a-d807-cfdf-81d45b7caf1e} {Suffrage:Voter ID:server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5 Address:5cfa8c24-ae0c-7b71-bd67-d713e599aef5}]" 2026-03-31T06:16:37.038Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:37.038Z [INFO] server-c5786428-a93a-d807-cfdf-81d45b7caf1e: entering follower state: follower="Node at c5786428-a93a-d807-cfdf-81d45b7caf1e [Follower]" leader-address= leader-id= 2026-03-31T06:16:37.038Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: entering follower state: follower="Node at 5cfa8c24-ae0c-7b71-bd67-d713e599aef5 [Follower]" leader-address= leader-id= 2026-03-31T06:16:37.789Z [WARN] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:37.789Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: entering candidate state: node="Node at 5cfa8c24-ae0c-7b71-bd67-d713e599aef5 [Candidate]" term=2 2026-03-31T06:16:37.789Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: election won: term=2 tally=2 2026-03-31T06:16:37.789Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: entering leader state: leader="Node at 5cfa8c24-ae0c-7b71-bd67-d713e599aef5 [Leader]" 2026-03-31T06:16:37.789Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: added peer, starting replication: peer=server-f990da9c-3bb4-1284-fe64-6b865bb760c8 2026-03-31T06:16:37.789Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: added peer, starting replication: peer=server-c5786428-a93a-d807-cfdf-81d45b7caf1e 2026-03-31T06:16:37.789Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: pipelining replication: peer="{Voter server-f990da9c-3bb4-1284-fe64-6b865bb760c8 f990da9c-3bb4-1284-fe64-6b865bb760c8}" 2026-03-31T06:16:37.789Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: pipelining replication: peer="{Voter server-c5786428-a93a-d807-cfdf-81d45b7caf1e c5786428-a93a-d807-cfdf-81d45b7caf1e}" 2026-03-31T06:16:38.794Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:37.78958926 +0000 UTC m=+26.067196641 (1 nodes), 751.0266ms from start of poll, 751.44592ms from cluster start. Timeout at 2026-03-31 06:16:38.79459046 +0000 UTC m=+27.072197881, 1.00500124s after stability 2026-03-31T06:16:38.795Z [INFO] creating new snapshot: path=/tmp/raft1541100578/snapshots/2-13-1774937798795.tmp 2026-03-31T06:16:38.795Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: copied to local snapshot: bytes=71 2026-03-31T06:16:38.795Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: snapshot restore progress: id=2-13-1774937798795 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:38.795Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: removing all old logs from log store 2026-03-31T06:16:38.795Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: compacting logs: from=1 to=2 2026-03-31T06:16:38.795Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: restored user snapshot: index=13 2026-03-31T06:16:38.796Z [ERROR] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: failed to get log: index=2 error="log not found" 2026-03-31T06:16:38.796Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: aborting pipeline replication: peer="{Voter server-f990da9c-3bb4-1284-fe64-6b865bb760c8 f990da9c-3bb4-1284-fe64-6b865bb760c8}" 2026-03-31T06:16:38.796Z [ERROR] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: failed to get log: index=2 error="log not found" 2026-03-31T06:16:38.796Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: aborting pipeline replication: peer="{Voter server-c5786428-a93a-d807-cfdf-81d45b7caf1e c5786428-a93a-d807-cfdf-81d45b7caf1e}" 2026-03-31T06:16:38.802Z [ERROR] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: failed to get log: index=2 error="log not found" 2026-03-31T06:16:38.802Z [INFO] creating new snapshot: path=/tmp/raft255633866/snapshots/2-13-1774937798802.tmp 2026-03-31T06:16:38.802Z [INFO] server-c5786428-a93a-d807-cfdf-81d45b7caf1e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:38.803Z [INFO] server-c5786428-a93a-d807-cfdf-81d45b7caf1e: copied to local snapshot: bytes=71 2026-03-31T06:16:38.803Z [INFO] server-c5786428-a93a-d807-cfdf-81d45b7caf1e: snapshot restore progress: id=2-13-1774937798802 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:38.803Z [INFO] server-c5786428-a93a-d807-cfdf-81d45b7caf1e: removing all old logs from log store 2026-03-31T06:16:38.803Z [INFO] server-c5786428-a93a-d807-cfdf-81d45b7caf1e: compacting logs: from=1 to=2 2026-03-31T06:16:38.803Z [INFO] server-c5786428-a93a-d807-cfdf-81d45b7caf1e: Installed remote snapshot 2026-03-31T06:16:38.803Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: pipelining replication: peer="{Voter server-c5786428-a93a-d807-cfdf-81d45b7caf1e c5786428-a93a-d807-cfdf-81d45b7caf1e}" 2026-03-31T06:16:38.803Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:38.803Z [ERROR] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: failed to get log: index=2 error="log not found" 2026-03-31T06:16:38.804Z [INFO] creating new snapshot: path=/tmp/raft3581378896/snapshots/2-13-1774937798804.tmp 2026-03-31T06:16:38.804Z [INFO] server-f990da9c-3bb4-1284-fe64-6b865bb760c8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:38.804Z [INFO] server-f990da9c-3bb4-1284-fe64-6b865bb760c8: copied to local snapshot: bytes=71 2026-03-31T06:16:38.804Z [INFO] server-f990da9c-3bb4-1284-fe64-6b865bb760c8: snapshot restore progress: id=2-13-1774937798804 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:38.805Z [INFO] server-f990da9c-3bb4-1284-fe64-6b865bb760c8: removing all old logs from log store 2026-03-31T06:16:38.805Z [INFO] server-f990da9c-3bb4-1284-fe64-6b865bb760c8: compacting logs: from=1 to=2 2026-03-31T06:16:38.805Z [INFO] server-f990da9c-3bb4-1284-fe64-6b865bb760c8: Installed remote snapshot 2026-03-31T06:16:38.805Z [INFO] server-5cfa8c24-ae0c-7b71-bd67-d713e599aef5: pipelining replication: peer="{Voter server-f990da9c-3bb4-1284-fe64-6b865bb760c8 f990da9c-3bb4-1284-fe64-6b865bb760c8}" 2026-03-31T06:16:39.809Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:38.80367718 +0000 UTC m=+27.081284561 (2 nodes), 720ns from start of poll, 1.76553384s from cluster start. Timeout at 2026-03-31 06:16:39.8086885 +0000 UTC m=+28.086295881, 1.00501132s after stability 2026-03-31T06:16:39.821Z [INFO] server-d1bf814d-82db-7f77-9976-78a3241bcb93: aborting pipeline replication: peer="{Voter server-4c12f43f-dd8d-f129-e09f-06cc29032f6e 4c12f43f-dd8d-f129-e09f-06cc29032f6e}" 2026-03-31T06:16:39.821Z [INFO] server-d1bf814d-82db-7f77-9976-78a3241bcb93: aborting pipeline replication: peer="{Voter server-e59c77f4-52fb-a5b4-e2ab-7614508a7ad7 e59c77f4-52fb-a5b4-e2ab-7614508a7ad7}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false 2026-03-31T06:16:39.823Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c34ba5d-1a3d-4d4e-087e-17def3913c15 Address:5c34ba5d-1a3d-4d4e-087e-17def3913c15} {Suffrage:Voter ID:server-b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c Address:b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c} {Suffrage:Voter ID:server-0be0a92d-d930-68fe-91b9-7a5ab0c0d00a Address:0be0a92d-d930-68fe-91b9-7a5ab0c0d00a}]" 2026-03-31T06:16:39.823Z [INFO] server-b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c34ba5d-1a3d-4d4e-087e-17def3913c15 Address:5c34ba5d-1a3d-4d4e-087e-17def3913c15} {Suffrage:Voter ID:server-b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c Address:b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c} {Suffrage:Voter ID:server-0be0a92d-d930-68fe-91b9-7a5ab0c0d00a Address:0be0a92d-d930-68fe-91b9-7a5ab0c0d00a}]" 2026-03-31T06:16:39.823Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: entering follower state: follower="Node at 5c34ba5d-1a3d-4d4e-087e-17def3913c15 [Follower]" leader-address= leader-id= 2026-03-31T06:16:39.823Z [INFO] server-b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c: entering follower state: follower="Node at b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c [Follower]" leader-address= leader-id= 2026-03-31T06:16:39.823Z [INFO] server-0be0a92d-d930-68fe-91b9-7a5ab0c0d00a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c34ba5d-1a3d-4d4e-087e-17def3913c15 Address:5c34ba5d-1a3d-4d4e-087e-17def3913c15} {Suffrage:Voter ID:server-b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c Address:b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c} {Suffrage:Voter ID:server-0be0a92d-d930-68fe-91b9-7a5ab0c0d00a Address:0be0a92d-d930-68fe-91b9-7a5ab0c0d00a}]" 2026-03-31T06:16:39.823Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:39.823Z [INFO] server-0be0a92d-d930-68fe-91b9-7a5ab0c0d00a: entering follower state: follower="Node at 0be0a92d-d930-68fe-91b9-7a5ab0c0d00a [Follower]" leader-address= leader-id= 2026-03-31T06:16:40.500Z [WARN] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:40.501Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: entering candidate state: node="Node at 5c34ba5d-1a3d-4d4e-087e-17def3913c15 [Candidate]" term=2 2026-03-31T06:16:40.501Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: election won: term=2 tally=2 2026-03-31T06:16:40.501Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: entering leader state: leader="Node at 5c34ba5d-1a3d-4d4e-087e-17def3913c15 [Leader]" 2026-03-31T06:16:40.501Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: added peer, starting replication: peer=server-b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c 2026-03-31T06:16:40.501Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: added peer, starting replication: peer=server-0be0a92d-d930-68fe-91b9-7a5ab0c0d00a 2026-03-31T06:16:40.501Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: pipelining replication: peer="{Voter server-b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c}" 2026-03-31T06:16:40.501Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: pipelining replication: peer="{Voter server-0be0a92d-d930-68fe-91b9-7a5ab0c0d00a 0be0a92d-d930-68fe-91b9-7a5ab0c0d00a}" 2026-03-31T06:16:41.507Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:40.50130974 +0000 UTC m=+28.778917121 (1 nodes), 677.496ms from start of poll, 678.51804ms from cluster start. Timeout at 2026-03-31 06:16:41.50631414 +0000 UTC m=+29.783921521, 1.0050044s after stability 2026-03-31T06:16:41.507Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: starting snapshot up to: index=12 2026-03-31T06:16:41.507Z [INFO] creating new snapshot: path=/tmp/raft1896642446/snapshots/2-12-1774937801507.tmp 2026-03-31T06:16:41.508Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: snapshot complete up to: index=12 2026-03-31T06:16:41.509Z [INFO] creating new snapshot: path=/tmp/raft1896642446/snapshots/2-23-1774937801509.tmp 2026-03-31T06:16:41.509Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: copied to local snapshot: bytes=71 2026-03-31T06:16:41.510Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: snapshot restore progress: id=2-23-1774937801509 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:41.510Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: restored user snapshot: index=23 2026-03-31T06:16:41.511Z [ERROR] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: failed to get log: index=23 error="log not found" 2026-03-31T06:16:41.511Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: aborting pipeline replication: peer="{Voter server-0be0a92d-d930-68fe-91b9-7a5ab0c0d00a 0be0a92d-d930-68fe-91b9-7a5ab0c0d00a}" 2026-03-31T06:16:41.511Z [ERROR] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: failed to get log: index=23 error="log not found" 2026-03-31T06:16:41.511Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: aborting pipeline replication: peer="{Voter server-b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c}" 2026-03-31T06:16:41.517Z [ERROR] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: failed to get log: index=23 error="log not found" 2026-03-31T06:16:41.517Z [INFO] creating new snapshot: path=/tmp/raft1122762306/snapshots/2-23-1774937801517.tmp 2026-03-31T06:16:41.518Z [INFO] server-0be0a92d-d930-68fe-91b9-7a5ab0c0d00a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:41.518Z [INFO] server-0be0a92d-d930-68fe-91b9-7a5ab0c0d00a: copied to local snapshot: bytes=71 2026-03-31T06:16:41.518Z [ERROR] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: failed to get log: index=23 error="log not found" 2026-03-31T06:16:41.518Z [INFO] server-0be0a92d-d930-68fe-91b9-7a5ab0c0d00a: snapshot restore progress: id=2-23-1774937801517 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:41.518Z [INFO] server-0be0a92d-d930-68fe-91b9-7a5ab0c0d00a: Installed remote snapshot 2026-03-31T06:16:41.518Z [WARN] server-0be0a92d-d930-68fe-91b9-7a5ab0c0d00a: skipping application of old log: index=22 2026-03-31T06:16:41.518Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: pipelining replication: peer="{Voter server-0be0a92d-d930-68fe-91b9-7a5ab0c0d00a 0be0a92d-d930-68fe-91b9-7a5ab0c0d00a}" 2026-03-31T06:16:41.518Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:41.518Z [INFO] creating new snapshot: path=/tmp/raft2636022118/snapshots/2-23-1774937801518.tmp 2026-03-31T06:16:41.519Z [INFO] server-b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:41.519Z [INFO] server-b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c: copied to local snapshot: bytes=71 2026-03-31T06:16:41.519Z [INFO] server-b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c: snapshot restore progress: id=2-23-1774937801518 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:41.519Z [INFO] server-b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c: Installed remote snapshot 2026-03-31T06:16:41.519Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: pipelining replication: peer="{Voter server-b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c}" 2026-03-31T06:16:42.524Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:41.51883926 +0000 UTC m=+29.796446641 (2 nodes), 920ns from start of poll, 1.69604756s from cluster start. Timeout at 2026-03-31 06:16:42.5238473 +0000 UTC m=+30.801454681, 1.00500804s after stability 2026-03-31T06:16:42.538Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: aborting pipeline replication: peer="{Voter server-0be0a92d-d930-68fe-91b9-7a5ab0c0d00a 0be0a92d-d930-68fe-91b9-7a5ab0c0d00a}" 2026-03-31T06:16:42.538Z [INFO] server-5c34ba5d-1a3d-4d4e-087e-17def3913c15: aborting pipeline replication: peer="{Voter server-b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c b2bae6bb-39c9-371e-0aa6-d6cd2d6cac5c}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false 2026-03-31T06:16:42.539Z [INFO] server-98dd69e3-084a-8ea0-013e-387f33f8c1bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98dd69e3-084a-8ea0-013e-387f33f8c1bc Address:98dd69e3-084a-8ea0-013e-387f33f8c1bc} {Suffrage:Voter ID:server-0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9 Address:0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9} {Suffrage:Voter ID:server-8c1fffcd-5833-5365-3d14-a7327fdfcbed Address:8c1fffcd-5833-5365-3d14-a7327fdfcbed}]" 2026-03-31T06:16:42.539Z [INFO] server-0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98dd69e3-084a-8ea0-013e-387f33f8c1bc Address:98dd69e3-084a-8ea0-013e-387f33f8c1bc} {Suffrage:Voter ID:server-0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9 Address:0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9} {Suffrage:Voter ID:server-8c1fffcd-5833-5365-3d14-a7327fdfcbed Address:8c1fffcd-5833-5365-3d14-a7327fdfcbed}]" 2026-03-31T06:16:42.539Z [INFO] server-98dd69e3-084a-8ea0-013e-387f33f8c1bc: entering follower state: follower="Node at 98dd69e3-084a-8ea0-013e-387f33f8c1bc [Follower]" leader-address= leader-id= 2026-03-31T06:16:42.539Z [INFO] server-0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9: entering follower state: follower="Node at 0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9 [Follower]" leader-address= leader-id= 2026-03-31T06:16:42.539Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98dd69e3-084a-8ea0-013e-387f33f8c1bc Address:98dd69e3-084a-8ea0-013e-387f33f8c1bc} {Suffrage:Voter ID:server-0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9 Address:0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9} {Suffrage:Voter ID:server-8c1fffcd-5833-5365-3d14-a7327fdfcbed Address:8c1fffcd-5833-5365-3d14-a7327fdfcbed}]" 2026-03-31T06:16:42.539Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:42.539Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: entering follower state: follower="Node at 8c1fffcd-5833-5365-3d14-a7327fdfcbed [Follower]" leader-address= leader-id= 2026-03-31T06:16:43.128Z [WARN] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:43.128Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: entering candidate state: node="Node at 8c1fffcd-5833-5365-3d14-a7327fdfcbed [Candidate]" term=2 2026-03-31T06:16:43.128Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: election won: term=2 tally=2 2026-03-31T06:16:43.128Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: entering leader state: leader="Node at 8c1fffcd-5833-5365-3d14-a7327fdfcbed [Leader]" 2026-03-31T06:16:43.128Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: added peer, starting replication: peer=server-98dd69e3-084a-8ea0-013e-387f33f8c1bc 2026-03-31T06:16:43.128Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: added peer, starting replication: peer=server-0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9 2026-03-31T06:16:43.128Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: pipelining replication: peer="{Voter server-0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9 0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9}" 2026-03-31T06:16:43.128Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: pipelining replication: peer="{Voter server-98dd69e3-084a-8ea0-013e-387f33f8c1bc 98dd69e3-084a-8ea0-013e-387f33f8c1bc}" 2026-03-31T06:16:44.133Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:43.12822926 +0000 UTC m=+31.405836641 (1 nodes), 588.70296ms from start of poll, 589.16124ms from cluster start. Timeout at 2026-03-31 06:16:44.13323598 +0000 UTC m=+32.410843361, 1.00500672s after stability 2026-03-31T06:16:44.134Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: starting snapshot up to: index=12 2026-03-31T06:16:44.134Z [INFO] creating new snapshot: path=/tmp/raft3710432933/snapshots/2-12-1774937804134.tmp 2026-03-31T06:16:44.135Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: snapshot complete up to: index=12 2026-03-31T06:16:44.136Z [INFO] creating new snapshot: path=/tmp/raft3710432933/snapshots/2-23-1774937804136.tmp 2026-03-31T06:16:44.136Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: copied to local snapshot: bytes=71 2026-03-31T06:16:44.136Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: snapshot restore progress: id=2-23-1774937804136 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:44.137Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: removing all old logs from log store 2026-03-31T06:16:44.137Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: compacting logs: from=1 to=22 2026-03-31T06:16:44.137Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: restored user snapshot: index=23 2026-03-31T06:16:44.137Z [ERROR] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: failed to get log: index=22 error="log not found" 2026-03-31T06:16:44.137Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: aborting pipeline replication: peer="{Voter server-98dd69e3-084a-8ea0-013e-387f33f8c1bc 98dd69e3-084a-8ea0-013e-387f33f8c1bc}" 2026-03-31T06:16:44.137Z [ERROR] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: failed to get log: index=22 error="log not found" 2026-03-31T06:16:44.137Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: aborting pipeline replication: peer="{Voter server-0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9 0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9}" 2026-03-31T06:16:44.143Z [ERROR] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: failed to get log: index=22 error="log not found" 2026-03-31T06:16:44.143Z [ERROR] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: failed to get log: index=22 error="log not found" 2026-03-31T06:16:44.145Z [INFO] creating new snapshot: path=/tmp/raft3472649605/snapshots/2-23-1774937804145.tmp 2026-03-31T06:16:44.146Z [INFO] creating new snapshot: path=/tmp/raft1077602195/snapshots/2-23-1774937804146.tmp 2026-03-31T06:16:44.146Z [INFO] server-0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:44.146Z [INFO] server-98dd69e3-084a-8ea0-013e-387f33f8c1bc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:44.146Z [INFO] server-0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9: copied to local snapshot: bytes=71 2026-03-31T06:16:44.146Z [INFO] server-98dd69e3-084a-8ea0-013e-387f33f8c1bc: copied to local snapshot: bytes=71 2026-03-31T06:16:44.146Z [INFO] server-0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9: snapshot restore progress: id=2-23-1774937804145 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:44.146Z [INFO] server-0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9: removing all old logs from log store 2026-03-31T06:16:44.146Z [INFO] server-0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9: compacting logs: from=1 to=22 2026-03-31T06:16:44.146Z [INFO] server-0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9: Installed remote snapshot 2026-03-31T06:16:44.146Z [INFO] server-98dd69e3-084a-8ea0-013e-387f33f8c1bc: snapshot restore progress: id=2-23-1774937804146 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:44.146Z [INFO] server-98dd69e3-084a-8ea0-013e-387f33f8c1bc: removing all old logs from log store 2026-03-31T06:16:44.146Z [INFO] server-98dd69e3-084a-8ea0-013e-387f33f8c1bc: compacting logs: from=1 to=22 2026-03-31T06:16:44.146Z [WARN] server-0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9: skipping application of old log: index=22 2026-03-31T06:16:44.146Z [INFO] server-98dd69e3-084a-8ea0-013e-387f33f8c1bc: Installed remote snapshot 2026-03-31T06:16:44.146Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: pipelining replication: peer="{Voter server-0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9 0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9}" 2026-03-31T06:16:44.146Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: pipelining replication: peer="{Voter server-98dd69e3-084a-8ea0-013e-387f33f8c1bc 98dd69e3-084a-8ea0-013e-387f33f8c1bc}" 2026-03-31T06:16:44.146Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:45.152Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:44.14699926 +0000 UTC m=+32.424606641 (2 nodes), 3.88µs from start of poll, 1.60793124s from cluster start. Timeout at 2026-03-31 06:16:45.1520085 +0000 UTC m=+33.429615881, 1.00500924s after stability 2026-03-31T06:16:45.165Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: aborting pipeline replication: peer="{Voter server-0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9 0af4e891-8cfd-6b4b-79a4-0e68bf25b3c9}" 2026-03-31T06:16:45.165Z [INFO] server-8c1fffcd-5833-5365-3d14-a7327fdfcbed: aborting pipeline replication: peer="{Voter server-98dd69e3-084a-8ea0-013e-387f33f8c1bc 98dd69e3-084a-8ea0-013e-387f33f8c1bc}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true 2026-03-31T06:16:45.167Z [INFO] server-e5b7b0ed-32c8-7309-a352-5645ef13f920: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e5b7b0ed-32c8-7309-a352-5645ef13f920 Address:e5b7b0ed-32c8-7309-a352-5645ef13f920} {Suffrage:Voter ID:server-f78d6045-bd91-9be4-0def-faddaffb7fba Address:f78d6045-bd91-9be4-0def-faddaffb7fba} {Suffrage:Voter ID:server-c24e2188-81de-f73d-f9e4-670caeb5a3b2 Address:c24e2188-81de-f73d-f9e4-670caeb5a3b2}]" 2026-03-31T06:16:45.167Z [INFO] server-e5b7b0ed-32c8-7309-a352-5645ef13f920: entering follower state: follower="Node at e5b7b0ed-32c8-7309-a352-5645ef13f920 [Follower]" leader-address= leader-id= 2026-03-31T06:16:45.167Z [INFO] server-f78d6045-bd91-9be4-0def-faddaffb7fba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e5b7b0ed-32c8-7309-a352-5645ef13f920 Address:e5b7b0ed-32c8-7309-a352-5645ef13f920} {Suffrage:Voter ID:server-f78d6045-bd91-9be4-0def-faddaffb7fba Address:f78d6045-bd91-9be4-0def-faddaffb7fba} {Suffrage:Voter ID:server-c24e2188-81de-f73d-f9e4-670caeb5a3b2 Address:c24e2188-81de-f73d-f9e4-670caeb5a3b2}]" 2026-03-31T06:16:45.167Z [INFO] server-f78d6045-bd91-9be4-0def-faddaffb7fba: entering follower state: follower="Node at f78d6045-bd91-9be4-0def-faddaffb7fba [Follower]" leader-address= leader-id= 2026-03-31T06:16:45.167Z [INFO] server-c24e2188-81de-f73d-f9e4-670caeb5a3b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e5b7b0ed-32c8-7309-a352-5645ef13f920 Address:e5b7b0ed-32c8-7309-a352-5645ef13f920} {Suffrage:Voter ID:server-f78d6045-bd91-9be4-0def-faddaffb7fba Address:f78d6045-bd91-9be4-0def-faddaffb7fba} {Suffrage:Voter ID:server-c24e2188-81de-f73d-f9e4-670caeb5a3b2 Address:c24e2188-81de-f73d-f9e4-670caeb5a3b2}]" 2026-03-31T06:16:45.168Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:45.168Z [INFO] server-c24e2188-81de-f73d-f9e4-670caeb5a3b2: entering follower state: follower="Node at c24e2188-81de-f73d-f9e4-670caeb5a3b2 [Follower]" leader-address= leader-id= 2026-03-31T06:16:45.933Z [WARN] server-e5b7b0ed-32c8-7309-a352-5645ef13f920: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:45.933Z [INFO] server-e5b7b0ed-32c8-7309-a352-5645ef13f920: entering candidate state: node="Node at e5b7b0ed-32c8-7309-a352-5645ef13f920 [Candidate]" term=2 2026-03-31T06:16:45.933Z [INFO] server-e5b7b0ed-32c8-7309-a352-5645ef13f920: election won: term=2 tally=2 2026-03-31T06:16:45.933Z [INFO] server-e5b7b0ed-32c8-7309-a352-5645ef13f920: entering leader state: leader="Node at e5b7b0ed-32c8-7309-a352-5645ef13f920 [Leader]" 2026-03-31T06:16:45.933Z [INFO] server-e5b7b0ed-32c8-7309-a352-5645ef13f920: added peer, starting replication: peer=server-f78d6045-bd91-9be4-0def-faddaffb7fba 2026-03-31T06:16:45.933Z [INFO] server-e5b7b0ed-32c8-7309-a352-5645ef13f920: added peer, starting replication: peer=server-c24e2188-81de-f73d-f9e4-670caeb5a3b2 2026-03-31T06:16:45.933Z [INFO] server-e5b7b0ed-32c8-7309-a352-5645ef13f920: pipelining replication: peer="{Voter server-c24e2188-81de-f73d-f9e4-670caeb5a3b2 c24e2188-81de-f73d-f9e4-670caeb5a3b2}" 2026-03-31T06:16:45.933Z [INFO] server-e5b7b0ed-32c8-7309-a352-5645ef13f920: pipelining replication: peer="{Voter server-f78d6045-bd91-9be4-0def-faddaffb7fba f78d6045-bd91-9be4-0def-faddaffb7fba}" 2026-03-31T06:16:46.939Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:45.93366334 +0000 UTC m=+34.211270721 (1 nodes), 765.55312ms from start of poll, 766.55288ms from cluster start. Timeout at 2026-03-31 06:16:46.9386689 +0000 UTC m=+35.216276281, 1.00500556s after stability 2026-03-31T06:16:46.939Z [INFO] server-e5b7b0ed-32c8-7309-a352-5645ef13f920: starting snapshot up to: index=12 2026-03-31T06:16:46.939Z [INFO] creating new snapshot: path=/tmp/raft933494939/snapshots/2-12-1774937806939.tmp 2026-03-31T06:16:46.940Z [INFO] server-e5b7b0ed-32c8-7309-a352-5645ef13f920: snapshot complete up to: index=12 2026-03-31T06:16:46.941Z [INFO] server-90eedf0d-3c89-750b-6864-4c3da44b7e91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90eedf0d-3c89-750b-6864-4c3da44b7e91 Address:90eedf0d-3c89-750b-6864-4c3da44b7e91} {Suffrage:Voter ID:server-aeee5fc3-30a2-d120-fcdc-983714743ad2 Address:aeee5fc3-30a2-d120-fcdc-983714743ad2} {Suffrage:Voter ID:server-28cbb796-4f9b-647d-2b44-2ff0186643dc Address:28cbb796-4f9b-647d-2b44-2ff0186643dc}]" 2026-03-31T06:16:46.941Z [INFO] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90eedf0d-3c89-750b-6864-4c3da44b7e91 Address:90eedf0d-3c89-750b-6864-4c3da44b7e91} {Suffrage:Voter ID:server-aeee5fc3-30a2-d120-fcdc-983714743ad2 Address:aeee5fc3-30a2-d120-fcdc-983714743ad2} {Suffrage:Voter ID:server-28cbb796-4f9b-647d-2b44-2ff0186643dc Address:28cbb796-4f9b-647d-2b44-2ff0186643dc}]" 2026-03-31T06:16:46.941Z [INFO] server-28cbb796-4f9b-647d-2b44-2ff0186643dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90eedf0d-3c89-750b-6864-4c3da44b7e91 Address:90eedf0d-3c89-750b-6864-4c3da44b7e91} {Suffrage:Voter ID:server-aeee5fc3-30a2-d120-fcdc-983714743ad2 Address:aeee5fc3-30a2-d120-fcdc-983714743ad2} {Suffrage:Voter ID:server-28cbb796-4f9b-647d-2b44-2ff0186643dc Address:28cbb796-4f9b-647d-2b44-2ff0186643dc}]" 2026-03-31T06:16:46.941Z [INFO] server-90eedf0d-3c89-750b-6864-4c3da44b7e91: entering follower state: follower="Node at 90eedf0d-3c89-750b-6864-4c3da44b7e91 [Follower]" leader-address= leader-id= 2026-03-31T06:16:46.941Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:46.941Z [INFO] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: entering follower state: follower="Node at aeee5fc3-30a2-d120-fcdc-983714743ad2 [Follower]" leader-address= leader-id= 2026-03-31T06:16:46.941Z [INFO] server-28cbb796-4f9b-647d-2b44-2ff0186643dc: entering follower state: follower="Node at 28cbb796-4f9b-647d-2b44-2ff0186643dc [Follower]" leader-address= leader-id= 2026-03-31T06:16:47.582Z [WARN] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:47.582Z [INFO] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: entering candidate state: node="Node at aeee5fc3-30a2-d120-fcdc-983714743ad2 [Candidate]" term=2 2026-03-31T06:16:47.582Z [INFO] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: election won: term=2 tally=2 2026-03-31T06:16:47.582Z [INFO] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: entering leader state: leader="Node at aeee5fc3-30a2-d120-fcdc-983714743ad2 [Leader]" 2026-03-31T06:16:47.582Z [INFO] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: added peer, starting replication: peer=server-90eedf0d-3c89-750b-6864-4c3da44b7e91 2026-03-31T06:16:47.582Z [INFO] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: added peer, starting replication: peer=server-28cbb796-4f9b-647d-2b44-2ff0186643dc 2026-03-31T06:16:47.582Z [INFO] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: pipelining replication: peer="{Voter server-28cbb796-4f9b-647d-2b44-2ff0186643dc 28cbb796-4f9b-647d-2b44-2ff0186643dc}" 2026-03-31T06:16:47.582Z [INFO] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: pipelining replication: peer="{Voter server-90eedf0d-3c89-750b-6864-4c3da44b7e91 90eedf0d-3c89-750b-6864-4c3da44b7e91}" 2026-03-31T06:16:48.587Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:47.5823271 +0000 UTC m=+35.859934481 (1 nodes), 640.9116ms from start of poll, 641.334ms from cluster start. Timeout at 2026-03-31 06:16:48.5874123 +0000 UTC m=+36.865019721, 1.00508524s after stability 2026-03-31T06:16:48.587Z [INFO] creating new snapshot: path=/tmp/raft2488077100/snapshots/2-14-1774937808587.tmp 2026-03-31T06:16:48.588Z [INFO] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: copied to local snapshot: bytes=71 2026-03-31T06:16:48.588Z [INFO] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: snapshot restore progress: id=2-14-1774937808587 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:48.588Z [INFO] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: restored user snapshot: index=14 2026-03-31T06:16:48.589Z [ERROR] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: failed to get log: index=3 error="log not found" 2026-03-31T06:16:48.589Z [INFO] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: aborting pipeline replication: peer="{Voter server-28cbb796-4f9b-647d-2b44-2ff0186643dc 28cbb796-4f9b-647d-2b44-2ff0186643dc}" 2026-03-31T06:16:48.589Z [ERROR] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: failed to get log: index=3 error="log not found" 2026-03-31T06:16:48.589Z [INFO] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: aborting pipeline replication: peer="{Voter server-90eedf0d-3c89-750b-6864-4c3da44b7e91 90eedf0d-3c89-750b-6864-4c3da44b7e91}" 2026-03-31T06:16:48.598Z [ERROR] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: failed to get log: index=3 error="log not found" 2026-03-31T06:16:48.598Z [ERROR] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: failed to get log: index=3 error="log not found" 2026-03-31T06:16:48.598Z [INFO] creating new snapshot: path=/tmp/raft3736991391/snapshots/2-14-1774937808598.tmp 2026-03-31T06:16:48.598Z [INFO] creating new snapshot: path=/tmp/raft568671114/snapshots/2-14-1774937808598.tmp 2026-03-31T06:16:48.599Z [INFO] server-28cbb796-4f9b-647d-2b44-2ff0186643dc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:48.599Z [INFO] server-90eedf0d-3c89-750b-6864-4c3da44b7e91: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:48.599Z [INFO] server-28cbb796-4f9b-647d-2b44-2ff0186643dc: copied to local snapshot: bytes=71 2026-03-31T06:16:48.599Z [INFO] server-90eedf0d-3c89-750b-6864-4c3da44b7e91: copied to local snapshot: bytes=71 2026-03-31T06:16:48.599Z [INFO] server-90eedf0d-3c89-750b-6864-4c3da44b7e91: snapshot restore progress: id=2-14-1774937808598 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:48.599Z [INFO] server-90eedf0d-3c89-750b-6864-4c3da44b7e91: Installed remote snapshot 2026-03-31T06:16:48.599Z [INFO] server-28cbb796-4f9b-647d-2b44-2ff0186643dc: snapshot restore progress: id=2-14-1774937808598 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:48.599Z [INFO] server-28cbb796-4f9b-647d-2b44-2ff0186643dc: Installed remote snapshot 2026-03-31T06:16:48.599Z [INFO] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: pipelining replication: peer="{Voter server-90eedf0d-3c89-750b-6864-4c3da44b7e91 90eedf0d-3c89-750b-6864-4c3da44b7e91}" 2026-03-31T06:16:48.599Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:48.600Z [INFO] server-aeee5fc3-30a2-d120-fcdc-983714743ad2: pipelining replication: peer="{Voter server-28cbb796-4f9b-647d-2b44-2ff0186643dc 28cbb796-4f9b-647d-2b44-2ff0186643dc}" 2026-03-31T06:16:49.605Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:48.60000786 +0000 UTC m=+36.877615241 (2 nodes), 1.08µs from start of poll, 1.65901476s from cluster start. Timeout at 2026-03-31 06:16:49.60502398 +0000 UTC m=+37.882631361, 1.00501612s after stability 2026-03-31T06:16:49.619Z [INFO] server-e5b7b0ed-32c8-7309-a352-5645ef13f920: aborting pipeline replication: peer="{Voter server-c24e2188-81de-f73d-f9e4-670caeb5a3b2 c24e2188-81de-f73d-f9e4-670caeb5a3b2}" 2026-03-31T06:16:49.619Z [INFO] server-e5b7b0ed-32c8-7309-a352-5645ef13f920: aborting pipeline replication: peer="{Voter server-f78d6045-bd91-9be4-0def-faddaffb7fba f78d6045-bd91-9be4-0def-faddaffb7fba}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true 2026-03-31T06:16:49.620Z [INFO] server-c69cb052-0d27-1ea9-113e-a81a09f0fa93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c69cb052-0d27-1ea9-113e-a81a09f0fa93 Address:c69cb052-0d27-1ea9-113e-a81a09f0fa93} {Suffrage:Voter ID:server-29f61a21-4097-8f91-9dd8-1aa115d62c1b Address:29f61a21-4097-8f91-9dd8-1aa115d62c1b} {Suffrage:Voter ID:server-b249238b-5ec9-11b5-2fc6-2f7a7a87a2da Address:b249238b-5ec9-11b5-2fc6-2f7a7a87a2da}]" 2026-03-31T06:16:49.620Z [INFO] server-29f61a21-4097-8f91-9dd8-1aa115d62c1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c69cb052-0d27-1ea9-113e-a81a09f0fa93 Address:c69cb052-0d27-1ea9-113e-a81a09f0fa93} {Suffrage:Voter ID:server-29f61a21-4097-8f91-9dd8-1aa115d62c1b Address:29f61a21-4097-8f91-9dd8-1aa115d62c1b} {Suffrage:Voter ID:server-b249238b-5ec9-11b5-2fc6-2f7a7a87a2da Address:b249238b-5ec9-11b5-2fc6-2f7a7a87a2da}]" 2026-03-31T06:16:49.620Z [INFO] server-c69cb052-0d27-1ea9-113e-a81a09f0fa93: entering follower state: follower="Node at c69cb052-0d27-1ea9-113e-a81a09f0fa93 [Follower]" leader-address= leader-id= 2026-03-31T06:16:49.620Z [INFO] server-29f61a21-4097-8f91-9dd8-1aa115d62c1b: entering follower state: follower="Node at 29f61a21-4097-8f91-9dd8-1aa115d62c1b [Follower]" leader-address= leader-id= 2026-03-31T06:16:49.620Z [INFO] server-b249238b-5ec9-11b5-2fc6-2f7a7a87a2da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c69cb052-0d27-1ea9-113e-a81a09f0fa93 Address:c69cb052-0d27-1ea9-113e-a81a09f0fa93} {Suffrage:Voter ID:server-29f61a21-4097-8f91-9dd8-1aa115d62c1b Address:29f61a21-4097-8f91-9dd8-1aa115d62c1b} {Suffrage:Voter ID:server-b249238b-5ec9-11b5-2fc6-2f7a7a87a2da Address:b249238b-5ec9-11b5-2fc6-2f7a7a87a2da}]" 2026-03-31T06:16:49.620Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:49.620Z [INFO] server-b249238b-5ec9-11b5-2fc6-2f7a7a87a2da: entering follower state: follower="Node at b249238b-5ec9-11b5-2fc6-2f7a7a87a2da [Follower]" leader-address= leader-id= 2026-03-31T06:16:50.331Z [WARN] server-b249238b-5ec9-11b5-2fc6-2f7a7a87a2da: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:50.331Z [INFO] server-b249238b-5ec9-11b5-2fc6-2f7a7a87a2da: entering candidate state: node="Node at b249238b-5ec9-11b5-2fc6-2f7a7a87a2da [Candidate]" term=2 2026-03-31T06:16:50.331Z [INFO] server-b249238b-5ec9-11b5-2fc6-2f7a7a87a2da: election won: term=2 tally=2 2026-03-31T06:16:50.331Z [INFO] server-b249238b-5ec9-11b5-2fc6-2f7a7a87a2da: entering leader state: leader="Node at b249238b-5ec9-11b5-2fc6-2f7a7a87a2da [Leader]" 2026-03-31T06:16:50.331Z [INFO] server-b249238b-5ec9-11b5-2fc6-2f7a7a87a2da: added peer, starting replication: peer=server-c69cb052-0d27-1ea9-113e-a81a09f0fa93 2026-03-31T06:16:50.331Z [INFO] server-b249238b-5ec9-11b5-2fc6-2f7a7a87a2da: added peer, starting replication: peer=server-29f61a21-4097-8f91-9dd8-1aa115d62c1b 2026-03-31T06:16:50.331Z [INFO] server-b249238b-5ec9-11b5-2fc6-2f7a7a87a2da: pipelining replication: peer="{Voter server-c69cb052-0d27-1ea9-113e-a81a09f0fa93 c69cb052-0d27-1ea9-113e-a81a09f0fa93}" 2026-03-31T06:16:50.331Z [INFO] server-b249238b-5ec9-11b5-2fc6-2f7a7a87a2da: pipelining replication: peer="{Voter server-29f61a21-4097-8f91-9dd8-1aa115d62c1b 29f61a21-4097-8f91-9dd8-1aa115d62c1b}" 2026-03-31T06:16:51.336Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:50.33150638 +0000 UTC m=+38.609113761 (1 nodes), 710.79532ms from start of poll, 711.21264ms from cluster start. Timeout at 2026-03-31 06:16:51.33651662 +0000 UTC m=+39.614124001, 1.00501024s after stability 2026-03-31T06:16:51.337Z [INFO] server-b249238b-5ec9-11b5-2fc6-2f7a7a87a2da: starting snapshot up to: index=12 2026-03-31T06:16:51.337Z [INFO] creating new snapshot: path=/tmp/raft810639377/snapshots/2-12-1774937811337.tmp 2026-03-31T06:16:51.337Z [INFO] server-b249238b-5ec9-11b5-2fc6-2f7a7a87a2da: snapshot complete up to: index=12 2026-03-31T06:16:51.338Z [INFO] server-4b659f42-3962-81fd-8af2-c8865b770f62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b659f42-3962-81fd-8af2-c8865b770f62 Address:4b659f42-3962-81fd-8af2-c8865b770f62} {Suffrage:Voter ID:server-0f4237d3-95a2-ffe7-11b5-b9b630241781 Address:0f4237d3-95a2-ffe7-11b5-b9b630241781} {Suffrage:Voter ID:server-1f568bcf-75bf-a19a-4ea3-6b75ad05af3f Address:1f568bcf-75bf-a19a-4ea3-6b75ad05af3f}]" 2026-03-31T06:16:51.338Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b659f42-3962-81fd-8af2-c8865b770f62 Address:4b659f42-3962-81fd-8af2-c8865b770f62} {Suffrage:Voter ID:server-0f4237d3-95a2-ffe7-11b5-b9b630241781 Address:0f4237d3-95a2-ffe7-11b5-b9b630241781} {Suffrage:Voter ID:server-1f568bcf-75bf-a19a-4ea3-6b75ad05af3f Address:1f568bcf-75bf-a19a-4ea3-6b75ad05af3f}]" 2026-03-31T06:16:51.338Z [INFO] server-4b659f42-3962-81fd-8af2-c8865b770f62: entering follower state: follower="Node at 4b659f42-3962-81fd-8af2-c8865b770f62 [Follower]" leader-address= leader-id= 2026-03-31T06:16:51.339Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: entering follower state: follower="Node at 0f4237d3-95a2-ffe7-11b5-b9b630241781 [Follower]" leader-address= leader-id= 2026-03-31T06:16:51.339Z [INFO] server-1f568bcf-75bf-a19a-4ea3-6b75ad05af3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b659f42-3962-81fd-8af2-c8865b770f62 Address:4b659f42-3962-81fd-8af2-c8865b770f62} {Suffrage:Voter ID:server-0f4237d3-95a2-ffe7-11b5-b9b630241781 Address:0f4237d3-95a2-ffe7-11b5-b9b630241781} {Suffrage:Voter ID:server-1f568bcf-75bf-a19a-4ea3-6b75ad05af3f Address:1f568bcf-75bf-a19a-4ea3-6b75ad05af3f}]" 2026-03-31T06:16:51.339Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:51.339Z [INFO] server-1f568bcf-75bf-a19a-4ea3-6b75ad05af3f: entering follower state: follower="Node at 1f568bcf-75bf-a19a-4ea3-6b75ad05af3f [Follower]" leader-address= leader-id= 2026-03-31T06:16:51.901Z [WARN] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:51.901Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: entering candidate state: node="Node at 0f4237d3-95a2-ffe7-11b5-b9b630241781 [Candidate]" term=2 2026-03-31T06:16:51.901Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: election won: term=2 tally=2 2026-03-31T06:16:51.901Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: entering leader state: leader="Node at 0f4237d3-95a2-ffe7-11b5-b9b630241781 [Leader]" 2026-03-31T06:16:51.901Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: added peer, starting replication: peer=server-4b659f42-3962-81fd-8af2-c8865b770f62 2026-03-31T06:16:51.901Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: added peer, starting replication: peer=server-1f568bcf-75bf-a19a-4ea3-6b75ad05af3f 2026-03-31T06:16:51.902Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: pipelining replication: peer="{Voter server-1f568bcf-75bf-a19a-4ea3-6b75ad05af3f 1f568bcf-75bf-a19a-4ea3-6b75ad05af3f}" 2026-03-31T06:16:51.902Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: pipelining replication: peer="{Voter server-4b659f42-3962-81fd-8af2-c8865b770f62 4b659f42-3962-81fd-8af2-c8865b770f62}" 2026-03-31T06:16:52.907Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:51.9019755 +0000 UTC m=+40.179582881 (1 nodes), 562.79412ms from start of poll, 563.27096ms from cluster start. Timeout at 2026-03-31 06:16:52.90697702 +0000 UTC m=+41.184584401, 1.00500152s after stability 2026-03-31T06:16:52.908Z [INFO] creating new snapshot: path=/tmp/raft2866743044/snapshots/2-14-1774937812908.tmp 2026-03-31T06:16:52.909Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: copied to local snapshot: bytes=71 2026-03-31T06:16:52.909Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: snapshot restore progress: id=2-14-1774937812908 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:52.909Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: removing all old logs from log store 2026-03-31T06:16:52.909Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: compacting logs: from=1 to=2 2026-03-31T06:16:52.909Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: restored user snapshot: index=14 2026-03-31T06:16:52.909Z [ERROR] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: failed to get log: index=2 error="log not found" 2026-03-31T06:16:52.909Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: aborting pipeline replication: peer="{Voter server-4b659f42-3962-81fd-8af2-c8865b770f62 4b659f42-3962-81fd-8af2-c8865b770f62}" 2026-03-31T06:16:52.909Z [ERROR] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: failed to get log: index=2 error="log not found" 2026-03-31T06:16:52.909Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: aborting pipeline replication: peer="{Voter server-1f568bcf-75bf-a19a-4ea3-6b75ad05af3f 1f568bcf-75bf-a19a-4ea3-6b75ad05af3f}" 2026-03-31T06:16:52.919Z [ERROR] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: failed to get log: index=2 error="log not found" 2026-03-31T06:16:52.919Z [ERROR] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: failed to get log: index=2 error="log not found" 2026-03-31T06:16:52.920Z [INFO] creating new snapshot: path=/tmp/raft901924995/snapshots/2-14-1774937812920.tmp 2026-03-31T06:16:52.920Z [INFO] creating new snapshot: path=/tmp/raft913811481/snapshots/2-14-1774937812920.tmp 2026-03-31T06:16:52.920Z [INFO] server-1f568bcf-75bf-a19a-4ea3-6b75ad05af3f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:52.920Z [INFO] server-4b659f42-3962-81fd-8af2-c8865b770f62: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:52.920Z [INFO] server-1f568bcf-75bf-a19a-4ea3-6b75ad05af3f: copied to local snapshot: bytes=71 2026-03-31T06:16:52.920Z [INFO] server-4b659f42-3962-81fd-8af2-c8865b770f62: copied to local snapshot: bytes=71 2026-03-31T06:16:52.920Z [INFO] server-1f568bcf-75bf-a19a-4ea3-6b75ad05af3f: snapshot restore progress: id=2-14-1774937812920 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:52.920Z [INFO] server-1f568bcf-75bf-a19a-4ea3-6b75ad05af3f: removing all old logs from log store 2026-03-31T06:16:52.920Z [INFO] server-1f568bcf-75bf-a19a-4ea3-6b75ad05af3f: compacting logs: from=1 to=2 2026-03-31T06:16:52.921Z [INFO] server-1f568bcf-75bf-a19a-4ea3-6b75ad05af3f: Installed remote snapshot 2026-03-31T06:16:52.920Z [INFO] server-4b659f42-3962-81fd-8af2-c8865b770f62: snapshot restore progress: id=2-14-1774937812920 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:52.921Z [INFO] server-4b659f42-3962-81fd-8af2-c8865b770f62: removing all old logs from log store 2026-03-31T06:16:52.921Z [INFO] server-4b659f42-3962-81fd-8af2-c8865b770f62: compacting logs: from=1 to=2 2026-03-31T06:16:52.921Z [INFO] server-4b659f42-3962-81fd-8af2-c8865b770f62: Installed remote snapshot 2026-03-31T06:16:52.921Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: pipelining replication: peer="{Voter server-1f568bcf-75bf-a19a-4ea3-6b75ad05af3f 1f568bcf-75bf-a19a-4ea3-6b75ad05af3f}" 2026-03-31T06:16:52.921Z [INFO] server-0f4237d3-95a2-ffe7-11b5-b9b630241781: pipelining replication: peer="{Voter server-4b659f42-3962-81fd-8af2-c8865b770f62 4b659f42-3962-81fd-8af2-c8865b770f62}" 2026-03-31T06:16:52.921Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:53.926Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:52.92122878 +0000 UTC m=+41.198836161 (2 nodes), 1.04µs from start of poll, 1.58252424s from cluster start. Timeout at 2026-03-31 06:16:53.92623334 +0000 UTC m=+42.203840681, 1.00500452s after stability 2026-03-31T06:16:53.938Z [INFO] server-b249238b-5ec9-11b5-2fc6-2f7a7a87a2da: aborting pipeline replication: peer="{Voter server-29f61a21-4097-8f91-9dd8-1aa115d62c1b 29f61a21-4097-8f91-9dd8-1aa115d62c1b}" 2026-03-31T06:16:53.938Z [INFO] server-b249238b-5ec9-11b5-2fc6-2f7a7a87a2da: aborting pipeline replication: peer="{Voter server-c69cb052-0d27-1ea9-113e-a81a09f0fa93 c69cb052-0d27-1ea9-113e-a81a09f0fa93}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false 2026-03-31T06:16:53.939Z [INFO] server-8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5 Address:8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5} {Suffrage:Voter ID:server-c75c71bc-c67b-8099-23cc-d9240b18590b Address:c75c71bc-c67b-8099-23cc-d9240b18590b} {Suffrage:Voter ID:server-c655b3df-3990-aff6-7632-79deda5c51c3 Address:c655b3df-3990-aff6-7632-79deda5c51c3}]" 2026-03-31T06:16:53.939Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5 Address:8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5} {Suffrage:Voter ID:server-c75c71bc-c67b-8099-23cc-d9240b18590b Address:c75c71bc-c67b-8099-23cc-d9240b18590b} {Suffrage:Voter ID:server-c655b3df-3990-aff6-7632-79deda5c51c3 Address:c655b3df-3990-aff6-7632-79deda5c51c3}]" 2026-03-31T06:16:53.939Z [INFO] server-8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5: entering follower state: follower="Node at 8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5 [Follower]" leader-address= leader-id= 2026-03-31T06:16:53.939Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: entering follower state: follower="Node at c75c71bc-c67b-8099-23cc-d9240b18590b [Follower]" leader-address= leader-id= 2026-03-31T06:16:53.939Z [INFO] server-c655b3df-3990-aff6-7632-79deda5c51c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5 Address:8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5} {Suffrage:Voter ID:server-c75c71bc-c67b-8099-23cc-d9240b18590b Address:c75c71bc-c67b-8099-23cc-d9240b18590b} {Suffrage:Voter ID:server-c655b3df-3990-aff6-7632-79deda5c51c3 Address:c655b3df-3990-aff6-7632-79deda5c51c3}]" 2026-03-31T06:16:53.939Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:53.939Z [INFO] server-c655b3df-3990-aff6-7632-79deda5c51c3: entering follower state: follower="Node at c655b3df-3990-aff6-7632-79deda5c51c3 [Follower]" leader-address= leader-id= 2026-03-31T06:16:54.777Z [WARN] server-c75c71bc-c67b-8099-23cc-d9240b18590b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:54.777Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: entering candidate state: node="Node at c75c71bc-c67b-8099-23cc-d9240b18590b [Candidate]" term=2 2026-03-31T06:16:54.777Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: election won: term=2 tally=2 2026-03-31T06:16:54.777Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: entering leader state: leader="Node at c75c71bc-c67b-8099-23cc-d9240b18590b [Leader]" 2026-03-31T06:16:54.778Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: added peer, starting replication: peer=server-8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5 2026-03-31T06:16:54.778Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: added peer, starting replication: peer=server-c655b3df-3990-aff6-7632-79deda5c51c3 2026-03-31T06:16:54.778Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: pipelining replication: peer="{Voter server-c655b3df-3990-aff6-7632-79deda5c51c3 c655b3df-3990-aff6-7632-79deda5c51c3}" 2026-03-31T06:16:54.778Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: pipelining replication: peer="{Voter server-8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5 8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5}" 2026-03-31T06:16:55.783Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:54.77811018 +0000 UTC m=+43.055717601 (1 nodes), 838.51344ms from start of poll, 839.00888ms from cluster start. Timeout at 2026-03-31 06:16:55.78313346 +0000 UTC m=+44.060740841, 1.00502324s after stability 2026-03-31T06:16:55.784Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: starting snapshot up to: index=12 2026-03-31T06:16:55.784Z [INFO] creating new snapshot: path=/tmp/raft1693134070/snapshots/2-12-1774937815784.tmp 2026-03-31T06:16:55.784Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: snapshot complete up to: index=12 2026-03-31T06:16:55.785Z [INFO] creating new snapshot: path=/tmp/raft1693134070/snapshots/2-23-1774937815785.tmp 2026-03-31T06:16:55.786Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: copied to local snapshot: bytes=71 2026-03-31T06:16:55.786Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: snapshot restore progress: id=2-23-1774937815785 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:55.786Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: restored user snapshot: index=23 2026-03-31T06:16:55.786Z [ERROR] server-c75c71bc-c67b-8099-23cc-d9240b18590b: failed to get log: index=23 error="log not found" 2026-03-31T06:16:55.786Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: aborting pipeline replication: peer="{Voter server-c655b3df-3990-aff6-7632-79deda5c51c3 c655b3df-3990-aff6-7632-79deda5c51c3}" 2026-03-31T06:16:55.786Z [ERROR] server-c75c71bc-c67b-8099-23cc-d9240b18590b: failed to get log: index=23 error="log not found" 2026-03-31T06:16:55.786Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: aborting pipeline replication: peer="{Voter server-8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5 8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5}" 2026-03-31T06:16:55.794Z [ERROR] server-c75c71bc-c67b-8099-23cc-d9240b18590b: failed to get log: index=23 error="log not found" 2026-03-31T06:16:55.794Z [INFO] creating new snapshot: path=/tmp/raft2178799551/snapshots/2-23-1774937815794.tmp 2026-03-31T06:16:55.795Z [INFO] server-8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:55.795Z [INFO] server-8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5: copied to local snapshot: bytes=71 2026-03-31T06:16:55.795Z [ERROR] server-c75c71bc-c67b-8099-23cc-d9240b18590b: failed to get log: index=23 error="log not found" 2026-03-31T06:16:55.795Z [INFO] server-8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5: snapshot restore progress: id=2-23-1774937815794 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:55.796Z [INFO] server-8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5: Installed remote snapshot 2026-03-31T06:16:55.796Z [WARN] server-8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5: skipping application of old log: index=22 2026-03-31T06:16:55.796Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: pipelining replication: peer="{Voter server-8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5 8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5}" 2026-03-31T06:16:55.796Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:55.796Z [INFO] creating new snapshot: path=/tmp/raft3677130111/snapshots/2-23-1774937815796.tmp 2026-03-31T06:16:55.796Z [INFO] server-c655b3df-3990-aff6-7632-79deda5c51c3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:55.796Z [INFO] server-c655b3df-3990-aff6-7632-79deda5c51c3: copied to local snapshot: bytes=71 2026-03-31T06:16:55.797Z [INFO] server-c655b3df-3990-aff6-7632-79deda5c51c3: snapshot restore progress: id=2-23-1774937815796 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:55.797Z [INFO] server-c655b3df-3990-aff6-7632-79deda5c51c3: Installed remote snapshot 2026-03-31T06:16:55.797Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: pipelining replication: peer="{Voter server-c655b3df-3990-aff6-7632-79deda5c51c3 c655b3df-3990-aff6-7632-79deda5c51c3}" 2026-03-31T06:16:56.802Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:55.79619798 +0000 UTC m=+44.073805361 (2 nodes), 1.08µs from start of poll, 1.85709664s from cluster start. Timeout at 2026-03-31 06:16:56.80121314 +0000 UTC m=+45.078820521, 1.00501516s after stability 2026-03-31T06:16:56.814Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: aborting pipeline replication: peer="{Voter server-c655b3df-3990-aff6-7632-79deda5c51c3 c655b3df-3990-aff6-7632-79deda5c51c3}" 2026-03-31T06:16:56.814Z [INFO] server-c75c71bc-c67b-8099-23cc-d9240b18590b: aborting pipeline replication: peer="{Voter server-8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5 8eb8fab8-b3f5-a50a-2cfb-0dc5b7ce71e5}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false 2026-03-31T06:16:56.815Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d4cbca73-380a-507a-3365-ae83b93cdc11 Address:d4cbca73-380a-507a-3365-ae83b93cdc11} {Suffrage:Voter ID:server-b940b8ae-0576-b95d-b87a-7d37ce7cc849 Address:b940b8ae-0576-b95d-b87a-7d37ce7cc849} {Suffrage:Voter ID:server-acbd1506-7795-c321-e1de-11abd27ef796 Address:acbd1506-7795-c321-e1de-11abd27ef796}]" 2026-03-31T06:16:56.816Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: entering follower state: follower="Node at d4cbca73-380a-507a-3365-ae83b93cdc11 [Follower]" leader-address= leader-id= 2026-03-31T06:16:56.816Z [INFO] server-b940b8ae-0576-b95d-b87a-7d37ce7cc849: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d4cbca73-380a-507a-3365-ae83b93cdc11 Address:d4cbca73-380a-507a-3365-ae83b93cdc11} {Suffrage:Voter ID:server-b940b8ae-0576-b95d-b87a-7d37ce7cc849 Address:b940b8ae-0576-b95d-b87a-7d37ce7cc849} {Suffrage:Voter ID:server-acbd1506-7795-c321-e1de-11abd27ef796 Address:acbd1506-7795-c321-e1de-11abd27ef796}]" 2026-03-31T06:16:56.816Z [INFO] server-b940b8ae-0576-b95d-b87a-7d37ce7cc849: entering follower state: follower="Node at b940b8ae-0576-b95d-b87a-7d37ce7cc849 [Follower]" leader-address= leader-id= 2026-03-31T06:16:56.816Z [INFO] server-acbd1506-7795-c321-e1de-11abd27ef796: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d4cbca73-380a-507a-3365-ae83b93cdc11 Address:d4cbca73-380a-507a-3365-ae83b93cdc11} {Suffrage:Voter ID:server-b940b8ae-0576-b95d-b87a-7d37ce7cc849 Address:b940b8ae-0576-b95d-b87a-7d37ce7cc849} {Suffrage:Voter ID:server-acbd1506-7795-c321-e1de-11abd27ef796 Address:acbd1506-7795-c321-e1de-11abd27ef796}]" 2026-03-31T06:16:56.816Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:56.816Z [INFO] server-acbd1506-7795-c321-e1de-11abd27ef796: entering follower state: follower="Node at acbd1506-7795-c321-e1de-11abd27ef796 [Follower]" leader-address= leader-id= 2026-03-31T06:16:57.386Z [WARN] server-d4cbca73-380a-507a-3365-ae83b93cdc11: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:57.386Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: entering candidate state: node="Node at d4cbca73-380a-507a-3365-ae83b93cdc11 [Candidate]" term=2 2026-03-31T06:16:57.386Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: election won: term=2 tally=2 2026-03-31T06:16:57.386Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: entering leader state: leader="Node at d4cbca73-380a-507a-3365-ae83b93cdc11 [Leader]" 2026-03-31T06:16:57.386Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: added peer, starting replication: peer=server-b940b8ae-0576-b95d-b87a-7d37ce7cc849 2026-03-31T06:16:57.386Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: added peer, starting replication: peer=server-acbd1506-7795-c321-e1de-11abd27ef796 2026-03-31T06:16:57.386Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: pipelining replication: peer="{Voter server-acbd1506-7795-c321-e1de-11abd27ef796 acbd1506-7795-c321-e1de-11abd27ef796}" 2026-03-31T06:16:57.386Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: pipelining replication: peer="{Voter server-b940b8ae-0576-b95d-b87a-7d37ce7cc849 b940b8ae-0576-b95d-b87a-7d37ce7cc849}" 2026-03-31T06:16:58.392Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:57.3866745 +0000 UTC m=+45.664281881 (1 nodes), 570.25096ms from start of poll, 570.99924ms from cluster start. Timeout at 2026-03-31 06:16:58.39167822 +0000 UTC m=+46.669285601, 1.00500372s after stability 2026-03-31T06:16:58.392Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: starting snapshot up to: index=12 2026-03-31T06:16:58.393Z [INFO] creating new snapshot: path=/tmp/raft3213364505/snapshots/2-12-1774937818392.tmp 2026-03-31T06:16:58.393Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: snapshot complete up to: index=12 2026-03-31T06:16:58.394Z [INFO] creating new snapshot: path=/tmp/raft3213364505/snapshots/2-23-1774937818394.tmp 2026-03-31T06:16:58.394Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: copied to local snapshot: bytes=71 2026-03-31T06:16:58.395Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: snapshot restore progress: id=2-23-1774937818394 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:58.395Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: removing all old logs from log store 2026-03-31T06:16:58.395Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: compacting logs: from=1 to=22 2026-03-31T06:16:58.395Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: restored user snapshot: index=23 2026-03-31T06:16:58.395Z [ERROR] server-d4cbca73-380a-507a-3365-ae83b93cdc11: failed to get log: index=22 error="log not found" 2026-03-31T06:16:58.395Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: aborting pipeline replication: peer="{Voter server-acbd1506-7795-c321-e1de-11abd27ef796 acbd1506-7795-c321-e1de-11abd27ef796}" 2026-03-31T06:16:58.395Z [ERROR] server-d4cbca73-380a-507a-3365-ae83b93cdc11: failed to get log: index=22 error="log not found" 2026-03-31T06:16:58.395Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: aborting pipeline replication: peer="{Voter server-b940b8ae-0576-b95d-b87a-7d37ce7cc849 b940b8ae-0576-b95d-b87a-7d37ce7cc849}" 2026-03-31T06:16:58.402Z [ERROR] server-d4cbca73-380a-507a-3365-ae83b93cdc11: failed to get log: index=22 error="log not found" 2026-03-31T06:16:58.402Z [INFO] creating new snapshot: path=/tmp/raft393117621/snapshots/2-23-1774937818402.tmp 2026-03-31T06:16:58.403Z [INFO] server-acbd1506-7795-c321-e1de-11abd27ef796: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:58.403Z [INFO] server-acbd1506-7795-c321-e1de-11abd27ef796: copied to local snapshot: bytes=71 2026-03-31T06:16:58.403Z [INFO] server-acbd1506-7795-c321-e1de-11abd27ef796: snapshot restore progress: id=2-23-1774937818402 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:58.403Z [ERROR] server-d4cbca73-380a-507a-3365-ae83b93cdc11: failed to get log: index=22 error="log not found" 2026-03-31T06:16:58.403Z [INFO] server-acbd1506-7795-c321-e1de-11abd27ef796: removing all old logs from log store 2026-03-31T06:16:58.403Z [INFO] server-acbd1506-7795-c321-e1de-11abd27ef796: compacting logs: from=1 to=22 2026-03-31T06:16:58.403Z [INFO] server-acbd1506-7795-c321-e1de-11abd27ef796: Installed remote snapshot 2026-03-31T06:16:58.403Z [WARN] server-acbd1506-7795-c321-e1de-11abd27ef796: skipping application of old log: index=22 2026-03-31T06:16:58.403Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: pipelining replication: peer="{Voter server-acbd1506-7795-c321-e1de-11abd27ef796 acbd1506-7795-c321-e1de-11abd27ef796}" 2026-03-31T06:16:58.403Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:16:58.403Z [INFO] creating new snapshot: path=/tmp/raft4033972450/snapshots/2-23-1774937818403.tmp 2026-03-31T06:16:58.404Z [INFO] server-b940b8ae-0576-b95d-b87a-7d37ce7cc849: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:58.404Z [INFO] server-b940b8ae-0576-b95d-b87a-7d37ce7cc849: copied to local snapshot: bytes=71 2026-03-31T06:16:58.404Z [INFO] server-b940b8ae-0576-b95d-b87a-7d37ce7cc849: snapshot restore progress: id=2-23-1774937818403 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:16:58.404Z [INFO] server-b940b8ae-0576-b95d-b87a-7d37ce7cc849: removing all old logs from log store 2026-03-31T06:16:58.404Z [INFO] server-b940b8ae-0576-b95d-b87a-7d37ce7cc849: compacting logs: from=1 to=22 2026-03-31T06:16:58.404Z [INFO] server-b940b8ae-0576-b95d-b87a-7d37ce7cc849: Installed remote snapshot 2026-03-31T06:16:58.404Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: pipelining replication: peer="{Voter server-b940b8ae-0576-b95d-b87a-7d37ce7cc849 b940b8ae-0576-b95d-b87a-7d37ce7cc849}" 2026-03-31T06:16:59.409Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:16:58.40377446 +0000 UTC m=+46.681381841 (2 nodes), 960ns from start of poll, 1.5880992s from cluster start. Timeout at 2026-03-31 06:16:59.40879114 +0000 UTC m=+47.686398521, 1.00501668s after stability 2026-03-31T06:16:59.421Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: aborting pipeline replication: peer="{Voter server-acbd1506-7795-c321-e1de-11abd27ef796 acbd1506-7795-c321-e1de-11abd27ef796}" 2026-03-31T06:16:59.421Z [INFO] server-d4cbca73-380a-507a-3365-ae83b93cdc11: aborting pipeline replication: peer="{Voter server-b940b8ae-0576-b95d-b87a-7d37ce7cc849 b940b8ae-0576-b95d-b87a-7d37ce7cc849}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true 2026-03-31T06:16:59.422Z [INFO] server-d120d7d0-ae87-696c-a3d3-a07c0c87f0f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d120d7d0-ae87-696c-a3d3-a07c0c87f0f9 Address:d120d7d0-ae87-696c-a3d3-a07c0c87f0f9} {Suffrage:Voter ID:server-4d59777c-b92d-9cf3-d7d1-43fc8ab92bd8 Address:4d59777c-b92d-9cf3-d7d1-43fc8ab92bd8} {Suffrage:Voter ID:server-b864b142-50a9-d4b8-a398-cfefc5fc2e3b Address:b864b142-50a9-d4b8-a398-cfefc5fc2e3b}]" 2026-03-31T06:16:59.422Z [INFO] server-4d59777c-b92d-9cf3-d7d1-43fc8ab92bd8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d120d7d0-ae87-696c-a3d3-a07c0c87f0f9 Address:d120d7d0-ae87-696c-a3d3-a07c0c87f0f9} {Suffrage:Voter ID:server-4d59777c-b92d-9cf3-d7d1-43fc8ab92bd8 Address:4d59777c-b92d-9cf3-d7d1-43fc8ab92bd8} {Suffrage:Voter ID:server-b864b142-50a9-d4b8-a398-cfefc5fc2e3b Address:b864b142-50a9-d4b8-a398-cfefc5fc2e3b}]" 2026-03-31T06:16:59.422Z [INFO] server-d120d7d0-ae87-696c-a3d3-a07c0c87f0f9: entering follower state: follower="Node at d120d7d0-ae87-696c-a3d3-a07c0c87f0f9 [Follower]" leader-address= leader-id= 2026-03-31T06:16:59.422Z [INFO] server-4d59777c-b92d-9cf3-d7d1-43fc8ab92bd8: entering follower state: follower="Node at 4d59777c-b92d-9cf3-d7d1-43fc8ab92bd8 [Follower]" leader-address= leader-id= 2026-03-31T06:16:59.422Z [INFO] server-b864b142-50a9-d4b8-a398-cfefc5fc2e3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d120d7d0-ae87-696c-a3d3-a07c0c87f0f9 Address:d120d7d0-ae87-696c-a3d3-a07c0c87f0f9} {Suffrage:Voter ID:server-4d59777c-b92d-9cf3-d7d1-43fc8ab92bd8 Address:4d59777c-b92d-9cf3-d7d1-43fc8ab92bd8} {Suffrage:Voter ID:server-b864b142-50a9-d4b8-a398-cfefc5fc2e3b Address:b864b142-50a9-d4b8-a398-cfefc5fc2e3b}]" 2026-03-31T06:16:59.422Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:16:59.422Z [INFO] server-b864b142-50a9-d4b8-a398-cfefc5fc2e3b: entering follower state: follower="Node at b864b142-50a9-d4b8-a398-cfefc5fc2e3b [Follower]" leader-address= leader-id= 2026-03-31T06:16:59.980Z [WARN] server-b864b142-50a9-d4b8-a398-cfefc5fc2e3b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:16:59.980Z [INFO] server-b864b142-50a9-d4b8-a398-cfefc5fc2e3b: entering candidate state: node="Node at b864b142-50a9-d4b8-a398-cfefc5fc2e3b [Candidate]" term=2 2026-03-31T06:16:59.981Z [INFO] server-b864b142-50a9-d4b8-a398-cfefc5fc2e3b: election won: term=2 tally=2 2026-03-31T06:16:59.981Z [INFO] server-b864b142-50a9-d4b8-a398-cfefc5fc2e3b: entering leader state: leader="Node at b864b142-50a9-d4b8-a398-cfefc5fc2e3b [Leader]" 2026-03-31T06:16:59.981Z [INFO] server-b864b142-50a9-d4b8-a398-cfefc5fc2e3b: added peer, starting replication: peer=server-d120d7d0-ae87-696c-a3d3-a07c0c87f0f9 2026-03-31T06:16:59.981Z [INFO] server-b864b142-50a9-d4b8-a398-cfefc5fc2e3b: added peer, starting replication: peer=server-4d59777c-b92d-9cf3-d7d1-43fc8ab92bd8 2026-03-31T06:16:59.981Z [INFO] server-b864b142-50a9-d4b8-a398-cfefc5fc2e3b: pipelining replication: peer="{Voter server-d120d7d0-ae87-696c-a3d3-a07c0c87f0f9 d120d7d0-ae87-696c-a3d3-a07c0c87f0f9}" 2026-03-31T06:16:59.981Z [INFO] server-b864b142-50a9-d4b8-a398-cfefc5fc2e3b: pipelining replication: peer="{Voter server-4d59777c-b92d-9cf3-d7d1-43fc8ab92bd8 4d59777c-b92d-9cf3-d7d1-43fc8ab92bd8}" 2026-03-31T06:17:00.986Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:16:59.98118638 +0000 UTC m=+48.258793761 (1 nodes), 558.5954ms from start of poll, 559.00568ms from cluster start. Timeout at 2026-03-31 06:17:00.98618818 +0000 UTC m=+49.263795521, 1.00500176s after stability 2026-03-31T06:17:00.987Z [INFO] server-b864b142-50a9-d4b8-a398-cfefc5fc2e3b: starting snapshot up to: index=12 2026-03-31T06:17:00.987Z [INFO] creating new snapshot: path=/tmp/raft510648591/snapshots/2-12-1774937820987.tmp 2026-03-31T06:17:00.988Z [INFO] server-b864b142-50a9-d4b8-a398-cfefc5fc2e3b: snapshot complete up to: index=12 2026-03-31T06:17:00.989Z [INFO] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-266bf53f-bab9-bd0b-2137-1dbe55e1d964 Address:266bf53f-bab9-bd0b-2137-1dbe55e1d964} {Suffrage:Voter ID:server-32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c Address:32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c} {Suffrage:Voter ID:server-2c2a2976-cfce-78ce-ffb3-d5a52e7d2882 Address:2c2a2976-cfce-78ce-ffb3-d5a52e7d2882}]" 2026-03-31T06:17:00.989Z [INFO] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: entering follower state: follower="Node at 266bf53f-bab9-bd0b-2137-1dbe55e1d964 [Follower]" leader-address= leader-id= 2026-03-31T06:17:00.989Z [INFO] server-32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-266bf53f-bab9-bd0b-2137-1dbe55e1d964 Address:266bf53f-bab9-bd0b-2137-1dbe55e1d964} {Suffrage:Voter ID:server-32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c Address:32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c} {Suffrage:Voter ID:server-2c2a2976-cfce-78ce-ffb3-d5a52e7d2882 Address:2c2a2976-cfce-78ce-ffb3-d5a52e7d2882}]" 2026-03-31T06:17:00.990Z [INFO] server-32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c: entering follower state: follower="Node at 32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c [Follower]" leader-address= leader-id= 2026-03-31T06:17:00.990Z [INFO] server-2c2a2976-cfce-78ce-ffb3-d5a52e7d2882: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-266bf53f-bab9-bd0b-2137-1dbe55e1d964 Address:266bf53f-bab9-bd0b-2137-1dbe55e1d964} {Suffrage:Voter ID:server-32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c Address:32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c} {Suffrage:Voter ID:server-2c2a2976-cfce-78ce-ffb3-d5a52e7d2882 Address:2c2a2976-cfce-78ce-ffb3-d5a52e7d2882}]" 2026-03-31T06:17:00.990Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:00.990Z [INFO] server-2c2a2976-cfce-78ce-ffb3-d5a52e7d2882: entering follower state: follower="Node at 2c2a2976-cfce-78ce-ffb3-d5a52e7d2882 [Follower]" leader-address= leader-id= 2026-03-31T06:17:01.691Z [WARN] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:01.691Z [INFO] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: entering candidate state: node="Node at 266bf53f-bab9-bd0b-2137-1dbe55e1d964 [Candidate]" term=2 2026-03-31T06:17:01.691Z [INFO] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: election won: term=2 tally=2 2026-03-31T06:17:01.691Z [INFO] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: entering leader state: leader="Node at 266bf53f-bab9-bd0b-2137-1dbe55e1d964 [Leader]" 2026-03-31T06:17:01.691Z [INFO] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: added peer, starting replication: peer=server-32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c 2026-03-31T06:17:01.691Z [INFO] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: added peer, starting replication: peer=server-2c2a2976-cfce-78ce-ffb3-d5a52e7d2882 2026-03-31T06:17:01.691Z [INFO] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: pipelining replication: peer="{Voter server-2c2a2976-cfce-78ce-ffb3-d5a52e7d2882 2c2a2976-cfce-78ce-ffb3-d5a52e7d2882}" 2026-03-31T06:17:01.692Z [INFO] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: pipelining replication: peer="{Voter server-32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c 32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c}" 2026-03-31T06:17:02.697Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:01.69193606 +0000 UTC m=+49.969543441 (1 nodes), 701.6848ms from start of poll, 702.36716ms from cluster start. Timeout at 2026-03-31 06:17:02.69694422 +0000 UTC m=+50.974551601, 1.00500816s after stability 2026-03-31T06:17:02.697Z [INFO] creating new snapshot: path=/tmp/raft1432414814/snapshots/2-15-1774937822697.tmp 2026-03-31T06:17:02.697Z [INFO] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: copied to local snapshot: bytes=71 2026-03-31T06:17:02.698Z [INFO] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: snapshot restore progress: id=2-15-1774937822697 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:02.698Z [INFO] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: restored user snapshot: index=15 2026-03-31T06:17:02.698Z [ERROR] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: failed to get log: index=3 error="log not found" 2026-03-31T06:17:02.698Z [INFO] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: aborting pipeline replication: peer="{Voter server-2c2a2976-cfce-78ce-ffb3-d5a52e7d2882 2c2a2976-cfce-78ce-ffb3-d5a52e7d2882}" 2026-03-31T06:17:02.698Z [ERROR] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: failed to get log: index=3 error="log not found" 2026-03-31T06:17:02.698Z [INFO] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: aborting pipeline replication: peer="{Voter server-32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c 32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c}" 2026-03-31T06:17:02.707Z [ERROR] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: failed to get log: index=3 error="log not found" 2026-03-31T06:17:02.708Z [INFO] creating new snapshot: path=/tmp/raft2332189455/snapshots/2-15-1774937822708.tmp 2026-03-31T06:17:02.708Z [ERROR] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: failed to get log: index=3 error="log not found" 2026-03-31T06:17:02.708Z [INFO] server-32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:02.708Z [INFO] creating new snapshot: path=/tmp/raft4221624487/snapshots/2-15-1774937822708.tmp 2026-03-31T06:17:02.708Z [INFO] server-32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c: copied to local snapshot: bytes=71 2026-03-31T06:17:02.708Z [INFO] server-2c2a2976-cfce-78ce-ffb3-d5a52e7d2882: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:02.709Z [INFO] server-32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c: snapshot restore progress: id=2-15-1774937822708 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:02.709Z [INFO] server-32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c: Installed remote snapshot 2026-03-31T06:17:02.709Z [INFO] server-2c2a2976-cfce-78ce-ffb3-d5a52e7d2882: copied to local snapshot: bytes=71 2026-03-31T06:17:02.709Z [INFO] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: pipelining replication: peer="{Voter server-32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c 32ca7425-0dbf-7b0b-4a45-09c2c64f6d3c}" 2026-03-31T06:17:02.709Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:02.709Z [INFO] server-2c2a2976-cfce-78ce-ffb3-d5a52e7d2882: snapshot restore progress: id=2-15-1774937822708 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:02.709Z [INFO] server-2c2a2976-cfce-78ce-ffb3-d5a52e7d2882: Installed remote snapshot 2026-03-31T06:17:02.709Z [INFO] server-266bf53f-bab9-bd0b-2137-1dbe55e1d964: pipelining replication: peer="{Voter server-2c2a2976-cfce-78ce-ffb3-d5a52e7d2882 2c2a2976-cfce-78ce-ffb3-d5a52e7d2882}" 2026-03-31T06:17:03.715Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:02.70940002 +0000 UTC m=+50.987007401 (2 nodes), 920ns from start of poll, 1.71983112s from cluster start. Timeout at 2026-03-31 06:17:03.71440738 +0000 UTC m=+51.992014761, 1.00500736s after stability 2026-03-31T06:17:03.727Z [INFO] server-b864b142-50a9-d4b8-a398-cfefc5fc2e3b: aborting pipeline replication: peer="{Voter server-4d59777c-b92d-9cf3-d7d1-43fc8ab92bd8 4d59777c-b92d-9cf3-d7d1-43fc8ab92bd8}" 2026-03-31T06:17:03.727Z [INFO] server-b864b142-50a9-d4b8-a398-cfefc5fc2e3b: aborting pipeline replication: peer="{Voter server-d120d7d0-ae87-696c-a3d3-a07c0c87f0f9 d120d7d0-ae87-696c-a3d3-a07c0c87f0f9}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true 2026-03-31T06:17:03.728Z [INFO] server-3678e8fd-f7b7-a5ef-b366-356c085ad0c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3678e8fd-f7b7-a5ef-b366-356c085ad0c1 Address:3678e8fd-f7b7-a5ef-b366-356c085ad0c1} {Suffrage:Voter ID:server-0479d444-945e-d40a-8c10-c1ce56887f57 Address:0479d444-945e-d40a-8c10-c1ce56887f57} {Suffrage:Voter ID:server-0d44823f-aba0-e4f1-9be1-886d4b964c71 Address:0d44823f-aba0-e4f1-9be1-886d4b964c71}]" 2026-03-31T06:17:03.728Z [INFO] server-0479d444-945e-d40a-8c10-c1ce56887f57: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3678e8fd-f7b7-a5ef-b366-356c085ad0c1 Address:3678e8fd-f7b7-a5ef-b366-356c085ad0c1} {Suffrage:Voter ID:server-0479d444-945e-d40a-8c10-c1ce56887f57 Address:0479d444-945e-d40a-8c10-c1ce56887f57} {Suffrage:Voter ID:server-0d44823f-aba0-e4f1-9be1-886d4b964c71 Address:0d44823f-aba0-e4f1-9be1-886d4b964c71}]" 2026-03-31T06:17:03.728Z [INFO] server-3678e8fd-f7b7-a5ef-b366-356c085ad0c1: entering follower state: follower="Node at 3678e8fd-f7b7-a5ef-b366-356c085ad0c1 [Follower]" leader-address= leader-id= 2026-03-31T06:17:03.728Z [INFO] server-0479d444-945e-d40a-8c10-c1ce56887f57: entering follower state: follower="Node at 0479d444-945e-d40a-8c10-c1ce56887f57 [Follower]" leader-address= leader-id= 2026-03-31T06:17:03.728Z [INFO] server-0d44823f-aba0-e4f1-9be1-886d4b964c71: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3678e8fd-f7b7-a5ef-b366-356c085ad0c1 Address:3678e8fd-f7b7-a5ef-b366-356c085ad0c1} {Suffrage:Voter ID:server-0479d444-945e-d40a-8c10-c1ce56887f57 Address:0479d444-945e-d40a-8c10-c1ce56887f57} {Suffrage:Voter ID:server-0d44823f-aba0-e4f1-9be1-886d4b964c71 Address:0d44823f-aba0-e4f1-9be1-886d4b964c71}]" 2026-03-31T06:17:03.728Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:03.728Z [INFO] server-0d44823f-aba0-e4f1-9be1-886d4b964c71: entering follower state: follower="Node at 0d44823f-aba0-e4f1-9be1-886d4b964c71 [Follower]" leader-address= leader-id= 2026-03-31T06:17:04.509Z [WARN] server-0479d444-945e-d40a-8c10-c1ce56887f57: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:04.509Z [INFO] server-0479d444-945e-d40a-8c10-c1ce56887f57: entering candidate state: node="Node at 0479d444-945e-d40a-8c10-c1ce56887f57 [Candidate]" term=2 2026-03-31T06:17:04.509Z [INFO] server-0479d444-945e-d40a-8c10-c1ce56887f57: election won: term=2 tally=2 2026-03-31T06:17:04.509Z [INFO] server-0479d444-945e-d40a-8c10-c1ce56887f57: entering leader state: leader="Node at 0479d444-945e-d40a-8c10-c1ce56887f57 [Leader]" 2026-03-31T06:17:04.509Z [INFO] server-0479d444-945e-d40a-8c10-c1ce56887f57: added peer, starting replication: peer=server-3678e8fd-f7b7-a5ef-b366-356c085ad0c1 2026-03-31T06:17:04.509Z [INFO] server-0479d444-945e-d40a-8c10-c1ce56887f57: added peer, starting replication: peer=server-0d44823f-aba0-e4f1-9be1-886d4b964c71 2026-03-31T06:17:04.509Z [INFO] server-0479d444-945e-d40a-8c10-c1ce56887f57: pipelining replication: peer="{Voter server-0d44823f-aba0-e4f1-9be1-886d4b964c71 0d44823f-aba0-e4f1-9be1-886d4b964c71}" 2026-03-31T06:17:04.509Z [INFO] server-0479d444-945e-d40a-8c10-c1ce56887f57: pipelining replication: peer="{Voter server-3678e8fd-f7b7-a5ef-b366-356c085ad0c1 3678e8fd-f7b7-a5ef-b366-356c085ad0c1}" 2026-03-31T06:17:05.514Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:04.50960038 +0000 UTC m=+52.787207761 (1 nodes), 780.77304ms from start of poll, 781.22464ms from cluster start. Timeout at 2026-03-31 06:17:05.51460294 +0000 UTC m=+53.792210361, 1.0050026s after stability 2026-03-31T06:17:05.515Z [INFO] server-0479d444-945e-d40a-8c10-c1ce56887f57: starting snapshot up to: index=12 2026-03-31T06:17:05.515Z [INFO] creating new snapshot: path=/tmp/raft4132315654/snapshots/2-12-1774937825515.tmp 2026-03-31T06:17:05.516Z [INFO] server-0479d444-945e-d40a-8c10-c1ce56887f57: snapshot complete up to: index=12 2026-03-31T06:17:05.517Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22940dab-7423-c433-e58c-c83986124b3e Address:22940dab-7423-c433-e58c-c83986124b3e} {Suffrage:Voter ID:server-b955cc52-c412-9694-f487-0708caadc00f Address:b955cc52-c412-9694-f487-0708caadc00f} {Suffrage:Voter ID:server-24043e19-27b6-b585-1d4d-86d38bb889f3 Address:24043e19-27b6-b585-1d4d-86d38bb889f3}]" 2026-03-31T06:17:05.517Z [INFO] server-b955cc52-c412-9694-f487-0708caadc00f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22940dab-7423-c433-e58c-c83986124b3e Address:22940dab-7423-c433-e58c-c83986124b3e} {Suffrage:Voter ID:server-b955cc52-c412-9694-f487-0708caadc00f Address:b955cc52-c412-9694-f487-0708caadc00f} {Suffrage:Voter ID:server-24043e19-27b6-b585-1d4d-86d38bb889f3 Address:24043e19-27b6-b585-1d4d-86d38bb889f3}]" 2026-03-31T06:17:05.517Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: entering follower state: follower="Node at 22940dab-7423-c433-e58c-c83986124b3e [Follower]" leader-address= leader-id= 2026-03-31T06:17:05.517Z [INFO] server-b955cc52-c412-9694-f487-0708caadc00f: entering follower state: follower="Node at b955cc52-c412-9694-f487-0708caadc00f [Follower]" leader-address= leader-id= 2026-03-31T06:17:05.517Z [INFO] server-24043e19-27b6-b585-1d4d-86d38bb889f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22940dab-7423-c433-e58c-c83986124b3e Address:22940dab-7423-c433-e58c-c83986124b3e} {Suffrage:Voter ID:server-b955cc52-c412-9694-f487-0708caadc00f Address:b955cc52-c412-9694-f487-0708caadc00f} {Suffrage:Voter ID:server-24043e19-27b6-b585-1d4d-86d38bb889f3 Address:24043e19-27b6-b585-1d4d-86d38bb889f3}]" 2026-03-31T06:17:05.517Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:05.517Z [INFO] server-24043e19-27b6-b585-1d4d-86d38bb889f3: entering follower state: follower="Node at 24043e19-27b6-b585-1d4d-86d38bb889f3 [Follower]" leader-address= leader-id= 2026-03-31T06:17:06.019Z [WARN] server-22940dab-7423-c433-e58c-c83986124b3e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:06.019Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: entering candidate state: node="Node at 22940dab-7423-c433-e58c-c83986124b3e [Candidate]" term=2 2026-03-31T06:17:06.019Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: election won: term=2 tally=2 2026-03-31T06:17:06.019Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: entering leader state: leader="Node at 22940dab-7423-c433-e58c-c83986124b3e [Leader]" 2026-03-31T06:17:06.019Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: added peer, starting replication: peer=server-b955cc52-c412-9694-f487-0708caadc00f 2026-03-31T06:17:06.019Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: added peer, starting replication: peer=server-24043e19-27b6-b585-1d4d-86d38bb889f3 2026-03-31T06:17:06.019Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: pipelining replication: peer="{Voter server-24043e19-27b6-b585-1d4d-86d38bb889f3 24043e19-27b6-b585-1d4d-86d38bb889f3}" 2026-03-31T06:17:06.020Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: pipelining replication: peer="{Voter server-b955cc52-c412-9694-f487-0708caadc00f b955cc52-c412-9694-f487-0708caadc00f}" 2026-03-31T06:17:07.025Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:06.0195635 +0000 UTC m=+54.297170881 (1 nodes), 502.08776ms from start of poll, 502.56564ms from cluster start. Timeout at 2026-03-31 06:17:07.0245677 +0000 UTC m=+55.302175121, 1.00500424s after stability 2026-03-31T06:17:07.025Z [INFO] creating new snapshot: path=/tmp/raft1803540490/snapshots/2-15-1774937827025.tmp 2026-03-31T06:17:07.026Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: copied to local snapshot: bytes=71 2026-03-31T06:17:07.027Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: snapshot restore progress: id=2-15-1774937827025 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:07.027Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: removing all old logs from log store 2026-03-31T06:17:07.027Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: compacting logs: from=1 to=2 2026-03-31T06:17:07.027Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: restored user snapshot: index=15 2026-03-31T06:17:07.027Z [ERROR] server-22940dab-7423-c433-e58c-c83986124b3e: failed to get log: index=2 error="log not found" 2026-03-31T06:17:07.027Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: aborting pipeline replication: peer="{Voter server-b955cc52-c412-9694-f487-0708caadc00f b955cc52-c412-9694-f487-0708caadc00f}" 2026-03-31T06:17:07.027Z [ERROR] server-22940dab-7423-c433-e58c-c83986124b3e: failed to get log: index=2 error="log not found" 2026-03-31T06:17:07.027Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: aborting pipeline replication: peer="{Voter server-24043e19-27b6-b585-1d4d-86d38bb889f3 24043e19-27b6-b585-1d4d-86d38bb889f3}" 2026-03-31T06:17:07.034Z [ERROR] server-22940dab-7423-c433-e58c-c83986124b3e: failed to get log: index=2 error="log not found" 2026-03-31T06:17:07.035Z [ERROR] server-22940dab-7423-c433-e58c-c83986124b3e: failed to get log: index=2 error="log not found" 2026-03-31T06:17:07.035Z [INFO] creating new snapshot: path=/tmp/raft864742639/snapshots/2-15-1774937827035.tmp 2026-03-31T06:17:07.035Z [INFO] creating new snapshot: path=/tmp/raft1848266708/snapshots/2-15-1774937827035.tmp 2026-03-31T06:17:07.035Z [INFO] server-24043e19-27b6-b585-1d4d-86d38bb889f3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:07.035Z [INFO] server-b955cc52-c412-9694-f487-0708caadc00f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:07.035Z [INFO] server-24043e19-27b6-b585-1d4d-86d38bb889f3: copied to local snapshot: bytes=71 2026-03-31T06:17:07.036Z [INFO] server-b955cc52-c412-9694-f487-0708caadc00f: copied to local snapshot: bytes=71 2026-03-31T06:17:07.036Z [INFO] server-24043e19-27b6-b585-1d4d-86d38bb889f3: snapshot restore progress: id=2-15-1774937827035 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:07.036Z [INFO] server-24043e19-27b6-b585-1d4d-86d38bb889f3: removing all old logs from log store 2026-03-31T06:17:07.036Z [INFO] server-24043e19-27b6-b585-1d4d-86d38bb889f3: compacting logs: from=1 to=2 2026-03-31T06:17:07.036Z [INFO] server-24043e19-27b6-b585-1d4d-86d38bb889f3: Installed remote snapshot 2026-03-31T06:17:07.036Z [INFO] server-b955cc52-c412-9694-f487-0708caadc00f: snapshot restore progress: id=2-15-1774937827035 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:07.036Z [INFO] server-b955cc52-c412-9694-f487-0708caadc00f: removing all old logs from log store 2026-03-31T06:17:07.036Z [INFO] server-b955cc52-c412-9694-f487-0708caadc00f: compacting logs: from=1 to=2 2026-03-31T06:17:07.036Z [INFO] server-b955cc52-c412-9694-f487-0708caadc00f: Installed remote snapshot 2026-03-31T06:17:07.036Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: pipelining replication: peer="{Voter server-24043e19-27b6-b585-1d4d-86d38bb889f3 24043e19-27b6-b585-1d4d-86d38bb889f3}" 2026-03-31T06:17:07.036Z [INFO] server-22940dab-7423-c433-e58c-c83986124b3e: pipelining replication: peer="{Voter server-b955cc52-c412-9694-f487-0708caadc00f b955cc52-c412-9694-f487-0708caadc00f}" 2026-03-31T06:17:07.036Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:08.041Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:07.03656026 +0000 UTC m=+55.314167641 (2 nodes), 1.12µs from start of poll, 1.5195624s from cluster start. Timeout at 2026-03-31 06:17:08.04158302 +0000 UTC m=+56.319190401, 1.00502276s after stability 2026-03-31T06:17:08.054Z [INFO] server-0479d444-945e-d40a-8c10-c1ce56887f57: aborting pipeline replication: peer="{Voter server-0d44823f-aba0-e4f1-9be1-886d4b964c71 0d44823f-aba0-e4f1-9be1-886d4b964c71}" 2026-03-31T06:17:08.054Z [INFO] server-0479d444-945e-d40a-8c10-c1ce56887f57: aborting pipeline replication: peer="{Voter server-3678e8fd-f7b7-a5ef-b366-356c085ad0c1 3678e8fd-f7b7-a5ef-b366-356c085ad0c1}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false 2026-03-31T06:17:08.055Z [INFO] server-a4743941-6289-1233-5944-02dcb5626528: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4743941-6289-1233-5944-02dcb5626528 Address:a4743941-6289-1233-5944-02dcb5626528} {Suffrage:Voter ID:server-1beb164c-d829-99d5-a391-4ba9596a5688 Address:1beb164c-d829-99d5-a391-4ba9596a5688} {Suffrage:Voter ID:server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8 Address:b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8}]" 2026-03-31T06:17:08.055Z [INFO] server-1beb164c-d829-99d5-a391-4ba9596a5688: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4743941-6289-1233-5944-02dcb5626528 Address:a4743941-6289-1233-5944-02dcb5626528} {Suffrage:Voter ID:server-1beb164c-d829-99d5-a391-4ba9596a5688 Address:1beb164c-d829-99d5-a391-4ba9596a5688} {Suffrage:Voter ID:server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8 Address:b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8}]" 2026-03-31T06:17:08.055Z [INFO] server-a4743941-6289-1233-5944-02dcb5626528: entering follower state: follower="Node at a4743941-6289-1233-5944-02dcb5626528 [Follower]" leader-address= leader-id= 2026-03-31T06:17:08.056Z [INFO] server-1beb164c-d829-99d5-a391-4ba9596a5688: entering follower state: follower="Node at 1beb164c-d829-99d5-a391-4ba9596a5688 [Follower]" leader-address= leader-id= 2026-03-31T06:17:08.056Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4743941-6289-1233-5944-02dcb5626528 Address:a4743941-6289-1233-5944-02dcb5626528} {Suffrage:Voter ID:server-1beb164c-d829-99d5-a391-4ba9596a5688 Address:1beb164c-d829-99d5-a391-4ba9596a5688} {Suffrage:Voter ID:server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8 Address:b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8}]" 2026-03-31T06:17:08.056Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:08.056Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: entering follower state: follower="Node at b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8 [Follower]" leader-address= leader-id= 2026-03-31T06:17:08.597Z [WARN] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:08.597Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: entering candidate state: node="Node at b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8 [Candidate]" term=2 2026-03-31T06:17:08.597Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: election won: term=2 tally=2 2026-03-31T06:17:08.597Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: entering leader state: leader="Node at b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8 [Leader]" 2026-03-31T06:17:08.597Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: added peer, starting replication: peer=server-a4743941-6289-1233-5944-02dcb5626528 2026-03-31T06:17:08.597Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: added peer, starting replication: peer=server-1beb164c-d829-99d5-a391-4ba9596a5688 2026-03-31T06:17:08.597Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: pipelining replication: peer="{Voter server-1beb164c-d829-99d5-a391-4ba9596a5688 1beb164c-d829-99d5-a391-4ba9596a5688}" 2026-03-31T06:17:08.598Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: pipelining replication: peer="{Voter server-a4743941-6289-1233-5944-02dcb5626528 a4743941-6289-1233-5944-02dcb5626528}" 2026-03-31T06:17:09.603Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:08.59791574 +0000 UTC m=+56.875523121 (1 nodes), 541.76404ms from start of poll, 542.24132ms from cluster start. Timeout at 2026-03-31 06:17:09.60291998 +0000 UTC m=+57.880527361, 1.00500424s after stability 2026-03-31T06:17:09.604Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: starting snapshot up to: index=12 2026-03-31T06:17:09.604Z [INFO] creating new snapshot: path=/tmp/raft2260129171/snapshots/2-12-1774937829604.tmp 2026-03-31T06:17:09.605Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: snapshot complete up to: index=12 2026-03-31T06:17:09.607Z [INFO] creating new snapshot: path=/tmp/raft2260129171/snapshots/2-113-1774937829607.tmp 2026-03-31T06:17:09.608Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: copied to local snapshot: bytes=71 2026-03-31T06:17:09.608Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: snapshot restore progress: id=2-113-1774937829607 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:09.608Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: restored user snapshot: index=113 2026-03-31T06:17:09.608Z [ERROR] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: failed to get log: index=23 error="log not found" 2026-03-31T06:17:09.608Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: aborting pipeline replication: peer="{Voter server-1beb164c-d829-99d5-a391-4ba9596a5688 1beb164c-d829-99d5-a391-4ba9596a5688}" 2026-03-31T06:17:09.608Z [ERROR] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: failed to get log: index=23 error="log not found" 2026-03-31T06:17:09.608Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: aborting pipeline replication: peer="{Voter server-a4743941-6289-1233-5944-02dcb5626528 a4743941-6289-1233-5944-02dcb5626528}" 2026-03-31T06:17:09.614Z [ERROR] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: failed to get log: index=23 error="log not found" 2026-03-31T06:17:09.615Z [ERROR] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: failed to get log: index=23 error="log not found" 2026-03-31T06:17:09.615Z [INFO] creating new snapshot: path=/tmp/raft1721862110/snapshots/2-113-1774937829615.tmp 2026-03-31T06:17:09.615Z [INFO] creating new snapshot: path=/tmp/raft2912833754/snapshots/2-113-1774937829615.tmp 2026-03-31T06:17:09.615Z [INFO] server-a4743941-6289-1233-5944-02dcb5626528: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:09.616Z [INFO] server-1beb164c-d829-99d5-a391-4ba9596a5688: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:09.616Z [INFO] server-a4743941-6289-1233-5944-02dcb5626528: copied to local snapshot: bytes=71 2026-03-31T06:17:09.616Z [INFO] server-1beb164c-d829-99d5-a391-4ba9596a5688: copied to local snapshot: bytes=71 2026-03-31T06:17:09.616Z [INFO] server-a4743941-6289-1233-5944-02dcb5626528: snapshot restore progress: id=2-113-1774937829615 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:09.616Z [INFO] server-a4743941-6289-1233-5944-02dcb5626528: Installed remote snapshot 2026-03-31T06:17:09.616Z [WARN] server-a4743941-6289-1233-5944-02dcb5626528: skipping application of old log: index=22 2026-03-31T06:17:09.616Z [INFO] server-1beb164c-d829-99d5-a391-4ba9596a5688: snapshot restore progress: id=2-113-1774937829615 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:09.616Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:09.616Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: pipelining replication: peer="{Voter server-a4743941-6289-1233-5944-02dcb5626528 a4743941-6289-1233-5944-02dcb5626528}" 2026-03-31T06:17:09.616Z [INFO] server-1beb164c-d829-99d5-a391-4ba9596a5688: Installed remote snapshot 2026-03-31T06:17:09.616Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: pipelining replication: peer="{Voter server-1beb164c-d829-99d5-a391-4ba9596a5688 1beb164c-d829-99d5-a391-4ba9596a5688}" 2026-03-31T06:17:10.622Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:09.6167285 +0000 UTC m=+57.894335881 (2 nodes), 920ns from start of poll, 1.56105408s from cluster start. Timeout at 2026-03-31 06:17:10.62174494 +0000 UTC m=+58.899352321, 1.00501644s after stability 2026-03-31T06:17:10.634Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: aborting pipeline replication: peer="{Voter server-1beb164c-d829-99d5-a391-4ba9596a5688 1beb164c-d829-99d5-a391-4ba9596a5688}" 2026-03-31T06:17:10.634Z [INFO] server-b6bc123e-1ffc-5060-96a6-2df4c8ddf8a8: aborting pipeline replication: peer="{Voter server-a4743941-6289-1233-5944-02dcb5626528 a4743941-6289-1233-5944-02dcb5626528}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false 2026-03-31T06:17:10.635Z [INFO] server-bfa88748-dc20-34e3-96d2-4431baabae9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfa88748-dc20-34e3-96d2-4431baabae9d Address:bfa88748-dc20-34e3-96d2-4431baabae9d} {Suffrage:Voter ID:server-4417bb25-3a54-caa0-765b-5d7b287cc1cf Address:4417bb25-3a54-caa0-765b-5d7b287cc1cf} {Suffrage:Voter ID:server-f0f2588a-bd40-55a3-ca48-4b05b3c48927 Address:f0f2588a-bd40-55a3-ca48-4b05b3c48927}]" 2026-03-31T06:17:10.635Z [INFO] server-4417bb25-3a54-caa0-765b-5d7b287cc1cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfa88748-dc20-34e3-96d2-4431baabae9d Address:bfa88748-dc20-34e3-96d2-4431baabae9d} {Suffrage:Voter ID:server-4417bb25-3a54-caa0-765b-5d7b287cc1cf Address:4417bb25-3a54-caa0-765b-5d7b287cc1cf} {Suffrage:Voter ID:server-f0f2588a-bd40-55a3-ca48-4b05b3c48927 Address:f0f2588a-bd40-55a3-ca48-4b05b3c48927}]" 2026-03-31T06:17:10.635Z [INFO] server-bfa88748-dc20-34e3-96d2-4431baabae9d: entering follower state: follower="Node at bfa88748-dc20-34e3-96d2-4431baabae9d [Follower]" leader-address= leader-id= 2026-03-31T06:17:10.636Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfa88748-dc20-34e3-96d2-4431baabae9d Address:bfa88748-dc20-34e3-96d2-4431baabae9d} {Suffrage:Voter ID:server-4417bb25-3a54-caa0-765b-5d7b287cc1cf Address:4417bb25-3a54-caa0-765b-5d7b287cc1cf} {Suffrage:Voter ID:server-f0f2588a-bd40-55a3-ca48-4b05b3c48927 Address:f0f2588a-bd40-55a3-ca48-4b05b3c48927}]" 2026-03-31T06:17:10.636Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:10.636Z [INFO] server-4417bb25-3a54-caa0-765b-5d7b287cc1cf: entering follower state: follower="Node at 4417bb25-3a54-caa0-765b-5d7b287cc1cf [Follower]" leader-address= leader-id= 2026-03-31T06:17:10.636Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: entering follower state: follower="Node at f0f2588a-bd40-55a3-ca48-4b05b3c48927 [Follower]" leader-address= leader-id= 2026-03-31T06:17:11.168Z [WARN] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:11.168Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: entering candidate state: node="Node at f0f2588a-bd40-55a3-ca48-4b05b3c48927 [Candidate]" term=2 2026-03-31T06:17:11.168Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: election won: term=2 tally=2 2026-03-31T06:17:11.168Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: entering leader state: leader="Node at f0f2588a-bd40-55a3-ca48-4b05b3c48927 [Leader]" 2026-03-31T06:17:11.168Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: added peer, starting replication: peer=server-bfa88748-dc20-34e3-96d2-4431baabae9d 2026-03-31T06:17:11.168Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: added peer, starting replication: peer=server-4417bb25-3a54-caa0-765b-5d7b287cc1cf 2026-03-31T06:17:11.168Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: pipelining replication: peer="{Voter server-4417bb25-3a54-caa0-765b-5d7b287cc1cf 4417bb25-3a54-caa0-765b-5d7b287cc1cf}" 2026-03-31T06:17:11.168Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: pipelining replication: peer="{Voter server-bfa88748-dc20-34e3-96d2-4431baabae9d bfa88748-dc20-34e3-96d2-4431baabae9d}" 2026-03-31T06:17:12.174Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:11.16860082 +0000 UTC m=+59.446208201 (1 nodes), 532.47936ms from start of poll, 532.89128ms from cluster start. Timeout at 2026-03-31 06:17:12.1736041 +0000 UTC m=+60.451211481, 1.00500328s after stability 2026-03-31T06:17:12.175Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: starting snapshot up to: index=12 2026-03-31T06:17:12.175Z [INFO] creating new snapshot: path=/tmp/raft3945630257/snapshots/2-12-1774937832175.tmp 2026-03-31T06:17:12.175Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: snapshot complete up to: index=12 2026-03-31T06:17:12.176Z [INFO] creating new snapshot: path=/tmp/raft3945630257/snapshots/2-113-1774937832176.tmp 2026-03-31T06:17:12.177Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: copied to local snapshot: bytes=71 2026-03-31T06:17:12.178Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: snapshot restore progress: id=2-113-1774937832176 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:12.178Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: removing all old logs from log store 2026-03-31T06:17:12.178Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: compacting logs: from=1 to=22 2026-03-31T06:17:12.178Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: restored user snapshot: index=113 2026-03-31T06:17:12.178Z [ERROR] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: failed to get log: index=22 error="log not found" 2026-03-31T06:17:12.178Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: aborting pipeline replication: peer="{Voter server-4417bb25-3a54-caa0-765b-5d7b287cc1cf 4417bb25-3a54-caa0-765b-5d7b287cc1cf}" 2026-03-31T06:17:12.178Z [ERROR] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: failed to get log: index=22 error="log not found" 2026-03-31T06:17:12.178Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: aborting pipeline replication: peer="{Voter server-bfa88748-dc20-34e3-96d2-4431baabae9d bfa88748-dc20-34e3-96d2-4431baabae9d}" 2026-03-31T06:17:12.188Z [ERROR] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: failed to get log: index=22 error="log not found" 2026-03-31T06:17:12.188Z [ERROR] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: failed to get log: index=22 error="log not found" 2026-03-31T06:17:12.189Z [INFO] creating new snapshot: path=/tmp/raft3254952146/snapshots/2-113-1774937832189.tmp 2026-03-31T06:17:12.189Z [INFO] creating new snapshot: path=/tmp/raft3786439934/snapshots/2-113-1774937832189.tmp 2026-03-31T06:17:12.189Z [INFO] server-4417bb25-3a54-caa0-765b-5d7b287cc1cf: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:12.189Z [INFO] server-bfa88748-dc20-34e3-96d2-4431baabae9d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:12.189Z [INFO] server-4417bb25-3a54-caa0-765b-5d7b287cc1cf: copied to local snapshot: bytes=71 2026-03-31T06:17:12.189Z [INFO] server-bfa88748-dc20-34e3-96d2-4431baabae9d: copied to local snapshot: bytes=71 2026-03-31T06:17:12.189Z [INFO] server-4417bb25-3a54-caa0-765b-5d7b287cc1cf: snapshot restore progress: id=2-113-1774937832189 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:12.189Z [INFO] server-4417bb25-3a54-caa0-765b-5d7b287cc1cf: removing all old logs from log store 2026-03-31T06:17:12.189Z [INFO] server-4417bb25-3a54-caa0-765b-5d7b287cc1cf: compacting logs: from=1 to=22 2026-03-31T06:17:12.189Z [INFO] server-4417bb25-3a54-caa0-765b-5d7b287cc1cf: Installed remote snapshot 2026-03-31T06:17:12.189Z [WARN] server-4417bb25-3a54-caa0-765b-5d7b287cc1cf: skipping application of old log: index=22 2026-03-31T06:17:12.189Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: pipelining replication: peer="{Voter server-4417bb25-3a54-caa0-765b-5d7b287cc1cf 4417bb25-3a54-caa0-765b-5d7b287cc1cf}" 2026-03-31T06:17:12.189Z [INFO] server-bfa88748-dc20-34e3-96d2-4431baabae9d: snapshot restore progress: id=2-113-1774937832189 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:12.190Z [INFO] server-bfa88748-dc20-34e3-96d2-4431baabae9d: removing all old logs from log store 2026-03-31T06:17:12.190Z [INFO] server-bfa88748-dc20-34e3-96d2-4431baabae9d: compacting logs: from=1 to=22 2026-03-31T06:17:12.190Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:12.190Z [INFO] server-bfa88748-dc20-34e3-96d2-4431baabae9d: Installed remote snapshot 2026-03-31T06:17:12.190Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: pipelining replication: peer="{Voter server-bfa88748-dc20-34e3-96d2-4431baabae9d bfa88748-dc20-34e3-96d2-4431baabae9d}" 2026-03-31T06:17:13.195Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:12.19003154 +0000 UTC m=+60.467638881 (2 nodes), 560ns from start of poll, 1.55432196s from cluster start. Timeout at 2026-03-31 06:17:13.19504794 +0000 UTC m=+61.472655321, 1.00501644s after stability 2026-03-31T06:17:13.206Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: aborting pipeline replication: peer="{Voter server-4417bb25-3a54-caa0-765b-5d7b287cc1cf 4417bb25-3a54-caa0-765b-5d7b287cc1cf}" 2026-03-31T06:17:13.206Z [INFO] server-f0f2588a-bd40-55a3-ca48-4b05b3c48927: aborting pipeline replication: peer="{Voter server-bfa88748-dc20-34e3-96d2-4431baabae9d bfa88748-dc20-34e3-96d2-4431baabae9d}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true 2026-03-31T06:17:13.207Z [INFO] server-dc0db525-da3a-2171-f9ad-bebffcc6b586: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc0db525-da3a-2171-f9ad-bebffcc6b586 Address:dc0db525-da3a-2171-f9ad-bebffcc6b586} {Suffrage:Voter ID:server-2246677e-332c-c3bb-f928-7bcf7ea7d52e Address:2246677e-332c-c3bb-f928-7bcf7ea7d52e} {Suffrage:Voter ID:server-02dd54c7-7f0e-14ef-d6f4-d8caf098a278 Address:02dd54c7-7f0e-14ef-d6f4-d8caf098a278}]" 2026-03-31T06:17:13.207Z [INFO] server-dc0db525-da3a-2171-f9ad-bebffcc6b586: entering follower state: follower="Node at dc0db525-da3a-2171-f9ad-bebffcc6b586 [Follower]" leader-address= leader-id= 2026-03-31T06:17:13.207Z [INFO] server-2246677e-332c-c3bb-f928-7bcf7ea7d52e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc0db525-da3a-2171-f9ad-bebffcc6b586 Address:dc0db525-da3a-2171-f9ad-bebffcc6b586} {Suffrage:Voter ID:server-2246677e-332c-c3bb-f928-7bcf7ea7d52e Address:2246677e-332c-c3bb-f928-7bcf7ea7d52e} {Suffrage:Voter ID:server-02dd54c7-7f0e-14ef-d6f4-d8caf098a278 Address:02dd54c7-7f0e-14ef-d6f4-d8caf098a278}]" 2026-03-31T06:17:13.207Z [INFO] server-2246677e-332c-c3bb-f928-7bcf7ea7d52e: entering follower state: follower="Node at 2246677e-332c-c3bb-f928-7bcf7ea7d52e [Follower]" leader-address= leader-id= 2026-03-31T06:17:13.207Z [INFO] server-02dd54c7-7f0e-14ef-d6f4-d8caf098a278: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc0db525-da3a-2171-f9ad-bebffcc6b586 Address:dc0db525-da3a-2171-f9ad-bebffcc6b586} {Suffrage:Voter ID:server-2246677e-332c-c3bb-f928-7bcf7ea7d52e Address:2246677e-332c-c3bb-f928-7bcf7ea7d52e} {Suffrage:Voter ID:server-02dd54c7-7f0e-14ef-d6f4-d8caf098a278 Address:02dd54c7-7f0e-14ef-d6f4-d8caf098a278}]" 2026-03-31T06:17:13.207Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:13.207Z [INFO] server-02dd54c7-7f0e-14ef-d6f4-d8caf098a278: entering follower state: follower="Node at 02dd54c7-7f0e-14ef-d6f4-d8caf098a278 [Follower]" leader-address= leader-id= 2026-03-31T06:17:13.865Z [WARN] server-dc0db525-da3a-2171-f9ad-bebffcc6b586: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:13.866Z [INFO] server-dc0db525-da3a-2171-f9ad-bebffcc6b586: entering candidate state: node="Node at dc0db525-da3a-2171-f9ad-bebffcc6b586 [Candidate]" term=2 2026-03-31T06:17:13.866Z [INFO] server-dc0db525-da3a-2171-f9ad-bebffcc6b586: election won: term=2 tally=2 2026-03-31T06:17:13.866Z [INFO] server-dc0db525-da3a-2171-f9ad-bebffcc6b586: entering leader state: leader="Node at dc0db525-da3a-2171-f9ad-bebffcc6b586 [Leader]" 2026-03-31T06:17:13.866Z [INFO] server-dc0db525-da3a-2171-f9ad-bebffcc6b586: added peer, starting replication: peer=server-2246677e-332c-c3bb-f928-7bcf7ea7d52e 2026-03-31T06:17:13.866Z [INFO] server-dc0db525-da3a-2171-f9ad-bebffcc6b586: added peer, starting replication: peer=server-02dd54c7-7f0e-14ef-d6f4-d8caf098a278 2026-03-31T06:17:13.866Z [INFO] server-dc0db525-da3a-2171-f9ad-bebffcc6b586: pipelining replication: peer="{Voter server-02dd54c7-7f0e-14ef-d6f4-d8caf098a278 02dd54c7-7f0e-14ef-d6f4-d8caf098a278}" 2026-03-31T06:17:13.866Z [INFO] server-dc0db525-da3a-2171-f9ad-bebffcc6b586: pipelining replication: peer="{Voter server-2246677e-332c-c3bb-f928-7bcf7ea7d52e 2246677e-332c-c3bb-f928-7bcf7ea7d52e}" 2026-03-31T06:17:14.871Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:13.86644078 +0000 UTC m=+62.144048121 (1 nodes), 658.59768ms from start of poll, 659.11952ms from cluster start. Timeout at 2026-03-31 06:17:14.87144194 +0000 UTC m=+63.149049281, 1.00500116s after stability 2026-03-31T06:17:14.872Z [INFO] server-dc0db525-da3a-2171-f9ad-bebffcc6b586: starting snapshot up to: index=12 2026-03-31T06:17:14.872Z [INFO] creating new snapshot: path=/tmp/raft1211106798/snapshots/2-12-1774937834872.tmp 2026-03-31T06:17:14.872Z [INFO] server-dc0db525-da3a-2171-f9ad-bebffcc6b586: snapshot complete up to: index=12 2026-03-31T06:17:14.873Z [INFO] server-c33788ab-1405-f9aa-f225-f6977710add8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c33788ab-1405-f9aa-f225-f6977710add8 Address:c33788ab-1405-f9aa-f225-f6977710add8} {Suffrage:Voter ID:server-3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a Address:3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a} {Suffrage:Voter ID:server-7103a22b-9cb3-50c6-d80e-148e3fd73724 Address:7103a22b-9cb3-50c6-d80e-148e3fd73724}]" 2026-03-31T06:17:14.873Z [INFO] server-3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c33788ab-1405-f9aa-f225-f6977710add8 Address:c33788ab-1405-f9aa-f225-f6977710add8} {Suffrage:Voter ID:server-3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a Address:3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a} {Suffrage:Voter ID:server-7103a22b-9cb3-50c6-d80e-148e3fd73724 Address:7103a22b-9cb3-50c6-d80e-148e3fd73724}]" 2026-03-31T06:17:14.873Z [INFO] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c33788ab-1405-f9aa-f225-f6977710add8 Address:c33788ab-1405-f9aa-f225-f6977710add8} {Suffrage:Voter ID:server-3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a Address:3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a} {Suffrage:Voter ID:server-7103a22b-9cb3-50c6-d80e-148e3fd73724 Address:7103a22b-9cb3-50c6-d80e-148e3fd73724}]" 2026-03-31T06:17:14.873Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:14.873Z [INFO] server-c33788ab-1405-f9aa-f225-f6977710add8: entering follower state: follower="Node at c33788ab-1405-f9aa-f225-f6977710add8 [Follower]" leader-address= leader-id= 2026-03-31T06:17:14.873Z [INFO] server-3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a: entering follower state: follower="Node at 3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a [Follower]" leader-address= leader-id= 2026-03-31T06:17:14.873Z [INFO] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: entering follower state: follower="Node at 7103a22b-9cb3-50c6-d80e-148e3fd73724 [Follower]" leader-address= leader-id= 2026-03-31T06:17:15.520Z [WARN] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:15.520Z [INFO] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: entering candidate state: node="Node at 7103a22b-9cb3-50c6-d80e-148e3fd73724 [Candidate]" term=2 2026-03-31T06:17:15.520Z [INFO] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: election won: term=2 tally=2 2026-03-31T06:17:15.520Z [INFO] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: entering leader state: leader="Node at 7103a22b-9cb3-50c6-d80e-148e3fd73724 [Leader]" 2026-03-31T06:17:15.520Z [INFO] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: added peer, starting replication: peer=server-c33788ab-1405-f9aa-f225-f6977710add8 2026-03-31T06:17:15.520Z [INFO] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: added peer, starting replication: peer=server-3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a 2026-03-31T06:17:15.520Z [INFO] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: pipelining replication: peer="{Voter server-c33788ab-1405-f9aa-f225-f6977710add8 c33788ab-1405-f9aa-f225-f6977710add8}" 2026-03-31T06:17:15.520Z [INFO] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: pipelining replication: peer="{Voter server-3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a 3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a}" 2026-03-31T06:17:16.526Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:15.52054198 +0000 UTC m=+63.798149361 (1 nodes), 647.16308ms from start of poll, 647.4322ms from cluster start. Timeout at 2026-03-31 06:17:16.52554382 +0000 UTC m=+64.803151201, 1.00500184s after stability 2026-03-31T06:17:16.526Z [INFO] creating new snapshot: path=/tmp/raft279774614/snapshots/2-113-1774937836526.tmp 2026-03-31T06:17:16.526Z [INFO] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: copied to local snapshot: bytes=71 2026-03-31T06:17:16.527Z [INFO] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: snapshot restore progress: id=2-113-1774937836526 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:16.527Z [INFO] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: restored user snapshot: index=113 2026-03-31T06:17:16.527Z [ERROR] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: failed to get log: index=3 error="log not found" 2026-03-31T06:17:16.527Z [INFO] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: aborting pipeline replication: peer="{Voter server-3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a 3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a}" 2026-03-31T06:17:16.527Z [ERROR] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: failed to get log: index=3 error="log not found" 2026-03-31T06:17:16.527Z [INFO] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: aborting pipeline replication: peer="{Voter server-c33788ab-1405-f9aa-f225-f6977710add8 c33788ab-1405-f9aa-f225-f6977710add8}" 2026-03-31T06:17:16.534Z [ERROR] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: failed to get log: index=3 error="log not found" 2026-03-31T06:17:16.535Z [INFO] creating new snapshot: path=/tmp/raft3000024908/snapshots/2-113-1774937836535.tmp 2026-03-31T06:17:16.535Z [ERROR] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: failed to get log: index=3 error="log not found" 2026-03-31T06:17:16.535Z [INFO] server-c33788ab-1405-f9aa-f225-f6977710add8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:16.535Z [INFO] creating new snapshot: path=/tmp/raft2889474590/snapshots/2-113-1774937836535.tmp 2026-03-31T06:17:16.536Z [INFO] server-c33788ab-1405-f9aa-f225-f6977710add8: copied to local snapshot: bytes=71 2026-03-31T06:17:16.536Z [INFO] server-3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:16.536Z [INFO] server-c33788ab-1405-f9aa-f225-f6977710add8: snapshot restore progress: id=2-113-1774937836535 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:16.537Z [INFO] server-c33788ab-1405-f9aa-f225-f6977710add8: Installed remote snapshot 2026-03-31T06:17:16.537Z [INFO] server-3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a: copied to local snapshot: bytes=71 2026-03-31T06:17:16.537Z [INFO] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: pipelining replication: peer="{Voter server-c33788ab-1405-f9aa-f225-f6977710add8 c33788ab-1405-f9aa-f225-f6977710add8}" 2026-03-31T06:17:16.537Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:16.537Z [INFO] server-3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a: snapshot restore progress: id=2-113-1774937836535 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:16.537Z [INFO] server-3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a: Installed remote snapshot 2026-03-31T06:17:16.537Z [INFO] server-7103a22b-9cb3-50c6-d80e-148e3fd73724: pipelining replication: peer="{Voter server-3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a 3c76c0d2-0bb9-ad86-76e0-ec71e7dd490a}" 2026-03-31T06:17:17.543Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:16.53732302 +0000 UTC m=+64.814930401 (2 nodes), 880ns from start of poll, 1.66421324s from cluster start. Timeout at 2026-03-31 06:17:17.54235126 +0000 UTC m=+65.819958641, 1.00502824s after stability 2026-03-31T06:17:17.554Z [INFO] server-dc0db525-da3a-2171-f9ad-bebffcc6b586: aborting pipeline replication: peer="{Voter server-02dd54c7-7f0e-14ef-d6f4-d8caf098a278 02dd54c7-7f0e-14ef-d6f4-d8caf098a278}" 2026-03-31T06:17:17.554Z [INFO] server-dc0db525-da3a-2171-f9ad-bebffcc6b586: aborting pipeline replication: peer="{Voter server-2246677e-332c-c3bb-f928-7bcf7ea7d52e 2246677e-332c-c3bb-f928-7bcf7ea7d52e}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true 2026-03-31T06:17:17.555Z [INFO] server-7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5 Address:7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5} {Suffrage:Voter ID:server-6903bdac-4d6e-0229-4552-b866adb4f266 Address:6903bdac-4d6e-0229-4552-b866adb4f266} {Suffrage:Voter ID:server-d382be12-0329-41de-5a8f-ec900400de82 Address:d382be12-0329-41de-5a8f-ec900400de82}]" 2026-03-31T06:17:17.555Z [INFO] server-6903bdac-4d6e-0229-4552-b866adb4f266: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5 Address:7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5} {Suffrage:Voter ID:server-6903bdac-4d6e-0229-4552-b866adb4f266 Address:6903bdac-4d6e-0229-4552-b866adb4f266} {Suffrage:Voter ID:server-d382be12-0329-41de-5a8f-ec900400de82 Address:d382be12-0329-41de-5a8f-ec900400de82}]" 2026-03-31T06:17:17.555Z [INFO] server-7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5: entering follower state: follower="Node at 7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5 [Follower]" leader-address= leader-id= 2026-03-31T06:17:17.555Z [INFO] server-6903bdac-4d6e-0229-4552-b866adb4f266: entering follower state: follower="Node at 6903bdac-4d6e-0229-4552-b866adb4f266 [Follower]" leader-address= leader-id= 2026-03-31T06:17:17.555Z [INFO] server-d382be12-0329-41de-5a8f-ec900400de82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5 Address:7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5} {Suffrage:Voter ID:server-6903bdac-4d6e-0229-4552-b866adb4f266 Address:6903bdac-4d6e-0229-4552-b866adb4f266} {Suffrage:Voter ID:server-d382be12-0329-41de-5a8f-ec900400de82 Address:d382be12-0329-41de-5a8f-ec900400de82}]" 2026-03-31T06:17:17.555Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:17.555Z [INFO] server-d382be12-0329-41de-5a8f-ec900400de82: entering follower state: follower="Node at d382be12-0329-41de-5a8f-ec900400de82 [Follower]" leader-address= leader-id= 2026-03-31T06:17:18.078Z [WARN] server-7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:18.078Z [INFO] server-7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5: entering candidate state: node="Node at 7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5 [Candidate]" term=2 2026-03-31T06:17:18.078Z [INFO] server-7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5: election won: term=2 tally=2 2026-03-31T06:17:18.078Z [INFO] server-7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5: entering leader state: leader="Node at 7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5 [Leader]" 2026-03-31T06:17:18.078Z [INFO] server-7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5: added peer, starting replication: peer=server-6903bdac-4d6e-0229-4552-b866adb4f266 2026-03-31T06:17:18.078Z [INFO] server-7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5: added peer, starting replication: peer=server-d382be12-0329-41de-5a8f-ec900400de82 2026-03-31T06:17:18.078Z [INFO] server-7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5: pipelining replication: peer="{Voter server-d382be12-0329-41de-5a8f-ec900400de82 d382be12-0329-41de-5a8f-ec900400de82}" 2026-03-31T06:17:18.078Z [INFO] server-7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5: pipelining replication: peer="{Voter server-6903bdac-4d6e-0229-4552-b866adb4f266 6903bdac-4d6e-0229-4552-b866adb4f266}" 2026-03-31T06:17:19.084Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:18.07859478 +0000 UTC m=+66.356202161 (1 nodes), 522.74712ms from start of poll, 523.24292ms from cluster start. Timeout at 2026-03-31 06:17:19.08360922 +0000 UTC m=+67.361216601, 1.00501444s after stability 2026-03-31T06:17:19.085Z [INFO] server-7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5: starting snapshot up to: index=12 2026-03-31T06:17:19.085Z [INFO] creating new snapshot: path=/tmp/raft250992349/snapshots/2-12-1774937839085.tmp 2026-03-31T06:17:19.085Z [INFO] server-7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5: snapshot complete up to: index=12 2026-03-31T06:17:19.086Z [INFO] server-1ea97fd9-8802-7f70-8534-9cde4f63a859: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ea97fd9-8802-7f70-8534-9cde4f63a859 Address:1ea97fd9-8802-7f70-8534-9cde4f63a859} {Suffrage:Voter ID:server-6fdcb464-59b9-f125-72bc-0c50ecd06dfc Address:6fdcb464-59b9-f125-72bc-0c50ecd06dfc} {Suffrage:Voter ID:server-7147a64a-30f0-2739-5437-bd868944a92f Address:7147a64a-30f0-2739-5437-bd868944a92f}]" 2026-03-31T06:17:19.087Z [INFO] server-6fdcb464-59b9-f125-72bc-0c50ecd06dfc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ea97fd9-8802-7f70-8534-9cde4f63a859 Address:1ea97fd9-8802-7f70-8534-9cde4f63a859} {Suffrage:Voter ID:server-6fdcb464-59b9-f125-72bc-0c50ecd06dfc Address:6fdcb464-59b9-f125-72bc-0c50ecd06dfc} {Suffrage:Voter ID:server-7147a64a-30f0-2739-5437-bd868944a92f Address:7147a64a-30f0-2739-5437-bd868944a92f}]" 2026-03-31T06:17:19.087Z [INFO] server-1ea97fd9-8802-7f70-8534-9cde4f63a859: entering follower state: follower="Node at 1ea97fd9-8802-7f70-8534-9cde4f63a859 [Follower]" leader-address= leader-id= 2026-03-31T06:17:19.087Z [INFO] server-6fdcb464-59b9-f125-72bc-0c50ecd06dfc: entering follower state: follower="Node at 6fdcb464-59b9-f125-72bc-0c50ecd06dfc [Follower]" leader-address= leader-id= 2026-03-31T06:17:19.087Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ea97fd9-8802-7f70-8534-9cde4f63a859 Address:1ea97fd9-8802-7f70-8534-9cde4f63a859} {Suffrage:Voter ID:server-6fdcb464-59b9-f125-72bc-0c50ecd06dfc Address:6fdcb464-59b9-f125-72bc-0c50ecd06dfc} {Suffrage:Voter ID:server-7147a64a-30f0-2739-5437-bd868944a92f Address:7147a64a-30f0-2739-5437-bd868944a92f}]" 2026-03-31T06:17:19.087Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:19.087Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: entering follower state: follower="Node at 7147a64a-30f0-2739-5437-bd868944a92f [Follower]" leader-address= leader-id= 2026-03-31T06:17:19.817Z [WARN] server-7147a64a-30f0-2739-5437-bd868944a92f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:19.817Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: entering candidate state: node="Node at 7147a64a-30f0-2739-5437-bd868944a92f [Candidate]" term=2 2026-03-31T06:17:19.817Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: election won: term=2 tally=2 2026-03-31T06:17:19.817Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: entering leader state: leader="Node at 7147a64a-30f0-2739-5437-bd868944a92f [Leader]" 2026-03-31T06:17:19.817Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: added peer, starting replication: peer=server-1ea97fd9-8802-7f70-8534-9cde4f63a859 2026-03-31T06:17:19.817Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: added peer, starting replication: peer=server-6fdcb464-59b9-f125-72bc-0c50ecd06dfc 2026-03-31T06:17:19.817Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: pipelining replication: peer="{Voter server-1ea97fd9-8802-7f70-8534-9cde4f63a859 1ea97fd9-8802-7f70-8534-9cde4f63a859}" 2026-03-31T06:17:19.817Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: pipelining replication: peer="{Voter server-6fdcb464-59b9-f125-72bc-0c50ecd06dfc 6fdcb464-59b9-f125-72bc-0c50ecd06dfc}" 2026-03-31T06:17:20.823Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:19.8175289 +0000 UTC m=+68.095136281 (1 nodes), 730.24484ms from start of poll, 730.68816ms from cluster start. Timeout at 2026-03-31 06:17:20.82253094 +0000 UTC m=+69.100138361, 1.00500208s after stability 2026-03-31T06:17:20.823Z [INFO] creating new snapshot: path=/tmp/raft2637456169/snapshots/2-113-1774937840823.tmp 2026-03-31T06:17:20.824Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: copied to local snapshot: bytes=71 2026-03-31T06:17:20.824Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: snapshot restore progress: id=2-113-1774937840823 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:20.824Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: removing all old logs from log store 2026-03-31T06:17:20.824Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: compacting logs: from=1 to=2 2026-03-31T06:17:20.824Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: restored user snapshot: index=113 2026-03-31T06:17:20.824Z [ERROR] server-7147a64a-30f0-2739-5437-bd868944a92f: failed to get log: index=2 error="log not found" 2026-03-31T06:17:20.824Z [ERROR] server-7147a64a-30f0-2739-5437-bd868944a92f: failed to get log: index=2 error="log not found" 2026-03-31T06:17:20.824Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: aborting pipeline replication: peer="{Voter server-1ea97fd9-8802-7f70-8534-9cde4f63a859 1ea97fd9-8802-7f70-8534-9cde4f63a859}" 2026-03-31T06:17:20.824Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: aborting pipeline replication: peer="{Voter server-6fdcb464-59b9-f125-72bc-0c50ecd06dfc 6fdcb464-59b9-f125-72bc-0c50ecd06dfc}" 2026-03-31T06:17:20.831Z [ERROR] server-7147a64a-30f0-2739-5437-bd868944a92f: failed to get log: index=2 error="log not found" 2026-03-31T06:17:20.832Z [INFO] creating new snapshot: path=/tmp/raft2657706990/snapshots/2-113-1774937840832.tmp 2026-03-31T06:17:20.832Z [INFO] server-1ea97fd9-8802-7f70-8534-9cde4f63a859: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:20.832Z [INFO] server-1ea97fd9-8802-7f70-8534-9cde4f63a859: copied to local snapshot: bytes=71 2026-03-31T06:17:20.833Z [INFO] server-1ea97fd9-8802-7f70-8534-9cde4f63a859: snapshot restore progress: id=2-113-1774937840832 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:20.833Z [INFO] server-1ea97fd9-8802-7f70-8534-9cde4f63a859: removing all old logs from log store 2026-03-31T06:17:20.833Z [INFO] server-1ea97fd9-8802-7f70-8534-9cde4f63a859: compacting logs: from=1 to=2 2026-03-31T06:17:20.833Z [INFO] server-1ea97fd9-8802-7f70-8534-9cde4f63a859: Installed remote snapshot 2026-03-31T06:17:20.833Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: pipelining replication: peer="{Voter server-1ea97fd9-8802-7f70-8534-9cde4f63a859 1ea97fd9-8802-7f70-8534-9cde4f63a859}" 2026-03-31T06:17:20.833Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:20.834Z [ERROR] server-7147a64a-30f0-2739-5437-bd868944a92f: failed to get log: index=2 error="log not found" 2026-03-31T06:17:20.834Z [INFO] creating new snapshot: path=/tmp/raft220395483/snapshots/2-113-1774937840834.tmp 2026-03-31T06:17:20.834Z [INFO] server-6fdcb464-59b9-f125-72bc-0c50ecd06dfc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:20.835Z [INFO] server-6fdcb464-59b9-f125-72bc-0c50ecd06dfc: copied to local snapshot: bytes=71 2026-03-31T06:17:20.835Z [INFO] server-6fdcb464-59b9-f125-72bc-0c50ecd06dfc: snapshot restore progress: id=2-113-1774937840834 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:20.835Z [INFO] server-6fdcb464-59b9-f125-72bc-0c50ecd06dfc: removing all old logs from log store 2026-03-31T06:17:20.835Z [INFO] server-6fdcb464-59b9-f125-72bc-0c50ecd06dfc: compacting logs: from=1 to=2 2026-03-31T06:17:20.835Z [INFO] server-6fdcb464-59b9-f125-72bc-0c50ecd06dfc: Installed remote snapshot 2026-03-31T06:17:20.835Z [INFO] server-7147a64a-30f0-2739-5437-bd868944a92f: pipelining replication: peer="{Voter server-6fdcb464-59b9-f125-72bc-0c50ecd06dfc 6fdcb464-59b9-f125-72bc-0c50ecd06dfc}" 2026-03-31T06:17:21.838Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:20.83340282 +0000 UTC m=+69.111010201 (2 nodes), 880ns from start of poll, 1.74656208s from cluster start. Timeout at 2026-03-31 06:17:21.83843322 +0000 UTC m=+70.116040641, 1.00503044s after stability 2026-03-31T06:17:21.851Z [INFO] server-7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5: aborting pipeline replication: peer="{Voter server-d382be12-0329-41de-5a8f-ec900400de82 d382be12-0329-41de-5a8f-ec900400de82}" 2026-03-31T06:17:21.851Z [INFO] server-7b2946a3-9dce-c8ef-e6f8-65a28bdef6c5: aborting pipeline replication: peer="{Voter server-6903bdac-4d6e-0229-4552-b866adb4f266 6903bdac-4d6e-0229-4552-b866adb4f266}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false 2026-03-31T06:17:21.852Z [INFO] server-01c982eb-e05b-b25a-14e8-f026c0e6be91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01c982eb-e05b-b25a-14e8-f026c0e6be91 Address:01c982eb-e05b-b25a-14e8-f026c0e6be91} {Suffrage:Voter ID:server-04f53e83-a7bb-1e95-46e1-792cd39035b4 Address:04f53e83-a7bb-1e95-46e1-792cd39035b4} {Suffrage:Voter ID:server-6091ddde-ff76-fd65-1c4e-209863160515 Address:6091ddde-ff76-fd65-1c4e-209863160515}]" 2026-03-31T06:17:21.852Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01c982eb-e05b-b25a-14e8-f026c0e6be91 Address:01c982eb-e05b-b25a-14e8-f026c0e6be91} {Suffrage:Voter ID:server-04f53e83-a7bb-1e95-46e1-792cd39035b4 Address:04f53e83-a7bb-1e95-46e1-792cd39035b4} {Suffrage:Voter ID:server-6091ddde-ff76-fd65-1c4e-209863160515 Address:6091ddde-ff76-fd65-1c4e-209863160515}]" 2026-03-31T06:17:21.852Z [INFO] server-01c982eb-e05b-b25a-14e8-f026c0e6be91: entering follower state: follower="Node at 01c982eb-e05b-b25a-14e8-f026c0e6be91 [Follower]" leader-address= leader-id= 2026-03-31T06:17:21.852Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: entering follower state: follower="Node at 04f53e83-a7bb-1e95-46e1-792cd39035b4 [Follower]" leader-address= leader-id= 2026-03-31T06:17:21.852Z [INFO] server-6091ddde-ff76-fd65-1c4e-209863160515: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01c982eb-e05b-b25a-14e8-f026c0e6be91 Address:01c982eb-e05b-b25a-14e8-f026c0e6be91} {Suffrage:Voter ID:server-04f53e83-a7bb-1e95-46e1-792cd39035b4 Address:04f53e83-a7bb-1e95-46e1-792cd39035b4} {Suffrage:Voter ID:server-6091ddde-ff76-fd65-1c4e-209863160515 Address:6091ddde-ff76-fd65-1c4e-209863160515}]" 2026-03-31T06:17:21.852Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:21.852Z [INFO] server-6091ddde-ff76-fd65-1c4e-209863160515: entering follower state: follower="Node at 6091ddde-ff76-fd65-1c4e-209863160515 [Follower]" leader-address= leader-id= 2026-03-31T06:17:22.533Z [WARN] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:22.533Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: entering candidate state: node="Node at 04f53e83-a7bb-1e95-46e1-792cd39035b4 [Candidate]" term=2 2026-03-31T06:17:22.534Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: election won: term=2 tally=2 2026-03-31T06:17:22.534Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: entering leader state: leader="Node at 04f53e83-a7bb-1e95-46e1-792cd39035b4 [Leader]" 2026-03-31T06:17:22.534Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: added peer, starting replication: peer=server-01c982eb-e05b-b25a-14e8-f026c0e6be91 2026-03-31T06:17:22.534Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: added peer, starting replication: peer=server-6091ddde-ff76-fd65-1c4e-209863160515 2026-03-31T06:17:22.534Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: pipelining replication: peer="{Voter server-6091ddde-ff76-fd65-1c4e-209863160515 6091ddde-ff76-fd65-1c4e-209863160515}" 2026-03-31T06:17:22.534Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: pipelining replication: peer="{Voter server-01c982eb-e05b-b25a-14e8-f026c0e6be91 01c982eb-e05b-b25a-14e8-f026c0e6be91}" 2026-03-31T06:17:23.539Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:22.53416798 +0000 UTC m=+70.811775361 (1 nodes), 681.207ms from start of poll, 681.63516ms from cluster start. Timeout at 2026-03-31 06:17:23.53917134 +0000 UTC m=+71.816778721, 1.00500336s after stability 2026-03-31T06:17:23.540Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: starting snapshot up to: index=12 2026-03-31T06:17:23.540Z [INFO] creating new snapshot: path=/tmp/raft3474682345/snapshots/2-12-1774937843540.tmp 2026-03-31T06:17:23.540Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: snapshot complete up to: index=12 2026-03-31T06:17:23.541Z [INFO] creating new snapshot: path=/tmp/raft3474682345/snapshots/2-1013-1774937843541.tmp 2026-03-31T06:17:23.542Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: copied to local snapshot: bytes=71 2026-03-31T06:17:23.542Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: snapshot restore progress: id=2-1013-1774937843541 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:23.542Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: restored user snapshot: index=1013 2026-03-31T06:17:23.542Z [ERROR] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: failed to get log: index=23 error="log not found" 2026-03-31T06:17:23.542Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: aborting pipeline replication: peer="{Voter server-6091ddde-ff76-fd65-1c4e-209863160515 6091ddde-ff76-fd65-1c4e-209863160515}" 2026-03-31T06:17:23.542Z [ERROR] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: failed to get log: index=23 error="log not found" 2026-03-31T06:17:23.542Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: aborting pipeline replication: peer="{Voter server-01c982eb-e05b-b25a-14e8-f026c0e6be91 01c982eb-e05b-b25a-14e8-f026c0e6be91}" 2026-03-31T06:17:23.549Z [ERROR] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: failed to get log: index=23 error="log not found" 2026-03-31T06:17:23.549Z [ERROR] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: failed to get log: index=23 error="log not found" 2026-03-31T06:17:23.550Z [INFO] creating new snapshot: path=/tmp/raft242502708/snapshots/2-1013-1774937843550.tmp 2026-03-31T06:17:23.550Z [INFO] creating new snapshot: path=/tmp/raft4168585622/snapshots/2-1013-1774937843550.tmp 2026-03-31T06:17:23.550Z [INFO] server-6091ddde-ff76-fd65-1c4e-209863160515: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:23.550Z [INFO] server-01c982eb-e05b-b25a-14e8-f026c0e6be91: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:23.550Z [INFO] server-6091ddde-ff76-fd65-1c4e-209863160515: copied to local snapshot: bytes=71 2026-03-31T06:17:23.551Z [INFO] server-6091ddde-ff76-fd65-1c4e-209863160515: snapshot restore progress: id=2-1013-1774937843550 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:23.551Z [INFO] server-6091ddde-ff76-fd65-1c4e-209863160515: Installed remote snapshot 2026-03-31T06:17:23.551Z [WARN] server-6091ddde-ff76-fd65-1c4e-209863160515: skipping application of old log: index=22 2026-03-31T06:17:23.551Z [INFO] server-01c982eb-e05b-b25a-14e8-f026c0e6be91: copied to local snapshot: bytes=71 2026-03-31T06:17:23.552Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: pipelining replication: peer="{Voter server-6091ddde-ff76-fd65-1c4e-209863160515 6091ddde-ff76-fd65-1c4e-209863160515}" 2026-03-31T06:17:23.552Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:23.552Z [INFO] server-01c982eb-e05b-b25a-14e8-f026c0e6be91: snapshot restore progress: id=2-1013-1774937843550 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:23.552Z [INFO] server-01c982eb-e05b-b25a-14e8-f026c0e6be91: Installed remote snapshot 2026-03-31T06:17:23.552Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: pipelining replication: peer="{Voter server-01c982eb-e05b-b25a-14e8-f026c0e6be91 01c982eb-e05b-b25a-14e8-f026c0e6be91}" 2026-03-31T06:17:24.558Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:23.55245846 +0000 UTC m=+71.830065841 (2 nodes), 1.64µs from start of poll, 1.69992564s from cluster start. Timeout at 2026-03-31 06:17:24.5574745 +0000 UTC m=+72.835081881, 1.00501604s after stability 2026-03-31T06:17:24.569Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: aborting pipeline replication: peer="{Voter server-6091ddde-ff76-fd65-1c4e-209863160515 6091ddde-ff76-fd65-1c4e-209863160515}" 2026-03-31T06:17:24.569Z [INFO] server-04f53e83-a7bb-1e95-46e1-792cd39035b4: aborting pipeline replication: peer="{Voter server-01c982eb-e05b-b25a-14e8-f026c0e6be91 01c982eb-e05b-b25a-14e8-f026c0e6be91}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false 2026-03-31T06:17:24.571Z [INFO] server-32f71c9e-2a0e-5da4-829b-7c61ce435bf6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32f71c9e-2a0e-5da4-829b-7c61ce435bf6 Address:32f71c9e-2a0e-5da4-829b-7c61ce435bf6} {Suffrage:Voter ID:server-48f913ad-39bc-88e0-7156-8f530c0a31d0 Address:48f913ad-39bc-88e0-7156-8f530c0a31d0} {Suffrage:Voter ID:server-0dd973e7-df15-9326-3d5e-f0151bedc105 Address:0dd973e7-df15-9326-3d5e-f0151bedc105}]" 2026-03-31T06:17:24.571Z [INFO] server-48f913ad-39bc-88e0-7156-8f530c0a31d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32f71c9e-2a0e-5da4-829b-7c61ce435bf6 Address:32f71c9e-2a0e-5da4-829b-7c61ce435bf6} {Suffrage:Voter ID:server-48f913ad-39bc-88e0-7156-8f530c0a31d0 Address:48f913ad-39bc-88e0-7156-8f530c0a31d0} {Suffrage:Voter ID:server-0dd973e7-df15-9326-3d5e-f0151bedc105 Address:0dd973e7-df15-9326-3d5e-f0151bedc105}]" 2026-03-31T06:17:24.571Z [INFO] server-32f71c9e-2a0e-5da4-829b-7c61ce435bf6: entering follower state: follower="Node at 32f71c9e-2a0e-5da4-829b-7c61ce435bf6 [Follower]" leader-address= leader-id= 2026-03-31T06:17:24.571Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32f71c9e-2a0e-5da4-829b-7c61ce435bf6 Address:32f71c9e-2a0e-5da4-829b-7c61ce435bf6} {Suffrage:Voter ID:server-48f913ad-39bc-88e0-7156-8f530c0a31d0 Address:48f913ad-39bc-88e0-7156-8f530c0a31d0} {Suffrage:Voter ID:server-0dd973e7-df15-9326-3d5e-f0151bedc105 Address:0dd973e7-df15-9326-3d5e-f0151bedc105}]" 2026-03-31T06:17:24.571Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:24.571Z [INFO] server-48f913ad-39bc-88e0-7156-8f530c0a31d0: entering follower state: follower="Node at 48f913ad-39bc-88e0-7156-8f530c0a31d0 [Follower]" leader-address= leader-id= 2026-03-31T06:17:24.571Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: entering follower state: follower="Node at 0dd973e7-df15-9326-3d5e-f0151bedc105 [Follower]" leader-address= leader-id= 2026-03-31T06:17:25.214Z [WARN] server-0dd973e7-df15-9326-3d5e-f0151bedc105: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:25.214Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: entering candidate state: node="Node at 0dd973e7-df15-9326-3d5e-f0151bedc105 [Candidate]" term=2 2026-03-31T06:17:25.214Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: election won: term=2 tally=2 2026-03-31T06:17:25.214Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: entering leader state: leader="Node at 0dd973e7-df15-9326-3d5e-f0151bedc105 [Leader]" 2026-03-31T06:17:25.214Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: added peer, starting replication: peer=server-32f71c9e-2a0e-5da4-829b-7c61ce435bf6 2026-03-31T06:17:25.214Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: added peer, starting replication: peer=server-48f913ad-39bc-88e0-7156-8f530c0a31d0 2026-03-31T06:17:25.214Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: pipelining replication: peer="{Voter server-48f913ad-39bc-88e0-7156-8f530c0a31d0 48f913ad-39bc-88e0-7156-8f530c0a31d0}" 2026-03-31T06:17:25.214Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: pipelining replication: peer="{Voter server-32f71c9e-2a0e-5da4-829b-7c61ce435bf6 32f71c9e-2a0e-5da4-829b-7c61ce435bf6}" 2026-03-31T06:17:26.220Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:25.21468022 +0000 UTC m=+73.492287601 (1 nodes), 643.33576ms from start of poll, 643.756ms from cluster start. Timeout at 2026-03-31 06:17:26.21968406 +0000 UTC m=+74.497291441, 1.00500384s after stability 2026-03-31T06:17:26.221Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: starting snapshot up to: index=12 2026-03-31T06:17:26.221Z [INFO] creating new snapshot: path=/tmp/raft1012957094/snapshots/2-12-1774937846221.tmp 2026-03-31T06:17:26.221Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: snapshot complete up to: index=12 2026-03-31T06:17:26.224Z [INFO] creating new snapshot: path=/tmp/raft1012957094/snapshots/2-1013-1774937846224.tmp 2026-03-31T06:17:26.225Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: copied to local snapshot: bytes=71 2026-03-31T06:17:26.225Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: snapshot restore progress: id=2-1013-1774937846224 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:26.225Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: removing all old logs from log store 2026-03-31T06:17:26.225Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: compacting logs: from=1 to=22 2026-03-31T06:17:26.225Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: restored user snapshot: index=1013 2026-03-31T06:17:26.225Z [ERROR] server-0dd973e7-df15-9326-3d5e-f0151bedc105: failed to get log: index=22 error="log not found" 2026-03-31T06:17:26.225Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: aborting pipeline replication: peer="{Voter server-32f71c9e-2a0e-5da4-829b-7c61ce435bf6 32f71c9e-2a0e-5da4-829b-7c61ce435bf6}" 2026-03-31T06:17:26.225Z [ERROR] server-0dd973e7-df15-9326-3d5e-f0151bedc105: failed to get log: index=22 error="log not found" 2026-03-31T06:17:26.225Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: aborting pipeline replication: peer="{Voter server-48f913ad-39bc-88e0-7156-8f530c0a31d0 48f913ad-39bc-88e0-7156-8f530c0a31d0}" 2026-03-31T06:17:26.234Z [ERROR] server-0dd973e7-df15-9326-3d5e-f0151bedc105: failed to get log: index=22 error="log not found" 2026-03-31T06:17:26.234Z [ERROR] server-0dd973e7-df15-9326-3d5e-f0151bedc105: failed to get log: index=22 error="log not found" 2026-03-31T06:17:26.235Z [INFO] creating new snapshot: path=/tmp/raft857983343/snapshots/2-1013-1774937846235.tmp 2026-03-31T06:17:26.235Z [INFO] server-32f71c9e-2a0e-5da4-829b-7c61ce435bf6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:26.235Z [INFO] creating new snapshot: path=/tmp/raft587305711/snapshots/2-1013-1774937846235.tmp 2026-03-31T06:17:26.235Z [INFO] server-48f913ad-39bc-88e0-7156-8f530c0a31d0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:26.235Z [INFO] server-32f71c9e-2a0e-5da4-829b-7c61ce435bf6: copied to local snapshot: bytes=71 2026-03-31T06:17:26.235Z [INFO] server-48f913ad-39bc-88e0-7156-8f530c0a31d0: copied to local snapshot: bytes=71 2026-03-31T06:17:26.236Z [INFO] server-32f71c9e-2a0e-5da4-829b-7c61ce435bf6: snapshot restore progress: id=2-1013-1774937846235 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:26.236Z [INFO] server-32f71c9e-2a0e-5da4-829b-7c61ce435bf6: removing all old logs from log store 2026-03-31T06:17:26.236Z [INFO] server-32f71c9e-2a0e-5da4-829b-7c61ce435bf6: compacting logs: from=1 to=22 2026-03-31T06:17:26.236Z [INFO] server-32f71c9e-2a0e-5da4-829b-7c61ce435bf6: Installed remote snapshot 2026-03-31T06:17:26.236Z [INFO] server-48f913ad-39bc-88e0-7156-8f530c0a31d0: snapshot restore progress: id=2-1013-1774937846235 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:26.236Z [INFO] server-48f913ad-39bc-88e0-7156-8f530c0a31d0: removing all old logs from log store 2026-03-31T06:17:26.236Z [INFO] server-48f913ad-39bc-88e0-7156-8f530c0a31d0: compacting logs: from=1 to=22 2026-03-31T06:17:26.236Z [INFO] server-48f913ad-39bc-88e0-7156-8f530c0a31d0: Installed remote snapshot 2026-03-31T06:17:26.236Z [WARN] server-32f71c9e-2a0e-5da4-829b-7c61ce435bf6: skipping application of old log: index=22 2026-03-31T06:17:26.236Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: pipelining replication: peer="{Voter server-32f71c9e-2a0e-5da4-829b-7c61ce435bf6 32f71c9e-2a0e-5da4-829b-7c61ce435bf6}" 2026-03-31T06:17:26.236Z [WARN] server-48f913ad-39bc-88e0-7156-8f530c0a31d0: skipping application of old log: index=1013 2026-03-31T06:17:26.236Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:26.236Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: pipelining replication: peer="{Voter server-48f913ad-39bc-88e0-7156-8f530c0a31d0 48f913ad-39bc-88e0-7156-8f530c0a31d0}" 2026-03-31T06:17:27.242Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:26.23648926 +0000 UTC m=+74.514096641 (2 nodes), 880ns from start of poll, 1.66556504s from cluster start. Timeout at 2026-03-31 06:17:27.24150802 +0000 UTC m=+75.519115401, 1.00501876s after stability 2026-03-31T06:17:27.254Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: aborting pipeline replication: peer="{Voter server-48f913ad-39bc-88e0-7156-8f530c0a31d0 48f913ad-39bc-88e0-7156-8f530c0a31d0}" 2026-03-31T06:17:27.254Z [INFO] server-0dd973e7-df15-9326-3d5e-f0151bedc105: aborting pipeline replication: peer="{Voter server-32f71c9e-2a0e-5da4-829b-7c61ce435bf6 32f71c9e-2a0e-5da4-829b-7c61ce435bf6}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true 2026-03-31T06:17:27.256Z [INFO] server-b04e078b-7e6f-c2f8-51cf-b3f479c47700: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b04e078b-7e6f-c2f8-51cf-b3f479c47700 Address:b04e078b-7e6f-c2f8-51cf-b3f479c47700} {Suffrage:Voter ID:server-b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a Address:b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a} {Suffrage:Voter ID:server-ce49da3b-259e-dcac-37f9-8e3e0df07a5f Address:ce49da3b-259e-dcac-37f9-8e3e0df07a5f}]" 2026-03-31T06:17:27.256Z [INFO] server-b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b04e078b-7e6f-c2f8-51cf-b3f479c47700 Address:b04e078b-7e6f-c2f8-51cf-b3f479c47700} {Suffrage:Voter ID:server-b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a Address:b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a} {Suffrage:Voter ID:server-ce49da3b-259e-dcac-37f9-8e3e0df07a5f Address:ce49da3b-259e-dcac-37f9-8e3e0df07a5f}]" 2026-03-31T06:17:27.256Z [INFO] server-b04e078b-7e6f-c2f8-51cf-b3f479c47700: entering follower state: follower="Node at b04e078b-7e6f-c2f8-51cf-b3f479c47700 [Follower]" leader-address= leader-id= 2026-03-31T06:17:27.256Z [INFO] server-b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a: entering follower state: follower="Node at b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a [Follower]" leader-address= leader-id= 2026-03-31T06:17:27.256Z [INFO] server-ce49da3b-259e-dcac-37f9-8e3e0df07a5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b04e078b-7e6f-c2f8-51cf-b3f479c47700 Address:b04e078b-7e6f-c2f8-51cf-b3f479c47700} {Suffrage:Voter ID:server-b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a Address:b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a} {Suffrage:Voter ID:server-ce49da3b-259e-dcac-37f9-8e3e0df07a5f Address:ce49da3b-259e-dcac-37f9-8e3e0df07a5f}]" 2026-03-31T06:17:27.256Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:27.256Z [INFO] server-ce49da3b-259e-dcac-37f9-8e3e0df07a5f: entering follower state: follower="Node at ce49da3b-259e-dcac-37f9-8e3e0df07a5f [Follower]" leader-address= leader-id= 2026-03-31T06:17:27.813Z [WARN] server-b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:27.813Z [INFO] server-b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a: entering candidate state: node="Node at b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a [Candidate]" term=2 2026-03-31T06:17:27.813Z [INFO] server-b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a: election won: term=2 tally=2 2026-03-31T06:17:27.814Z [INFO] server-b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a: entering leader state: leader="Node at b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a [Leader]" 2026-03-31T06:17:27.814Z [INFO] server-b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a: added peer, starting replication: peer=server-b04e078b-7e6f-c2f8-51cf-b3f479c47700 2026-03-31T06:17:27.814Z [INFO] server-b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a: added peer, starting replication: peer=server-ce49da3b-259e-dcac-37f9-8e3e0df07a5f 2026-03-31T06:17:27.814Z [INFO] server-b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a: pipelining replication: peer="{Voter server-ce49da3b-259e-dcac-37f9-8e3e0df07a5f ce49da3b-259e-dcac-37f9-8e3e0df07a5f}" 2026-03-31T06:17:27.814Z [INFO] server-b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a: pipelining replication: peer="{Voter server-b04e078b-7e6f-c2f8-51cf-b3f479c47700 b04e078b-7e6f-c2f8-51cf-b3f479c47700}" 2026-03-31T06:17:28.819Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:27.8140397 +0000 UTC m=+76.091647081 (1 nodes), 557.54036ms from start of poll, 558.1398ms from cluster start. Timeout at 2026-03-31 06:17:28.81904298 +0000 UTC m=+77.096650401, 1.00500332s after stability 2026-03-31T06:17:28.820Z [INFO] server-b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a: starting snapshot up to: index=12 2026-03-31T06:17:28.820Z [INFO] creating new snapshot: path=/tmp/raft1723467927/snapshots/2-12-1774937848820.tmp 2026-03-31T06:17:28.820Z [INFO] server-b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a: snapshot complete up to: index=12 2026-03-31T06:17:28.821Z [INFO] server-5e78dc5d-334c-dfb1-a343-2cf89b7999c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e78dc5d-334c-dfb1-a343-2cf89b7999c1 Address:5e78dc5d-334c-dfb1-a343-2cf89b7999c1} {Suffrage:Voter ID:server-de9ce7ab-af71-0f4c-032f-1ed2f0d097f0 Address:de9ce7ab-af71-0f4c-032f-1ed2f0d097f0} {Suffrage:Voter ID:server-e5ab2f62-b4b7-26a5-2bbc-f96793927052 Address:e5ab2f62-b4b7-26a5-2bbc-f96793927052}]" 2026-03-31T06:17:28.821Z [INFO] server-5e78dc5d-334c-dfb1-a343-2cf89b7999c1: entering follower state: follower="Node at 5e78dc5d-334c-dfb1-a343-2cf89b7999c1 [Follower]" leader-address= leader-id= 2026-03-31T06:17:28.821Z [INFO] server-de9ce7ab-af71-0f4c-032f-1ed2f0d097f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e78dc5d-334c-dfb1-a343-2cf89b7999c1 Address:5e78dc5d-334c-dfb1-a343-2cf89b7999c1} {Suffrage:Voter ID:server-de9ce7ab-af71-0f4c-032f-1ed2f0d097f0 Address:de9ce7ab-af71-0f4c-032f-1ed2f0d097f0} {Suffrage:Voter ID:server-e5ab2f62-b4b7-26a5-2bbc-f96793927052 Address:e5ab2f62-b4b7-26a5-2bbc-f96793927052}]" 2026-03-31T06:17:28.821Z [INFO] server-de9ce7ab-af71-0f4c-032f-1ed2f0d097f0: entering follower state: follower="Node at de9ce7ab-af71-0f4c-032f-1ed2f0d097f0 [Follower]" leader-address= leader-id= 2026-03-31T06:17:28.821Z [INFO] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e78dc5d-334c-dfb1-a343-2cf89b7999c1 Address:5e78dc5d-334c-dfb1-a343-2cf89b7999c1} {Suffrage:Voter ID:server-de9ce7ab-af71-0f4c-032f-1ed2f0d097f0 Address:de9ce7ab-af71-0f4c-032f-1ed2f0d097f0} {Suffrage:Voter ID:server-e5ab2f62-b4b7-26a5-2bbc-f96793927052 Address:e5ab2f62-b4b7-26a5-2bbc-f96793927052}]" 2026-03-31T06:17:28.821Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:28.822Z [INFO] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: entering follower state: follower="Node at e5ab2f62-b4b7-26a5-2bbc-f96793927052 [Follower]" leader-address= leader-id= 2026-03-31T06:17:29.425Z [WARN] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:29.425Z [INFO] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: entering candidate state: node="Node at e5ab2f62-b4b7-26a5-2bbc-f96793927052 [Candidate]" term=2 2026-03-31T06:17:29.425Z [INFO] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: election won: term=2 tally=2 2026-03-31T06:17:29.425Z [INFO] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: entering leader state: leader="Node at e5ab2f62-b4b7-26a5-2bbc-f96793927052 [Leader]" 2026-03-31T06:17:29.425Z [INFO] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: added peer, starting replication: peer=server-5e78dc5d-334c-dfb1-a343-2cf89b7999c1 2026-03-31T06:17:29.425Z [INFO] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: added peer, starting replication: peer=server-de9ce7ab-af71-0f4c-032f-1ed2f0d097f0 2026-03-31T06:17:29.425Z [INFO] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: pipelining replication: peer="{Voter server-5e78dc5d-334c-dfb1-a343-2cf89b7999c1 5e78dc5d-334c-dfb1-a343-2cf89b7999c1}" 2026-03-31T06:17:29.425Z [INFO] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: pipelining replication: peer="{Voter server-de9ce7ab-af71-0f4c-032f-1ed2f0d097f0 de9ce7ab-af71-0f4c-032f-1ed2f0d097f0}" 2026-03-31T06:17:30.430Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:29.42530562 +0000 UTC m=+77.702913001 (1 nodes), 603.31344ms from start of poll, 603.75548ms from cluster start. Timeout at 2026-03-31 06:17:30.43030962 +0000 UTC m=+78.707917041, 1.00500404s after stability 2026-03-31T06:17:30.431Z [INFO] creating new snapshot: path=/tmp/raft1546839382/snapshots/2-1013-1774937850431.tmp 2026-03-31T06:17:30.431Z [INFO] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: copied to local snapshot: bytes=71 2026-03-31T06:17:30.431Z [INFO] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: snapshot restore progress: id=2-1013-1774937850431 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:30.431Z [INFO] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: restored user snapshot: index=1013 2026-03-31T06:17:30.431Z [ERROR] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: failed to get log: index=3 error="log not found" 2026-03-31T06:17:30.431Z [INFO] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: aborting pipeline replication: peer="{Voter server-de9ce7ab-af71-0f4c-032f-1ed2f0d097f0 de9ce7ab-af71-0f4c-032f-1ed2f0d097f0}" 2026-03-31T06:17:30.431Z [ERROR] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: failed to get log: index=3 error="log not found" 2026-03-31T06:17:30.431Z [INFO] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: aborting pipeline replication: peer="{Voter server-5e78dc5d-334c-dfb1-a343-2cf89b7999c1 5e78dc5d-334c-dfb1-a343-2cf89b7999c1}" 2026-03-31T06:17:30.437Z [ERROR] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: failed to get log: index=3 error="log not found" 2026-03-31T06:17:30.438Z [INFO] creating new snapshot: path=/tmp/raft1406951030/snapshots/2-1013-1774937850438.tmp 2026-03-31T06:17:30.438Z [INFO] server-de9ce7ab-af71-0f4c-032f-1ed2f0d097f0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:30.438Z [INFO] server-de9ce7ab-af71-0f4c-032f-1ed2f0d097f0: copied to local snapshot: bytes=71 2026-03-31T06:17:30.439Z [INFO] server-de9ce7ab-af71-0f4c-032f-1ed2f0d097f0: snapshot restore progress: id=2-1013-1774937850438 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:30.439Z [INFO] server-de9ce7ab-af71-0f4c-032f-1ed2f0d097f0: Installed remote snapshot 2026-03-31T06:17:30.439Z [INFO] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: pipelining replication: peer="{Voter server-de9ce7ab-af71-0f4c-032f-1ed2f0d097f0 de9ce7ab-af71-0f4c-032f-1ed2f0d097f0}" 2026-03-31T06:17:30.439Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:30.441Z [ERROR] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: failed to get log: index=3 error="log not found" 2026-03-31T06:17:30.441Z [INFO] creating new snapshot: path=/tmp/raft93875238/snapshots/2-1013-1774937850441.tmp 2026-03-31T06:17:30.442Z [INFO] server-5e78dc5d-334c-dfb1-a343-2cf89b7999c1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:30.442Z [INFO] server-5e78dc5d-334c-dfb1-a343-2cf89b7999c1: copied to local snapshot: bytes=71 2026-03-31T06:17:30.442Z [INFO] server-5e78dc5d-334c-dfb1-a343-2cf89b7999c1: snapshot restore progress: id=2-1013-1774937850441 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:30.442Z [INFO] server-5e78dc5d-334c-dfb1-a343-2cf89b7999c1: Installed remote snapshot 2026-03-31T06:17:30.442Z [INFO] server-e5ab2f62-b4b7-26a5-2bbc-f96793927052: pipelining replication: peer="{Voter server-5e78dc5d-334c-dfb1-a343-2cf89b7999c1 5e78dc5d-334c-dfb1-a343-2cf89b7999c1}" 2026-03-31T06:17:31.445Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:30.43943306 +0000 UTC m=+78.717040441 (2 nodes), 920ns from start of poll, 1.61788292s from cluster start. Timeout at 2026-03-31 06:17:31.44444586 +0000 UTC m=+79.722053241, 1.0050128s after stability 2026-03-31T06:17:31.456Z [INFO] server-b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a: aborting pipeline replication: peer="{Voter server-b04e078b-7e6f-c2f8-51cf-b3f479c47700 b04e078b-7e6f-c2f8-51cf-b3f479c47700}" 2026-03-31T06:17:31.456Z [INFO] server-b0ccbb7f-42ac-62c4-c57c-c184fc16bc2a: aborting pipeline replication: peer="{Voter server-ce49da3b-259e-dcac-37f9-8e3e0df07a5f ce49da3b-259e-dcac-37f9-8e3e0df07a5f}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true 2026-03-31T06:17:31.457Z [INFO] server-95d8e299-d28d-9fbc-2197-0a286385dd9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95d8e299-d28d-9fbc-2197-0a286385dd9c Address:95d8e299-d28d-9fbc-2197-0a286385dd9c} {Suffrage:Voter ID:server-63e6bab9-a3fb-f266-d4a2-b02f380b07d5 Address:63e6bab9-a3fb-f266-d4a2-b02f380b07d5} {Suffrage:Voter ID:server-06bf44c7-7190-e36e-350b-48fa5debf5b9 Address:06bf44c7-7190-e36e-350b-48fa5debf5b9}]" 2026-03-31T06:17:31.457Z [INFO] server-63e6bab9-a3fb-f266-d4a2-b02f380b07d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95d8e299-d28d-9fbc-2197-0a286385dd9c Address:95d8e299-d28d-9fbc-2197-0a286385dd9c} {Suffrage:Voter ID:server-63e6bab9-a3fb-f266-d4a2-b02f380b07d5 Address:63e6bab9-a3fb-f266-d4a2-b02f380b07d5} {Suffrage:Voter ID:server-06bf44c7-7190-e36e-350b-48fa5debf5b9 Address:06bf44c7-7190-e36e-350b-48fa5debf5b9}]" 2026-03-31T06:17:31.457Z [INFO] server-95d8e299-d28d-9fbc-2197-0a286385dd9c: entering follower state: follower="Node at 95d8e299-d28d-9fbc-2197-0a286385dd9c [Follower]" leader-address= leader-id= 2026-03-31T06:17:31.457Z [INFO] server-63e6bab9-a3fb-f266-d4a2-b02f380b07d5: entering follower state: follower="Node at 63e6bab9-a3fb-f266-d4a2-b02f380b07d5 [Follower]" leader-address= leader-id= 2026-03-31T06:17:31.457Z [INFO] server-06bf44c7-7190-e36e-350b-48fa5debf5b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95d8e299-d28d-9fbc-2197-0a286385dd9c Address:95d8e299-d28d-9fbc-2197-0a286385dd9c} {Suffrage:Voter ID:server-63e6bab9-a3fb-f266-d4a2-b02f380b07d5 Address:63e6bab9-a3fb-f266-d4a2-b02f380b07d5} {Suffrage:Voter ID:server-06bf44c7-7190-e36e-350b-48fa5debf5b9 Address:06bf44c7-7190-e36e-350b-48fa5debf5b9}]" 2026-03-31T06:17:31.457Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:31.457Z [INFO] server-06bf44c7-7190-e36e-350b-48fa5debf5b9: entering follower state: follower="Node at 06bf44c7-7190-e36e-350b-48fa5debf5b9 [Follower]" leader-address= leader-id= 2026-03-31T06:17:32.186Z [WARN] server-06bf44c7-7190-e36e-350b-48fa5debf5b9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:32.186Z [INFO] server-06bf44c7-7190-e36e-350b-48fa5debf5b9: entering candidate state: node="Node at 06bf44c7-7190-e36e-350b-48fa5debf5b9 [Candidate]" term=2 2026-03-31T06:17:32.186Z [INFO] server-06bf44c7-7190-e36e-350b-48fa5debf5b9: election won: term=2 tally=2 2026-03-31T06:17:32.186Z [INFO] server-06bf44c7-7190-e36e-350b-48fa5debf5b9: entering leader state: leader="Node at 06bf44c7-7190-e36e-350b-48fa5debf5b9 [Leader]" 2026-03-31T06:17:32.186Z [INFO] server-06bf44c7-7190-e36e-350b-48fa5debf5b9: added peer, starting replication: peer=server-95d8e299-d28d-9fbc-2197-0a286385dd9c 2026-03-31T06:17:32.186Z [INFO] server-06bf44c7-7190-e36e-350b-48fa5debf5b9: added peer, starting replication: peer=server-63e6bab9-a3fb-f266-d4a2-b02f380b07d5 2026-03-31T06:17:32.186Z [INFO] server-06bf44c7-7190-e36e-350b-48fa5debf5b9: pipelining replication: peer="{Voter server-63e6bab9-a3fb-f266-d4a2-b02f380b07d5 63e6bab9-a3fb-f266-d4a2-b02f380b07d5}" 2026-03-31T06:17:32.186Z [INFO] server-06bf44c7-7190-e36e-350b-48fa5debf5b9: pipelining replication: peer="{Voter server-95d8e299-d28d-9fbc-2197-0a286385dd9c 95d8e299-d28d-9fbc-2197-0a286385dd9c}" 2026-03-31T06:17:33.192Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:32.18671458 +0000 UTC m=+80.464321961 (1 nodes), 729.15724ms from start of poll, 729.59016ms from cluster start. Timeout at 2026-03-31 06:17:33.1917381 +0000 UTC m=+81.469345481, 1.00502352s after stability 2026-03-31T06:17:33.192Z [INFO] server-06bf44c7-7190-e36e-350b-48fa5debf5b9: starting snapshot up to: index=12 2026-03-31T06:17:33.192Z [INFO] creating new snapshot: path=/tmp/raft901330092/snapshots/2-12-1774937853192.tmp 2026-03-31T06:17:33.193Z [INFO] server-06bf44c7-7190-e36e-350b-48fa5debf5b9: snapshot complete up to: index=12 2026-03-31T06:17:33.194Z [INFO] server-138684a6-3969-fa11-0ec2-b6ae697f77de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138684a6-3969-fa11-0ec2-b6ae697f77de Address:138684a6-3969-fa11-0ec2-b6ae697f77de} {Suffrage:Voter ID:server-27a15905-12a1-0071-aec6-5346e06f87f9 Address:27a15905-12a1-0071-aec6-5346e06f87f9} {Suffrage:Voter ID:server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5 Address:8b4ad946-840c-fba6-e77c-d74b2a44e3c5}]" 2026-03-31T06:17:33.194Z [INFO] server-27a15905-12a1-0071-aec6-5346e06f87f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138684a6-3969-fa11-0ec2-b6ae697f77de Address:138684a6-3969-fa11-0ec2-b6ae697f77de} {Suffrage:Voter ID:server-27a15905-12a1-0071-aec6-5346e06f87f9 Address:27a15905-12a1-0071-aec6-5346e06f87f9} {Suffrage:Voter ID:server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5 Address:8b4ad946-840c-fba6-e77c-d74b2a44e3c5}]" 2026-03-31T06:17:33.194Z [INFO] server-138684a6-3969-fa11-0ec2-b6ae697f77de: entering follower state: follower="Node at 138684a6-3969-fa11-0ec2-b6ae697f77de [Follower]" leader-address= leader-id= 2026-03-31T06:17:33.194Z [INFO] server-27a15905-12a1-0071-aec6-5346e06f87f9: entering follower state: follower="Node at 27a15905-12a1-0071-aec6-5346e06f87f9 [Follower]" leader-address= leader-id= 2026-03-31T06:17:33.194Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138684a6-3969-fa11-0ec2-b6ae697f77de Address:138684a6-3969-fa11-0ec2-b6ae697f77de} {Suffrage:Voter ID:server-27a15905-12a1-0071-aec6-5346e06f87f9 Address:27a15905-12a1-0071-aec6-5346e06f87f9} {Suffrage:Voter ID:server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5 Address:8b4ad946-840c-fba6-e77c-d74b2a44e3c5}]" 2026-03-31T06:17:33.194Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:33.194Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: entering follower state: follower="Node at 8b4ad946-840c-fba6-e77c-d74b2a44e3c5 [Follower]" leader-address= leader-id= 2026-03-31T06:17:33.805Z [WARN] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:33.805Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: entering candidate state: node="Node at 8b4ad946-840c-fba6-e77c-d74b2a44e3c5 [Candidate]" term=2 2026-03-31T06:17:33.805Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: election won: term=2 tally=2 2026-03-31T06:17:33.805Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: entering leader state: leader="Node at 8b4ad946-840c-fba6-e77c-d74b2a44e3c5 [Leader]" 2026-03-31T06:17:33.806Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: added peer, starting replication: peer=server-138684a6-3969-fa11-0ec2-b6ae697f77de 2026-03-31T06:17:33.806Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: added peer, starting replication: peer=server-27a15905-12a1-0071-aec6-5346e06f87f9 2026-03-31T06:17:33.806Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: pipelining replication: peer="{Voter server-138684a6-3969-fa11-0ec2-b6ae697f77de 138684a6-3969-fa11-0ec2-b6ae697f77de}" 2026-03-31T06:17:33.806Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: pipelining replication: peer="{Voter server-27a15905-12a1-0071-aec6-5346e06f87f9 27a15905-12a1-0071-aec6-5346e06f87f9}" 2026-03-31T06:17:34.811Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:33.80607538 +0000 UTC m=+82.083682801 (1 nodes), 611.34296ms from start of poll, 611.75772ms from cluster start. Timeout at 2026-03-31 06:17:34.81107898 +0000 UTC m=+83.088686361, 1.00500356s after stability 2026-03-31T06:17:34.811Z [INFO] creating new snapshot: path=/tmp/raft86392860/snapshots/2-1013-1774937854811.tmp 2026-03-31T06:17:34.812Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: copied to local snapshot: bytes=71 2026-03-31T06:17:34.812Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: snapshot restore progress: id=2-1013-1774937854811 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:34.812Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: removing all old logs from log store 2026-03-31T06:17:34.812Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: compacting logs: from=1 to=2 2026-03-31T06:17:34.812Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: restored user snapshot: index=1013 2026-03-31T06:17:34.812Z [ERROR] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: failed to get log: index=2 error="log not found" 2026-03-31T06:17:34.812Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: aborting pipeline replication: peer="{Voter server-27a15905-12a1-0071-aec6-5346e06f87f9 27a15905-12a1-0071-aec6-5346e06f87f9}" 2026-03-31T06:17:34.812Z [ERROR] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: failed to get log: index=2 error="log not found" 2026-03-31T06:17:34.812Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: aborting pipeline replication: peer="{Voter server-138684a6-3969-fa11-0ec2-b6ae697f77de 138684a6-3969-fa11-0ec2-b6ae697f77de}" 2026-03-31T06:17:34.818Z [ERROR] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: failed to get log: index=2 error="log not found" 2026-03-31T06:17:34.818Z [INFO] creating new snapshot: path=/tmp/raft1839776734/snapshots/2-1013-1774937854818.tmp 2026-03-31T06:17:34.818Z [INFO] server-138684a6-3969-fa11-0ec2-b6ae697f77de: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:34.819Z [INFO] server-138684a6-3969-fa11-0ec2-b6ae697f77de: copied to local snapshot: bytes=71 2026-03-31T06:17:34.819Z [ERROR] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: failed to get log: index=2 error="log not found" 2026-03-31T06:17:34.819Z [INFO] server-138684a6-3969-fa11-0ec2-b6ae697f77de: snapshot restore progress: id=2-1013-1774937854818 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:34.819Z [INFO] server-138684a6-3969-fa11-0ec2-b6ae697f77de: removing all old logs from log store 2026-03-31T06:17:34.819Z [INFO] server-138684a6-3969-fa11-0ec2-b6ae697f77de: compacting logs: from=1 to=2 2026-03-31T06:17:34.819Z [INFO] server-138684a6-3969-fa11-0ec2-b6ae697f77de: Installed remote snapshot 2026-03-31T06:17:34.819Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: pipelining replication: peer="{Voter server-138684a6-3969-fa11-0ec2-b6ae697f77de 138684a6-3969-fa11-0ec2-b6ae697f77de}" 2026-03-31T06:17:34.819Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:34.819Z [INFO] creating new snapshot: path=/tmp/raft1188761005/snapshots/2-1013-1774937854819.tmp 2026-03-31T06:17:34.819Z [INFO] server-27a15905-12a1-0071-aec6-5346e06f87f9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:34.820Z [INFO] server-27a15905-12a1-0071-aec6-5346e06f87f9: copied to local snapshot: bytes=71 2026-03-31T06:17:34.820Z [INFO] server-27a15905-12a1-0071-aec6-5346e06f87f9: snapshot restore progress: id=2-1013-1774937854819 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:34.820Z [INFO] server-27a15905-12a1-0071-aec6-5346e06f87f9: removing all old logs from log store 2026-03-31T06:17:34.820Z [INFO] server-27a15905-12a1-0071-aec6-5346e06f87f9: compacting logs: from=1 to=2 2026-03-31T06:17:34.820Z [INFO] server-27a15905-12a1-0071-aec6-5346e06f87f9: Installed remote snapshot 2026-03-31T06:17:34.820Z [INFO] server-8b4ad946-840c-fba6-e77c-d74b2a44e3c5: pipelining replication: peer="{Voter server-27a15905-12a1-0071-aec6-5346e06f87f9 27a15905-12a1-0071-aec6-5346e06f87f9}" 2026-03-31T06:17:35.825Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:34.8197535 +0000 UTC m=+83.097360881 (2 nodes), 880ns from start of poll, 1.6254358s from cluster start. Timeout at 2026-03-31 06:17:35.8247813 +0000 UTC m=+84.102388681, 1.0050278s after stability 2026-03-31T06:17:35.841Z [INFO] server-06bf44c7-7190-e36e-350b-48fa5debf5b9: aborting pipeline replication: peer="{Voter server-63e6bab9-a3fb-f266-d4a2-b02f380b07d5 63e6bab9-a3fb-f266-d4a2-b02f380b07d5}" 2026-03-31T06:17:35.841Z [INFO] server-06bf44c7-7190-e36e-350b-48fa5debf5b9: aborting pipeline replication: peer="{Voter server-95d8e299-d28d-9fbc-2197-0a286385dd9c 95d8e299-d28d-9fbc-2197-0a286385dd9c}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false 2026-03-31T06:17:35.842Z [INFO] server-662a5961-f839-31f5-77fa-5808ccf8d8c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-662a5961-f839-31f5-77fa-5808ccf8d8c3 Address:662a5961-f839-31f5-77fa-5808ccf8d8c3} {Suffrage:Voter ID:server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8 Address:9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8} {Suffrage:Voter ID:server-6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb Address:6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb}]" 2026-03-31T06:17:35.842Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-662a5961-f839-31f5-77fa-5808ccf8d8c3 Address:662a5961-f839-31f5-77fa-5808ccf8d8c3} {Suffrage:Voter ID:server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8 Address:9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8} {Suffrage:Voter ID:server-6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb Address:6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb}]" 2026-03-31T06:17:35.842Z [INFO] server-662a5961-f839-31f5-77fa-5808ccf8d8c3: entering follower state: follower="Node at 662a5961-f839-31f5-77fa-5808ccf8d8c3 [Follower]" leader-address= leader-id= 2026-03-31T06:17:35.842Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: entering follower state: follower="Node at 9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8 [Follower]" leader-address= leader-id= 2026-03-31T06:17:35.842Z [INFO] server-6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-662a5961-f839-31f5-77fa-5808ccf8d8c3 Address:662a5961-f839-31f5-77fa-5808ccf8d8c3} {Suffrage:Voter ID:server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8 Address:9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8} {Suffrage:Voter ID:server-6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb Address:6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb}]" 2026-03-31T06:17:35.842Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:35.842Z [INFO] server-6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb: entering follower state: follower="Node at 6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb [Follower]" leader-address= leader-id= 2026-03-31T06:17:36.347Z [WARN] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:36.347Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: entering candidate state: node="Node at 9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8 [Candidate]" term=2 2026-03-31T06:17:36.347Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: election won: term=2 tally=2 2026-03-31T06:17:36.347Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: entering leader state: leader="Node at 9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8 [Leader]" 2026-03-31T06:17:36.347Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: added peer, starting replication: peer=server-662a5961-f839-31f5-77fa-5808ccf8d8c3 2026-03-31T06:17:36.347Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: added peer, starting replication: peer=server-6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb 2026-03-31T06:17:36.347Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: pipelining replication: peer="{Voter server-6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb 6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb}" 2026-03-31T06:17:36.347Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: pipelining replication: peer="{Voter server-662a5961-f839-31f5-77fa-5808ccf8d8c3 662a5961-f839-31f5-77fa-5808ccf8d8c3}" 2026-03-31T06:17:37.352Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:36.34744726 +0000 UTC m=+84.625054681 (1 nodes), 504.62996ms from start of poll, 505.04ms from cluster start. Timeout at 2026-03-31 06:17:37.3524549 +0000 UTC m=+85.630062281, 1.0050076s after stability 2026-03-31T06:17:37.353Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: starting snapshot up to: index=12 2026-03-31T06:17:37.353Z [INFO] creating new snapshot: path=/tmp/raft392822834/snapshots/2-12-1774937857353.tmp 2026-03-31T06:17:37.353Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: snapshot complete up to: index=12 2026-03-31T06:17:37.354Z [INFO] creating new snapshot: path=/tmp/raft392822834/snapshots/2-10013-1774937857354.tmp 2026-03-31T06:17:37.355Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: copied to local snapshot: bytes=71 2026-03-31T06:17:37.355Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: snapshot restore progress: id=2-10013-1774937857354 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:37.355Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: restored user snapshot: index=10013 2026-03-31T06:17:37.355Z [ERROR] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: failed to get log: index=23 error="log not found" 2026-03-31T06:17:37.355Z [ERROR] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: failed to get log: index=23 error="log not found" 2026-03-31T06:17:37.355Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: aborting pipeline replication: peer="{Voter server-6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb 6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb}" 2026-03-31T06:17:37.355Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: aborting pipeline replication: peer="{Voter server-662a5961-f839-31f5-77fa-5808ccf8d8c3 662a5961-f839-31f5-77fa-5808ccf8d8c3}" 2026-03-31T06:17:37.362Z [ERROR] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: failed to get log: index=23 error="log not found" 2026-03-31T06:17:37.362Z [INFO] creating new snapshot: path=/tmp/raft2564497719/snapshots/2-10013-1774937857362.tmp 2026-03-31T06:17:37.362Z [INFO] server-662a5961-f839-31f5-77fa-5808ccf8d8c3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:37.363Z [INFO] server-662a5961-f839-31f5-77fa-5808ccf8d8c3: copied to local snapshot: bytes=71 2026-03-31T06:17:37.363Z [ERROR] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: failed to get log: index=23 error="log not found" 2026-03-31T06:17:37.363Z [INFO] server-662a5961-f839-31f5-77fa-5808ccf8d8c3: snapshot restore progress: id=2-10013-1774937857362 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:37.363Z [INFO] server-662a5961-f839-31f5-77fa-5808ccf8d8c3: Installed remote snapshot 2026-03-31T06:17:37.363Z [WARN] server-662a5961-f839-31f5-77fa-5808ccf8d8c3: skipping application of old log: index=22 2026-03-31T06:17:37.363Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: pipelining replication: peer="{Voter server-662a5961-f839-31f5-77fa-5808ccf8d8c3 662a5961-f839-31f5-77fa-5808ccf8d8c3}" 2026-03-31T06:17:37.363Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:37.363Z [INFO] creating new snapshot: path=/tmp/raft288657270/snapshots/2-10013-1774937857363.tmp 2026-03-31T06:17:37.364Z [INFO] server-6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:37.364Z [INFO] server-6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb: copied to local snapshot: bytes=71 2026-03-31T06:17:37.364Z [INFO] server-6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb: snapshot restore progress: id=2-10013-1774937857363 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:37.364Z [INFO] server-6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb: Installed remote snapshot 2026-03-31T06:17:37.364Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: pipelining replication: peer="{Voter server-6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb 6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb}" 2026-03-31T06:17:38.369Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:37.36378762 +0000 UTC m=+85.641395001 (2 nodes), 880ns from start of poll, 1.52138032s from cluster start. Timeout at 2026-03-31 06:17:38.36881086 +0000 UTC m=+86.646418241, 1.00502324s after stability 2026-03-31T06:17:38.380Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: aborting pipeline replication: peer="{Voter server-662a5961-f839-31f5-77fa-5808ccf8d8c3 662a5961-f839-31f5-77fa-5808ccf8d8c3}" 2026-03-31T06:17:38.380Z [INFO] server-9d54cb97-cfc0-9b3a-c60d-edc4f7b5d6c8: aborting pipeline replication: peer="{Voter server-6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb 6e8797f1-8dd7-a675-2e1f-c01a4e03d5fb}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false 2026-03-31T06:17:38.381Z [INFO] server-070fdaaf-c568-b25c-5836-a8617e1a27c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-070fdaaf-c568-b25c-5836-a8617e1a27c5 Address:070fdaaf-c568-b25c-5836-a8617e1a27c5} {Suffrage:Voter ID:server-e77d88a3-122c-4fd3-0110-6d5866fcf60a Address:e77d88a3-122c-4fd3-0110-6d5866fcf60a} {Suffrage:Voter ID:server-38d1d610-9da4-b706-e060-2a2bd571957f Address:38d1d610-9da4-b706-e060-2a2bd571957f}]" 2026-03-31T06:17:38.381Z [INFO] server-e77d88a3-122c-4fd3-0110-6d5866fcf60a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-070fdaaf-c568-b25c-5836-a8617e1a27c5 Address:070fdaaf-c568-b25c-5836-a8617e1a27c5} {Suffrage:Voter ID:server-e77d88a3-122c-4fd3-0110-6d5866fcf60a Address:e77d88a3-122c-4fd3-0110-6d5866fcf60a} {Suffrage:Voter ID:server-38d1d610-9da4-b706-e060-2a2bd571957f Address:38d1d610-9da4-b706-e060-2a2bd571957f}]" 2026-03-31T06:17:38.381Z [INFO] server-070fdaaf-c568-b25c-5836-a8617e1a27c5: entering follower state: follower="Node at 070fdaaf-c568-b25c-5836-a8617e1a27c5 [Follower]" leader-address= leader-id= 2026-03-31T06:17:38.381Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-070fdaaf-c568-b25c-5836-a8617e1a27c5 Address:070fdaaf-c568-b25c-5836-a8617e1a27c5} {Suffrage:Voter ID:server-e77d88a3-122c-4fd3-0110-6d5866fcf60a Address:e77d88a3-122c-4fd3-0110-6d5866fcf60a} {Suffrage:Voter ID:server-38d1d610-9da4-b706-e060-2a2bd571957f Address:38d1d610-9da4-b706-e060-2a2bd571957f}]" 2026-03-31T06:17:38.381Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:38.381Z [INFO] server-e77d88a3-122c-4fd3-0110-6d5866fcf60a: entering follower state: follower="Node at e77d88a3-122c-4fd3-0110-6d5866fcf60a [Follower]" leader-address= leader-id= 2026-03-31T06:17:38.381Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: entering follower state: follower="Node at 38d1d610-9da4-b706-e060-2a2bd571957f [Follower]" leader-address= leader-id= 2026-03-31T06:17:39.089Z [WARN] server-38d1d610-9da4-b706-e060-2a2bd571957f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:39.089Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: entering candidate state: node="Node at 38d1d610-9da4-b706-e060-2a2bd571957f [Candidate]" term=2 2026-03-31T06:17:39.089Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: election won: term=2 tally=2 2026-03-31T06:17:39.089Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: entering leader state: leader="Node at 38d1d610-9da4-b706-e060-2a2bd571957f [Leader]" 2026-03-31T06:17:39.089Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: added peer, starting replication: peer=server-070fdaaf-c568-b25c-5836-a8617e1a27c5 2026-03-31T06:17:39.089Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: added peer, starting replication: peer=server-e77d88a3-122c-4fd3-0110-6d5866fcf60a 2026-03-31T06:17:39.089Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: pipelining replication: peer="{Voter server-070fdaaf-c568-b25c-5836-a8617e1a27c5 070fdaaf-c568-b25c-5836-a8617e1a27c5}" 2026-03-31T06:17:39.089Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: pipelining replication: peer="{Voter server-e77d88a3-122c-4fd3-0110-6d5866fcf60a e77d88a3-122c-4fd3-0110-6d5866fcf60a}" 2026-03-31T06:17:40.095Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:39.0894793 +0000 UTC m=+87.367086681 (1 nodes), 707.68024ms from start of poll, 708.14324ms from cluster start. Timeout at 2026-03-31 06:17:40.09448262 +0000 UTC m=+88.372090001, 1.00500332s after stability 2026-03-31T06:17:40.095Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: starting snapshot up to: index=12 2026-03-31T06:17:40.095Z [INFO] creating new snapshot: path=/tmp/raft3738045731/snapshots/2-12-1774937860095.tmp 2026-03-31T06:17:40.096Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: snapshot complete up to: index=12 2026-03-31T06:17:40.097Z [INFO] creating new snapshot: path=/tmp/raft3738045731/snapshots/2-10013-1774937860097.tmp 2026-03-31T06:17:40.098Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: copied to local snapshot: bytes=71 2026-03-31T06:17:40.099Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: snapshot restore progress: id=2-10013-1774937860097 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:40.099Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: removing all old logs from log store 2026-03-31T06:17:40.099Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: compacting logs: from=1 to=22 2026-03-31T06:17:40.099Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: restored user snapshot: index=10013 2026-03-31T06:17:40.099Z [ERROR] server-38d1d610-9da4-b706-e060-2a2bd571957f: failed to get log: index=22 error="log not found" 2026-03-31T06:17:40.099Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: aborting pipeline replication: peer="{Voter server-e77d88a3-122c-4fd3-0110-6d5866fcf60a e77d88a3-122c-4fd3-0110-6d5866fcf60a}" 2026-03-31T06:17:40.099Z [ERROR] server-38d1d610-9da4-b706-e060-2a2bd571957f: failed to get log: index=22 error="log not found" 2026-03-31T06:17:40.099Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: aborting pipeline replication: peer="{Voter server-070fdaaf-c568-b25c-5836-a8617e1a27c5 070fdaaf-c568-b25c-5836-a8617e1a27c5}" 2026-03-31T06:17:40.106Z [ERROR] server-38d1d610-9da4-b706-e060-2a2bd571957f: failed to get log: index=22 error="log not found" 2026-03-31T06:17:40.106Z [INFO] creating new snapshot: path=/tmp/raft188702885/snapshots/2-10013-1774937860106.tmp 2026-03-31T06:17:40.107Z [INFO] server-e77d88a3-122c-4fd3-0110-6d5866fcf60a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:40.107Z [INFO] server-e77d88a3-122c-4fd3-0110-6d5866fcf60a: copied to local snapshot: bytes=71 2026-03-31T06:17:40.107Z [INFO] server-e77d88a3-122c-4fd3-0110-6d5866fcf60a: snapshot restore progress: id=2-10013-1774937860106 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:40.107Z [INFO] server-e77d88a3-122c-4fd3-0110-6d5866fcf60a: removing all old logs from log store 2026-03-31T06:17:40.107Z [INFO] server-e77d88a3-122c-4fd3-0110-6d5866fcf60a: compacting logs: from=1 to=22 2026-03-31T06:17:40.107Z [INFO] server-e77d88a3-122c-4fd3-0110-6d5866fcf60a: Installed remote snapshot 2026-03-31T06:17:40.107Z [WARN] server-e77d88a3-122c-4fd3-0110-6d5866fcf60a: skipping application of old log: index=22 2026-03-31T06:17:40.107Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: pipelining replication: peer="{Voter server-e77d88a3-122c-4fd3-0110-6d5866fcf60a e77d88a3-122c-4fd3-0110-6d5866fcf60a}" 2026-03-31T06:17:40.107Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:40.109Z [ERROR] server-38d1d610-9da4-b706-e060-2a2bd571957f: failed to get log: index=22 error="log not found" 2026-03-31T06:17:40.110Z [INFO] creating new snapshot: path=/tmp/raft1808133049/snapshots/2-10013-1774937860110.tmp 2026-03-31T06:17:40.110Z [INFO] server-070fdaaf-c568-b25c-5836-a8617e1a27c5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:40.110Z [INFO] server-070fdaaf-c568-b25c-5836-a8617e1a27c5: copied to local snapshot: bytes=71 2026-03-31T06:17:40.110Z [INFO] server-070fdaaf-c568-b25c-5836-a8617e1a27c5: snapshot restore progress: id=2-10013-1774937860110 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:40.110Z [INFO] server-070fdaaf-c568-b25c-5836-a8617e1a27c5: removing all old logs from log store 2026-03-31T06:17:40.110Z [INFO] server-070fdaaf-c568-b25c-5836-a8617e1a27c5: compacting logs: from=1 to=22 2026-03-31T06:17:40.110Z [INFO] server-070fdaaf-c568-b25c-5836-a8617e1a27c5: Installed remote snapshot 2026-03-31T06:17:40.111Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: pipelining replication: peer="{Voter server-070fdaaf-c568-b25c-5836-a8617e1a27c5 070fdaaf-c568-b25c-5836-a8617e1a27c5}" 2026-03-31T06:17:41.113Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:40.10786374 +0000 UTC m=+88.385471121 (2 nodes), 840ns from start of poll, 1.72652768s from cluster start. Timeout at 2026-03-31 06:17:41.11286998 +0000 UTC m=+89.390477361, 1.00500624s after stability 2026-03-31T06:17:41.125Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: aborting pipeline replication: peer="{Voter server-e77d88a3-122c-4fd3-0110-6d5866fcf60a e77d88a3-122c-4fd3-0110-6d5866fcf60a}" 2026-03-31T06:17:41.125Z [INFO] server-38d1d610-9da4-b706-e060-2a2bd571957f: aborting pipeline replication: peer="{Voter server-070fdaaf-c568-b25c-5836-a8617e1a27c5 070fdaaf-c568-b25c-5836-a8617e1a27c5}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true 2026-03-31T06:17:41.126Z [INFO] server-7d198484-1da5-d3e9-67ee-b5e0c7ff966d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d198484-1da5-d3e9-67ee-b5e0c7ff966d Address:7d198484-1da5-d3e9-67ee-b5e0c7ff966d} {Suffrage:Voter ID:server-bcbbbe14-3fe5-5155-f2d2-03a19db7ca51 Address:bcbbbe14-3fe5-5155-f2d2-03a19db7ca51} {Suffrage:Voter ID:server-fb346e13-1906-433b-70f8-b4bc4d163f27 Address:fb346e13-1906-433b-70f8-b4bc4d163f27}]" 2026-03-31T06:17:41.126Z [INFO] server-bcbbbe14-3fe5-5155-f2d2-03a19db7ca51: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d198484-1da5-d3e9-67ee-b5e0c7ff966d Address:7d198484-1da5-d3e9-67ee-b5e0c7ff966d} {Suffrage:Voter ID:server-bcbbbe14-3fe5-5155-f2d2-03a19db7ca51 Address:bcbbbe14-3fe5-5155-f2d2-03a19db7ca51} {Suffrage:Voter ID:server-fb346e13-1906-433b-70f8-b4bc4d163f27 Address:fb346e13-1906-433b-70f8-b4bc4d163f27}]" 2026-03-31T06:17:41.126Z [INFO] server-7d198484-1da5-d3e9-67ee-b5e0c7ff966d: entering follower state: follower="Node at 7d198484-1da5-d3e9-67ee-b5e0c7ff966d [Follower]" leader-address= leader-id= 2026-03-31T06:17:41.127Z [INFO] server-fb346e13-1906-433b-70f8-b4bc4d163f27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d198484-1da5-d3e9-67ee-b5e0c7ff966d Address:7d198484-1da5-d3e9-67ee-b5e0c7ff966d} {Suffrage:Voter ID:server-bcbbbe14-3fe5-5155-f2d2-03a19db7ca51 Address:bcbbbe14-3fe5-5155-f2d2-03a19db7ca51} {Suffrage:Voter ID:server-fb346e13-1906-433b-70f8-b4bc4d163f27 Address:fb346e13-1906-433b-70f8-b4bc4d163f27}]" 2026-03-31T06:17:41.127Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:41.127Z [INFO] server-bcbbbe14-3fe5-5155-f2d2-03a19db7ca51: entering follower state: follower="Node at bcbbbe14-3fe5-5155-f2d2-03a19db7ca51 [Follower]" leader-address= leader-id= 2026-03-31T06:17:41.127Z [INFO] server-fb346e13-1906-433b-70f8-b4bc4d163f27: entering follower state: follower="Node at fb346e13-1906-433b-70f8-b4bc4d163f27 [Follower]" leader-address= leader-id= 2026-03-31T06:17:41.853Z [WARN] server-fb346e13-1906-433b-70f8-b4bc4d163f27: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:41.853Z [INFO] server-fb346e13-1906-433b-70f8-b4bc4d163f27: entering candidate state: node="Node at fb346e13-1906-433b-70f8-b4bc4d163f27 [Candidate]" term=2 2026-03-31T06:17:41.854Z [INFO] server-fb346e13-1906-433b-70f8-b4bc4d163f27: election won: term=2 tally=2 2026-03-31T06:17:41.854Z [INFO] server-fb346e13-1906-433b-70f8-b4bc4d163f27: entering leader state: leader="Node at fb346e13-1906-433b-70f8-b4bc4d163f27 [Leader]" 2026-03-31T06:17:41.854Z [INFO] server-fb346e13-1906-433b-70f8-b4bc4d163f27: added peer, starting replication: peer=server-7d198484-1da5-d3e9-67ee-b5e0c7ff966d 2026-03-31T06:17:41.854Z [INFO] server-fb346e13-1906-433b-70f8-b4bc4d163f27: added peer, starting replication: peer=server-bcbbbe14-3fe5-5155-f2d2-03a19db7ca51 2026-03-31T06:17:41.854Z [INFO] server-fb346e13-1906-433b-70f8-b4bc4d163f27: pipelining replication: peer="{Voter server-bcbbbe14-3fe5-5155-f2d2-03a19db7ca51 bcbbbe14-3fe5-5155-f2d2-03a19db7ca51}" 2026-03-31T06:17:41.854Z [INFO] server-fb346e13-1906-433b-70f8-b4bc4d163f27: pipelining replication: peer="{Voter server-7d198484-1da5-d3e9-67ee-b5e0c7ff966d 7d198484-1da5-d3e9-67ee-b5e0c7ff966d}" 2026-03-31T06:17:42.860Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:41.85419886 +0000 UTC m=+90.131806241 (1 nodes), 727.10584ms from start of poll, 727.5166ms from cluster start. Timeout at 2026-03-31 06:17:42.85920158 +0000 UTC m=+91.136808961, 1.00500272s after stability 2026-03-31T06:17:42.861Z [INFO] server-fb346e13-1906-433b-70f8-b4bc4d163f27: starting snapshot up to: index=12 2026-03-31T06:17:42.861Z [INFO] creating new snapshot: path=/tmp/raft1544705166/snapshots/2-12-1774937862861.tmp 2026-03-31T06:17:42.861Z [INFO] server-fb346e13-1906-433b-70f8-b4bc4d163f27: snapshot complete up to: index=12 2026-03-31T06:17:42.863Z [INFO] server-64677f9e-4fe3-9c57-bb25-edb3b9bc8403: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64677f9e-4fe3-9c57-bb25-edb3b9bc8403 Address:64677f9e-4fe3-9c57-bb25-edb3b9bc8403} {Suffrage:Voter ID:server-2718e9e8-abc3-7046-450d-cd403200c67b Address:2718e9e8-abc3-7046-450d-cd403200c67b} {Suffrage:Voter ID:server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d Address:3bdb1798-ffdc-2974-b381-f6ab4c9c537d}]" 2026-03-31T06:17:42.863Z [INFO] server-64677f9e-4fe3-9c57-bb25-edb3b9bc8403: entering follower state: follower="Node at 64677f9e-4fe3-9c57-bb25-edb3b9bc8403 [Follower]" leader-address= leader-id= 2026-03-31T06:17:42.863Z [INFO] server-2718e9e8-abc3-7046-450d-cd403200c67b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64677f9e-4fe3-9c57-bb25-edb3b9bc8403 Address:64677f9e-4fe3-9c57-bb25-edb3b9bc8403} {Suffrage:Voter ID:server-2718e9e8-abc3-7046-450d-cd403200c67b Address:2718e9e8-abc3-7046-450d-cd403200c67b} {Suffrage:Voter ID:server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d Address:3bdb1798-ffdc-2974-b381-f6ab4c9c537d}]" 2026-03-31T06:17:42.863Z [INFO] server-2718e9e8-abc3-7046-450d-cd403200c67b: entering follower state: follower="Node at 2718e9e8-abc3-7046-450d-cd403200c67b [Follower]" leader-address= leader-id= 2026-03-31T06:17:42.863Z [INFO] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64677f9e-4fe3-9c57-bb25-edb3b9bc8403 Address:64677f9e-4fe3-9c57-bb25-edb3b9bc8403} {Suffrage:Voter ID:server-2718e9e8-abc3-7046-450d-cd403200c67b Address:2718e9e8-abc3-7046-450d-cd403200c67b} {Suffrage:Voter ID:server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d Address:3bdb1798-ffdc-2974-b381-f6ab4c9c537d}]" 2026-03-31T06:17:42.863Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:42.863Z [INFO] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: entering follower state: follower="Node at 3bdb1798-ffdc-2974-b381-f6ab4c9c537d [Follower]" leader-address= leader-id= 2026-03-31T06:17:43.400Z [WARN] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:43.400Z [INFO] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: entering candidate state: node="Node at 3bdb1798-ffdc-2974-b381-f6ab4c9c537d [Candidate]" term=2 2026-03-31T06:17:43.400Z [INFO] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: election won: term=2 tally=2 2026-03-31T06:17:43.400Z [INFO] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: entering leader state: leader="Node at 3bdb1798-ffdc-2974-b381-f6ab4c9c537d [Leader]" 2026-03-31T06:17:43.400Z [INFO] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: added peer, starting replication: peer=server-64677f9e-4fe3-9c57-bb25-edb3b9bc8403 2026-03-31T06:17:43.400Z [INFO] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: added peer, starting replication: peer=server-2718e9e8-abc3-7046-450d-cd403200c67b 2026-03-31T06:17:43.400Z [INFO] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: pipelining replication: peer="{Voter server-64677f9e-4fe3-9c57-bb25-edb3b9bc8403 64677f9e-4fe3-9c57-bb25-edb3b9bc8403}" 2026-03-31T06:17:43.400Z [INFO] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: pipelining replication: peer="{Voter server-2718e9e8-abc3-7046-450d-cd403200c67b 2718e9e8-abc3-7046-450d-cd403200c67b}" 2026-03-31T06:17:44.406Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:43.40056254 +0000 UTC m=+91.678169961 (1 nodes), 536.57456ms from start of poll, 537.047ms from cluster start. Timeout at 2026-03-31 06:17:44.40557166 +0000 UTC m=+92.683179041, 1.00500908s after stability 2026-03-31T06:17:44.406Z [INFO] creating new snapshot: path=/tmp/raft1598952552/snapshots/2-10013-1774937864406.tmp 2026-03-31T06:17:44.406Z [INFO] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: copied to local snapshot: bytes=71 2026-03-31T06:17:44.407Z [INFO] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: snapshot restore progress: id=2-10013-1774937864406 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:44.407Z [INFO] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: restored user snapshot: index=10013 2026-03-31T06:17:44.407Z [ERROR] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: failed to get log: index=3 error="log not found" 2026-03-31T06:17:44.407Z [INFO] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: aborting pipeline replication: peer="{Voter server-2718e9e8-abc3-7046-450d-cd403200c67b 2718e9e8-abc3-7046-450d-cd403200c67b}" 2026-03-31T06:17:44.407Z [ERROR] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: failed to get log: index=3 error="log not found" 2026-03-31T06:17:44.407Z [INFO] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: aborting pipeline replication: peer="{Voter server-64677f9e-4fe3-9c57-bb25-edb3b9bc8403 64677f9e-4fe3-9c57-bb25-edb3b9bc8403}" 2026-03-31T06:17:44.414Z [ERROR] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: failed to get log: index=3 error="log not found" 2026-03-31T06:17:44.414Z [ERROR] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: failed to get log: index=3 error="log not found" 2026-03-31T06:17:44.415Z [INFO] creating new snapshot: path=/tmp/raft4253972539/snapshots/2-10013-1774937864415.tmp 2026-03-31T06:17:44.415Z [INFO] creating new snapshot: path=/tmp/raft1472649542/snapshots/2-10013-1774937864415.tmp 2026-03-31T06:17:44.415Z [INFO] server-64677f9e-4fe3-9c57-bb25-edb3b9bc8403: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:44.415Z [INFO] server-2718e9e8-abc3-7046-450d-cd403200c67b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:44.415Z [INFO] server-64677f9e-4fe3-9c57-bb25-edb3b9bc8403: copied to local snapshot: bytes=71 2026-03-31T06:17:44.415Z [INFO] server-2718e9e8-abc3-7046-450d-cd403200c67b: copied to local snapshot: bytes=71 2026-03-31T06:17:44.415Z [INFO] server-64677f9e-4fe3-9c57-bb25-edb3b9bc8403: snapshot restore progress: id=2-10013-1774937864415 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:44.416Z [INFO] server-64677f9e-4fe3-9c57-bb25-edb3b9bc8403: Installed remote snapshot 2026-03-31T06:17:44.416Z [INFO] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: pipelining replication: peer="{Voter server-64677f9e-4fe3-9c57-bb25-edb3b9bc8403 64677f9e-4fe3-9c57-bb25-edb3b9bc8403}" 2026-03-31T06:17:44.416Z [INFO] server-2718e9e8-abc3-7046-450d-cd403200c67b: snapshot restore progress: id=2-10013-1774937864415 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:44.416Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:44.416Z [INFO] server-2718e9e8-abc3-7046-450d-cd403200c67b: Installed remote snapshot 2026-03-31T06:17:44.416Z [INFO] server-3bdb1798-ffdc-2974-b381-f6ab4c9c537d: pipelining replication: peer="{Voter server-2718e9e8-abc3-7046-450d-cd403200c67b 2718e9e8-abc3-7046-450d-cd403200c67b}" 2026-03-31T06:17:45.422Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:44.41619858 +0000 UTC m=+92.693805961 (2 nodes), 920ns from start of poll, 1.552683s from cluster start. Timeout at 2026-03-31 06:17:45.42122686 +0000 UTC m=+93.698834241, 1.00502828s after stability 2026-03-31T06:17:45.433Z [INFO] server-fb346e13-1906-433b-70f8-b4bc4d163f27: aborting pipeline replication: peer="{Voter server-bcbbbe14-3fe5-5155-f2d2-03a19db7ca51 bcbbbe14-3fe5-5155-f2d2-03a19db7ca51}" 2026-03-31T06:17:45.433Z [INFO] server-fb346e13-1906-433b-70f8-b4bc4d163f27: aborting pipeline replication: peer="{Voter server-7d198484-1da5-d3e9-67ee-b5e0c7ff966d 7d198484-1da5-d3e9-67ee-b5e0c7ff966d}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true 2026-03-31T06:17:45.435Z [INFO] server-7ebe48ce-1c6c-e92c-7179-691658b6c4fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ebe48ce-1c6c-e92c-7179-691658b6c4fd Address:7ebe48ce-1c6c-e92c-7179-691658b6c4fd} {Suffrage:Voter ID:server-f59039a3-393c-9f23-6ff5-3022b4943eed Address:f59039a3-393c-9f23-6ff5-3022b4943eed} {Suffrage:Voter ID:server-a02ac7cc-899e-2ff3-83a3-346f2c92b4d0 Address:a02ac7cc-899e-2ff3-83a3-346f2c92b4d0}]" 2026-03-31T06:17:45.435Z [INFO] server-f59039a3-393c-9f23-6ff5-3022b4943eed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ebe48ce-1c6c-e92c-7179-691658b6c4fd Address:7ebe48ce-1c6c-e92c-7179-691658b6c4fd} {Suffrage:Voter ID:server-f59039a3-393c-9f23-6ff5-3022b4943eed Address:f59039a3-393c-9f23-6ff5-3022b4943eed} {Suffrage:Voter ID:server-a02ac7cc-899e-2ff3-83a3-346f2c92b4d0 Address:a02ac7cc-899e-2ff3-83a3-346f2c92b4d0}]" 2026-03-31T06:17:45.435Z [INFO] server-7ebe48ce-1c6c-e92c-7179-691658b6c4fd: entering follower state: follower="Node at 7ebe48ce-1c6c-e92c-7179-691658b6c4fd [Follower]" leader-address= leader-id= 2026-03-31T06:17:45.436Z [INFO] server-a02ac7cc-899e-2ff3-83a3-346f2c92b4d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ebe48ce-1c6c-e92c-7179-691658b6c4fd Address:7ebe48ce-1c6c-e92c-7179-691658b6c4fd} {Suffrage:Voter ID:server-f59039a3-393c-9f23-6ff5-3022b4943eed Address:f59039a3-393c-9f23-6ff5-3022b4943eed} {Suffrage:Voter ID:server-a02ac7cc-899e-2ff3-83a3-346f2c92b4d0 Address:a02ac7cc-899e-2ff3-83a3-346f2c92b4d0}]" 2026-03-31T06:17:45.436Z [INFO] server-f59039a3-393c-9f23-6ff5-3022b4943eed: entering follower state: follower="Node at f59039a3-393c-9f23-6ff5-3022b4943eed [Follower]" leader-address= leader-id= 2026-03-31T06:17:45.436Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:45.436Z [INFO] server-a02ac7cc-899e-2ff3-83a3-346f2c92b4d0: entering follower state: follower="Node at a02ac7cc-899e-2ff3-83a3-346f2c92b4d0 [Follower]" leader-address= leader-id= 2026-03-31T06:17:46.077Z [WARN] server-a02ac7cc-899e-2ff3-83a3-346f2c92b4d0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:46.077Z [INFO] server-a02ac7cc-899e-2ff3-83a3-346f2c92b4d0: entering candidate state: node="Node at a02ac7cc-899e-2ff3-83a3-346f2c92b4d0 [Candidate]" term=2 2026-03-31T06:17:46.077Z [INFO] server-a02ac7cc-899e-2ff3-83a3-346f2c92b4d0: election won: term=2 tally=2 2026-03-31T06:17:46.077Z [INFO] server-a02ac7cc-899e-2ff3-83a3-346f2c92b4d0: entering leader state: leader="Node at a02ac7cc-899e-2ff3-83a3-346f2c92b4d0 [Leader]" 2026-03-31T06:17:46.077Z [INFO] server-a02ac7cc-899e-2ff3-83a3-346f2c92b4d0: added peer, starting replication: peer=server-7ebe48ce-1c6c-e92c-7179-691658b6c4fd 2026-03-31T06:17:46.077Z [INFO] server-a02ac7cc-899e-2ff3-83a3-346f2c92b4d0: added peer, starting replication: peer=server-f59039a3-393c-9f23-6ff5-3022b4943eed 2026-03-31T06:17:46.077Z [INFO] server-a02ac7cc-899e-2ff3-83a3-346f2c92b4d0: pipelining replication: peer="{Voter server-7ebe48ce-1c6c-e92c-7179-691658b6c4fd 7ebe48ce-1c6c-e92c-7179-691658b6c4fd}" 2026-03-31T06:17:46.077Z [INFO] server-a02ac7cc-899e-2ff3-83a3-346f2c92b4d0: pipelining replication: peer="{Voter server-f59039a3-393c-9f23-6ff5-3022b4943eed f59039a3-393c-9f23-6ff5-3022b4943eed}" 2026-03-31T06:17:47.083Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:46.07746898 +0000 UTC m=+94.355076361 (1 nodes), 641.33208ms from start of poll, 641.8792ms from cluster start. Timeout at 2026-03-31 06:17:47.08248222 +0000 UTC m=+95.360089601, 1.00501324s after stability 2026-03-31T06:17:47.084Z [INFO] server-a02ac7cc-899e-2ff3-83a3-346f2c92b4d0: starting snapshot up to: index=12 2026-03-31T06:17:47.084Z [INFO] creating new snapshot: path=/tmp/raft99171621/snapshots/2-12-1774937867084.tmp 2026-03-31T06:17:47.084Z [INFO] server-a02ac7cc-899e-2ff3-83a3-346f2c92b4d0: snapshot complete up to: index=12 2026-03-31T06:17:47.086Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da2f6390-895e-7c71-7fa1-78a3d93381b5 Address:da2f6390-895e-7c71-7fa1-78a3d93381b5} {Suffrage:Voter ID:server-26ca502a-b3b7-bc56-c689-5c17590b14a6 Address:26ca502a-b3b7-bc56-c689-5c17590b14a6} {Suffrage:Voter ID:server-67081f75-4633-6a81-dcec-133fa7aefac9 Address:67081f75-4633-6a81-dcec-133fa7aefac9}]" 2026-03-31T06:17:47.086Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: entering follower state: follower="Node at da2f6390-895e-7c71-7fa1-78a3d93381b5 [Follower]" leader-address= leader-id= 2026-03-31T06:17:47.086Z [INFO] server-26ca502a-b3b7-bc56-c689-5c17590b14a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da2f6390-895e-7c71-7fa1-78a3d93381b5 Address:da2f6390-895e-7c71-7fa1-78a3d93381b5} {Suffrage:Voter ID:server-26ca502a-b3b7-bc56-c689-5c17590b14a6 Address:26ca502a-b3b7-bc56-c689-5c17590b14a6} {Suffrage:Voter ID:server-67081f75-4633-6a81-dcec-133fa7aefac9 Address:67081f75-4633-6a81-dcec-133fa7aefac9}]" 2026-03-31T06:17:47.087Z [INFO] server-26ca502a-b3b7-bc56-c689-5c17590b14a6: entering follower state: follower="Node at 26ca502a-b3b7-bc56-c689-5c17590b14a6 [Follower]" leader-address= leader-id= 2026-03-31T06:17:47.087Z [INFO] server-67081f75-4633-6a81-dcec-133fa7aefac9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da2f6390-895e-7c71-7fa1-78a3d93381b5 Address:da2f6390-895e-7c71-7fa1-78a3d93381b5} {Suffrage:Voter ID:server-26ca502a-b3b7-bc56-c689-5c17590b14a6 Address:26ca502a-b3b7-bc56-c689-5c17590b14a6} {Suffrage:Voter ID:server-67081f75-4633-6a81-dcec-133fa7aefac9 Address:67081f75-4633-6a81-dcec-133fa7aefac9}]" 2026-03-31T06:17:47.087Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:47.087Z [INFO] server-67081f75-4633-6a81-dcec-133fa7aefac9: entering follower state: follower="Node at 67081f75-4633-6a81-dcec-133fa7aefac9 [Follower]" leader-address= leader-id= 2026-03-31T06:17:47.616Z [WARN] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:47.616Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: entering candidate state: node="Node at da2f6390-895e-7c71-7fa1-78a3d93381b5 [Candidate]" term=2 2026-03-31T06:17:47.616Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: election won: term=2 tally=2 2026-03-31T06:17:47.616Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: entering leader state: leader="Node at da2f6390-895e-7c71-7fa1-78a3d93381b5 [Leader]" 2026-03-31T06:17:47.616Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: added peer, starting replication: peer=server-26ca502a-b3b7-bc56-c689-5c17590b14a6 2026-03-31T06:17:47.616Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: added peer, starting replication: peer=server-67081f75-4633-6a81-dcec-133fa7aefac9 2026-03-31T06:17:47.616Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: pipelining replication: peer="{Voter server-26ca502a-b3b7-bc56-c689-5c17590b14a6 26ca502a-b3b7-bc56-c689-5c17590b14a6}" 2026-03-31T06:17:47.616Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: pipelining replication: peer="{Voter server-67081f75-4633-6a81-dcec-133fa7aefac9 67081f75-4633-6a81-dcec-133fa7aefac9}" 2026-03-31T06:17:48.622Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:47.61659094 +0000 UTC m=+95.894198321 (1 nodes), 528.80156ms from start of poll, 530.26848ms from cluster start. Timeout at 2026-03-31 06:17:48.6216011 +0000 UTC m=+96.899208481, 1.00501016s after stability 2026-03-31T06:17:48.622Z [INFO] creating new snapshot: path=/tmp/raft2797902974/snapshots/2-10013-1774937868622.tmp 2026-03-31T06:17:48.622Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: copied to local snapshot: bytes=71 2026-03-31T06:17:48.623Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: snapshot restore progress: id=2-10013-1774937868622 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:48.623Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: removing all old logs from log store 2026-03-31T06:17:48.623Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: compacting logs: from=1 to=2 2026-03-31T06:17:48.623Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: restored user snapshot: index=10013 2026-03-31T06:17:48.623Z [ERROR] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: failed to get log: index=2 error="log not found" 2026-03-31T06:17:48.623Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: aborting pipeline replication: peer="{Voter server-67081f75-4633-6a81-dcec-133fa7aefac9 67081f75-4633-6a81-dcec-133fa7aefac9}" 2026-03-31T06:17:48.623Z [ERROR] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: failed to get log: index=2 error="log not found" 2026-03-31T06:17:48.623Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: aborting pipeline replication: peer="{Voter server-26ca502a-b3b7-bc56-c689-5c17590b14a6 26ca502a-b3b7-bc56-c689-5c17590b14a6}" 2026-03-31T06:17:48.630Z [ERROR] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: failed to get log: index=2 error="log not found" 2026-03-31T06:17:48.630Z [INFO] creating new snapshot: path=/tmp/raft4274536991/snapshots/2-10013-1774937868630.tmp 2026-03-31T06:17:48.631Z [INFO] server-26ca502a-b3b7-bc56-c689-5c17590b14a6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:48.631Z [INFO] server-26ca502a-b3b7-bc56-c689-5c17590b14a6: copied to local snapshot: bytes=71 2026-03-31T06:17:48.631Z [INFO] server-26ca502a-b3b7-bc56-c689-5c17590b14a6: snapshot restore progress: id=2-10013-1774937868630 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:48.631Z [INFO] server-26ca502a-b3b7-bc56-c689-5c17590b14a6: removing all old logs from log store 2026-03-31T06:17:48.631Z [INFO] server-26ca502a-b3b7-bc56-c689-5c17590b14a6: compacting logs: from=1 to=2 2026-03-31T06:17:48.631Z [INFO] server-26ca502a-b3b7-bc56-c689-5c17590b14a6: Installed remote snapshot 2026-03-31T06:17:48.631Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: pipelining replication: peer="{Voter server-26ca502a-b3b7-bc56-c689-5c17590b14a6 26ca502a-b3b7-bc56-c689-5c17590b14a6}" 2026-03-31T06:17:48.631Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:48.632Z [ERROR] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: failed to get log: index=2 error="log not found" 2026-03-31T06:17:48.633Z [INFO] creating new snapshot: path=/tmp/raft2817878173/snapshots/2-10013-1774937868633.tmp 2026-03-31T06:17:48.633Z [INFO] server-67081f75-4633-6a81-dcec-133fa7aefac9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:48.633Z [INFO] server-67081f75-4633-6a81-dcec-133fa7aefac9: copied to local snapshot: bytes=71 2026-03-31T06:17:48.634Z [INFO] server-67081f75-4633-6a81-dcec-133fa7aefac9: snapshot restore progress: id=2-10013-1774937868633 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-03-31T06:17:48.634Z [INFO] server-67081f75-4633-6a81-dcec-133fa7aefac9: removing all old logs from log store 2026-03-31T06:17:48.634Z [INFO] server-67081f75-4633-6a81-dcec-133fa7aefac9: compacting logs: from=1 to=2 2026-03-31T06:17:48.634Z [INFO] server-67081f75-4633-6a81-dcec-133fa7aefac9: Installed remote snapshot 2026-03-31T06:17:48.634Z [INFO] server-da2f6390-895e-7c71-7fa1-78a3d93381b5: pipelining replication: peer="{Voter server-67081f75-4633-6a81-dcec-133fa7aefac9 67081f75-4633-6a81-dcec-133fa7aefac9}" 2026-03-31T06:17:49.637Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:48.6317991 +0000 UTC m=+96.909406481 (2 nodes), 1µs from start of poll, 1.54547664s from cluster start. Timeout at 2026-03-31 06:17:49.6368217 +0000 UTC m=+97.914429121, 1.00502264s after stability 2026-03-31T06:17:49.649Z [INFO] server-a02ac7cc-899e-2ff3-83a3-346f2c92b4d0: aborting pipeline replication: peer="{Voter server-f59039a3-393c-9f23-6ff5-3022b4943eed f59039a3-393c-9f23-6ff5-3022b4943eed}" 2026-03-31T06:17:49.649Z [INFO] server-a02ac7cc-899e-2ff3-83a3-346f2c92b4d0: aborting pipeline replication: peer="{Voter server-7ebe48ce-1c6c-e92c-7179-691658b6c4fd 7ebe48ce-1c6c-e92c-7179-691658b6c4fd}" --- PASS: TestRaft_UserRestore (83.82s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.55s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.56s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.22s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.66s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.72s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.63s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.45s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.32s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.88s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.61s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.31s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.33s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.58s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.57s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.35s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.30s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.72s) --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.68s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.20s) --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.39s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.54s) --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.75s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.31s) --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.21s) === RUN TestRaft_SendSnapshotFollower 2026-03-31T06:17:49.650Z [INFO] server-a2cbb12f-846a-e8c9-ba36-d3436e519df8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a2cbb12f-846a-e8c9-ba36-d3436e519df8 Address:a2cbb12f-846a-e8c9-ba36-d3436e519df8} {Suffrage:Voter ID:server-8ee4efb1-6c7d-f6f2-353c-07328eb60cbd Address:8ee4efb1-6c7d-f6f2-353c-07328eb60cbd} {Suffrage:Voter ID:server-b45d2b32-2a18-365a-9803-64d02476f42d Address:b45d2b32-2a18-365a-9803-64d02476f42d}]" 2026-03-31T06:17:49.650Z [INFO] server-8ee4efb1-6c7d-f6f2-353c-07328eb60cbd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a2cbb12f-846a-e8c9-ba36-d3436e519df8 Address:a2cbb12f-846a-e8c9-ba36-d3436e519df8} {Suffrage:Voter ID:server-8ee4efb1-6c7d-f6f2-353c-07328eb60cbd Address:8ee4efb1-6c7d-f6f2-353c-07328eb60cbd} {Suffrage:Voter ID:server-b45d2b32-2a18-365a-9803-64d02476f42d Address:b45d2b32-2a18-365a-9803-64d02476f42d}]" 2026-03-31T06:17:49.650Z [INFO] server-a2cbb12f-846a-e8c9-ba36-d3436e519df8: entering follower state: follower="Node at a2cbb12f-846a-e8c9-ba36-d3436e519df8 [Follower]" leader-address= leader-id= 2026-03-31T06:17:49.650Z [INFO] server-8ee4efb1-6c7d-f6f2-353c-07328eb60cbd: entering follower state: follower="Node at 8ee4efb1-6c7d-f6f2-353c-07328eb60cbd [Follower]" leader-address= leader-id= 2026-03-31T06:17:49.650Z [INFO] server-b45d2b32-2a18-365a-9803-64d02476f42d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a2cbb12f-846a-e8c9-ba36-d3436e519df8 Address:a2cbb12f-846a-e8c9-ba36-d3436e519df8} {Suffrage:Voter ID:server-8ee4efb1-6c7d-f6f2-353c-07328eb60cbd Address:8ee4efb1-6c7d-f6f2-353c-07328eb60cbd} {Suffrage:Voter ID:server-b45d2b32-2a18-365a-9803-64d02476f42d Address:b45d2b32-2a18-365a-9803-64d02476f42d}]" 2026-03-31T06:17:49.650Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:49.650Z [INFO] server-b45d2b32-2a18-365a-9803-64d02476f42d: entering follower state: follower="Node at b45d2b32-2a18-365a-9803-64d02476f42d [Follower]" leader-address= leader-id= 2026-03-31T06:17:49.716Z [WARN] server-b45d2b32-2a18-365a-9803-64d02476f42d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:49.716Z [INFO] server-b45d2b32-2a18-365a-9803-64d02476f42d: entering candidate state: node="Node at b45d2b32-2a18-365a-9803-64d02476f42d [Candidate]" term=2 2026-03-31T06:17:49.716Z [INFO] server-b45d2b32-2a18-365a-9803-64d02476f42d: election won: term=2 tally=2 2026-03-31T06:17:49.716Z [INFO] server-b45d2b32-2a18-365a-9803-64d02476f42d: entering leader state: leader="Node at b45d2b32-2a18-365a-9803-64d02476f42d [Leader]" 2026-03-31T06:17:49.716Z [INFO] server-b45d2b32-2a18-365a-9803-64d02476f42d: added peer, starting replication: peer=server-a2cbb12f-846a-e8c9-ba36-d3436e519df8 2026-03-31T06:17:49.716Z [INFO] server-b45d2b32-2a18-365a-9803-64d02476f42d: added peer, starting replication: peer=server-8ee4efb1-6c7d-f6f2-353c-07328eb60cbd 2026-03-31T06:17:49.716Z [INFO] server-b45d2b32-2a18-365a-9803-64d02476f42d: pipelining replication: peer="{Voter server-a2cbb12f-846a-e8c9-ba36-d3436e519df8 a2cbb12f-846a-e8c9-ba36-d3436e519df8}" 2026-03-31T06:17:49.716Z [INFO] server-b45d2b32-2a18-365a-9803-64d02476f42d: pipelining replication: peer="{Voter server-8ee4efb1-6c7d-f6f2-353c-07328eb60cbd 8ee4efb1-6c7d-f6f2-353c-07328eb60cbd}" 2026-03-31T06:17:49.822Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:49.71663382 +0000 UTC m=+97.994241201 (2 nodes), 66.04228ms from start of poll, 66.4384ms from cluster start. Timeout at 2026-03-31 06:17:49.8216785 +0000 UTC m=+98.099285881, 105.04468ms after stability 2026-03-31T06:17:49.822Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:49.928Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:49.8226477 +0000 UTC m=+98.100255081 (1 nodes), 1.88µs from start of poll, 172.45228ms from cluster start. Timeout at 2026-03-31 06:17:49.92773646 +0000 UTC m=+98.205343961, 105.08888ms after stability 2026-03-31T06:17:49.928Z [ERROR] server-b45d2b32-2a18-365a-9803-64d02476f42d: failed to pipeline appendEntries: peer="{Voter server-a2cbb12f-846a-e8c9-ba36-d3436e519df8 a2cbb12f-846a-e8c9-ba36-d3436e519df8}" error="append pipeline closed" 2026-03-31T06:17:49.928Z [INFO] server-b45d2b32-2a18-365a-9803-64d02476f42d: aborting pipeline replication: peer="{Voter server-a2cbb12f-846a-e8c9-ba36-d3436e519df8 a2cbb12f-846a-e8c9-ba36-d3436e519df8}" 2026-03-31T06:17:49.928Z [ERROR] server-b45d2b32-2a18-365a-9803-64d02476f42d: failed to appendEntries to: peer="{Voter server-a2cbb12f-846a-e8c9-ba36-d3436e519df8 a2cbb12f-846a-e8c9-ba36-d3436e519df8}" error="failed to connect to peer: a2cbb12f-846a-e8c9-ba36-d3436e519df8" 2026-03-31T06:17:49.929Z [ERROR] server-b45d2b32-2a18-365a-9803-64d02476f42d: failed to heartbeat to: peer=a2cbb12f-846a-e8c9-ba36-d3436e519df8 backoff time=10ms error="failed to connect to peer: a2cbb12f-846a-e8c9-ba36-d3436e519df8" raft_test.go:1627: [INFO] Finished apply without behind follower 2026-03-31T06:17:49.932Z [ERROR] server-a2cbb12f-846a-e8c9-ba36-d3436e519df8: failed to take snapshot: error="nothing new to snapshot" 2026-03-31T06:17:49.932Z [INFO] server-8ee4efb1-6c7d-f6f2-353c-07328eb60cbd: starting snapshot up to: index=98 2026-03-31T06:17:49.932Z [INFO] creating new snapshot: path=/tmp/raft1606399254/snapshots/2-98-1774937869932.tmp 2026-03-31T06:17:49.933Z [INFO] server-8ee4efb1-6c7d-f6f2-353c-07328eb60cbd: compacting logs: from=1 to=92 2026-03-31T06:17:49.933Z [INFO] server-8ee4efb1-6c7d-f6f2-353c-07328eb60cbd: snapshot complete up to: index=98 2026-03-31T06:17:49.933Z [INFO] server-b45d2b32-2a18-365a-9803-64d02476f42d: starting snapshot up to: index=102 2026-03-31T06:17:49.933Z [INFO] creating new snapshot: path=/tmp/raft3252572363/snapshots/2-102-1774937869933.tmp 2026-03-31T06:17:49.933Z [INFO] server-b45d2b32-2a18-365a-9803-64d02476f42d: compacting logs: from=1 to=92 2026-03-31T06:17:49.933Z [INFO] server-b45d2b32-2a18-365a-9803-64d02476f42d: snapshot complete up to: index=102 2026-03-31T06:17:49.939Z [ERROR] server-b45d2b32-2a18-365a-9803-64d02476f42d: failed to get log: index=2 error="log not found" 2026-03-31T06:17:49.940Z [INFO] creating new snapshot: path=/tmp/raft2753935963/snapshots/2-102-1774937869940.tmp 2026-03-31T06:17:49.940Z [INFO] server-a2cbb12f-846a-e8c9-ba36-d3436e519df8: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2026-03-31T06:17:49.941Z [INFO] server-a2cbb12f-846a-e8c9-ba36-d3436e519df8: copied to local snapshot: bytes=693 2026-03-31T06:17:49.941Z [INFO] server-a2cbb12f-846a-e8c9-ba36-d3436e519df8: snapshot restore progress: id=2-102-1774937869940 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2026-03-31T06:17:49.941Z [INFO] server-a2cbb12f-846a-e8c9-ba36-d3436e519df8: Installed remote snapshot 2026-03-31T06:17:49.941Z [WARN] server-a2cbb12f-846a-e8c9-ba36-d3436e519df8: skipping application of old log: index=102 2026-03-31T06:17:49.941Z [INFO] server-b45d2b32-2a18-365a-9803-64d02476f42d: pipelining replication: peer="{Voter server-a2cbb12f-846a-e8c9-ba36-d3436e519df8 a2cbb12f-846a-e8c9-ba36-d3436e519df8}" 2026-03-31T06:17:49.944Z [INFO] server-b45d2b32-2a18-365a-9803-64d02476f42d: aborting pipeline replication: peer="{Voter server-a2cbb12f-846a-e8c9-ba36-d3436e519df8 a2cbb12f-846a-e8c9-ba36-d3436e519df8}" 2026-03-31T06:17:49.944Z [INFO] server-b45d2b32-2a18-365a-9803-64d02476f42d: aborting pipeline replication: peer="{Voter server-8ee4efb1-6c7d-f6f2-353c-07328eb60cbd 8ee4efb1-6c7d-f6f2-353c-07328eb60cbd}" --- PASS: TestRaft_SendSnapshotFollower (0.30s) === RUN TestRaft_SendSnapshotAndLogsFollower 2026-03-31T06:17:49.945Z [INFO] server-e9b0f684-efef-534a-26c5-684612eb504c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9b0f684-efef-534a-26c5-684612eb504c Address:e9b0f684-efef-534a-26c5-684612eb504c} {Suffrage:Voter ID:server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2 Address:7c27d8a9-9ca6-22ce-351d-9ce8522e65f2} {Suffrage:Voter ID:server-801241bb-5066-f390-cc76-d3f8dbd8fb67 Address:801241bb-5066-f390-cc76-d3f8dbd8fb67}]" 2026-03-31T06:17:49.945Z [INFO] server-e9b0f684-efef-534a-26c5-684612eb504c: entering follower state: follower="Node at e9b0f684-efef-534a-26c5-684612eb504c [Follower]" leader-address= leader-id= 2026-03-31T06:17:49.945Z [INFO] server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9b0f684-efef-534a-26c5-684612eb504c Address:e9b0f684-efef-534a-26c5-684612eb504c} {Suffrage:Voter ID:server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2 Address:7c27d8a9-9ca6-22ce-351d-9ce8522e65f2} {Suffrage:Voter ID:server-801241bb-5066-f390-cc76-d3f8dbd8fb67 Address:801241bb-5066-f390-cc76-d3f8dbd8fb67}]" 2026-03-31T06:17:49.946Z [INFO] server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2: entering follower state: follower="Node at 7c27d8a9-9ca6-22ce-351d-9ce8522e65f2 [Follower]" leader-address= leader-id= 2026-03-31T06:17:49.946Z [INFO] server-801241bb-5066-f390-cc76-d3f8dbd8fb67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9b0f684-efef-534a-26c5-684612eb504c Address:e9b0f684-efef-534a-26c5-684612eb504c} {Suffrage:Voter ID:server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2 Address:7c27d8a9-9ca6-22ce-351d-9ce8522e65f2} {Suffrage:Voter ID:server-801241bb-5066-f390-cc76-d3f8dbd8fb67 Address:801241bb-5066-f390-cc76-d3f8dbd8fb67}]" 2026-03-31T06:17:49.946Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:49.946Z [INFO] server-801241bb-5066-f390-cc76-d3f8dbd8fb67: entering follower state: follower="Node at 801241bb-5066-f390-cc76-d3f8dbd8fb67 [Follower]" leader-address= leader-id= 2026-03-31T06:17:50.008Z [WARN] server-e9b0f684-efef-534a-26c5-684612eb504c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:50.008Z [INFO] server-e9b0f684-efef-534a-26c5-684612eb504c: entering candidate state: node="Node at e9b0f684-efef-534a-26c5-684612eb504c [Candidate]" term=2 2026-03-31T06:17:50.008Z [INFO] server-e9b0f684-efef-534a-26c5-684612eb504c: election won: term=2 tally=2 2026-03-31T06:17:50.008Z [INFO] server-e9b0f684-efef-534a-26c5-684612eb504c: entering leader state: leader="Node at e9b0f684-efef-534a-26c5-684612eb504c [Leader]" 2026-03-31T06:17:50.008Z [INFO] server-e9b0f684-efef-534a-26c5-684612eb504c: added peer, starting replication: peer=server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2 2026-03-31T06:17:50.008Z [INFO] server-e9b0f684-efef-534a-26c5-684612eb504c: added peer, starting replication: peer=server-801241bb-5066-f390-cc76-d3f8dbd8fb67 2026-03-31T06:17:50.009Z [INFO] server-e9b0f684-efef-534a-26c5-684612eb504c: pipelining replication: peer="{Voter server-801241bb-5066-f390-cc76-d3f8dbd8fb67 801241bb-5066-f390-cc76-d3f8dbd8fb67}" 2026-03-31T06:17:50.009Z [INFO] server-e9b0f684-efef-534a-26c5-684612eb504c: pipelining replication: peer="{Voter server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2 7c27d8a9-9ca6-22ce-351d-9ce8522e65f2}" 2026-03-31T06:17:50.114Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:50.00896598 +0000 UTC m=+98.286573401 (2 nodes), 62.70888ms from start of poll, 63.24508ms from cluster start. Timeout at 2026-03-31 06:17:50.11397342 +0000 UTC m=+98.391580841, 105.00744ms after stability 2026-03-31T06:17:50.114Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:50.219Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:50.11469518 +0000 UTC m=+98.392302561 (1 nodes), 2.44µs from start of poll, 168.97424ms from cluster start. Timeout at 2026-03-31 06:17:50.21972342 +0000 UTC m=+98.497330841, 105.02828ms after stability 2026-03-31T06:17:50.219Z [ERROR] server-e9b0f684-efef-534a-26c5-684612eb504c: failed to pipeline appendEntries: peer="{Voter server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2 7c27d8a9-9ca6-22ce-351d-9ce8522e65f2}" error="append pipeline closed" 2026-03-31T06:17:50.219Z [INFO] server-e9b0f684-efef-534a-26c5-684612eb504c: aborting pipeline replication: peer="{Voter server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2 7c27d8a9-9ca6-22ce-351d-9ce8522e65f2}" 2026-03-31T06:17:50.219Z [ERROR] server-e9b0f684-efef-534a-26c5-684612eb504c: failed to appendEntries to: peer="{Voter server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2 7c27d8a9-9ca6-22ce-351d-9ce8522e65f2}" error="failed to connect to peer: 7c27d8a9-9ca6-22ce-351d-9ce8522e65f2" 2026-03-31T06:17:50.223Z [ERROR] server-e9b0f684-efef-534a-26c5-684612eb504c: failed to heartbeat to: peer=7c27d8a9-9ca6-22ce-351d-9ce8522e65f2 backoff time=10ms error="failed to connect to peer: 7c27d8a9-9ca6-22ce-351d-9ce8522e65f2" raft_test.go:1669: [INFO] Finished apply without behind follower 2026-03-31T06:17:50.223Z [INFO] server-e9b0f684-efef-534a-26c5-684612eb504c: starting snapshot up to: index=102 2026-03-31T06:17:50.223Z [INFO] creating new snapshot: path=/tmp/raft3704746840/snapshots/2-102-1774937870223.tmp 2026-03-31T06:17:50.224Z [INFO] server-e9b0f684-efef-534a-26c5-684612eb504c: compacting logs: from=1 to=92 2026-03-31T06:17:50.224Z [INFO] server-e9b0f684-efef-534a-26c5-684612eb504c: snapshot complete up to: index=102 2026-03-31T06:17:50.225Z [ERROR] server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2: failed to take snapshot: error="nothing new to snapshot" 2026-03-31T06:17:50.225Z [INFO] server-801241bb-5066-f390-cc76-d3f8dbd8fb67: starting snapshot up to: index=96 2026-03-31T06:17:50.225Z [INFO] creating new snapshot: path=/tmp/raft1809535513/snapshots/2-96-1774937870225.tmp 2026-03-31T06:17:50.226Z [INFO] server-801241bb-5066-f390-cc76-d3f8dbd8fb67: compacting logs: from=1 to=92 2026-03-31T06:17:50.226Z [INFO] server-801241bb-5066-f390-cc76-d3f8dbd8fb67: snapshot complete up to: index=96 2026-03-31T06:17:50.230Z [ERROR] server-e9b0f684-efef-534a-26c5-684612eb504c: failed to get log: index=2 error="log not found" 2026-03-31T06:17:50.230Z [ERROR] server-e9b0f684-efef-534a-26c5-684612eb504c: failed to install snapshot: id=2-102-1774937870223 error="failed to connect to peer: 7c27d8a9-9ca6-22ce-351d-9ce8522e65f2" 2026-03-31T06:17:50.230Z [ERROR] server-e9b0f684-efef-534a-26c5-684612eb504c: failed to send snapshot to: peer="{Voter server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2 7c27d8a9-9ca6-22ce-351d-9ce8522e65f2}" error="failed to connect to peer: 7c27d8a9-9ca6-22ce-351d-9ce8522e65f2" raft_test.go:1690: [INFO] Finished apply without behind follower 2026-03-31T06:17:50.240Z [ERROR] server-e9b0f684-efef-534a-26c5-684612eb504c: failed to get log: index=2 error="log not found" 2026-03-31T06:17:50.241Z [INFO] creating new snapshot: path=/tmp/raft3671030991/snapshots/2-102-1774937870241.tmp 2026-03-31T06:17:50.241Z [INFO] server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2026-03-31T06:17:50.241Z [INFO] server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2: copied to local snapshot: bytes=693 2026-03-31T06:17:50.242Z [INFO] server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2: snapshot restore progress: id=2-102-1774937870241 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2026-03-31T06:17:50.242Z [INFO] server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2: Installed remote snapshot 2026-03-31T06:17:50.242Z [INFO] server-e9b0f684-efef-534a-26c5-684612eb504c: pipelining replication: peer="{Voter server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2 7c27d8a9-9ca6-22ce-351d-9ce8522e65f2}" 2026-03-31T06:17:50.248Z [INFO] server-e9b0f684-efef-534a-26c5-684612eb504c: aborting pipeline replication: peer="{Voter server-801241bb-5066-f390-cc76-d3f8dbd8fb67 801241bb-5066-f390-cc76-d3f8dbd8fb67}" 2026-03-31T06:17:50.248Z [INFO] server-e9b0f684-efef-534a-26c5-684612eb504c: aborting pipeline replication: peer="{Voter server-7c27d8a9-9ca6-22ce-351d-9ce8522e65f2 7c27d8a9-9ca6-22ce-351d-9ce8522e65f2}" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.30s) === RUN TestRaft_ReJoinFollower 2026-03-31T06:17:50.249Z [INFO] server-a1597eb8-a47c-21a0-99f8-c805d54978f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1597eb8-a47c-21a0-99f8-c805d54978f6 Address:a1597eb8-a47c-21a0-99f8-c805d54978f6} {Suffrage:Voter ID:server-d0eb7b68-a89b-fc4c-732a-b6b19965d047 Address:d0eb7b68-a89b-fc4c-732a-b6b19965d047} {Suffrage:Voter ID:server-f0b0c6ec-7413-242e-8987-4caa199e4719 Address:f0b0c6ec-7413-242e-8987-4caa199e4719}]" 2026-03-31T06:17:50.249Z [INFO] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1597eb8-a47c-21a0-99f8-c805d54978f6 Address:a1597eb8-a47c-21a0-99f8-c805d54978f6} {Suffrage:Voter ID:server-d0eb7b68-a89b-fc4c-732a-b6b19965d047 Address:d0eb7b68-a89b-fc4c-732a-b6b19965d047} {Suffrage:Voter ID:server-f0b0c6ec-7413-242e-8987-4caa199e4719 Address:f0b0c6ec-7413-242e-8987-4caa199e4719}]" 2026-03-31T06:17:50.249Z [INFO] server-a1597eb8-a47c-21a0-99f8-c805d54978f6: entering follower state: follower="Node at a1597eb8-a47c-21a0-99f8-c805d54978f6 [Follower]" leader-address= leader-id= 2026-03-31T06:17:50.249Z [INFO] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: entering follower state: follower="Node at d0eb7b68-a89b-fc4c-732a-b6b19965d047 [Follower]" leader-address= leader-id= 2026-03-31T06:17:50.249Z [INFO] server-f0b0c6ec-7413-242e-8987-4caa199e4719: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1597eb8-a47c-21a0-99f8-c805d54978f6 Address:a1597eb8-a47c-21a0-99f8-c805d54978f6} {Suffrage:Voter ID:server-d0eb7b68-a89b-fc4c-732a-b6b19965d047 Address:d0eb7b68-a89b-fc4c-732a-b6b19965d047} {Suffrage:Voter ID:server-f0b0c6ec-7413-242e-8987-4caa199e4719 Address:f0b0c6ec-7413-242e-8987-4caa199e4719}]" 2026-03-31T06:17:50.249Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:50.249Z [INFO] server-f0b0c6ec-7413-242e-8987-4caa199e4719: entering follower state: follower="Node at f0b0c6ec-7413-242e-8987-4caa199e4719 [Follower]" leader-address= leader-id= 2026-03-31T06:17:50.332Z [WARN] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:50.332Z [INFO] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: entering candidate state: node="Node at d0eb7b68-a89b-fc4c-732a-b6b19965d047 [Candidate]" term=2 2026-03-31T06:17:50.332Z [INFO] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: election won: term=2 tally=2 2026-03-31T06:17:50.332Z [INFO] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: entering leader state: leader="Node at d0eb7b68-a89b-fc4c-732a-b6b19965d047 [Leader]" 2026-03-31T06:17:50.332Z [INFO] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: added peer, starting replication: peer=server-a1597eb8-a47c-21a0-99f8-c805d54978f6 2026-03-31T06:17:50.332Z [INFO] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: added peer, starting replication: peer=server-f0b0c6ec-7413-242e-8987-4caa199e4719 2026-03-31T06:17:50.332Z [INFO] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: pipelining replication: peer="{Voter server-a1597eb8-a47c-21a0-99f8-c805d54978f6 a1597eb8-a47c-21a0-99f8-c805d54978f6}" 2026-03-31T06:17:50.332Z [INFO] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: pipelining replication: peer="{Voter server-f0b0c6ec-7413-242e-8987-4caa199e4719 f0b0c6ec-7413-242e-8987-4caa199e4719}" 2026-03-31T06:17:50.438Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:50.33282978 +0000 UTC m=+98.610437161 (1 nodes), 83.08992ms from start of poll, 83.54456ms from cluster start. Timeout at 2026-03-31 06:17:50.4378345 +0000 UTC m=+98.715441881, 105.00472ms after stability 2026-03-31T06:17:50.444Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:50.550Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:50.4450013 +0000 UTC m=+98.722608681 (2 nodes), 1.4µs from start of poll, 195.71608ms from cluster start. Timeout at 2026-03-31 06:17:50.55001666 +0000 UTC m=+98.827624081, 105.0154ms after stability 2026-03-31T06:17:50.550Z [INFO] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: updating configuration: command=RemoveServer server-id=server-a1597eb8-a47c-21a0-99f8-c805d54978f6 server-addr= servers="[{Suffrage:Voter ID:server-d0eb7b68-a89b-fc4c-732a-b6b19965d047 Address:d0eb7b68-a89b-fc4c-732a-b6b19965d047} {Suffrage:Voter ID:server-f0b0c6ec-7413-242e-8987-4caa199e4719 Address:f0b0c6ec-7413-242e-8987-4caa199e4719}]" 2026-03-31T06:17:50.550Z [INFO] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: removed peer, stopping replication: peer=server-a1597eb8-a47c-21a0-99f8-c805d54978f6 last-index=3 2026-03-31T06:17:50.550Z [INFO] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: aborting pipeline replication: peer="{Voter server-a1597eb8-a47c-21a0-99f8-c805d54978f6 a1597eb8-a47c-21a0-99f8-c805d54978f6}" 2026-03-31T06:17:50.633Z [WARN] server-a1597eb8-a47c-21a0-99f8-c805d54978f6: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2026-03-31T06:17:50.656Z [INFO] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: updating configuration: command=AddVoter server-id=server-a1597eb8-a47c-21a0-99f8-c805d54978f6 server-addr=a1597eb8-a47c-21a0-99f8-c805d54978f6 servers="[{Suffrage:Voter ID:server-d0eb7b68-a89b-fc4c-732a-b6b19965d047 Address:d0eb7b68-a89b-fc4c-732a-b6b19965d047} {Suffrage:Voter ID:server-f0b0c6ec-7413-242e-8987-4caa199e4719 Address:f0b0c6ec-7413-242e-8987-4caa199e4719} {Suffrage:Voter ID:server-a1597eb8-a47c-21a0-99f8-c805d54978f6 Address:a1597eb8-a47c-21a0-99f8-c805d54978f6}]" 2026-03-31T06:17:50.656Z [INFO] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: added peer, starting replication: peer=server-a1597eb8-a47c-21a0-99f8-c805d54978f6 2026-03-31T06:17:50.656Z [WARN] server-a1597eb8-a47c-21a0-99f8-c805d54978f6: failed to get previous log: previous-index=4 last-index=3 error="log not found" 2026-03-31T06:17:50.657Z [WARN] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: appendEntries rejected, sending older logs: peer="{Voter server-a1597eb8-a47c-21a0-99f8-c805d54978f6 a1597eb8-a47c-21a0-99f8-c805d54978f6}" next=4 2026-03-31T06:17:50.657Z [INFO] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: pipelining replication: peer="{Voter server-a1597eb8-a47c-21a0-99f8-c805d54978f6 a1597eb8-a47c-21a0-99f8-c805d54978f6}" 2026-03-31T06:17:50.657Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:50.762Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:50.65722478 +0000 UTC m=+98.934832161 (1 nodes), 2.28µs from start of poll, 407.93956ms from cluster start. Timeout at 2026-03-31 06:17:50.76226394 +0000 UTC m=+99.039871321, 105.03916ms after stability 2026-03-31T06:17:50.762Z [INFO] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: aborting pipeline replication: peer="{Voter server-a1597eb8-a47c-21a0-99f8-c805d54978f6 a1597eb8-a47c-21a0-99f8-c805d54978f6}" 2026-03-31T06:17:50.762Z [INFO] server-d0eb7b68-a89b-fc4c-732a-b6b19965d047: aborting pipeline replication: peer="{Voter server-f0b0c6ec-7413-242e-8987-4caa199e4719 f0b0c6ec-7413-242e-8987-4caa199e4719}" --- PASS: TestRaft_ReJoinFollower (0.51s) === RUN TestRaft_LeaderLeaseExpire 2026-03-31T06:17:50.763Z [INFO] server-71b20d2e-cd70-0e29-74a5-a54159011605: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-71b20d2e-cd70-0e29-74a5-a54159011605 Address:71b20d2e-cd70-0e29-74a5-a54159011605} {Suffrage:Voter ID:server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98 Address:a7f38cf4-329f-9cc0-3dff-ef43ab723d98}]" 2026-03-31T06:17:50.763Z [INFO] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-71b20d2e-cd70-0e29-74a5-a54159011605 Address:71b20d2e-cd70-0e29-74a5-a54159011605} {Suffrage:Voter ID:server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98 Address:a7f38cf4-329f-9cc0-3dff-ef43ab723d98}]" 2026-03-31T06:17:50.763Z [INFO] server-71b20d2e-cd70-0e29-74a5-a54159011605: entering follower state: follower="Node at 71b20d2e-cd70-0e29-74a5-a54159011605 [Follower]" leader-address= leader-id= 2026-03-31T06:17:50.763Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:50.763Z [INFO] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: entering follower state: follower="Node at a7f38cf4-329f-9cc0-3dff-ef43ab723d98 [Follower]" leader-address= leader-id= 2026-03-31T06:17:50.825Z [WARN] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:50.825Z [INFO] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: entering candidate state: node="Node at a7f38cf4-329f-9cc0-3dff-ef43ab723d98 [Candidate]" term=2 2026-03-31T06:17:50.825Z [INFO] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: election won: term=2 tally=2 2026-03-31T06:17:50.825Z [INFO] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: entering leader state: leader="Node at a7f38cf4-329f-9cc0-3dff-ef43ab723d98 [Leader]" 2026-03-31T06:17:50.825Z [INFO] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: added peer, starting replication: peer=server-71b20d2e-cd70-0e29-74a5-a54159011605 2026-03-31T06:17:50.825Z [INFO] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: pipelining replication: peer="{Voter server-71b20d2e-cd70-0e29-74a5-a54159011605 71b20d2e-cd70-0e29-74a5-a54159011605}" 2026-03-31T06:17:50.931Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:50.82577278 +0000 UTC m=+99.103380161 (1 nodes), 62.1074ms from start of poll, 62.52144ms from cluster start. Timeout at 2026-03-31 06:17:50.93079406 +0000 UTC m=+99.208401441, 105.02128ms after stability 2026-03-31T06:17:50.936Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:51.041Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:50.93653534 +0000 UTC m=+99.214142721 (1 nodes), 1.76µs from start of poll, 173.284ms from cluster start. Timeout at 2026-03-31 06:17:51.04156346 +0000 UTC m=+99.319170881, 105.02816ms after stability raft_test.go:1799: [INFO] Disconnecting Node at 71b20d2e-cd70-0e29-74a5-a54159011605 [Follower] 2026-03-31T06:17:51.049Z [ERROR] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: failed to pipeline appendEntries: peer="{Voter server-71b20d2e-cd70-0e29-74a5-a54159011605 71b20d2e-cd70-0e29-74a5-a54159011605}" error="append pipeline closed" 2026-03-31T06:17:51.049Z [INFO] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: aborting pipeline replication: peer="{Voter server-71b20d2e-cd70-0e29-74a5-a54159011605 71b20d2e-cd70-0e29-74a5-a54159011605}" 2026-03-31T06:17:51.051Z [ERROR] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: failed to heartbeat to: peer=71b20d2e-cd70-0e29-74a5-a54159011605 backoff time=10ms error="failed to connect to peer: 71b20d2e-cd70-0e29-74a5-a54159011605" 2026-03-31T06:17:51.057Z [ERROR] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: failed to appendEntries to: peer="{Voter server-71b20d2e-cd70-0e29-74a5-a54159011605 71b20d2e-cd70-0e29-74a5-a54159011605}" error="failed to connect to peer: 71b20d2e-cd70-0e29-74a5-a54159011605" 2026-03-31T06:17:51.068Z [ERROR] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: failed to heartbeat to: peer=71b20d2e-cd70-0e29-74a5-a54159011605 backoff time=10ms error="failed to connect to peer: 71b20d2e-cd70-0e29-74a5-a54159011605" 2026-03-31T06:17:51.075Z [ERROR] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: failed to appendEntries to: peer="{Voter server-71b20d2e-cd70-0e29-74a5-a54159011605 71b20d2e-cd70-0e29-74a5-a54159011605}" error="failed to connect to peer: 71b20d2e-cd70-0e29-74a5-a54159011605" 2026-03-31T06:17:51.089Z [ERROR] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: failed to heartbeat to: peer=71b20d2e-cd70-0e29-74a5-a54159011605 backoff time=10ms error="failed to connect to peer: 71b20d2e-cd70-0e29-74a5-a54159011605" 2026-03-31T06:17:51.092Z [WARN] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: failed to contact: server-id=server-71b20d2e-cd70-0e29-74a5-a54159011605 time=50.91724ms 2026-03-31T06:17:51.092Z [WARN] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: failed to contact quorum of nodes, stepping down 2026-03-31T06:17:51.093Z [INFO] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: entering follower state: follower="Node at a7f38cf4-329f-9cc0-3dff-ef43ab723d98 [Follower]" leader-address= leader-id= 2026-03-31T06:17:51.093Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:51.093Z [ERROR] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: failed to appendEntries to: peer="{Voter server-71b20d2e-cd70-0e29-74a5-a54159011605 71b20d2e-cd70-0e29-74a5-a54159011605}" error="failed to connect to peer: 71b20d2e-cd70-0e29-74a5-a54159011605" 2026-03-31T06:17:51.156Z [WARN] server-71b20d2e-cd70-0e29-74a5-a54159011605: heartbeat timeout reached, starting election: last-leader-addr=a7f38cf4-329f-9cc0-3dff-ef43ab723d98 last-leader-id=server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98 2026-03-31T06:17:51.156Z [INFO] server-71b20d2e-cd70-0e29-74a5-a54159011605: entering candidate state: node="Node at 71b20d2e-cd70-0e29-74a5-a54159011605 [Candidate]" term=3 2026-03-31T06:17:51.156Z [ERROR] server-71b20d2e-cd70-0e29-74a5-a54159011605: failed to make requestVote RPC: target="{Voter server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98 a7f38cf4-329f-9cc0-3dff-ef43ab723d98}" error="failed to connect to peer: a7f38cf4-329f-9cc0-3dff-ef43ab723d98" term=3 2026-03-31T06:17:51.183Z [WARN] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:51.183Z [INFO] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: entering candidate state: node="Node at a7f38cf4-329f-9cc0-3dff-ef43ab723d98 [Candidate]" term=3 2026-03-31T06:17:51.183Z [ERROR] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: failed to make requestVote RPC: target="{Voter server-71b20d2e-cd70-0e29-74a5-a54159011605 71b20d2e-cd70-0e29-74a5-a54159011605}" error="failed to connect to peer: 71b20d2e-cd70-0e29-74a5-a54159011605" term=3 2026-03-31T06:17:51.227Z [WARN] server-71b20d2e-cd70-0e29-74a5-a54159011605: Election timeout reached, restarting election 2026-03-31T06:17:51.227Z [INFO] server-71b20d2e-cd70-0e29-74a5-a54159011605: entering candidate state: node="Node at 71b20d2e-cd70-0e29-74a5-a54159011605 [Candidate]" term=4 2026-03-31T06:17:51.227Z [ERROR] server-71b20d2e-cd70-0e29-74a5-a54159011605: failed to make requestVote RPC: target="{Voter server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98 a7f38cf4-329f-9cc0-3dff-ef43ab723d98}" error="failed to connect to peer: a7f38cf4-329f-9cc0-3dff-ef43ab723d98" term=4 2026-03-31T06:17:51.242Z [WARN] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: Election timeout reached, restarting election 2026-03-31T06:17:51.242Z [INFO] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: entering candidate state: node="Node at a7f38cf4-329f-9cc0-3dff-ef43ab723d98 [Candidate]" term=4 2026-03-31T06:17:51.242Z [ERROR] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: failed to make requestVote RPC: target="{Voter server-71b20d2e-cd70-0e29-74a5-a54159011605 71b20d2e-cd70-0e29-74a5-a54159011605}" error="failed to connect to peer: 71b20d2e-cd70-0e29-74a5-a54159011605" term=4 2026-03-31T06:17:51.288Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:51.18339026 +0000 UTC m=+99.460997681 (0 nodes), 89.88912ms from start of poll, 420.13896ms from cluster start. Timeout at 2026-03-31 06:17:51.28840082 +0000 UTC m=+99.566008201, 105.01052ms after stability 2026-03-31T06:17:51.306Z [WARN] server-71b20d2e-cd70-0e29-74a5-a54159011605: Election timeout reached, restarting election 2026-03-31T06:17:51.306Z [INFO] server-71b20d2e-cd70-0e29-74a5-a54159011605: entering candidate state: node="Node at 71b20d2e-cd70-0e29-74a5-a54159011605 [Candidate]" term=5 2026-03-31T06:17:51.306Z [ERROR] server-71b20d2e-cd70-0e29-74a5-a54159011605: failed to make requestVote RPC: target="{Voter server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98 a7f38cf4-329f-9cc0-3dff-ef43ab723d98}" error="failed to connect to peer: a7f38cf4-329f-9cc0-3dff-ef43ab723d98" term=5 2026-03-31T06:17:51.337Z [WARN] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: Election timeout reached, restarting election 2026-03-31T06:17:51.337Z [INFO] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: entering candidate state: node="Node at a7f38cf4-329f-9cc0-3dff-ef43ab723d98 [Candidate]" term=5 2026-03-31T06:17:51.337Z [ERROR] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: failed to make requestVote RPC: target="{Voter server-71b20d2e-cd70-0e29-74a5-a54159011605 71b20d2e-cd70-0e29-74a5-a54159011605}" error="failed to connect to peer: 71b20d2e-cd70-0e29-74a5-a54159011605" term=5 2026-03-31T06:17:51.363Z [WARN] server-71b20d2e-cd70-0e29-74a5-a54159011605: Election timeout reached, restarting election 2026-03-31T06:17:51.363Z [INFO] server-71b20d2e-cd70-0e29-74a5-a54159011605: entering candidate state: node="Node at 71b20d2e-cd70-0e29-74a5-a54159011605 [Candidate]" term=6 2026-03-31T06:17:51.363Z [ERROR] server-71b20d2e-cd70-0e29-74a5-a54159011605: failed to make requestVote RPC: target="{Voter server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98 a7f38cf4-329f-9cc0-3dff-ef43ab723d98}" error="failed to connect to peer: a7f38cf4-329f-9cc0-3dff-ef43ab723d98" term=6 2026-03-31T06:17:51.392Z [WARN] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: Election timeout reached, restarting election 2026-03-31T06:17:51.392Z [INFO] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: entering candidate state: node="Node at a7f38cf4-329f-9cc0-3dff-ef43ab723d98 [Candidate]" term=6 2026-03-31T06:17:51.392Z [ERROR] server-a7f38cf4-329f-9cc0-3dff-ef43ab723d98: failed to make requestVote RPC: target="{Voter server-71b20d2e-cd70-0e29-74a5-a54159011605 71b20d2e-cd70-0e29-74a5-a54159011605}" error="failed to connect to peer: 71b20d2e-cd70-0e29-74a5-a54159011605" term=6 --- PASS: TestRaft_LeaderLeaseExpire (0.63s) === RUN TestRaft_Barrier 2026-03-31T06:17:51.395Z [INFO] server-34f0a963-2d6c-59ca-96d0-b9f75dc86b2b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f0a963-2d6c-59ca-96d0-b9f75dc86b2b Address:34f0a963-2d6c-59ca-96d0-b9f75dc86b2b} {Suffrage:Voter ID:server-3f6f0386-ee87-fa1e-e03e-dfe8ae0502cb Address:3f6f0386-ee87-fa1e-e03e-dfe8ae0502cb} {Suffrage:Voter ID:server-c1a650a5-e22b-b074-15db-e605969ca9bb Address:c1a650a5-e22b-b074-15db-e605969ca9bb}]" 2026-03-31T06:17:51.395Z [INFO] server-34f0a963-2d6c-59ca-96d0-b9f75dc86b2b: entering follower state: follower="Node at 34f0a963-2d6c-59ca-96d0-b9f75dc86b2b [Follower]" leader-address= leader-id= 2026-03-31T06:17:51.395Z [INFO] server-3f6f0386-ee87-fa1e-e03e-dfe8ae0502cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f0a963-2d6c-59ca-96d0-b9f75dc86b2b Address:34f0a963-2d6c-59ca-96d0-b9f75dc86b2b} {Suffrage:Voter ID:server-3f6f0386-ee87-fa1e-e03e-dfe8ae0502cb Address:3f6f0386-ee87-fa1e-e03e-dfe8ae0502cb} {Suffrage:Voter ID:server-c1a650a5-e22b-b074-15db-e605969ca9bb Address:c1a650a5-e22b-b074-15db-e605969ca9bb}]" 2026-03-31T06:17:51.395Z [INFO] server-3f6f0386-ee87-fa1e-e03e-dfe8ae0502cb: entering follower state: follower="Node at 3f6f0386-ee87-fa1e-e03e-dfe8ae0502cb [Follower]" leader-address= leader-id= 2026-03-31T06:17:51.395Z [INFO] server-c1a650a5-e22b-b074-15db-e605969ca9bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f0a963-2d6c-59ca-96d0-b9f75dc86b2b Address:34f0a963-2d6c-59ca-96d0-b9f75dc86b2b} {Suffrage:Voter ID:server-3f6f0386-ee87-fa1e-e03e-dfe8ae0502cb Address:3f6f0386-ee87-fa1e-e03e-dfe8ae0502cb} {Suffrage:Voter ID:server-c1a650a5-e22b-b074-15db-e605969ca9bb Address:c1a650a5-e22b-b074-15db-e605969ca9bb}]" 2026-03-31T06:17:51.395Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:51.395Z [INFO] server-c1a650a5-e22b-b074-15db-e605969ca9bb: entering follower state: follower="Node at c1a650a5-e22b-b074-15db-e605969ca9bb [Follower]" leader-address= leader-id= 2026-03-31T06:17:51.449Z [WARN] server-34f0a963-2d6c-59ca-96d0-b9f75dc86b2b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:51.449Z [INFO] server-34f0a963-2d6c-59ca-96d0-b9f75dc86b2b: entering candidate state: node="Node at 34f0a963-2d6c-59ca-96d0-b9f75dc86b2b [Candidate]" term=2 2026-03-31T06:17:51.449Z [INFO] server-34f0a963-2d6c-59ca-96d0-b9f75dc86b2b: election won: term=2 tally=2 2026-03-31T06:17:51.449Z [INFO] server-34f0a963-2d6c-59ca-96d0-b9f75dc86b2b: entering leader state: leader="Node at 34f0a963-2d6c-59ca-96d0-b9f75dc86b2b [Leader]" 2026-03-31T06:17:51.449Z [INFO] server-34f0a963-2d6c-59ca-96d0-b9f75dc86b2b: added peer, starting replication: peer=server-3f6f0386-ee87-fa1e-e03e-dfe8ae0502cb 2026-03-31T06:17:51.449Z [INFO] server-34f0a963-2d6c-59ca-96d0-b9f75dc86b2b: added peer, starting replication: peer=server-c1a650a5-e22b-b074-15db-e605969ca9bb 2026-03-31T06:17:51.450Z [INFO] server-34f0a963-2d6c-59ca-96d0-b9f75dc86b2b: pipelining replication: peer="{Voter server-3f6f0386-ee87-fa1e-e03e-dfe8ae0502cb 3f6f0386-ee87-fa1e-e03e-dfe8ae0502cb}" 2026-03-31T06:17:51.450Z [INFO] server-34f0a963-2d6c-59ca-96d0-b9f75dc86b2b: pipelining replication: peer="{Voter server-c1a650a5-e22b-b074-15db-e605969ca9bb c1a650a5-e22b-b074-15db-e605969ca9bb}" 2026-03-31T06:17:51.554Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:51.44989866 +0000 UTC m=+99.727506041 (1 nodes), 54.11ms from start of poll, 54.58356ms from cluster start. Timeout at 2026-03-31 06:17:51.55490866 +0000 UTC m=+99.832516081, 105.01004ms after stability 2026-03-31T06:17:51.573Z [INFO] server-34f0a963-2d6c-59ca-96d0-b9f75dc86b2b: aborting pipeline replication: peer="{Voter server-c1a650a5-e22b-b074-15db-e605969ca9bb c1a650a5-e22b-b074-15db-e605969ca9bb}" 2026-03-31T06:17:51.573Z [INFO] server-34f0a963-2d6c-59ca-96d0-b9f75dc86b2b: aborting pipeline replication: peer="{Voter server-3f6f0386-ee87-fa1e-e03e-dfe8ae0502cb 3f6f0386-ee87-fa1e-e03e-dfe8ae0502cb}" --- PASS: TestRaft_Barrier (0.18s) === RUN TestRaft_VerifyLeader 2026-03-31T06:17:51.574Z [INFO] server-06c85147-c5a6-5cba-b5c7-e5d14d114d07: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06c85147-c5a6-5cba-b5c7-e5d14d114d07 Address:06c85147-c5a6-5cba-b5c7-e5d14d114d07} {Suffrage:Voter ID:server-9124fbc9-38f7-5775-26e0-d782db1645f5 Address:9124fbc9-38f7-5775-26e0-d782db1645f5} {Suffrage:Voter ID:server-9c164685-f2f0-af6c-b2a8-1bb3a69d9ca1 Address:9c164685-f2f0-af6c-b2a8-1bb3a69d9ca1}]" 2026-03-31T06:17:51.574Z [INFO] server-9124fbc9-38f7-5775-26e0-d782db1645f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06c85147-c5a6-5cba-b5c7-e5d14d114d07 Address:06c85147-c5a6-5cba-b5c7-e5d14d114d07} {Suffrage:Voter ID:server-9124fbc9-38f7-5775-26e0-d782db1645f5 Address:9124fbc9-38f7-5775-26e0-d782db1645f5} {Suffrage:Voter ID:server-9c164685-f2f0-af6c-b2a8-1bb3a69d9ca1 Address:9c164685-f2f0-af6c-b2a8-1bb3a69d9ca1}]" 2026-03-31T06:17:51.574Z [INFO] server-06c85147-c5a6-5cba-b5c7-e5d14d114d07: entering follower state: follower="Node at 06c85147-c5a6-5cba-b5c7-e5d14d114d07 [Follower]" leader-address= leader-id= 2026-03-31T06:17:51.574Z [INFO] server-9124fbc9-38f7-5775-26e0-d782db1645f5: entering follower state: follower="Node at 9124fbc9-38f7-5775-26e0-d782db1645f5 [Follower]" leader-address= leader-id= 2026-03-31T06:17:51.574Z [INFO] server-9c164685-f2f0-af6c-b2a8-1bb3a69d9ca1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06c85147-c5a6-5cba-b5c7-e5d14d114d07 Address:06c85147-c5a6-5cba-b5c7-e5d14d114d07} {Suffrage:Voter ID:server-9124fbc9-38f7-5775-26e0-d782db1645f5 Address:9124fbc9-38f7-5775-26e0-d782db1645f5} {Suffrage:Voter ID:server-9c164685-f2f0-af6c-b2a8-1bb3a69d9ca1 Address:9c164685-f2f0-af6c-b2a8-1bb3a69d9ca1}]" 2026-03-31T06:17:51.574Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:51.574Z [INFO] server-9c164685-f2f0-af6c-b2a8-1bb3a69d9ca1: entering follower state: follower="Node at 9c164685-f2f0-af6c-b2a8-1bb3a69d9ca1 [Follower]" leader-address= leader-id= 2026-03-31T06:17:51.642Z [WARN] server-06c85147-c5a6-5cba-b5c7-e5d14d114d07: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:51.642Z [INFO] server-06c85147-c5a6-5cba-b5c7-e5d14d114d07: entering candidate state: node="Node at 06c85147-c5a6-5cba-b5c7-e5d14d114d07 [Candidate]" term=2 2026-03-31T06:17:51.642Z [INFO] server-06c85147-c5a6-5cba-b5c7-e5d14d114d07: election won: term=2 tally=2 2026-03-31T06:17:51.642Z [INFO] server-06c85147-c5a6-5cba-b5c7-e5d14d114d07: entering leader state: leader="Node at 06c85147-c5a6-5cba-b5c7-e5d14d114d07 [Leader]" 2026-03-31T06:17:51.642Z [INFO] server-06c85147-c5a6-5cba-b5c7-e5d14d114d07: added peer, starting replication: peer=server-9124fbc9-38f7-5775-26e0-d782db1645f5 2026-03-31T06:17:51.642Z [INFO] server-06c85147-c5a6-5cba-b5c7-e5d14d114d07: added peer, starting replication: peer=server-9c164685-f2f0-af6c-b2a8-1bb3a69d9ca1 2026-03-31T06:17:51.642Z [INFO] server-06c85147-c5a6-5cba-b5c7-e5d14d114d07: pipelining replication: peer="{Voter server-9c164685-f2f0-af6c-b2a8-1bb3a69d9ca1 9c164685-f2f0-af6c-b2a8-1bb3a69d9ca1}" 2026-03-31T06:17:51.642Z [INFO] server-06c85147-c5a6-5cba-b5c7-e5d14d114d07: pipelining replication: peer="{Voter server-9124fbc9-38f7-5775-26e0-d782db1645f5 9124fbc9-38f7-5775-26e0-d782db1645f5}" 2026-03-31T06:17:51.747Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:51.64263514 +0000 UTC m=+99.920242561 (1 nodes), 68.15388ms from start of poll, 68.57048ms from cluster start. Timeout at 2026-03-31 06:17:51.74763914 +0000 UTC m=+100.025246521, 105.00396ms after stability 2026-03-31T06:17:51.748Z [INFO] server-06c85147-c5a6-5cba-b5c7-e5d14d114d07: aborting pipeline replication: peer="{Voter server-9c164685-f2f0-af6c-b2a8-1bb3a69d9ca1 9c164685-f2f0-af6c-b2a8-1bb3a69d9ca1}" 2026-03-31T06:17:51.748Z [INFO] server-06c85147-c5a6-5cba-b5c7-e5d14d114d07: aborting pipeline replication: peer="{Voter server-9124fbc9-38f7-5775-26e0-d782db1645f5 9124fbc9-38f7-5775-26e0-d782db1645f5}" --- PASS: TestRaft_VerifyLeader (0.17s) === RUN TestRaft_VerifyLeader_Single 2026-03-31T06:17:51.748Z [INFO] server-0c901148-d2fa-f628-e258-3274be0ccd3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c901148-d2fa-f628-e258-3274be0ccd3b Address:0c901148-d2fa-f628-e258-3274be0ccd3b}]" 2026-03-31T06:17:51.749Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:51.749Z [INFO] server-0c901148-d2fa-f628-e258-3274be0ccd3b: entering follower state: follower="Node at 0c901148-d2fa-f628-e258-3274be0ccd3b [Follower]" leader-address= leader-id= 2026-03-31T06:17:51.820Z [WARN] server-0c901148-d2fa-f628-e258-3274be0ccd3b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:51.821Z [INFO] server-0c901148-d2fa-f628-e258-3274be0ccd3b: entering candidate state: node="Node at 0c901148-d2fa-f628-e258-3274be0ccd3b [Candidate]" term=2 2026-03-31T06:17:51.821Z [INFO] server-0c901148-d2fa-f628-e258-3274be0ccd3b: election won: term=2 tally=1 2026-03-31T06:17:51.821Z [INFO] server-0c901148-d2fa-f628-e258-3274be0ccd3b: entering leader state: leader="Node at 0c901148-d2fa-f628-e258-3274be0ccd3b [Leader]" 2026-03-31T06:17:51.927Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:51.82235826 +0000 UTC m=+100.099965681 (1 nodes), 73.31152ms from start of poll, 73.53468ms from cluster start. Timeout at 2026-03-31 06:17:51.92738034 +0000 UTC m=+100.204987721, 105.02204ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.18s) === RUN TestRaft_VerifyLeader_Fail 2026-03-31T06:17:51.928Z [INFO] server-efc6e291-a578-0069-368f-41258eaf6e28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-efc6e291-a578-0069-368f-41258eaf6e28 Address:efc6e291-a578-0069-368f-41258eaf6e28} {Suffrage:Voter ID:server-5e9563d4-1a6d-1fa3-b7d8-c8f63f652082 Address:5e9563d4-1a6d-1fa3-b7d8-c8f63f652082}]" 2026-03-31T06:17:51.928Z [INFO] server-5e9563d4-1a6d-1fa3-b7d8-c8f63f652082: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-efc6e291-a578-0069-368f-41258eaf6e28 Address:efc6e291-a578-0069-368f-41258eaf6e28} {Suffrage:Voter ID:server-5e9563d4-1a6d-1fa3-b7d8-c8f63f652082 Address:5e9563d4-1a6d-1fa3-b7d8-c8f63f652082}]" 2026-03-31T06:17:51.928Z [INFO] server-efc6e291-a578-0069-368f-41258eaf6e28: entering follower state: follower="Node at efc6e291-a578-0069-368f-41258eaf6e28 [Follower]" leader-address= leader-id= 2026-03-31T06:17:51.928Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:51.928Z [INFO] server-5e9563d4-1a6d-1fa3-b7d8-c8f63f652082: entering follower state: follower="Node at 5e9563d4-1a6d-1fa3-b7d8-c8f63f652082 [Follower]" leader-address= leader-id= 2026-03-31T06:17:51.978Z [WARN] server-efc6e291-a578-0069-368f-41258eaf6e28: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:51.978Z [INFO] server-efc6e291-a578-0069-368f-41258eaf6e28: entering candidate state: node="Node at efc6e291-a578-0069-368f-41258eaf6e28 [Candidate]" term=2 2026-03-31T06:17:51.979Z [INFO] server-efc6e291-a578-0069-368f-41258eaf6e28: election won: term=2 tally=2 2026-03-31T06:17:51.979Z [INFO] server-efc6e291-a578-0069-368f-41258eaf6e28: entering leader state: leader="Node at efc6e291-a578-0069-368f-41258eaf6e28 [Leader]" 2026-03-31T06:17:51.979Z [INFO] server-efc6e291-a578-0069-368f-41258eaf6e28: added peer, starting replication: peer=server-5e9563d4-1a6d-1fa3-b7d8-c8f63f652082 2026-03-31T06:17:51.979Z [INFO] server-efc6e291-a578-0069-368f-41258eaf6e28: pipelining replication: peer="{Voter server-5e9563d4-1a6d-1fa3-b7d8-c8f63f652082 5e9563d4-1a6d-1fa3-b7d8-c8f63f652082}" 2026-03-31T06:17:52.085Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:51.97921158 +0000 UTC m=+100.256818961 (1 nodes), 50.6304ms from start of poll, 50.93872ms from cluster start. Timeout at 2026-03-31 06:17:52.08421498 +0000 UTC m=+100.361822401, 105.00344ms after stability 2026-03-31T06:17:52.085Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:52.190Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:52.08540646 +0000 UTC m=+100.363013841 (1 nodes), 2.64µs from start of poll, 157.1336ms from cluster start. Timeout at 2026-03-31 06:17:52.19043466 +0000 UTC m=+100.468042081, 105.02824ms after stability 2026-03-31T06:17:52.190Z [ERROR] server-efc6e291-a578-0069-368f-41258eaf6e28: peer has newer term, stopping replication: peer="{Voter server-5e9563d4-1a6d-1fa3-b7d8-c8f63f652082 5e9563d4-1a6d-1fa3-b7d8-c8f63f652082}" 2026-03-31T06:17:52.190Z [INFO] server-efc6e291-a578-0069-368f-41258eaf6e28: aborting pipeline replication: peer="{Voter server-5e9563d4-1a6d-1fa3-b7d8-c8f63f652082 5e9563d4-1a6d-1fa3-b7d8-c8f63f652082}" 2026-03-31T06:17:52.191Z [INFO] server-efc6e291-a578-0069-368f-41258eaf6e28: entering follower state: follower="Node at efc6e291-a578-0069-368f-41258eaf6e28 [Follower]" leader-address= leader-id= --- PASS: TestRaft_VerifyLeader_Fail (0.26s) === RUN TestRaft_VerifyLeader_PartialConnect 2026-03-31T06:17:52.191Z [INFO] server-5c84d6b3-b164-8151-16b5-adb6f5d2f633: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c84d6b3-b164-8151-16b5-adb6f5d2f633 Address:5c84d6b3-b164-8151-16b5-adb6f5d2f633} {Suffrage:Voter ID:server-1d4f6e84-e27e-f44d-e253-fcfed5c8395c Address:1d4f6e84-e27e-f44d-e253-fcfed5c8395c} {Suffrage:Voter ID:server-f1811d58-2110-798e-957e-feefe2a74fca Address:f1811d58-2110-798e-957e-feefe2a74fca}]" 2026-03-31T06:17:52.191Z [INFO] server-5c84d6b3-b164-8151-16b5-adb6f5d2f633: entering follower state: follower="Node at 5c84d6b3-b164-8151-16b5-adb6f5d2f633 [Follower]" leader-address= leader-id= 2026-03-31T06:17:52.192Z [INFO] server-1d4f6e84-e27e-f44d-e253-fcfed5c8395c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c84d6b3-b164-8151-16b5-adb6f5d2f633 Address:5c84d6b3-b164-8151-16b5-adb6f5d2f633} {Suffrage:Voter ID:server-1d4f6e84-e27e-f44d-e253-fcfed5c8395c Address:1d4f6e84-e27e-f44d-e253-fcfed5c8395c} {Suffrage:Voter ID:server-f1811d58-2110-798e-957e-feefe2a74fca Address:f1811d58-2110-798e-957e-feefe2a74fca}]" 2026-03-31T06:17:52.192Z [INFO] server-1d4f6e84-e27e-f44d-e253-fcfed5c8395c: entering follower state: follower="Node at 1d4f6e84-e27e-f44d-e253-fcfed5c8395c [Follower]" leader-address= leader-id= 2026-03-31T06:17:52.192Z [INFO] server-f1811d58-2110-798e-957e-feefe2a74fca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c84d6b3-b164-8151-16b5-adb6f5d2f633 Address:5c84d6b3-b164-8151-16b5-adb6f5d2f633} {Suffrage:Voter ID:server-1d4f6e84-e27e-f44d-e253-fcfed5c8395c Address:1d4f6e84-e27e-f44d-e253-fcfed5c8395c} {Suffrage:Voter ID:server-f1811d58-2110-798e-957e-feefe2a74fca Address:f1811d58-2110-798e-957e-feefe2a74fca}]" 2026-03-31T06:17:52.192Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:52.192Z [INFO] server-f1811d58-2110-798e-957e-feefe2a74fca: entering follower state: follower="Node at f1811d58-2110-798e-957e-feefe2a74fca [Follower]" leader-address= leader-id= 2026-03-31T06:17:52.255Z [WARN] server-f1811d58-2110-798e-957e-feefe2a74fca: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:52.255Z [INFO] server-f1811d58-2110-798e-957e-feefe2a74fca: entering candidate state: node="Node at f1811d58-2110-798e-957e-feefe2a74fca [Candidate]" term=2 2026-03-31T06:17:52.255Z [INFO] server-f1811d58-2110-798e-957e-feefe2a74fca: election won: term=2 tally=2 2026-03-31T06:17:52.255Z [INFO] server-f1811d58-2110-798e-957e-feefe2a74fca: entering leader state: leader="Node at f1811d58-2110-798e-957e-feefe2a74fca [Leader]" 2026-03-31T06:17:52.255Z [INFO] server-f1811d58-2110-798e-957e-feefe2a74fca: added peer, starting replication: peer=server-5c84d6b3-b164-8151-16b5-adb6f5d2f633 2026-03-31T06:17:52.255Z [INFO] server-f1811d58-2110-798e-957e-feefe2a74fca: added peer, starting replication: peer=server-1d4f6e84-e27e-f44d-e253-fcfed5c8395c 2026-03-31T06:17:52.255Z [INFO] server-f1811d58-2110-798e-957e-feefe2a74fca: pipelining replication: peer="{Voter server-5c84d6b3-b164-8151-16b5-adb6f5d2f633 5c84d6b3-b164-8151-16b5-adb6f5d2f633}" 2026-03-31T06:17:52.255Z [INFO] server-f1811d58-2110-798e-957e-feefe2a74fca: pipelining replication: peer="{Voter server-1d4f6e84-e27e-f44d-e253-fcfed5c8395c 1d4f6e84-e27e-f44d-e253-fcfed5c8395c}" 2026-03-31T06:17:52.360Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:52.25558254 +0000 UTC m=+100.533189961 (1 nodes), 63.2532ms from start of poll, 63.8298ms from cluster start. Timeout at 2026-03-31 06:17:52.3605857 +0000 UTC m=+100.638193081, 105.00312ms after stability 2026-03-31T06:17:52.365Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:52.471Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:52.36599006 +0000 UTC m=+100.643597441 (2 nodes), 2.48µs from start of poll, 174.23728ms from cluster start. Timeout at 2026-03-31 06:17:52.47101458 +0000 UTC m=+100.748621961, 105.02452ms after stability raft_test.go:1968: [INFO] Disconnecting Node at 5c84d6b3-b164-8151-16b5-adb6f5d2f633 [Follower] 2026-03-31T06:17:52.471Z [ERROR] server-f1811d58-2110-798e-957e-feefe2a74fca: failed to heartbeat to: peer=5c84d6b3-b164-8151-16b5-adb6f5d2f633 backoff time=10ms error="failed to connect to peer: 5c84d6b3-b164-8151-16b5-adb6f5d2f633" 2026-03-31T06:17:52.471Z [INFO] server-f1811d58-2110-798e-957e-feefe2a74fca: aborting pipeline replication: peer="{Voter server-1d4f6e84-e27e-f44d-e253-fcfed5c8395c 1d4f6e84-e27e-f44d-e253-fcfed5c8395c}" 2026-03-31T06:17:52.471Z [INFO] server-f1811d58-2110-798e-957e-feefe2a74fca: aborting pipeline replication: peer="{Voter server-5c84d6b3-b164-8151-16b5-adb6f5d2f633 5c84d6b3-b164-8151-16b5-adb6f5d2f633}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.28s) === RUN TestRaft_NotifyCh 2026-03-31T06:17:52.472Z [INFO] server-232020e1-56ca-e0b2-2225-ca6a73b39f37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-232020e1-56ca-e0b2-2225-ca6a73b39f37 Address:232020e1-56ca-e0b2-2225-ca6a73b39f37}]" 2026-03-31T06:17:52.472Z [INFO] server-232020e1-56ca-e0b2-2225-ca6a73b39f37: entering follower state: follower="Node at 232020e1-56ca-e0b2-2225-ca6a73b39f37 [Follower]" leader-address= leader-id= 2026-03-31T06:17:52.542Z [WARN] server-232020e1-56ca-e0b2-2225-ca6a73b39f37: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:52.542Z [INFO] server-232020e1-56ca-e0b2-2225-ca6a73b39f37: entering candidate state: node="Node at 232020e1-56ca-e0b2-2225-ca6a73b39f37 [Candidate]" term=2 2026-03-31T06:17:52.542Z [INFO] server-232020e1-56ca-e0b2-2225-ca6a73b39f37: election won: term=2 tally=1 2026-03-31T06:17:52.542Z [INFO] server-232020e1-56ca-e0b2-2225-ca6a73b39f37: entering leader state: leader="Node at 232020e1-56ca-e0b2-2225-ca6a73b39f37 [Leader]" --- PASS: TestRaft_NotifyCh (0.07s) === RUN TestRaft_AppendEntry 2026-03-31T06:17:52.543Z [INFO] server-13117c58-ed7b-7f03-2abb-a6368f6c285e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-13117c58-ed7b-7f03-2abb-a6368f6c285e Address:13117c58-ed7b-7f03-2abb-a6368f6c285e} {Suffrage:Voter ID:server-38e0d323-0955-2963-6f5c-a788e75a9575 Address:38e0d323-0955-2963-6f5c-a788e75a9575} {Suffrage:Voter ID:server-1db37322-0578-6728-0ae6-98133a9eec9c Address:1db37322-0578-6728-0ae6-98133a9eec9c}]" 2026-03-31T06:17:52.543Z [INFO] server-38e0d323-0955-2963-6f5c-a788e75a9575: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-13117c58-ed7b-7f03-2abb-a6368f6c285e Address:13117c58-ed7b-7f03-2abb-a6368f6c285e} {Suffrage:Voter ID:server-38e0d323-0955-2963-6f5c-a788e75a9575 Address:38e0d323-0955-2963-6f5c-a788e75a9575} {Suffrage:Voter ID:server-1db37322-0578-6728-0ae6-98133a9eec9c Address:1db37322-0578-6728-0ae6-98133a9eec9c}]" 2026-03-31T06:17:52.543Z [INFO] server-13117c58-ed7b-7f03-2abb-a6368f6c285e: entering follower state: follower="Node at 13117c58-ed7b-7f03-2abb-a6368f6c285e [Follower]" leader-address= leader-id= 2026-03-31T06:17:52.543Z [INFO] server-38e0d323-0955-2963-6f5c-a788e75a9575: entering follower state: follower="Node at 38e0d323-0955-2963-6f5c-a788e75a9575 [Follower]" leader-address= leader-id= 2026-03-31T06:17:52.543Z [INFO] server-1db37322-0578-6728-0ae6-98133a9eec9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-13117c58-ed7b-7f03-2abb-a6368f6c285e Address:13117c58-ed7b-7f03-2abb-a6368f6c285e} {Suffrage:Voter ID:server-38e0d323-0955-2963-6f5c-a788e75a9575 Address:38e0d323-0955-2963-6f5c-a788e75a9575} {Suffrage:Voter ID:server-1db37322-0578-6728-0ae6-98133a9eec9c Address:1db37322-0578-6728-0ae6-98133a9eec9c}]" 2026-03-31T06:17:52.543Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:52.543Z [INFO] server-1db37322-0578-6728-0ae6-98133a9eec9c: entering follower state: follower="Node at 1db37322-0578-6728-0ae6-98133a9eec9c [Follower]" leader-address= leader-id= 2026-03-31T06:17:52.598Z [WARN] server-38e0d323-0955-2963-6f5c-a788e75a9575: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:52.598Z [INFO] server-38e0d323-0955-2963-6f5c-a788e75a9575: entering candidate state: node="Node at 38e0d323-0955-2963-6f5c-a788e75a9575 [Candidate]" term=2 2026-03-31T06:17:52.598Z [WARN] server-13117c58-ed7b-7f03-2abb-a6368f6c285e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:52.598Z [INFO] server-38e0d323-0955-2963-6f5c-a788e75a9575: election won: term=2 tally=2 2026-03-31T06:17:52.598Z [INFO] server-13117c58-ed7b-7f03-2abb-a6368f6c285e: entering candidate state: node="Node at 13117c58-ed7b-7f03-2abb-a6368f6c285e [Candidate]" term=2 2026-03-31T06:17:52.598Z [INFO] server-38e0d323-0955-2963-6f5c-a788e75a9575: entering leader state: leader="Node at 38e0d323-0955-2963-6f5c-a788e75a9575 [Leader]" 2026-03-31T06:17:52.598Z [INFO] server-38e0d323-0955-2963-6f5c-a788e75a9575: added peer, starting replication: peer=server-13117c58-ed7b-7f03-2abb-a6368f6c285e 2026-03-31T06:17:52.598Z [INFO] server-13117c58-ed7b-7f03-2abb-a6368f6c285e: duplicate requestVote for same term: term=2 2026-03-31T06:17:52.598Z [INFO] server-38e0d323-0955-2963-6f5c-a788e75a9575: added peer, starting replication: peer=server-1db37322-0578-6728-0ae6-98133a9eec9c 2026-03-31T06:17:52.598Z [INFO] server-1db37322-0578-6728-0ae6-98133a9eec9c: duplicate requestVote for same term: term=2 2026-03-31T06:17:52.598Z [WARN] server-38e0d323-0955-2963-6f5c-a788e75a9575: rejecting vote request since we have a leader: from=13117c58-ed7b-7f03-2abb-a6368f6c285e leader=38e0d323-0955-2963-6f5c-a788e75a9575 leader-id=server-38e0d323-0955-2963-6f5c-a788e75a9575 2026-03-31T06:17:52.598Z [INFO] server-13117c58-ed7b-7f03-2abb-a6368f6c285e: entering follower state: follower="Node at 13117c58-ed7b-7f03-2abb-a6368f6c285e [Follower]" leader-address=38e0d323-0955-2963-6f5c-a788e75a9575 leader-id=server-38e0d323-0955-2963-6f5c-a788e75a9575 2026-03-31T06:17:52.598Z [INFO] server-38e0d323-0955-2963-6f5c-a788e75a9575: pipelining replication: peer="{Voter server-1db37322-0578-6728-0ae6-98133a9eec9c 1db37322-0578-6728-0ae6-98133a9eec9c}" 2026-03-31T06:17:52.598Z [INFO] server-38e0d323-0955-2963-6f5c-a788e75a9575: pipelining replication: peer="{Voter server-13117c58-ed7b-7f03-2abb-a6368f6c285e 13117c58-ed7b-7f03-2abb-a6368f6c285e}" 2026-03-31T06:17:52.703Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:52.59839506 +0000 UTC m=+100.876002441 (2 nodes), 54.6056ms from start of poll, 55.05252ms from cluster start. Timeout at 2026-03-31 06:17:52.70339638 +0000 UTC m=+100.981003761, 105.00132ms after stability 2026-03-31T06:17:52.703Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:52.809Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:52.70350334 +0000 UTC m=+100.981110721 (1 nodes), 1µs from start of poll, 160.1608ms from cluster start. Timeout at 2026-03-31 06:17:52.80852314 +0000 UTC m=+101.086130561, 105.01984ms after stability 2026-03-31T06:17:52.809Z [WARN] server-13117c58-ed7b-7f03-2abb-a6368f6c285e: clearing log suffix: from=1 to=2 2026-03-31T06:17:52.809Z [WARN] server-13117c58-ed7b-7f03-2abb-a6368f6c285e: skipping application of old log: index=1 2026-03-31T06:17:52.809Z [WARN] server-13117c58-ed7b-7f03-2abb-a6368f6c285e: skipping application of old log: index=1 2026-03-31T06:17:52.809Z [INFO] server-38e0d323-0955-2963-6f5c-a788e75a9575: aborting pipeline replication: peer="{Voter server-1db37322-0578-6728-0ae6-98133a9eec9c 1db37322-0578-6728-0ae6-98133a9eec9c}" 2026-03-31T06:17:52.809Z [INFO] server-38e0d323-0955-2963-6f5c-a788e75a9575: aborting pipeline replication: peer="{Voter server-13117c58-ed7b-7f03-2abb-a6368f6c285e 13117c58-ed7b-7f03-2abb-a6368f6c285e}" --- PASS: TestRaft_AppendEntry (0.27s) === RUN TestRaft_VotingGrant_WhenLeaderAvailable 2026-03-31T06:17:52.810Z [INFO] server-8a580dde-e09a-6ab6-bebd-0cf78571a2ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a580dde-e09a-6ab6-bebd-0cf78571a2ca Address:8a580dde-e09a-6ab6-bebd-0cf78571a2ca} {Suffrage:Voter ID:server-143813ea-fa67-2096-4511-2c63c049437d Address:143813ea-fa67-2096-4511-2c63c049437d} {Suffrage:Voter ID:server-f48eacd1-ba0f-f5ae-d84d-85fb723fc820 Address:f48eacd1-ba0f-f5ae-d84d-85fb723fc820}]" 2026-03-31T06:17:52.810Z [INFO] server-143813ea-fa67-2096-4511-2c63c049437d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a580dde-e09a-6ab6-bebd-0cf78571a2ca Address:8a580dde-e09a-6ab6-bebd-0cf78571a2ca} {Suffrage:Voter ID:server-143813ea-fa67-2096-4511-2c63c049437d Address:143813ea-fa67-2096-4511-2c63c049437d} {Suffrage:Voter ID:server-f48eacd1-ba0f-f5ae-d84d-85fb723fc820 Address:f48eacd1-ba0f-f5ae-d84d-85fb723fc820}]" 2026-03-31T06:17:52.810Z [INFO] server-8a580dde-e09a-6ab6-bebd-0cf78571a2ca: entering follower state: follower="Node at 8a580dde-e09a-6ab6-bebd-0cf78571a2ca [Follower]" leader-address= leader-id= 2026-03-31T06:17:52.810Z [INFO] server-143813ea-fa67-2096-4511-2c63c049437d: entering follower state: follower="Node at 143813ea-fa67-2096-4511-2c63c049437d [Follower]" leader-address= leader-id= 2026-03-31T06:17:52.810Z [INFO] server-f48eacd1-ba0f-f5ae-d84d-85fb723fc820: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a580dde-e09a-6ab6-bebd-0cf78571a2ca Address:8a580dde-e09a-6ab6-bebd-0cf78571a2ca} {Suffrage:Voter ID:server-143813ea-fa67-2096-4511-2c63c049437d Address:143813ea-fa67-2096-4511-2c63c049437d} {Suffrage:Voter ID:server-f48eacd1-ba0f-f5ae-d84d-85fb723fc820 Address:f48eacd1-ba0f-f5ae-d84d-85fb723fc820}]" 2026-03-31T06:17:52.810Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:52.810Z [INFO] server-f48eacd1-ba0f-f5ae-d84d-85fb723fc820: entering follower state: follower="Node at f48eacd1-ba0f-f5ae-d84d-85fb723fc820 [Follower]" leader-address= leader-id= 2026-03-31T06:17:52.869Z [WARN] server-143813ea-fa67-2096-4511-2c63c049437d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:52.869Z [INFO] server-143813ea-fa67-2096-4511-2c63c049437d: entering candidate state: node="Node at 143813ea-fa67-2096-4511-2c63c049437d [Candidate]" term=2 2026-03-31T06:17:52.869Z [INFO] server-143813ea-fa67-2096-4511-2c63c049437d: election won: term=2 tally=2 2026-03-31T06:17:52.869Z [INFO] server-143813ea-fa67-2096-4511-2c63c049437d: entering leader state: leader="Node at 143813ea-fa67-2096-4511-2c63c049437d [Leader]" 2026-03-31T06:17:52.869Z [INFO] server-143813ea-fa67-2096-4511-2c63c049437d: added peer, starting replication: peer=server-8a580dde-e09a-6ab6-bebd-0cf78571a2ca 2026-03-31T06:17:52.869Z [INFO] server-143813ea-fa67-2096-4511-2c63c049437d: added peer, starting replication: peer=server-f48eacd1-ba0f-f5ae-d84d-85fb723fc820 2026-03-31T06:17:52.869Z [INFO] server-143813ea-fa67-2096-4511-2c63c049437d: pipelining replication: peer="{Voter server-f48eacd1-ba0f-f5ae-d84d-85fb723fc820 f48eacd1-ba0f-f5ae-d84d-85fb723fc820}" 2026-03-31T06:17:52.869Z [INFO] server-143813ea-fa67-2096-4511-2c63c049437d: pipelining replication: peer="{Voter server-8a580dde-e09a-6ab6-bebd-0cf78571a2ca 8a580dde-e09a-6ab6-bebd-0cf78571a2ca}" 2026-03-31T06:17:52.974Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:52.8693937 +0000 UTC m=+101.147001121 (2 nodes), 58.8408ms from start of poll, 59.24272ms from cluster start. Timeout at 2026-03-31 06:17:52.9743969 +0000 UTC m=+101.252004321, 105.0032ms after stability 2026-03-31T06:17:52.974Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:53.080Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:52.97452866 +0000 UTC m=+101.252136081 (1 nodes), 2µs from start of poll, 164.37768ms from cluster start. Timeout at 2026-03-31 06:17:53.07953118 +0000 UTC m=+101.357138601, 105.00252ms after stability 2026-03-31T06:17:53.080Z [WARN] server-f48eacd1-ba0f-f5ae-d84d-85fb723fc820: rejecting vote request since we have a leader: from=8a580dde-e09a-6ab6-bebd-0cf78571a2ca leader=143813ea-fa67-2096-4511-2c63c049437d leader-id=server-143813ea-fa67-2096-4511-2c63c049437d 2026-03-31T06:17:53.080Z [INFO] server-143813ea-fa67-2096-4511-2c63c049437d: aborting pipeline replication: peer="{Voter server-f48eacd1-ba0f-f5ae-d84d-85fb723fc820 f48eacd1-ba0f-f5ae-d84d-85fb723fc820}" 2026-03-31T06:17:53.080Z [INFO] server-143813ea-fa67-2096-4511-2c63c049437d: aborting pipeline replication: peer="{Voter server-8a580dde-e09a-6ab6-bebd-0cf78571a2ca 8a580dde-e09a-6ab6-bebd-0cf78571a2ca}" --- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.27s) === RUN TestRaft_ProtocolVersion_RejectRPC 2026-03-31T06:17:53.081Z [INFO] server-f9be55d4-f3b1-825e-a181-fa5eb5af30e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f9be55d4-f3b1-825e-a181-fa5eb5af30e3 Address:f9be55d4-f3b1-825e-a181-fa5eb5af30e3} {Suffrage:Voter ID:server-54149d2a-7229-85d0-1708-24bcf87330d6 Address:54149d2a-7229-85d0-1708-24bcf87330d6} {Suffrage:Voter ID:server-83f25b2e-0c19-1ec4-0369-ffc49f72e10a Address:83f25b2e-0c19-1ec4-0369-ffc49f72e10a}]" 2026-03-31T06:17:53.081Z [INFO] server-f9be55d4-f3b1-825e-a181-fa5eb5af30e3: entering follower state: follower="Node at f9be55d4-f3b1-825e-a181-fa5eb5af30e3 [Follower]" leader-address= leader-id= 2026-03-31T06:17:53.081Z [INFO] server-54149d2a-7229-85d0-1708-24bcf87330d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f9be55d4-f3b1-825e-a181-fa5eb5af30e3 Address:f9be55d4-f3b1-825e-a181-fa5eb5af30e3} {Suffrage:Voter ID:server-54149d2a-7229-85d0-1708-24bcf87330d6 Address:54149d2a-7229-85d0-1708-24bcf87330d6} {Suffrage:Voter ID:server-83f25b2e-0c19-1ec4-0369-ffc49f72e10a Address:83f25b2e-0c19-1ec4-0369-ffc49f72e10a}]" 2026-03-31T06:17:53.081Z [INFO] server-54149d2a-7229-85d0-1708-24bcf87330d6: entering follower state: follower="Node at 54149d2a-7229-85d0-1708-24bcf87330d6 [Follower]" leader-address= leader-id= 2026-03-31T06:17:53.081Z [INFO] server-83f25b2e-0c19-1ec4-0369-ffc49f72e10a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f9be55d4-f3b1-825e-a181-fa5eb5af30e3 Address:f9be55d4-f3b1-825e-a181-fa5eb5af30e3} {Suffrage:Voter ID:server-54149d2a-7229-85d0-1708-24bcf87330d6 Address:54149d2a-7229-85d0-1708-24bcf87330d6} {Suffrage:Voter ID:server-83f25b2e-0c19-1ec4-0369-ffc49f72e10a Address:83f25b2e-0c19-1ec4-0369-ffc49f72e10a}]" 2026-03-31T06:17:53.081Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:53.081Z [INFO] server-83f25b2e-0c19-1ec4-0369-ffc49f72e10a: entering follower state: follower="Node at 83f25b2e-0c19-1ec4-0369-ffc49f72e10a [Follower]" leader-address= leader-id= 2026-03-31T06:17:53.139Z [WARN] server-83f25b2e-0c19-1ec4-0369-ffc49f72e10a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:53.139Z [INFO] server-83f25b2e-0c19-1ec4-0369-ffc49f72e10a: entering candidate state: node="Node at 83f25b2e-0c19-1ec4-0369-ffc49f72e10a [Candidate]" term=2 2026-03-31T06:17:53.139Z [INFO] server-83f25b2e-0c19-1ec4-0369-ffc49f72e10a: election won: term=2 tally=2 2026-03-31T06:17:53.139Z [INFO] server-83f25b2e-0c19-1ec4-0369-ffc49f72e10a: entering leader state: leader="Node at 83f25b2e-0c19-1ec4-0369-ffc49f72e10a [Leader]" 2026-03-31T06:17:53.139Z [INFO] server-83f25b2e-0c19-1ec4-0369-ffc49f72e10a: added peer, starting replication: peer=server-f9be55d4-f3b1-825e-a181-fa5eb5af30e3 2026-03-31T06:17:53.139Z [INFO] server-83f25b2e-0c19-1ec4-0369-ffc49f72e10a: added peer, starting replication: peer=server-54149d2a-7229-85d0-1708-24bcf87330d6 2026-03-31T06:17:53.139Z [INFO] server-83f25b2e-0c19-1ec4-0369-ffc49f72e10a: pipelining replication: peer="{Voter server-f9be55d4-f3b1-825e-a181-fa5eb5af30e3 f9be55d4-f3b1-825e-a181-fa5eb5af30e3}" 2026-03-31T06:17:53.139Z [INFO] server-83f25b2e-0c19-1ec4-0369-ffc49f72e10a: pipelining replication: peer="{Voter server-54149d2a-7229-85d0-1708-24bcf87330d6 54149d2a-7229-85d0-1708-24bcf87330d6}" 2026-03-31T06:17:53.244Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:53.13930058 +0000 UTC m=+101.416907961 (2 nodes), 57.32572ms from start of poll, 57.72376ms from cluster start. Timeout at 2026-03-31 06:17:53.24430442 +0000 UTC m=+101.521911801, 105.00384ms after stability 2026-03-31T06:17:53.244Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:53.350Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:53.24449466 +0000 UTC m=+101.522102081 (1 nodes), 2.44µs from start of poll, 162.91788ms from cluster start. Timeout at 2026-03-31 06:17:53.3495183 +0000 UTC m=+101.627125641, 105.02356ms after stability 2026-03-31T06:17:53.350Z [INFO] server-83f25b2e-0c19-1ec4-0369-ffc49f72e10a: aborting pipeline replication: peer="{Voter server-54149d2a-7229-85d0-1708-24bcf87330d6 54149d2a-7229-85d0-1708-24bcf87330d6}" 2026-03-31T06:17:53.350Z [INFO] server-83f25b2e-0c19-1ec4-0369-ffc49f72e10a: aborting pipeline replication: peer="{Voter server-f9be55d4-f3b1-825e-a181-fa5eb5af30e3 f9be55d4-f3b1-825e-a181-fa5eb5af30e3}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 raft_test.go:2151: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 raft_test.go:2196: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s) === RUN TestRaft_LeaderID_Propagated 2026-03-31T06:17:53.351Z [INFO] server-9ad06214-66a0-99eb-f673-8adbf19e1e74: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ad06214-66a0-99eb-f673-8adbf19e1e74 Address:9ad06214-66a0-99eb-f673-8adbf19e1e74} {Suffrage:Voter ID:server-238cff00-2b22-e068-d7a4-a961a616da7f Address:238cff00-2b22-e068-d7a4-a961a616da7f} {Suffrage:Voter ID:server-312d2f75-792a-52a3-f3e7-59f6c644f871 Address:312d2f75-792a-52a3-f3e7-59f6c644f871}]" 2026-03-31T06:17:53.351Z [INFO] server-9ad06214-66a0-99eb-f673-8adbf19e1e74: entering follower state: follower="Node at 9ad06214-66a0-99eb-f673-8adbf19e1e74 [Follower]" leader-address= leader-id= 2026-03-31T06:17:53.351Z [INFO] server-238cff00-2b22-e068-d7a4-a961a616da7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ad06214-66a0-99eb-f673-8adbf19e1e74 Address:9ad06214-66a0-99eb-f673-8adbf19e1e74} {Suffrage:Voter ID:server-238cff00-2b22-e068-d7a4-a961a616da7f Address:238cff00-2b22-e068-d7a4-a961a616da7f} {Suffrage:Voter ID:server-312d2f75-792a-52a3-f3e7-59f6c644f871 Address:312d2f75-792a-52a3-f3e7-59f6c644f871}]" 2026-03-31T06:17:53.351Z [INFO] server-238cff00-2b22-e068-d7a4-a961a616da7f: entering follower state: follower="Node at 238cff00-2b22-e068-d7a4-a961a616da7f [Follower]" leader-address= leader-id= 2026-03-31T06:17:53.351Z [INFO] server-312d2f75-792a-52a3-f3e7-59f6c644f871: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ad06214-66a0-99eb-f673-8adbf19e1e74 Address:9ad06214-66a0-99eb-f673-8adbf19e1e74} {Suffrage:Voter ID:server-238cff00-2b22-e068-d7a4-a961a616da7f Address:238cff00-2b22-e068-d7a4-a961a616da7f} {Suffrage:Voter ID:server-312d2f75-792a-52a3-f3e7-59f6c644f871 Address:312d2f75-792a-52a3-f3e7-59f6c644f871}]" 2026-03-31T06:17:53.351Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:53.351Z [INFO] server-312d2f75-792a-52a3-f3e7-59f6c644f871: entering follower state: follower="Node at 312d2f75-792a-52a3-f3e7-59f6c644f871 [Follower]" leader-address= leader-id= 2026-03-31T06:17:53.415Z [WARN] server-238cff00-2b22-e068-d7a4-a961a616da7f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:53.415Z [INFO] server-238cff00-2b22-e068-d7a4-a961a616da7f: entering candidate state: node="Node at 238cff00-2b22-e068-d7a4-a961a616da7f [Candidate]" term=2 2026-03-31T06:17:53.415Z [INFO] server-238cff00-2b22-e068-d7a4-a961a616da7f: election won: term=2 tally=2 2026-03-31T06:17:53.415Z [INFO] server-238cff00-2b22-e068-d7a4-a961a616da7f: entering leader state: leader="Node at 238cff00-2b22-e068-d7a4-a961a616da7f [Leader]" 2026-03-31T06:17:53.415Z [INFO] server-238cff00-2b22-e068-d7a4-a961a616da7f: added peer, starting replication: peer=server-9ad06214-66a0-99eb-f673-8adbf19e1e74 2026-03-31T06:17:53.415Z [INFO] server-238cff00-2b22-e068-d7a4-a961a616da7f: added peer, starting replication: peer=server-312d2f75-792a-52a3-f3e7-59f6c644f871 2026-03-31T06:17:53.415Z [INFO] server-238cff00-2b22-e068-d7a4-a961a616da7f: pipelining replication: peer="{Voter server-312d2f75-792a-52a3-f3e7-59f6c644f871 312d2f75-792a-52a3-f3e7-59f6c644f871}" 2026-03-31T06:17:53.415Z [INFO] server-238cff00-2b22-e068-d7a4-a961a616da7f: pipelining replication: peer="{Voter server-9ad06214-66a0-99eb-f673-8adbf19e1e74 9ad06214-66a0-99eb-f673-8adbf19e1e74}" 2026-03-31T06:17:53.521Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:53.41589786 +0000 UTC m=+101.693505281 (1 nodes), 64.2676ms from start of poll, 64.70696ms from cluster start. Timeout at 2026-03-31 06:17:53.52090242 +0000 UTC m=+101.798509801, 105.00452ms after stability 2026-03-31T06:17:53.521Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:53.626Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:53.52110262 +0000 UTC m=+101.798710001 (1 nodes), 1.6µs from start of poll, 169.91168ms from cluster start. Timeout at 2026-03-31 06:17:53.62613534 +0000 UTC m=+101.903742761, 105.03276ms after stability 2026-03-31T06:17:53.626Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:53.732Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:53.62644658 +0000 UTC m=+101.904053961 (1 nodes), 1.12µs from start of poll, 275.25564ms from cluster start. Timeout at 2026-03-31 06:17:53.73145758 +0000 UTC m=+102.009065001, 105.01104ms after stability 2026-03-31T06:17:53.732Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:53.838Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:53.73273042 +0000 UTC m=+102.010337801 (1 nodes), 1.12µs from start of poll, 381.53948ms from cluster start. Timeout at 2026-03-31 06:17:53.8378405 +0000 UTC m=+102.115447921, 105.11012ms after stability 2026-03-31T06:17:53.839Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:53.946Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:53.83991818 +0000 UTC m=+102.117525561 (1 nodes), 1.44µs from start of poll, 488.72724ms from cluster start. Timeout at 2026-03-31 06:17:53.94494674 +0000 UTC m=+102.222554121, 105.02856ms after stability 2026-03-31T06:17:53.946Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:54.051Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:53.94634642 +0000 UTC m=+102.223953801 (1 nodes), 920ns from start of poll, 595.15548ms from cluster start. Timeout at 2026-03-31 06:17:54.05137686 +0000 UTC m=+102.328984241, 105.03044ms after stability 2026-03-31T06:17:54.157Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:54.263Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:54.15784026 +0000 UTC m=+102.435447641 (1 nodes), 1.48µs from start of poll, 806.64932ms from cluster start. Timeout at 2026-03-31 06:17:54.26287306 +0000 UTC m=+102.540480441, 105.0328ms after stability 2026-03-31T06:17:54.263Z [INFO] server-238cff00-2b22-e068-d7a4-a961a616da7f: aborting pipeline replication: peer="{Voter server-312d2f75-792a-52a3-f3e7-59f6c644f871 312d2f75-792a-52a3-f3e7-59f6c644f871}" 2026-03-31T06:17:54.263Z [INFO] server-238cff00-2b22-e068-d7a4-a961a616da7f: aborting pipeline replication: peer="{Voter server-9ad06214-66a0-99eb-f673-8adbf19e1e74 9ad06214-66a0-99eb-f673-8adbf19e1e74}" --- PASS: TestRaft_LeaderID_Propagated (0.91s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer 2026-03-31T06:17:54.264Z [INFO] server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e Address:28bafaf5-b703-a348-8ff2-d64ce13d2a7e} {Suffrage:Voter ID:server-42321323-0935-74f8-14d4-11529617ffd5 Address:42321323-0935-74f8-14d4-11529617ffd5} {Suffrage:Voter ID:server-5f4161dd-7402-be8e-b193-c378fe7341f4 Address:5f4161dd-7402-be8e-b193-c378fe7341f4}]" 2026-03-31T06:17:54.264Z [INFO] server-42321323-0935-74f8-14d4-11529617ffd5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e Address:28bafaf5-b703-a348-8ff2-d64ce13d2a7e} {Suffrage:Voter ID:server-42321323-0935-74f8-14d4-11529617ffd5 Address:42321323-0935-74f8-14d4-11529617ffd5} {Suffrage:Voter ID:server-5f4161dd-7402-be8e-b193-c378fe7341f4 Address:5f4161dd-7402-be8e-b193-c378fe7341f4}]" 2026-03-31T06:17:54.264Z [INFO] server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e: entering follower state: follower="Node at 28bafaf5-b703-a348-8ff2-d64ce13d2a7e [Follower]" leader-address= leader-id= 2026-03-31T06:17:54.264Z [INFO] server-42321323-0935-74f8-14d4-11529617ffd5: entering follower state: follower="Node at 42321323-0935-74f8-14d4-11529617ffd5 [Follower]" leader-address= leader-id= 2026-03-31T06:17:54.264Z [INFO] server-5f4161dd-7402-be8e-b193-c378fe7341f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e Address:28bafaf5-b703-a348-8ff2-d64ce13d2a7e} {Suffrage:Voter ID:server-42321323-0935-74f8-14d4-11529617ffd5 Address:42321323-0935-74f8-14d4-11529617ffd5} {Suffrage:Voter ID:server-5f4161dd-7402-be8e-b193-c378fe7341f4 Address:5f4161dd-7402-be8e-b193-c378fe7341f4}]" 2026-03-31T06:17:54.264Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:54.264Z [INFO] server-5f4161dd-7402-be8e-b193-c378fe7341f4: entering follower state: follower="Node at 5f4161dd-7402-be8e-b193-c378fe7341f4 [Follower]" leader-address= leader-id= 2026-03-31T06:17:54.318Z [WARN] server-42321323-0935-74f8-14d4-11529617ffd5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:54.318Z [INFO] server-42321323-0935-74f8-14d4-11529617ffd5: entering candidate state: node="Node at 42321323-0935-74f8-14d4-11529617ffd5 [Candidate]" term=2 2026-03-31T06:17:54.318Z [INFO] server-42321323-0935-74f8-14d4-11529617ffd5: election won: term=2 tally=2 2026-03-31T06:17:54.318Z [INFO] server-42321323-0935-74f8-14d4-11529617ffd5: entering leader state: leader="Node at 42321323-0935-74f8-14d4-11529617ffd5 [Leader]" 2026-03-31T06:17:54.318Z [INFO] server-42321323-0935-74f8-14d4-11529617ffd5: added peer, starting replication: peer=server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e 2026-03-31T06:17:54.319Z [INFO] server-42321323-0935-74f8-14d4-11529617ffd5: added peer, starting replication: peer=server-5f4161dd-7402-be8e-b193-c378fe7341f4 2026-03-31T06:17:54.319Z [INFO] server-42321323-0935-74f8-14d4-11529617ffd5: pipelining replication: peer="{Voter server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e 28bafaf5-b703-a348-8ff2-d64ce13d2a7e}" 2026-03-31T06:17:54.319Z [INFO] server-42321323-0935-74f8-14d4-11529617ffd5: pipelining replication: peer="{Voter server-5f4161dd-7402-be8e-b193-c378fe7341f4 5f4161dd-7402-be8e-b193-c378fe7341f4}" 2026-03-31T06:17:54.424Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:54.3190529 +0000 UTC m=+102.596660241 (1 nodes), 54.33736ms from start of poll, 54.74944ms from cluster start. Timeout at 2026-03-31 06:17:54.4240541 +0000 UTC m=+102.701661481, 105.00124ms after stability 2026-03-31T06:17:54.424Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:54.532Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:54.42417582 +0000 UTC m=+102.701783201 (1 nodes), 2.24µs from start of poll, 159.8724ms from cluster start. Timeout at 2026-03-31 06:17:54.52920934 +0000 UTC m=+102.806816721, 105.03352ms after stability 2026-03-31T06:17:54.532Z [INFO] server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e: entering candidate state: node="Node at 28bafaf5-b703-a348-8ff2-d64ce13d2a7e [Candidate]" term=3 2026-03-31T06:17:54.532Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:54.533Z [INFO] server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e: election won: term=3 tally=2 2026-03-31T06:17:54.533Z [INFO] server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e: entering leader state: leader="Node at 28bafaf5-b703-a348-8ff2-d64ce13d2a7e [Leader]" 2026-03-31T06:17:54.533Z [INFO] server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e: added peer, starting replication: peer=server-42321323-0935-74f8-14d4-11529617ffd5 2026-03-31T06:17:54.533Z [INFO] server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e: added peer, starting replication: peer=server-5f4161dd-7402-be8e-b193-c378fe7341f4 2026-03-31T06:17:54.533Z [INFO] server-42321323-0935-74f8-14d4-11529617ffd5: entering follower state: follower="Node at 42321323-0935-74f8-14d4-11529617ffd5 [Follower]" leader-address= leader-id= 2026-03-31T06:17:54.533Z [INFO] server-42321323-0935-74f8-14d4-11529617ffd5: aborting pipeline replication: peer="{Voter server-5f4161dd-7402-be8e-b193-c378fe7341f4 5f4161dd-7402-be8e-b193-c378fe7341f4}" 2026-03-31T06:17:54.533Z [INFO] server-42321323-0935-74f8-14d4-11529617ffd5: aborting pipeline replication: peer="{Voter server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e 28bafaf5-b703-a348-8ff2-d64ce13d2a7e}" 2026-03-31T06:17:54.533Z [INFO] server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e: pipelining replication: peer="{Voter server-5f4161dd-7402-be8e-b193-c378fe7341f4 5f4161dd-7402-be8e-b193-c378fe7341f4}" 2026-03-31T06:17:54.533Z [INFO] server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e: pipelining replication: peer="{Voter server-42321323-0935-74f8-14d4-11529617ffd5 42321323-0935-74f8-14d4-11529617ffd5}" 2026-03-31T06:17:54.639Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:54.53317618 +0000 UTC m=+102.810783561 (1 nodes), 159.16µs from start of poll, 268.87276ms from cluster start. Timeout at 2026-03-31 06:17:54.63822706 +0000 UTC m=+102.915834481, 105.05092ms after stability 2026-03-31T06:17:54.639Z [INFO] server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e: aborting pipeline replication: peer="{Voter server-5f4161dd-7402-be8e-b193-c378fe7341f4 5f4161dd-7402-be8e-b193-c378fe7341f4}" 2026-03-31T06:17:54.639Z [INFO] server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e: aborting pipeline replication: peer="{Voter server-42321323-0935-74f8-14d4-11529617ffd5 42321323-0935-74f8-14d4-11529617ffd5}" 2026-03-31T06:17:55.140Z [ERROR] server-28bafaf5-b703-a348-8ff2-d64ce13d2a7e: failed to heartbeat to: peer=42321323-0935-74f8-14d4-11529617ffd5 backoff time=10ms error="command timed out" --- PASS: TestRaft_LeadershipTransfer (0.88s) === RUN TestRaft_LeadershipTransferWithOneNode 2026-03-31T06:17:55.140Z [INFO] server-c87bd9b4-9167-0128-e20e-1c0c45390350: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c87bd9b4-9167-0128-e20e-1c0c45390350 Address:c87bd9b4-9167-0128-e20e-1c0c45390350}]" 2026-03-31T06:17:55.140Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:55.140Z [INFO] server-c87bd9b4-9167-0128-e20e-1c0c45390350: entering follower state: follower="Node at c87bd9b4-9167-0128-e20e-1c0c45390350 [Follower]" leader-address= leader-id= 2026-03-31T06:17:55.230Z [WARN] server-c87bd9b4-9167-0128-e20e-1c0c45390350: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:55.230Z [INFO] server-c87bd9b4-9167-0128-e20e-1c0c45390350: entering candidate state: node="Node at c87bd9b4-9167-0128-e20e-1c0c45390350 [Candidate]" term=2 2026-03-31T06:17:55.230Z [INFO] server-c87bd9b4-9167-0128-e20e-1c0c45390350: election won: term=2 tally=1 2026-03-31T06:17:55.230Z [INFO] server-c87bd9b4-9167-0128-e20e-1c0c45390350: entering leader state: leader="Node at c87bd9b4-9167-0128-e20e-1c0c45390350 [Leader]" 2026-03-31T06:17:55.336Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:55.23077618 +0000 UTC m=+103.508383561 (1 nodes), 89.97208ms from start of poll, 90.17472ms from cluster start. Timeout at 2026-03-31 06:17:55.33578742 +0000 UTC m=+103.613394841, 105.01128ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.20s) === RUN TestRaft_LeadershipTransferWithSevenNodes 2026-03-31T06:17:55.337Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a7ed411-0692-79f0-8010-6607591de316 Address:9a7ed411-0692-79f0-8010-6607591de316} {Suffrage:Voter ID:server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e Address:feae006a-0b80-ffd2-cb0c-c56f84f3a37e} {Suffrage:Voter ID:server-cb7301ff-2b28-8211-f4b1-8cfd8f000433 Address:cb7301ff-2b28-8211-f4b1-8cfd8f000433} {Suffrage:Voter ID:server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0 Address:c43c11f2-dcf5-530a-0d7f-c8b7802c61c0} {Suffrage:Voter ID:server-4138e958-1dbb-9700-a44a-c7ee468a4105 Address:4138e958-1dbb-9700-a44a-c7ee468a4105} {Suffrage:Voter ID:server-5439fe31-2692-fca0-5c05-cba1f545aaad Address:5439fe31-2692-fca0-5c05-cba1f545aaad} {Suffrage:Voter ID:server-61802f6b-b1ae-9489-8ef7-908db117bfe4 Address:61802f6b-b1ae-9489-8ef7-908db117bfe4}]" 2026-03-31T06:17:55.337Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a7ed411-0692-79f0-8010-6607591de316 Address:9a7ed411-0692-79f0-8010-6607591de316} {Suffrage:Voter ID:server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e Address:feae006a-0b80-ffd2-cb0c-c56f84f3a37e} {Suffrage:Voter ID:server-cb7301ff-2b28-8211-f4b1-8cfd8f000433 Address:cb7301ff-2b28-8211-f4b1-8cfd8f000433} {Suffrage:Voter ID:server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0 Address:c43c11f2-dcf5-530a-0d7f-c8b7802c61c0} {Suffrage:Voter ID:server-4138e958-1dbb-9700-a44a-c7ee468a4105 Address:4138e958-1dbb-9700-a44a-c7ee468a4105} {Suffrage:Voter ID:server-5439fe31-2692-fca0-5c05-cba1f545aaad Address:5439fe31-2692-fca0-5c05-cba1f545aaad} {Suffrage:Voter ID:server-61802f6b-b1ae-9489-8ef7-908db117bfe4 Address:61802f6b-b1ae-9489-8ef7-908db117bfe4}]" 2026-03-31T06:17:55.337Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: entering follower state: follower="Node at 9a7ed411-0692-79f0-8010-6607591de316 [Follower]" leader-address= leader-id= 2026-03-31T06:17:55.337Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: entering follower state: follower="Node at feae006a-0b80-ffd2-cb0c-c56f84f3a37e [Follower]" leader-address= leader-id= 2026-03-31T06:17:55.337Z [INFO] server-cb7301ff-2b28-8211-f4b1-8cfd8f000433: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a7ed411-0692-79f0-8010-6607591de316 Address:9a7ed411-0692-79f0-8010-6607591de316} {Suffrage:Voter ID:server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e Address:feae006a-0b80-ffd2-cb0c-c56f84f3a37e} {Suffrage:Voter ID:server-cb7301ff-2b28-8211-f4b1-8cfd8f000433 Address:cb7301ff-2b28-8211-f4b1-8cfd8f000433} {Suffrage:Voter ID:server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0 Address:c43c11f2-dcf5-530a-0d7f-c8b7802c61c0} {Suffrage:Voter ID:server-4138e958-1dbb-9700-a44a-c7ee468a4105 Address:4138e958-1dbb-9700-a44a-c7ee468a4105} {Suffrage:Voter ID:server-5439fe31-2692-fca0-5c05-cba1f545aaad Address:5439fe31-2692-fca0-5c05-cba1f545aaad} {Suffrage:Voter ID:server-61802f6b-b1ae-9489-8ef7-908db117bfe4 Address:61802f6b-b1ae-9489-8ef7-908db117bfe4}]" 2026-03-31T06:17:55.337Z [INFO] server-cb7301ff-2b28-8211-f4b1-8cfd8f000433: entering follower state: follower="Node at cb7301ff-2b28-8211-f4b1-8cfd8f000433 [Follower]" leader-address= leader-id= 2026-03-31T06:17:55.337Z [INFO] server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a7ed411-0692-79f0-8010-6607591de316 Address:9a7ed411-0692-79f0-8010-6607591de316} {Suffrage:Voter ID:server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e Address:feae006a-0b80-ffd2-cb0c-c56f84f3a37e} {Suffrage:Voter ID:server-cb7301ff-2b28-8211-f4b1-8cfd8f000433 Address:cb7301ff-2b28-8211-f4b1-8cfd8f000433} {Suffrage:Voter ID:server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0 Address:c43c11f2-dcf5-530a-0d7f-c8b7802c61c0} {Suffrage:Voter ID:server-4138e958-1dbb-9700-a44a-c7ee468a4105 Address:4138e958-1dbb-9700-a44a-c7ee468a4105} {Suffrage:Voter ID:server-5439fe31-2692-fca0-5c05-cba1f545aaad Address:5439fe31-2692-fca0-5c05-cba1f545aaad} {Suffrage:Voter ID:server-61802f6b-b1ae-9489-8ef7-908db117bfe4 Address:61802f6b-b1ae-9489-8ef7-908db117bfe4}]" 2026-03-31T06:17:55.337Z [INFO] server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0: entering follower state: follower="Node at c43c11f2-dcf5-530a-0d7f-c8b7802c61c0 [Follower]" leader-address= leader-id= 2026-03-31T06:17:55.337Z [INFO] server-4138e958-1dbb-9700-a44a-c7ee468a4105: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a7ed411-0692-79f0-8010-6607591de316 Address:9a7ed411-0692-79f0-8010-6607591de316} {Suffrage:Voter ID:server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e Address:feae006a-0b80-ffd2-cb0c-c56f84f3a37e} {Suffrage:Voter ID:server-cb7301ff-2b28-8211-f4b1-8cfd8f000433 Address:cb7301ff-2b28-8211-f4b1-8cfd8f000433} {Suffrage:Voter ID:server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0 Address:c43c11f2-dcf5-530a-0d7f-c8b7802c61c0} {Suffrage:Voter ID:server-4138e958-1dbb-9700-a44a-c7ee468a4105 Address:4138e958-1dbb-9700-a44a-c7ee468a4105} {Suffrage:Voter ID:server-5439fe31-2692-fca0-5c05-cba1f545aaad Address:5439fe31-2692-fca0-5c05-cba1f545aaad} {Suffrage:Voter ID:server-61802f6b-b1ae-9489-8ef7-908db117bfe4 Address:61802f6b-b1ae-9489-8ef7-908db117bfe4}]" 2026-03-31T06:17:55.338Z [INFO] server-4138e958-1dbb-9700-a44a-c7ee468a4105: entering follower state: follower="Node at 4138e958-1dbb-9700-a44a-c7ee468a4105 [Follower]" leader-address= leader-id= 2026-03-31T06:17:55.338Z [INFO] server-5439fe31-2692-fca0-5c05-cba1f545aaad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a7ed411-0692-79f0-8010-6607591de316 Address:9a7ed411-0692-79f0-8010-6607591de316} {Suffrage:Voter ID:server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e Address:feae006a-0b80-ffd2-cb0c-c56f84f3a37e} {Suffrage:Voter ID:server-cb7301ff-2b28-8211-f4b1-8cfd8f000433 Address:cb7301ff-2b28-8211-f4b1-8cfd8f000433} {Suffrage:Voter ID:server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0 Address:c43c11f2-dcf5-530a-0d7f-c8b7802c61c0} {Suffrage:Voter ID:server-4138e958-1dbb-9700-a44a-c7ee468a4105 Address:4138e958-1dbb-9700-a44a-c7ee468a4105} {Suffrage:Voter ID:server-5439fe31-2692-fca0-5c05-cba1f545aaad Address:5439fe31-2692-fca0-5c05-cba1f545aaad} {Suffrage:Voter ID:server-61802f6b-b1ae-9489-8ef7-908db117bfe4 Address:61802f6b-b1ae-9489-8ef7-908db117bfe4}]" 2026-03-31T06:17:55.338Z [INFO] server-5439fe31-2692-fca0-5c05-cba1f545aaad: entering follower state: follower="Node at 5439fe31-2692-fca0-5c05-cba1f545aaad [Follower]" leader-address= leader-id= 2026-03-31T06:17:55.338Z [INFO] server-61802f6b-b1ae-9489-8ef7-908db117bfe4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a7ed411-0692-79f0-8010-6607591de316 Address:9a7ed411-0692-79f0-8010-6607591de316} {Suffrage:Voter ID:server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e Address:feae006a-0b80-ffd2-cb0c-c56f84f3a37e} {Suffrage:Voter ID:server-cb7301ff-2b28-8211-f4b1-8cfd8f000433 Address:cb7301ff-2b28-8211-f4b1-8cfd8f000433} {Suffrage:Voter ID:server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0 Address:c43c11f2-dcf5-530a-0d7f-c8b7802c61c0} {Suffrage:Voter ID:server-4138e958-1dbb-9700-a44a-c7ee468a4105 Address:4138e958-1dbb-9700-a44a-c7ee468a4105} {Suffrage:Voter ID:server-5439fe31-2692-fca0-5c05-cba1f545aaad Address:5439fe31-2692-fca0-5c05-cba1f545aaad} {Suffrage:Voter ID:server-61802f6b-b1ae-9489-8ef7-908db117bfe4 Address:61802f6b-b1ae-9489-8ef7-908db117bfe4}]" 2026-03-31T06:17:55.338Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:55.338Z [INFO] server-61802f6b-b1ae-9489-8ef7-908db117bfe4: entering follower state: follower="Node at 61802f6b-b1ae-9489-8ef7-908db117bfe4 [Follower]" leader-address= leader-id= 2026-03-31T06:17:55.390Z [WARN] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:55.390Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: entering candidate state: node="Node at feae006a-0b80-ffd2-cb0c-c56f84f3a37e [Candidate]" term=2 2026-03-31T06:17:55.390Z [WARN] server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:55.390Z [INFO] server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0: entering candidate state: node="Node at c43c11f2-dcf5-530a-0d7f-c8b7802c61c0 [Candidate]" term=2 2026-03-31T06:17:55.390Z [INFO] server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0: duplicate requestVote for same term: term=2 2026-03-31T06:17:55.390Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: election won: term=2 tally=4 2026-03-31T06:17:55.390Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: entering leader state: leader="Node at feae006a-0b80-ffd2-cb0c-c56f84f3a37e [Leader]" 2026-03-31T06:17:55.390Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: duplicate requestVote for same term: term=2 2026-03-31T06:17:55.390Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: added peer, starting replication: peer=server-9a7ed411-0692-79f0-8010-6607591de316 2026-03-31T06:17:55.390Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: added peer, starting replication: peer=server-cb7301ff-2b28-8211-f4b1-8cfd8f000433 2026-03-31T06:17:55.390Z [INFO] server-5439fe31-2692-fca0-5c05-cba1f545aaad: duplicate requestVote for same term: term=2 2026-03-31T06:17:55.390Z [INFO] server-61802f6b-b1ae-9489-8ef7-908db117bfe4: duplicate requestVote for same term: term=2 2026-03-31T06:17:55.390Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: added peer, starting replication: peer=server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0 2026-03-31T06:17:55.390Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: added peer, starting replication: peer=server-4138e958-1dbb-9700-a44a-c7ee468a4105 2026-03-31T06:17:55.390Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: added peer, starting replication: peer=server-5439fe31-2692-fca0-5c05-cba1f545aaad 2026-03-31T06:17:55.390Z [INFO] server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0: entering follower state: follower="Node at c43c11f2-dcf5-530a-0d7f-c8b7802c61c0 [Follower]" leader-address=feae006a-0b80-ffd2-cb0c-c56f84f3a37e leader-id=server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e 2026-03-31T06:17:55.390Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: pipelining replication: peer="{Voter server-9a7ed411-0692-79f0-8010-6607591de316 9a7ed411-0692-79f0-8010-6607591de316}" 2026-03-31T06:17:55.390Z [INFO] server-cb7301ff-2b28-8211-f4b1-8cfd8f000433: duplicate requestVote for same term: term=2 2026-03-31T06:17:55.390Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: added peer, starting replication: peer=server-61802f6b-b1ae-9489-8ef7-908db117bfe4 2026-03-31T06:17:55.390Z [INFO] server-4138e958-1dbb-9700-a44a-c7ee468a4105: duplicate requestVote for same term: term=2 2026-03-31T06:17:55.390Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: pipelining replication: peer="{Voter server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0 c43c11f2-dcf5-530a-0d7f-c8b7802c61c0}" 2026-03-31T06:17:55.390Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: pipelining replication: peer="{Voter server-5439fe31-2692-fca0-5c05-cba1f545aaad 5439fe31-2692-fca0-5c05-cba1f545aaad}" 2026-03-31T06:17:55.390Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: pipelining replication: peer="{Voter server-cb7301ff-2b28-8211-f4b1-8cfd8f000433 cb7301ff-2b28-8211-f4b1-8cfd8f000433}" 2026-03-31T06:17:55.390Z [WARN] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: rejecting vote request since we have a leader: from=c43c11f2-dcf5-530a-0d7f-c8b7802c61c0 leader=feae006a-0b80-ffd2-cb0c-c56f84f3a37e leader-id=server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e 2026-03-31T06:17:55.390Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: pipelining replication: peer="{Voter server-4138e958-1dbb-9700-a44a-c7ee468a4105 4138e958-1dbb-9700-a44a-c7ee468a4105}" 2026-03-31T06:17:55.390Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: pipelining replication: peer="{Voter server-61802f6b-b1ae-9489-8ef7-908db117bfe4 61802f6b-b1ae-9489-8ef7-908db117bfe4}" 2026-03-31T06:17:55.497Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:55.39068054 +0000 UTC m=+103.668287921 (1 nodes), 52.41224ms from start of poll, 53.37788ms from cluster start. Timeout at 2026-03-31 06:17:55.49568286 +0000 UTC m=+103.773290281, 105.00236ms after stability 2026-03-31T06:17:55.497Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:55.602Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:55.49748122 +0000 UTC m=+103.775088601 (6 nodes), 3.12µs from start of poll, 160.17856ms from cluster start. Timeout at 2026-03-31 06:17:55.60251366 +0000 UTC m=+103.880121041, 105.03244ms after stability 2026-03-31T06:17:55.603Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:55.708Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:55.60302198 +0000 UTC m=+103.880629401 (1 nodes), 2.88µs from start of poll, 265.71936ms from cluster start. Timeout at 2026-03-31 06:17:55.70806326 +0000 UTC m=+103.985670681, 105.04128ms after stability 2026-03-31T06:17:55.708Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: entering candidate state: node="Node at 9a7ed411-0692-79f0-8010-6607591de316 [Candidate]" term=3 2026-03-31T06:17:55.708Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: election won: term=3 tally=4 2026-03-31T06:17:55.708Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: entering leader state: leader="Node at 9a7ed411-0692-79f0-8010-6607591de316 [Leader]" 2026-03-31T06:17:55.708Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: added peer, starting replication: peer=server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e 2026-03-31T06:17:55.708Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: entering follower state: follower="Node at feae006a-0b80-ffd2-cb0c-c56f84f3a37e [Follower]" leader-address= leader-id= 2026-03-31T06:17:55.709Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: added peer, starting replication: peer=server-cb7301ff-2b28-8211-f4b1-8cfd8f000433 2026-03-31T06:17:55.709Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: added peer, starting replication: peer=server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0 2026-03-31T06:17:55.709Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: aborting pipeline replication: peer="{Voter server-4138e958-1dbb-9700-a44a-c7ee468a4105 4138e958-1dbb-9700-a44a-c7ee468a4105}" 2026-03-31T06:17:55.709Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: added peer, starting replication: peer=server-4138e958-1dbb-9700-a44a-c7ee468a4105 2026-03-31T06:17:55.709Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: added peer, starting replication: peer=server-5439fe31-2692-fca0-5c05-cba1f545aaad 2026-03-31T06:17:55.709Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: added peer, starting replication: peer=server-61802f6b-b1ae-9489-8ef7-908db117bfe4 2026-03-31T06:17:55.709Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: pipelining replication: peer="{Voter server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0 c43c11f2-dcf5-530a-0d7f-c8b7802c61c0}" 2026-03-31T06:17:55.708Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:55.709Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: aborting pipeline replication: peer="{Voter server-9a7ed411-0692-79f0-8010-6607591de316 9a7ed411-0692-79f0-8010-6607591de316}" 2026-03-31T06:17:55.709Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: aborting pipeline replication: peer="{Voter server-5439fe31-2692-fca0-5c05-cba1f545aaad 5439fe31-2692-fca0-5c05-cba1f545aaad}" 2026-03-31T06:17:55.709Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: pipelining replication: peer="{Voter server-cb7301ff-2b28-8211-f4b1-8cfd8f000433 cb7301ff-2b28-8211-f4b1-8cfd8f000433}" 2026-03-31T06:17:55.709Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: aborting pipeline replication: peer="{Voter server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0 c43c11f2-dcf5-530a-0d7f-c8b7802c61c0}" 2026-03-31T06:17:55.709Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: aborting pipeline replication: peer="{Voter server-61802f6b-b1ae-9489-8ef7-908db117bfe4 61802f6b-b1ae-9489-8ef7-908db117bfe4}" 2026-03-31T06:17:55.709Z [INFO] server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e: aborting pipeline replication: peer="{Voter server-cb7301ff-2b28-8211-f4b1-8cfd8f000433 cb7301ff-2b28-8211-f4b1-8cfd8f000433}" 2026-03-31T06:17:55.709Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: pipelining replication: peer="{Voter server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e feae006a-0b80-ffd2-cb0c-c56f84f3a37e}" 2026-03-31T06:17:55.709Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: pipelining replication: peer="{Voter server-61802f6b-b1ae-9489-8ef7-908db117bfe4 61802f6b-b1ae-9489-8ef7-908db117bfe4}" 2026-03-31T06:17:55.709Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: pipelining replication: peer="{Voter server-4138e958-1dbb-9700-a44a-c7ee468a4105 4138e958-1dbb-9700-a44a-c7ee468a4105}" 2026-03-31T06:17:55.709Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: pipelining replication: peer="{Voter server-5439fe31-2692-fca0-5c05-cba1f545aaad 5439fe31-2692-fca0-5c05-cba1f545aaad}" 2026-03-31T06:17:55.814Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:55.7092651 +0000 UTC m=+103.986872521 (1 nodes), 67.76µs from start of poll, 371.96248ms from cluster start. Timeout at 2026-03-31 06:17:55.81428674 +0000 UTC m=+104.091894121, 105.0216ms after stability 2026-03-31T06:17:55.815Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: aborting pipeline replication: peer="{Voter server-4138e958-1dbb-9700-a44a-c7ee468a4105 4138e958-1dbb-9700-a44a-c7ee468a4105}" 2026-03-31T06:17:55.815Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: aborting pipeline replication: peer="{Voter server-feae006a-0b80-ffd2-cb0c-c56f84f3a37e feae006a-0b80-ffd2-cb0c-c56f84f3a37e}" 2026-03-31T06:17:55.815Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: aborting pipeline replication: peer="{Voter server-cb7301ff-2b28-8211-f4b1-8cfd8f000433 cb7301ff-2b28-8211-f4b1-8cfd8f000433}" 2026-03-31T06:17:55.815Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: aborting pipeline replication: peer="{Voter server-c43c11f2-dcf5-530a-0d7f-c8b7802c61c0 c43c11f2-dcf5-530a-0d7f-c8b7802c61c0}" 2026-03-31T06:17:55.815Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: aborting pipeline replication: peer="{Voter server-5439fe31-2692-fca0-5c05-cba1f545aaad 5439fe31-2692-fca0-5c05-cba1f545aaad}" 2026-03-31T06:17:55.815Z [INFO] server-9a7ed411-0692-79f0-8010-6607591de316: aborting pipeline replication: peer="{Voter server-61802f6b-b1ae-9489-8ef7-908db117bfe4 61802f6b-b1ae-9489-8ef7-908db117bfe4}" 2026-03-31T06:17:56.315Z [ERROR] server-9a7ed411-0692-79f0-8010-6607591de316: failed to heartbeat to: peer=61802f6b-b1ae-9489-8ef7-908db117bfe4 backoff time=10ms error="command timed out" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.98s) === RUN TestRaft_LeadershipTransferToInvalidID 2026-03-31T06:17:56.317Z [INFO] server-7308c39c-024d-0376-de4b-a2770ef69f2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7308c39c-024d-0376-de4b-a2770ef69f2a Address:7308c39c-024d-0376-de4b-a2770ef69f2a} {Suffrage:Voter ID:server-a02a1841-0706-856f-0dd0-97446733c13f Address:a02a1841-0706-856f-0dd0-97446733c13f} {Suffrage:Voter ID:server-59d494bb-b3db-67e0-91ae-161345bcab27 Address:59d494bb-b3db-67e0-91ae-161345bcab27}]" 2026-03-31T06:17:56.317Z [INFO] server-a02a1841-0706-856f-0dd0-97446733c13f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7308c39c-024d-0376-de4b-a2770ef69f2a Address:7308c39c-024d-0376-de4b-a2770ef69f2a} {Suffrage:Voter ID:server-a02a1841-0706-856f-0dd0-97446733c13f Address:a02a1841-0706-856f-0dd0-97446733c13f} {Suffrage:Voter ID:server-59d494bb-b3db-67e0-91ae-161345bcab27 Address:59d494bb-b3db-67e0-91ae-161345bcab27}]" 2026-03-31T06:17:56.317Z [INFO] server-7308c39c-024d-0376-de4b-a2770ef69f2a: entering follower state: follower="Node at 7308c39c-024d-0376-de4b-a2770ef69f2a [Follower]" leader-address= leader-id= 2026-03-31T06:17:56.317Z [INFO] server-a02a1841-0706-856f-0dd0-97446733c13f: entering follower state: follower="Node at a02a1841-0706-856f-0dd0-97446733c13f [Follower]" leader-address= leader-id= 2026-03-31T06:17:56.317Z [INFO] server-59d494bb-b3db-67e0-91ae-161345bcab27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7308c39c-024d-0376-de4b-a2770ef69f2a Address:7308c39c-024d-0376-de4b-a2770ef69f2a} {Suffrage:Voter ID:server-a02a1841-0706-856f-0dd0-97446733c13f Address:a02a1841-0706-856f-0dd0-97446733c13f} {Suffrage:Voter ID:server-59d494bb-b3db-67e0-91ae-161345bcab27 Address:59d494bb-b3db-67e0-91ae-161345bcab27}]" 2026-03-31T06:17:56.317Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:56.317Z [INFO] server-59d494bb-b3db-67e0-91ae-161345bcab27: entering follower state: follower="Node at 59d494bb-b3db-67e0-91ae-161345bcab27 [Follower]" leader-address= leader-id= 2026-03-31T06:17:56.401Z [WARN] server-59d494bb-b3db-67e0-91ae-161345bcab27: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:56.401Z [INFO] server-59d494bb-b3db-67e0-91ae-161345bcab27: entering candidate state: node="Node at 59d494bb-b3db-67e0-91ae-161345bcab27 [Candidate]" term=2 2026-03-31T06:17:56.401Z [INFO] server-59d494bb-b3db-67e0-91ae-161345bcab27: election won: term=2 tally=2 2026-03-31T06:17:56.401Z [INFO] server-59d494bb-b3db-67e0-91ae-161345bcab27: entering leader state: leader="Node at 59d494bb-b3db-67e0-91ae-161345bcab27 [Leader]" 2026-03-31T06:17:56.401Z [INFO] server-59d494bb-b3db-67e0-91ae-161345bcab27: added peer, starting replication: peer=server-7308c39c-024d-0376-de4b-a2770ef69f2a 2026-03-31T06:17:56.401Z [INFO] server-59d494bb-b3db-67e0-91ae-161345bcab27: added peer, starting replication: peer=server-a02a1841-0706-856f-0dd0-97446733c13f 2026-03-31T06:17:56.401Z [INFO] server-59d494bb-b3db-67e0-91ae-161345bcab27: pipelining replication: peer="{Voter server-7308c39c-024d-0376-de4b-a2770ef69f2a 7308c39c-024d-0376-de4b-a2770ef69f2a}" 2026-03-31T06:17:56.401Z [INFO] server-59d494bb-b3db-67e0-91ae-161345bcab27: pipelining replication: peer="{Voter server-a02a1841-0706-856f-0dd0-97446733c13f a02a1841-0706-856f-0dd0-97446733c13f}" 2026-03-31T06:17:56.506Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:56.40129838 +0000 UTC m=+104.678905761 (1 nodes), 83.9126ms from start of poll, 84.34808ms from cluster start. Timeout at 2026-03-31 06:17:56.5063025 +0000 UTC m=+104.783909881, 105.00412ms after stability 2026-03-31T06:17:56.506Z [INFO] server-59d494bb-b3db-67e0-91ae-161345bcab27: aborting pipeline replication: peer="{Voter server-a02a1841-0706-856f-0dd0-97446733c13f a02a1841-0706-856f-0dd0-97446733c13f}" 2026-03-31T06:17:56.506Z [INFO] server-59d494bb-b3db-67e0-91ae-161345bcab27: aborting pipeline replication: peer="{Voter server-7308c39c-024d-0376-de4b-a2770ef69f2a 7308c39c-024d-0376-de4b-a2770ef69f2a}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.19s) === RUN TestRaft_LeadershipTransferToInvalidAddress 2026-03-31T06:17:56.507Z [INFO] server-530e477f-8a4c-247a-d538-5773a5ec89a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-530e477f-8a4c-247a-d538-5773a5ec89a8 Address:530e477f-8a4c-247a-d538-5773a5ec89a8} {Suffrage:Voter ID:server-19ceaed7-d136-909d-247c-48be09838e4e Address:19ceaed7-d136-909d-247c-48be09838e4e} {Suffrage:Voter ID:server-c7d91836-612a-3bfa-4219-9137517ab460 Address:c7d91836-612a-3bfa-4219-9137517ab460}]" 2026-03-31T06:17:56.507Z [INFO] server-19ceaed7-d136-909d-247c-48be09838e4e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-530e477f-8a4c-247a-d538-5773a5ec89a8 Address:530e477f-8a4c-247a-d538-5773a5ec89a8} {Suffrage:Voter ID:server-19ceaed7-d136-909d-247c-48be09838e4e Address:19ceaed7-d136-909d-247c-48be09838e4e} {Suffrage:Voter ID:server-c7d91836-612a-3bfa-4219-9137517ab460 Address:c7d91836-612a-3bfa-4219-9137517ab460}]" 2026-03-31T06:17:56.507Z [INFO] server-530e477f-8a4c-247a-d538-5773a5ec89a8: entering follower state: follower="Node at 530e477f-8a4c-247a-d538-5773a5ec89a8 [Follower]" leader-address= leader-id= 2026-03-31T06:17:56.508Z [INFO] server-c7d91836-612a-3bfa-4219-9137517ab460: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-530e477f-8a4c-247a-d538-5773a5ec89a8 Address:530e477f-8a4c-247a-d538-5773a5ec89a8} {Suffrage:Voter ID:server-19ceaed7-d136-909d-247c-48be09838e4e Address:19ceaed7-d136-909d-247c-48be09838e4e} {Suffrage:Voter ID:server-c7d91836-612a-3bfa-4219-9137517ab460 Address:c7d91836-612a-3bfa-4219-9137517ab460}]" 2026-03-31T06:17:56.508Z [INFO] server-19ceaed7-d136-909d-247c-48be09838e4e: entering follower state: follower="Node at 19ceaed7-d136-909d-247c-48be09838e4e [Follower]" leader-address= leader-id= 2026-03-31T06:17:56.508Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:56.508Z [INFO] server-c7d91836-612a-3bfa-4219-9137517ab460: entering follower state: follower="Node at c7d91836-612a-3bfa-4219-9137517ab460 [Follower]" leader-address= leader-id= 2026-03-31T06:17:56.569Z [WARN] server-19ceaed7-d136-909d-247c-48be09838e4e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:56.569Z [INFO] server-19ceaed7-d136-909d-247c-48be09838e4e: entering candidate state: node="Node at 19ceaed7-d136-909d-247c-48be09838e4e [Candidate]" term=2 2026-03-31T06:17:56.570Z [INFO] server-19ceaed7-d136-909d-247c-48be09838e4e: election won: term=2 tally=2 2026-03-31T06:17:56.570Z [INFO] server-19ceaed7-d136-909d-247c-48be09838e4e: entering leader state: leader="Node at 19ceaed7-d136-909d-247c-48be09838e4e [Leader]" 2026-03-31T06:17:56.570Z [INFO] server-19ceaed7-d136-909d-247c-48be09838e4e: added peer, starting replication: peer=server-530e477f-8a4c-247a-d538-5773a5ec89a8 2026-03-31T06:17:56.570Z [INFO] server-19ceaed7-d136-909d-247c-48be09838e4e: added peer, starting replication: peer=server-c7d91836-612a-3bfa-4219-9137517ab460 2026-03-31T06:17:56.570Z [INFO] server-19ceaed7-d136-909d-247c-48be09838e4e: pipelining replication: peer="{Voter server-c7d91836-612a-3bfa-4219-9137517ab460 c7d91836-612a-3bfa-4219-9137517ab460}" 2026-03-31T06:17:56.570Z [INFO] server-19ceaed7-d136-909d-247c-48be09838e4e: pipelining replication: peer="{Voter server-530e477f-8a4c-247a-d538-5773a5ec89a8 530e477f-8a4c-247a-d538-5773a5ec89a8}" 2026-03-31T06:17:56.676Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:56.57008306 +0000 UTC m=+104.847690481 (2 nodes), 61.95264ms from start of poll, 62.40308ms from cluster start. Timeout at 2026-03-31 06:17:56.6750867 +0000 UTC m=+104.952694081, 105.0036ms after stability 2026-03-31T06:17:56.676Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:56.781Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:56.67615486 +0000 UTC m=+104.953762281 (1 nodes), 2.68µs from start of poll, 168.47488ms from cluster start. Timeout at 2026-03-31 06:17:56.7811755 +0000 UTC m=+105.058782881, 105.0206ms after stability 2026-03-31T06:17:56.782Z [INFO] server-19ceaed7-d136-909d-247c-48be09838e4e: aborting pipeline replication: peer="{Voter server-c7d91836-612a-3bfa-4219-9137517ab460 c7d91836-612a-3bfa-4219-9137517ab460}" 2026-03-31T06:17:56.782Z [INFO] server-19ceaed7-d136-909d-247c-48be09838e4e: aborting pipeline replication: peer="{Voter server-530e477f-8a4c-247a-d538-5773a5ec89a8 530e477f-8a4c-247a-d538-5773a5ec89a8}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.28s) === RUN TestRaft_LeadershipTransferToBehindServer 2026-03-31T06:17:56.782Z [INFO] server-410a994b-6a3b-2b28-0bde-68e060f39212: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-410a994b-6a3b-2b28-0bde-68e060f39212 Address:410a994b-6a3b-2b28-0bde-68e060f39212} {Suffrage:Voter ID:server-fdf78898-f326-6af3-f60e-508402f26571 Address:fdf78898-f326-6af3-f60e-508402f26571} {Suffrage:Voter ID:server-b4b0ba40-536d-ab91-160d-32af43e4f87d Address:b4b0ba40-536d-ab91-160d-32af43e4f87d}]" 2026-03-31T06:17:56.783Z [INFO] server-fdf78898-f326-6af3-f60e-508402f26571: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-410a994b-6a3b-2b28-0bde-68e060f39212 Address:410a994b-6a3b-2b28-0bde-68e060f39212} {Suffrage:Voter ID:server-fdf78898-f326-6af3-f60e-508402f26571 Address:fdf78898-f326-6af3-f60e-508402f26571} {Suffrage:Voter ID:server-b4b0ba40-536d-ab91-160d-32af43e4f87d Address:b4b0ba40-536d-ab91-160d-32af43e4f87d}]" 2026-03-31T06:17:56.783Z [INFO] server-410a994b-6a3b-2b28-0bde-68e060f39212: entering follower state: follower="Node at 410a994b-6a3b-2b28-0bde-68e060f39212 [Follower]" leader-address= leader-id= 2026-03-31T06:17:56.783Z [INFO] server-fdf78898-f326-6af3-f60e-508402f26571: entering follower state: follower="Node at fdf78898-f326-6af3-f60e-508402f26571 [Follower]" leader-address= leader-id= 2026-03-31T06:17:56.783Z [INFO] server-b4b0ba40-536d-ab91-160d-32af43e4f87d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-410a994b-6a3b-2b28-0bde-68e060f39212 Address:410a994b-6a3b-2b28-0bde-68e060f39212} {Suffrage:Voter ID:server-fdf78898-f326-6af3-f60e-508402f26571 Address:fdf78898-f326-6af3-f60e-508402f26571} {Suffrage:Voter ID:server-b4b0ba40-536d-ab91-160d-32af43e4f87d Address:b4b0ba40-536d-ab91-160d-32af43e4f87d}]" 2026-03-31T06:17:56.783Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:56.783Z [INFO] server-b4b0ba40-536d-ab91-160d-32af43e4f87d: entering follower state: follower="Node at b4b0ba40-536d-ab91-160d-32af43e4f87d [Follower]" leader-address= leader-id= 2026-03-31T06:17:56.844Z [WARN] server-fdf78898-f326-6af3-f60e-508402f26571: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:56.844Z [INFO] server-fdf78898-f326-6af3-f60e-508402f26571: entering candidate state: node="Node at fdf78898-f326-6af3-f60e-508402f26571 [Candidate]" term=2 2026-03-31T06:17:56.844Z [INFO] server-fdf78898-f326-6af3-f60e-508402f26571: election won: term=2 tally=2 2026-03-31T06:17:56.844Z [INFO] server-fdf78898-f326-6af3-f60e-508402f26571: entering leader state: leader="Node at fdf78898-f326-6af3-f60e-508402f26571 [Leader]" 2026-03-31T06:17:56.844Z [INFO] server-fdf78898-f326-6af3-f60e-508402f26571: added peer, starting replication: peer=server-410a994b-6a3b-2b28-0bde-68e060f39212 2026-03-31T06:17:56.844Z [INFO] server-fdf78898-f326-6af3-f60e-508402f26571: added peer, starting replication: peer=server-b4b0ba40-536d-ab91-160d-32af43e4f87d 2026-03-31T06:17:56.845Z [INFO] server-fdf78898-f326-6af3-f60e-508402f26571: pipelining replication: peer="{Voter server-410a994b-6a3b-2b28-0bde-68e060f39212 410a994b-6a3b-2b28-0bde-68e060f39212}" 2026-03-31T06:17:56.845Z [INFO] server-fdf78898-f326-6af3-f60e-508402f26571: pipelining replication: peer="{Voter server-b4b0ba40-536d-ab91-160d-32af43e4f87d b4b0ba40-536d-ab91-160d-32af43e4f87d}" 2026-03-31T06:17:56.949Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:56.84491154 +0000 UTC m=+105.122518961 (1 nodes), 61.65216ms from start of poll, 62.0702ms from cluster start. Timeout at 2026-03-31 06:17:56.94991474 +0000 UTC m=+105.227522121, 105.00316ms after stability 2026-03-31T06:17:56.949Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:57.055Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:56.9500099 +0000 UTC m=+105.227617281 (2 nodes), 1.28µs from start of poll, 167.16852ms from cluster start. Timeout at 2026-03-31 06:17:57.05502306 +0000 UTC m=+105.332630441, 105.01316ms after stability 2026-03-31T06:17:57.106Z [INFO] server-410a994b-6a3b-2b28-0bde-68e060f39212: entering candidate state: node="Node at 410a994b-6a3b-2b28-0bde-68e060f39212 [Candidate]" term=3 2026-03-31T06:17:57.106Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:57.107Z [INFO] server-410a994b-6a3b-2b28-0bde-68e060f39212: election won: term=3 tally=2 2026-03-31T06:17:57.107Z [INFO] server-410a994b-6a3b-2b28-0bde-68e060f39212: entering leader state: leader="Node at 410a994b-6a3b-2b28-0bde-68e060f39212 [Leader]" 2026-03-31T06:17:57.107Z [INFO] server-410a994b-6a3b-2b28-0bde-68e060f39212: added peer, starting replication: peer=server-fdf78898-f326-6af3-f60e-508402f26571 2026-03-31T06:17:57.107Z [INFO] server-410a994b-6a3b-2b28-0bde-68e060f39212: added peer, starting replication: peer=server-b4b0ba40-536d-ab91-160d-32af43e4f87d 2026-03-31T06:17:57.107Z [INFO] server-fdf78898-f326-6af3-f60e-508402f26571: entering follower state: follower="Node at fdf78898-f326-6af3-f60e-508402f26571 [Follower]" leader-address= leader-id= 2026-03-31T06:17:57.107Z [INFO] server-fdf78898-f326-6af3-f60e-508402f26571: aborting pipeline replication: peer="{Voter server-410a994b-6a3b-2b28-0bde-68e060f39212 410a994b-6a3b-2b28-0bde-68e060f39212}" 2026-03-31T06:17:57.107Z [INFO] server-fdf78898-f326-6af3-f60e-508402f26571: aborting pipeline replication: peer="{Voter server-b4b0ba40-536d-ab91-160d-32af43e4f87d b4b0ba40-536d-ab91-160d-32af43e4f87d}" 2026-03-31T06:17:57.107Z [INFO] server-410a994b-6a3b-2b28-0bde-68e060f39212: pipelining replication: peer="{Voter server-fdf78898-f326-6af3-f60e-508402f26571 fdf78898-f326-6af3-f60e-508402f26571}" 2026-03-31T06:17:57.107Z [INFO] server-410a994b-6a3b-2b28-0bde-68e060f39212: pipelining replication: peer="{Voter server-b4b0ba40-536d-ab91-160d-32af43e4f87d b4b0ba40-536d-ab91-160d-32af43e4f87d}" 2026-03-31T06:17:57.213Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:57.1071569 +0000 UTC m=+105.384764281 (1 nodes), 160.6µs from start of poll, 324.31552ms from cluster start. Timeout at 2026-03-31 06:17:57.2122197 +0000 UTC m=+105.489827121, 105.06284ms after stability 2026-03-31T06:17:57.213Z [INFO] server-410a994b-6a3b-2b28-0bde-68e060f39212: aborting pipeline replication: peer="{Voter server-b4b0ba40-536d-ab91-160d-32af43e4f87d b4b0ba40-536d-ab91-160d-32af43e4f87d}" 2026-03-31T06:17:57.213Z [INFO] server-410a994b-6a3b-2b28-0bde-68e060f39212: aborting pipeline replication: peer="{Voter server-fdf78898-f326-6af3-f60e-508402f26571 fdf78898-f326-6af3-f60e-508402f26571}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.43s) === RUN TestRaft_LeadershipTransferToItself 2026-03-31T06:17:57.214Z [INFO] server-3cf28049-3696-dfd2-72ae-4ecd2a802741: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3cf28049-3696-dfd2-72ae-4ecd2a802741 Address:3cf28049-3696-dfd2-72ae-4ecd2a802741} {Suffrage:Voter ID:server-914abc7f-d0aa-0e06-e82f-111a51896b80 Address:914abc7f-d0aa-0e06-e82f-111a51896b80} {Suffrage:Voter ID:server-8c04d648-d57b-b00c-805f-0a7300d54212 Address:8c04d648-d57b-b00c-805f-0a7300d54212}]" 2026-03-31T06:17:57.214Z [INFO] server-914abc7f-d0aa-0e06-e82f-111a51896b80: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3cf28049-3696-dfd2-72ae-4ecd2a802741 Address:3cf28049-3696-dfd2-72ae-4ecd2a802741} {Suffrage:Voter ID:server-914abc7f-d0aa-0e06-e82f-111a51896b80 Address:914abc7f-d0aa-0e06-e82f-111a51896b80} {Suffrage:Voter ID:server-8c04d648-d57b-b00c-805f-0a7300d54212 Address:8c04d648-d57b-b00c-805f-0a7300d54212}]" 2026-03-31T06:17:57.214Z [INFO] server-3cf28049-3696-dfd2-72ae-4ecd2a802741: entering follower state: follower="Node at 3cf28049-3696-dfd2-72ae-4ecd2a802741 [Follower]" leader-address= leader-id= 2026-03-31T06:17:57.214Z [INFO] server-8c04d648-d57b-b00c-805f-0a7300d54212: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3cf28049-3696-dfd2-72ae-4ecd2a802741 Address:3cf28049-3696-dfd2-72ae-4ecd2a802741} {Suffrage:Voter ID:server-914abc7f-d0aa-0e06-e82f-111a51896b80 Address:914abc7f-d0aa-0e06-e82f-111a51896b80} {Suffrage:Voter ID:server-8c04d648-d57b-b00c-805f-0a7300d54212 Address:8c04d648-d57b-b00c-805f-0a7300d54212}]" 2026-03-31T06:17:57.214Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:57.214Z [INFO] server-914abc7f-d0aa-0e06-e82f-111a51896b80: entering follower state: follower="Node at 914abc7f-d0aa-0e06-e82f-111a51896b80 [Follower]" leader-address= leader-id= 2026-03-31T06:17:57.214Z [INFO] server-8c04d648-d57b-b00c-805f-0a7300d54212: entering follower state: follower="Node at 8c04d648-d57b-b00c-805f-0a7300d54212 [Follower]" leader-address= leader-id= 2026-03-31T06:17:57.281Z [WARN] server-914abc7f-d0aa-0e06-e82f-111a51896b80: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:57.281Z [INFO] server-914abc7f-d0aa-0e06-e82f-111a51896b80: entering candidate state: node="Node at 914abc7f-d0aa-0e06-e82f-111a51896b80 [Candidate]" term=2 2026-03-31T06:17:57.281Z [INFO] server-914abc7f-d0aa-0e06-e82f-111a51896b80: election won: term=2 tally=2 2026-03-31T06:17:57.281Z [INFO] server-914abc7f-d0aa-0e06-e82f-111a51896b80: entering leader state: leader="Node at 914abc7f-d0aa-0e06-e82f-111a51896b80 [Leader]" 2026-03-31T06:17:57.281Z [INFO] server-914abc7f-d0aa-0e06-e82f-111a51896b80: added peer, starting replication: peer=server-3cf28049-3696-dfd2-72ae-4ecd2a802741 2026-03-31T06:17:57.281Z [INFO] server-914abc7f-d0aa-0e06-e82f-111a51896b80: added peer, starting replication: peer=server-8c04d648-d57b-b00c-805f-0a7300d54212 2026-03-31T06:17:57.281Z [INFO] server-914abc7f-d0aa-0e06-e82f-111a51896b80: pipelining replication: peer="{Voter server-8c04d648-d57b-b00c-805f-0a7300d54212 8c04d648-d57b-b00c-805f-0a7300d54212}" 2026-03-31T06:17:57.281Z [INFO] server-914abc7f-d0aa-0e06-e82f-111a51896b80: pipelining replication: peer="{Voter server-3cf28049-3696-dfd2-72ae-4ecd2a802741 3cf28049-3696-dfd2-72ae-4ecd2a802741}" 2026-03-31T06:17:57.387Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:57.2818681 +0000 UTC m=+105.559475481 (1 nodes), 67.37676ms from start of poll, 67.74928ms from cluster start. Timeout at 2026-03-31 06:17:57.38687714 +0000 UTC m=+105.664484521, 105.00904ms after stability 2026-03-31T06:17:57.387Z [INFO] server-914abc7f-d0aa-0e06-e82f-111a51896b80: cannot transfer leadership to itself 2026-03-31T06:17:57.387Z [INFO] server-914abc7f-d0aa-0e06-e82f-111a51896b80: aborting pipeline replication: peer="{Voter server-8c04d648-d57b-b00c-805f-0a7300d54212 8c04d648-d57b-b00c-805f-0a7300d54212}" 2026-03-31T06:17:57.387Z [INFO] server-914abc7f-d0aa-0e06-e82f-111a51896b80: aborting pipeline replication: peer="{Voter server-3cf28049-3696-dfd2-72ae-4ecd2a802741 3cf28049-3696-dfd2-72ae-4ecd2a802741}" --- PASS: TestRaft_LeadershipTransferToItself (0.17s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests 2026-03-31T06:17:57.388Z [INFO] server-e0aa6848-7d91-db0e-48f8-d9663fac06a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0aa6848-7d91-db0e-48f8-d9663fac06a4 Address:e0aa6848-7d91-db0e-48f8-d9663fac06a4} {Suffrage:Voter ID:server-49901a17-ee22-d565-ff39-e66eeabb453c Address:49901a17-ee22-d565-ff39-e66eeabb453c} {Suffrage:Voter ID:server-7353fe1b-be16-b5b4-d65d-e26f58773e52 Address:7353fe1b-be16-b5b4-d65d-e26f58773e52}]" 2026-03-31T06:17:57.388Z [INFO] server-49901a17-ee22-d565-ff39-e66eeabb453c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0aa6848-7d91-db0e-48f8-d9663fac06a4 Address:e0aa6848-7d91-db0e-48f8-d9663fac06a4} {Suffrage:Voter ID:server-49901a17-ee22-d565-ff39-e66eeabb453c Address:49901a17-ee22-d565-ff39-e66eeabb453c} {Suffrage:Voter ID:server-7353fe1b-be16-b5b4-d65d-e26f58773e52 Address:7353fe1b-be16-b5b4-d65d-e26f58773e52}]" 2026-03-31T06:17:57.388Z [INFO] server-e0aa6848-7d91-db0e-48f8-d9663fac06a4: entering follower state: follower="Node at e0aa6848-7d91-db0e-48f8-d9663fac06a4 [Follower]" leader-address= leader-id= 2026-03-31T06:17:57.388Z [INFO] server-49901a17-ee22-d565-ff39-e66eeabb453c: entering follower state: follower="Node at 49901a17-ee22-d565-ff39-e66eeabb453c [Follower]" leader-address= leader-id= 2026-03-31T06:17:57.388Z [INFO] server-7353fe1b-be16-b5b4-d65d-e26f58773e52: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0aa6848-7d91-db0e-48f8-d9663fac06a4 Address:e0aa6848-7d91-db0e-48f8-d9663fac06a4} {Suffrage:Voter ID:server-49901a17-ee22-d565-ff39-e66eeabb453c Address:49901a17-ee22-d565-ff39-e66eeabb453c} {Suffrage:Voter ID:server-7353fe1b-be16-b5b4-d65d-e26f58773e52 Address:7353fe1b-be16-b5b4-d65d-e26f58773e52}]" 2026-03-31T06:17:57.388Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:57.388Z [INFO] server-7353fe1b-be16-b5b4-d65d-e26f58773e52: entering follower state: follower="Node at 7353fe1b-be16-b5b4-d65d-e26f58773e52 [Follower]" leader-address= leader-id= 2026-03-31T06:17:57.448Z [WARN] server-49901a17-ee22-d565-ff39-e66eeabb453c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:57.448Z [INFO] server-49901a17-ee22-d565-ff39-e66eeabb453c: entering candidate state: node="Node at 49901a17-ee22-d565-ff39-e66eeabb453c [Candidate]" term=2 2026-03-31T06:17:57.448Z [INFO] server-49901a17-ee22-d565-ff39-e66eeabb453c: election won: term=2 tally=2 2026-03-31T06:17:57.448Z [INFO] server-49901a17-ee22-d565-ff39-e66eeabb453c: entering leader state: leader="Node at 49901a17-ee22-d565-ff39-e66eeabb453c [Leader]" 2026-03-31T06:17:57.448Z [INFO] server-49901a17-ee22-d565-ff39-e66eeabb453c: added peer, starting replication: peer=server-e0aa6848-7d91-db0e-48f8-d9663fac06a4 2026-03-31T06:17:57.448Z [INFO] server-49901a17-ee22-d565-ff39-e66eeabb453c: added peer, starting replication: peer=server-7353fe1b-be16-b5b4-d65d-e26f58773e52 2026-03-31T06:17:57.449Z [INFO] server-49901a17-ee22-d565-ff39-e66eeabb453c: pipelining replication: peer="{Voter server-7353fe1b-be16-b5b4-d65d-e26f58773e52 7353fe1b-be16-b5b4-d65d-e26f58773e52}" 2026-03-31T06:17:57.449Z [INFO] server-49901a17-ee22-d565-ff39-e66eeabb453c: pipelining replication: peer="{Voter server-e0aa6848-7d91-db0e-48f8-d9663fac06a4 e0aa6848-7d91-db0e-48f8-d9663fac06a4}" 2026-03-31T06:17:57.555Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:57.44902482 +0000 UTC m=+105.726632201 (1 nodes), 60.45024ms from start of poll, 60.8544ms from cluster start. Timeout at 2026-03-31 06:17:57.55402694 +0000 UTC m=+105.831634321, 105.00212ms after stability raft_test.go:2457: waiting on future 0 raft_test.go:2457: waiting on future 1 raft_test.go:2457: waiting on future 2 raft_test.go:2457: waiting on future 3 raft_test.go:2457: waiting on future 4 raft_test.go:2457: waiting on future 5 raft_test.go:2457: waiting on future 6 2026-03-31T06:17:57.555Z [INFO] server-49901a17-ee22-d565-ff39-e66eeabb453c: aborting pipeline replication: peer="{Voter server-e0aa6848-7d91-db0e-48f8-d9663fac06a4 e0aa6848-7d91-db0e-48f8-d9663fac06a4}" 2026-03-31T06:17:57.555Z [INFO] server-49901a17-ee22-d565-ff39-e66eeabb453c: aborting pipeline replication: peer="{Voter server-7353fe1b-be16-b5b4-d65d-e26f58773e52 7353fe1b-be16-b5b4-d65d-e26f58773e52}" --- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.17s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout 2026-03-31T06:17:57.556Z [INFO] server-7aaffcd6-7066-9165-8f18-702e92f5771c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7aaffcd6-7066-9165-8f18-702e92f5771c Address:7aaffcd6-7066-9165-8f18-702e92f5771c} {Suffrage:Voter ID:server-98e391dc-371f-67d8-4572-63e0e8c543a8 Address:98e391dc-371f-67d8-4572-63e0e8c543a8} {Suffrage:Voter ID:server-8825253c-5314-a371-19b8-4b749b8d546e Address:8825253c-5314-a371-19b8-4b749b8d546e}]" 2026-03-31T06:17:57.556Z [INFO] server-7aaffcd6-7066-9165-8f18-702e92f5771c: entering follower state: follower="Node at 7aaffcd6-7066-9165-8f18-702e92f5771c [Follower]" leader-address= leader-id= 2026-03-31T06:17:57.556Z [INFO] server-98e391dc-371f-67d8-4572-63e0e8c543a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7aaffcd6-7066-9165-8f18-702e92f5771c Address:7aaffcd6-7066-9165-8f18-702e92f5771c} {Suffrage:Voter ID:server-98e391dc-371f-67d8-4572-63e0e8c543a8 Address:98e391dc-371f-67d8-4572-63e0e8c543a8} {Suffrage:Voter ID:server-8825253c-5314-a371-19b8-4b749b8d546e Address:8825253c-5314-a371-19b8-4b749b8d546e}]" 2026-03-31T06:17:57.556Z [INFO] server-98e391dc-371f-67d8-4572-63e0e8c543a8: entering follower state: follower="Node at 98e391dc-371f-67d8-4572-63e0e8c543a8 [Follower]" leader-address= leader-id= 2026-03-31T06:17:57.556Z [INFO] server-8825253c-5314-a371-19b8-4b749b8d546e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7aaffcd6-7066-9165-8f18-702e92f5771c Address:7aaffcd6-7066-9165-8f18-702e92f5771c} {Suffrage:Voter ID:server-98e391dc-371f-67d8-4572-63e0e8c543a8 Address:98e391dc-371f-67d8-4572-63e0e8c543a8} {Suffrage:Voter ID:server-8825253c-5314-a371-19b8-4b749b8d546e Address:8825253c-5314-a371-19b8-4b749b8d546e}]" 2026-03-31T06:17:57.556Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:57.556Z [INFO] server-8825253c-5314-a371-19b8-4b749b8d546e: entering follower state: follower="Node at 8825253c-5314-a371-19b8-4b749b8d546e [Follower]" leader-address= leader-id= 2026-03-31T06:17:57.612Z [WARN] server-8825253c-5314-a371-19b8-4b749b8d546e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:57.612Z [INFO] server-8825253c-5314-a371-19b8-4b749b8d546e: entering candidate state: node="Node at 8825253c-5314-a371-19b8-4b749b8d546e [Candidate]" term=2 2026-03-31T06:17:57.612Z [INFO] server-8825253c-5314-a371-19b8-4b749b8d546e: election won: term=2 tally=2 2026-03-31T06:17:57.612Z [INFO] server-8825253c-5314-a371-19b8-4b749b8d546e: entering leader state: leader="Node at 8825253c-5314-a371-19b8-4b749b8d546e [Leader]" 2026-03-31T06:17:57.612Z [INFO] server-8825253c-5314-a371-19b8-4b749b8d546e: added peer, starting replication: peer=server-7aaffcd6-7066-9165-8f18-702e92f5771c 2026-03-31T06:17:57.613Z [INFO] server-8825253c-5314-a371-19b8-4b749b8d546e: added peer, starting replication: peer=server-98e391dc-371f-67d8-4572-63e0e8c543a8 2026-03-31T06:17:57.613Z [INFO] server-8825253c-5314-a371-19b8-4b749b8d546e: pipelining replication: peer="{Voter server-98e391dc-371f-67d8-4572-63e0e8c543a8 98e391dc-371f-67d8-4572-63e0e8c543a8}" 2026-03-31T06:17:57.613Z [INFO] server-8825253c-5314-a371-19b8-4b749b8d546e: pipelining replication: peer="{Voter server-7aaffcd6-7066-9165-8f18-702e92f5771c 7aaffcd6-7066-9165-8f18-702e92f5771c}" 2026-03-31T06:17:57.718Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:57.61301882 +0000 UTC m=+105.890626201 (1 nodes), 56.315ms from start of poll, 56.8678ms from cluster start. Timeout at 2026-03-31 06:17:57.71802374 +0000 UTC m=+105.995631121, 105.00492ms after stability 2026-03-31T06:17:57.718Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:57.823Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:57.71858594 +0000 UTC m=+105.996193321 (2 nodes), 1.8µs from start of poll, 162.43492ms from cluster start. Timeout at 2026-03-31 06:17:57.8236125 +0000 UTC m=+106.101219881, 105.02656ms after stability 2026-03-31T06:17:58.309Z [INFO] server-8825253c-5314-a371-19b8-4b749b8d546e: aborting pipeline replication: peer="{Voter server-98e391dc-371f-67d8-4572-63e0e8c543a8 98e391dc-371f-67d8-4572-63e0e8c543a8}" 2026-03-31T06:17:58.309Z [INFO] server-8825253c-5314-a371-19b8-4b749b8d546e: aborting pipeline replication: peer="{Voter server-7aaffcd6-7066-9165-8f18-702e92f5771c 7aaffcd6-7066-9165-8f18-702e92f5771c}" --- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.75s) === RUN TestRaft_LeadershipTransferIgnoresNonvoters 2026-03-31T06:17:58.310Z [INFO] server-9f21f33d-f57d-ce75-6d3a-735c4ba3ef42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f21f33d-f57d-ce75-6d3a-735c4ba3ef42 Address:9f21f33d-f57d-ce75-6d3a-735c4ba3ef42} {Suffrage:Voter ID:server-e79eff2e-27d0-4e14-0e7e-850918d256fc Address:e79eff2e-27d0-4e14-0e7e-850918d256fc}]" 2026-03-31T06:17:58.310Z [INFO] server-e79eff2e-27d0-4e14-0e7e-850918d256fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f21f33d-f57d-ce75-6d3a-735c4ba3ef42 Address:9f21f33d-f57d-ce75-6d3a-735c4ba3ef42} {Suffrage:Voter ID:server-e79eff2e-27d0-4e14-0e7e-850918d256fc Address:e79eff2e-27d0-4e14-0e7e-850918d256fc}]" 2026-03-31T06:17:58.310Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:58.310Z [INFO] server-9f21f33d-f57d-ce75-6d3a-735c4ba3ef42: entering follower state: follower="Node at 9f21f33d-f57d-ce75-6d3a-735c4ba3ef42 [Follower]" leader-address= leader-id= 2026-03-31T06:17:58.310Z [INFO] server-e79eff2e-27d0-4e14-0e7e-850918d256fc: entering follower state: follower="Node at e79eff2e-27d0-4e14-0e7e-850918d256fc [Follower]" leader-address= leader-id= 2026-03-31T06:17:58.382Z [WARN] server-9f21f33d-f57d-ce75-6d3a-735c4ba3ef42: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:58.382Z [INFO] server-9f21f33d-f57d-ce75-6d3a-735c4ba3ef42: entering candidate state: node="Node at 9f21f33d-f57d-ce75-6d3a-735c4ba3ef42 [Candidate]" term=2 2026-03-31T06:17:58.382Z [INFO] server-9f21f33d-f57d-ce75-6d3a-735c4ba3ef42: election won: term=2 tally=2 2026-03-31T06:17:58.382Z [INFO] server-9f21f33d-f57d-ce75-6d3a-735c4ba3ef42: entering leader state: leader="Node at 9f21f33d-f57d-ce75-6d3a-735c4ba3ef42 [Leader]" 2026-03-31T06:17:58.382Z [INFO] server-9f21f33d-f57d-ce75-6d3a-735c4ba3ef42: added peer, starting replication: peer=server-e79eff2e-27d0-4e14-0e7e-850918d256fc 2026-03-31T06:17:58.382Z [INFO] server-9f21f33d-f57d-ce75-6d3a-735c4ba3ef42: pipelining replication: peer="{Voter server-e79eff2e-27d0-4e14-0e7e-850918d256fc e79eff2e-27d0-4e14-0e7e-850918d256fc}" 2026-03-31T06:17:58.487Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:58.38245542 +0000 UTC m=+106.660062841 (1 nodes), 72.10244ms from start of poll, 72.3904ms from cluster start. Timeout at 2026-03-31 06:17:58.4874697 +0000 UTC m=+106.765077081, 105.01424ms after stability 2026-03-31T06:17:58.488Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:58.593Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:58.48804598 +0000 UTC m=+106.765653401 (1 nodes), 760ns from start of poll, 177.98096ms from cluster start. Timeout at 2026-03-31 06:17:58.59313274 +0000 UTC m=+106.870740121, 105.08672ms after stability 2026-03-31T06:17:58.593Z [INFO] server-9f21f33d-f57d-ce75-6d3a-735c4ba3ef42: updating configuration: command=DemoteVoter server-id=server-e79eff2e-27d0-4e14-0e7e-850918d256fc server-addr= servers="[{Suffrage:Voter ID:server-9f21f33d-f57d-ce75-6d3a-735c4ba3ef42 Address:9f21f33d-f57d-ce75-6d3a-735c4ba3ef42} {Suffrage:Nonvoter ID:server-e79eff2e-27d0-4e14-0e7e-850918d256fc Address:e79eff2e-27d0-4e14-0e7e-850918d256fc}]" 2026-03-31T06:17:58.593Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:58.699Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:58.59384682 +0000 UTC m=+106.871454201 (1 nodes), 800ns from start of poll, 283.78176ms from cluster start. Timeout at 2026-03-31 06:17:58.6989115 +0000 UTC m=+106.976518881, 105.06468ms after stability 2026-03-31T06:17:58.699Z [INFO] server-9f21f33d-f57d-ce75-6d3a-735c4ba3ef42: aborting pipeline replication: peer="{Voter server-e79eff2e-27d0-4e14-0e7e-850918d256fc e79eff2e-27d0-4e14-0e7e-850918d256fc}" --- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.39s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap 2026-03-31T06:17:58.699Z [INFO] server-22141115-be83-8732-4c77-85cb7fe5263b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22141115-be83-8732-4c77-85cb7fe5263b Address:22141115-be83-8732-4c77-85cb7fe5263b} {Suffrage:Voter ID:server-a84443d0-d3fa-2389-1710-861e3a302887 Address:a84443d0-d3fa-2389-1710-861e3a302887}]" 2026-03-31T06:17:58.700Z [INFO] server-a84443d0-d3fa-2389-1710-861e3a302887: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22141115-be83-8732-4c77-85cb7fe5263b Address:22141115-be83-8732-4c77-85cb7fe5263b} {Suffrage:Voter ID:server-a84443d0-d3fa-2389-1710-861e3a302887 Address:a84443d0-d3fa-2389-1710-861e3a302887}]" 2026-03-31T06:17:58.699Z [INFO] server-22141115-be83-8732-4c77-85cb7fe5263b: entering follower state: follower="Node at 22141115-be83-8732-4c77-85cb7fe5263b [Follower]" leader-address= leader-id= 2026-03-31T06:17:58.700Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:58.700Z [INFO] server-a84443d0-d3fa-2389-1710-861e3a302887: entering follower state: follower="Node at a84443d0-d3fa-2389-1710-861e3a302887 [Follower]" leader-address= leader-id= 2026-03-31T06:17:58.757Z [WARN] server-22141115-be83-8732-4c77-85cb7fe5263b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:58.757Z [INFO] server-22141115-be83-8732-4c77-85cb7fe5263b: entering candidate state: node="Node at 22141115-be83-8732-4c77-85cb7fe5263b [Candidate]" term=2 2026-03-31T06:17:58.757Z [INFO] server-22141115-be83-8732-4c77-85cb7fe5263b: election won: term=2 tally=2 2026-03-31T06:17:58.757Z [INFO] server-22141115-be83-8732-4c77-85cb7fe5263b: entering leader state: leader="Node at 22141115-be83-8732-4c77-85cb7fe5263b [Leader]" 2026-03-31T06:17:58.757Z [INFO] server-22141115-be83-8732-4c77-85cb7fe5263b: added peer, starting replication: peer=server-a84443d0-d3fa-2389-1710-861e3a302887 2026-03-31T06:17:58.757Z [INFO] server-22141115-be83-8732-4c77-85cb7fe5263b: pipelining replication: peer="{Voter server-a84443d0-d3fa-2389-1710-861e3a302887 a84443d0-d3fa-2389-1710-861e3a302887}" 2026-03-31T06:17:58.864Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:58.75787026 +0000 UTC m=+107.035477641 (1 nodes), 57.7766ms from start of poll, 58.09584ms from cluster start. Timeout at 2026-03-31 06:17:58.86287418 +0000 UTC m=+107.140481561, 105.00392ms after stability 2026-03-31T06:17:58.864Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:58.969Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:58.86413506 +0000 UTC m=+107.141742481 (1 nodes), 2.48µs from start of poll, 164.36068ms from cluster start. Timeout at 2026-03-31 06:17:58.96916722 +0000 UTC m=+107.246774601, 105.03212ms after stability 2026-03-31T06:17:58.980Z [INFO] server-a84443d0-d3fa-2389-1710-861e3a302887: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22141115-be83-8732-4c77-85cb7fe5263b Address:22141115-be83-8732-4c77-85cb7fe5263b} {Suffrage:Voter ID:server-a84443d0-d3fa-2389-1710-861e3a302887 Address:a84443d0-d3fa-2389-1710-861e3a302887}]" 2026-03-31T06:17:58.980Z [INFO] server-22141115-be83-8732-4c77-85cb7fe5263b: aborting pipeline replication: peer="{Voter server-a84443d0-d3fa-2389-1710-861e3a302887 a84443d0-d3fa-2389-1710-861e3a302887}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.28s) === RUN TestRaft_LogStoreIsMonotonic 2026-03-31T06:17:58.981Z [INFO] server-e834d2e0-eacf-df0d-6e59-30218780d82f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e834d2e0-eacf-df0d-6e59-30218780d82f Address:e834d2e0-eacf-df0d-6e59-30218780d82f}]" 2026-03-31T06:17:58.981Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:58.981Z [INFO] server-e834d2e0-eacf-df0d-6e59-30218780d82f: entering follower state: follower="Node at e834d2e0-eacf-df0d-6e59-30218780d82f [Follower]" leader-address= leader-id= 2026-03-31T06:17:59.063Z [WARN] server-e834d2e0-eacf-df0d-6e59-30218780d82f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:59.063Z [INFO] server-e834d2e0-eacf-df0d-6e59-30218780d82f: entering candidate state: node="Node at e834d2e0-eacf-df0d-6e59-30218780d82f [Candidate]" term=2 2026-03-31T06:17:59.063Z [INFO] server-e834d2e0-eacf-df0d-6e59-30218780d82f: election won: term=2 tally=1 2026-03-31T06:17:59.063Z [INFO] server-e834d2e0-eacf-df0d-6e59-30218780d82f: entering leader state: leader="Node at e834d2e0-eacf-df0d-6e59-30218780d82f [Leader]" 2026-03-31T06:17:59.169Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:59.06402178 +0000 UTC m=+107.341629161 (1 nodes), 82.39224ms from start of poll, 82.59324ms from cluster start. Timeout at 2026-03-31 06:17:59.16902514 +0000 UTC m=+107.446632521, 105.00336ms after stability --- PASS: TestRaft_LogStoreIsMonotonic (0.19s) === RUN TestRaft_CacheLogWithStoreError 2026-03-31T06:17:59.170Z [INFO] server-660eb37a-f256-7278-f569-a3cbda40d03f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-660eb37a-f256-7278-f569-a3cbda40d03f Address:660eb37a-f256-7278-f569-a3cbda40d03f} {Suffrage:Voter ID:server-3a63d033-e17a-08cf-8103-c9451d5bfcf4 Address:3a63d033-e17a-08cf-8103-c9451d5bfcf4}]" 2026-03-31T06:17:59.170Z [INFO] server-3a63d033-e17a-08cf-8103-c9451d5bfcf4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-660eb37a-f256-7278-f569-a3cbda40d03f Address:660eb37a-f256-7278-f569-a3cbda40d03f} {Suffrage:Voter ID:server-3a63d033-e17a-08cf-8103-c9451d5bfcf4 Address:3a63d033-e17a-08cf-8103-c9451d5bfcf4}]" 2026-03-31T06:17:59.170Z [INFO] server-660eb37a-f256-7278-f569-a3cbda40d03f: entering follower state: follower="Node at 660eb37a-f256-7278-f569-a3cbda40d03f [Follower]" leader-address= leader-id= 2026-03-31T06:17:59.170Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:59.170Z [INFO] server-3a63d033-e17a-08cf-8103-c9451d5bfcf4: entering follower state: follower="Node at 3a63d033-e17a-08cf-8103-c9451d5bfcf4 [Follower]" leader-address= leader-id= 2026-03-31T06:17:59.241Z [WARN] server-660eb37a-f256-7278-f569-a3cbda40d03f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:59.241Z [INFO] server-660eb37a-f256-7278-f569-a3cbda40d03f: entering candidate state: node="Node at 660eb37a-f256-7278-f569-a3cbda40d03f [Candidate]" term=2 2026-03-31T06:17:59.242Z [INFO] server-660eb37a-f256-7278-f569-a3cbda40d03f: election won: term=2 tally=2 2026-03-31T06:17:59.242Z [INFO] server-660eb37a-f256-7278-f569-a3cbda40d03f: entering leader state: leader="Node at 660eb37a-f256-7278-f569-a3cbda40d03f [Leader]" 2026-03-31T06:17:59.242Z [INFO] server-660eb37a-f256-7278-f569-a3cbda40d03f: added peer, starting replication: peer=server-3a63d033-e17a-08cf-8103-c9451d5bfcf4 2026-03-31T06:17:59.242Z [INFO] server-660eb37a-f256-7278-f569-a3cbda40d03f: pipelining replication: peer="{Voter server-3a63d033-e17a-08cf-8103-c9451d5bfcf4 3a63d033-e17a-08cf-8103-c9451d5bfcf4}" 2026-03-31T06:17:59.347Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:59.24209958 +0000 UTC m=+107.519706961 (1 nodes), 71.5582ms from start of poll, 71.86396ms from cluster start. Timeout at 2026-03-31 06:17:59.3471043 +0000 UTC m=+107.624711681, 105.00472ms after stability 2026-03-31T06:17:59.347Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:59.452Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:59.3472811 +0000 UTC m=+107.624888481 (1 nodes), 2.08µs from start of poll, 177.04548ms from cluster start. Timeout at 2026-03-31 06:17:59.45230326 +0000 UTC m=+107.729910641, 105.02216ms after stability 2026-03-31T06:17:59.453Z [ERROR] server-3a63d033-e17a-08cf-8103-c9451d5bfcf4: failed to append to logs: error="unable to store logs within log store, err: "some error"" 2026-03-31T06:17:59.453Z [WARN] server-3a63d033-e17a-08cf-8103-c9451d5bfcf4: failed to get previous log: previous-index=9 last-index=7 error="log not found" 2026-03-31T06:17:59.453Z [INFO] server-660eb37a-f256-7278-f569-a3cbda40d03f: aborting pipeline replication: peer="{Voter server-3a63d033-e17a-08cf-8103-c9451d5bfcf4 3a63d033-e17a-08cf-8103-c9451d5bfcf4}" 2026-03-31T06:17:59.461Z [INFO] server-660eb37a-f256-7278-f569-a3cbda40d03f: pipelining replication: peer="{Voter server-3a63d033-e17a-08cf-8103-c9451d5bfcf4 3a63d033-e17a-08cf-8103-c9451d5bfcf4}" 2026-03-31T06:17:59.461Z [INFO] server-3a63d033-e17a-08cf-8103-c9451d5bfcf4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-660eb37a-f256-7278-f569-a3cbda40d03f Address:660eb37a-f256-7278-f569-a3cbda40d03f} {Suffrage:Voter ID:server-3a63d033-e17a-08cf-8103-c9451d5bfcf4 Address:3a63d033-e17a-08cf-8103-c9451d5bfcf4}]" 2026-03-31T06:17:59.461Z [INFO] server-660eb37a-f256-7278-f569-a3cbda40d03f: aborting pipeline replication: peer="{Voter server-3a63d033-e17a-08cf-8103-c9451d5bfcf4 3a63d033-e17a-08cf-8103-c9451d5bfcf4}" --- PASS: TestRaft_CacheLogWithStoreError (0.29s) === RUN TestRaft_ReloadConfig 2026-03-31T06:17:59.462Z [INFO] server-162dd517-40d4-5f0e-d46c-2cab1a292c82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-162dd517-40d4-5f0e-d46c-2cab1a292c82 Address:162dd517-40d4-5f0e-d46c-2cab1a292c82}]" --- PASS: TestRaft_ReloadConfig (0.00s) === RUN TestRaft_ReloadConfigValidates 2026-03-31T06:17:59.462Z [INFO] server-554109fb-a6b8-5063-69aa-0a57324dc2bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-554109fb-a6b8-5063-69aa-0a57324dc2bc Address:554109fb-a6b8-5063-69aa-0a57324dc2bc}]" --- PASS: TestRaft_ReloadConfigValidates (0.00s) === RUN TestRaft_InstallSnapshot_InvalidPeers 2026-03-31T06:17:59.462Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)" --- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s) === RUN TestRaft_VoteNotGranted_WhenNodeNotInCluster 2026-03-31T06:17:59.463Z [INFO] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ffc338e-4794-c2a1-5b04-4961401f4d7f Address:4ffc338e-4794-c2a1-5b04-4961401f4d7f} {Suffrage:Voter ID:server-a3618a08-3134-1c50-85ec-b8d5bec6badd Address:a3618a08-3134-1c50-85ec-b8d5bec6badd} {Suffrage:Voter ID:server-c1094d08-d0f4-666d-def3-8d427748d4ab Address:c1094d08-d0f4-666d-def3-8d427748d4ab}]" 2026-03-31T06:17:59.463Z [INFO] server-a3618a08-3134-1c50-85ec-b8d5bec6badd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ffc338e-4794-c2a1-5b04-4961401f4d7f Address:4ffc338e-4794-c2a1-5b04-4961401f4d7f} {Suffrage:Voter ID:server-a3618a08-3134-1c50-85ec-b8d5bec6badd Address:a3618a08-3134-1c50-85ec-b8d5bec6badd} {Suffrage:Voter ID:server-c1094d08-d0f4-666d-def3-8d427748d4ab Address:c1094d08-d0f4-666d-def3-8d427748d4ab}]" 2026-03-31T06:17:59.463Z [INFO] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: entering follower state: follower="Node at 4ffc338e-4794-c2a1-5b04-4961401f4d7f [Follower]" leader-address= leader-id= 2026-03-31T06:17:59.463Z [INFO] server-a3618a08-3134-1c50-85ec-b8d5bec6badd: entering follower state: follower="Node at a3618a08-3134-1c50-85ec-b8d5bec6badd [Follower]" leader-address= leader-id= 2026-03-31T06:17:59.463Z [INFO] server-c1094d08-d0f4-666d-def3-8d427748d4ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ffc338e-4794-c2a1-5b04-4961401f4d7f Address:4ffc338e-4794-c2a1-5b04-4961401f4d7f} {Suffrage:Voter ID:server-a3618a08-3134-1c50-85ec-b8d5bec6badd Address:a3618a08-3134-1c50-85ec-b8d5bec6badd} {Suffrage:Voter ID:server-c1094d08-d0f4-666d-def3-8d427748d4ab Address:c1094d08-d0f4-666d-def3-8d427748d4ab}]" 2026-03-31T06:17:59.463Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:59.463Z [INFO] server-c1094d08-d0f4-666d-def3-8d427748d4ab: entering follower state: follower="Node at c1094d08-d0f4-666d-def3-8d427748d4ab [Follower]" leader-address= leader-id= 2026-03-31T06:17:59.530Z [WARN] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:17:59.530Z [INFO] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: entering candidate state: node="Node at 4ffc338e-4794-c2a1-5b04-4961401f4d7f [Candidate]" term=2 2026-03-31T06:17:59.530Z [INFO] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: election won: term=2 tally=2 2026-03-31T06:17:59.530Z [INFO] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: entering leader state: leader="Node at 4ffc338e-4794-c2a1-5b04-4961401f4d7f [Leader]" 2026-03-31T06:17:59.530Z [INFO] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: added peer, starting replication: peer=server-a3618a08-3134-1c50-85ec-b8d5bec6badd 2026-03-31T06:17:59.530Z [INFO] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: added peer, starting replication: peer=server-c1094d08-d0f4-666d-def3-8d427748d4ab 2026-03-31T06:17:59.530Z [INFO] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: pipelining replication: peer="{Voter server-a3618a08-3134-1c50-85ec-b8d5bec6badd a3618a08-3134-1c50-85ec-b8d5bec6badd}" 2026-03-31T06:17:59.530Z [INFO] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: pipelining replication: peer="{Voter server-c1094d08-d0f4-666d-def3-8d427748d4ab c1094d08-d0f4-666d-def3-8d427748d4ab}" 2026-03-31T06:17:59.636Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:17:59.53071866 +0000 UTC m=+107.808326041 (1 nodes), 67.18752ms from start of poll, 67.58488ms from cluster start. Timeout at 2026-03-31 06:17:59.63572934 +0000 UTC m=+107.913336721, 105.01068ms after stability 2026-03-31T06:17:59.642Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:17:59.747Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:17:59.6420537 +0000 UTC m=+107.919661121 (2 nodes), 3.56µs from start of poll, 178.91996ms from cluster start. Timeout at 2026-03-31 06:17:59.7470781 +0000 UTC m=+108.024685481, 105.02436ms after stability 2026-03-31T06:17:59.747Z [INFO] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: updating configuration: command=RemoveServer server-id=server-a3618a08-3134-1c50-85ec-b8d5bec6badd server-addr= servers="[{Suffrage:Voter ID:server-4ffc338e-4794-c2a1-5b04-4961401f4d7f Address:4ffc338e-4794-c2a1-5b04-4961401f4d7f} {Suffrage:Voter ID:server-c1094d08-d0f4-666d-def3-8d427748d4ab Address:c1094d08-d0f4-666d-def3-8d427748d4ab}]" 2026-03-31T06:17:59.747Z [INFO] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: removed peer, stopping replication: peer=server-a3618a08-3134-1c50-85ec-b8d5bec6badd last-index=3 2026-03-31T06:17:59.747Z [INFO] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: aborting pipeline replication: peer="{Voter server-a3618a08-3134-1c50-85ec-b8d5bec6badd a3618a08-3134-1c50-85ec-b8d5bec6badd}" 2026-03-31T06:17:59.835Z [WARN] server-a3618a08-3134-1c50-85ec-b8d5bec6badd: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2026-03-31T06:17:59.854Z [ERROR] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: failed to heartbeat to: peer=c1094d08-d0f4-666d-def3-8d427748d4ab backoff time=10ms error="failed to connect to peer: c1094d08-d0f4-666d-def3-8d427748d4ab" 2026-03-31T06:17:59.862Z [ERROR] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: failed to pipeline appendEntries: peer="{Voter server-c1094d08-d0f4-666d-def3-8d427748d4ab c1094d08-d0f4-666d-def3-8d427748d4ab}" error="append pipeline closed" 2026-03-31T06:17:59.862Z [INFO] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: aborting pipeline replication: peer="{Voter server-c1094d08-d0f4-666d-def3-8d427748d4ab c1094d08-d0f4-666d-def3-8d427748d4ab}" 2026-03-31T06:17:59.869Z [ERROR] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: failed to appendEntries to: peer="{Voter server-c1094d08-d0f4-666d-def3-8d427748d4ab c1094d08-d0f4-666d-def3-8d427748d4ab}" error="failed to connect to peer: c1094d08-d0f4-666d-def3-8d427748d4ab" 2026-03-31T06:17:59.876Z [ERROR] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: failed to heartbeat to: peer=c1094d08-d0f4-666d-def3-8d427748d4ab backoff time=10ms error="failed to connect to peer: c1094d08-d0f4-666d-def3-8d427748d4ab" 2026-03-31T06:17:59.889Z [ERROR] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: failed to appendEntries to: peer="{Voter server-c1094d08-d0f4-666d-def3-8d427748d4ab c1094d08-d0f4-666d-def3-8d427748d4ab}" error="failed to connect to peer: c1094d08-d0f4-666d-def3-8d427748d4ab" 2026-03-31T06:17:59.893Z [ERROR] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: failed to heartbeat to: peer=c1094d08-d0f4-666d-def3-8d427748d4ab backoff time=10ms error="failed to connect to peer: c1094d08-d0f4-666d-def3-8d427748d4ab" 2026-03-31T06:17:59.903Z [WARN] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: failed to contact: server-id=server-c1094d08-d0f4-666d-def3-8d427748d4ab time=50.06728ms 2026-03-31T06:17:59.903Z [WARN] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: failed to contact quorum of nodes, stepping down 2026-03-31T06:17:59.903Z [INFO] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: entering follower state: follower="Node at 4ffc338e-4794-c2a1-5b04-4961401f4d7f [Follower]" leader-address= leader-id= 2026-03-31T06:17:59.908Z [ERROR] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: failed to heartbeat to: peer=c1094d08-d0f4-666d-def3-8d427748d4ab backoff time=20ms error="failed to connect to peer: c1094d08-d0f4-666d-def3-8d427748d4ab" 2026-03-31T06:17:59.910Z [ERROR] server-4ffc338e-4794-c2a1-5b04-4961401f4d7f: failed to appendEntries to: peer="{Voter server-c1094d08-d0f4-666d-def3-8d427748d4ab c1094d08-d0f4-666d-def3-8d427748d4ab}" error="failed to connect to peer: c1094d08-d0f4-666d-def3-8d427748d4ab" 2026-03-31T06:17:59.939Z [WARN] server-c1094d08-d0f4-666d-def3-8d427748d4ab: heartbeat timeout reached, starting election: last-leader-addr=4ffc338e-4794-c2a1-5b04-4961401f4d7f last-leader-id=server-4ffc338e-4794-c2a1-5b04-4961401f4d7f 2026-03-31T06:17:59.939Z [INFO] server-c1094d08-d0f4-666d-def3-8d427748d4ab: entering candidate state: node="Node at c1094d08-d0f4-666d-def3-8d427748d4ab [Candidate]" term=3 2026-03-31T06:17:59.939Z [ERROR] server-c1094d08-d0f4-666d-def3-8d427748d4ab: failed to make requestVote RPC: target="{Voter server-4ffc338e-4794-c2a1-5b04-4961401f4d7f 4ffc338e-4794-c2a1-5b04-4961401f4d7f}" error="failed to connect to peer: 4ffc338e-4794-c2a1-5b04-4961401f4d7f" term=3 2026-03-31T06:17:59.958Z [WARN] server-c1094d08-d0f4-666d-def3-8d427748d4ab: rejecting vote request since node is not in configuration: from=a3618a08-3134-1c50-85ec-b8d5bec6badd --- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.50s) === RUN TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin 2026-03-31T06:17:59.959Z [INFO] server-1069ee2b-29b0-04ed-d3f8-9147982749e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1069ee2b-29b0-04ed-d3f8-9147982749e2 Address:1069ee2b-29b0-04ed-d3f8-9147982749e2} {Suffrage:Voter ID:server-6b399796-cfe2-e9d2-fdb4-9813a1894569 Address:6b399796-cfe2-e9d2-fdb4-9813a1894569} {Suffrage:Voter ID:server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91 Address:d3dcd5fc-9f21-d4d4-397a-1f4073024c91}]" 2026-03-31T06:17:59.960Z [INFO] server-6b399796-cfe2-e9d2-fdb4-9813a1894569: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1069ee2b-29b0-04ed-d3f8-9147982749e2 Address:1069ee2b-29b0-04ed-d3f8-9147982749e2} {Suffrage:Voter ID:server-6b399796-cfe2-e9d2-fdb4-9813a1894569 Address:6b399796-cfe2-e9d2-fdb4-9813a1894569} {Suffrage:Voter ID:server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91 Address:d3dcd5fc-9f21-d4d4-397a-1f4073024c91}]" 2026-03-31T06:17:59.960Z [INFO] server-1069ee2b-29b0-04ed-d3f8-9147982749e2: entering follower state: follower="Node at 1069ee2b-29b0-04ed-d3f8-9147982749e2 [Follower]" leader-address= leader-id= 2026-03-31T06:17:59.960Z [INFO] server-6b399796-cfe2-e9d2-fdb4-9813a1894569: entering follower state: follower="Node at 6b399796-cfe2-e9d2-fdb4-9813a1894569 [Follower]" leader-address= leader-id= 2026-03-31T06:17:59.960Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1069ee2b-29b0-04ed-d3f8-9147982749e2 Address:1069ee2b-29b0-04ed-d3f8-9147982749e2} {Suffrage:Voter ID:server-6b399796-cfe2-e9d2-fdb4-9813a1894569 Address:6b399796-cfe2-e9d2-fdb4-9813a1894569} {Suffrage:Voter ID:server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91 Address:d3dcd5fc-9f21-d4d4-397a-1f4073024c91}]" 2026-03-31T06:17:59.960Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:17:59.960Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: entering follower state: follower="Node at d3dcd5fc-9f21-d4d4-397a-1f4073024c91 [Follower]" leader-address= leader-id= 2026-03-31T06:18:00.027Z [WARN] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:18:00.027Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: entering candidate state: node="Node at d3dcd5fc-9f21-d4d4-397a-1f4073024c91 [Candidate]" term=2 2026-03-31T06:18:00.027Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: election won: term=2 tally=2 2026-03-31T06:18:00.027Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: entering leader state: leader="Node at d3dcd5fc-9f21-d4d4-397a-1f4073024c91 [Leader]" 2026-03-31T06:18:00.027Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: added peer, starting replication: peer=server-1069ee2b-29b0-04ed-d3f8-9147982749e2 2026-03-31T06:18:00.027Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: added peer, starting replication: peer=server-6b399796-cfe2-e9d2-fdb4-9813a1894569 2026-03-31T06:18:00.027Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: pipelining replication: peer="{Voter server-6b399796-cfe2-e9d2-fdb4-9813a1894569 6b399796-cfe2-e9d2-fdb4-9813a1894569}" 2026-03-31T06:18:00.028Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: pipelining replication: peer="{Voter server-1069ee2b-29b0-04ed-d3f8-9147982749e2 1069ee2b-29b0-04ed-d3f8-9147982749e2}" 2026-03-31T06:18:00.133Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:18:00.02797118 +0000 UTC m=+108.305578601 (1 nodes), 67.7486ms from start of poll, 68.19228ms from cluster start. Timeout at 2026-03-31 06:18:00.13297418 +0000 UTC m=+108.410581561, 105.00296ms after stability 2026-03-31T06:18:00.139Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:18:00.244Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:18:00.13909882 +0000 UTC m=+108.416706201 (2 nodes), 1.92µs from start of poll, 179.31988ms from cluster start. Timeout at 2026-03-31 06:18:00.24412982 +0000 UTC m=+108.521737201, 105.031ms after stability 2026-03-31T06:18:00.245Z [ERROR] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: failed to heartbeat to: peer=1069ee2b-29b0-04ed-d3f8-9147982749e2 backoff time=10ms error="failed to connect to peer: 1069ee2b-29b0-04ed-d3f8-9147982749e2" 2026-03-31T06:18:00.251Z [ERROR] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: failed to pipeline appendEntries: peer="{Voter server-1069ee2b-29b0-04ed-d3f8-9147982749e2 1069ee2b-29b0-04ed-d3f8-9147982749e2}" error="append pipeline closed" 2026-03-31T06:18:00.251Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: aborting pipeline replication: peer="{Voter server-1069ee2b-29b0-04ed-d3f8-9147982749e2 1069ee2b-29b0-04ed-d3f8-9147982749e2}" 2026-03-31T06:18:00.261Z [ERROR] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: failed to appendEntries to: peer="{Voter server-1069ee2b-29b0-04ed-d3f8-9147982749e2 1069ee2b-29b0-04ed-d3f8-9147982749e2}" error="failed to connect to peer: 1069ee2b-29b0-04ed-d3f8-9147982749e2" 2026-03-31T06:18:00.264Z [ERROR] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: failed to heartbeat to: peer=1069ee2b-29b0-04ed-d3f8-9147982749e2 backoff time=10ms error="failed to connect to peer: 1069ee2b-29b0-04ed-d3f8-9147982749e2" 2026-03-31T06:18:00.281Z [ERROR] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: failed to heartbeat to: peer=1069ee2b-29b0-04ed-d3f8-9147982749e2 backoff time=10ms error="failed to connect to peer: 1069ee2b-29b0-04ed-d3f8-9147982749e2" 2026-03-31T06:18:00.282Z [ERROR] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: failed to appendEntries to: peer="{Voter server-1069ee2b-29b0-04ed-d3f8-9147982749e2 1069ee2b-29b0-04ed-d3f8-9147982749e2}" error="failed to connect to peer: 1069ee2b-29b0-04ed-d3f8-9147982749e2" 2026-03-31T06:18:00.293Z [WARN] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: failed to contact: server-id=server-1069ee2b-29b0-04ed-d3f8-9147982749e2 time=50.37372ms 2026-03-31T06:18:00.298Z [ERROR] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: failed to heartbeat to: peer=1069ee2b-29b0-04ed-d3f8-9147982749e2 backoff time=20ms error="failed to connect to peer: 1069ee2b-29b0-04ed-d3f8-9147982749e2" 2026-03-31T06:18:00.301Z [ERROR] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: failed to appendEntries to: peer="{Voter server-1069ee2b-29b0-04ed-d3f8-9147982749e2 1069ee2b-29b0-04ed-d3f8-9147982749e2}" error="failed to connect to peer: 1069ee2b-29b0-04ed-d3f8-9147982749e2" 2026-03-31T06:18:00.327Z [ERROR] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: failed to heartbeat to: peer=1069ee2b-29b0-04ed-d3f8-9147982749e2 backoff time=25ms error="failed to connect to peer: 1069ee2b-29b0-04ed-d3f8-9147982749e2" 2026-03-31T06:18:00.328Z [ERROR] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: failed to appendEntries to: peer="{Voter server-1069ee2b-29b0-04ed-d3f8-9147982749e2 1069ee2b-29b0-04ed-d3f8-9147982749e2}" error="failed to connect to peer: 1069ee2b-29b0-04ed-d3f8-9147982749e2" 2026-03-31T06:18:00.342Z [WARN] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: failed to contact: server-id=server-1069ee2b-29b0-04ed-d3f8-9147982749e2 time=99.50116ms 2026-03-31T06:18:00.349Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: updating configuration: command=RemoveServer server-id=server-1069ee2b-29b0-04ed-d3f8-9147982749e2 server-addr= servers="[{Suffrage:Voter ID:server-6b399796-cfe2-e9d2-fdb4-9813a1894569 Address:6b399796-cfe2-e9d2-fdb4-9813a1894569} {Suffrage:Voter ID:server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91 Address:d3dcd5fc-9f21-d4d4-397a-1f4073024c91}]" 2026-03-31T06:18:00.349Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: removed peer, stopping replication: peer=server-1069ee2b-29b0-04ed-d3f8-9147982749e2 last-index=3 2026-03-31T06:18:00.350Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: updating configuration: command=AddNonvoter server-id=server-1069ee2b-29b0-04ed-d3f8-9147982749e2 server-addr=1069ee2b-29b0-04ed-d3f8-9147982749e2 servers="[{Suffrage:Voter ID:server-6b399796-cfe2-e9d2-fdb4-9813a1894569 Address:6b399796-cfe2-e9d2-fdb4-9813a1894569} {Suffrage:Voter ID:server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91 Address:d3dcd5fc-9f21-d4d4-397a-1f4073024c91} {Suffrage:Nonvoter ID:server-1069ee2b-29b0-04ed-d3f8-9147982749e2 Address:1069ee2b-29b0-04ed-d3f8-9147982749e2}]" 2026-03-31T06:18:00.350Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: added peer, starting replication: peer=server-1069ee2b-29b0-04ed-d3f8-9147982749e2 2026-03-31T06:18:00.350Z [ERROR] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: failed to appendEntries to: peer="{Nonvoter server-1069ee2b-29b0-04ed-d3f8-9147982749e2 1069ee2b-29b0-04ed-d3f8-9147982749e2}" error="failed to connect to peer: 1069ee2b-29b0-04ed-d3f8-9147982749e2" 2026-03-31T06:18:00.364Z [WARN] server-1069ee2b-29b0-04ed-d3f8-9147982749e2: heartbeat timeout reached, starting election: last-leader-addr=d3dcd5fc-9f21-d4d4-397a-1f4073024c91 last-leader-id=server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91 2026-03-31T06:18:00.364Z [INFO] server-1069ee2b-29b0-04ed-d3f8-9147982749e2: entering candidate state: node="Node at 1069ee2b-29b0-04ed-d3f8-9147982749e2 [Candidate]" term=23 2026-03-31T06:18:00.364Z [WARN] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: rejecting vote request since we have a leader: from=1069ee2b-29b0-04ed-d3f8-9147982749e2 leader=d3dcd5fc-9f21-d4d4-397a-1f4073024c91 leader-id=server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91 2026-03-31T06:18:00.364Z [WARN] server-6b399796-cfe2-e9d2-fdb4-9813a1894569: rejecting vote request since we have a leader: from=1069ee2b-29b0-04ed-d3f8-9147982749e2 leader=d3dcd5fc-9f21-d4d4-397a-1f4073024c91 leader-id=server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91 2026-03-31T06:18:00.371Z [ERROR] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: peer has newer term, stopping replication: peer="{Nonvoter server-1069ee2b-29b0-04ed-d3f8-9147982749e2 1069ee2b-29b0-04ed-d3f8-9147982749e2}" 2026-03-31T06:18:00.371Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: entering follower state: follower="Node at d3dcd5fc-9f21-d4d4-397a-1f4073024c91 [Follower]" leader-address= leader-id= 2026-03-31T06:18:00.371Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: aborting pipeline replication: peer="{Voter server-6b399796-cfe2-e9d2-fdb4-9813a1894569 6b399796-cfe2-e9d2-fdb4-9813a1894569}" 2026-03-31T06:18:00.377Z [ERROR] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: peer has newer term, stopping replication: peer="{Voter server-1069ee2b-29b0-04ed-d3f8-9147982749e2 1069ee2b-29b0-04ed-d3f8-9147982749e2}" 2026-03-31T06:18:00.428Z [WARN] server-1069ee2b-29b0-04ed-d3f8-9147982749e2: Election timeout reached, restarting election 2026-03-31T06:18:00.428Z [INFO] server-1069ee2b-29b0-04ed-d3f8-9147982749e2: entering candidate state: node="Node at 1069ee2b-29b0-04ed-d3f8-9147982749e2 [Candidate]" term=24 2026-03-31T06:18:00.428Z [WARN] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: rejecting vote request since node is not a voter: from=1069ee2b-29b0-04ed-d3f8-9147982749e2 2026-03-31T06:18:00.428Z [WARN] server-6b399796-cfe2-e9d2-fdb4-9813a1894569: rejecting vote request since we have a leader: from=1069ee2b-29b0-04ed-d3f8-9147982749e2 leader=d3dcd5fc-9f21-d4d4-397a-1f4073024c91 leader-id=server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91 2026-03-31T06:18:00.456Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:18:00.464Z [WARN] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:18:00.464Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: entering candidate state: node="Node at d3dcd5fc-9f21-d4d4-397a-1f4073024c91 [Candidate]" term=25 2026-03-31T06:18:00.464Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: election won: term=25 tally=2 2026-03-31T06:18:00.464Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: entering leader state: leader="Node at d3dcd5fc-9f21-d4d4-397a-1f4073024c91 [Leader]" 2026-03-31T06:18:00.464Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: added peer, starting replication: peer=server-6b399796-cfe2-e9d2-fdb4-9813a1894569 2026-03-31T06:18:00.464Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: added peer, starting replication: peer=server-1069ee2b-29b0-04ed-d3f8-9147982749e2 2026-03-31T06:18:00.464Z [WARN] server-1069ee2b-29b0-04ed-d3f8-9147982749e2: failed to get previous log: previous-index=4 last-index=2 error="log not found" 2026-03-31T06:18:00.464Z [INFO] server-1069ee2b-29b0-04ed-d3f8-9147982749e2: entering follower state: follower="Node at 1069ee2b-29b0-04ed-d3f8-9147982749e2 [Follower]" leader-address=d3dcd5fc-9f21-d4d4-397a-1f4073024c91 leader-id=server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91 2026-03-31T06:18:00.464Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: pipelining replication: peer="{Voter server-6b399796-cfe2-e9d2-fdb4-9813a1894569 6b399796-cfe2-e9d2-fdb4-9813a1894569}" 2026-03-31T06:18:00.464Z [WARN] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: appendEntries rejected, sending older logs: peer="{Nonvoter server-1069ee2b-29b0-04ed-d3f8-9147982749e2 1069ee2b-29b0-04ed-d3f8-9147982749e2}" next=3 2026-03-31T06:18:00.464Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: pipelining replication: peer="{Nonvoter server-1069ee2b-29b0-04ed-d3f8-9147982749e2 1069ee2b-29b0-04ed-d3f8-9147982749e2}" 2026-03-31T06:18:00.569Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:18:00.4644643 +0000 UTC m=+108.742071681 (1 nodes), 7.72128ms from start of poll, 504.68536ms from cluster start. Timeout at 2026-03-31 06:18:00.56946746 +0000 UTC m=+108.847074841, 105.00316ms after stability 2026-03-31T06:18:00.595Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: updating configuration: command=RemoveServer server-id=server-1069ee2b-29b0-04ed-d3f8-9147982749e2 server-addr= servers="[{Suffrage:Voter ID:server-6b399796-cfe2-e9d2-fdb4-9813a1894569 Address:6b399796-cfe2-e9d2-fdb4-9813a1894569} {Suffrage:Voter ID:server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91 Address:d3dcd5fc-9f21-d4d4-397a-1f4073024c91}]" 2026-03-31T06:18:00.595Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: removed peer, stopping replication: peer=server-1069ee2b-29b0-04ed-d3f8-9147982749e2 last-index=106 2026-03-31T06:18:00.596Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: aborting pipeline replication: peer="{Nonvoter server-1069ee2b-29b0-04ed-d3f8-9147982749e2 1069ee2b-29b0-04ed-d3f8-9147982749e2}" 2026-03-31T06:18:00.596Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: updating configuration: command=AddVoter server-id=server-1069ee2b-29b0-04ed-d3f8-9147982749e2 server-addr=1069ee2b-29b0-04ed-d3f8-9147982749e2 servers="[{Suffrage:Voter ID:server-6b399796-cfe2-e9d2-fdb4-9813a1894569 Address:6b399796-cfe2-e9d2-fdb4-9813a1894569} {Suffrage:Voter ID:server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91 Address:d3dcd5fc-9f21-d4d4-397a-1f4073024c91} {Suffrage:Voter ID:server-1069ee2b-29b0-04ed-d3f8-9147982749e2 Address:1069ee2b-29b0-04ed-d3f8-9147982749e2}]" 2026-03-31T06:18:00.596Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: added peer, starting replication: peer=server-1069ee2b-29b0-04ed-d3f8-9147982749e2 2026-03-31T06:18:00.596Z [WARN] server-1069ee2b-29b0-04ed-d3f8-9147982749e2: failed to get previous log: previous-index=107 last-index=106 error="log not found" 2026-03-31T06:18:00.596Z [WARN] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: appendEntries rejected, sending older logs: peer="{Voter server-1069ee2b-29b0-04ed-d3f8-9147982749e2 1069ee2b-29b0-04ed-d3f8-9147982749e2}" next=107 2026-03-31T06:18:00.596Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: pipelining replication: peer="{Voter server-1069ee2b-29b0-04ed-d3f8-9147982749e2 1069ee2b-29b0-04ed-d3f8-9147982749e2}" 2026-03-31T06:18:01.669Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:18:01.775Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:18:01.66926138 +0000 UTC m=+109.946868761 (1 nodes), 1.12µs from start of poll, 1.70948244s from cluster start. Timeout at 2026-03-31 06:18:01.77429386 +0000 UTC m=+110.051901281, 105.03252ms after stability 2026-03-31T06:18:01.775Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: aborting pipeline replication: peer="{Voter server-1069ee2b-29b0-04ed-d3f8-9147982749e2 1069ee2b-29b0-04ed-d3f8-9147982749e2}" 2026-03-31T06:18:01.775Z [INFO] server-d3dcd5fc-9f21-d4d4-397a-1f4073024c91: aborting pipeline replication: peer="{Voter server-6b399796-cfe2-e9d2-fdb4-9813a1894569 6b399796-cfe2-e9d2-fdb4-9813a1894569}" --- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.82s) === RUN TestRaft_FollowerRemovalNoElection 2026-03-31T06:18:01.776Z [INFO] server-1532fafc-d768-7ed9-e44e-da1908165de7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1532fafc-d768-7ed9-e44e-da1908165de7 Address:1532fafc-d768-7ed9-e44e-da1908165de7} {Suffrage:Voter ID:server-e4fdf3b0-1333-71be-5b89-134979621997 Address:e4fdf3b0-1333-71be-5b89-134979621997} {Suffrage:Voter ID:server-5a1b57c6-f45c-10f9-bacd-73d7bf5adfb5 Address:5a1b57c6-f45c-10f9-bacd-73d7bf5adfb5}]" 2026-03-31T06:18:01.776Z [INFO] server-e4fdf3b0-1333-71be-5b89-134979621997: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1532fafc-d768-7ed9-e44e-da1908165de7 Address:1532fafc-d768-7ed9-e44e-da1908165de7} {Suffrage:Voter ID:server-e4fdf3b0-1333-71be-5b89-134979621997 Address:e4fdf3b0-1333-71be-5b89-134979621997} {Suffrage:Voter ID:server-5a1b57c6-f45c-10f9-bacd-73d7bf5adfb5 Address:5a1b57c6-f45c-10f9-bacd-73d7bf5adfb5}]" 2026-03-31T06:18:01.776Z [INFO] server-1532fafc-d768-7ed9-e44e-da1908165de7: entering follower state: follower="Node at 1532fafc-d768-7ed9-e44e-da1908165de7 [Follower]" leader-address= leader-id= 2026-03-31T06:18:01.776Z [INFO] server-e4fdf3b0-1333-71be-5b89-134979621997: entering follower state: follower="Node at e4fdf3b0-1333-71be-5b89-134979621997 [Follower]" leader-address= leader-id= 2026-03-31T06:18:01.776Z [INFO] server-5a1b57c6-f45c-10f9-bacd-73d7bf5adfb5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1532fafc-d768-7ed9-e44e-da1908165de7 Address:1532fafc-d768-7ed9-e44e-da1908165de7} {Suffrage:Voter ID:server-e4fdf3b0-1333-71be-5b89-134979621997 Address:e4fdf3b0-1333-71be-5b89-134979621997} {Suffrage:Voter ID:server-5a1b57c6-f45c-10f9-bacd-73d7bf5adfb5 Address:5a1b57c6-f45c-10f9-bacd-73d7bf5adfb5}]" 2026-03-31T06:18:01.776Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:18:01.776Z [INFO] server-5a1b57c6-f45c-10f9-bacd-73d7bf5adfb5: entering follower state: follower="Node at 5a1b57c6-f45c-10f9-bacd-73d7bf5adfb5 [Follower]" leader-address= leader-id= 2026-03-31T06:18:01.934Z [WARN] server-1532fafc-d768-7ed9-e44e-da1908165de7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:18:01.934Z [INFO] server-1532fafc-d768-7ed9-e44e-da1908165de7: entering candidate state: node="Node at 1532fafc-d768-7ed9-e44e-da1908165de7 [Candidate]" term=2 2026-03-31T06:18:01.934Z [WARN] server-e4fdf3b0-1333-71be-5b89-134979621997: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:18:01.934Z [INFO] server-e4fdf3b0-1333-71be-5b89-134979621997: entering candidate state: node="Node at e4fdf3b0-1333-71be-5b89-134979621997 [Candidate]" term=2 2026-03-31T06:18:01.934Z [INFO] server-1532fafc-d768-7ed9-e44e-da1908165de7: election won: term=2 tally=2 2026-03-31T06:18:01.934Z [INFO] server-e4fdf3b0-1333-71be-5b89-134979621997: duplicate requestVote for same term: term=2 2026-03-31T06:18:01.934Z [INFO] server-1532fafc-d768-7ed9-e44e-da1908165de7: entering leader state: leader="Node at 1532fafc-d768-7ed9-e44e-da1908165de7 [Leader]" 2026-03-31T06:18:01.934Z [INFO] server-1532fafc-d768-7ed9-e44e-da1908165de7: added peer, starting replication: peer=server-e4fdf3b0-1333-71be-5b89-134979621997 2026-03-31T06:18:01.934Z [INFO] server-1532fafc-d768-7ed9-e44e-da1908165de7: added peer, starting replication: peer=server-5a1b57c6-f45c-10f9-bacd-73d7bf5adfb5 2026-03-31T06:18:01.934Z [INFO] server-5a1b57c6-f45c-10f9-bacd-73d7bf5adfb5: duplicate requestVote for same term: term=2 2026-03-31T06:18:01.934Z [WARN] server-1532fafc-d768-7ed9-e44e-da1908165de7: rejecting vote request since we have a leader: from=e4fdf3b0-1333-71be-5b89-134979621997 leader=1532fafc-d768-7ed9-e44e-da1908165de7 leader-id=server-1532fafc-d768-7ed9-e44e-da1908165de7 2026-03-31T06:18:01.935Z [INFO] server-1532fafc-d768-7ed9-e44e-da1908165de7: pipelining replication: peer="{Voter server-5a1b57c6-f45c-10f9-bacd-73d7bf5adfb5 5a1b57c6-f45c-10f9-bacd-73d7bf5adfb5}" 2026-03-31T06:18:01.935Z [INFO] server-1532fafc-d768-7ed9-e44e-da1908165de7: pipelining replication: peer="{Voter server-e4fdf3b0-1333-71be-5b89-134979621997 e4fdf3b0-1333-71be-5b89-134979621997}" 2026-03-31T06:18:01.935Z [INFO] server-e4fdf3b0-1333-71be-5b89-134979621997: entering follower state: follower="Node at e4fdf3b0-1333-71be-5b89-134979621997 [Follower]" leader-address=1532fafc-d768-7ed9-e44e-da1908165de7 leader-id=server-1532fafc-d768-7ed9-e44e-da1908165de7 2026-03-31T06:18:02.140Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:18:01.93503534 +0000 UTC m=+110.212642761 (1 nodes), 158.52876ms from start of poll, 158.93716ms from cluster start. Timeout at 2026-03-31 06:18:02.14003662 +0000 UTC m=+110.417644001, 205.00124ms after stability 2026-03-31T06:18:02.140Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:18:02.346Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:18:02.1401679 +0000 UTC m=+110.417775281 (1 nodes), 2.4µs from start of poll, 364.06968ms from cluster start. Timeout at 2026-03-31 06:18:02.3452037 +0000 UTC m=+110.622811081, 205.0358ms after stability 2026-03-31T06:18:02.351Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:18:02.556Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:18:02.3514961 +0000 UTC m=+110.629103521 (2 nodes), 2.92µs from start of poll, 575.39792ms from cluster start. Timeout at 2026-03-31 06:18:02.55652598 +0000 UTC m=+110.834133361, 205.02984ms after stability 2026-03-31T06:18:02.556Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:18:02.762Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:18:02.5566793 +0000 UTC m=+110.834286721 (1 nodes), 2.48µs from start of poll, 780.58112ms from cluster start. Timeout at 2026-03-31 06:18:02.76171398 +0000 UTC m=+111.039321401, 205.03468ms after stability raft_test.go:2961: [INFO] restarting Node at e4fdf3b0-1333-71be-5b89-134979621997 [Follower] 2026-03-31T06:18:02.762Z [INFO] server-e4fdf3b0-1333-71be-5b89-134979621997: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1532fafc-d768-7ed9-e44e-da1908165de7 Address:1532fafc-d768-7ed9-e44e-da1908165de7} {Suffrage:Voter ID:server-e4fdf3b0-1333-71be-5b89-134979621997 Address:e4fdf3b0-1333-71be-5b89-134979621997} {Suffrage:Voter ID:server-5a1b57c6-f45c-10f9-bacd-73d7bf5adfb5 Address:5a1b57c6-f45c-10f9-bacd-73d7bf5adfb5}]" 2026-03-31T06:18:02.762Z [INFO] server-e4fdf3b0-1333-71be-5b89-134979621997: entering follower state: follower="Node at e4fdf3b0-1333-71be-5b89-134979621997 [Follower]" leader-address= leader-id= 2026-03-31T06:18:03.012Z [INFO] server-1532fafc-d768-7ed9-e44e-da1908165de7: aborting pipeline replication: peer="{Voter server-5a1b57c6-f45c-10f9-bacd-73d7bf5adfb5 5a1b57c6-f45c-10f9-bacd-73d7bf5adfb5}" 2026-03-31T06:18:03.385Z [ERROR] server-1532fafc-d768-7ed9-e44e-da1908165de7: failed to pipeline appendEntries: peer="{Voter server-e4fdf3b0-1333-71be-5b89-134979621997 e4fdf3b0-1333-71be-5b89-134979621997}" error="command enqueue timeout" 2026-03-31T06:18:03.385Z [INFO] server-1532fafc-d768-7ed9-e44e-da1908165de7: aborting pipeline replication: peer="{Voter server-e4fdf3b0-1333-71be-5b89-134979621997 e4fdf3b0-1333-71be-5b89-134979621997}" 2026-03-31T06:18:03.518Z [ERROR] server-1532fafc-d768-7ed9-e44e-da1908165de7: failed to heartbeat to: peer=e4fdf3b0-1333-71be-5b89-134979621997 backoff time=10ms error="command timed out" --- PASS: TestRaft_FollowerRemovalNoElection (1.74s) === RUN TestRaft_VoteWithNoIDNoAddr 2026-03-31T06:18:03.518Z [INFO] server-0ad9bd12-e59f-1841-e4ea-49eb37419d45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ad9bd12-e59f-1841-e4ea-49eb37419d45 Address:0ad9bd12-e59f-1841-e4ea-49eb37419d45} {Suffrage:Voter ID:server-7944fac8-8013-88c0-c5fa-eff7f4f47452 Address:7944fac8-8013-88c0-c5fa-eff7f4f47452} {Suffrage:Voter ID:server-19c5b7ee-f229-f986-b138-7fb6144ed634 Address:19c5b7ee-f229-f986-b138-7fb6144ed634}]" 2026-03-31T06:18:03.518Z [INFO] server-0ad9bd12-e59f-1841-e4ea-49eb37419d45: entering follower state: follower="Node at 0ad9bd12-e59f-1841-e4ea-49eb37419d45 [Follower]" leader-address= leader-id= 2026-03-31T06:18:03.519Z [INFO] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ad9bd12-e59f-1841-e4ea-49eb37419d45 Address:0ad9bd12-e59f-1841-e4ea-49eb37419d45} {Suffrage:Voter ID:server-7944fac8-8013-88c0-c5fa-eff7f4f47452 Address:7944fac8-8013-88c0-c5fa-eff7f4f47452} {Suffrage:Voter ID:server-19c5b7ee-f229-f986-b138-7fb6144ed634 Address:19c5b7ee-f229-f986-b138-7fb6144ed634}]" 2026-03-31T06:18:03.519Z [INFO] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: entering follower state: follower="Node at 7944fac8-8013-88c0-c5fa-eff7f4f47452 [Follower]" leader-address= leader-id= 2026-03-31T06:18:03.519Z [INFO] server-19c5b7ee-f229-f986-b138-7fb6144ed634: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ad9bd12-e59f-1841-e4ea-49eb37419d45 Address:0ad9bd12-e59f-1841-e4ea-49eb37419d45} {Suffrage:Voter ID:server-7944fac8-8013-88c0-c5fa-eff7f4f47452 Address:7944fac8-8013-88c0-c5fa-eff7f4f47452} {Suffrage:Voter ID:server-19c5b7ee-f229-f986-b138-7fb6144ed634 Address:19c5b7ee-f229-f986-b138-7fb6144ed634}]" 2026-03-31T06:18:03.519Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:18:03.519Z [INFO] server-19c5b7ee-f229-f986-b138-7fb6144ed634: entering follower state: follower="Node at 19c5b7ee-f229-f986-b138-7fb6144ed634 [Follower]" leader-address= leader-id= 2026-03-31T06:18:03.584Z [WARN] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-03-31T06:18:03.584Z [INFO] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: entering candidate state: node="Node at 7944fac8-8013-88c0-c5fa-eff7f4f47452 [Candidate]" term=2 2026-03-31T06:18:03.584Z [INFO] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: election won: term=2 tally=2 2026-03-31T06:18:03.584Z [INFO] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: entering leader state: leader="Node at 7944fac8-8013-88c0-c5fa-eff7f4f47452 [Leader]" 2026-03-31T06:18:03.584Z [INFO] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: added peer, starting replication: peer=server-0ad9bd12-e59f-1841-e4ea-49eb37419d45 2026-03-31T06:18:03.584Z [INFO] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: added peer, starting replication: peer=server-19c5b7ee-f229-f986-b138-7fb6144ed634 2026-03-31T06:18:03.584Z [INFO] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: pipelining replication: peer="{Voter server-0ad9bd12-e59f-1841-e4ea-49eb37419d45 0ad9bd12-e59f-1841-e4ea-49eb37419d45}" 2026-03-31T06:18:03.584Z [INFO] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: pipelining replication: peer="{Voter server-19c5b7ee-f229-f986-b138-7fb6144ed634 19c5b7ee-f229-f986-b138-7fb6144ed634}" 2026-03-31T06:18:03.690Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:18:03.58453818 +0000 UTC m=+111.862145561 (1 nodes), 65.21984ms from start of poll, 65.7668ms from cluster start. Timeout at 2026-03-31 06:18:03.68954226 +0000 UTC m=+111.967149641, 105.00408ms after stability 2026-03-31T06:18:03.690Z [INFO] cluster: starting stability test: raft-state=Leader 2026-03-31T06:18:03.796Z [INFO] cluster: stable state for Leader reached at 2026-03-31 06:18:03.69077102 +0000 UTC m=+111.968378441 (1 nodes), 2.48µs from start of poll, 171.99968ms from cluster start. Timeout at 2026-03-31 06:18:03.79578802 +0000 UTC m=+112.073395401, 105.01696ms after stability 2026-03-31T06:18:03.801Z [INFO] cluster: starting stability test: raft-state=Follower 2026-03-31T06:18:03.906Z [INFO] cluster: stable state for Follower reached at 2026-03-31 06:18:03.80130458 +0000 UTC m=+112.078911961 (2 nodes), 3.08µs from start of poll, 282.5332ms from cluster start. Timeout at 2026-03-31 06:18:03.90634314 +0000 UTC m=+112.183950561, 105.0386ms after stability 2026-03-31T06:18:03.910Z [ERROR] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to heartbeat to: peer=0ad9bd12-e59f-1841-e4ea-49eb37419d45 backoff time=10ms error="failed to connect to peer: 0ad9bd12-e59f-1841-e4ea-49eb37419d45" 2026-03-31T06:18:03.910Z [ERROR] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to pipeline appendEntries: peer="{Voter server-0ad9bd12-e59f-1841-e4ea-49eb37419d45 0ad9bd12-e59f-1841-e4ea-49eb37419d45}" error="append pipeline closed" 2026-03-31T06:18:03.910Z [INFO] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: aborting pipeline replication: peer="{Voter server-0ad9bd12-e59f-1841-e4ea-49eb37419d45 0ad9bd12-e59f-1841-e4ea-49eb37419d45}" 2026-03-31T06:18:03.910Z [ERROR] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to pipeline appendEntries: peer="{Voter server-19c5b7ee-f229-f986-b138-7fb6144ed634 19c5b7ee-f229-f986-b138-7fb6144ed634}" error="append pipeline closed" 2026-03-31T06:18:03.910Z [INFO] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: aborting pipeline replication: peer="{Voter server-19c5b7ee-f229-f986-b138-7fb6144ed634 19c5b7ee-f229-f986-b138-7fb6144ed634}" 2026-03-31T06:18:03.911Z [ERROR] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to heartbeat to: peer=19c5b7ee-f229-f986-b138-7fb6144ed634 backoff time=10ms error="failed to connect to peer: 19c5b7ee-f229-f986-b138-7fb6144ed634" 2026-03-31T06:18:03.916Z [ERROR] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to appendEntries to: peer="{Voter server-0ad9bd12-e59f-1841-e4ea-49eb37419d45 0ad9bd12-e59f-1841-e4ea-49eb37419d45}" error="failed to connect to peer: 0ad9bd12-e59f-1841-e4ea-49eb37419d45" 2026-03-31T06:18:03.919Z [ERROR] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to appendEntries to: peer="{Voter server-19c5b7ee-f229-f986-b138-7fb6144ed634 19c5b7ee-f229-f986-b138-7fb6144ed634}" error="failed to connect to peer: 19c5b7ee-f229-f986-b138-7fb6144ed634" 2026-03-31T06:18:03.928Z [ERROR] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to heartbeat to: peer=19c5b7ee-f229-f986-b138-7fb6144ed634 backoff time=10ms error="failed to connect to peer: 19c5b7ee-f229-f986-b138-7fb6144ed634" 2026-03-31T06:18:03.928Z [ERROR] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to heartbeat to: peer=0ad9bd12-e59f-1841-e4ea-49eb37419d45 backoff time=10ms error="failed to connect to peer: 0ad9bd12-e59f-1841-e4ea-49eb37419d45" 2026-03-31T06:18:03.934Z [ERROR] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to appendEntries to: peer="{Voter server-0ad9bd12-e59f-1841-e4ea-49eb37419d45 0ad9bd12-e59f-1841-e4ea-49eb37419d45}" error="failed to connect to peer: 0ad9bd12-e59f-1841-e4ea-49eb37419d45" 2026-03-31T06:18:03.937Z [ERROR] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to appendEntries to: peer="{Voter server-19c5b7ee-f229-f986-b138-7fb6144ed634 19c5b7ee-f229-f986-b138-7fb6144ed634}" error="failed to connect to peer: 19c5b7ee-f229-f986-b138-7fb6144ed634" 2026-03-31T06:18:03.945Z [ERROR] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to heartbeat to: peer=19c5b7ee-f229-f986-b138-7fb6144ed634 backoff time=10ms error="failed to connect to peer: 19c5b7ee-f229-f986-b138-7fb6144ed634" 2026-03-31T06:18:03.946Z [ERROR] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to heartbeat to: peer=0ad9bd12-e59f-1841-e4ea-49eb37419d45 backoff time=10ms error="failed to connect to peer: 0ad9bd12-e59f-1841-e4ea-49eb37419d45" 2026-03-31T06:18:03.953Z [WARN] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to contact: server-id=server-0ad9bd12-e59f-1841-e4ea-49eb37419d45 time=50.17272ms 2026-03-31T06:18:03.954Z [ERROR] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to appendEntries to: peer="{Voter server-0ad9bd12-e59f-1841-e4ea-49eb37419d45 0ad9bd12-e59f-1841-e4ea-49eb37419d45}" error="failed to connect to peer: 0ad9bd12-e59f-1841-e4ea-49eb37419d45" 2026-03-31T06:18:03.954Z [ERROR] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to appendEntries to: peer="{Voter server-19c5b7ee-f229-f986-b138-7fb6144ed634 19c5b7ee-f229-f986-b138-7fb6144ed634}" error="failed to connect to peer: 19c5b7ee-f229-f986-b138-7fb6144ed634" 2026-03-31T06:18:03.963Z [WARN] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to contact: server-id=server-0ad9bd12-e59f-1841-e4ea-49eb37419d45 time=60.25508ms 2026-03-31T06:18:03.963Z [WARN] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to contact: server-id=server-19c5b7ee-f229-f986-b138-7fb6144ed634 time=57.7394ms 2026-03-31T06:18:03.963Z [WARN] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to contact quorum of nodes, stepping down 2026-03-31T06:18:03.963Z [ERROR] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to heartbeat to: peer=0ad9bd12-e59f-1841-e4ea-49eb37419d45 backoff time=20ms error="failed to connect to peer: 0ad9bd12-e59f-1841-e4ea-49eb37419d45" 2026-03-31T06:18:03.963Z [INFO] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: entering follower state: follower="Node at 7944fac8-8013-88c0-c5fa-eff7f4f47452 [Follower]" leader-address= leader-id= 2026-03-31T06:18:03.973Z [WARN] server-19c5b7ee-f229-f986-b138-7fb6144ed634: heartbeat timeout reached, starting election: last-leader-addr=7944fac8-8013-88c0-c5fa-eff7f4f47452 last-leader-id=server-7944fac8-8013-88c0-c5fa-eff7f4f47452 2026-03-31T06:18:03.973Z [INFO] server-19c5b7ee-f229-f986-b138-7fb6144ed634: entering candidate state: node="Node at 19c5b7ee-f229-f986-b138-7fb6144ed634 [Candidate]" term=3 2026-03-31T06:18:03.973Z [ERROR] server-19c5b7ee-f229-f986-b138-7fb6144ed634: failed to make requestVote RPC: target="{Voter server-7944fac8-8013-88c0-c5fa-eff7f4f47452 7944fac8-8013-88c0-c5fa-eff7f4f47452}" error="failed to connect to peer: 7944fac8-8013-88c0-c5fa-eff7f4f47452" term=3 2026-03-31T06:18:03.973Z [WARN] server-0ad9bd12-e59f-1841-e4ea-49eb37419d45: rejecting vote request since we have a leader: from=19c5b7ee-f229-f986-b138-7fb6144ed634 leader=7944fac8-8013-88c0-c5fa-eff7f4f47452 leader-id=server-7944fac8-8013-88c0-c5fa-eff7f4f47452 2026-03-31T06:18:03.982Z [ERROR] server-7944fac8-8013-88c0-c5fa-eff7f4f47452: failed to appendEntries to: peer="{Voter server-0ad9bd12-e59f-1841-e4ea-49eb37419d45 0ad9bd12-e59f-1841-e4ea-49eb37419d45}" error="failed to connect to peer: 0ad9bd12-e59f-1841-e4ea-49eb37419d45" 2026-03-31T06:18:04.002Z [WARN] server-0ad9bd12-e59f-1841-e4ea-49eb37419d45: heartbeat timeout reached, starting election: last-leader-addr=7944fac8-8013-88c0-c5fa-eff7f4f47452 last-leader-id=server-7944fac8-8013-88c0-c5fa-eff7f4f47452 2026-03-31T06:18:04.002Z [INFO] server-0ad9bd12-e59f-1841-e4ea-49eb37419d45: entering candidate state: node="Node at 0ad9bd12-e59f-1841-e4ea-49eb37419d45 [Candidate]" term=3 2026-03-31T06:18:04.003Z [INFO] server-19c5b7ee-f229-f986-b138-7fb6144ed634: duplicate requestVote for same term: term=3 2026-03-31T06:18:04.003Z [ERROR] server-0ad9bd12-e59f-1841-e4ea-49eb37419d45: failed to make requestVote RPC: target="{Voter server-7944fac8-8013-88c0-c5fa-eff7f4f47452 7944fac8-8013-88c0-c5fa-eff7f4f47452}" error="failed to connect to peer: 7944fac8-8013-88c0-c5fa-eff7f4f47452" term=3 2026-03-31T06:18:04.012Z [INFO] server-0ad9bd12-e59f-1841-e4ea-49eb37419d45: entering follower state: follower="Node at 0ad9bd12-e59f-1841-e4ea-49eb37419d45 [Follower]" leader-address= leader-id= --- PASS: TestRaft_VoteWithNoIDNoAddr (0.49s) === RUN TestRaft_runFollower_State_Transition === RUN TestRaft_runFollower_State_Transition/NonVoter 2026-03-31T06:18:04.013Z [INFO] 127.0.0.1:40615 :: starting node: addr=127.0.0.1:40615 2026-03-31T06:18:04.013Z [INFO] 127.0.0.1:40615 :: initial configuration: index=0 servers=[] 2026-03-31T06:18:04.013Z [INFO] 127.0.0.1:40615 :: entering follower state: follower="Node at 127.0.0.1:40615 [Follower]" leader-address= leader-id= 2026-03-31T06:18:04.095Z [WARN] 127.0.0.1:40615 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election === RUN TestRaft_runFollower_State_Transition/Voter 2026-03-31T06:18:04.164Z [INFO] 127.0.0.1:40191 :: starting node: addr=127.0.0.1:40191 2026-03-31T06:18:04.164Z [INFO] 127.0.0.1:40191 :: initial configuration: index=0 servers=[] 2026-03-31T06:18:04.164Z [INFO] 127.0.0.1:40191 :: entering follower state: follower="Node at 127.0.0.1:40191 [Follower]" leader-address= leader-id= 2026-03-31T06:18:04.254Z [WARN] 127.0.0.1:40191 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= === RUN TestRaft_runFollower_State_Transition/Not_in_Config 2026-03-31T06:18:04.315Z [INFO] 127.0.0.1:34483 :: starting node: addr=127.0.0.1:34483 2026-03-31T06:18:04.315Z [INFO] 127.0.0.1:34483 :: initial configuration: index=0 servers=[] 2026-03-31T06:18:04.315Z [INFO] 127.0.0.1:34483 :: entering follower state: follower="Node at 127.0.0.1:34483 [Follower]" leader-address= leader-id= 2026-03-31T06:18:04.389Z [WARN] 127.0.0.1:34483 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election --- PASS: TestRaft_runFollower_State_Transition (0.45s) --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s) === RUN TestRaft_runFollower_ReloadTimeoutConfigs 2026-03-31T06:18:04.467Z [INFO] 127.0.0.1:36609 :: starting node: addr=127.0.0.1:36609 2026-03-31T06:18:04.467Z [INFO] 127.0.0.1:36609 :: initial configuration: index=0 servers=[] 2026-03-31T06:18:04.467Z [INFO] 127.0.0.1:36609 :: entering follower state: follower="Node at 127.0.0.1:36609 [Follower]" leader-address= leader-id= 2026-03-31T06:18:04.467Z [WARN] 127.0.0.1:36609 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= --- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s) === RUN TestSaturationMetric === RUN TestSaturationMetric/without_smoothing --- PASS: TestSaturationMetric (0.00s) --- PASS: TestSaturationMetric/without_smoothing (0.00s) === RUN TestSaturationMetric_IncorrectUsage === RUN TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_first --- PASS: TestSaturationMetric_IncorrectUsage (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_EmptyAddr --- PASS: TestTCPTransport_EmptyAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) === RUN TestOverrideNotifyBool --- PASS: TestOverrideNotifyBool (0.00s) PASS ok github.com/hashicorp/raft 112.915s ? github.com/hashicorp/raft/bench [no test files] make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' create-stamp debian/debhelper-build-stamp dh_testroot -O--buildsystem=golang dh_prep -O--buildsystem=golang dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang dh_installdocs -O--buildsystem=golang dh_installchangelogs -O--buildsystem=golang dh_installsystemduser -O--buildsystem=golang dh_perl -O--buildsystem=golang dh_link -O--buildsystem=golang dh_strip_nondeterminism -O--buildsystem=golang dh_compress -O--buildsystem=golang dh_fixperms -O--buildsystem=golang dh_missing -O--buildsystem=golang dh_installdeb -O--buildsystem=golang dh_golang -O--buildsystem=golang dh_gencontrol -O--buildsystem=golang dh_md5sums -O--buildsystem=golang dh_builddeb -O--buildsystem=golang dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'. dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_arm64.buildinfo dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_arm64.changes dpkg-genchanges: info: binary-only upload (no source code included) dpkg-source --after-build . dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/674357 and its subdirectories I: Current time: Mon Mar 30 18:18:10 -12 2026 I: pbuilder-time-stamp: 1774937890 Tue Feb 25 23:55:13 UTC 2025 I: 1st build successful. Starting 2nd build on remote node codethink04-arm64.debian.net. Tue Feb 25 23:55:13 UTC 2025 I: Preparing to do remote build '2' on codethink04-arm64.debian.net. Tue Feb 25 23:59:31 UTC 2025 I: Deleting $TMPDIR on codethink04-arm64.debian.net.