Wed Mar 12 07:50:16 UTC 2025 I: starting to build golang-github-hashicorp-raft/unstable/amd64 on jenkins on '2025-03-12 07:49' Wed Mar 12 07:50:16 UTC 2025 I: The jenkins build log is/was available at https://jenkins.debian.net/userContent/reproducible/debian/build_service/amd64_8/50541/console.log Wed Mar 12 07:50:17 UTC 2025 I: Downloading source for unstable/golang-github-hashicorp-raft=1.5.0-1 --2025-03-12 07:50:17-- http://deb.debian.org/debian/pool/main/g/golang-github-hashicorp-raft/golang-github-hashicorp-raft_1.5.0-1.dsc Connecting to 46.16.76.132:3128... connected. Proxy request sent, awaiting response... 200 OK Length: 2480 (2.4K) [text/prs.lines.tag] Saving to: ‘golang-github-hashicorp-raft_1.5.0-1.dsc’ 0K .. 100% 346M=0s 2025-03-12 07:50:17 (346 MB/s) - ‘golang-github-hashicorp-raft_1.5.0-1.dsc’ saved [2480/2480] Wed Mar 12 07:50:17 UTC 2025 I: golang-github-hashicorp-raft_1.5.0-1.dsc -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Format: 3.0 (quilt) Source: golang-github-hashicorp-raft Binary: golang-github-hashicorp-raft-dev Architecture: all Version: 1.5.0-1 Maintainer: Debian Go Packaging Team Uploaders: Tim Potter , Dmitry Smirnov , Homepage: https://github.com/hashicorp/raft Standards-Version: 4.6.2 Vcs-Browser: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft Vcs-Git: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git Testsuite: autopkgtest-pkg-go Build-Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev Package-List: golang-github-hashicorp-raft-dev deb golang optional arch=all Checksums-Sha1: cea1c93d584c6f8dfbf53aef3438ab18ec7e7222 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz 015fac87182a135b38ce31d1eb99f68ca251ae23 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz Checksums-Sha256: f4dc9aabc7784a1b9b5da36b85a3797ac37911f2a6dc54a6fd6cb60a6fd643e9 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz 0d95ee03abc0b447cfb92e400d914ecee5113fd181a4ea1d51fa5b6c37288658 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz Files: 7ebc703c8f7e2d7fc02568216f2e3b0a 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz 1f0ee65ab76bca0e05a75b8f9ad32d8d 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz Go-Import-Path: github.com/hashicorp/raft -----BEGIN PGP SIGNATURE----- iQJKBAEBCgA0FiEEMD9oek78sa58GjWjtwAXP7uAWikFAmTCiiIWHGRzd2FyYnJp Y2tAZGViaWFuLm9yZwAKCRC3ABc/u4BaKYS6D/wN81mTx6pt43be/9H7Ovy6z+5p nv0+kgrOLtH5y4KUjNLZJuPljAp+uoELmotxp1wqkoLSYs1AwCZ11X4NVs8YoLXj sLd+Ub+QcWb0Gejqjw7Nk9VZqqXjwfFJW54n8NC2Zy8D6OelCl47sDPmiu03v3uz oRu4dp2DW/uKkC0vb4RczT81GBmJ19Tu2ZdDvtk7kwnvF48cqDhXMhUTQAEjoyVU Q8kfY13blza+uXZk6xhWyYsc+wpTh6RHZIWsbpgejKmEYnZCoCdaR0UqOoau33Bs T1OXVNM0P6vrmeXyoiEy66Q2XyHeei+HYRUgED16Ujg0qxPxSEVE5timSwcXHa/u xb5ZcKQwqEoJISjWTFDG8gww8MBPOKNGW5BKRdQSdtDhNEsnmn58jl1lsRs5L6Xg +DrufjU6KKqqYyOBS+H+s68KXleg2t/qP+K8GZjJl/eX3+Tw5xOdt3ZWDMWxB2qF gS+ASmxYV0Uu/pd0MLWXjAFcllnl93MzPVkF14A8eYrghujLjN3+oUOD0RQzxpAb LHc5b0iPnN4/YQtJX3fZi2y6c62BCZ5Slb5sB0bsgNPbk1T3qQrtMPWJXUfr32ys VOow6eNA+VDxNxHZ3aSUiVC0lcUm6dJDYqcfk/PgDdBR/E7iigkTi3O/2rgW0Y7Z 14FQMVvbK31veGXJjA== =okTI -----END PGP SIGNATURE----- Wed Mar 12 07:50:17 UTC 2025 I: Checking whether the package is not for us Wed Mar 12 07:50:17 UTC 2025 I: Starting 1st build on remote node ionos15-amd64.debian.net. Wed Mar 12 07:50:17 UTC 2025 I: Preparing to do remote build '1' on ionos15-amd64.debian.net. Wed Mar 12 07:53:32 UTC 2025 I: Deleting $TMPDIR on ionos15-amd64.debian.net. W: cgroups are not available on the host, not using them. I: pbuilder: network access will be disabled during build I: Current time: Tue Apr 14 02:13:15 -12 2026 I: pbuilder-time-stamp: 1776175995 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/unstable-reproducible-base.tgz] I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: Copying source file I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc] I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz] I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz] I: Extracting source dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: unsupported subcommand dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying 0001-Skip-unreliable-tests.patch I: Not using root during the build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/946523/tmp/hooks/D02_print_environment starting I: set BUILDDIR='/build/reproducible-path' BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' BUILDUSERNAME='pbuilder1' BUILD_ARCH='amd64' DEBIAN_FRONTEND='noninteractive' DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=42 ' DISTRIBUTION='unstable' HOME='/root' HOST_ARCH='amd64' IFS=' ' LANG='C' LANGUAGE='en_US:en' LC_ALL='C' MAIL='/var/mail/root' OPTIND='1' PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' PBCURRENTCOMMANDLINEOPERATION='build' PBUILDER_OPERATION='build' PBUILDER_PKGDATADIR='/usr/share/pbuilder' PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' PBUILDER_SYSCONFDIR='/etc' PPID='946523' PS1='# ' PS2='> ' PS4='+ ' PWD='/' SHELL='/bin/bash' SHLVL='2' SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.wUggbdD7/pbuilderrc_L4zc --distribution unstable --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/unstable-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.wUggbdD7/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc' SUDO_GID='111' SUDO_UID='106' SUDO_USER='jenkins' TERM='unknown' TZ='/usr/share/zoneinfo/Etc/GMT+12' USER='root' _='/usr/sbin/chroot' http_proxy='http://213.165.73.152:3128' I: uname -a Linux ionos15-amd64 6.12.12+bpo-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.12-1~bpo12+1 (2025-02-23) x86_64 GNU/Linux I: ls -l /bin lrwxrwxrwx 1 root root 7 Mar 4 2025 /bin -> usr/bin I: user script /srv/workspace/pbuilder/946523/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: amd64 Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 19783 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however: Package debhelper-compat is not installed. pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however: Package dh-sequence-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however: Package golang-github-stretchr-testify-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.24-go{a} golang-1.24-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-prometheus-sigv4-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libunistring5{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 0 packages upgraded, 107 newly installed, 0 to remove and 0 not upgraded. Need to get 132 MB of archives. After unpacking 714 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian unstable/main amd64 sensible-utils all 0.0.24 [24.8 kB] Get: 2 http://deb.debian.org/debian unstable/main amd64 libmagic-mgc amd64 1:5.45-3+b1 [314 kB] Get: 3 http://deb.debian.org/debian unstable/main amd64 libmagic1t64 amd64 1:5.45-3+b1 [108 kB] Get: 4 http://deb.debian.org/debian unstable/main amd64 file amd64 1:5.45-3+b1 [43.3 kB] Get: 5 http://deb.debian.org/debian unstable/main amd64 gettext-base amd64 0.23.1-1 [243 kB] Get: 6 http://deb.debian.org/debian unstable/main amd64 libuchardet0 amd64 0.0.8-1+b2 [68.9 kB] Get: 7 http://deb.debian.org/debian unstable/main amd64 groff-base amd64 1.23.0-7 [1185 kB] Get: 8 http://deb.debian.org/debian unstable/main amd64 bsdextrautils amd64 2.40.4-5 [92.4 kB] Get: 9 http://deb.debian.org/debian unstable/main amd64 libpipeline1 amd64 1.5.8-1 [42.0 kB] Get: 10 http://deb.debian.org/debian unstable/main amd64 man-db amd64 2.13.0-1 [1420 kB] Get: 11 http://deb.debian.org/debian unstable/main amd64 m4 amd64 1.4.19-7 [294 kB] Get: 12 http://deb.debian.org/debian unstable/main amd64 autoconf all 2.72-3 [493 kB] Get: 13 http://deb.debian.org/debian unstable/main amd64 autotools-dev all 20220109.1 [51.6 kB] Get: 14 http://deb.debian.org/debian unstable/main amd64 automake all 1:1.17-3 [862 kB] Get: 15 http://deb.debian.org/debian unstable/main amd64 autopoint all 0.23.1-1 [770 kB] Get: 16 http://deb.debian.org/debian unstable/main amd64 libdebhelper-perl all 13.24.1 [90.9 kB] Get: 17 http://deb.debian.org/debian unstable/main amd64 libtool all 2.5.4-4 [539 kB] Get: 18 http://deb.debian.org/debian unstable/main amd64 dh-autoreconf all 20 [17.1 kB] Get: 19 http://deb.debian.org/debian unstable/main amd64 libarchive-zip-perl all 1.68-1 [104 kB] Get: 20 http://deb.debian.org/debian unstable/main amd64 libfile-stripnondeterminism-perl all 1.14.1-2 [19.7 kB] Get: 21 http://deb.debian.org/debian unstable/main amd64 dh-strip-nondeterminism all 1.14.1-2 [8620 B] Get: 22 http://deb.debian.org/debian unstable/main amd64 libelf1t64 amd64 0.192-4 [189 kB] Get: 23 http://deb.debian.org/debian unstable/main amd64 dwz amd64 0.15-1+b1 [110 kB] Get: 24 http://deb.debian.org/debian unstable/main amd64 libunistring5 amd64 1.3-1 [476 kB] Get: 25 http://deb.debian.org/debian unstable/main amd64 libicu72 amd64 72.1-6 [9421 kB] Get: 26 http://deb.debian.org/debian unstable/main amd64 libxml2 amd64 2.12.7+dfsg+really2.9.14-0.2+b2 [699 kB] Get: 27 http://deb.debian.org/debian unstable/main amd64 gettext amd64 0.23.1-1 [1680 kB] Get: 28 http://deb.debian.org/debian unstable/main amd64 intltool-debian all 0.35.0+20060710.6 [22.9 kB] Get: 29 http://deb.debian.org/debian unstable/main amd64 po-debconf all 1.0.21+nmu1 [248 kB] Get: 30 http://deb.debian.org/debian unstable/main amd64 debhelper all 13.24.1 [920 kB] Get: 31 http://deb.debian.org/debian unstable/main amd64 dh-golang all 1.63 [24.4 kB] Get: 32 http://deb.debian.org/debian unstable/main amd64 golang-1.24-src all 1.24.1-1 [21.2 MB] Get: 33 http://deb.debian.org/debian unstable/main amd64 golang-1.24-go amd64 1.24.1-1 [28.6 MB] Get: 34 http://deb.debian.org/debian unstable/main amd64 golang-src all 2:1.24~2 [5136 B] Get: 35 http://deb.debian.org/debian unstable/main amd64 golang-go amd64 2:1.24~2 [44.3 kB] Get: 36 http://deb.debian.org/debian unstable/main amd64 golang-any amd64 2:1.24~2 [5216 B] Get: 37 http://deb.debian.org/debian unstable/main amd64 golang-github-alecthomas-units-dev all 0.0~git20240626.61d1e34-1 [9032 B] Get: 38 http://deb.debian.org/debian unstable/main amd64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 39 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 40 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-sys-dev all 0.22.0-1 [431 kB] Get: 41 http://deb.debian.org/debian unstable/main amd64 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B] Get: 42 http://deb.debian.org/debian unstable/main amd64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB] Get: 43 http://deb.debian.org/debian unstable/main amd64 golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB] Get: 44 http://deb.debian.org/debian unstable/main amd64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB] Get: 45 http://deb.debian.org/debian unstable/main amd64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 46 http://deb.debian.org/debian unstable/main amd64 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB] Get: 47 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB] Get: 48 http://deb.debian.org/debian unstable/main amd64 golang-github-stretchr-testify-dev all 1.10.0-1 [78.4 kB] Get: 49 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 50 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB] Get: 51 http://deb.debian.org/debian unstable/main amd64 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB] Get: 52 http://deb.debian.org/debian unstable/main amd64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B] Get: 53 http://deb.debian.org/debian unstable/main amd64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB] Get: 54 http://deb.debian.org/debian unstable/main amd64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 55 http://deb.debian.org/debian unstable/main amd64 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB] Get: 56 http://deb.debian.org/debian unstable/main amd64 golang-google-protobuf-dev all 1.36.5-1 [796 kB] Get: 57 http://deb.debian.org/debian unstable/main amd64 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB] Get: 58 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B] Get: 59 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 60 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB] Get: 61 http://deb.debian.org/debian unstable/main amd64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB] Get: 62 http://deb.debian.org/debian unstable/main amd64 golang-github-beorn7-perks-dev all 1.0.1-3 [12.2 kB] Get: 63 http://deb.debian.org/debian unstable/main amd64 golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB] Get: 64 http://deb.debian.org/debian unstable/main amd64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 65 http://deb.debian.org/debian unstable/main amd64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB] Get: 66 http://deb.debian.org/debian unstable/main amd64 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB] Get: 67 http://deb.debian.org/debian unstable/main amd64 golang-github-klauspost-compress-dev all 1.18.0+ds1-1 [33.3 MB] Get: 68 http://deb.debian.org/debian unstable/main amd64 golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB] Get: 69 http://deb.debian.org/debian unstable/main amd64 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB] Get: 70 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-yaml.v2-dev all 2.4.0-5 [60.5 kB] Get: 71 http://deb.debian.org/debian unstable/main amd64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB] Get: 72 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-term-dev all 0.22.0-1 [14.8 kB] Get: 73 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-text-dev all 0.22.0-1 [3995 kB] Get: 74 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-crypto-dev all 1:0.25.0-1 [1682 kB] Get: 75 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-net-dev all 1:0.27.0-1 [898 kB] Get: 76 http://deb.debian.org/debian unstable/main amd64 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB] Get: 77 http://deb.debian.org/debian unstable/main amd64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 78 http://deb.debian.org/debian unstable/main amd64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB] Get: 79 http://deb.debian.org/debian unstable/main amd64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB] Get: 80 http://deb.debian.org/debian unstable/main amd64 golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B] Get: 81 http://deb.debian.org/debian unstable/main amd64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 82 http://deb.debian.org/debian unstable/main amd64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-4 [14.6 kB] Get: 83 http://deb.debian.org/debian unstable/main amd64 golang-github-prometheus-sigv4-dev all 0.1.2-2 [8484 B] Get: 84 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-oauth2-dev all 0.27.0-1 [37.2 kB] Get: 85 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB] Get: 86 http://deb.debian.org/debian unstable/main amd64 golang-github-prometheus-common-dev all 0.62.0-3 [118 kB] Get: 87 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB] Get: 88 http://deb.debian.org/debian unstable/main amd64 golang-github-prometheus-procfs-dev all 0.15.1-1 [184 kB] Get: 89 http://deb.debian.org/debian unstable/main amd64 golang-github-prometheus-client-golang-dev all 1.21.1-1 [194 kB] Get: 90 http://deb.debian.org/debian unstable/main amd64 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB] Get: 91 http://deb.debian.org/debian unstable/main amd64 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB] Get: 92 http://deb.debian.org/debian unstable/main amd64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B] Get: 93 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 94 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 95 http://deb.debian.org/debian unstable/main amd64 libsasl2-modules-db amd64 2.1.28+dfsg1-9 [19.8 kB] Get: 96 http://deb.debian.org/debian unstable/main amd64 libsasl2-2 amd64 2.1.28+dfsg1-9 [57.5 kB] Get: 97 http://deb.debian.org/debian unstable/main amd64 libssl-dev amd64 3.4.1-1 [2787 kB] Get: 98 http://deb.debian.org/debian unstable/main amd64 libsasl2-dev amd64 2.1.28+dfsg1-9 [234 kB] Get: 99 http://deb.debian.org/debian unstable/main amd64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB] Get: 100 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-mod-dev all 0.19.0-1 [104 kB] Get: 101 http://deb.debian.org/debian unstable/main amd64 golang-github-yuin-goldmark-dev all 1.7.4-1 [113 kB] Get: 102 http://deb.debian.org/debian unstable/main amd64 golang-golang-x-tools-dev all 1:0.25.0+ds-1 [1608 kB] Get: 103 http://deb.debian.org/debian unstable/main amd64 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB] Get: 104 http://deb.debian.org/debian unstable/main amd64 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB] Get: 105 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-2 [31.2 kB] Get: 106 http://deb.debian.org/debian unstable/main amd64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 107 http://deb.debian.org/debian unstable/main amd64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 132 MB in 2s (71.1 MB/s) Preconfiguring packages ... Selecting previously unselected package sensible-utils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19783 files and directories currently installed.) Preparing to unpack .../000-sensible-utils_0.0.24_all.deb ... Unpacking sensible-utils (0.0.24) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3+b1_amd64.deb ... Unpacking libmagic-mgc (1:5.45-3+b1) ... Selecting previously unselected package libmagic1t64:amd64. Preparing to unpack .../002-libmagic1t64_1%3a5.45-3+b1_amd64.deb ... Unpacking libmagic1t64:amd64 (1:5.45-3+b1) ... Selecting previously unselected package file. Preparing to unpack .../003-file_1%3a5.45-3+b1_amd64.deb ... Unpacking file (1:5.45-3+b1) ... Selecting previously unselected package gettext-base. Preparing to unpack .../004-gettext-base_0.23.1-1_amd64.deb ... Unpacking gettext-base (0.23.1-1) ... Selecting previously unselected package libuchardet0:amd64. Preparing to unpack .../005-libuchardet0_0.0.8-1+b2_amd64.deb ... Unpacking libuchardet0:amd64 (0.0.8-1+b2) ... Selecting previously unselected package groff-base. Preparing to unpack .../006-groff-base_1.23.0-7_amd64.deb ... Unpacking groff-base (1.23.0-7) ... Selecting previously unselected package bsdextrautils. Preparing to unpack .../007-bsdextrautils_2.40.4-5_amd64.deb ... Unpacking bsdextrautils (2.40.4-5) ... Selecting previously unselected package libpipeline1:amd64. Preparing to unpack .../008-libpipeline1_1.5.8-1_amd64.deb ... Unpacking libpipeline1:amd64 (1.5.8-1) ... Selecting previously unselected package man-db. Preparing to unpack .../009-man-db_2.13.0-1_amd64.deb ... Unpacking man-db (2.13.0-1) ... Selecting previously unselected package m4. Preparing to unpack .../010-m4_1.4.19-7_amd64.deb ... Unpacking m4 (1.4.19-7) ... Selecting previously unselected package autoconf. Preparing to unpack .../011-autoconf_2.72-3_all.deb ... Unpacking autoconf (2.72-3) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ... Unpacking autotools-dev (20220109.1) ... Selecting previously unselected package automake. Preparing to unpack .../013-automake_1%3a1.17-3_all.deb ... Unpacking automake (1:1.17-3) ... Selecting previously unselected package autopoint. Preparing to unpack .../014-autopoint_0.23.1-1_all.deb ... Unpacking autopoint (0.23.1-1) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../015-libdebhelper-perl_13.24.1_all.deb ... Unpacking libdebhelper-perl (13.24.1) ... Selecting previously unselected package libtool. Preparing to unpack .../016-libtool_2.5.4-4_all.deb ... Unpacking libtool (2.5.4-4) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../017-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.1-2_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.14.1-2) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../020-dh-strip-nondeterminism_1.14.1-2_all.deb ... Unpacking dh-strip-nondeterminism (1.14.1-2) ... Selecting previously unselected package libelf1t64:amd64. Preparing to unpack .../021-libelf1t64_0.192-4_amd64.deb ... Unpacking libelf1t64:amd64 (0.192-4) ... Selecting previously unselected package dwz. Preparing to unpack .../022-dwz_0.15-1+b1_amd64.deb ... Unpacking dwz (0.15-1+b1) ... Selecting previously unselected package libunistring5:amd64. Preparing to unpack .../023-libunistring5_1.3-1_amd64.deb ... Unpacking libunistring5:amd64 (1.3-1) ... Selecting previously unselected package libicu72:amd64. Preparing to unpack .../024-libicu72_72.1-6_amd64.deb ... Unpacking libicu72:amd64 (72.1-6) ... Selecting previously unselected package libxml2:amd64. Preparing to unpack .../025-libxml2_2.12.7+dfsg+really2.9.14-0.2+b2_amd64.deb ... Unpacking libxml2:amd64 (2.12.7+dfsg+really2.9.14-0.2+b2) ... Selecting previously unselected package gettext. Preparing to unpack .../026-gettext_0.23.1-1_amd64.deb ... Unpacking gettext (0.23.1-1) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../027-intltool-debian_0.35.0+20060710.6_all.deb ... Unpacking intltool-debian (0.35.0+20060710.6) ... Selecting previously unselected package po-debconf. Preparing to unpack .../028-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../029-debhelper_13.24.1_all.deb ... Unpacking debhelper (13.24.1) ... Selecting previously unselected package dh-golang. Preparing to unpack .../030-dh-golang_1.63_all.deb ... Unpacking dh-golang (1.63) ... Selecting previously unselected package golang-1.24-src. Preparing to unpack .../031-golang-1.24-src_1.24.1-1_all.deb ... Unpacking golang-1.24-src (1.24.1-1) ... Selecting previously unselected package golang-1.24-go. Preparing to unpack .../032-golang-1.24-go_1.24.1-1_amd64.deb ... Unpacking golang-1.24-go (1.24.1-1) ... Selecting previously unselected package golang-src. Preparing to unpack .../033-golang-src_2%3a1.24~2_all.deb ... Unpacking golang-src (2:1.24~2) ... Selecting previously unselected package golang-go:amd64. Preparing to unpack .../034-golang-go_2%3a1.24~2_amd64.deb ... Unpacking golang-go:amd64 (2:1.24~2) ... Selecting previously unselected package golang-any:amd64. Preparing to unpack .../035-golang-any_2%3a1.24~2_amd64.deb ... Unpacking golang-any:amd64 (2:1.24~2) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../036-golang-github-alecthomas-units-dev_0.0~git20240626.61d1e34-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../037-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../038-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../039-golang-golang-x-sys-dev_0.22.0-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.22.0-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../040-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../041-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../042-golang-github-fatih-color-dev_1.17.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.17.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../043-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../044-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../045-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../046-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../047-golang-github-stretchr-testify-dev_1.10.0-1_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.10.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../048-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../049-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../050-golang-github-pkg-errors-dev_0.9.1-3_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-3) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../051-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../052-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../053-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-google-go-cmp-dev. Preparing to unpack .../054-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ... Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ... Selecting previously unselected package golang-google-protobuf-dev. Preparing to unpack .../055-golang-google-protobuf-dev_1.36.5-1_all.deb ... Unpacking golang-google-protobuf-dev (1.36.5-1) ... Selecting previously unselected package golang-github-golang-protobuf-1-5-dev. Preparing to unpack .../056-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ... Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../057-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../058-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../059-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../060-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../061-golang-github-beorn7-perks-dev_1.0.1-3_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-3) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../062-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../063-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../064-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../065-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ... Selecting previously unselected package golang-github-klauspost-compress-dev. Preparing to unpack .../066-golang-github-klauspost-compress-dev_1.18.0+ds1-1_all.deb ... Unpacking golang-github-klauspost-compress-dev (1.18.0+ds1-1) ... Selecting previously unselected package golang-github-kylelemons-godebug-dev. Preparing to unpack .../067-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ... Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../068-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../069-golang-gopkg-yaml.v2-dev_2.4.0-5_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.0-5) ... Selecting previously unselected package golang-github-jmespath-go-jmespath-dev. Preparing to unpack .../070-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ... Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Selecting previously unselected package golang-golang-x-term-dev. Preparing to unpack .../071-golang-golang-x-term-dev_0.22.0-1_all.deb ... Unpacking golang-golang-x-term-dev (0.22.0-1) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../072-golang-golang-x-text-dev_0.22.0-1_all.deb ... Unpacking golang-golang-x-text-dev (0.22.0-1) ... Selecting previously unselected package golang-golang-x-crypto-dev. Preparing to unpack .../073-golang-golang-x-crypto-dev_1%3a0.25.0-1_all.deb ... Unpacking golang-golang-x-crypto-dev (1:0.25.0-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../074-golang-golang-x-net-dev_1%3a0.27.0-1_all.deb ... Unpacking golang-golang-x-net-dev (1:0.27.0-1) ... Selecting previously unselected package golang-github-aws-aws-sdk-go-dev. Preparing to unpack .../075-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ... Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../076-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-kit-log-dev. Preparing to unpack .../077-golang-github-go-kit-log-dev_0.2.1-1_all.deb ... Unpacking golang-github-go-kit-log-dev (0.2.1-1) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../078-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Selecting previously unselected package golang-github-munnerz-goautoneg-dev. Preparing to unpack .../079-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ... Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../080-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../081-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-4_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ... Selecting previously unselected package golang-github-prometheus-sigv4-dev. Preparing to unpack .../082-golang-github-prometheus-sigv4-dev_0.1.2-2_all.deb ... Unpacking golang-github-prometheus-sigv4-dev (0.1.2-2) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../083-golang-golang-x-oauth2-dev_0.27.0-1_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.27.0-1) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../084-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../085-golang-github-prometheus-common-dev_0.62.0-3_all.deb ... Unpacking golang-github-prometheus-common-dev (0.62.0-3) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../086-golang-golang-x-sync-dev_0.7.0-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.7.0-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../087-golang-github-prometheus-procfs-dev_0.15.1-1_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.15.1-1) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../088-golang-github-prometheus-client-golang-dev_1.21.1-1_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.21.1-1) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../089-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../090-golang-github-creack-pty-dev_1.1.21-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.21-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../091-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../092-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../093-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:amd64. Preparing to unpack .../094-libsasl2-modules-db_2.1.28+dfsg1-9_amd64.deb ... Unpacking libsasl2-modules-db:amd64 (2.1.28+dfsg1-9) ... Selecting previously unselected package libsasl2-2:amd64. Preparing to unpack .../095-libsasl2-2_2.1.28+dfsg1-9_amd64.deb ... Unpacking libsasl2-2:amd64 (2.1.28+dfsg1-9) ... Selecting previously unselected package libssl-dev:amd64. Preparing to unpack .../096-libssl-dev_3.4.1-1_amd64.deb ... Unpacking libssl-dev:amd64 (3.4.1-1) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../097-libsasl2-dev_2.1.28+dfsg1-9_amd64.deb ... Unpacking libsasl2-dev (2.1.28+dfsg1-9) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../098-golang-github-kr-text-dev_0.2.0-2_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-2) ... Selecting previously unselected package golang-golang-x-mod-dev. Preparing to unpack .../099-golang-golang-x-mod-dev_0.19.0-1_all.deb ... Unpacking golang-golang-x-mod-dev (0.19.0-1) ... Selecting previously unselected package golang-github-yuin-goldmark-dev. Preparing to unpack .../100-golang-github-yuin-goldmark-dev_1.7.4-1_all.deb ... Unpacking golang-github-yuin-goldmark-dev (1.7.4-1) ... Selecting previously unselected package golang-golang-x-tools-dev. Preparing to unpack .../101-golang-golang-x-tools-dev_1%3a0.25.0+ds-1_all.deb ... Unpacking golang-golang-x-tools-dev (1:0.25.0+ds-1) ... Selecting previously unselected package golang-github-rogpeppe-go-internal-dev. Preparing to unpack .../102-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ... Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../103-golang-github-kr-pretty-dev_0.3.1-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.3.1-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../104-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-2_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../105-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../106-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up golang-1.24-src (1.24.1-1) ... Setting up libpipeline1:amd64 (1.5.8-1) ... Setting up golang-github-google-go-cmp-dev (0.6.0-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.0-5) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Setting up libicu72:amd64 (72.1-6) ... Setting up bsdextrautils (2.40.4-5) ... Setting up golang-github-creack-pty-dev (1.1.21-1) ... Setting up golang-github-stretchr-objx-dev (0.5.2-1) ... Setting up libmagic-mgc (1:5.45-3+b1) ... Setting up golang-github-pkg-errors-dev (0.9.1-3) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ... Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ... Setting up libdebhelper-perl (13.24.1) ... Setting up golang-golang-x-sys-dev (0.22.0-1) ... Setting up libmagic1t64:amd64 (1:5.45-3+b1) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.23.1-1) ... Setting up m4 (1.4.19-7) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up golang-golang-x-term-dev (0.22.0-1) ... Setting up file (1:5.45-3+b1) ... Setting up golang-github-beorn7-perks-dev (1.0.1-3) ... Setting up libelf1t64:amd64 (0.192-4) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:amd64 (2.1.28+dfsg1-9) ... Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ... Setting up autotools-dev (20220109.1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-google-protobuf-dev (1.36.5-1) ... Setting up golang-github-kr-text-dev (0.2.0-2) ... Setting up golang-github-stretchr-testify-dev (1.10.0-1) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Setting up golang-github-yuin-goldmark-dev (1.7.4-1) ... Setting up libunistring5:amd64 (1.3-1) ... Setting up libssl-dev:amd64 (3.4.1-1) ... Setting up golang-golang-x-sync-dev (0.7.0-1) ... Setting up golang-golang-x-mod-dev (0.19.0-1) ... Setting up autopoint (0.23.1-1) ... Setting up golang-1.24-go (1.24.1-1) ... Setting up golang-github-klauspost-compress-dev (1.18.0+ds1-1) ... Setting up libsasl2-2:amd64 (2.1.28+dfsg1-9) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Setting up autoconf (2.72-3) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Setting up dwz (0.15-1+b1) ... Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ... Setting up sensible-utils (0.0.24) ... Setting up golang-golang-x-text-dev (0.22.0-1) ... Setting up libuchardet0:amd64 (0.0.8-1+b2) ... Setting up golang-github-prometheus-procfs-dev (0.15.1-1) ... Setting up golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ... Setting up golang-golang-x-crypto-dev (1:0.25.0-1) ... Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-golang-x-oauth2-dev (0.27.0-1) ... Setting up golang-src (2:1.24~2) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Setting up libxml2:amd64 (2.12.7+dfsg+really2.9.14-0.2+b2) ... Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Setting up automake (1:1.17-3) ... update-alternatives: using /usr/bin/automake-1.17 to provide /usr/bin/automake (automake) in auto mode Setting up libfile-stripnondeterminism-perl (1.14.1-2) ... Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ... Setting up golang-github-go-kit-log-dev (0.2.1-1) ... Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ... Setting up gettext (0.23.1-1) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Setting up libtool (2.5.4-4) ... Setting up golang-go:amd64 (2:1.24~2) ... Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ... Setting up golang-github-json-iterator-go-dev (1.1.12-2) ... Setting up golang-any:amd64 (2:1.24~2) ... Setting up intltool-debian (0.35.0+20060710.6) ... Setting up dh-autoreconf (20) ... Setting up libsasl2-dev (2.1.28+dfsg1-9) ... Setting up golang-golang-x-net-dev (1:0.27.0-1) ... Setting up dh-strip-nondeterminism (1.14.1-2) ... Setting up groff-base (1.23.0-7) ... Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ... Setting up golang-github-fatih-color-dev (1.17.0-1) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up golang-golang-x-tools-dev (1:0.25.0+ds-1) ... Setting up man-db (2.13.0-1) ... Not building database; man-db/auto-update is not 'true'. Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up golang-github-kr-pretty-dev (0.3.1-1) ... Setting up debhelper (13.24.1) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Setting up dh-golang (1.63) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ... Setting up golang-github-prometheus-sigv4-dev (0.1.2-2) ... Setting up golang-github-prometheus-common-dev (0.62.0-3) ... Setting up golang-github-prometheus-client-golang-dev (1.21.1-1) ... Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ... Processing triggers for libc-bin (2.41-4) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps I: Building the package I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.5.0-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.5.0-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Daniel Swarbrick dpkg-source --before-build . dpkg-buildpackage: info: host architecture amd64 debian/rules clean dh clean --buildsystem=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules binary dh binary --buildsystem=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-x86_64-linux-gnu && go install -trimpath -v -p 42 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/asan internal/unsafeheader internal/byteorder internal/goos unicode/utf8 encoding vendor/golang.org/x/crypto/internal/alias vendor/golang.org/x/crypto/cryptobyte/asn1 cmp internal/godebugs internal/coverage/rtcov crypto/internal/fips140/alias log/internal internal/goarch internal/goexperiment unicode internal/itoa internal/profilerecord internal/nettrace internal/msan unicode/utf16 math/bits crypto/internal/boring/sig container/list internal/cpu sync/atomic internal/runtime/atomic internal/runtime/syscall internal/runtime/math crypto/internal/fips140/subtle internal/runtime/sys internal/abi crypto/internal/fips140deps/byteorder internal/chacha8rand crypto/internal/fips140deps/cpu internal/bytealg internal/runtime/exithook math internal/stringslite internal/race internal/sync internal/runtime/maps runtime iter weak internal/reflectlite crypto/subtle sync maps slices errors sort internal/bisect internal/testlog internal/singleflight unique internal/oserror github.com/hashicorp/golang-lru/simplelru math/rand/v2 vendor/golang.org/x/net/dns/dnsmessage io path strconv runtime/cgo syscall hash internal/godebug strings bytes internal/saferio crypto/internal/randutil hash/crc32 hash/crc64 crypto net/netip reflect crypto/internal/fips140deps/godebug vendor/golang.org/x/text/transform math/rand crypto/internal/impl net/http/internal/ascii bufio github.com/hashicorp/go-immutable-radix html regexp/syntax crypto/internal/fips140 crypto/tls/internal/fips140tls crypto/internal/fips140/sha3 crypto/internal/fips140/sha256 crypto/internal/fips140/sha512 crypto/sha3 crypto/internal/fips140/hmac crypto/internal/fips140/check crypto/internal/fips140hash crypto/internal/fips140/edwards25519/field crypto/internal/fips140/hkdf crypto/internal/fips140/bigmod crypto/internal/fips140/tls12 crypto/internal/fips140/aes crypto/internal/fips140/nistec/fiat internal/syscall/execenv internal/syscall/unix time crypto/internal/fips140/tls13 regexp crypto/internal/fips140/edwards25519 context io/fs internal/poll internal/filepathlite crypto/internal/fips140/nistec os internal/fmtsort encoding/binary encoding/base64 vendor/golang.org/x/crypto/internal/poly1305 golang.org/x/sys/unix encoding/pem crypto/internal/sysrand vendor/golang.org/x/sys/cpu path/filepath io/ioutil os/signal internal/sysinfo fmt net crypto/internal/entropy crypto/internal/fips140/drbg crypto/internal/fips140only crypto/internal/fips140/ed25519 crypto/internal/fips140/ecdh crypto/internal/fips140/ecdsa crypto/internal/fips140/mlkem crypto/internal/fips140/rsa crypto/internal/fips140/aes/gcm crypto/rc4 crypto/md5 crypto/cipher crypto/des crypto/internal/boring vendor/golang.org/x/crypto/chacha20 log net/http/internal net/url mime vendor/golang.org/x/net/http2/hpack go/token encoding/hex mime/quotedprintable encoding/json compress/flate text/template/parse flag crypto/sha512 runtime/trace crypto/aes runtime/debug crypto/ecdh encoding/gob crypto/hmac crypto/sha256 crypto/sha1 vendor/golang.org/x/text/unicode/norm math/big vendor/golang.org/x/crypto/chacha20poly1305 vendor/golang.org/x/text/unicode/bidi testing compress/gzip text/template vendor/golang.org/x/text/secure/bidirule vendor/golang.org/x/net/idna github.com/mattn/go-isatty github.com/mattn/go-colorable github.com/fatih/color github.com/hashicorp/go-hclog crypto/internal/boring/bbig crypto/rand crypto/dsa crypto/elliptic encoding/asn1 crypto/ed25519 crypto/internal/hpke crypto/rsa html/template crypto/x509/pkix vendor/golang.org/x/crypto/cryptobyte crypto/ecdsa vendor/golang.org/x/net/http/httpproxy net/textproto crypto/x509 vendor/golang.org/x/net/http/httpguts mime/multipart crypto/tls net/http/httptrace net/http github.com/armon/go-metrics net/rpc github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench debian/rules override_dh_auto_test make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' dh_auto_test cd obj-x86_64-linux-gnu && go test -vet=off -v -p 42 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir 2026-04-14T14:14:28.529Z [INFO] creating new snapshot: path=/tmp/raft3273654170/raft2470565036/snapshots/3-10-1776176068529.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s) === RUN TestFileSS_CreateSnapshot 2026-04-14T14:14:28.534Z [INFO] creating new snapshot: path=/tmp/raft1437671531/snapshots/3-10-1776176068534.tmp 2026-04-14T14:14:28.536Z [WARN] found temporary snapshot: name=3-10-1776176068534.tmp --- PASS: TestFileSS_CreateSnapshot (0.01s) === RUN TestFileSS_CancelSnapshot 2026-04-14T14:14:28.542Z [INFO] creating new snapshot: path=/tmp/raft1406195829/snapshots/3-10-1776176068542.tmp --- PASS: TestFileSS_CancelSnapshot (0.00s) === RUN TestFileSS_Retention 2026-04-14T14:14:28.545Z [INFO] creating new snapshot: path=/tmp/raft1402793155/snapshots/3-10-1776176068545.tmp 2026-04-14T14:14:28.552Z [INFO] creating new snapshot: path=/tmp/raft1402793155/snapshots/3-11-1776176068552.tmp 2026-04-14T14:14:28.560Z [INFO] creating new snapshot: path=/tmp/raft1402793155/snapshots/3-12-1776176068560.tmp 2026-04-14T14:14:28.568Z [INFO] reaping snapshot: path=/tmp/raft1402793155/snapshots/3-10-1776176068545 2026-04-14T14:14:28.568Z [INFO] creating new snapshot: path=/tmp/raft1402793155/snapshots/3-13-1776176068568.tmp 2026-04-14T14:14:28.577Z [INFO] reaping snapshot: path=/tmp/raft1402793155/snapshots/3-11-1776176068552 2026-04-14T14:14:28.577Z [INFO] creating new snapshot: path=/tmp/raft1402793155/snapshots/3-14-1776176068577.tmp 2026-04-14T14:14:28.585Z [INFO] reaping snapshot: path=/tmp/raft1402793155/snapshots/3-12-1776176068560 --- PASS: TestFileSS_Retention (0.04s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering 2026-04-14T14:14:28.586Z [INFO] creating new snapshot: path=/tmp/raft243076123/snapshots/5-130350-1776176068586.tmp 2026-04-14T14:14:28.592Z [INFO] creating new snapshot: path=/tmp/raft243076123/snapshots/36-204917-1776176068592.tmp --- PASS: TestFileSS_Ordering (0.01s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestRaft_RestartFollower_LongInitialHeartbeat --- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestLogCacheWithBackendStoreError --- PASS: TestLogCacheWithBackendStoreError (0.00s) === RUN TestOldestLog === RUN TestOldestLog/empty_logs === RUN TestOldestLog/simple_case --- PASS: TestOldestLog (0.00s) --- PASS: TestOldestLog/empty_logs (0.00s) --- PASS: TestOldestLog/simple_case (0.00s) === RUN TestEmitsLogStoreMetrics --- PASS: TestEmitsLogStoreMetrics (0.01s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.00s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams --- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.02s) === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 2026-04-14T14:14:28.697Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 2026-04-14T14:14:28.749Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s) === RUN TestNetworkTransport_RequestVote 2026-04-14T14:14:28.802Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:839: countingWriter: 2026-04-14T14:14:28.810Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-14T14:14:28.815Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-14T14:14:28.826Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-14T14:14:28.846Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-14T14:14:28.887Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-14T14:14:28.968Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-14T14:14:29.128Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-14T14:14:29.449Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-14T14:14:30.090Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-14T14:14:31.092Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-14T14:14:32.092Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop 2026-04-14T14:14:32.811Z [INFO] server-fa061a52-ea11-0436-d1ae-7e33d1b59089: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa061a52-ea11-0436-d1ae-7e33d1b59089 Address:fa061a52-ea11-0436-d1ae-7e33d1b59089}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown 2026-04-14T14:14:32.812Z [INFO] server-5aec95f7-4725-5c6a-3cba-7c5767dc9a6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5aec95f7-4725-5c6a-3cba-7c5767dc9a6e Address:5aec95f7-4725-5c6a-3cba-7c5767dc9a6e}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap 2026-04-14T14:14:32.812Z [INFO] server-e08be983-5e84-c34b-ce6f-14beee08f491: initial configuration: index=0 servers=[] 2026-04-14T14:14:32.813Z [INFO] server-8e7129b8-80bc-3121-5931-83cab702ad75: initial configuration: index=0 servers=[] 2026-04-14T14:14:32.813Z [INFO] server-ad31bb0a-9e2f-5713-91d9-a3d4d29e4095: initial configuration: index=0 servers=[] 2026-04-14T14:14:32.813Z [INFO] server-ad31bb0a-9e2f-5713-91d9-a3d4d29e4095: entering follower state: follower="Node at ad31bb0a-9e2f-5713-91d9-a3d4d29e4095 [Follower]" leader-address= leader-id= 2026-04-14T14:14:32.813Z [INFO] server-e08be983-5e84-c34b-ce6f-14beee08f491: entering follower state: follower="Node at e08be983-5e84-c34b-ce6f-14beee08f491 [Follower]" leader-address= leader-id= 2026-04-14T14:14:32.813Z [INFO] server-8e7129b8-80bc-3121-5931-83cab702ad75: entering follower state: follower="Node at 8e7129b8-80bc-3121-5931-83cab702ad75 [Follower]" leader-address= leader-id= 2026-04-14T14:14:32.813Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:14:32.868Z [WARN] server-ad31bb0a-9e2f-5713-91d9-a3d4d29e4095: no known peers, aborting election 2026-04-14T14:14:32.897Z [WARN] server-8e7129b8-80bc-3121-5931-83cab702ad75: no known peers, aborting election 2026-04-14T14:14:32.913Z [WARN] server-e08be983-5e84-c34b-ce6f-14beee08f491: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:32.913Z [INFO] server-e08be983-5e84-c34b-ce6f-14beee08f491: entering candidate state: node="Node at e08be983-5e84-c34b-ce6f-14beee08f491 [Candidate]" term=2 2026-04-14T14:14:32.913Z [INFO] server-e08be983-5e84-c34b-ce6f-14beee08f491: election won: term=2 tally=2 2026-04-14T14:14:32.913Z [INFO] server-e08be983-5e84-c34b-ce6f-14beee08f491: entering leader state: leader="Node at e08be983-5e84-c34b-ce6f-14beee08f491 [Leader]" 2026-04-14T14:14:32.913Z [INFO] server-e08be983-5e84-c34b-ce6f-14beee08f491: added peer, starting replication: peer=server-8e7129b8-80bc-3121-5931-83cab702ad75 2026-04-14T14:14:32.913Z [INFO] server-e08be983-5e84-c34b-ce6f-14beee08f491: added peer, starting replication: peer=server-ad31bb0a-9e2f-5713-91d9-a3d4d29e4095 2026-04-14T14:14:32.913Z [WARN] server-ad31bb0a-9e2f-5713-91d9-a3d4d29e4095: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2026-04-14T14:14:32.913Z [WARN] server-8e7129b8-80bc-3121-5931-83cab702ad75: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2026-04-14T14:14:32.913Z [WARN] server-e08be983-5e84-c34b-ce6f-14beee08f491: appendEntries rejected, sending older logs: peer="{Voter server-ad31bb0a-9e2f-5713-91d9-a3d4d29e4095 ad31bb0a-9e2f-5713-91d9-a3d4d29e4095}" next=1 2026-04-14T14:14:32.913Z [INFO] server-e08be983-5e84-c34b-ce6f-14beee08f491: pipelining replication: peer="{Voter server-ad31bb0a-9e2f-5713-91d9-a3d4d29e4095 ad31bb0a-9e2f-5713-91d9-a3d4d29e4095}" 2026-04-14T14:14:32.913Z [WARN] server-e08be983-5e84-c34b-ce6f-14beee08f491: appendEntries rejected, sending older logs: peer="{Voter server-8e7129b8-80bc-3121-5931-83cab702ad75 8e7129b8-80bc-3121-5931-83cab702ad75}" next=1 2026-04-14T14:14:32.913Z [INFO] server-e08be983-5e84-c34b-ce6f-14beee08f491: pipelining replication: peer="{Voter server-8e7129b8-80bc-3121-5931-83cab702ad75 8e7129b8-80bc-3121-5931-83cab702ad75}" 2026-04-14T14:14:33.018Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:14:32.913279299 +0000 UTC m=+4.385465090 (2 nodes), 99.889983ms from start of poll, 100.322937ms from cluster start. Timeout at 2026-04-14 14:14:33.018283185 +0000 UTC m=+4.490468952, 105.003862ms after stability 2026-04-14T14:14:33.018Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:33.124Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:33.018593345 +0000 UTC m=+4.490779111 (1 nodes), 949ns from start of poll, 205.636958ms from cluster start. Timeout at 2026-04-14 14:14:33.123596684 +0000 UTC m=+4.595782453, 105.003342ms after stability 2026-04-14T14:14:33.135Z [INFO] server-e08be983-5e84-c34b-ce6f-14beee08f491: aborting pipeline replication: peer="{Voter server-ad31bb0a-9e2f-5713-91d9-a3d4d29e4095 ad31bb0a-9e2f-5713-91d9-a3d4d29e4095}" 2026-04-14T14:14:33.135Z [INFO] server-e08be983-5e84-c34b-ce6f-14beee08f491: aborting pipeline replication: peer="{Voter server-8e7129b8-80bc-3121-5931-83cab702ad75 8e7129b8-80bc-3121-5931-83cab702ad75}" 2026-04-14T14:14:33.635Z [ERROR] server-e08be983-5e84-c34b-ce6f-14beee08f491: failed to heartbeat to: peer=ad31bb0a-9e2f-5713-91d9-a3d4d29e4095 backoff time=10ms error="command timed out" --- PASS: TestRaft_LiveBootstrap (0.82s) === RUN TestRaft_LiveBootstrap_From_NonVoter 2026-04-14T14:14:33.636Z [INFO] server-17295990-93fc-79a0-a986-8f108ebb210d: initial configuration: index=0 servers=[] 2026-04-14T14:14:33.637Z [INFO] server-2cf97cb2-07bf-a89d-0f32-ca3140dccc23: initial configuration: index=0 servers=[] 2026-04-14T14:14:33.637Z [INFO] server-2cf97cb2-07bf-a89d-0f32-ca3140dccc23: entering follower state: follower="Node at 2cf97cb2-07bf-a89d-0f32-ca3140dccc23 [Follower]" leader-address= leader-id= 2026-04-14T14:14:33.637Z [INFO] server-17295990-93fc-79a0-a986-8f108ebb210d: entering follower state: follower="Node at 17295990-93fc-79a0-a986-8f108ebb210d [Follower]" leader-address= leader-id= --- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s) === RUN TestRaft_RecoverCluster_NoState 2026-04-14T14:14:33.637Z [INFO] server-5ee84d9a-ea92-0018-ca39-4340f9f9448b: initial configuration: index=0 servers=[] 2026-04-14T14:14:33.637Z [INFO] server-5ee84d9a-ea92-0018-ca39-4340f9f9448b: entering follower state: follower="Node at 5ee84d9a-ea92-0018-ca39-4340f9f9448b [Shutdown]" leader-address= leader-id= --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster === RUN TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs 2026-04-14T14:14:33.638Z [INFO] server-366b82ae-9112-5a7f-8f70-13d47f686d2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-366b82ae-9112-5a7f-8f70-13d47f686d2a Address:366b82ae-9112-5a7f-8f70-13d47f686d2a} {Suffrage:Voter ID:server-e33184a2-9862-f8c8-5c38-57144b7b1407 Address:e33184a2-9862-f8c8-5c38-57144b7b1407} {Suffrage:Voter ID:server-03efaa0b-03bb-4d0f-bb82-c73748e95063 Address:03efaa0b-03bb-4d0f-bb82-c73748e95063}]" 2026-04-14T14:14:33.639Z [INFO] server-366b82ae-9112-5a7f-8f70-13d47f686d2a: entering follower state: follower="Node at 366b82ae-9112-5a7f-8f70-13d47f686d2a [Follower]" leader-address= leader-id= 2026-04-14T14:14:33.639Z [INFO] server-e33184a2-9862-f8c8-5c38-57144b7b1407: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-366b82ae-9112-5a7f-8f70-13d47f686d2a Address:366b82ae-9112-5a7f-8f70-13d47f686d2a} {Suffrage:Voter ID:server-e33184a2-9862-f8c8-5c38-57144b7b1407 Address:e33184a2-9862-f8c8-5c38-57144b7b1407} {Suffrage:Voter ID:server-03efaa0b-03bb-4d0f-bb82-c73748e95063 Address:03efaa0b-03bb-4d0f-bb82-c73748e95063}]" 2026-04-14T14:14:33.639Z [INFO] server-e33184a2-9862-f8c8-5c38-57144b7b1407: entering follower state: follower="Node at e33184a2-9862-f8c8-5c38-57144b7b1407 [Follower]" leader-address= leader-id= 2026-04-14T14:14:33.639Z [INFO] server-03efaa0b-03bb-4d0f-bb82-c73748e95063: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-366b82ae-9112-5a7f-8f70-13d47f686d2a Address:366b82ae-9112-5a7f-8f70-13d47f686d2a} {Suffrage:Voter ID:server-e33184a2-9862-f8c8-5c38-57144b7b1407 Address:e33184a2-9862-f8c8-5c38-57144b7b1407} {Suffrage:Voter ID:server-03efaa0b-03bb-4d0f-bb82-c73748e95063 Address:03efaa0b-03bb-4d0f-bb82-c73748e95063}]" 2026-04-14T14:14:33.639Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:33.639Z [INFO] server-03efaa0b-03bb-4d0f-bb82-c73748e95063: entering follower state: follower="Node at 03efaa0b-03bb-4d0f-bb82-c73748e95063 [Follower]" leader-address= leader-id= 2026-04-14T14:14:33.689Z [WARN] server-e33184a2-9862-f8c8-5c38-57144b7b1407: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:33.689Z [INFO] server-e33184a2-9862-f8c8-5c38-57144b7b1407: entering candidate state: node="Node at e33184a2-9862-f8c8-5c38-57144b7b1407 [Candidate]" term=2 2026-04-14T14:14:33.689Z [INFO] server-e33184a2-9862-f8c8-5c38-57144b7b1407: election won: term=2 tally=2 2026-04-14T14:14:33.689Z [INFO] server-e33184a2-9862-f8c8-5c38-57144b7b1407: entering leader state: leader="Node at e33184a2-9862-f8c8-5c38-57144b7b1407 [Leader]" 2026-04-14T14:14:33.689Z [INFO] server-e33184a2-9862-f8c8-5c38-57144b7b1407: added peer, starting replication: peer=server-366b82ae-9112-5a7f-8f70-13d47f686d2a 2026-04-14T14:14:33.689Z [INFO] server-e33184a2-9862-f8c8-5c38-57144b7b1407: added peer, starting replication: peer=server-03efaa0b-03bb-4d0f-bb82-c73748e95063 2026-04-14T14:14:33.689Z [INFO] server-e33184a2-9862-f8c8-5c38-57144b7b1407: pipelining replication: peer="{Voter server-366b82ae-9112-5a7f-8f70-13d47f686d2a 366b82ae-9112-5a7f-8f70-13d47f686d2a}" 2026-04-14T14:14:33.690Z [INFO] server-e33184a2-9862-f8c8-5c38-57144b7b1407: pipelining replication: peer="{Voter server-03efaa0b-03bb-4d0f-bb82-c73748e95063 03efaa0b-03bb-4d0f-bb82-c73748e95063}" 2026-04-14T14:14:33.795Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:33.68977951 +0000 UTC m=+5.161965274 (1 nodes), 50.542301ms from start of poll, 50.967498ms from cluster start. Timeout at 2026-04-14 14:14:33.794783509 +0000 UTC m=+5.266969278, 105.004004ms after stability 2026-04-14T14:14:33.795Z [INFO] server-e33184a2-9862-f8c8-5c38-57144b7b1407: aborting pipeline replication: peer="{Voter server-03efaa0b-03bb-4d0f-bb82-c73748e95063 03efaa0b-03bb-4d0f-bb82-c73748e95063}" 2026-04-14T14:14:33.795Z [INFO] server-e33184a2-9862-f8c8-5c38-57144b7b1407: aborting pipeline replication: peer="{Voter server-366b82ae-9112-5a7f-8f70-13d47f686d2a 366b82ae-9112-5a7f-8f70-13d47f686d2a}" 2026-04-14T14:14:33.795Z [INFO] creating new snapshot: path=/tmp/raft20232376/snapshots/2-2-1776176073795.tmp 2026-04-14T14:14:33.836Z [INFO] server-366b82ae-9112-5a7f-8f70-13d47f686d2a: starting restore from snapshot: id=2-2-1776176073795 last-index=2 last-term=2 size-in-bytes=1 2026-04-14T14:14:33.837Z [INFO] server-366b82ae-9112-5a7f-8f70-13d47f686d2a: snapshot restore progress: id=2-2-1776176073795 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2026-04-14T14:14:33.837Z [INFO] server-366b82ae-9112-5a7f-8f70-13d47f686d2a: restored from snapshot: id=2-2-1776176073795 last-index=2 last-term=2 size-in-bytes=1 2026-04-14T14:14:33.837Z [INFO] server-366b82ae-9112-5a7f-8f70-13d47f686d2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-366b82ae-9112-5a7f-8f70-13d47f686d2a Address:366b82ae-9112-5a7f-8f70-13d47f686d2a} {Suffrage:Voter ID:server-e33184a2-9862-f8c8-5c38-57144b7b1407 Address:e33184a2-9862-f8c8-5c38-57144b7b1407} {Suffrage:Voter ID:server-03efaa0b-03bb-4d0f-bb82-c73748e95063 Address:03efaa0b-03bb-4d0f-bb82-c73748e95063}]" 2026-04-14T14:14:33.837Z [INFO] creating new snapshot: path=/tmp/raft2273708208/snapshots/2-2-1776176073837.tmp 2026-04-14T14:14:33.837Z [INFO] server-366b82ae-9112-5a7f-8f70-13d47f686d2a: entering follower state: follower="Node at 366b82ae-9112-5a7f-8f70-13d47f686d2a [Follower]" leader-address= leader-id= 2026-04-14T14:14:33.837Z [INFO] server-e33184a2-9862-f8c8-5c38-57144b7b1407: starting restore from snapshot: id=2-2-1776176073837 last-index=2 last-term=2 size-in-bytes=1 2026-04-14T14:14:33.837Z [INFO] server-e33184a2-9862-f8c8-5c38-57144b7b1407: snapshot restore progress: id=2-2-1776176073837 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2026-04-14T14:14:33.837Z [INFO] server-e33184a2-9862-f8c8-5c38-57144b7b1407: restored from snapshot: id=2-2-1776176073837 last-index=2 last-term=2 size-in-bytes=1 2026-04-14T14:14:33.837Z [INFO] server-e33184a2-9862-f8c8-5c38-57144b7b1407: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-366b82ae-9112-5a7f-8f70-13d47f686d2a Address:366b82ae-9112-5a7f-8f70-13d47f686d2a} {Suffrage:Voter ID:server-e33184a2-9862-f8c8-5c38-57144b7b1407 Address:e33184a2-9862-f8c8-5c38-57144b7b1407} {Suffrage:Voter ID:server-03efaa0b-03bb-4d0f-bb82-c73748e95063 Address:03efaa0b-03bb-4d0f-bb82-c73748e95063}]" 2026-04-14T14:14:33.837Z [INFO] server-e33184a2-9862-f8c8-5c38-57144b7b1407: entering follower state: follower="Node at e33184a2-9862-f8c8-5c38-57144b7b1407 [Follower]" leader-address= leader-id= 2026-04-14T14:14:33.838Z [INFO] creating new snapshot: path=/tmp/raft1847057021/snapshots/2-2-1776176073838.tmp 2026-04-14T14:14:33.838Z [INFO] server-03efaa0b-03bb-4d0f-bb82-c73748e95063: starting restore from snapshot: id=2-2-1776176073838 last-index=2 last-term=2 size-in-bytes=1 2026-04-14T14:14:33.838Z [INFO] server-03efaa0b-03bb-4d0f-bb82-c73748e95063: snapshot restore progress: id=2-2-1776176073838 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2026-04-14T14:14:33.838Z [INFO] server-03efaa0b-03bb-4d0f-bb82-c73748e95063: restored from snapshot: id=2-2-1776176073838 last-index=2 last-term=2 size-in-bytes=1 2026-04-14T14:14:33.838Z [INFO] server-03efaa0b-03bb-4d0f-bb82-c73748e95063: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-366b82ae-9112-5a7f-8f70-13d47f686d2a Address:366b82ae-9112-5a7f-8f70-13d47f686d2a} {Suffrage:Voter ID:server-e33184a2-9862-f8c8-5c38-57144b7b1407 Address:e33184a2-9862-f8c8-5c38-57144b7b1407} {Suffrage:Voter ID:server-03efaa0b-03bb-4d0f-bb82-c73748e95063 Address:03efaa0b-03bb-4d0f-bb82-c73748e95063}]" 2026-04-14T14:14:33.838Z [INFO] server-03efaa0b-03bb-4d0f-bb82-c73748e95063: entering follower state: follower="Node at 03efaa0b-03bb-4d0f-bb82-c73748e95063 [Follower]" leader-address= leader-id= 2026-04-14T14:14:33.897Z [WARN] server-03efaa0b-03bb-4d0f-bb82-c73748e95063: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:33.897Z [INFO] server-03efaa0b-03bb-4d0f-bb82-c73748e95063: entering candidate state: node="Node at 03efaa0b-03bb-4d0f-bb82-c73748e95063 [Candidate]" term=3 2026-04-14T14:14:33.897Z [INFO] server-03efaa0b-03bb-4d0f-bb82-c73748e95063: election won: term=3 tally=2 2026-04-14T14:14:33.897Z [INFO] server-03efaa0b-03bb-4d0f-bb82-c73748e95063: entering leader state: leader="Node at 03efaa0b-03bb-4d0f-bb82-c73748e95063 [Leader]" 2026-04-14T14:14:33.897Z [INFO] server-03efaa0b-03bb-4d0f-bb82-c73748e95063: added peer, starting replication: peer=server-366b82ae-9112-5a7f-8f70-13d47f686d2a 2026-04-14T14:14:33.897Z [INFO] server-03efaa0b-03bb-4d0f-bb82-c73748e95063: added peer, starting replication: peer=server-e33184a2-9862-f8c8-5c38-57144b7b1407 2026-04-14T14:14:33.897Z [INFO] server-03efaa0b-03bb-4d0f-bb82-c73748e95063: pipelining replication: peer="{Voter server-e33184a2-9862-f8c8-5c38-57144b7b1407 e33184a2-9862-f8c8-5c38-57144b7b1407}" 2026-04-14T14:14:33.897Z [INFO] server-03efaa0b-03bb-4d0f-bb82-c73748e95063: pipelining replication: peer="{Voter server-366b82ae-9112-5a7f-8f70-13d47f686d2a 366b82ae-9112-5a7f-8f70-13d47f686d2a}" 2026-04-14T14:14:34.154Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:34.260Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:34.15436868 +0000 UTC m=+5.626554449 (1 nodes), 21.703µs from start of poll, 515.556673ms from cluster start. Timeout at 2026-04-14 14:14:34.2593751 +0000 UTC m=+5.731560868, 105.006419ms after stability 2026-04-14T14:14:34.260Z [INFO] server-03efaa0b-03bb-4d0f-bb82-c73748e95063: aborting pipeline replication: peer="{Voter server-e33184a2-9862-f8c8-5c38-57144b7b1407 e33184a2-9862-f8c8-5c38-57144b7b1407}" 2026-04-14T14:14:34.260Z [INFO] server-03efaa0b-03bb-4d0f-bb82-c73748e95063: aborting pipeline replication: peer="{Voter server-366b82ae-9112-5a7f-8f70-13d47f686d2a 366b82ae-9112-5a7f-8f70-13d47f686d2a}" === RUN TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs 2026-04-14T14:14:34.277Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2841277-5598-83b4-252d-a817aea710b7 Address:d2841277-5598-83b4-252d-a817aea710b7} {Suffrage:Voter ID:server-045e5ad8-868a-b639-2f7c-108f64ae7994 Address:045e5ad8-868a-b639-2f7c-108f64ae7994} {Suffrage:Voter ID:server-eca68486-f28a-b2f9-89c5-23de74b0b6e3 Address:eca68486-f28a-b2f9-89c5-23de74b0b6e3}]" 2026-04-14T14:14:34.277Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: entering follower state: follower="Node at d2841277-5598-83b4-252d-a817aea710b7 [Follower]" leader-address= leader-id= 2026-04-14T14:14:34.277Z [INFO] server-045e5ad8-868a-b639-2f7c-108f64ae7994: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2841277-5598-83b4-252d-a817aea710b7 Address:d2841277-5598-83b4-252d-a817aea710b7} {Suffrage:Voter ID:server-045e5ad8-868a-b639-2f7c-108f64ae7994 Address:045e5ad8-868a-b639-2f7c-108f64ae7994} {Suffrage:Voter ID:server-eca68486-f28a-b2f9-89c5-23de74b0b6e3 Address:eca68486-f28a-b2f9-89c5-23de74b0b6e3}]" 2026-04-14T14:14:34.277Z [INFO] server-045e5ad8-868a-b639-2f7c-108f64ae7994: entering follower state: follower="Node at 045e5ad8-868a-b639-2f7c-108f64ae7994 [Follower]" leader-address= leader-id= 2026-04-14T14:14:34.277Z [INFO] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2841277-5598-83b4-252d-a817aea710b7 Address:d2841277-5598-83b4-252d-a817aea710b7} {Suffrage:Voter ID:server-045e5ad8-868a-b639-2f7c-108f64ae7994 Address:045e5ad8-868a-b639-2f7c-108f64ae7994} {Suffrage:Voter ID:server-eca68486-f28a-b2f9-89c5-23de74b0b6e3 Address:eca68486-f28a-b2f9-89c5-23de74b0b6e3}]" 2026-04-14T14:14:34.277Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:34.277Z [INFO] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: entering follower state: follower="Node at eca68486-f28a-b2f9-89c5-23de74b0b6e3 [Follower]" leader-address= leader-id= 2026-04-14T14:14:34.335Z [WARN] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:34.336Z [INFO] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: entering candidate state: node="Node at eca68486-f28a-b2f9-89c5-23de74b0b6e3 [Candidate]" term=2 2026-04-14T14:14:34.336Z [INFO] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: election won: term=2 tally=2 2026-04-14T14:14:34.336Z [INFO] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: entering leader state: leader="Node at eca68486-f28a-b2f9-89c5-23de74b0b6e3 [Leader]" 2026-04-14T14:14:34.336Z [INFO] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: added peer, starting replication: peer=server-d2841277-5598-83b4-252d-a817aea710b7 2026-04-14T14:14:34.336Z [INFO] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: added peer, starting replication: peer=server-045e5ad8-868a-b639-2f7c-108f64ae7994 2026-04-14T14:14:34.336Z [INFO] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: pipelining replication: peer="{Voter server-045e5ad8-868a-b639-2f7c-108f64ae7994 045e5ad8-868a-b639-2f7c-108f64ae7994}" 2026-04-14T14:14:34.336Z [INFO] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: pipelining replication: peer="{Voter server-d2841277-5598-83b4-252d-a817aea710b7 d2841277-5598-83b4-252d-a817aea710b7}" 2026-04-14T14:14:34.442Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:34.336182054 +0000 UTC m=+5.808367824 (1 nodes), 58.214408ms from start of poll, 58.888715ms from cluster start. Timeout at 2026-04-14 14:14:34.441187206 +0000 UTC m=+5.913372970, 105.005146ms after stability 2026-04-14T14:14:34.442Z [INFO] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: aborting pipeline replication: peer="{Voter server-045e5ad8-868a-b639-2f7c-108f64ae7994 045e5ad8-868a-b639-2f7c-108f64ae7994}" 2026-04-14T14:14:34.442Z [INFO] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: aborting pipeline replication: peer="{Voter server-d2841277-5598-83b4-252d-a817aea710b7 d2841277-5598-83b4-252d-a817aea710b7}" 2026-04-14T14:14:34.942Z [ERROR] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: failed to heartbeat to: peer=d2841277-5598-83b4-252d-a817aea710b7 backoff time=10ms error="command timed out" 2026-04-14T14:14:35.223Z [INFO] creating new snapshot: path=/tmp/raft3641732358/snapshots/2-6-1776176075223.tmp 2026-04-14T14:14:35.309Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: starting restore from snapshot: id=2-6-1776176075223 last-index=6 last-term=2 size-in-bytes=25 2026-04-14T14:14:35.309Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: snapshot restore progress: id=2-6-1776176075223 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2026-04-14T14:14:35.309Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: restored from snapshot: id=2-6-1776176075223 last-index=6 last-term=2 size-in-bytes=25 2026-04-14T14:14:35.309Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2841277-5598-83b4-252d-a817aea710b7 Address:d2841277-5598-83b4-252d-a817aea710b7} {Suffrage:Voter ID:server-045e5ad8-868a-b639-2f7c-108f64ae7994 Address:045e5ad8-868a-b639-2f7c-108f64ae7994} {Suffrage:Voter ID:server-eca68486-f28a-b2f9-89c5-23de74b0b6e3 Address:eca68486-f28a-b2f9-89c5-23de74b0b6e3}]" 2026-04-14T14:14:35.310Z [INFO] creating new snapshot: path=/tmp/raft849779909/snapshots/2-6-1776176075310.tmp 2026-04-14T14:14:35.310Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: entering follower state: follower="Node at d2841277-5598-83b4-252d-a817aea710b7 [Follower]" leader-address= leader-id= 2026-04-14T14:14:35.310Z [INFO] server-045e5ad8-868a-b639-2f7c-108f64ae7994: starting restore from snapshot: id=2-6-1776176075310 last-index=6 last-term=2 size-in-bytes=25 2026-04-14T14:14:35.310Z [INFO] server-045e5ad8-868a-b639-2f7c-108f64ae7994: snapshot restore progress: id=2-6-1776176075310 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2026-04-14T14:14:35.310Z [INFO] server-045e5ad8-868a-b639-2f7c-108f64ae7994: restored from snapshot: id=2-6-1776176075310 last-index=6 last-term=2 size-in-bytes=25 2026-04-14T14:14:35.310Z [INFO] server-045e5ad8-868a-b639-2f7c-108f64ae7994: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2841277-5598-83b4-252d-a817aea710b7 Address:d2841277-5598-83b4-252d-a817aea710b7} {Suffrage:Voter ID:server-045e5ad8-868a-b639-2f7c-108f64ae7994 Address:045e5ad8-868a-b639-2f7c-108f64ae7994} {Suffrage:Voter ID:server-eca68486-f28a-b2f9-89c5-23de74b0b6e3 Address:eca68486-f28a-b2f9-89c5-23de74b0b6e3}]" 2026-04-14T14:14:35.310Z [INFO] creating new snapshot: path=/tmp/raft3910022976/snapshots/2-6-1776176075310.tmp 2026-04-14T14:14:35.310Z [INFO] server-045e5ad8-868a-b639-2f7c-108f64ae7994: entering follower state: follower="Node at 045e5ad8-868a-b639-2f7c-108f64ae7994 [Follower]" leader-address= leader-id= 2026-04-14T14:14:35.363Z [WARN] server-d2841277-5598-83b4-252d-a817aea710b7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:35.363Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: entering candidate state: node="Node at d2841277-5598-83b4-252d-a817aea710b7 [Candidate]" term=3 2026-04-14T14:14:35.363Z [ERROR] server-d2841277-5598-83b4-252d-a817aea710b7: failed to make requestVote RPC: target="{Voter server-eca68486-f28a-b2f9-89c5-23de74b0b6e3 eca68486-f28a-b2f9-89c5-23de74b0b6e3}" error="failed to connect to peer: eca68486-f28a-b2f9-89c5-23de74b0b6e3" term=3 2026-04-14T14:14:35.363Z [ERROR] server-d2841277-5598-83b4-252d-a817aea710b7: failed to make requestVote RPC: target="{Voter server-045e5ad8-868a-b639-2f7c-108f64ae7994 045e5ad8-868a-b639-2f7c-108f64ae7994}" error="failed to connect to peer: 045e5ad8-868a-b639-2f7c-108f64ae7994" term=3 2026-04-14T14:14:35.383Z [WARN] server-045e5ad8-868a-b639-2f7c-108f64ae7994: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:35.383Z [INFO] server-045e5ad8-868a-b639-2f7c-108f64ae7994: entering candidate state: node="Node at 045e5ad8-868a-b639-2f7c-108f64ae7994 [Candidate]" term=3 2026-04-14T14:14:35.383Z [ERROR] server-045e5ad8-868a-b639-2f7c-108f64ae7994: failed to make requestVote RPC: target="{Voter server-eca68486-f28a-b2f9-89c5-23de74b0b6e3 eca68486-f28a-b2f9-89c5-23de74b0b6e3}" error="failed to connect to peer: eca68486-f28a-b2f9-89c5-23de74b0b6e3" term=3 2026-04-14T14:14:35.383Z [ERROR] server-045e5ad8-868a-b639-2f7c-108f64ae7994: failed to make requestVote RPC: target="{Voter server-d2841277-5598-83b4-252d-a817aea710b7 d2841277-5598-83b4-252d-a817aea710b7}" error="failed to connect to peer: d2841277-5598-83b4-252d-a817aea710b7" term=3 2026-04-14T14:14:35.419Z [WARN] server-d2841277-5598-83b4-252d-a817aea710b7: Election timeout reached, restarting election 2026-04-14T14:14:35.420Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: entering candidate state: node="Node at d2841277-5598-83b4-252d-a817aea710b7 [Candidate]" term=4 2026-04-14T14:14:35.420Z [ERROR] server-d2841277-5598-83b4-252d-a817aea710b7: failed to make requestVote RPC: target="{Voter server-eca68486-f28a-b2f9-89c5-23de74b0b6e3 eca68486-f28a-b2f9-89c5-23de74b0b6e3}" error="failed to connect to peer: eca68486-f28a-b2f9-89c5-23de74b0b6e3" term=4 2026-04-14T14:14:35.420Z [ERROR] server-d2841277-5598-83b4-252d-a817aea710b7: failed to make requestVote RPC: target="{Voter server-045e5ad8-868a-b639-2f7c-108f64ae7994 045e5ad8-868a-b639-2f7c-108f64ae7994}" error="failed to connect to peer: 045e5ad8-868a-b639-2f7c-108f64ae7994" term=4 2026-04-14T14:14:35.438Z [WARN] server-045e5ad8-868a-b639-2f7c-108f64ae7994: Election timeout reached, restarting election 2026-04-14T14:14:35.438Z [INFO] server-045e5ad8-868a-b639-2f7c-108f64ae7994: entering candidate state: node="Node at 045e5ad8-868a-b639-2f7c-108f64ae7994 [Candidate]" term=4 2026-04-14T14:14:35.438Z [ERROR] server-045e5ad8-868a-b639-2f7c-108f64ae7994: failed to make requestVote RPC: target="{Voter server-eca68486-f28a-b2f9-89c5-23de74b0b6e3 eca68486-f28a-b2f9-89c5-23de74b0b6e3}" error="failed to connect to peer: eca68486-f28a-b2f9-89c5-23de74b0b6e3" term=4 2026-04-14T14:14:35.438Z [ERROR] server-045e5ad8-868a-b639-2f7c-108f64ae7994: failed to make requestVote RPC: target="{Voter server-d2841277-5598-83b4-252d-a817aea710b7 d2841277-5598-83b4-252d-a817aea710b7}" error="failed to connect to peer: d2841277-5598-83b4-252d-a817aea710b7" term=4 2026-04-14T14:14:35.468Z [INFO] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: starting restore from snapshot: id=2-6-1776176075310 last-index=6 last-term=2 size-in-bytes=25 2026-04-14T14:14:35.468Z [INFO] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: snapshot restore progress: id=2-6-1776176075310 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2026-04-14T14:14:35.468Z [INFO] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: restored from snapshot: id=2-6-1776176075310 last-index=6 last-term=2 size-in-bytes=25 2026-04-14T14:14:35.468Z [INFO] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2841277-5598-83b4-252d-a817aea710b7 Address:d2841277-5598-83b4-252d-a817aea710b7} {Suffrage:Voter ID:server-045e5ad8-868a-b639-2f7c-108f64ae7994 Address:045e5ad8-868a-b639-2f7c-108f64ae7994} {Suffrage:Voter ID:server-eca68486-f28a-b2f9-89c5-23de74b0b6e3 Address:eca68486-f28a-b2f9-89c5-23de74b0b6e3}]" 2026-04-14T14:14:35.468Z [INFO] server-eca68486-f28a-b2f9-89c5-23de74b0b6e3: entering follower state: follower="Node at eca68486-f28a-b2f9-89c5-23de74b0b6e3 [Follower]" leader-address= leader-id= 2026-04-14T14:14:35.500Z [WARN] server-d2841277-5598-83b4-252d-a817aea710b7: Election timeout reached, restarting election 2026-04-14T14:14:35.500Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: entering candidate state: node="Node at d2841277-5598-83b4-252d-a817aea710b7 [Candidate]" term=5 2026-04-14T14:14:35.500Z [INFO] server-045e5ad8-868a-b639-2f7c-108f64ae7994: entering follower state: follower="Node at 045e5ad8-868a-b639-2f7c-108f64ae7994 [Follower]" leader-address= leader-id= 2026-04-14T14:14:35.500Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: election won: term=5 tally=2 2026-04-14T14:14:35.500Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: entering leader state: leader="Node at d2841277-5598-83b4-252d-a817aea710b7 [Leader]" 2026-04-14T14:14:35.501Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: added peer, starting replication: peer=server-045e5ad8-868a-b639-2f7c-108f64ae7994 2026-04-14T14:14:35.501Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: added peer, starting replication: peer=server-eca68486-f28a-b2f9-89c5-23de74b0b6e3 2026-04-14T14:14:35.501Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: pipelining replication: peer="{Voter server-eca68486-f28a-b2f9-89c5-23de74b0b6e3 eca68486-f28a-b2f9-89c5-23de74b0b6e3}" 2026-04-14T14:14:35.501Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: pipelining replication: peer="{Voter server-045e5ad8-868a-b639-2f7c-108f64ae7994 045e5ad8-868a-b639-2f7c-108f64ae7994}" 2026-04-14T14:14:35.783Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:35.889Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:35.783933892 +0000 UTC m=+7.256119660 (1 nodes), 31.24µs from start of poll, 1.506640551s from cluster start. Timeout at 2026-04-14 14:14:35.88894108 +0000 UTC m=+7.361126849, 105.007189ms after stability 2026-04-14T14:14:35.889Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: aborting pipeline replication: peer="{Voter server-eca68486-f28a-b2f9-89c5-23de74b0b6e3 eca68486-f28a-b2f9-89c5-23de74b0b6e3}" 2026-04-14T14:14:35.889Z [INFO] server-d2841277-5598-83b4-252d-a817aea710b7: aborting pipeline replication: peer="{Voter server-045e5ad8-868a-b639-2f7c-108f64ae7994 045e5ad8-868a-b639-2f7c-108f64ae7994}" === RUN TestRaft_RecoverCluster/snapshot,_with_trailing_logs 2026-04-14T14:14:35.890Z [INFO] server-6611c0ea-a56d-281f-67be-f44b60b189b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6611c0ea-a56d-281f-67be-f44b60b189b3 Address:6611c0ea-a56d-281f-67be-f44b60b189b3} {Suffrage:Voter ID:server-177aedc7-d18b-4291-4bc0-ebbef6472f33 Address:177aedc7-d18b-4291-4bc0-ebbef6472f33} {Suffrage:Voter ID:server-ac172b42-222e-4220-d64f-987077ed341b Address:ac172b42-222e-4220-d64f-987077ed341b}]" 2026-04-14T14:14:35.890Z [INFO] server-177aedc7-d18b-4291-4bc0-ebbef6472f33: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6611c0ea-a56d-281f-67be-f44b60b189b3 Address:6611c0ea-a56d-281f-67be-f44b60b189b3} {Suffrage:Voter ID:server-177aedc7-d18b-4291-4bc0-ebbef6472f33 Address:177aedc7-d18b-4291-4bc0-ebbef6472f33} {Suffrage:Voter ID:server-ac172b42-222e-4220-d64f-987077ed341b Address:ac172b42-222e-4220-d64f-987077ed341b}]" 2026-04-14T14:14:35.890Z [INFO] server-6611c0ea-a56d-281f-67be-f44b60b189b3: entering follower state: follower="Node at 6611c0ea-a56d-281f-67be-f44b60b189b3 [Follower]" leader-address= leader-id= 2026-04-14T14:14:35.890Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6611c0ea-a56d-281f-67be-f44b60b189b3 Address:6611c0ea-a56d-281f-67be-f44b60b189b3} {Suffrage:Voter ID:server-177aedc7-d18b-4291-4bc0-ebbef6472f33 Address:177aedc7-d18b-4291-4bc0-ebbef6472f33} {Suffrage:Voter ID:server-ac172b42-222e-4220-d64f-987077ed341b Address:ac172b42-222e-4220-d64f-987077ed341b}]" 2026-04-14T14:14:35.890Z [INFO] server-177aedc7-d18b-4291-4bc0-ebbef6472f33: entering follower state: follower="Node at 177aedc7-d18b-4291-4bc0-ebbef6472f33 [Follower]" leader-address= leader-id= 2026-04-14T14:14:35.890Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:35.890Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: entering follower state: follower="Node at ac172b42-222e-4220-d64f-987077ed341b [Follower]" leader-address= leader-id= 2026-04-14T14:14:35.942Z [WARN] server-ac172b42-222e-4220-d64f-987077ed341b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:35.942Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: entering candidate state: node="Node at ac172b42-222e-4220-d64f-987077ed341b [Candidate]" term=2 2026-04-14T14:14:35.942Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: election won: term=2 tally=2 2026-04-14T14:14:35.942Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: entering leader state: leader="Node at ac172b42-222e-4220-d64f-987077ed341b [Leader]" 2026-04-14T14:14:35.942Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: added peer, starting replication: peer=server-6611c0ea-a56d-281f-67be-f44b60b189b3 2026-04-14T14:14:35.942Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: added peer, starting replication: peer=server-177aedc7-d18b-4291-4bc0-ebbef6472f33 2026-04-14T14:14:35.942Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: pipelining replication: peer="{Voter server-6611c0ea-a56d-281f-67be-f44b60b189b3 6611c0ea-a56d-281f-67be-f44b60b189b3}" 2026-04-14T14:14:35.942Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: pipelining replication: peer="{Voter server-177aedc7-d18b-4291-4bc0-ebbef6472f33 177aedc7-d18b-4291-4bc0-ebbef6472f33}" 2026-04-14T14:14:36.047Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:35.942399829 +0000 UTC m=+7.414585597 (1 nodes), 51.678676ms from start of poll, 51.936769ms from cluster start. Timeout at 2026-04-14 14:14:36.047404003 +0000 UTC m=+7.519589767, 105.00417ms after stability 2026-04-14T14:14:36.049Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: aborting pipeline replication: peer="{Voter server-177aedc7-d18b-4291-4bc0-ebbef6472f33 177aedc7-d18b-4291-4bc0-ebbef6472f33}" 2026-04-14T14:14:36.049Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: aborting pipeline replication: peer="{Voter server-6611c0ea-a56d-281f-67be-f44b60b189b3 6611c0ea-a56d-281f-67be-f44b60b189b3}" 2026-04-14T14:14:36.162Z [INFO] creating new snapshot: path=/tmp/raft2100499242/snapshots/2-27-1776176076162.tmp 2026-04-14T14:14:36.200Z [INFO] server-6611c0ea-a56d-281f-67be-f44b60b189b3: starting restore from snapshot: id=2-27-1776176076162 last-index=27 last-term=2 size-in-bytes=168 2026-04-14T14:14:36.200Z [INFO] server-6611c0ea-a56d-281f-67be-f44b60b189b3: snapshot restore progress: id=2-27-1776176076162 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2026-04-14T14:14:36.200Z [INFO] server-6611c0ea-a56d-281f-67be-f44b60b189b3: restored from snapshot: id=2-27-1776176076162 last-index=27 last-term=2 size-in-bytes=168 2026-04-14T14:14:36.200Z [INFO] server-6611c0ea-a56d-281f-67be-f44b60b189b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6611c0ea-a56d-281f-67be-f44b60b189b3 Address:6611c0ea-a56d-281f-67be-f44b60b189b3} {Suffrage:Voter ID:server-177aedc7-d18b-4291-4bc0-ebbef6472f33 Address:177aedc7-d18b-4291-4bc0-ebbef6472f33} {Suffrage:Voter ID:server-ac172b42-222e-4220-d64f-987077ed341b Address:ac172b42-222e-4220-d64f-987077ed341b}]" 2026-04-14T14:14:36.200Z [INFO] creating new snapshot: path=/tmp/raft2300880345/snapshots/2-27-1776176076200.tmp 2026-04-14T14:14:36.200Z [INFO] server-6611c0ea-a56d-281f-67be-f44b60b189b3: entering follower state: follower="Node at 6611c0ea-a56d-281f-67be-f44b60b189b3 [Follower]" leader-address= leader-id= 2026-04-14T14:14:36.201Z [INFO] server-177aedc7-d18b-4291-4bc0-ebbef6472f33: starting restore from snapshot: id=2-27-1776176076200 last-index=27 last-term=2 size-in-bytes=168 2026-04-14T14:14:36.201Z [INFO] server-177aedc7-d18b-4291-4bc0-ebbef6472f33: snapshot restore progress: id=2-27-1776176076200 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2026-04-14T14:14:36.201Z [INFO] server-177aedc7-d18b-4291-4bc0-ebbef6472f33: restored from snapshot: id=2-27-1776176076200 last-index=27 last-term=2 size-in-bytes=168 2026-04-14T14:14:36.201Z [INFO] server-177aedc7-d18b-4291-4bc0-ebbef6472f33: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6611c0ea-a56d-281f-67be-f44b60b189b3 Address:6611c0ea-a56d-281f-67be-f44b60b189b3} {Suffrage:Voter ID:server-177aedc7-d18b-4291-4bc0-ebbef6472f33 Address:177aedc7-d18b-4291-4bc0-ebbef6472f33} {Suffrage:Voter ID:server-ac172b42-222e-4220-d64f-987077ed341b Address:ac172b42-222e-4220-d64f-987077ed341b}]" 2026-04-14T14:14:36.201Z [INFO] creating new snapshot: path=/tmp/raft943014117/snapshots/2-27-1776176076201.tmp 2026-04-14T14:14:36.201Z [INFO] server-177aedc7-d18b-4291-4bc0-ebbef6472f33: entering follower state: follower="Node at 177aedc7-d18b-4291-4bc0-ebbef6472f33 [Follower]" leader-address= leader-id= 2026-04-14T14:14:36.201Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: starting restore from snapshot: id=2-27-1776176076201 last-index=27 last-term=2 size-in-bytes=168 2026-04-14T14:14:36.201Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: snapshot restore progress: id=2-27-1776176076201 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2026-04-14T14:14:36.201Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: restored from snapshot: id=2-27-1776176076201 last-index=27 last-term=2 size-in-bytes=168 2026-04-14T14:14:36.201Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6611c0ea-a56d-281f-67be-f44b60b189b3 Address:6611c0ea-a56d-281f-67be-f44b60b189b3} {Suffrage:Voter ID:server-177aedc7-d18b-4291-4bc0-ebbef6472f33 Address:177aedc7-d18b-4291-4bc0-ebbef6472f33} {Suffrage:Voter ID:server-ac172b42-222e-4220-d64f-987077ed341b Address:ac172b42-222e-4220-d64f-987077ed341b}]" 2026-04-14T14:14:36.201Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: entering follower state: follower="Node at ac172b42-222e-4220-d64f-987077ed341b [Follower]" leader-address= leader-id= 2026-04-14T14:14:36.256Z [WARN] server-ac172b42-222e-4220-d64f-987077ed341b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:36.256Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: entering candidate state: node="Node at ac172b42-222e-4220-d64f-987077ed341b [Candidate]" term=3 2026-04-14T14:14:36.256Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: election won: term=3 tally=2 2026-04-14T14:14:36.256Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: entering leader state: leader="Node at ac172b42-222e-4220-d64f-987077ed341b [Leader]" 2026-04-14T14:14:36.256Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: added peer, starting replication: peer=server-6611c0ea-a56d-281f-67be-f44b60b189b3 2026-04-14T14:14:36.256Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: added peer, starting replication: peer=server-177aedc7-d18b-4291-4bc0-ebbef6472f33 2026-04-14T14:14:36.256Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: pipelining replication: peer="{Voter server-177aedc7-d18b-4291-4bc0-ebbef6472f33 177aedc7-d18b-4291-4bc0-ebbef6472f33}" 2026-04-14T14:14:36.256Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: pipelining replication: peer="{Voter server-6611c0ea-a56d-281f-67be-f44b60b189b3 6611c0ea-a56d-281f-67be-f44b60b189b3}" 2026-04-14T14:14:36.517Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:36.622Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:36.517481362 +0000 UTC m=+7.989667125 (1 nodes), 29.191µs from start of poll, 627.018297ms from cluster start. Timeout at 2026-04-14 14:14:36.6224884 +0000 UTC m=+8.094674169, 105.007044ms after stability 2026-04-14T14:14:36.623Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: aborting pipeline replication: peer="{Voter server-177aedc7-d18b-4291-4bc0-ebbef6472f33 177aedc7-d18b-4291-4bc0-ebbef6472f33}" 2026-04-14T14:14:36.623Z [INFO] server-ac172b42-222e-4220-d64f-987077ed341b: aborting pipeline replication: peer="{Voter server-6611c0ea-a56d-281f-67be-f44b60b189b3 6611c0ea-a56d-281f-67be-f44b60b189b3}" --- PASS: TestRaft_RecoverCluster (2.99s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.64s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (1.61s) --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.73s) === RUN TestRaft_HasExistingState raft_test.go:258: Skipping unreliable test --- SKIP: TestRaft_HasExistingState (0.00s) === RUN TestRaft_SingleNode 2026-04-14T14:14:36.624Z [INFO] server-c8c76b81-b74d-b5e9-20e2-95fc863d8ef6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8c76b81-b74d-b5e9-20e2-95fc863d8ef6 Address:c8c76b81-b74d-b5e9-20e2-95fc863d8ef6}]" 2026-04-14T14:14:36.624Z [INFO] server-c8c76b81-b74d-b5e9-20e2-95fc863d8ef6: entering follower state: follower="Node at c8c76b81-b74d-b5e9-20e2-95fc863d8ef6 [Follower]" leader-address= leader-id= 2026-04-14T14:14:36.696Z [WARN] server-c8c76b81-b74d-b5e9-20e2-95fc863d8ef6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:36.696Z [INFO] server-c8c76b81-b74d-b5e9-20e2-95fc863d8ef6: entering candidate state: node="Node at c8c76b81-b74d-b5e9-20e2-95fc863d8ef6 [Candidate]" term=2 2026-04-14T14:14:36.696Z [INFO] server-c8c76b81-b74d-b5e9-20e2-95fc863d8ef6: election won: term=2 tally=1 2026-04-14T14:14:36.696Z [INFO] server-c8c76b81-b74d-b5e9-20e2-95fc863d8ef6: entering leader state: leader="Node at c8c76b81-b74d-b5e9-20e2-95fc863d8ef6 [Leader]" --- PASS: TestRaft_SingleNode (0.10s) === RUN TestRaft_TripleNode 2026-04-14T14:14:36.727Z [INFO] server-a20408b8-5eb0-c056-a119-faa63bb15034: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a20408b8-5eb0-c056-a119-faa63bb15034 Address:a20408b8-5eb0-c056-a119-faa63bb15034} {Suffrage:Voter ID:server-385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86 Address:385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86} {Suffrage:Voter ID:server-4e486fd0-e5a5-413d-60de-1340a2aaaae0 Address:4e486fd0-e5a5-413d-60de-1340a2aaaae0}]" 2026-04-14T14:14:36.727Z [INFO] server-a20408b8-5eb0-c056-a119-faa63bb15034: entering follower state: follower="Node at a20408b8-5eb0-c056-a119-faa63bb15034 [Follower]" leader-address= leader-id= 2026-04-14T14:14:36.727Z [INFO] server-385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a20408b8-5eb0-c056-a119-faa63bb15034 Address:a20408b8-5eb0-c056-a119-faa63bb15034} {Suffrage:Voter ID:server-385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86 Address:385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86} {Suffrage:Voter ID:server-4e486fd0-e5a5-413d-60de-1340a2aaaae0 Address:4e486fd0-e5a5-413d-60de-1340a2aaaae0}]" 2026-04-14T14:14:36.727Z [INFO] server-4e486fd0-e5a5-413d-60de-1340a2aaaae0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a20408b8-5eb0-c056-a119-faa63bb15034 Address:a20408b8-5eb0-c056-a119-faa63bb15034} {Suffrage:Voter ID:server-385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86 Address:385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86} {Suffrage:Voter ID:server-4e486fd0-e5a5-413d-60de-1340a2aaaae0 Address:4e486fd0-e5a5-413d-60de-1340a2aaaae0}]" 2026-04-14T14:14:36.727Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:14:36.727Z [INFO] server-385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86: entering follower state: follower="Node at 385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86 [Follower]" leader-address= leader-id= 2026-04-14T14:14:36.727Z [INFO] server-4e486fd0-e5a5-413d-60de-1340a2aaaae0: entering follower state: follower="Node at 4e486fd0-e5a5-413d-60de-1340a2aaaae0 [Follower]" leader-address= leader-id= 2026-04-14T14:14:36.779Z [WARN] server-385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:36.779Z [INFO] server-385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86: entering candidate state: node="Node at 385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86 [Candidate]" term=2 2026-04-14T14:14:36.779Z [INFO] server-385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86: election won: term=2 tally=2 2026-04-14T14:14:36.779Z [INFO] server-385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86: entering leader state: leader="Node at 385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86 [Leader]" 2026-04-14T14:14:36.779Z [INFO] server-385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86: added peer, starting replication: peer=server-a20408b8-5eb0-c056-a119-faa63bb15034 2026-04-14T14:14:36.779Z [INFO] server-385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86: added peer, starting replication: peer=server-4e486fd0-e5a5-413d-60de-1340a2aaaae0 2026-04-14T14:14:36.779Z [INFO] server-385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86: pipelining replication: peer="{Voter server-4e486fd0-e5a5-413d-60de-1340a2aaaae0 4e486fd0-e5a5-413d-60de-1340a2aaaae0}" 2026-04-14T14:14:36.779Z [INFO] server-385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86: pipelining replication: peer="{Voter server-a20408b8-5eb0-c056-a119-faa63bb15034 a20408b8-5eb0-c056-a119-faa63bb15034}" 2026-04-14T14:14:36.885Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:14:36.779303582 +0000 UTC m=+8.251489362 (2 nodes), 51.908248ms from start of poll, 52.285624ms from cluster start. Timeout at 2026-04-14 14:14:36.884306179 +0000 UTC m=+8.356491948, 105.002586ms after stability 2026-04-14T14:14:36.885Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:36.990Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:36.885174687 +0000 UTC m=+8.357360451 (1 nodes), 628ns from start of poll, 158.156713ms from cluster start. Timeout at 2026-04-14 14:14:36.990178149 +0000 UTC m=+8.462363915, 105.003464ms after stability 2026-04-14T14:14:37.001Z [INFO] server-385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86: aborting pipeline replication: peer="{Voter server-4e486fd0-e5a5-413d-60de-1340a2aaaae0 4e486fd0-e5a5-413d-60de-1340a2aaaae0}" 2026-04-14T14:14:37.001Z [INFO] server-385dd2f6-c6fe-a9ac-c1c6-698e6d7b0b86: aborting pipeline replication: peer="{Voter server-a20408b8-5eb0-c056-a119-faa63bb15034 a20408b8-5eb0-c056-a119-faa63bb15034}" --- PASS: TestRaft_TripleNode (0.28s) === RUN TestRaft_LeaderFail 2026-04-14T14:14:37.003Z [INFO] server-1cd3256d-89a3-23e6-d394-af44898f4637: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cd3256d-89a3-23e6-d394-af44898f4637 Address:1cd3256d-89a3-23e6-d394-af44898f4637} {Suffrage:Voter ID:server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 Address:62afb84b-8e5e-4979-061c-f4cc9b0b1037} {Suffrage:Voter ID:server-8f2835fe-942b-99c9-b58b-4af8d881b12c Address:8f2835fe-942b-99c9-b58b-4af8d881b12c}]" 2026-04-14T14:14:37.003Z [INFO] server-1cd3256d-89a3-23e6-d394-af44898f4637: entering follower state: follower="Node at 1cd3256d-89a3-23e6-d394-af44898f4637 [Follower]" leader-address= leader-id= 2026-04-14T14:14:37.003Z [INFO] server-62afb84b-8e5e-4979-061c-f4cc9b0b1037: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cd3256d-89a3-23e6-d394-af44898f4637 Address:1cd3256d-89a3-23e6-d394-af44898f4637} {Suffrage:Voter ID:server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 Address:62afb84b-8e5e-4979-061c-f4cc9b0b1037} {Suffrage:Voter ID:server-8f2835fe-942b-99c9-b58b-4af8d881b12c Address:8f2835fe-942b-99c9-b58b-4af8d881b12c}]" 2026-04-14T14:14:37.003Z [INFO] server-62afb84b-8e5e-4979-061c-f4cc9b0b1037: entering follower state: follower="Node at 62afb84b-8e5e-4979-061c-f4cc9b0b1037 [Follower]" leader-address= leader-id= 2026-04-14T14:14:37.003Z [INFO] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cd3256d-89a3-23e6-d394-af44898f4637 Address:1cd3256d-89a3-23e6-d394-af44898f4637} {Suffrage:Voter ID:server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 Address:62afb84b-8e5e-4979-061c-f4cc9b0b1037} {Suffrage:Voter ID:server-8f2835fe-942b-99c9-b58b-4af8d881b12c Address:8f2835fe-942b-99c9-b58b-4af8d881b12c}]" 2026-04-14T14:14:37.003Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:14:37.003Z [INFO] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: entering follower state: follower="Node at 8f2835fe-942b-99c9-b58b-4af8d881b12c [Follower]" leader-address= leader-id= 2026-04-14T14:14:37.060Z [WARN] server-1cd3256d-89a3-23e6-d394-af44898f4637: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:37.060Z [INFO] server-1cd3256d-89a3-23e6-d394-af44898f4637: entering candidate state: node="Node at 1cd3256d-89a3-23e6-d394-af44898f4637 [Candidate]" term=2 2026-04-14T14:14:37.060Z [INFO] server-1cd3256d-89a3-23e6-d394-af44898f4637: election won: term=2 tally=2 2026-04-14T14:14:37.060Z [INFO] server-1cd3256d-89a3-23e6-d394-af44898f4637: entering leader state: leader="Node at 1cd3256d-89a3-23e6-d394-af44898f4637 [Leader]" 2026-04-14T14:14:37.060Z [INFO] server-1cd3256d-89a3-23e6-d394-af44898f4637: added peer, starting replication: peer=server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 2026-04-14T14:14:37.060Z [INFO] server-1cd3256d-89a3-23e6-d394-af44898f4637: added peer, starting replication: peer=server-8f2835fe-942b-99c9-b58b-4af8d881b12c 2026-04-14T14:14:37.060Z [INFO] server-1cd3256d-89a3-23e6-d394-af44898f4637: pipelining replication: peer="{Voter server-8f2835fe-942b-99c9-b58b-4af8d881b12c 8f2835fe-942b-99c9-b58b-4af8d881b12c}" 2026-04-14T14:14:37.061Z [INFO] server-1cd3256d-89a3-23e6-d394-af44898f4637: pipelining replication: peer="{Voter server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 62afb84b-8e5e-4979-061c-f4cc9b0b1037}" 2026-04-14T14:14:37.165Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:14:37.060857796 +0000 UTC m=+8.533043566 (2 nodes), 57.396313ms from start of poll, 57.819665ms from cluster start. Timeout at 2026-04-14 14:14:37.165861329 +0000 UTC m=+8.638047098, 105.003532ms after stability 2026-04-14T14:14:37.165Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:37.271Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:37.165979634 +0000 UTC m=+8.638165402 (1 nodes), 710ns from start of poll, 162.941501ms from cluster start. Timeout at 2026-04-14 14:14:37.270987988 +0000 UTC m=+8.743173770, 105.008368ms after stability raft_test.go:378: [INFO] Disconnecting Node at 1cd3256d-89a3-23e6-d394-af44898f4637 [Leader] 2026-04-14T14:14:37.286Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to heartbeat to: peer=62afb84b-8e5e-4979-061c-f4cc9b0b1037 backoff time=10ms error="failed to connect to peer: 62afb84b-8e5e-4979-061c-f4cc9b0b1037" 2026-04-14T14:14:37.286Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:37.286Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to pipeline appendEntries: peer="{Voter server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 62afb84b-8e5e-4979-061c-f4cc9b0b1037}" error="append pipeline closed" 2026-04-14T14:14:37.286Z [INFO] server-1cd3256d-89a3-23e6-d394-af44898f4637: aborting pipeline replication: peer="{Voter server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 62afb84b-8e5e-4979-061c-f4cc9b0b1037}" 2026-04-14T14:14:37.289Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to pipeline appendEntries: peer="{Voter server-8f2835fe-942b-99c9-b58b-4af8d881b12c 8f2835fe-942b-99c9-b58b-4af8d881b12c}" error="append pipeline closed" 2026-04-14T14:14:37.289Z [INFO] server-1cd3256d-89a3-23e6-d394-af44898f4637: aborting pipeline replication: peer="{Voter server-8f2835fe-942b-99c9-b58b-4af8d881b12c 8f2835fe-942b-99c9-b58b-4af8d881b12c}" 2026-04-14T14:14:37.291Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to heartbeat to: peer=8f2835fe-942b-99c9-b58b-4af8d881b12c backoff time=10ms error="failed to connect to peer: 8f2835fe-942b-99c9-b58b-4af8d881b12c" 2026-04-14T14:14:37.293Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to appendEntries to: peer="{Voter server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 62afb84b-8e5e-4979-061c-f4cc9b0b1037}" error="failed to connect to peer: 62afb84b-8e5e-4979-061c-f4cc9b0b1037" 2026-04-14T14:14:37.298Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to appendEntries to: peer="{Voter server-8f2835fe-942b-99c9-b58b-4af8d881b12c 8f2835fe-942b-99c9-b58b-4af8d881b12c}" error="failed to connect to peer: 8f2835fe-942b-99c9-b58b-4af8d881b12c" 2026-04-14T14:14:37.305Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to heartbeat to: peer=62afb84b-8e5e-4979-061c-f4cc9b0b1037 backoff time=10ms error="failed to connect to peer: 62afb84b-8e5e-4979-061c-f4cc9b0b1037" 2026-04-14T14:14:37.311Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to appendEntries to: peer="{Voter server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 62afb84b-8e5e-4979-061c-f4cc9b0b1037}" error="failed to connect to peer: 62afb84b-8e5e-4979-061c-f4cc9b0b1037" 2026-04-14T14:14:37.311Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to heartbeat to: peer=8f2835fe-942b-99c9-b58b-4af8d881b12c backoff time=10ms error="failed to connect to peer: 8f2835fe-942b-99c9-b58b-4af8d881b12c" 2026-04-14T14:14:37.316Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to appendEntries to: peer="{Voter server-8f2835fe-942b-99c9-b58b-4af8d881b12c 8f2835fe-942b-99c9-b58b-4af8d881b12c}" error="failed to connect to peer: 8f2835fe-942b-99c9-b58b-4af8d881b12c" 2026-04-14T14:14:37.323Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to heartbeat to: peer=62afb84b-8e5e-4979-061c-f4cc9b0b1037 backoff time=10ms error="failed to connect to peer: 62afb84b-8e5e-4979-061c-f4cc9b0b1037" 2026-04-14T14:14:37.331Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to appendEntries to: peer="{Voter server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 62afb84b-8e5e-4979-061c-f4cc9b0b1037}" error="failed to connect to peer: 62afb84b-8e5e-4979-061c-f4cc9b0b1037" 2026-04-14T14:14:37.331Z [WARN] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to contact: server-id=server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 time=50.951112ms 2026-04-14T14:14:37.331Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to heartbeat to: peer=8f2835fe-942b-99c9-b58b-4af8d881b12c backoff time=10ms error="failed to connect to peer: 8f2835fe-942b-99c9-b58b-4af8d881b12c" 2026-04-14T14:14:37.333Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to appendEntries to: peer="{Voter server-8f2835fe-942b-99c9-b58b-4af8d881b12c 8f2835fe-942b-99c9-b58b-4af8d881b12c}" error="failed to connect to peer: 8f2835fe-942b-99c9-b58b-4af8d881b12c" 2026-04-14T14:14:37.342Z [WARN] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to contact: server-id=server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 time=61.721214ms 2026-04-14T14:14:37.342Z [WARN] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to contact: server-id=server-8f2835fe-942b-99c9-b58b-4af8d881b12c time=60.454297ms 2026-04-14T14:14:37.342Z [WARN] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to contact quorum of nodes, stepping down 2026-04-14T14:14:37.342Z [INFO] server-1cd3256d-89a3-23e6-d394-af44898f4637: entering follower state: follower="Node at 1cd3256d-89a3-23e6-d394-af44898f4637 [Follower]" leader-address= leader-id= 2026-04-14T14:14:37.344Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to heartbeat to: peer=62afb84b-8e5e-4979-061c-f4cc9b0b1037 backoff time=20ms error="failed to connect to peer: 62afb84b-8e5e-4979-061c-f4cc9b0b1037" 2026-04-14T14:14:37.349Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to heartbeat to: peer=8f2835fe-942b-99c9-b58b-4af8d881b12c backoff time=20ms error="failed to connect to peer: 8f2835fe-942b-99c9-b58b-4af8d881b12c" 2026-04-14T14:14:37.357Z [WARN] server-62afb84b-8e5e-4979-061c-f4cc9b0b1037: heartbeat timeout reached, starting election: last-leader-addr=1cd3256d-89a3-23e6-d394-af44898f4637 last-leader-id=server-1cd3256d-89a3-23e6-d394-af44898f4637 2026-04-14T14:14:37.357Z [INFO] server-62afb84b-8e5e-4979-061c-f4cc9b0b1037: entering candidate state: node="Node at 62afb84b-8e5e-4979-061c-f4cc9b0b1037 [Candidate]" term=3 2026-04-14T14:14:37.357Z [WARN] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: rejecting vote request since we have a leader: from=62afb84b-8e5e-4979-061c-f4cc9b0b1037 leader=1cd3256d-89a3-23e6-d394-af44898f4637 leader-id=server-1cd3256d-89a3-23e6-d394-af44898f4637 2026-04-14T14:14:37.357Z [ERROR] server-62afb84b-8e5e-4979-061c-f4cc9b0b1037: failed to make requestVote RPC: target="{Voter server-1cd3256d-89a3-23e6-d394-af44898f4637 1cd3256d-89a3-23e6-d394-af44898f4637}" error="failed to connect to peer: 1cd3256d-89a3-23e6-d394-af44898f4637" term=3 2026-04-14T14:14:37.361Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to appendEntries to: peer="{Voter server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 62afb84b-8e5e-4979-061c-f4cc9b0b1037}" error="failed to connect to peer: 62afb84b-8e5e-4979-061c-f4cc9b0b1037" 2026-04-14T14:14:37.361Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to appendEntries to: peer="{Voter server-8f2835fe-942b-99c9-b58b-4af8d881b12c 8f2835fe-942b-99c9-b58b-4af8d881b12c}" error="failed to connect to peer: 8f2835fe-942b-99c9-b58b-4af8d881b12c" 2026-04-14T14:14:37.364Z [WARN] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: heartbeat timeout reached, starting election: last-leader-addr=1cd3256d-89a3-23e6-d394-af44898f4637 last-leader-id=server-1cd3256d-89a3-23e6-d394-af44898f4637 2026-04-14T14:14:37.364Z [INFO] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: entering candidate state: node="Node at 8f2835fe-942b-99c9-b58b-4af8d881b12c [Candidate]" term=3 2026-04-14T14:14:37.364Z [ERROR] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: failed to make requestVote RPC: target="{Voter server-1cd3256d-89a3-23e6-d394-af44898f4637 1cd3256d-89a3-23e6-d394-af44898f4637}" error="failed to connect to peer: 1cd3256d-89a3-23e6-d394-af44898f4637" term=3 2026-04-14T14:14:37.365Z [INFO] server-62afb84b-8e5e-4979-061c-f4cc9b0b1037: duplicate requestVote for same term: term=3 2026-04-14T14:14:37.422Z [WARN] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: Election timeout reached, restarting election 2026-04-14T14:14:37.422Z [INFO] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: entering candidate state: node="Node at 8f2835fe-942b-99c9-b58b-4af8d881b12c [Candidate]" term=4 2026-04-14T14:14:37.423Z [INFO] server-62afb84b-8e5e-4979-061c-f4cc9b0b1037: entering follower state: follower="Node at 62afb84b-8e5e-4979-061c-f4cc9b0b1037 [Follower]" leader-address= leader-id= 2026-04-14T14:14:37.423Z [ERROR] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: failed to make requestVote RPC: target="{Voter server-1cd3256d-89a3-23e6-d394-af44898f4637 1cd3256d-89a3-23e6-d394-af44898f4637}" error="failed to connect to peer: 1cd3256d-89a3-23e6-d394-af44898f4637" term=4 2026-04-14T14:14:37.423Z [INFO] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: election won: term=4 tally=2 2026-04-14T14:14:37.423Z [INFO] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: entering leader state: leader="Node at 8f2835fe-942b-99c9-b58b-4af8d881b12c [Leader]" 2026-04-14T14:14:37.423Z [INFO] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: added peer, starting replication: peer=server-1cd3256d-89a3-23e6-d394-af44898f4637 2026-04-14T14:14:37.423Z [INFO] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: added peer, starting replication: peer=server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 2026-04-14T14:14:37.423Z [ERROR] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: failed to appendEntries to: peer="{Voter server-1cd3256d-89a3-23e6-d394-af44898f4637 1cd3256d-89a3-23e6-d394-af44898f4637}" error="failed to connect to peer: 1cd3256d-89a3-23e6-d394-af44898f4637" 2026-04-14T14:14:37.423Z [INFO] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: pipelining replication: peer="{Voter server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 62afb84b-8e5e-4979-061c-f4cc9b0b1037}" 2026-04-14T14:14:37.433Z [ERROR] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: failed to heartbeat to: peer=1cd3256d-89a3-23e6-d394-af44898f4637 backoff time=10ms error="failed to connect to peer: 1cd3256d-89a3-23e6-d394-af44898f4637" 2026-04-14T14:14:37.435Z [WARN] server-1cd3256d-89a3-23e6-d394-af44898f4637: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:37.436Z [INFO] server-1cd3256d-89a3-23e6-d394-af44898f4637: entering candidate state: node="Node at 1cd3256d-89a3-23e6-d394-af44898f4637 [Candidate]" term=3 2026-04-14T14:14:37.436Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to make requestVote RPC: target="{Voter server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 62afb84b-8e5e-4979-061c-f4cc9b0b1037}" error="failed to connect to peer: 62afb84b-8e5e-4979-061c-f4cc9b0b1037" term=3 2026-04-14T14:14:37.436Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to make requestVote RPC: target="{Voter server-8f2835fe-942b-99c9-b58b-4af8d881b12c 8f2835fe-942b-99c9-b58b-4af8d881b12c}" error="failed to connect to peer: 8f2835fe-942b-99c9-b58b-4af8d881b12c" term=3 2026-04-14T14:14:37.440Z [ERROR] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: failed to appendEntries to: peer="{Voter server-1cd3256d-89a3-23e6-d394-af44898f4637 1cd3256d-89a3-23e6-d394-af44898f4637}" error="failed to connect to peer: 1cd3256d-89a3-23e6-d394-af44898f4637" 2026-04-14T14:14:37.450Z [ERROR] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: failed to heartbeat to: peer=1cd3256d-89a3-23e6-d394-af44898f4637 backoff time=10ms error="failed to connect to peer: 1cd3256d-89a3-23e6-d394-af44898f4637" 2026-04-14T14:14:37.460Z [ERROR] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: failed to appendEntries to: peer="{Voter server-1cd3256d-89a3-23e6-d394-af44898f4637 1cd3256d-89a3-23e6-d394-af44898f4637}" error="failed to connect to peer: 1cd3256d-89a3-23e6-d394-af44898f4637" 2026-04-14T14:14:37.470Z [ERROR] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: failed to heartbeat to: peer=1cd3256d-89a3-23e6-d394-af44898f4637 backoff time=10ms error="failed to connect to peer: 1cd3256d-89a3-23e6-d394-af44898f4637" 2026-04-14T14:14:37.473Z [WARN] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: failed to contact: server-id=server-1cd3256d-89a3-23e6-d394-af44898f4637 time=50.818227ms 2026-04-14T14:14:37.487Z [ERROR] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: failed to appendEntries to: peer="{Voter server-1cd3256d-89a3-23e6-d394-af44898f4637 1cd3256d-89a3-23e6-d394-af44898f4637}" error="failed to connect to peer: 1cd3256d-89a3-23e6-d394-af44898f4637" 2026-04-14T14:14:37.488Z [ERROR] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: failed to heartbeat to: peer=1cd3256d-89a3-23e6-d394-af44898f4637 backoff time=20ms error="failed to connect to peer: 1cd3256d-89a3-23e6-d394-af44898f4637" 2026-04-14T14:14:37.506Z [WARN] server-1cd3256d-89a3-23e6-d394-af44898f4637: Election timeout reached, restarting election 2026-04-14T14:14:37.506Z [INFO] server-1cd3256d-89a3-23e6-d394-af44898f4637: entering candidate state: node="Node at 1cd3256d-89a3-23e6-d394-af44898f4637 [Candidate]" term=4 2026-04-14T14:14:37.506Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to make requestVote RPC: target="{Voter server-8f2835fe-942b-99c9-b58b-4af8d881b12c 8f2835fe-942b-99c9-b58b-4af8d881b12c}" error="failed to connect to peer: 8f2835fe-942b-99c9-b58b-4af8d881b12c" term=4 2026-04-14T14:14:37.506Z [ERROR] server-1cd3256d-89a3-23e6-d394-af44898f4637: failed to make requestVote RPC: target="{Voter server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 62afb84b-8e5e-4979-061c-f4cc9b0b1037}" error="failed to connect to peer: 62afb84b-8e5e-4979-061c-f4cc9b0b1037" term=4 2026-04-14T14:14:37.517Z [ERROR] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: failed to heartbeat to: peer=1cd3256d-89a3-23e6-d394-af44898f4637 backoff time=25ms error="failed to connect to peer: 1cd3256d-89a3-23e6-d394-af44898f4637" 2026-04-14T14:14:37.520Z [WARN] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: failed to contact: server-id=server-1cd3256d-89a3-23e6-d394-af44898f4637 time=97.831923ms 2026-04-14T14:14:37.536Z [ERROR] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: failed to appendEntries to: peer="{Voter server-1cd3256d-89a3-23e6-d394-af44898f4637 1cd3256d-89a3-23e6-d394-af44898f4637}" error="failed to connect to peer: 1cd3256d-89a3-23e6-d394-af44898f4637" 2026-04-14T14:14:37.541Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:37.436042721 +0000 UTC m=+8.908228502 (1 nodes), 149.832996ms from start of poll, 433.004601ms from cluster start. Timeout at 2026-04-14 14:14:37.541043427 +0000 UTC m=+9.013229196, 105.000694ms after stability raft_test.go:413: [INFO] Reconnecting Node at 1cd3256d-89a3-23e6-d394-af44898f4637 [Candidate] 2026-04-14T14:14:37.551Z [INFO] server-1cd3256d-89a3-23e6-d394-af44898f4637: entering follower state: follower="Node at 1cd3256d-89a3-23e6-d394-af44898f4637 [Follower]" leader-address=8f2835fe-942b-99c9-b58b-4af8d881b12c leader-id=server-8f2835fe-942b-99c9-b58b-4af8d881b12c 2026-04-14T14:14:37.622Z [INFO] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: pipelining replication: peer="{Voter server-1cd3256d-89a3-23e6-d394-af44898f4637 1cd3256d-89a3-23e6-d394-af44898f4637}" 2026-04-14T14:14:37.627Z [INFO] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: aborting pipeline replication: peer="{Voter server-1cd3256d-89a3-23e6-d394-af44898f4637 1cd3256d-89a3-23e6-d394-af44898f4637}" 2026-04-14T14:14:37.627Z [INFO] server-8f2835fe-942b-99c9-b58b-4af8d881b12c: aborting pipeline replication: peer="{Voter server-62afb84b-8e5e-4979-061c-f4cc9b0b1037 62afb84b-8e5e-4979-061c-f4cc9b0b1037}" --- PASS: TestRaft_LeaderFail (0.63s) === RUN TestRaft_BehindFollower 2026-04-14T14:14:37.628Z [INFO] server-2f68d70d-15c3-006a-6165-fb56cf7d687b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f68d70d-15c3-006a-6165-fb56cf7d687b Address:2f68d70d-15c3-006a-6165-fb56cf7d687b} {Suffrage:Voter ID:server-9ba15cb8-80a6-eeaf-15bf-58b59cbfba4d Address:9ba15cb8-80a6-eeaf-15bf-58b59cbfba4d} {Suffrage:Voter ID:server-025151b6-04bc-b4fb-86df-4d3e0e7ea826 Address:025151b6-04bc-b4fb-86df-4d3e0e7ea826}]" 2026-04-14T14:14:37.628Z [INFO] server-9ba15cb8-80a6-eeaf-15bf-58b59cbfba4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f68d70d-15c3-006a-6165-fb56cf7d687b Address:2f68d70d-15c3-006a-6165-fb56cf7d687b} {Suffrage:Voter ID:server-9ba15cb8-80a6-eeaf-15bf-58b59cbfba4d Address:9ba15cb8-80a6-eeaf-15bf-58b59cbfba4d} {Suffrage:Voter ID:server-025151b6-04bc-b4fb-86df-4d3e0e7ea826 Address:025151b6-04bc-b4fb-86df-4d3e0e7ea826}]" 2026-04-14T14:14:37.628Z [INFO] server-2f68d70d-15c3-006a-6165-fb56cf7d687b: entering follower state: follower="Node at 2f68d70d-15c3-006a-6165-fb56cf7d687b [Follower]" leader-address= leader-id= 2026-04-14T14:14:37.628Z [INFO] server-025151b6-04bc-b4fb-86df-4d3e0e7ea826: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f68d70d-15c3-006a-6165-fb56cf7d687b Address:2f68d70d-15c3-006a-6165-fb56cf7d687b} {Suffrage:Voter ID:server-9ba15cb8-80a6-eeaf-15bf-58b59cbfba4d Address:9ba15cb8-80a6-eeaf-15bf-58b59cbfba4d} {Suffrage:Voter ID:server-025151b6-04bc-b4fb-86df-4d3e0e7ea826 Address:025151b6-04bc-b4fb-86df-4d3e0e7ea826}]" 2026-04-14T14:14:37.628Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:37.628Z [INFO] server-9ba15cb8-80a6-eeaf-15bf-58b59cbfba4d: entering follower state: follower="Node at 9ba15cb8-80a6-eeaf-15bf-58b59cbfba4d [Follower]" leader-address= leader-id= 2026-04-14T14:14:37.628Z [INFO] server-025151b6-04bc-b4fb-86df-4d3e0e7ea826: entering follower state: follower="Node at 025151b6-04bc-b4fb-86df-4d3e0e7ea826 [Follower]" leader-address= leader-id= 2026-04-14T14:14:37.684Z [WARN] server-025151b6-04bc-b4fb-86df-4d3e0e7ea826: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:37.684Z [INFO] server-025151b6-04bc-b4fb-86df-4d3e0e7ea826: entering candidate state: node="Node at 025151b6-04bc-b4fb-86df-4d3e0e7ea826 [Candidate]" term=2 2026-04-14T14:14:37.685Z [INFO] server-025151b6-04bc-b4fb-86df-4d3e0e7ea826: election won: term=2 tally=2 2026-04-14T14:14:37.685Z [INFO] server-025151b6-04bc-b4fb-86df-4d3e0e7ea826: entering leader state: leader="Node at 025151b6-04bc-b4fb-86df-4d3e0e7ea826 [Leader]" 2026-04-14T14:14:37.685Z [INFO] server-025151b6-04bc-b4fb-86df-4d3e0e7ea826: added peer, starting replication: peer=server-2f68d70d-15c3-006a-6165-fb56cf7d687b 2026-04-14T14:14:37.685Z [INFO] server-025151b6-04bc-b4fb-86df-4d3e0e7ea826: added peer, starting replication: peer=server-9ba15cb8-80a6-eeaf-15bf-58b59cbfba4d 2026-04-14T14:14:37.685Z [INFO] server-025151b6-04bc-b4fb-86df-4d3e0e7ea826: pipelining replication: peer="{Voter server-2f68d70d-15c3-006a-6165-fb56cf7d687b 2f68d70d-15c3-006a-6165-fb56cf7d687b}" 2026-04-14T14:14:37.685Z [INFO] server-025151b6-04bc-b4fb-86df-4d3e0e7ea826: pipelining replication: peer="{Voter server-9ba15cb8-80a6-eeaf-15bf-58b59cbfba4d 9ba15cb8-80a6-eeaf-15bf-58b59cbfba4d}" 2026-04-14T14:14:37.790Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:37.685100544 +0000 UTC m=+9.157286317 (1 nodes), 56.876438ms from start of poll, 57.12517ms from cluster start. Timeout at 2026-04-14 14:14:37.790101327 +0000 UTC m=+9.262287092, 105.000775ms after stability 2026-04-14T14:14:37.790Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:14:37.896Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:14:37.790506933 +0000 UTC m=+9.262692698 (2 nodes), 970ns from start of poll, 162.531551ms from cluster start. Timeout at 2026-04-14 14:14:37.895511746 +0000 UTC m=+9.367697514, 105.004816ms after stability 2026-04-14T14:14:37.896Z [ERROR] server-025151b6-04bc-b4fb-86df-4d3e0e7ea826: failed to pipeline appendEntries: peer="{Voter server-2f68d70d-15c3-006a-6165-fb56cf7d687b 2f68d70d-15c3-006a-6165-fb56cf7d687b}" error="append pipeline closed" 2026-04-14T14:14:37.896Z [INFO] server-025151b6-04bc-b4fb-86df-4d3e0e7ea826: aborting pipeline replication: peer="{Voter server-2f68d70d-15c3-006a-6165-fb56cf7d687b 2f68d70d-15c3-006a-6165-fb56cf7d687b}" 2026-04-14T14:14:37.896Z [ERROR] server-025151b6-04bc-b4fb-86df-4d3e0e7ea826: failed to appendEntries to: peer="{Voter server-2f68d70d-15c3-006a-6165-fb56cf7d687b 2f68d70d-15c3-006a-6165-fb56cf7d687b}" error="failed to connect to peer: 2f68d70d-15c3-006a-6165-fb56cf7d687b" raft_test.go:459: [INFO] Finished apply without behind follower 2026-04-14T14:14:37.907Z [INFO] server-025151b6-04bc-b4fb-86df-4d3e0e7ea826: pipelining replication: peer="{Voter server-2f68d70d-15c3-006a-6165-fb56cf7d687b 2f68d70d-15c3-006a-6165-fb56cf7d687b}" 2026-04-14T14:14:37.919Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:38.025Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:37.919979725 +0000 UTC m=+9.392165506 (1 nodes), 2.1µs from start of poll, 292.004359ms from cluster start. Timeout at 2026-04-14 14:14:38.024983007 +0000 UTC m=+9.497168771, 105.003265ms after stability 2026-04-14T14:14:38.025Z [INFO] server-025151b6-04bc-b4fb-86df-4d3e0e7ea826: aborting pipeline replication: peer="{Voter server-2f68d70d-15c3-006a-6165-fb56cf7d687b 2f68d70d-15c3-006a-6165-fb56cf7d687b}" 2026-04-14T14:14:38.025Z [INFO] server-025151b6-04bc-b4fb-86df-4d3e0e7ea826: aborting pipeline replication: peer="{Voter server-9ba15cb8-80a6-eeaf-15bf-58b59cbfba4d 9ba15cb8-80a6-eeaf-15bf-58b59cbfba4d}" --- PASS: TestRaft_BehindFollower (0.40s) === RUN TestRaft_ApplyNonLeader 2026-04-14T14:14:38.027Z [INFO] server-7bee06b5-5577-cda3-6da2-c23b76611724: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bee06b5-5577-cda3-6da2-c23b76611724 Address:7bee06b5-5577-cda3-6da2-c23b76611724} {Suffrage:Voter ID:server-939d44c6-ce1d-3538-0e08-5575b690fe45 Address:939d44c6-ce1d-3538-0e08-5575b690fe45} {Suffrage:Voter ID:server-ce4dc009-4924-99c8-345b-7d272c9b27de Address:ce4dc009-4924-99c8-345b-7d272c9b27de}]" 2026-04-14T14:14:38.027Z [INFO] server-939d44c6-ce1d-3538-0e08-5575b690fe45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bee06b5-5577-cda3-6da2-c23b76611724 Address:7bee06b5-5577-cda3-6da2-c23b76611724} {Suffrage:Voter ID:server-939d44c6-ce1d-3538-0e08-5575b690fe45 Address:939d44c6-ce1d-3538-0e08-5575b690fe45} {Suffrage:Voter ID:server-ce4dc009-4924-99c8-345b-7d272c9b27de Address:ce4dc009-4924-99c8-345b-7d272c9b27de}]" 2026-04-14T14:14:38.027Z [INFO] server-7bee06b5-5577-cda3-6da2-c23b76611724: entering follower state: follower="Node at 7bee06b5-5577-cda3-6da2-c23b76611724 [Follower]" leader-address= leader-id= 2026-04-14T14:14:38.027Z [INFO] server-939d44c6-ce1d-3538-0e08-5575b690fe45: entering follower state: follower="Node at 939d44c6-ce1d-3538-0e08-5575b690fe45 [Follower]" leader-address= leader-id= 2026-04-14T14:14:38.027Z [INFO] server-ce4dc009-4924-99c8-345b-7d272c9b27de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bee06b5-5577-cda3-6da2-c23b76611724 Address:7bee06b5-5577-cda3-6da2-c23b76611724} {Suffrage:Voter ID:server-939d44c6-ce1d-3538-0e08-5575b690fe45 Address:939d44c6-ce1d-3538-0e08-5575b690fe45} {Suffrage:Voter ID:server-ce4dc009-4924-99c8-345b-7d272c9b27de Address:ce4dc009-4924-99c8-345b-7d272c9b27de}]" 2026-04-14T14:14:38.027Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:38.027Z [INFO] server-ce4dc009-4924-99c8-345b-7d272c9b27de: entering follower state: follower="Node at ce4dc009-4924-99c8-345b-7d272c9b27de [Follower]" leader-address= leader-id= 2026-04-14T14:14:38.097Z [WARN] server-7bee06b5-5577-cda3-6da2-c23b76611724: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:38.097Z [INFO] server-7bee06b5-5577-cda3-6da2-c23b76611724: entering candidate state: node="Node at 7bee06b5-5577-cda3-6da2-c23b76611724 [Candidate]" term=2 2026-04-14T14:14:38.097Z [INFO] server-7bee06b5-5577-cda3-6da2-c23b76611724: election won: term=2 tally=2 2026-04-14T14:14:38.097Z [INFO] server-7bee06b5-5577-cda3-6da2-c23b76611724: entering leader state: leader="Node at 7bee06b5-5577-cda3-6da2-c23b76611724 [Leader]" 2026-04-14T14:14:38.097Z [INFO] server-7bee06b5-5577-cda3-6da2-c23b76611724: added peer, starting replication: peer=server-939d44c6-ce1d-3538-0e08-5575b690fe45 2026-04-14T14:14:38.097Z [INFO] server-7bee06b5-5577-cda3-6da2-c23b76611724: added peer, starting replication: peer=server-ce4dc009-4924-99c8-345b-7d272c9b27de 2026-04-14T14:14:38.097Z [INFO] server-7bee06b5-5577-cda3-6da2-c23b76611724: pipelining replication: peer="{Voter server-ce4dc009-4924-99c8-345b-7d272c9b27de ce4dc009-4924-99c8-345b-7d272c9b27de}" 2026-04-14T14:14:38.097Z [INFO] server-7bee06b5-5577-cda3-6da2-c23b76611724: pipelining replication: peer="{Voter server-939d44c6-ce1d-3538-0e08-5575b690fe45 939d44c6-ce1d-3538-0e08-5575b690fe45}" 2026-04-14T14:14:38.203Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:38.097882629 +0000 UTC m=+9.570068400 (1 nodes), 70.225617ms from start of poll, 71.472105ms from cluster start. Timeout at 2026-04-14 14:14:38.202886986 +0000 UTC m=+9.675072753, 105.004353ms after stability 2026-04-14T14:14:38.203Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:14:38.309Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:14:38.203762848 +0000 UTC m=+9.675948614 (2 nodes), 698ns from start of poll, 177.352319ms from cluster start. Timeout at 2026-04-14 14:14:38.308767735 +0000 UTC m=+9.780953502, 105.004888ms after stability 2026-04-14T14:14:38.309Z [INFO] server-7bee06b5-5577-cda3-6da2-c23b76611724: aborting pipeline replication: peer="{Voter server-ce4dc009-4924-99c8-345b-7d272c9b27de ce4dc009-4924-99c8-345b-7d272c9b27de}" 2026-04-14T14:14:38.309Z [INFO] server-7bee06b5-5577-cda3-6da2-c23b76611724: aborting pipeline replication: peer="{Voter server-939d44c6-ce1d-3538-0e08-5575b690fe45 939d44c6-ce1d-3538-0e08-5575b690fe45}" --- PASS: TestRaft_ApplyNonLeader (0.28s) === RUN TestRaft_ApplyConcurrent 2026-04-14T14:14:38.310Z [INFO] server-59aa635f-b628-8c3b-a114-85b40f48fcde: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59aa635f-b628-8c3b-a114-85b40f48fcde Address:59aa635f-b628-8c3b-a114-85b40f48fcde} {Suffrage:Voter ID:server-4e4a7d12-e499-cd4b-0164-a2979f99a41c Address:4e4a7d12-e499-cd4b-0164-a2979f99a41c} {Suffrage:Voter ID:server-1b737291-18a7-7c2d-ebae-e785e2af3a96 Address:1b737291-18a7-7c2d-ebae-e785e2af3a96}]" 2026-04-14T14:14:38.310Z [INFO] server-59aa635f-b628-8c3b-a114-85b40f48fcde: entering follower state: follower="Node at 59aa635f-b628-8c3b-a114-85b40f48fcde [Follower]" leader-address= leader-id= 2026-04-14T14:14:38.310Z [INFO] server-4e4a7d12-e499-cd4b-0164-a2979f99a41c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59aa635f-b628-8c3b-a114-85b40f48fcde Address:59aa635f-b628-8c3b-a114-85b40f48fcde} {Suffrage:Voter ID:server-4e4a7d12-e499-cd4b-0164-a2979f99a41c Address:4e4a7d12-e499-cd4b-0164-a2979f99a41c} {Suffrage:Voter ID:server-1b737291-18a7-7c2d-ebae-e785e2af3a96 Address:1b737291-18a7-7c2d-ebae-e785e2af3a96}]" 2026-04-14T14:14:38.310Z [INFO] server-4e4a7d12-e499-cd4b-0164-a2979f99a41c: entering follower state: follower="Node at 4e4a7d12-e499-cd4b-0164-a2979f99a41c [Follower]" leader-address= leader-id= 2026-04-14T14:14:38.310Z [INFO] server-1b737291-18a7-7c2d-ebae-e785e2af3a96: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59aa635f-b628-8c3b-a114-85b40f48fcde Address:59aa635f-b628-8c3b-a114-85b40f48fcde} {Suffrage:Voter ID:server-4e4a7d12-e499-cd4b-0164-a2979f99a41c Address:4e4a7d12-e499-cd4b-0164-a2979f99a41c} {Suffrage:Voter ID:server-1b737291-18a7-7c2d-ebae-e785e2af3a96 Address:1b737291-18a7-7c2d-ebae-e785e2af3a96}]" 2026-04-14T14:14:38.311Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:38.311Z [INFO] server-1b737291-18a7-7c2d-ebae-e785e2af3a96: entering follower state: follower="Node at 1b737291-18a7-7c2d-ebae-e785e2af3a96 [Follower]" leader-address= leader-id= 2026-04-14T14:14:38.428Z [WARN] server-59aa635f-b628-8c3b-a114-85b40f48fcde: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:38.428Z [INFO] server-59aa635f-b628-8c3b-a114-85b40f48fcde: entering candidate state: node="Node at 59aa635f-b628-8c3b-a114-85b40f48fcde [Candidate]" term=2 2026-04-14T14:14:38.428Z [INFO] server-59aa635f-b628-8c3b-a114-85b40f48fcde: election won: term=2 tally=2 2026-04-14T14:14:38.428Z [INFO] server-59aa635f-b628-8c3b-a114-85b40f48fcde: entering leader state: leader="Node at 59aa635f-b628-8c3b-a114-85b40f48fcde [Leader]" 2026-04-14T14:14:38.428Z [INFO] server-59aa635f-b628-8c3b-a114-85b40f48fcde: added peer, starting replication: peer=server-4e4a7d12-e499-cd4b-0164-a2979f99a41c 2026-04-14T14:14:38.428Z [INFO] server-59aa635f-b628-8c3b-a114-85b40f48fcde: added peer, starting replication: peer=server-1b737291-18a7-7c2d-ebae-e785e2af3a96 2026-04-14T14:14:38.428Z [INFO] server-59aa635f-b628-8c3b-a114-85b40f48fcde: pipelining replication: peer="{Voter server-4e4a7d12-e499-cd4b-0164-a2979f99a41c 4e4a7d12-e499-cd4b-0164-a2979f99a41c}" 2026-04-14T14:14:38.429Z [INFO] server-59aa635f-b628-8c3b-a114-85b40f48fcde: pipelining replication: peer="{Voter server-1b737291-18a7-7c2d-ebae-e785e2af3a96 1b737291-18a7-7c2d-ebae-e785e2af3a96}" 2026-04-14T14:14:38.634Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:38.428944258 +0000 UTC m=+9.901130028 (1 nodes), 117.911399ms from start of poll, 118.269276ms from cluster start. Timeout at 2026-04-14 14:14:38.633946517 +0000 UTC m=+10.106132282, 205.002254ms after stability 2026-04-14T14:14:38.648Z [INFO] server-59aa635f-b628-8c3b-a114-85b40f48fcde: aborting pipeline replication: peer="{Voter server-1b737291-18a7-7c2d-ebae-e785e2af3a96 1b737291-18a7-7c2d-ebae-e785e2af3a96}" 2026-04-14T14:14:38.648Z [INFO] server-59aa635f-b628-8c3b-a114-85b40f48fcde: aborting pipeline replication: peer="{Voter server-4e4a7d12-e499-cd4b-0164-a2979f99a41c 4e4a7d12-e499-cd4b-0164-a2979f99a41c}" --- PASS: TestRaft_ApplyConcurrent (0.46s) === RUN TestRaft_ApplyConcurrent_Timeout 2026-04-14T14:14:38.772Z [INFO] server-adb654b6-c22d-b78c-1943-ce2426b3d17a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-adb654b6-c22d-b78c-1943-ce2426b3d17a Address:adb654b6-c22d-b78c-1943-ce2426b3d17a}]" 2026-04-14T14:14:38.772Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:38.772Z [INFO] server-adb654b6-c22d-b78c-1943-ce2426b3d17a: entering follower state: follower="Node at adb654b6-c22d-b78c-1943-ce2426b3d17a [Follower]" leader-address= leader-id= 2026-04-14T14:14:38.954Z [WARN] server-adb654b6-c22d-b78c-1943-ce2426b3d17a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:38.954Z [INFO] server-adb654b6-c22d-b78c-1943-ce2426b3d17a: entering candidate state: node="Node at adb654b6-c22d-b78c-1943-ce2426b3d17a [Candidate]" term=2 2026-04-14T14:14:38.954Z [INFO] server-adb654b6-c22d-b78c-1943-ce2426b3d17a: election won: term=2 tally=1 2026-04-14T14:14:38.954Z [INFO] server-adb654b6-c22d-b78c-1943-ce2426b3d17a: entering leader state: leader="Node at adb654b6-c22d-b78c-1943-ce2426b3d17a [Leader]" 2026-04-14T14:14:39.155Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:38.954879377 +0000 UTC m=+10.427065145 (1 nodes), 182.340367ms from start of poll, 182.499266ms from cluster start. Timeout at 2026-04-14 14:14:39.155882234 +0000 UTC m=+10.628067999, 201.002854ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.43s) === RUN TestRaft_JoinNode 2026-04-14T14:14:39.198Z [INFO] server-022f31bf-cd33-c40b-f106-f6a47e43e888: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-022f31bf-cd33-c40b-f106-f6a47e43e888 Address:022f31bf-cd33-c40b-f106-f6a47e43e888} {Suffrage:Voter ID:server-d1d918a8-f18c-1c96-5609-b06d20fdd2f4 Address:d1d918a8-f18c-1c96-5609-b06d20fdd2f4}]" 2026-04-14T14:14:39.198Z [INFO] server-d1d918a8-f18c-1c96-5609-b06d20fdd2f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-022f31bf-cd33-c40b-f106-f6a47e43e888 Address:022f31bf-cd33-c40b-f106-f6a47e43e888} {Suffrage:Voter ID:server-d1d918a8-f18c-1c96-5609-b06d20fdd2f4 Address:d1d918a8-f18c-1c96-5609-b06d20fdd2f4}]" 2026-04-14T14:14:39.198Z [INFO] server-022f31bf-cd33-c40b-f106-f6a47e43e888: entering follower state: follower="Node at 022f31bf-cd33-c40b-f106-f6a47e43e888 [Follower]" leader-address= leader-id= 2026-04-14T14:14:39.198Z [INFO] server-d1d918a8-f18c-1c96-5609-b06d20fdd2f4: entering follower state: follower="Node at d1d918a8-f18c-1c96-5609-b06d20fdd2f4 [Follower]" leader-address= leader-id= 2026-04-14T14:14:39.198Z [INFO] server-f60f42e7-f071-2ce4-d330-a282102d4c66: initial configuration: index=0 servers=[] 2026-04-14T14:14:39.198Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:39.198Z [INFO] server-f60f42e7-f071-2ce4-d330-a282102d4c66: entering follower state: follower="Node at f60f42e7-f071-2ce4-d330-a282102d4c66 [Follower]" leader-address= leader-id= 2026-04-14T14:14:39.286Z [WARN] server-f60f42e7-f071-2ce4-d330-a282102d4c66: no known peers, aborting election 2026-04-14T14:14:39.288Z [WARN] server-022f31bf-cd33-c40b-f106-f6a47e43e888: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:39.289Z [INFO] server-022f31bf-cd33-c40b-f106-f6a47e43e888: entering candidate state: node="Node at 022f31bf-cd33-c40b-f106-f6a47e43e888 [Candidate]" term=2 2026-04-14T14:14:39.289Z [INFO] server-022f31bf-cd33-c40b-f106-f6a47e43e888: election won: term=2 tally=2 2026-04-14T14:14:39.289Z [INFO] server-022f31bf-cd33-c40b-f106-f6a47e43e888: entering leader state: leader="Node at 022f31bf-cd33-c40b-f106-f6a47e43e888 [Leader]" 2026-04-14T14:14:39.289Z [INFO] server-022f31bf-cd33-c40b-f106-f6a47e43e888: added peer, starting replication: peer=server-d1d918a8-f18c-1c96-5609-b06d20fdd2f4 2026-04-14T14:14:39.289Z [INFO] server-022f31bf-cd33-c40b-f106-f6a47e43e888: pipelining replication: peer="{Voter server-d1d918a8-f18c-1c96-5609-b06d20fdd2f4 d1d918a8-f18c-1c96-5609-b06d20fdd2f4}" 2026-04-14T14:14:39.394Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:39.289170583 +0000 UTC m=+10.761356374 (1 nodes), 90.336723ms from start of poll, 90.759499ms from cluster start. Timeout at 2026-04-14 14:14:39.394173938 +0000 UTC m=+10.866359705, 105.003331ms after stability 2026-04-14T14:14:39.394Z [INFO] server-022f31bf-cd33-c40b-f106-f6a47e43e888: updating configuration: command=AddVoter server-id=server-f60f42e7-f071-2ce4-d330-a282102d4c66 server-addr=f60f42e7-f071-2ce4-d330-a282102d4c66 servers="[{Suffrage:Voter ID:server-022f31bf-cd33-c40b-f106-f6a47e43e888 Address:022f31bf-cd33-c40b-f106-f6a47e43e888} {Suffrage:Voter ID:server-d1d918a8-f18c-1c96-5609-b06d20fdd2f4 Address:d1d918a8-f18c-1c96-5609-b06d20fdd2f4} {Suffrage:Voter ID:server-f60f42e7-f071-2ce4-d330-a282102d4c66 Address:f60f42e7-f071-2ce4-d330-a282102d4c66}]" 2026-04-14T14:14:39.394Z [INFO] server-022f31bf-cd33-c40b-f106-f6a47e43e888: added peer, starting replication: peer=server-f60f42e7-f071-2ce4-d330-a282102d4c66 2026-04-14T14:14:39.394Z [WARN] server-f60f42e7-f071-2ce4-d330-a282102d4c66: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2026-04-14T14:14:39.394Z [WARN] server-022f31bf-cd33-c40b-f106-f6a47e43e888: appendEntries rejected, sending older logs: peer="{Voter server-f60f42e7-f071-2ce4-d330-a282102d4c66 f60f42e7-f071-2ce4-d330-a282102d4c66}" next=1 2026-04-14T14:14:39.394Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:39.394Z [INFO] server-022f31bf-cd33-c40b-f106-f6a47e43e888: pipelining replication: peer="{Voter server-f60f42e7-f071-2ce4-d330-a282102d4c66 f60f42e7-f071-2ce4-d330-a282102d4c66}" 2026-04-14T14:14:39.500Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:39.39487086 +0000 UTC m=+10.867056628 (1 nodes), 621ns from start of poll, 196.459753ms from cluster start. Timeout at 2026-04-14 14:14:39.499875028 +0000 UTC m=+10.972060793, 105.004165ms after stability 2026-04-14T14:14:39.500Z [INFO] server-022f31bf-cd33-c40b-f106-f6a47e43e888: aborting pipeline replication: peer="{Voter server-f60f42e7-f071-2ce4-d330-a282102d4c66 f60f42e7-f071-2ce4-d330-a282102d4c66}" 2026-04-14T14:14:39.500Z [INFO] server-022f31bf-cd33-c40b-f106-f6a47e43e888: aborting pipeline replication: peer="{Voter server-d1d918a8-f18c-1c96-5609-b06d20fdd2f4 d1d918a8-f18c-1c96-5609-b06d20fdd2f4}" --- PASS: TestRaft_JoinNode (0.30s) === RUN TestRaft_JoinNode_ConfigStore 2026-04-14T14:14:39.502Z [INFO] server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0 Address:f419cfcf-0b0c-182e-53ec-bbcd8c9940e0}]" 2026-04-14T14:14:39.502Z [INFO] server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0: entering follower state: follower="Node at f419cfcf-0b0c-182e-53ec-bbcd8c9940e0 [Follower]" leader-address= leader-id= 2026-04-14T14:14:39.502Z [INFO] server-6951c8cb-3834-e670-7c7a-e7655390be86: initial configuration: index=0 servers=[] 2026-04-14T14:14:39.502Z [INFO] server-6951c8cb-3834-e670-7c7a-e7655390be86: entering follower state: follower="Node at 6951c8cb-3834-e670-7c7a-e7655390be86 [Follower]" leader-address= leader-id= 2026-04-14T14:14:39.502Z [INFO] server-2b72a090-3083-c566-7d01-f0b638417b7e: initial configuration: index=0 servers=[] 2026-04-14T14:14:39.502Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:39.502Z [INFO] server-2b72a090-3083-c566-7d01-f0b638417b7e: entering follower state: follower="Node at 2b72a090-3083-c566-7d01-f0b638417b7e [Follower]" leader-address= leader-id= 2026-04-14T14:14:39.552Z [WARN] server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:39.552Z [INFO] server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0: entering candidate state: node="Node at f419cfcf-0b0c-182e-53ec-bbcd8c9940e0 [Candidate]" term=2 2026-04-14T14:14:39.552Z [INFO] server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0: election won: term=2 tally=1 2026-04-14T14:14:39.552Z [INFO] server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0: entering leader state: leader="Node at f419cfcf-0b0c-182e-53ec-bbcd8c9940e0 [Leader]" 2026-04-14T14:14:39.569Z [WARN] server-2b72a090-3083-c566-7d01-f0b638417b7e: no known peers, aborting election 2026-04-14T14:14:39.589Z [WARN] server-6951c8cb-3834-e670-7c7a-e7655390be86: no known peers, aborting election 2026-04-14T14:14:39.658Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:39.553023168 +0000 UTC m=+11.025208934 (1 nodes), 50.380359ms from start of poll, 51.109717ms from cluster start. Timeout at 2026-04-14 14:14:39.658024069 +0000 UTC m=+11.130209836, 105.000902ms after stability 2026-04-14T14:14:39.658Z [INFO] server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0: updating configuration: command=AddVoter server-id=server-6951c8cb-3834-e670-7c7a-e7655390be86 server-addr=6951c8cb-3834-e670-7c7a-e7655390be86 servers="[{Suffrage:Voter ID:server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0 Address:f419cfcf-0b0c-182e-53ec-bbcd8c9940e0} {Suffrage:Voter ID:server-6951c8cb-3834-e670-7c7a-e7655390be86 Address:6951c8cb-3834-e670-7c7a-e7655390be86}]" 2026-04-14T14:14:39.658Z [INFO] server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0: added peer, starting replication: peer=server-6951c8cb-3834-e670-7c7a-e7655390be86 2026-04-14T14:14:39.658Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:39.658Z [WARN] server-6951c8cb-3834-e670-7c7a-e7655390be86: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2026-04-14T14:14:39.658Z [WARN] server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0: appendEntries rejected, sending older logs: peer="{Voter server-6951c8cb-3834-e670-7c7a-e7655390be86 6951c8cb-3834-e670-7c7a-e7655390be86}" next=1 2026-04-14T14:14:39.658Z [INFO] server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0: pipelining replication: peer="{Voter server-6951c8cb-3834-e670-7c7a-e7655390be86 6951c8cb-3834-e670-7c7a-e7655390be86}" 2026-04-14T14:14:39.764Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:39.658406084 +0000 UTC m=+11.130591848 (1 nodes), 742ns from start of poll, 156.492631ms from cluster start. Timeout at 2026-04-14 14:14:39.763412135 +0000 UTC m=+11.235597904, 105.006056ms after stability 2026-04-14T14:14:39.764Z [INFO] server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0: updating configuration: command=AddVoter server-id=server-2b72a090-3083-c566-7d01-f0b638417b7e server-addr=2b72a090-3083-c566-7d01-f0b638417b7e servers="[{Suffrage:Voter ID:server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0 Address:f419cfcf-0b0c-182e-53ec-bbcd8c9940e0} {Suffrage:Voter ID:server-6951c8cb-3834-e670-7c7a-e7655390be86 Address:6951c8cb-3834-e670-7c7a-e7655390be86} {Suffrage:Voter ID:server-2b72a090-3083-c566-7d01-f0b638417b7e Address:2b72a090-3083-c566-7d01-f0b638417b7e}]" 2026-04-14T14:14:39.764Z [INFO] server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0: added peer, starting replication: peer=server-2b72a090-3083-c566-7d01-f0b638417b7e 2026-04-14T14:14:39.764Z [WARN] server-2b72a090-3083-c566-7d01-f0b638417b7e: failed to get previous log: previous-index=4 last-index=0 error="log not found" 2026-04-14T14:14:39.764Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:39.764Z [WARN] server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0: appendEntries rejected, sending older logs: peer="{Voter server-2b72a090-3083-c566-7d01-f0b638417b7e 2b72a090-3083-c566-7d01-f0b638417b7e}" next=1 2026-04-14T14:14:39.764Z [INFO] server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0: pipelining replication: peer="{Voter server-2b72a090-3083-c566-7d01-f0b638417b7e 2b72a090-3083-c566-7d01-f0b638417b7e}" 2026-04-14T14:14:39.870Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:39.76463955 +0000 UTC m=+11.236825321 (1 nodes), 850ns from start of poll, 262.726104ms from cluster start. Timeout at 2026-04-14 14:14:39.869642334 +0000 UTC m=+11.341828101, 105.00278ms after stability 2026-04-14T14:14:39.870Z [INFO] server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0: aborting pipeline replication: peer="{Voter server-2b72a090-3083-c566-7d01-f0b638417b7e 2b72a090-3083-c566-7d01-f0b638417b7e}" 2026-04-14T14:14:39.870Z [INFO] server-f419cfcf-0b0c-182e-53ec-bbcd8c9940e0: aborting pipeline replication: peer="{Voter server-6951c8cb-3834-e670-7c7a-e7655390be86 6951c8cb-3834-e670-7c7a-e7655390be86}" --- PASS: TestRaft_JoinNode_ConfigStore (0.37s) === RUN TestRaft_RemoveFollower 2026-04-14T14:14:39.871Z [INFO] server-5b7353c3-e980-53d3-9d04-3ad2bfb33b99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5b7353c3-e980-53d3-9d04-3ad2bfb33b99 Address:5b7353c3-e980-53d3-9d04-3ad2bfb33b99} {Suffrage:Voter ID:server-67cefa96-ff7f-0643-9734-9770bf1b3501 Address:67cefa96-ff7f-0643-9734-9770bf1b3501} {Suffrage:Voter ID:server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3 Address:a3df33d3-2ac1-410e-a5e8-1238697fd7e3}]" 2026-04-14T14:14:39.871Z [INFO] server-5b7353c3-e980-53d3-9d04-3ad2bfb33b99: entering follower state: follower="Node at 5b7353c3-e980-53d3-9d04-3ad2bfb33b99 [Follower]" leader-address= leader-id= 2026-04-14T14:14:39.872Z [INFO] server-67cefa96-ff7f-0643-9734-9770bf1b3501: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5b7353c3-e980-53d3-9d04-3ad2bfb33b99 Address:5b7353c3-e980-53d3-9d04-3ad2bfb33b99} {Suffrage:Voter ID:server-67cefa96-ff7f-0643-9734-9770bf1b3501 Address:67cefa96-ff7f-0643-9734-9770bf1b3501} {Suffrage:Voter ID:server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3 Address:a3df33d3-2ac1-410e-a5e8-1238697fd7e3}]" 2026-04-14T14:14:39.872Z [INFO] server-67cefa96-ff7f-0643-9734-9770bf1b3501: entering follower state: follower="Node at 67cefa96-ff7f-0643-9734-9770bf1b3501 [Follower]" leader-address= leader-id= 2026-04-14T14:14:39.872Z [INFO] server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5b7353c3-e980-53d3-9d04-3ad2bfb33b99 Address:5b7353c3-e980-53d3-9d04-3ad2bfb33b99} {Suffrage:Voter ID:server-67cefa96-ff7f-0643-9734-9770bf1b3501 Address:67cefa96-ff7f-0643-9734-9770bf1b3501} {Suffrage:Voter ID:server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3 Address:a3df33d3-2ac1-410e-a5e8-1238697fd7e3}]" 2026-04-14T14:14:39.872Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:39.872Z [INFO] server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3: entering follower state: follower="Node at a3df33d3-2ac1-410e-a5e8-1238697fd7e3 [Follower]" leader-address= leader-id= 2026-04-14T14:14:39.944Z [WARN] server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:39.944Z [INFO] server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3: entering candidate state: node="Node at a3df33d3-2ac1-410e-a5e8-1238697fd7e3 [Candidate]" term=2 2026-04-14T14:14:39.944Z [INFO] server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3: election won: term=2 tally=2 2026-04-14T14:14:39.944Z [INFO] server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3: entering leader state: leader="Node at a3df33d3-2ac1-410e-a5e8-1238697fd7e3 [Leader]" 2026-04-14T14:14:39.944Z [INFO] server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3: added peer, starting replication: peer=server-5b7353c3-e980-53d3-9d04-3ad2bfb33b99 2026-04-14T14:14:39.944Z [INFO] server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3: added peer, starting replication: peer=server-67cefa96-ff7f-0643-9734-9770bf1b3501 2026-04-14T14:14:39.944Z [INFO] server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3: pipelining replication: peer="{Voter server-67cefa96-ff7f-0643-9734-9770bf1b3501 67cefa96-ff7f-0643-9734-9770bf1b3501}" 2026-04-14T14:14:39.944Z [INFO] server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3: pipelining replication: peer="{Voter server-5b7353c3-e980-53d3-9d04-3ad2bfb33b99 5b7353c3-e980-53d3-9d04-3ad2bfb33b99}" 2026-04-14T14:14:40.050Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:39.944665563 +0000 UTC m=+11.416851361 (1 nodes), 72.206256ms from start of poll, 73.516469ms from cluster start. Timeout at 2026-04-14 14:14:40.04966887 +0000 UTC m=+11.521854638, 105.003277ms after stability 2026-04-14T14:14:40.055Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:14:40.160Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:14:40.055719393 +0000 UTC m=+11.527905161 (2 nodes), 1.215µs from start of poll, 184.570269ms from cluster start. Timeout at 2026-04-14 14:14:40.160723646 +0000 UTC m=+11.632909414, 105.004253ms after stability 2026-04-14T14:14:40.160Z [INFO] server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3: updating configuration: command=RemoveServer server-id=server-5b7353c3-e980-53d3-9d04-3ad2bfb33b99 server-addr= servers="[{Suffrage:Voter ID:server-67cefa96-ff7f-0643-9734-9770bf1b3501 Address:67cefa96-ff7f-0643-9734-9770bf1b3501} {Suffrage:Voter ID:server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3 Address:a3df33d3-2ac1-410e-a5e8-1238697fd7e3}]" 2026-04-14T14:14:40.160Z [INFO] server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3: removed peer, stopping replication: peer=server-5b7353c3-e980-53d3-9d04-3ad2bfb33b99 last-index=3 2026-04-14T14:14:40.160Z [INFO] server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3: aborting pipeline replication: peer="{Voter server-5b7353c3-e980-53d3-9d04-3ad2bfb33b99 5b7353c3-e980-53d3-9d04-3ad2bfb33b99}" 2026-04-14T14:14:40.266Z [INFO] server-a3df33d3-2ac1-410e-a5e8-1238697fd7e3: aborting pipeline replication: peer="{Voter server-67cefa96-ff7f-0643-9734-9770bf1b3501 67cefa96-ff7f-0643-9734-9770bf1b3501}" --- PASS: TestRaft_RemoveFollower (0.40s) === RUN TestRaft_RemoveLeader 2026-04-14T14:14:40.267Z [INFO] server-43073cab-ae60-019d-1b05-cffdfe7b87f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-43073cab-ae60-019d-1b05-cffdfe7b87f2 Address:43073cab-ae60-019d-1b05-cffdfe7b87f2} {Suffrage:Voter ID:server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0 Address:a9a40df5-2072-7fc7-4eff-ca192a2c17f0} {Suffrage:Voter ID:server-34c3f0b5-059c-eb26-9c53-d738a77600dc Address:34c3f0b5-059c-eb26-9c53-d738a77600dc}]" 2026-04-14T14:14:40.267Z [INFO] server-43073cab-ae60-019d-1b05-cffdfe7b87f2: entering follower state: follower="Node at 43073cab-ae60-019d-1b05-cffdfe7b87f2 [Follower]" leader-address= leader-id= 2026-04-14T14:14:40.267Z [INFO] server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-43073cab-ae60-019d-1b05-cffdfe7b87f2 Address:43073cab-ae60-019d-1b05-cffdfe7b87f2} {Suffrage:Voter ID:server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0 Address:a9a40df5-2072-7fc7-4eff-ca192a2c17f0} {Suffrage:Voter ID:server-34c3f0b5-059c-eb26-9c53-d738a77600dc Address:34c3f0b5-059c-eb26-9c53-d738a77600dc}]" 2026-04-14T14:14:40.267Z [INFO] server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0: entering follower state: follower="Node at a9a40df5-2072-7fc7-4eff-ca192a2c17f0 [Follower]" leader-address= leader-id= 2026-04-14T14:14:40.267Z [INFO] server-34c3f0b5-059c-eb26-9c53-d738a77600dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-43073cab-ae60-019d-1b05-cffdfe7b87f2 Address:43073cab-ae60-019d-1b05-cffdfe7b87f2} {Suffrage:Voter ID:server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0 Address:a9a40df5-2072-7fc7-4eff-ca192a2c17f0} {Suffrage:Voter ID:server-34c3f0b5-059c-eb26-9c53-d738a77600dc Address:34c3f0b5-059c-eb26-9c53-d738a77600dc}]" 2026-04-14T14:14:40.268Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:40.268Z [INFO] server-34c3f0b5-059c-eb26-9c53-d738a77600dc: entering follower state: follower="Node at 34c3f0b5-059c-eb26-9c53-d738a77600dc [Follower]" leader-address= leader-id= 2026-04-14T14:14:40.320Z [WARN] server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:40.320Z [INFO] server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0: entering candidate state: node="Node at a9a40df5-2072-7fc7-4eff-ca192a2c17f0 [Candidate]" term=2 2026-04-14T14:14:40.320Z [INFO] server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0: election won: term=2 tally=2 2026-04-14T14:14:40.320Z [INFO] server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0: entering leader state: leader="Node at a9a40df5-2072-7fc7-4eff-ca192a2c17f0 [Leader]" 2026-04-14T14:14:40.320Z [INFO] server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0: added peer, starting replication: peer=server-43073cab-ae60-019d-1b05-cffdfe7b87f2 2026-04-14T14:14:40.320Z [INFO] server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0: added peer, starting replication: peer=server-34c3f0b5-059c-eb26-9c53-d738a77600dc 2026-04-14T14:14:40.320Z [INFO] server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0: pipelining replication: peer="{Voter server-34c3f0b5-059c-eb26-9c53-d738a77600dc 34c3f0b5-059c-eb26-9c53-d738a77600dc}" 2026-04-14T14:14:40.320Z [INFO] server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0: pipelining replication: peer="{Voter server-43073cab-ae60-019d-1b05-cffdfe7b87f2 43073cab-ae60-019d-1b05-cffdfe7b87f2}" 2026-04-14T14:14:40.426Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:40.320600017 +0000 UTC m=+11.792785786 (1 nodes), 52.005434ms from start of poll, 53.101017ms from cluster start. Timeout at 2026-04-14 14:14:40.425602362 +0000 UTC m=+11.897788130, 105.002344ms after stability 2026-04-14T14:14:40.432Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:14:40.538Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:14:40.432123683 +0000 UTC m=+11.904309452 (2 nodes), 3.087µs from start of poll, 164.624683ms from cluster start. Timeout at 2026-04-14 14:14:40.53712653 +0000 UTC m=+12.009312298, 105.002846ms after stability 2026-04-14T14:14:40.538Z [INFO] server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0: updating configuration: command=RemoveServer server-id=server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0 server-addr= servers="[{Suffrage:Voter ID:server-43073cab-ae60-019d-1b05-cffdfe7b87f2 Address:43073cab-ae60-019d-1b05-cffdfe7b87f2} {Suffrage:Voter ID:server-34c3f0b5-059c-eb26-9c53-d738a77600dc Address:34c3f0b5-059c-eb26-9c53-d738a77600dc}]" 2026-04-14T14:14:40.538Z [INFO] server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0: removed ourself, shutting down 2026-04-14T14:14:40.538Z [INFO] server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0: aborting pipeline replication: peer="{Voter server-34c3f0b5-059c-eb26-9c53-d738a77600dc 34c3f0b5-059c-eb26-9c53-d738a77600dc}" 2026-04-14T14:14:40.538Z [INFO] server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0: aborting pipeline replication: peer="{Voter server-43073cab-ae60-019d-1b05-cffdfe7b87f2 43073cab-ae60-019d-1b05-cffdfe7b87f2}" 2026-04-14T14:14:40.608Z [WARN] server-34c3f0b5-059c-eb26-9c53-d738a77600dc: heartbeat timeout reached, starting election: last-leader-addr=a9a40df5-2072-7fc7-4eff-ca192a2c17f0 last-leader-id=server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0 2026-04-14T14:14:40.608Z [INFO] server-34c3f0b5-059c-eb26-9c53-d738a77600dc: entering candidate state: node="Node at 34c3f0b5-059c-eb26-9c53-d738a77600dc [Candidate]" term=3 2026-04-14T14:14:40.608Z [WARN] server-43073cab-ae60-019d-1b05-cffdfe7b87f2: rejecting vote request since we have a leader: from=34c3f0b5-059c-eb26-9c53-d738a77600dc leader=a9a40df5-2072-7fc7-4eff-ca192a2c17f0 leader-id=server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0 2026-04-14T14:14:40.630Z [WARN] server-43073cab-ae60-019d-1b05-cffdfe7b87f2: heartbeat timeout reached, starting election: last-leader-addr=a9a40df5-2072-7fc7-4eff-ca192a2c17f0 last-leader-id=server-a9a40df5-2072-7fc7-4eff-ca192a2c17f0 2026-04-14T14:14:40.630Z [INFO] server-43073cab-ae60-019d-1b05-cffdfe7b87f2: entering candidate state: node="Node at 43073cab-ae60-019d-1b05-cffdfe7b87f2 [Candidate]" term=3 2026-04-14T14:14:40.630Z [INFO] server-34c3f0b5-059c-eb26-9c53-d738a77600dc: duplicate requestVote for same term: term=3 2026-04-14T14:14:40.691Z [WARN] server-43073cab-ae60-019d-1b05-cffdfe7b87f2: Election timeout reached, restarting election 2026-04-14T14:14:40.691Z [INFO] server-43073cab-ae60-019d-1b05-cffdfe7b87f2: entering candidate state: node="Node at 43073cab-ae60-019d-1b05-cffdfe7b87f2 [Candidate]" term=4 2026-04-14T14:14:40.691Z [INFO] server-34c3f0b5-059c-eb26-9c53-d738a77600dc: entering follower state: follower="Node at 34c3f0b5-059c-eb26-9c53-d738a77600dc [Follower]" leader-address= leader-id= 2026-04-14T14:14:40.691Z [INFO] server-43073cab-ae60-019d-1b05-cffdfe7b87f2: election won: term=4 tally=2 2026-04-14T14:14:40.691Z [INFO] server-43073cab-ae60-019d-1b05-cffdfe7b87f2: entering leader state: leader="Node at 43073cab-ae60-019d-1b05-cffdfe7b87f2 [Leader]" 2026-04-14T14:14:40.692Z [INFO] server-43073cab-ae60-019d-1b05-cffdfe7b87f2: added peer, starting replication: peer=server-34c3f0b5-059c-eb26-9c53-d738a77600dc 2026-04-14T14:14:40.692Z [INFO] server-43073cab-ae60-019d-1b05-cffdfe7b87f2: pipelining replication: peer="{Voter server-34c3f0b5-059c-eb26-9c53-d738a77600dc 34c3f0b5-059c-eb26-9c53-d738a77600dc}" 2026-04-14T14:14:40.750Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:40.855Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:40.750267284 +0000 UTC m=+12.222453055 (1 nodes), 44.31µs from start of poll, 482.768286ms from cluster start. Timeout at 2026-04-14 14:14:40.855269499 +0000 UTC m=+12.327455268, 105.002213ms after stability 2026-04-14T14:14:40.855Z [INFO] server-43073cab-ae60-019d-1b05-cffdfe7b87f2: aborting pipeline replication: peer="{Voter server-34c3f0b5-059c-eb26-9c53-d738a77600dc 34c3f0b5-059c-eb26-9c53-d738a77600dc}" --- PASS: TestRaft_RemoveLeader (0.59s) === RUN TestRaft_RemoveLeader_NoShutdown 2026-04-14T14:14:40.856Z [INFO] server-21d1be7a-ef84-3ede-56bc-c3b787f0213b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21d1be7a-ef84-3ede-56bc-c3b787f0213b Address:21d1be7a-ef84-3ede-56bc-c3b787f0213b} {Suffrage:Voter ID:server-6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6 Address:6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6} {Suffrage:Voter ID:server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201 Address:2d160fd3-b8d2-dced-efdf-c9e5dbb27201}]" 2026-04-14T14:14:40.856Z [INFO] server-21d1be7a-ef84-3ede-56bc-c3b787f0213b: entering follower state: follower="Node at 21d1be7a-ef84-3ede-56bc-c3b787f0213b [Follower]" leader-address= leader-id= 2026-04-14T14:14:40.856Z [INFO] server-6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21d1be7a-ef84-3ede-56bc-c3b787f0213b Address:21d1be7a-ef84-3ede-56bc-c3b787f0213b} {Suffrage:Voter ID:server-6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6 Address:6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6} {Suffrage:Voter ID:server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201 Address:2d160fd3-b8d2-dced-efdf-c9e5dbb27201}]" 2026-04-14T14:14:40.856Z [INFO] server-6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6: entering follower state: follower="Node at 6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6 [Follower]" leader-address= leader-id= 2026-04-14T14:14:40.856Z [INFO] server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21d1be7a-ef84-3ede-56bc-c3b787f0213b Address:21d1be7a-ef84-3ede-56bc-c3b787f0213b} {Suffrage:Voter ID:server-6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6 Address:6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6} {Suffrage:Voter ID:server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201 Address:2d160fd3-b8d2-dced-efdf-c9e5dbb27201}]" 2026-04-14T14:14:40.856Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:14:40.856Z [INFO] server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201: entering follower state: follower="Node at 2d160fd3-b8d2-dced-efdf-c9e5dbb27201 [Follower]" leader-address= leader-id= 2026-04-14T14:14:40.932Z [WARN] server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:40.932Z [INFO] server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201: entering candidate state: node="Node at 2d160fd3-b8d2-dced-efdf-c9e5dbb27201 [Candidate]" term=2 2026-04-14T14:14:40.932Z [INFO] server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201: election won: term=2 tally=2 2026-04-14T14:14:40.932Z [INFO] server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201: entering leader state: leader="Node at 2d160fd3-b8d2-dced-efdf-c9e5dbb27201 [Leader]" 2026-04-14T14:14:40.932Z [INFO] server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201: added peer, starting replication: peer=server-21d1be7a-ef84-3ede-56bc-c3b787f0213b 2026-04-14T14:14:40.932Z [INFO] server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201: added peer, starting replication: peer=server-6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6 2026-04-14T14:14:40.932Z [INFO] server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201: pipelining replication: peer="{Voter server-6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6 6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6}" 2026-04-14T14:14:40.932Z [INFO] server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201: pipelining replication: peer="{Voter server-21d1be7a-ef84-3ede-56bc-c3b787f0213b 21d1be7a-ef84-3ede-56bc-c3b787f0213b}" 2026-04-14T14:14:41.037Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:14:40.932194535 +0000 UTC m=+12.404380299 (2 nodes), 75.301392ms from start of poll, 75.6705ms from cluster start. Timeout at 2026-04-14 14:14:41.037197086 +0000 UTC m=+12.509382853, 105.002554ms after stability 2026-04-14T14:14:41.037Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:41.142Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:41.037558833 +0000 UTC m=+12.509744603 (1 nodes), 2.875µs from start of poll, 181.034804ms from cluster start. Timeout at 2026-04-14 14:14:41.142563573 +0000 UTC m=+12.614749343, 105.00474ms after stability 2026-04-14T14:14:41.146Z [INFO] server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201: updating configuration: command=RemoveServer server-id=server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201 server-addr= servers="[{Suffrage:Voter ID:server-21d1be7a-ef84-3ede-56bc-c3b787f0213b Address:21d1be7a-ef84-3ede-56bc-c3b787f0213b} {Suffrage:Voter ID:server-6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6 Address:6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6}]" 2026-04-14T14:14:41.146Z [INFO] server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201: removed ourself, transitioning to follower 2026-04-14T14:14:41.146Z [INFO] server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201: entering follower state: follower="Node at 2d160fd3-b8d2-dced-efdf-c9e5dbb27201 [Follower]" leader-address= leader-id= 2026-04-14T14:14:41.146Z [INFO] server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201: aborting pipeline replication: peer="{Voter server-6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6 6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6}" 2026-04-14T14:14:41.146Z [INFO] server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201: aborting pipeline replication: peer="{Voter server-21d1be7a-ef84-3ede-56bc-c3b787f0213b 21d1be7a-ef84-3ede-56bc-c3b787f0213b}" 2026-04-14T14:14:41.212Z [WARN] server-21d1be7a-ef84-3ede-56bc-c3b787f0213b: heartbeat timeout reached, starting election: last-leader-addr=2d160fd3-b8d2-dced-efdf-c9e5dbb27201 last-leader-id=server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201 2026-04-14T14:14:41.212Z [INFO] server-21d1be7a-ef84-3ede-56bc-c3b787f0213b: entering candidate state: node="Node at 21d1be7a-ef84-3ede-56bc-c3b787f0213b [Candidate]" term=3 2026-04-14T14:14:41.213Z [WARN] server-6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6: rejecting vote request since we have a leader: from=21d1be7a-ef84-3ede-56bc-c3b787f0213b leader=2d160fd3-b8d2-dced-efdf-c9e5dbb27201 leader-id=server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201 2026-04-14T14:14:41.237Z [WARN] server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201: not part of stable configuration, aborting election 2026-04-14T14:14:41.239Z [WARN] server-6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6: heartbeat timeout reached, starting election: last-leader-addr=2d160fd3-b8d2-dced-efdf-c9e5dbb27201 last-leader-id=server-2d160fd3-b8d2-dced-efdf-c9e5dbb27201 2026-04-14T14:14:41.239Z [INFO] server-6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6: entering candidate state: node="Node at 6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6 [Candidate]" term=3 2026-04-14T14:14:41.239Z [INFO] server-21d1be7a-ef84-3ede-56bc-c3b787f0213b: duplicate requestVote for same term: term=3 2026-04-14T14:14:41.251Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:41.296Z [WARN] server-21d1be7a-ef84-3ede-56bc-c3b787f0213b: Election timeout reached, restarting election 2026-04-14T14:14:41.296Z [INFO] server-21d1be7a-ef84-3ede-56bc-c3b787f0213b: entering candidate state: node="Node at 21d1be7a-ef84-3ede-56bc-c3b787f0213b [Candidate]" term=4 2026-04-14T14:14:41.296Z [INFO] server-6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6: entering follower state: follower="Node at 6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6 [Follower]" leader-address= leader-id= 2026-04-14T14:14:41.296Z [INFO] server-21d1be7a-ef84-3ede-56bc-c3b787f0213b: election won: term=4 tally=2 2026-04-14T14:14:41.296Z [INFO] server-21d1be7a-ef84-3ede-56bc-c3b787f0213b: entering leader state: leader="Node at 21d1be7a-ef84-3ede-56bc-c3b787f0213b [Leader]" 2026-04-14T14:14:41.296Z [INFO] server-21d1be7a-ef84-3ede-56bc-c3b787f0213b: added peer, starting replication: peer=server-6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6 2026-04-14T14:14:41.296Z [INFO] server-21d1be7a-ef84-3ede-56bc-c3b787f0213b: pipelining replication: peer="{Voter server-6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6 6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6}" 2026-04-14T14:14:41.401Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:41.296499911 +0000 UTC m=+12.768685682 (1 nodes), 44.615403ms from start of poll, 439.975883ms from cluster start. Timeout at 2026-04-14 14:14:41.40150232 +0000 UTC m=+12.873688084, 105.002402ms after stability 2026-04-14T14:14:41.507Z [INFO] server-21d1be7a-ef84-3ede-56bc-c3b787f0213b: aborting pipeline replication: peer="{Voter server-6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6 6dcf65ee-e9c9-dd69-53e4-ebf5a93066b6}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.65s) === RUN TestRaft_RemoveFollower_SplitCluster 2026-04-14T14:14:41.508Z [INFO] server-e652b523-8f7d-0b03-40ad-3fc1f4b4d576: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e652b523-8f7d-0b03-40ad-3fc1f4b4d576 Address:e652b523-8f7d-0b03-40ad-3fc1f4b4d576} {Suffrage:Voter ID:server-f0808778-e412-11d8-9a42-d0953571b1c1 Address:f0808778-e412-11d8-9a42-d0953571b1c1} {Suffrage:Voter ID:server-be030a87-cccc-adc2-cfac-864b5da59931 Address:be030a87-cccc-adc2-cfac-864b5da59931} {Suffrage:Voter ID:server-0f9c8e0e-f37c-1349-cfaf-7f0a7453846d Address:0f9c8e0e-f37c-1349-cfaf-7f0a7453846d}]" 2026-04-14T14:14:41.508Z [INFO] server-e652b523-8f7d-0b03-40ad-3fc1f4b4d576: entering follower state: follower="Node at e652b523-8f7d-0b03-40ad-3fc1f4b4d576 [Follower]" leader-address= leader-id= 2026-04-14T14:14:41.508Z [INFO] server-f0808778-e412-11d8-9a42-d0953571b1c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e652b523-8f7d-0b03-40ad-3fc1f4b4d576 Address:e652b523-8f7d-0b03-40ad-3fc1f4b4d576} {Suffrage:Voter ID:server-f0808778-e412-11d8-9a42-d0953571b1c1 Address:f0808778-e412-11d8-9a42-d0953571b1c1} {Suffrage:Voter ID:server-be030a87-cccc-adc2-cfac-864b5da59931 Address:be030a87-cccc-adc2-cfac-864b5da59931} {Suffrage:Voter ID:server-0f9c8e0e-f37c-1349-cfaf-7f0a7453846d Address:0f9c8e0e-f37c-1349-cfaf-7f0a7453846d}]" 2026-04-14T14:14:41.508Z [INFO] server-f0808778-e412-11d8-9a42-d0953571b1c1: entering follower state: follower="Node at f0808778-e412-11d8-9a42-d0953571b1c1 [Follower]" leader-address= leader-id= 2026-04-14T14:14:41.508Z [INFO] server-be030a87-cccc-adc2-cfac-864b5da59931: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e652b523-8f7d-0b03-40ad-3fc1f4b4d576 Address:e652b523-8f7d-0b03-40ad-3fc1f4b4d576} {Suffrage:Voter ID:server-f0808778-e412-11d8-9a42-d0953571b1c1 Address:f0808778-e412-11d8-9a42-d0953571b1c1} {Suffrage:Voter ID:server-be030a87-cccc-adc2-cfac-864b5da59931 Address:be030a87-cccc-adc2-cfac-864b5da59931} {Suffrage:Voter ID:server-0f9c8e0e-f37c-1349-cfaf-7f0a7453846d Address:0f9c8e0e-f37c-1349-cfaf-7f0a7453846d}]" 2026-04-14T14:14:41.508Z [INFO] server-be030a87-cccc-adc2-cfac-864b5da59931: entering follower state: follower="Node at be030a87-cccc-adc2-cfac-864b5da59931 [Follower]" leader-address= leader-id= 2026-04-14T14:14:41.508Z [INFO] server-0f9c8e0e-f37c-1349-cfaf-7f0a7453846d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e652b523-8f7d-0b03-40ad-3fc1f4b4d576 Address:e652b523-8f7d-0b03-40ad-3fc1f4b4d576} {Suffrage:Voter ID:server-f0808778-e412-11d8-9a42-d0953571b1c1 Address:f0808778-e412-11d8-9a42-d0953571b1c1} {Suffrage:Voter ID:server-be030a87-cccc-adc2-cfac-864b5da59931 Address:be030a87-cccc-adc2-cfac-864b5da59931} {Suffrage:Voter ID:server-0f9c8e0e-f37c-1349-cfaf-7f0a7453846d Address:0f9c8e0e-f37c-1349-cfaf-7f0a7453846d}]" 2026-04-14T14:14:41.509Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:41.509Z [INFO] server-0f9c8e0e-f37c-1349-cfaf-7f0a7453846d: entering follower state: follower="Node at 0f9c8e0e-f37c-1349-cfaf-7f0a7453846d [Follower]" leader-address= leader-id= 2026-04-14T14:14:41.581Z [WARN] server-be030a87-cccc-adc2-cfac-864b5da59931: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:41.581Z [INFO] server-be030a87-cccc-adc2-cfac-864b5da59931: entering candidate state: node="Node at be030a87-cccc-adc2-cfac-864b5da59931 [Candidate]" term=2 2026-04-14T14:14:41.581Z [INFO] server-be030a87-cccc-adc2-cfac-864b5da59931: election won: term=2 tally=3 2026-04-14T14:14:41.581Z [INFO] server-be030a87-cccc-adc2-cfac-864b5da59931: entering leader state: leader="Node at be030a87-cccc-adc2-cfac-864b5da59931 [Leader]" 2026-04-14T14:14:41.581Z [INFO] server-be030a87-cccc-adc2-cfac-864b5da59931: added peer, starting replication: peer=server-e652b523-8f7d-0b03-40ad-3fc1f4b4d576 2026-04-14T14:14:41.581Z [INFO] server-be030a87-cccc-adc2-cfac-864b5da59931: added peer, starting replication: peer=server-f0808778-e412-11d8-9a42-d0953571b1c1 2026-04-14T14:14:41.581Z [INFO] server-be030a87-cccc-adc2-cfac-864b5da59931: added peer, starting replication: peer=server-0f9c8e0e-f37c-1349-cfaf-7f0a7453846d 2026-04-14T14:14:41.582Z [INFO] server-be030a87-cccc-adc2-cfac-864b5da59931: pipelining replication: peer="{Voter server-0f9c8e0e-f37c-1349-cfaf-7f0a7453846d 0f9c8e0e-f37c-1349-cfaf-7f0a7453846d}" 2026-04-14T14:14:41.582Z [INFO] server-be030a87-cccc-adc2-cfac-864b5da59931: pipelining replication: peer="{Voter server-e652b523-8f7d-0b03-40ad-3fc1f4b4d576 e652b523-8f7d-0b03-40ad-3fc1f4b4d576}" 2026-04-14T14:14:41.582Z [INFO] server-be030a87-cccc-adc2-cfac-864b5da59931: pipelining replication: peer="{Voter server-f0808778-e412-11d8-9a42-d0953571b1c1 f0808778-e412-11d8-9a42-d0953571b1c1}" 2026-04-14T14:14:41.686Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:41.581647824 +0000 UTC m=+13.053833605 (1 nodes), 72.604382ms from start of poll, 73.266338ms from cluster start. Timeout at 2026-04-14 14:14:41.686650279 +0000 UTC m=+13.158836043, 105.002438ms after stability 2026-04-14T14:14:41.792Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:14:41.897Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:14:41.79239696 +0000 UTC m=+13.264582727 (3 nodes), 2.448µs from start of poll, 284.01546ms from cluster start. Timeout at 2026-04-14 14:14:41.897471556 +0000 UTC m=+13.369657324, 105.074597ms after stability 2026-04-14T14:14:41.897Z [INFO] server-be030a87-cccc-adc2-cfac-864b5da59931: updating configuration: command=RemoveServer server-id=server-0f9c8e0e-f37c-1349-cfaf-7f0a7453846d server-addr= servers="[{Suffrage:Voter ID:server-e652b523-8f7d-0b03-40ad-3fc1f4b4d576 Address:e652b523-8f7d-0b03-40ad-3fc1f4b4d576} {Suffrage:Voter ID:server-f0808778-e412-11d8-9a42-d0953571b1c1 Address:f0808778-e412-11d8-9a42-d0953571b1c1} {Suffrage:Voter ID:server-be030a87-cccc-adc2-cfac-864b5da59931 Address:be030a87-cccc-adc2-cfac-864b5da59931}]" 2026-04-14T14:14:41.898Z [INFO] server-be030a87-cccc-adc2-cfac-864b5da59931: removed peer, stopping replication: peer=server-0f9c8e0e-f37c-1349-cfaf-7f0a7453846d last-index=3 2026-04-14T14:14:41.898Z [ERROR] server-be030a87-cccc-adc2-cfac-864b5da59931: failed to pipeline appendEntries: peer="{Voter server-e652b523-8f7d-0b03-40ad-3fc1f4b4d576 e652b523-8f7d-0b03-40ad-3fc1f4b4d576}" error="append pipeline closed" 2026-04-14T14:14:41.898Z [ERROR] server-be030a87-cccc-adc2-cfac-864b5da59931: failed to pipeline appendEntries: peer="{Voter server-f0808778-e412-11d8-9a42-d0953571b1c1 f0808778-e412-11d8-9a42-d0953571b1c1}" error="append pipeline closed" 2026-04-14T14:14:41.898Z [INFO] server-be030a87-cccc-adc2-cfac-864b5da59931: aborting pipeline replication: peer="{Voter server-f0808778-e412-11d8-9a42-d0953571b1c1 f0808778-e412-11d8-9a42-d0953571b1c1}" 2026-04-14T14:14:41.898Z [INFO] server-be030a87-cccc-adc2-cfac-864b5da59931: aborting pipeline replication: peer="{Voter server-0f9c8e0e-f37c-1349-cfaf-7f0a7453846d 0f9c8e0e-f37c-1349-cfaf-7f0a7453846d}" 2026-04-14T14:14:41.898Z [INFO] server-be030a87-cccc-adc2-cfac-864b5da59931: aborting pipeline replication: peer="{Voter server-e652b523-8f7d-0b03-40ad-3fc1f4b4d576 e652b523-8f7d-0b03-40ad-3fc1f4b4d576}" 2026-04-14T14:14:41.902Z [ERROR] server-be030a87-cccc-adc2-cfac-864b5da59931: failed to heartbeat to: peer=f0808778-e412-11d8-9a42-d0953571b1c1 backoff time=10ms error="failed to connect to peer: f0808778-e412-11d8-9a42-d0953571b1c1" 2026-04-14T14:14:41.905Z [ERROR] server-be030a87-cccc-adc2-cfac-864b5da59931: failed to heartbeat to: peer=e652b523-8f7d-0b03-40ad-3fc1f4b4d576 backoff time=10ms error="failed to connect to peer: e652b523-8f7d-0b03-40ad-3fc1f4b4d576" 2026-04-14T14:14:41.906Z [ERROR] server-be030a87-cccc-adc2-cfac-864b5da59931: failed to appendEntries to: peer="{Voter server-e652b523-8f7d-0b03-40ad-3fc1f4b4d576 e652b523-8f7d-0b03-40ad-3fc1f4b4d576}" error="failed to connect to peer: e652b523-8f7d-0b03-40ad-3fc1f4b4d576" 2026-04-14T14:14:41.906Z [ERROR] server-be030a87-cccc-adc2-cfac-864b5da59931: failed to appendEntries to: peer="{Voter server-f0808778-e412-11d8-9a42-d0953571b1c1 f0808778-e412-11d8-9a42-d0953571b1c1}" error="failed to connect to peer: f0808778-e412-11d8-9a42-d0953571b1c1" 2026-04-14T14:14:41.920Z [ERROR] server-be030a87-cccc-adc2-cfac-864b5da59931: failed to heartbeat to: peer=f0808778-e412-11d8-9a42-d0953571b1c1 backoff time=10ms error="failed to connect to peer: f0808778-e412-11d8-9a42-d0953571b1c1" 2026-04-14T14:14:41.921Z [ERROR] server-be030a87-cccc-adc2-cfac-864b5da59931: failed to heartbeat to: peer=e652b523-8f7d-0b03-40ad-3fc1f4b4d576 backoff time=10ms error="failed to connect to peer: e652b523-8f7d-0b03-40ad-3fc1f4b4d576" 2026-04-14T14:14:41.925Z [ERROR] server-be030a87-cccc-adc2-cfac-864b5da59931: failed to appendEntries to: peer="{Voter server-f0808778-e412-11d8-9a42-d0953571b1c1 f0808778-e412-11d8-9a42-d0953571b1c1}" error="failed to connect to peer: f0808778-e412-11d8-9a42-d0953571b1c1" 2026-04-14T14:14:41.925Z [ERROR] server-be030a87-cccc-adc2-cfac-864b5da59931: failed to appendEntries to: peer="{Voter server-e652b523-8f7d-0b03-40ad-3fc1f4b4d576 e652b523-8f7d-0b03-40ad-3fc1f4b4d576}" error="failed to connect to peer: e652b523-8f7d-0b03-40ad-3fc1f4b4d576" 2026-04-14T14:14:41.937Z [ERROR] server-be030a87-cccc-adc2-cfac-864b5da59931: failed to heartbeat to: peer=f0808778-e412-11d8-9a42-d0953571b1c1 backoff time=10ms error="failed to connect to peer: f0808778-e412-11d8-9a42-d0953571b1c1" 2026-04-14T14:14:41.941Z [ERROR] server-be030a87-cccc-adc2-cfac-864b5da59931: failed to heartbeat to: peer=e652b523-8f7d-0b03-40ad-3fc1f4b4d576 backoff time=10ms error="failed to connect to peer: e652b523-8f7d-0b03-40ad-3fc1f4b4d576" 2026-04-14T14:14:41.945Z [ERROR] server-be030a87-cccc-adc2-cfac-864b5da59931: failed to appendEntries to: peer="{Voter server-e652b523-8f7d-0b03-40ad-3fc1f4b4d576 e652b523-8f7d-0b03-40ad-3fc1f4b4d576}" error="failed to connect to peer: e652b523-8f7d-0b03-40ad-3fc1f4b4d576" 2026-04-14T14:14:41.946Z [ERROR] server-be030a87-cccc-adc2-cfac-864b5da59931: failed to appendEntries to: peer="{Voter server-f0808778-e412-11d8-9a42-d0953571b1c1 f0808778-e412-11d8-9a42-d0953571b1c1}" error="failed to connect to peer: f0808778-e412-11d8-9a42-d0953571b1c1" 2026-04-14T14:14:41.946Z [WARN] server-be030a87-cccc-adc2-cfac-864b5da59931: failed to contact: server-id=server-e652b523-8f7d-0b03-40ad-3fc1f4b4d576 time=50.256265ms 2026-04-14T14:14:41.947Z [WARN] server-be030a87-cccc-adc2-cfac-864b5da59931: failed to contact: server-id=server-f0808778-e412-11d8-9a42-d0953571b1c1 time=50.275908ms 2026-04-14T14:14:41.947Z [WARN] server-be030a87-cccc-adc2-cfac-864b5da59931: failed to contact quorum of nodes, stepping down 2026-04-14T14:14:41.947Z [INFO] server-be030a87-cccc-adc2-cfac-864b5da59931: entering follower state: follower="Node at be030a87-cccc-adc2-cfac-864b5da59931 [Follower]" leader-address= leader-id= --- PASS: TestRaft_RemoveFollower_SplitCluster (0.44s) === RUN TestRaft_AddKnownPeer 2026-04-14T14:14:41.948Z [INFO] server-1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd Address:1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd} {Suffrage:Voter ID:server-d1bd2af8-5c25-41ef-5c84-ecd75347a3ae Address:d1bd2af8-5c25-41ef-5c84-ecd75347a3ae} {Suffrage:Voter ID:server-75b7d8ed-2c03-25d3-6b8b-36805ac49cae Address:75b7d8ed-2c03-25d3-6b8b-36805ac49cae}]" 2026-04-14T14:14:41.948Z [INFO] server-1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd: entering follower state: follower="Node at 1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd [Follower]" leader-address= leader-id= 2026-04-14T14:14:41.949Z [INFO] server-d1bd2af8-5c25-41ef-5c84-ecd75347a3ae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd Address:1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd} {Suffrage:Voter ID:server-d1bd2af8-5c25-41ef-5c84-ecd75347a3ae Address:d1bd2af8-5c25-41ef-5c84-ecd75347a3ae} {Suffrage:Voter ID:server-75b7d8ed-2c03-25d3-6b8b-36805ac49cae Address:75b7d8ed-2c03-25d3-6b8b-36805ac49cae}]" 2026-04-14T14:14:41.949Z [INFO] server-d1bd2af8-5c25-41ef-5c84-ecd75347a3ae: entering follower state: follower="Node at d1bd2af8-5c25-41ef-5c84-ecd75347a3ae [Follower]" leader-address= leader-id= 2026-04-14T14:14:41.949Z [INFO] server-75b7d8ed-2c03-25d3-6b8b-36805ac49cae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd Address:1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd} {Suffrage:Voter ID:server-d1bd2af8-5c25-41ef-5c84-ecd75347a3ae Address:d1bd2af8-5c25-41ef-5c84-ecd75347a3ae} {Suffrage:Voter ID:server-75b7d8ed-2c03-25d3-6b8b-36805ac49cae Address:75b7d8ed-2c03-25d3-6b8b-36805ac49cae}]" 2026-04-14T14:14:41.949Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:41.949Z [INFO] server-75b7d8ed-2c03-25d3-6b8b-36805ac49cae: entering follower state: follower="Node at 75b7d8ed-2c03-25d3-6b8b-36805ac49cae [Follower]" leader-address= leader-id= 2026-04-14T14:14:42.000Z [WARN] server-d1bd2af8-5c25-41ef-5c84-ecd75347a3ae: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:42.000Z [INFO] server-d1bd2af8-5c25-41ef-5c84-ecd75347a3ae: entering candidate state: node="Node at d1bd2af8-5c25-41ef-5c84-ecd75347a3ae [Candidate]" term=2 2026-04-14T14:14:42.000Z [INFO] server-d1bd2af8-5c25-41ef-5c84-ecd75347a3ae: election won: term=2 tally=2 2026-04-14T14:14:42.000Z [INFO] server-d1bd2af8-5c25-41ef-5c84-ecd75347a3ae: entering leader state: leader="Node at d1bd2af8-5c25-41ef-5c84-ecd75347a3ae [Leader]" 2026-04-14T14:14:42.000Z [INFO] server-d1bd2af8-5c25-41ef-5c84-ecd75347a3ae: added peer, starting replication: peer=server-1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd 2026-04-14T14:14:42.000Z [INFO] server-d1bd2af8-5c25-41ef-5c84-ecd75347a3ae: added peer, starting replication: peer=server-75b7d8ed-2c03-25d3-6b8b-36805ac49cae 2026-04-14T14:14:42.000Z [INFO] server-d1bd2af8-5c25-41ef-5c84-ecd75347a3ae: pipelining replication: peer="{Voter server-75b7d8ed-2c03-25d3-6b8b-36805ac49cae 75b7d8ed-2c03-25d3-6b8b-36805ac49cae}" 2026-04-14T14:14:42.000Z [INFO] server-d1bd2af8-5c25-41ef-5c84-ecd75347a3ae: pipelining replication: peer="{Voter server-1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd 1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd}" 2026-04-14T14:14:42.106Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:42.000326203 +0000 UTC m=+13.472511974 (1 nodes), 51.038195ms from start of poll, 52.274424ms from cluster start. Timeout at 2026-04-14 14:14:42.105328994 +0000 UTC m=+13.577514763, 105.002789ms after stability 2026-04-14T14:14:42.106Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:14:42.212Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:14:42.106311708 +0000 UTC m=+13.578497475 (2 nodes), 1.116µs from start of poll, 158.259925ms from cluster start. Timeout at 2026-04-14 14:14:42.211318532 +0000 UTC m=+13.683504299, 105.006824ms after stability 2026-04-14T14:14:42.212Z [INFO] server-d1bd2af8-5c25-41ef-5c84-ecd75347a3ae: updating configuration: command=AddVoter server-id=server-1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd server-addr=1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd servers="[{Suffrage:Voter ID:server-1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd Address:1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd} {Suffrage:Voter ID:server-d1bd2af8-5c25-41ef-5c84-ecd75347a3ae Address:d1bd2af8-5c25-41ef-5c84-ecd75347a3ae} {Suffrage:Voter ID:server-75b7d8ed-2c03-25d3-6b8b-36805ac49cae Address:75b7d8ed-2c03-25d3-6b8b-36805ac49cae}]" 2026-04-14T14:14:42.212Z [INFO] server-d1bd2af8-5c25-41ef-5c84-ecd75347a3ae: aborting pipeline replication: peer="{Voter server-75b7d8ed-2c03-25d3-6b8b-36805ac49cae 75b7d8ed-2c03-25d3-6b8b-36805ac49cae}" 2026-04-14T14:14:42.212Z [INFO] server-d1bd2af8-5c25-41ef-5c84-ecd75347a3ae: aborting pipeline replication: peer="{Voter server-1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd 1ad9e2c8-9be8-4ba4-5132-c2f50bbc79bd}" --- PASS: TestRaft_AddKnownPeer (0.27s) === RUN TestRaft_RemoveUnknownPeer 2026-04-14T14:14:42.213Z [INFO] server-0f6cf8d0-816a-8449-8e7f-cb2869c7e0a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f6cf8d0-816a-8449-8e7f-cb2869c7e0a8 Address:0f6cf8d0-816a-8449-8e7f-cb2869c7e0a8} {Suffrage:Voter ID:server-6b58e954-5ed3-2c00-7c5a-ce03aba66777 Address:6b58e954-5ed3-2c00-7c5a-ce03aba66777} {Suffrage:Voter ID:server-64a1ef69-2f41-538c-6f9a-ffd6ce5ff186 Address:64a1ef69-2f41-538c-6f9a-ffd6ce5ff186}]" 2026-04-14T14:14:42.213Z [INFO] server-6b58e954-5ed3-2c00-7c5a-ce03aba66777: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f6cf8d0-816a-8449-8e7f-cb2869c7e0a8 Address:0f6cf8d0-816a-8449-8e7f-cb2869c7e0a8} {Suffrage:Voter ID:server-6b58e954-5ed3-2c00-7c5a-ce03aba66777 Address:6b58e954-5ed3-2c00-7c5a-ce03aba66777} {Suffrage:Voter ID:server-64a1ef69-2f41-538c-6f9a-ffd6ce5ff186 Address:64a1ef69-2f41-538c-6f9a-ffd6ce5ff186}]" 2026-04-14T14:14:42.213Z [INFO] server-0f6cf8d0-816a-8449-8e7f-cb2869c7e0a8: entering follower state: follower="Node at 0f6cf8d0-816a-8449-8e7f-cb2869c7e0a8 [Follower]" leader-address= leader-id= 2026-04-14T14:14:42.213Z [INFO] server-64a1ef69-2f41-538c-6f9a-ffd6ce5ff186: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f6cf8d0-816a-8449-8e7f-cb2869c7e0a8 Address:0f6cf8d0-816a-8449-8e7f-cb2869c7e0a8} {Suffrage:Voter ID:server-6b58e954-5ed3-2c00-7c5a-ce03aba66777 Address:6b58e954-5ed3-2c00-7c5a-ce03aba66777} {Suffrage:Voter ID:server-64a1ef69-2f41-538c-6f9a-ffd6ce5ff186 Address:64a1ef69-2f41-538c-6f9a-ffd6ce5ff186}]" 2026-04-14T14:14:42.213Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:42.213Z [INFO] server-6b58e954-5ed3-2c00-7c5a-ce03aba66777: entering follower state: follower="Node at 6b58e954-5ed3-2c00-7c5a-ce03aba66777 [Follower]" leader-address= leader-id= 2026-04-14T14:14:42.213Z [INFO] server-64a1ef69-2f41-538c-6f9a-ffd6ce5ff186: entering follower state: follower="Node at 64a1ef69-2f41-538c-6f9a-ffd6ce5ff186 [Follower]" leader-address= leader-id= 2026-04-14T14:14:42.270Z [WARN] server-6b58e954-5ed3-2c00-7c5a-ce03aba66777: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:42.270Z [INFO] server-6b58e954-5ed3-2c00-7c5a-ce03aba66777: entering candidate state: node="Node at 6b58e954-5ed3-2c00-7c5a-ce03aba66777 [Candidate]" term=2 2026-04-14T14:14:42.270Z [INFO] server-6b58e954-5ed3-2c00-7c5a-ce03aba66777: election won: term=2 tally=2 2026-04-14T14:14:42.270Z [INFO] server-6b58e954-5ed3-2c00-7c5a-ce03aba66777: entering leader state: leader="Node at 6b58e954-5ed3-2c00-7c5a-ce03aba66777 [Leader]" 2026-04-14T14:14:42.270Z [INFO] server-6b58e954-5ed3-2c00-7c5a-ce03aba66777: added peer, starting replication: peer=server-0f6cf8d0-816a-8449-8e7f-cb2869c7e0a8 2026-04-14T14:14:42.270Z [INFO] server-6b58e954-5ed3-2c00-7c5a-ce03aba66777: added peer, starting replication: peer=server-64a1ef69-2f41-538c-6f9a-ffd6ce5ff186 2026-04-14T14:14:42.270Z [INFO] server-6b58e954-5ed3-2c00-7c5a-ce03aba66777: pipelining replication: peer="{Voter server-64a1ef69-2f41-538c-6f9a-ffd6ce5ff186 64a1ef69-2f41-538c-6f9a-ffd6ce5ff186}" 2026-04-14T14:14:42.270Z [INFO] server-6b58e954-5ed3-2c00-7c5a-ce03aba66777: pipelining replication: peer="{Voter server-0f6cf8d0-816a-8449-8e7f-cb2869c7e0a8 0f6cf8d0-816a-8449-8e7f-cb2869c7e0a8}" 2026-04-14T14:14:42.376Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:42.270552836 +0000 UTC m=+13.742738660 (1 nodes), 57.073858ms from start of poll, 57.329864ms from cluster start. Timeout at 2026-04-14 14:14:42.375555924 +0000 UTC m=+13.847741688, 105.003028ms after stability 2026-04-14T14:14:42.376Z [INFO] server-6b58e954-5ed3-2c00-7c5a-ce03aba66777: updating configuration: command=RemoveServer server-id=6f896e60-f512-52db-445a-18ec0991bdcd server-addr= servers="[{Suffrage:Voter ID:server-0f6cf8d0-816a-8449-8e7f-cb2869c7e0a8 Address:0f6cf8d0-816a-8449-8e7f-cb2869c7e0a8} {Suffrage:Voter ID:server-6b58e954-5ed3-2c00-7c5a-ce03aba66777 Address:6b58e954-5ed3-2c00-7c5a-ce03aba66777} {Suffrage:Voter ID:server-64a1ef69-2f41-538c-6f9a-ffd6ce5ff186 Address:64a1ef69-2f41-538c-6f9a-ffd6ce5ff186}]" 2026-04-14T14:14:42.376Z [INFO] server-6b58e954-5ed3-2c00-7c5a-ce03aba66777: aborting pipeline replication: peer="{Voter server-64a1ef69-2f41-538c-6f9a-ffd6ce5ff186 64a1ef69-2f41-538c-6f9a-ffd6ce5ff186}" 2026-04-14T14:14:42.376Z [INFO] server-6b58e954-5ed3-2c00-7c5a-ce03aba66777: aborting pipeline replication: peer="{Voter server-0f6cf8d0-816a-8449-8e7f-cb2869c7e0a8 0f6cf8d0-816a-8449-8e7f-cb2869c7e0a8}" --- PASS: TestRaft_RemoveUnknownPeer (0.16s) === RUN TestRaft_SnapshotRestore 2026-04-14T14:14:42.377Z [INFO] server-67daa427-8169-9ffb-50b1-7d55affe82c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67daa427-8169-9ffb-50b1-7d55affe82c0 Address:67daa427-8169-9ffb-50b1-7d55affe82c0}]" 2026-04-14T14:14:42.377Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:42.377Z [INFO] server-67daa427-8169-9ffb-50b1-7d55affe82c0: entering follower state: follower="Node at 67daa427-8169-9ffb-50b1-7d55affe82c0 [Follower]" leader-address= leader-id= 2026-04-14T14:14:42.446Z [WARN] server-67daa427-8169-9ffb-50b1-7d55affe82c0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:42.446Z [INFO] server-67daa427-8169-9ffb-50b1-7d55affe82c0: entering candidate state: node="Node at 67daa427-8169-9ffb-50b1-7d55affe82c0 [Candidate]" term=2 2026-04-14T14:14:42.446Z [INFO] server-67daa427-8169-9ffb-50b1-7d55affe82c0: election won: term=2 tally=1 2026-04-14T14:14:42.446Z [INFO] server-67daa427-8169-9ffb-50b1-7d55affe82c0: entering leader state: leader="Node at 67daa427-8169-9ffb-50b1-7d55affe82c0 [Leader]" 2026-04-14T14:14:42.552Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:42.446546697 +0000 UTC m=+13.918732466 (1 nodes), 68.980233ms from start of poll, 69.092312ms from cluster start. Timeout at 2026-04-14 14:14:42.551549105 +0000 UTC m=+14.023734870, 105.002404ms after stability 2026-04-14T14:14:42.554Z [INFO] server-67daa427-8169-9ffb-50b1-7d55affe82c0: starting snapshot up to: index=102 2026-04-14T14:14:42.554Z [INFO] creating new snapshot: path=/tmp/raft2493296816/snapshots/2-102-1776176082554.tmp 2026-04-14T14:14:42.555Z [INFO] server-67daa427-8169-9ffb-50b1-7d55affe82c0: compacting logs: from=1 to=92 2026-04-14T14:14:42.555Z [INFO] server-67daa427-8169-9ffb-50b1-7d55affe82c0: snapshot complete up to: index=102 2026-04-14T14:14:42.555Z [INFO] server-67daa427-8169-9ffb-50b1-7d55affe82c0: starting restore from snapshot: id=2-102-1776176082554 last-index=102 last-term=2 size-in-bytes=693 2026-04-14T14:14:42.556Z [INFO] server-67daa427-8169-9ffb-50b1-7d55affe82c0: snapshot restore progress: id=2-102-1776176082554 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2026-04-14T14:14:42.556Z [INFO] server-67daa427-8169-9ffb-50b1-7d55affe82c0: restored from snapshot: id=2-102-1776176082554 last-index=102 last-term=2 size-in-bytes=693 2026-04-14T14:14:42.556Z [INFO] server-67daa427-8169-9ffb-50b1-7d55affe82c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67daa427-8169-9ffb-50b1-7d55affe82c0 Address:67daa427-8169-9ffb-50b1-7d55affe82c0}]" --- PASS: TestRaft_SnapshotRestore (0.18s) === RUN TestRaft_RestoreSnapshotOnStartup_Monotonic 2026-04-14T14:14:42.558Z [INFO] server-29224890-0860-7aff-4513-369ee7b75755: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29224890-0860-7aff-4513-369ee7b75755 Address:29224890-0860-7aff-4513-369ee7b75755}]" 2026-04-14T14:14:42.558Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:42.558Z [INFO] server-29224890-0860-7aff-4513-369ee7b75755: entering follower state: follower="Node at 29224890-0860-7aff-4513-369ee7b75755 [Follower]" leader-address= leader-id= 2026-04-14T14:14:42.648Z [WARN] server-29224890-0860-7aff-4513-369ee7b75755: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:42.648Z [INFO] server-29224890-0860-7aff-4513-369ee7b75755: entering candidate state: node="Node at 29224890-0860-7aff-4513-369ee7b75755 [Candidate]" term=2 2026-04-14T14:14:42.648Z [INFO] server-29224890-0860-7aff-4513-369ee7b75755: election won: term=2 tally=1 2026-04-14T14:14:42.648Z [INFO] server-29224890-0860-7aff-4513-369ee7b75755: entering leader state: leader="Node at 29224890-0860-7aff-4513-369ee7b75755 [Leader]" 2026-04-14T14:14:42.754Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:42.648300395 +0000 UTC m=+14.120486166 (1 nodes), 89.580608ms from start of poll, 89.799562ms from cluster start. Timeout at 2026-04-14 14:14:42.753303399 +0000 UTC m=+14.225489166, 105.003ms after stability 2026-04-14T14:14:42.756Z [INFO] server-29224890-0860-7aff-4513-369ee7b75755: starting snapshot up to: index=102 2026-04-14T14:14:42.756Z [INFO] creating new snapshot: path=/tmp/raft1892056464/snapshots/2-102-1776176082756.tmp 2026-04-14T14:14:42.756Z [INFO] server-29224890-0860-7aff-4513-369ee7b75755: compacting logs: from=1 to=92 2026-04-14T14:14:42.756Z [INFO] server-29224890-0860-7aff-4513-369ee7b75755: snapshot complete up to: index=102 2026-04-14T14:14:42.756Z [INFO] server-29224890-0860-7aff-4513-369ee7b75755: starting restore from snapshot: id=2-102-1776176082756 last-index=102 last-term=2 size-in-bytes=693 2026-04-14T14:14:42.757Z [INFO] server-29224890-0860-7aff-4513-369ee7b75755: snapshot restore progress: id=2-102-1776176082756 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2026-04-14T14:14:42.757Z [INFO] server-29224890-0860-7aff-4513-369ee7b75755: restored from snapshot: id=2-102-1776176082756 last-index=102 last-term=2 size-in-bytes=693 2026-04-14T14:14:42.757Z [INFO] server-29224890-0860-7aff-4513-369ee7b75755: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29224890-0860-7aff-4513-369ee7b75755 Address:29224890-0860-7aff-4513-369ee7b75755}]" --- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.20s) === RUN TestRaft_SnapshotRestore_Progress 2026-04-14T14:14:42.758Z [INFO] server-5a6d7eac-0901-c818-5e9a-d007614736f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a6d7eac-0901-c818-5e9a-d007614736f0 Address:5a6d7eac-0901-c818-5e9a-d007614736f0}]" 2026-04-14T14:14:42.758Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:42.758Z [INFO] server-5a6d7eac-0901-c818-5e9a-d007614736f0: entering follower state: follower="Node at 5a6d7eac-0901-c818-5e9a-d007614736f0 [Follower]" leader-address= leader-id= 2026-04-14T14:14:42.835Z [WARN] server-5a6d7eac-0901-c818-5e9a-d007614736f0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:42.835Z [INFO] server-5a6d7eac-0901-c818-5e9a-d007614736f0: entering candidate state: node="Node at 5a6d7eac-0901-c818-5e9a-d007614736f0 [Candidate]" term=2 2026-04-14T14:14:42.835Z [INFO] server-5a6d7eac-0901-c818-5e9a-d007614736f0: election won: term=2 tally=1 2026-04-14T14:14:42.835Z [INFO] server-5a6d7eac-0901-c818-5e9a-d007614736f0: entering leader state: leader="Node at 5a6d7eac-0901-c818-5e9a-d007614736f0 [Leader]" 2026-04-14T14:14:42.941Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:42.83595055 +0000 UTC m=+14.308136317 (1 nodes), 77.115828ms from start of poll, 77.233439ms from cluster start. Timeout at 2026-04-14 14:14:42.940951162 +0000 UTC m=+14.413136926, 105.000609ms after stability 2026-04-14T14:14:42.944Z [INFO] server-5a6d7eac-0901-c818-5e9a-d007614736f0: starting snapshot up to: index=102 2026-04-14T14:14:42.944Z [INFO] creating new snapshot: path=/tmp/raft1053136177/snapshots/2-102-1776176082944.tmp 2026-04-14T14:14:42.944Z [INFO] server-5a6d7eac-0901-c818-5e9a-d007614736f0: compacting logs: from=1 to=92 2026-04-14T14:14:42.944Z [INFO] server-5a6d7eac-0901-c818-5e9a-d007614736f0: snapshot complete up to: index=102 --- PASS: TestRaft_SnapshotRestore_Progress (0.19s) === RUN TestRaft_NoRestoreOnStart 2026-04-14T14:14:42.945Z [INFO] server-2b0e7a8a-513b-9288-ac6c-f4de706efb16: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b0e7a8a-513b-9288-ac6c-f4de706efb16 Address:2b0e7a8a-513b-9288-ac6c-f4de706efb16}]" 2026-04-14T14:14:42.946Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:42.946Z [INFO] server-2b0e7a8a-513b-9288-ac6c-f4de706efb16: entering follower state: follower="Node at 2b0e7a8a-513b-9288-ac6c-f4de706efb16 [Follower]" leader-address= leader-id= 2026-04-14T14:14:43.035Z [WARN] server-2b0e7a8a-513b-9288-ac6c-f4de706efb16: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:43.035Z [INFO] server-2b0e7a8a-513b-9288-ac6c-f4de706efb16: entering candidate state: node="Node at 2b0e7a8a-513b-9288-ac6c-f4de706efb16 [Candidate]" term=2 2026-04-14T14:14:43.035Z [INFO] server-2b0e7a8a-513b-9288-ac6c-f4de706efb16: election won: term=2 tally=1 2026-04-14T14:14:43.035Z [INFO] server-2b0e7a8a-513b-9288-ac6c-f4de706efb16: entering leader state: leader="Node at 2b0e7a8a-513b-9288-ac6c-f4de706efb16 [Leader]" 2026-04-14T14:14:43.140Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:43.035238802 +0000 UTC m=+14.507424571 (1 nodes), 89.225268ms from start of poll, 89.348178ms from cluster start. Timeout at 2026-04-14 14:14:43.140242545 +0000 UTC m=+14.612428317, 105.003746ms after stability 2026-04-14T14:14:43.141Z [INFO] server-2b0e7a8a-513b-9288-ac6c-f4de706efb16: starting snapshot up to: index=102 2026-04-14T14:14:43.141Z [INFO] creating new snapshot: path=/tmp/raft1407947192/snapshots/2-102-1776176083141.tmp 2026-04-14T14:14:43.142Z [INFO] server-2b0e7a8a-513b-9288-ac6c-f4de706efb16: compacting logs: from=1 to=92 2026-04-14T14:14:43.142Z [INFO] server-2b0e7a8a-513b-9288-ac6c-f4de706efb16: snapshot complete up to: index=102 2026-04-14T14:14:43.142Z [INFO] server-2b0e7a8a-513b-9288-ac6c-f4de706efb16: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b0e7a8a-513b-9288-ac6c-f4de706efb16 Address:2b0e7a8a-513b-9288-ac6c-f4de706efb16}]" --- PASS: TestRaft_NoRestoreOnStart (0.20s) === RUN TestRaft_SnapshotRestore_PeerChange 2026-04-14T14:14:43.142Z [INFO] server-2b0e7a8a-513b-9288-ac6c-f4de706efb16: entering follower state: follower="Node at 2b0e7a8a-513b-9288-ac6c-f4de706efb16 [Follower]" leader-address= leader-id= raft_test.go:1266: Skipping unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot 2026-04-14T14:14:43.143Z [INFO] server-fddcba18-0db5-4c21-7b5b-5c64d0961167: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fddcba18-0db5-4c21-7b5b-5c64d0961167 Address:fddcba18-0db5-4c21-7b5b-5c64d0961167}]" 2026-04-14T14:14:43.143Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:43.143Z [INFO] server-fddcba18-0db5-4c21-7b5b-5c64d0961167: entering follower state: follower="Node at fddcba18-0db5-4c21-7b5b-5c64d0961167 [Follower]" leader-address= leader-id= 2026-04-14T14:14:43.197Z [WARN] server-2b0e7a8a-513b-9288-ac6c-f4de706efb16: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:43.197Z [INFO] server-2b0e7a8a-513b-9288-ac6c-f4de706efb16: entering candidate state: node="Node at 2b0e7a8a-513b-9288-ac6c-f4de706efb16 [Candidate]" term=3 2026-04-14T14:14:43.197Z [INFO] server-2b0e7a8a-513b-9288-ac6c-f4de706efb16: election won: term=3 tally=1 2026-04-14T14:14:43.197Z [INFO] server-2b0e7a8a-513b-9288-ac6c-f4de706efb16: entering leader state: leader="Node at 2b0e7a8a-513b-9288-ac6c-f4de706efb16 [Leader]" 2026-04-14T14:14:43.199Z [WARN] server-fddcba18-0db5-4c21-7b5b-5c64d0961167: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:43.199Z [INFO] server-fddcba18-0db5-4c21-7b5b-5c64d0961167: entering candidate state: node="Node at fddcba18-0db5-4c21-7b5b-5c64d0961167 [Candidate]" term=2 2026-04-14T14:14:43.199Z [INFO] server-fddcba18-0db5-4c21-7b5b-5c64d0961167: election won: term=2 tally=1 2026-04-14T14:14:43.199Z [INFO] server-fddcba18-0db5-4c21-7b5b-5c64d0961167: entering leader state: leader="Node at fddcba18-0db5-4c21-7b5b-5c64d0961167 [Leader]" 2026-04-14T14:14:43.305Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:43.199412923 +0000 UTC m=+14.671598703 (1 nodes), 55.514113ms from start of poll, 55.977288ms from cluster start. Timeout at 2026-04-14 14:14:43.304413566 +0000 UTC m=+14.776599335, 105.000632ms after stability 2026-04-14T14:14:43.319Z [INFO] server-fddcba18-0db5-4c21-7b5b-5c64d0961167: starting snapshot up to: index=102 2026-04-14T14:14:43.319Z [INFO] creating new snapshot: path=/tmp/raft3975492585/snapshots/2-102-1776176083319.tmp 2026-04-14T14:14:43.320Z [INFO] server-fddcba18-0db5-4c21-7b5b-5c64d0961167: compacting logs: from=1 to=92 2026-04-14T14:14:43.320Z [INFO] server-fddcba18-0db5-4c21-7b5b-5c64d0961167: snapshot complete up to: index=102 --- PASS: TestRaft_AutoSnapshot (0.27s) === RUN TestRaft_UserSnapshot 2026-04-14T14:14:43.413Z [INFO] server-c5d57b10-a975-7c58-18f6-efc78d6d1c95: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c5d57b10-a975-7c58-18f6-efc78d6d1c95 Address:c5d57b10-a975-7c58-18f6-efc78d6d1c95}]" 2026-04-14T14:14:43.413Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:43.413Z [INFO] server-c5d57b10-a975-7c58-18f6-efc78d6d1c95: entering follower state: follower="Node at c5d57b10-a975-7c58-18f6-efc78d6d1c95 [Follower]" leader-address= leader-id= 2026-04-14T14:14:43.501Z [WARN] server-c5d57b10-a975-7c58-18f6-efc78d6d1c95: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:43.501Z [INFO] server-c5d57b10-a975-7c58-18f6-efc78d6d1c95: entering candidate state: node="Node at c5d57b10-a975-7c58-18f6-efc78d6d1c95 [Candidate]" term=2 2026-04-14T14:14:43.501Z [INFO] server-c5d57b10-a975-7c58-18f6-efc78d6d1c95: election won: term=2 tally=1 2026-04-14T14:14:43.501Z [INFO] server-c5d57b10-a975-7c58-18f6-efc78d6d1c95: entering leader state: leader="Node at c5d57b10-a975-7c58-18f6-efc78d6d1c95 [Leader]" 2026-04-14T14:14:43.607Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:43.501601507 +0000 UTC m=+14.973787275 (1 nodes), 88.245963ms from start of poll, 88.387434ms from cluster start. Timeout at 2026-04-14 14:14:43.606602808 +0000 UTC m=+15.078788576, 105.001301ms after stability 2026-04-14T14:14:43.607Z [ERROR] server-c5d57b10-a975-7c58-18f6-efc78d6d1c95: failed to take snapshot: error="nothing new to snapshot" 2026-04-14T14:14:43.607Z [INFO] server-c5d57b10-a975-7c58-18f6-efc78d6d1c95: starting snapshot up to: index=12 2026-04-14T14:14:43.607Z [INFO] creating new snapshot: path=/tmp/raft2766275142/snapshots/2-12-1776176083607.tmp 2026-04-14T14:14:43.608Z [INFO] server-c5d57b10-a975-7c58-18f6-efc78d6d1c95: compacting logs: from=1 to=2 2026-04-14T14:14:43.608Z [INFO] server-c5d57b10-a975-7c58-18f6-efc78d6d1c95: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.26s) === RUN TestRaft_UserRestore === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false 2026-04-14T14:14:43.676Z [INFO] server-e839da20-6de3-7423-ea1c-0ecda0dded6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e839da20-6de3-7423-ea1c-0ecda0dded6f Address:e839da20-6de3-7423-ea1c-0ecda0dded6f} {Suffrage:Voter ID:server-41a56509-311d-96da-e9ea-da6f404af727 Address:41a56509-311d-96da-e9ea-da6f404af727} {Suffrage:Voter ID:server-df553f25-90e4-5f04-eff1-0716c7733dbd Address:df553f25-90e4-5f04-eff1-0716c7733dbd}]" 2026-04-14T14:14:43.676Z [INFO] server-41a56509-311d-96da-e9ea-da6f404af727: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e839da20-6de3-7423-ea1c-0ecda0dded6f Address:e839da20-6de3-7423-ea1c-0ecda0dded6f} {Suffrage:Voter ID:server-41a56509-311d-96da-e9ea-da6f404af727 Address:41a56509-311d-96da-e9ea-da6f404af727} {Suffrage:Voter ID:server-df553f25-90e4-5f04-eff1-0716c7733dbd Address:df553f25-90e4-5f04-eff1-0716c7733dbd}]" 2026-04-14T14:14:43.676Z [INFO] server-e839da20-6de3-7423-ea1c-0ecda0dded6f: entering follower state: follower="Node at e839da20-6de3-7423-ea1c-0ecda0dded6f [Follower]" leader-address= leader-id= 2026-04-14T14:14:43.676Z [INFO] server-41a56509-311d-96da-e9ea-da6f404af727: entering follower state: follower="Node at 41a56509-311d-96da-e9ea-da6f404af727 [Follower]" leader-address= leader-id= 2026-04-14T14:14:43.677Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e839da20-6de3-7423-ea1c-0ecda0dded6f Address:e839da20-6de3-7423-ea1c-0ecda0dded6f} {Suffrage:Voter ID:server-41a56509-311d-96da-e9ea-da6f404af727 Address:41a56509-311d-96da-e9ea-da6f404af727} {Suffrage:Voter ID:server-df553f25-90e4-5f04-eff1-0716c7733dbd Address:df553f25-90e4-5f04-eff1-0716c7733dbd}]" 2026-04-14T14:14:43.677Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:43.677Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: entering follower state: follower="Node at df553f25-90e4-5f04-eff1-0716c7733dbd [Follower]" leader-address= leader-id= 2026-04-14T14:14:44.272Z [WARN] server-df553f25-90e4-5f04-eff1-0716c7733dbd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:44.272Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: entering candidate state: node="Node at df553f25-90e4-5f04-eff1-0716c7733dbd [Candidate]" term=2 2026-04-14T14:14:44.272Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: election won: term=2 tally=2 2026-04-14T14:14:44.272Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: entering leader state: leader="Node at df553f25-90e4-5f04-eff1-0716c7733dbd [Leader]" 2026-04-14T14:14:44.272Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: added peer, starting replication: peer=server-e839da20-6de3-7423-ea1c-0ecda0dded6f 2026-04-14T14:14:44.272Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: added peer, starting replication: peer=server-41a56509-311d-96da-e9ea-da6f404af727 2026-04-14T14:14:44.272Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: pipelining replication: peer="{Voter server-41a56509-311d-96da-e9ea-da6f404af727 41a56509-311d-96da-e9ea-da6f404af727}" 2026-04-14T14:14:44.272Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: pipelining replication: peer="{Voter server-e839da20-6de3-7423-ea1c-0ecda0dded6f e839da20-6de3-7423-ea1c-0ecda0dded6f}" 2026-04-14T14:14:45.278Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:44.272258407 +0000 UTC m=+15.744444171 (1 nodes), 595.180818ms from start of poll, 595.71088ms from cluster start. Timeout at 2026-04-14 14:14:45.277261075 +0000 UTC m=+16.749446842, 1.005002671s after stability 2026-04-14T14:14:45.278Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: starting snapshot up to: index=12 2026-04-14T14:14:45.278Z [INFO] creating new snapshot: path=/tmp/raft3069781946/snapshots/2-12-1776176085278.tmp 2026-04-14T14:14:45.306Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: snapshot complete up to: index=12 2026-04-14T14:14:45.306Z [INFO] creating new snapshot: path=/tmp/raft3069781946/snapshots/2-23-1776176085306.tmp 2026-04-14T14:14:45.307Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: copied to local snapshot: bytes=71 2026-04-14T14:14:45.307Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: snapshot restore progress: id=2-23-1776176085306 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:45.307Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: restored user snapshot: index=23 2026-04-14T14:14:45.307Z [ERROR] server-df553f25-90e4-5f04-eff1-0716c7733dbd: failed to get log: index=23 error="log not found" 2026-04-14T14:14:45.307Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: aborting pipeline replication: peer="{Voter server-e839da20-6de3-7423-ea1c-0ecda0dded6f e839da20-6de3-7423-ea1c-0ecda0dded6f}" 2026-04-14T14:14:45.307Z [ERROR] server-df553f25-90e4-5f04-eff1-0716c7733dbd: failed to get log: index=23 error="log not found" 2026-04-14T14:14:45.307Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: aborting pipeline replication: peer="{Voter server-41a56509-311d-96da-e9ea-da6f404af727 41a56509-311d-96da-e9ea-da6f404af727}" 2026-04-14T14:14:45.316Z [ERROR] server-df553f25-90e4-5f04-eff1-0716c7733dbd: failed to get log: index=23 error="log not found" 2026-04-14T14:14:45.316Z [INFO] creating new snapshot: path=/tmp/raft2549282982/snapshots/2-23-1776176085316.tmp 2026-04-14T14:14:45.317Z [ERROR] server-df553f25-90e4-5f04-eff1-0716c7733dbd: failed to get log: index=23 error="log not found" 2026-04-14T14:14:45.317Z [INFO] server-41a56509-311d-96da-e9ea-da6f404af727: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:45.317Z [INFO] creating new snapshot: path=/tmp/raft2564509648/snapshots/2-23-1776176085317.tmp 2026-04-14T14:14:45.317Z [INFO] server-41a56509-311d-96da-e9ea-da6f404af727: copied to local snapshot: bytes=71 2026-04-14T14:14:45.317Z [INFO] server-e839da20-6de3-7423-ea1c-0ecda0dded6f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:45.317Z [INFO] server-41a56509-311d-96da-e9ea-da6f404af727: snapshot restore progress: id=2-23-1776176085316 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:45.317Z [INFO] server-41a56509-311d-96da-e9ea-da6f404af727: Installed remote snapshot 2026-04-14T14:14:45.318Z [WARN] server-41a56509-311d-96da-e9ea-da6f404af727: skipping application of old log: index=22 2026-04-14T14:14:45.318Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: pipelining replication: peer="{Voter server-41a56509-311d-96da-e9ea-da6f404af727 41a56509-311d-96da-e9ea-da6f404af727}" 2026-04-14T14:14:45.318Z [INFO] server-e839da20-6de3-7423-ea1c-0ecda0dded6f: copied to local snapshot: bytes=71 2026-04-14T14:14:45.318Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:14:45.319Z [INFO] server-e839da20-6de3-7423-ea1c-0ecda0dded6f: snapshot restore progress: id=2-23-1776176085317 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:45.319Z [INFO] server-e839da20-6de3-7423-ea1c-0ecda0dded6f: Installed remote snapshot 2026-04-14T14:14:45.319Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: pipelining replication: peer="{Voter server-e839da20-6de3-7423-ea1c-0ecda0dded6f e839da20-6de3-7423-ea1c-0ecda0dded6f}" 2026-04-14T14:14:46.324Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:14:45.318586014 +0000 UTC m=+16.790771796 (2 nodes), 1.557µs from start of poll, 1.642038505s from cluster start. Timeout at 2026-04-14 14:14:46.323589551 +0000 UTC m=+17.795775317, 1.005003521s after stability 2026-04-14T14:14:46.336Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: aborting pipeline replication: peer="{Voter server-41a56509-311d-96da-e9ea-da6f404af727 41a56509-311d-96da-e9ea-da6f404af727}" 2026-04-14T14:14:46.336Z [INFO] server-df553f25-90e4-5f04-eff1-0716c7733dbd: aborting pipeline replication: peer="{Voter server-e839da20-6de3-7423-ea1c-0ecda0dded6f e839da20-6de3-7423-ea1c-0ecda0dded6f}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false 2026-04-14T14:14:46.339Z [INFO] server-9b1e1f01-e18e-52ba-af68-4c68677a21d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b1e1f01-e18e-52ba-af68-4c68677a21d5 Address:9b1e1f01-e18e-52ba-af68-4c68677a21d5} {Suffrage:Voter ID:server-2918ce69-ecfc-6e84-b87b-57b0f6baccc6 Address:2918ce69-ecfc-6e84-b87b-57b0f6baccc6} {Suffrage:Voter ID:server-36aee4a9-4ca9-0995-acb0-90314b33f32d Address:36aee4a9-4ca9-0995-acb0-90314b33f32d}]" 2026-04-14T14:14:46.339Z [INFO] server-9b1e1f01-e18e-52ba-af68-4c68677a21d5: entering follower state: follower="Node at 9b1e1f01-e18e-52ba-af68-4c68677a21d5 [Follower]" leader-address= leader-id= 2026-04-14T14:14:46.339Z [INFO] server-2918ce69-ecfc-6e84-b87b-57b0f6baccc6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b1e1f01-e18e-52ba-af68-4c68677a21d5 Address:9b1e1f01-e18e-52ba-af68-4c68677a21d5} {Suffrage:Voter ID:server-2918ce69-ecfc-6e84-b87b-57b0f6baccc6 Address:2918ce69-ecfc-6e84-b87b-57b0f6baccc6} {Suffrage:Voter ID:server-36aee4a9-4ca9-0995-acb0-90314b33f32d Address:36aee4a9-4ca9-0995-acb0-90314b33f32d}]" 2026-04-14T14:14:46.339Z [INFO] server-2918ce69-ecfc-6e84-b87b-57b0f6baccc6: entering follower state: follower="Node at 2918ce69-ecfc-6e84-b87b-57b0f6baccc6 [Follower]" leader-address= leader-id= 2026-04-14T14:14:46.339Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b1e1f01-e18e-52ba-af68-4c68677a21d5 Address:9b1e1f01-e18e-52ba-af68-4c68677a21d5} {Suffrage:Voter ID:server-2918ce69-ecfc-6e84-b87b-57b0f6baccc6 Address:2918ce69-ecfc-6e84-b87b-57b0f6baccc6} {Suffrage:Voter ID:server-36aee4a9-4ca9-0995-acb0-90314b33f32d Address:36aee4a9-4ca9-0995-acb0-90314b33f32d}]" 2026-04-14T14:14:46.339Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:46.339Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: entering follower state: follower="Node at 36aee4a9-4ca9-0995-acb0-90314b33f32d [Follower]" leader-address= leader-id= 2026-04-14T14:14:46.898Z [WARN] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:46.898Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: entering candidate state: node="Node at 36aee4a9-4ca9-0995-acb0-90314b33f32d [Candidate]" term=2 2026-04-14T14:14:46.898Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: election won: term=2 tally=2 2026-04-14T14:14:46.898Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: entering leader state: leader="Node at 36aee4a9-4ca9-0995-acb0-90314b33f32d [Leader]" 2026-04-14T14:14:46.898Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: added peer, starting replication: peer=server-9b1e1f01-e18e-52ba-af68-4c68677a21d5 2026-04-14T14:14:46.898Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: added peer, starting replication: peer=server-2918ce69-ecfc-6e84-b87b-57b0f6baccc6 2026-04-14T14:14:46.899Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: pipelining replication: peer="{Voter server-2918ce69-ecfc-6e84-b87b-57b0f6baccc6 2918ce69-ecfc-6e84-b87b-57b0f6baccc6}" 2026-04-14T14:14:46.899Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: pipelining replication: peer="{Voter server-9b1e1f01-e18e-52ba-af68-4c68677a21d5 9b1e1f01-e18e-52ba-af68-4c68677a21d5}" 2026-04-14T14:14:47.904Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:46.898992477 +0000 UTC m=+18.371178271 (1 nodes), 559.040694ms from start of poll, 559.750548ms from cluster start. Timeout at 2026-04-14 14:14:47.903996942 +0000 UTC m=+19.376182711, 1.00500444s after stability 2026-04-14T14:14:47.906Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: starting snapshot up to: index=12 2026-04-14T14:14:47.906Z [INFO] creating new snapshot: path=/tmp/raft3776698725/snapshots/2-12-1776176087906.tmp 2026-04-14T14:14:47.922Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: snapshot complete up to: index=12 2026-04-14T14:14:47.922Z [INFO] creating new snapshot: path=/tmp/raft3776698725/snapshots/2-23-1776176087922.tmp 2026-04-14T14:14:47.923Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: copied to local snapshot: bytes=71 2026-04-14T14:14:47.923Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: snapshot restore progress: id=2-23-1776176087922 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:47.923Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: removing all old logs from log store 2026-04-14T14:14:47.923Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: compacting logs: from=1 to=22 2026-04-14T14:14:47.923Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: restored user snapshot: index=23 2026-04-14T14:14:47.923Z [ERROR] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: failed to get log: index=22 error="log not found" 2026-04-14T14:14:47.923Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: aborting pipeline replication: peer="{Voter server-2918ce69-ecfc-6e84-b87b-57b0f6baccc6 2918ce69-ecfc-6e84-b87b-57b0f6baccc6}" 2026-04-14T14:14:47.923Z [ERROR] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: failed to get log: index=22 error="log not found" 2026-04-14T14:14:47.923Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: aborting pipeline replication: peer="{Voter server-9b1e1f01-e18e-52ba-af68-4c68677a21d5 9b1e1f01-e18e-52ba-af68-4c68677a21d5}" 2026-04-14T14:14:47.930Z [ERROR] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: failed to get log: index=22 error="log not found" 2026-04-14T14:14:47.930Z [INFO] creating new snapshot: path=/tmp/raft153927908/snapshots/2-23-1776176087930.tmp 2026-04-14T14:14:47.930Z [ERROR] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: failed to get log: index=22 error="log not found" 2026-04-14T14:14:47.930Z [INFO] server-9b1e1f01-e18e-52ba-af68-4c68677a21d5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:47.931Z [INFO] creating new snapshot: path=/tmp/raft1835315632/snapshots/2-23-1776176087931.tmp 2026-04-14T14:14:47.931Z [INFO] server-2918ce69-ecfc-6e84-b87b-57b0f6baccc6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:48.016Z [INFO] server-9b1e1f01-e18e-52ba-af68-4c68677a21d5: copied to local snapshot: bytes=71 2026-04-14T14:14:48.016Z [INFO] server-2918ce69-ecfc-6e84-b87b-57b0f6baccc6: copied to local snapshot: bytes=71 2026-04-14T14:14:48.016Z [INFO] server-2918ce69-ecfc-6e84-b87b-57b0f6baccc6: snapshot restore progress: id=2-23-1776176087931 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:48.016Z [INFO] server-2918ce69-ecfc-6e84-b87b-57b0f6baccc6: removing all old logs from log store 2026-04-14T14:14:48.016Z [INFO] server-2918ce69-ecfc-6e84-b87b-57b0f6baccc6: compacting logs: from=1 to=22 2026-04-14T14:14:48.016Z [INFO] server-2918ce69-ecfc-6e84-b87b-57b0f6baccc6: Installed remote snapshot 2026-04-14T14:14:48.016Z [INFO] server-9b1e1f01-e18e-52ba-af68-4c68677a21d5: snapshot restore progress: id=2-23-1776176087930 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:48.016Z [INFO] server-9b1e1f01-e18e-52ba-af68-4c68677a21d5: removing all old logs from log store 2026-04-14T14:14:48.016Z [INFO] server-9b1e1f01-e18e-52ba-af68-4c68677a21d5: compacting logs: from=1 to=22 2026-04-14T14:14:48.017Z [INFO] server-9b1e1f01-e18e-52ba-af68-4c68677a21d5: Installed remote snapshot 2026-04-14T14:14:48.017Z [WARN] server-2918ce69-ecfc-6e84-b87b-57b0f6baccc6: skipping application of old log: index=22 2026-04-14T14:14:48.017Z [WARN] server-9b1e1f01-e18e-52ba-af68-4c68677a21d5: skipping application of old log: index=23 2026-04-14T14:14:48.017Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: pipelining replication: peer="{Voter server-2918ce69-ecfc-6e84-b87b-57b0f6baccc6 2918ce69-ecfc-6e84-b87b-57b0f6baccc6}" 2026-04-14T14:14:48.017Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: pipelining replication: peer="{Voter server-9b1e1f01-e18e-52ba-af68-4c68677a21d5 9b1e1f01-e18e-52ba-af68-4c68677a21d5}" 2026-04-14T14:14:48.017Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:14:49.022Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:14:48.017173416 +0000 UTC m=+19.489359190 (2 nodes), 1.026µs from start of poll, 1.677931467s from cluster start. Timeout at 2026-04-14 14:14:49.022178954 +0000 UTC m=+20.494364722, 1.005005532s after stability 2026-04-14T14:14:49.034Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: aborting pipeline replication: peer="{Voter server-2918ce69-ecfc-6e84-b87b-57b0f6baccc6 2918ce69-ecfc-6e84-b87b-57b0f6baccc6}" 2026-04-14T14:14:49.034Z [INFO] server-36aee4a9-4ca9-0995-acb0-90314b33f32d: aborting pipeline replication: peer="{Voter server-9b1e1f01-e18e-52ba-af68-4c68677a21d5 9b1e1f01-e18e-52ba-af68-4c68677a21d5}" === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true 2026-04-14T14:14:49.035Z [INFO] server-ffe1821d-d3d5-c457-d0bb-0de58c0ff7ae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ffe1821d-d3d5-c457-d0bb-0de58c0ff7ae Address:ffe1821d-d3d5-c457-d0bb-0de58c0ff7ae} {Suffrage:Voter ID:server-6051bb60-1b25-cb66-52ee-54a4a56a70b3 Address:6051bb60-1b25-cb66-52ee-54a4a56a70b3} {Suffrage:Voter ID:server-b5101e81-2e88-b556-e905-c022c0911aab Address:b5101e81-2e88-b556-e905-c022c0911aab}]" 2026-04-14T14:14:49.035Z [INFO] server-ffe1821d-d3d5-c457-d0bb-0de58c0ff7ae: entering follower state: follower="Node at ffe1821d-d3d5-c457-d0bb-0de58c0ff7ae [Follower]" leader-address= leader-id= 2026-04-14T14:14:49.035Z [INFO] server-6051bb60-1b25-cb66-52ee-54a4a56a70b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ffe1821d-d3d5-c457-d0bb-0de58c0ff7ae Address:ffe1821d-d3d5-c457-d0bb-0de58c0ff7ae} {Suffrage:Voter ID:server-6051bb60-1b25-cb66-52ee-54a4a56a70b3 Address:6051bb60-1b25-cb66-52ee-54a4a56a70b3} {Suffrage:Voter ID:server-b5101e81-2e88-b556-e905-c022c0911aab Address:b5101e81-2e88-b556-e905-c022c0911aab}]" 2026-04-14T14:14:49.035Z [INFO] server-b5101e81-2e88-b556-e905-c022c0911aab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ffe1821d-d3d5-c457-d0bb-0de58c0ff7ae Address:ffe1821d-d3d5-c457-d0bb-0de58c0ff7ae} {Suffrage:Voter ID:server-6051bb60-1b25-cb66-52ee-54a4a56a70b3 Address:6051bb60-1b25-cb66-52ee-54a4a56a70b3} {Suffrage:Voter ID:server-b5101e81-2e88-b556-e905-c022c0911aab Address:b5101e81-2e88-b556-e905-c022c0911aab}]" 2026-04-14T14:14:49.036Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:49.035Z [INFO] server-6051bb60-1b25-cb66-52ee-54a4a56a70b3: entering follower state: follower="Node at 6051bb60-1b25-cb66-52ee-54a4a56a70b3 [Follower]" leader-address= leader-id= 2026-04-14T14:14:49.036Z [INFO] server-b5101e81-2e88-b556-e905-c022c0911aab: entering follower state: follower="Node at b5101e81-2e88-b556-e905-c022c0911aab [Follower]" leader-address= leader-id= 2026-04-14T14:14:49.650Z [WARN] server-6051bb60-1b25-cb66-52ee-54a4a56a70b3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:49.650Z [INFO] server-6051bb60-1b25-cb66-52ee-54a4a56a70b3: entering candidate state: node="Node at 6051bb60-1b25-cb66-52ee-54a4a56a70b3 [Candidate]" term=2 2026-04-14T14:14:49.651Z [INFO] server-6051bb60-1b25-cb66-52ee-54a4a56a70b3: election won: term=2 tally=2 2026-04-14T14:14:49.651Z [INFO] server-6051bb60-1b25-cb66-52ee-54a4a56a70b3: entering leader state: leader="Node at 6051bb60-1b25-cb66-52ee-54a4a56a70b3 [Leader]" 2026-04-14T14:14:49.651Z [INFO] server-6051bb60-1b25-cb66-52ee-54a4a56a70b3: added peer, starting replication: peer=server-ffe1821d-d3d5-c457-d0bb-0de58c0ff7ae 2026-04-14T14:14:49.651Z [INFO] server-6051bb60-1b25-cb66-52ee-54a4a56a70b3: added peer, starting replication: peer=server-b5101e81-2e88-b556-e905-c022c0911aab 2026-04-14T14:14:49.651Z [INFO] server-6051bb60-1b25-cb66-52ee-54a4a56a70b3: pipelining replication: peer="{Voter server-b5101e81-2e88-b556-e905-c022c0911aab b5101e81-2e88-b556-e905-c022c0911aab}" 2026-04-14T14:14:49.651Z [INFO] server-6051bb60-1b25-cb66-52ee-54a4a56a70b3: pipelining replication: peer="{Voter server-ffe1821d-d3d5-c457-d0bb-0de58c0ff7ae ffe1821d-d3d5-c457-d0bb-0de58c0ff7ae}" 2026-04-14T14:14:50.656Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:49.651088275 +0000 UTC m=+21.123274064 (1 nodes), 615.07187ms from start of poll, 615.353955ms from cluster start. Timeout at 2026-04-14 14:14:50.656090878 +0000 UTC m=+22.128276647, 1.005002583s after stability 2026-04-14T14:14:50.657Z [INFO] server-6051bb60-1b25-cb66-52ee-54a4a56a70b3: starting snapshot up to: index=12 2026-04-14T14:14:50.657Z [INFO] creating new snapshot: path=/tmp/raft3322974237/snapshots/2-12-1776176090657.tmp 2026-04-14T14:14:50.765Z [INFO] server-6051bb60-1b25-cb66-52ee-54a4a56a70b3: snapshot complete up to: index=12 2026-04-14T14:14:50.765Z [INFO] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b64472c-8658-50c4-63a7-84d323e4eaa3 Address:3b64472c-8658-50c4-63a7-84d323e4eaa3} {Suffrage:Voter ID:server-37ff153e-6d72-3828-8a6d-c9c9a9ce31e2 Address:37ff153e-6d72-3828-8a6d-c9c9a9ce31e2} {Suffrage:Voter ID:server-372d5da1-0400-6f00-a0c6-be882a9fa10c Address:372d5da1-0400-6f00-a0c6-be882a9fa10c}]" 2026-04-14T14:14:50.765Z [INFO] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: entering follower state: follower="Node at 3b64472c-8658-50c4-63a7-84d323e4eaa3 [Follower]" leader-address= leader-id= 2026-04-14T14:14:50.765Z [INFO] server-37ff153e-6d72-3828-8a6d-c9c9a9ce31e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b64472c-8658-50c4-63a7-84d323e4eaa3 Address:3b64472c-8658-50c4-63a7-84d323e4eaa3} {Suffrage:Voter ID:server-37ff153e-6d72-3828-8a6d-c9c9a9ce31e2 Address:37ff153e-6d72-3828-8a6d-c9c9a9ce31e2} {Suffrage:Voter ID:server-372d5da1-0400-6f00-a0c6-be882a9fa10c Address:372d5da1-0400-6f00-a0c6-be882a9fa10c}]" 2026-04-14T14:14:50.766Z [INFO] server-372d5da1-0400-6f00-a0c6-be882a9fa10c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b64472c-8658-50c4-63a7-84d323e4eaa3 Address:3b64472c-8658-50c4-63a7-84d323e4eaa3} {Suffrage:Voter ID:server-37ff153e-6d72-3828-8a6d-c9c9a9ce31e2 Address:37ff153e-6d72-3828-8a6d-c9c9a9ce31e2} {Suffrage:Voter ID:server-372d5da1-0400-6f00-a0c6-be882a9fa10c Address:372d5da1-0400-6f00-a0c6-be882a9fa10c}]" 2026-04-14T14:14:50.766Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:50.766Z [INFO] server-37ff153e-6d72-3828-8a6d-c9c9a9ce31e2: entering follower state: follower="Node at 37ff153e-6d72-3828-8a6d-c9c9a9ce31e2 [Follower]" leader-address= leader-id= 2026-04-14T14:14:50.766Z [INFO] server-372d5da1-0400-6f00-a0c6-be882a9fa10c: entering follower state: follower="Node at 372d5da1-0400-6f00-a0c6-be882a9fa10c [Follower]" leader-address= leader-id= 2026-04-14T14:14:51.577Z [WARN] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:51.577Z [INFO] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: entering candidate state: node="Node at 3b64472c-8658-50c4-63a7-84d323e4eaa3 [Candidate]" term=2 2026-04-14T14:14:51.577Z [INFO] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: election won: term=2 tally=2 2026-04-14T14:14:51.577Z [INFO] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: entering leader state: leader="Node at 3b64472c-8658-50c4-63a7-84d323e4eaa3 [Leader]" 2026-04-14T14:14:51.577Z [INFO] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: added peer, starting replication: peer=server-37ff153e-6d72-3828-8a6d-c9c9a9ce31e2 2026-04-14T14:14:51.577Z [INFO] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: added peer, starting replication: peer=server-372d5da1-0400-6f00-a0c6-be882a9fa10c 2026-04-14T14:14:51.577Z [INFO] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: pipelining replication: peer="{Voter server-372d5da1-0400-6f00-a0c6-be882a9fa10c 372d5da1-0400-6f00-a0c6-be882a9fa10c}" 2026-04-14T14:14:51.578Z [INFO] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: pipelining replication: peer="{Voter server-37ff153e-6d72-3828-8a6d-c9c9a9ce31e2 37ff153e-6d72-3828-8a6d-c9c9a9ce31e2}" 2026-04-14T14:14:52.583Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:51.57744452 +0000 UTC m=+23.049630286 (1 nodes), 811.353862ms from start of poll, 811.645905ms from cluster start. Timeout at 2026-04-14 14:14:52.582447591 +0000 UTC m=+24.054633358, 1.005003072s after stability 2026-04-14T14:14:52.583Z [INFO] creating new snapshot: path=/tmp/raft3984945538/snapshots/2-13-1776176092583.tmp 2026-04-14T14:14:52.671Z [INFO] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: copied to local snapshot: bytes=71 2026-04-14T14:14:52.671Z [INFO] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: snapshot restore progress: id=2-13-1776176092583 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:52.671Z [INFO] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: restored user snapshot: index=13 2026-04-14T14:14:52.671Z [ERROR] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: failed to get log: index=3 error="log not found" 2026-04-14T14:14:52.671Z [INFO] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: aborting pipeline replication: peer="{Voter server-372d5da1-0400-6f00-a0c6-be882a9fa10c 372d5da1-0400-6f00-a0c6-be882a9fa10c}" 2026-04-14T14:14:52.671Z [ERROR] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: failed to get log: index=3 error="log not found" 2026-04-14T14:14:52.672Z [INFO] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: aborting pipeline replication: peer="{Voter server-37ff153e-6d72-3828-8a6d-c9c9a9ce31e2 37ff153e-6d72-3828-8a6d-c9c9a9ce31e2}" 2026-04-14T14:14:52.681Z [ERROR] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: failed to get log: index=3 error="log not found" 2026-04-14T14:14:52.681Z [ERROR] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: failed to get log: index=3 error="log not found" 2026-04-14T14:14:52.681Z [INFO] creating new snapshot: path=/tmp/raft2985668030/snapshots/2-13-1776176092681.tmp 2026-04-14T14:14:52.681Z [INFO] creating new snapshot: path=/tmp/raft3910367128/snapshots/2-13-1776176092681.tmp 2026-04-14T14:14:52.681Z [INFO] server-37ff153e-6d72-3828-8a6d-c9c9a9ce31e2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:52.682Z [INFO] server-372d5da1-0400-6f00-a0c6-be882a9fa10c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:52.877Z [INFO] server-372d5da1-0400-6f00-a0c6-be882a9fa10c: copied to local snapshot: bytes=71 2026-04-14T14:14:52.877Z [INFO] server-37ff153e-6d72-3828-8a6d-c9c9a9ce31e2: copied to local snapshot: bytes=71 2026-04-14T14:14:52.878Z [INFO] server-37ff153e-6d72-3828-8a6d-c9c9a9ce31e2: snapshot restore progress: id=2-13-1776176092681 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:52.878Z [INFO] server-37ff153e-6d72-3828-8a6d-c9c9a9ce31e2: Installed remote snapshot 2026-04-14T14:14:52.878Z [INFO] server-372d5da1-0400-6f00-a0c6-be882a9fa10c: snapshot restore progress: id=2-13-1776176092681 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:52.878Z [INFO] server-372d5da1-0400-6f00-a0c6-be882a9fa10c: Installed remote snapshot 2026-04-14T14:14:52.878Z [INFO] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: pipelining replication: peer="{Voter server-37ff153e-6d72-3828-8a6d-c9c9a9ce31e2 37ff153e-6d72-3828-8a6d-c9c9a9ce31e2}" 2026-04-14T14:14:52.878Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:14:52.878Z [INFO] server-3b64472c-8658-50c4-63a7-84d323e4eaa3: pipelining replication: peer="{Voter server-372d5da1-0400-6f00-a0c6-be882a9fa10c 372d5da1-0400-6f00-a0c6-be882a9fa10c}" 2026-04-14T14:14:53.883Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:14:52.878487015 +0000 UTC m=+24.350672783 (2 nodes), 871ns from start of poll, 2.112688402s from cluster start. Timeout at 2026-04-14 14:14:53.883492636 +0000 UTC m=+25.355678405, 1.005005622s after stability 2026-04-14T14:14:53.895Z [INFO] server-6051bb60-1b25-cb66-52ee-54a4a56a70b3: aborting pipeline replication: peer="{Voter server-b5101e81-2e88-b556-e905-c022c0911aab b5101e81-2e88-b556-e905-c022c0911aab}" 2026-04-14T14:14:53.895Z [INFO] server-6051bb60-1b25-cb66-52ee-54a4a56a70b3: aborting pipeline replication: peer="{Voter server-ffe1821d-d3d5-c457-d0bb-0de58c0ff7ae ffe1821d-d3d5-c457-d0bb-0de58c0ff7ae}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true 2026-04-14T14:14:53.897Z [INFO] server-ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6 Address:ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6} {Suffrage:Voter ID:server-a6718353-d5ea-d73d-7e58-880b2689cad4 Address:a6718353-d5ea-d73d-7e58-880b2689cad4} {Suffrage:Voter ID:server-a8631b17-fbdd-3673-607e-51c0f4600fe3 Address:a8631b17-fbdd-3673-607e-51c0f4600fe3}]" 2026-04-14T14:14:53.897Z [INFO] server-ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6: entering follower state: follower="Node at ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6 [Follower]" leader-address= leader-id= 2026-04-14T14:14:53.897Z [INFO] server-a6718353-d5ea-d73d-7e58-880b2689cad4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6 Address:ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6} {Suffrage:Voter ID:server-a6718353-d5ea-d73d-7e58-880b2689cad4 Address:a6718353-d5ea-d73d-7e58-880b2689cad4} {Suffrage:Voter ID:server-a8631b17-fbdd-3673-607e-51c0f4600fe3 Address:a8631b17-fbdd-3673-607e-51c0f4600fe3}]" 2026-04-14T14:14:53.897Z [INFO] server-a6718353-d5ea-d73d-7e58-880b2689cad4: entering follower state: follower="Node at a6718353-d5ea-d73d-7e58-880b2689cad4 [Follower]" leader-address= leader-id= 2026-04-14T14:14:53.897Z [INFO] server-a8631b17-fbdd-3673-607e-51c0f4600fe3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6 Address:ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6} {Suffrage:Voter ID:server-a6718353-d5ea-d73d-7e58-880b2689cad4 Address:a6718353-d5ea-d73d-7e58-880b2689cad4} {Suffrage:Voter ID:server-a8631b17-fbdd-3673-607e-51c0f4600fe3 Address:a8631b17-fbdd-3673-607e-51c0f4600fe3}]" 2026-04-14T14:14:53.897Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:53.897Z [INFO] server-a8631b17-fbdd-3673-607e-51c0f4600fe3: entering follower state: follower="Node at a8631b17-fbdd-3673-607e-51c0f4600fe3 [Follower]" leader-address= leader-id= 2026-04-14T14:14:54.436Z [WARN] server-ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:54.436Z [INFO] server-ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6: entering candidate state: node="Node at ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6 [Candidate]" term=2 2026-04-14T14:14:54.436Z [INFO] server-ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6: election won: term=2 tally=2 2026-04-14T14:14:54.436Z [INFO] server-ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6: entering leader state: leader="Node at ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6 [Leader]" 2026-04-14T14:14:54.436Z [INFO] server-ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6: added peer, starting replication: peer=server-a6718353-d5ea-d73d-7e58-880b2689cad4 2026-04-14T14:14:54.436Z [INFO] server-ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6: added peer, starting replication: peer=server-a8631b17-fbdd-3673-607e-51c0f4600fe3 2026-04-14T14:14:54.436Z [INFO] server-ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6: pipelining replication: peer="{Voter server-a6718353-d5ea-d73d-7e58-880b2689cad4 a6718353-d5ea-d73d-7e58-880b2689cad4}" 2026-04-14T14:14:54.436Z [INFO] server-ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6: pipelining replication: peer="{Voter server-a8631b17-fbdd-3673-607e-51c0f4600fe3 a8631b17-fbdd-3673-607e-51c0f4600fe3}" 2026-04-14T14:14:55.441Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:54.436837229 +0000 UTC m=+25.909022997 (1 nodes), 539.253466ms from start of poll, 539.698374ms from cluster start. Timeout at 2026-04-14 14:14:55.441840893 +0000 UTC m=+26.914026662, 1.005003665s after stability 2026-04-14T14:14:55.443Z [INFO] server-ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6: starting snapshot up to: index=12 2026-04-14T14:14:55.443Z [INFO] creating new snapshot: path=/tmp/raft2840832797/snapshots/2-12-1776176095443.tmp 2026-04-14T14:14:55.444Z [INFO] server-ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6: snapshot complete up to: index=12 2026-04-14T14:14:55.445Z [INFO] server-da3321df-2b93-cfc1-3c31-21d8ffd87e47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da3321df-2b93-cfc1-3c31-21d8ffd87e47 Address:da3321df-2b93-cfc1-3c31-21d8ffd87e47} {Suffrage:Voter ID:server-ca070695-22d8-933f-86d1-8d195aeb9a6b Address:ca070695-22d8-933f-86d1-8d195aeb9a6b} {Suffrage:Voter ID:server-ffced5b9-8288-6bf6-e7bc-d04b30ce4069 Address:ffced5b9-8288-6bf6-e7bc-d04b30ce4069}]" 2026-04-14T14:14:55.445Z [INFO] server-da3321df-2b93-cfc1-3c31-21d8ffd87e47: entering follower state: follower="Node at da3321df-2b93-cfc1-3c31-21d8ffd87e47 [Follower]" leader-address= leader-id= 2026-04-14T14:14:55.445Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da3321df-2b93-cfc1-3c31-21d8ffd87e47 Address:da3321df-2b93-cfc1-3c31-21d8ffd87e47} {Suffrage:Voter ID:server-ca070695-22d8-933f-86d1-8d195aeb9a6b Address:ca070695-22d8-933f-86d1-8d195aeb9a6b} {Suffrage:Voter ID:server-ffced5b9-8288-6bf6-e7bc-d04b30ce4069 Address:ffced5b9-8288-6bf6-e7bc-d04b30ce4069}]" 2026-04-14T14:14:55.445Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: entering follower state: follower="Node at ca070695-22d8-933f-86d1-8d195aeb9a6b [Follower]" leader-address= leader-id= 2026-04-14T14:14:55.445Z [INFO] server-ffced5b9-8288-6bf6-e7bc-d04b30ce4069: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da3321df-2b93-cfc1-3c31-21d8ffd87e47 Address:da3321df-2b93-cfc1-3c31-21d8ffd87e47} {Suffrage:Voter ID:server-ca070695-22d8-933f-86d1-8d195aeb9a6b Address:ca070695-22d8-933f-86d1-8d195aeb9a6b} {Suffrage:Voter ID:server-ffced5b9-8288-6bf6-e7bc-d04b30ce4069 Address:ffced5b9-8288-6bf6-e7bc-d04b30ce4069}]" 2026-04-14T14:14:55.445Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:55.445Z [INFO] server-ffced5b9-8288-6bf6-e7bc-d04b30ce4069: entering follower state: follower="Node at ffced5b9-8288-6bf6-e7bc-d04b30ce4069 [Follower]" leader-address= leader-id= 2026-04-14T14:14:55.953Z [WARN] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:55.953Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: entering candidate state: node="Node at ca070695-22d8-933f-86d1-8d195aeb9a6b [Candidate]" term=2 2026-04-14T14:14:55.953Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: election won: term=2 tally=2 2026-04-14T14:14:55.953Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: entering leader state: leader="Node at ca070695-22d8-933f-86d1-8d195aeb9a6b [Leader]" 2026-04-14T14:14:55.953Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: added peer, starting replication: peer=server-da3321df-2b93-cfc1-3c31-21d8ffd87e47 2026-04-14T14:14:55.953Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: added peer, starting replication: peer=server-ffced5b9-8288-6bf6-e7bc-d04b30ce4069 2026-04-14T14:14:55.953Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: pipelining replication: peer="{Voter server-ffced5b9-8288-6bf6-e7bc-d04b30ce4069 ffced5b9-8288-6bf6-e7bc-d04b30ce4069}" 2026-04-14T14:14:55.953Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: pipelining replication: peer="{Voter server-da3321df-2b93-cfc1-3c31-21d8ffd87e47 da3321df-2b93-cfc1-3c31-21d8ffd87e47}" 2026-04-14T14:14:56.959Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:55.95352412 +0000 UTC m=+27.425709885 (1 nodes), 508.098284ms from start of poll, 508.484007ms from cluster start. Timeout at 2026-04-14 14:14:56.9585284 +0000 UTC m=+28.430714179, 1.005004294s after stability 2026-04-14T14:14:56.959Z [INFO] creating new snapshot: path=/tmp/raft4060836894/snapshots/2-13-1776176096959.tmp 2026-04-14T14:14:56.960Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: copied to local snapshot: bytes=71 2026-04-14T14:14:56.960Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: snapshot restore progress: id=2-13-1776176096959 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:56.960Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: removing all old logs from log store 2026-04-14T14:14:56.960Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: compacting logs: from=1 to=2 2026-04-14T14:14:56.961Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: restored user snapshot: index=13 2026-04-14T14:14:56.961Z [ERROR] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: failed to get log: index=2 error="log not found" 2026-04-14T14:14:56.961Z [ERROR] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: failed to get log: index=2 error="log not found" 2026-04-14T14:14:56.961Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: aborting pipeline replication: peer="{Voter server-da3321df-2b93-cfc1-3c31-21d8ffd87e47 da3321df-2b93-cfc1-3c31-21d8ffd87e47}" 2026-04-14T14:14:56.961Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: aborting pipeline replication: peer="{Voter server-ffced5b9-8288-6bf6-e7bc-d04b30ce4069 ffced5b9-8288-6bf6-e7bc-d04b30ce4069}" 2026-04-14T14:14:56.969Z [ERROR] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: failed to get log: index=2 error="log not found" 2026-04-14T14:14:56.969Z [ERROR] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: failed to get log: index=2 error="log not found" 2026-04-14T14:14:56.970Z [INFO] creating new snapshot: path=/tmp/raft413372277/snapshots/2-13-1776176096970.tmp 2026-04-14T14:14:56.970Z [INFO] creating new snapshot: path=/tmp/raft2255083951/snapshots/2-13-1776176096970.tmp 2026-04-14T14:14:56.970Z [INFO] server-da3321df-2b93-cfc1-3c31-21d8ffd87e47: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:56.970Z [INFO] server-ffced5b9-8288-6bf6-e7bc-d04b30ce4069: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:56.970Z [INFO] server-da3321df-2b93-cfc1-3c31-21d8ffd87e47: copied to local snapshot: bytes=71 2026-04-14T14:14:56.970Z [INFO] server-ffced5b9-8288-6bf6-e7bc-d04b30ce4069: copied to local snapshot: bytes=71 2026-04-14T14:14:56.970Z [INFO] server-da3321df-2b93-cfc1-3c31-21d8ffd87e47: snapshot restore progress: id=2-13-1776176096970 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:56.970Z [INFO] server-da3321df-2b93-cfc1-3c31-21d8ffd87e47: removing all old logs from log store 2026-04-14T14:14:56.970Z [INFO] server-da3321df-2b93-cfc1-3c31-21d8ffd87e47: compacting logs: from=1 to=2 2026-04-14T14:14:56.970Z [INFO] server-da3321df-2b93-cfc1-3c31-21d8ffd87e47: Installed remote snapshot 2026-04-14T14:14:56.971Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: pipelining replication: peer="{Voter server-da3321df-2b93-cfc1-3c31-21d8ffd87e47 da3321df-2b93-cfc1-3c31-21d8ffd87e47}" 2026-04-14T14:14:56.971Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:14:56.971Z [INFO] server-ffced5b9-8288-6bf6-e7bc-d04b30ce4069: snapshot restore progress: id=2-13-1776176096970 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:56.971Z [INFO] server-ffced5b9-8288-6bf6-e7bc-d04b30ce4069: removing all old logs from log store 2026-04-14T14:14:56.971Z [INFO] server-ffced5b9-8288-6bf6-e7bc-d04b30ce4069: compacting logs: from=1 to=2 2026-04-14T14:14:56.971Z [INFO] server-ffced5b9-8288-6bf6-e7bc-d04b30ce4069: Installed remote snapshot 2026-04-14T14:14:56.971Z [INFO] server-ca070695-22d8-933f-86d1-8d195aeb9a6b: pipelining replication: peer="{Voter server-ffced5b9-8288-6bf6-e7bc-d04b30ce4069 ffced5b9-8288-6bf6-e7bc-d04b30ce4069}" 2026-04-14T14:14:57.977Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:14:56.971057839 +0000 UTC m=+28.443243612 (2 nodes), 1.518µs from start of poll, 1.526017734s from cluster start. Timeout at 2026-04-14 14:14:57.976069319 +0000 UTC m=+29.448255085, 1.005011473s after stability 2026-04-14T14:14:57.989Z [INFO] server-ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6: aborting pipeline replication: peer="{Voter server-a8631b17-fbdd-3673-607e-51c0f4600fe3 a8631b17-fbdd-3673-607e-51c0f4600fe3}" 2026-04-14T14:14:57.989Z [INFO] server-ecdcbb5e-ca94-c9a7-1b45-c6db7f3840a6: aborting pipeline replication: peer="{Voter server-a6718353-d5ea-d73d-7e58-880b2689cad4 a6718353-d5ea-d73d-7e58-880b2689cad4}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false 2026-04-14T14:14:57.992Z [INFO] server-4beb320b-f4e3-8504-2d14-e6a8404a42d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4beb320b-f4e3-8504-2d14-e6a8404a42d8 Address:4beb320b-f4e3-8504-2d14-e6a8404a42d8} {Suffrage:Voter ID:server-659e9be3-0a95-5fdb-a383-24d440a149af Address:659e9be3-0a95-5fdb-a383-24d440a149af} {Suffrage:Voter ID:server-2334388f-3804-b5b0-3adb-8e8063a4915f Address:2334388f-3804-b5b0-3adb-8e8063a4915f}]" 2026-04-14T14:14:57.992Z [INFO] server-659e9be3-0a95-5fdb-a383-24d440a149af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4beb320b-f4e3-8504-2d14-e6a8404a42d8 Address:4beb320b-f4e3-8504-2d14-e6a8404a42d8} {Suffrage:Voter ID:server-659e9be3-0a95-5fdb-a383-24d440a149af Address:659e9be3-0a95-5fdb-a383-24d440a149af} {Suffrage:Voter ID:server-2334388f-3804-b5b0-3adb-8e8063a4915f Address:2334388f-3804-b5b0-3adb-8e8063a4915f}]" 2026-04-14T14:14:57.992Z [INFO] server-4beb320b-f4e3-8504-2d14-e6a8404a42d8: entering follower state: follower="Node at 4beb320b-f4e3-8504-2d14-e6a8404a42d8 [Follower]" leader-address= leader-id= 2026-04-14T14:14:57.992Z [INFO] server-659e9be3-0a95-5fdb-a383-24d440a149af: entering follower state: follower="Node at 659e9be3-0a95-5fdb-a383-24d440a149af [Follower]" leader-address= leader-id= 2026-04-14T14:14:57.992Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4beb320b-f4e3-8504-2d14-e6a8404a42d8 Address:4beb320b-f4e3-8504-2d14-e6a8404a42d8} {Suffrage:Voter ID:server-659e9be3-0a95-5fdb-a383-24d440a149af Address:659e9be3-0a95-5fdb-a383-24d440a149af} {Suffrage:Voter ID:server-2334388f-3804-b5b0-3adb-8e8063a4915f Address:2334388f-3804-b5b0-3adb-8e8063a4915f}]" 2026-04-14T14:14:57.992Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:14:57.992Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: entering follower state: follower="Node at 2334388f-3804-b5b0-3adb-8e8063a4915f [Follower]" leader-address= leader-id= 2026-04-14T14:14:58.565Z [WARN] server-2334388f-3804-b5b0-3adb-8e8063a4915f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:14:58.565Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: entering candidate state: node="Node at 2334388f-3804-b5b0-3adb-8e8063a4915f [Candidate]" term=2 2026-04-14T14:14:58.565Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: election won: term=2 tally=2 2026-04-14T14:14:58.565Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: entering leader state: leader="Node at 2334388f-3804-b5b0-3adb-8e8063a4915f [Leader]" 2026-04-14T14:14:58.565Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: added peer, starting replication: peer=server-4beb320b-f4e3-8504-2d14-e6a8404a42d8 2026-04-14T14:14:58.565Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: added peer, starting replication: peer=server-659e9be3-0a95-5fdb-a383-24d440a149af 2026-04-14T14:14:58.566Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: pipelining replication: peer="{Voter server-659e9be3-0a95-5fdb-a383-24d440a149af 659e9be3-0a95-5fdb-a383-24d440a149af}" 2026-04-14T14:14:58.566Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: pipelining replication: peer="{Voter server-4beb320b-f4e3-8504-2d14-e6a8404a42d8 4beb320b-f4e3-8504-2d14-e6a8404a42d8}" 2026-04-14T14:14:59.571Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:14:58.565952192 +0000 UTC m=+30.038137975 (1 nodes), 573.668564ms from start of poll, 573.985829ms from cluster start. Timeout at 2026-04-14 14:14:59.571016872 +0000 UTC m=+31.043202638, 1.005064663s after stability 2026-04-14T14:14:59.571Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: starting snapshot up to: index=12 2026-04-14T14:14:59.571Z [INFO] creating new snapshot: path=/tmp/raft2714978711/snapshots/2-12-1776176099571.tmp 2026-04-14T14:14:59.572Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: snapshot complete up to: index=12 2026-04-14T14:14:59.574Z [INFO] creating new snapshot: path=/tmp/raft2714978711/snapshots/2-23-1776176099574.tmp 2026-04-14T14:14:59.574Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: copied to local snapshot: bytes=71 2026-04-14T14:14:59.574Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: snapshot restore progress: id=2-23-1776176099574 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:59.574Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: restored user snapshot: index=23 2026-04-14T14:14:59.575Z [ERROR] server-2334388f-3804-b5b0-3adb-8e8063a4915f: failed to get log: index=23 error="log not found" 2026-04-14T14:14:59.575Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: aborting pipeline replication: peer="{Voter server-659e9be3-0a95-5fdb-a383-24d440a149af 659e9be3-0a95-5fdb-a383-24d440a149af}" 2026-04-14T14:14:59.575Z [ERROR] server-2334388f-3804-b5b0-3adb-8e8063a4915f: failed to get log: index=23 error="log not found" 2026-04-14T14:14:59.575Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: aborting pipeline replication: peer="{Voter server-4beb320b-f4e3-8504-2d14-e6a8404a42d8 4beb320b-f4e3-8504-2d14-e6a8404a42d8}" 2026-04-14T14:14:59.581Z [ERROR] server-2334388f-3804-b5b0-3adb-8e8063a4915f: failed to get log: index=23 error="log not found" 2026-04-14T14:14:59.582Z [INFO] creating new snapshot: path=/tmp/raft2616857768/snapshots/2-23-1776176099582.tmp 2026-04-14T14:14:59.582Z [INFO] server-659e9be3-0a95-5fdb-a383-24d440a149af: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:59.582Z [ERROR] server-2334388f-3804-b5b0-3adb-8e8063a4915f: failed to get log: index=23 error="log not found" 2026-04-14T14:14:59.582Z [INFO] creating new snapshot: path=/tmp/raft1832832468/snapshots/2-23-1776176099582.tmp 2026-04-14T14:14:59.583Z [INFO] server-659e9be3-0a95-5fdb-a383-24d440a149af: copied to local snapshot: bytes=71 2026-04-14T14:14:59.583Z [INFO] server-4beb320b-f4e3-8504-2d14-e6a8404a42d8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:59.584Z [INFO] server-659e9be3-0a95-5fdb-a383-24d440a149af: snapshot restore progress: id=2-23-1776176099582 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:59.584Z [INFO] server-659e9be3-0a95-5fdb-a383-24d440a149af: Installed remote snapshot 2026-04-14T14:14:59.584Z [INFO] server-4beb320b-f4e3-8504-2d14-e6a8404a42d8: copied to local snapshot: bytes=71 2026-04-14T14:14:59.584Z [WARN] server-659e9be3-0a95-5fdb-a383-24d440a149af: skipping application of old log: index=22 2026-04-14T14:14:59.584Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: pipelining replication: peer="{Voter server-659e9be3-0a95-5fdb-a383-24d440a149af 659e9be3-0a95-5fdb-a383-24d440a149af}" 2026-04-14T14:14:59.584Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:14:59.584Z [INFO] server-4beb320b-f4e3-8504-2d14-e6a8404a42d8: snapshot restore progress: id=2-23-1776176099582 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:14:59.584Z [INFO] server-4beb320b-f4e3-8504-2d14-e6a8404a42d8: Installed remote snapshot 2026-04-14T14:14:59.584Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: pipelining replication: peer="{Voter server-4beb320b-f4e3-8504-2d14-e6a8404a42d8 4beb320b-f4e3-8504-2d14-e6a8404a42d8}" 2026-04-14T14:15:00.590Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:14:59.584204881 +0000 UTC m=+31.056390663 (2 nodes), 728ns from start of poll, 1.592238517s from cluster start. Timeout at 2026-04-14 14:15:00.589214091 +0000 UTC m=+32.061399861, 1.005009198s after stability 2026-04-14T14:15:00.601Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: aborting pipeline replication: peer="{Voter server-659e9be3-0a95-5fdb-a383-24d440a149af 659e9be3-0a95-5fdb-a383-24d440a149af}" 2026-04-14T14:15:00.601Z [INFO] server-2334388f-3804-b5b0-3adb-8e8063a4915f: aborting pipeline replication: peer="{Voter server-4beb320b-f4e3-8504-2d14-e6a8404a42d8 4beb320b-f4e3-8504-2d14-e6a8404a42d8}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false 2026-04-14T14:15:00.602Z [INFO] server-88e97c2c-215a-bafc-f0f4-1aa7545bd4b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88e97c2c-215a-bafc-f0f4-1aa7545bd4b7 Address:88e97c2c-215a-bafc-f0f4-1aa7545bd4b7} {Suffrage:Voter ID:server-aac4549e-be13-f83f-021f-4487274648c9 Address:aac4549e-be13-f83f-021f-4487274648c9} {Suffrage:Voter ID:server-e8fb191c-fbd5-fe39-85d2-93be82a9a902 Address:e8fb191c-fbd5-fe39-85d2-93be82a9a902}]" 2026-04-14T14:15:00.602Z [INFO] server-88e97c2c-215a-bafc-f0f4-1aa7545bd4b7: entering follower state: follower="Node at 88e97c2c-215a-bafc-f0f4-1aa7545bd4b7 [Follower]" leader-address= leader-id= 2026-04-14T14:15:00.602Z [INFO] server-aac4549e-be13-f83f-021f-4487274648c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88e97c2c-215a-bafc-f0f4-1aa7545bd4b7 Address:88e97c2c-215a-bafc-f0f4-1aa7545bd4b7} {Suffrage:Voter ID:server-aac4549e-be13-f83f-021f-4487274648c9 Address:aac4549e-be13-f83f-021f-4487274648c9} {Suffrage:Voter ID:server-e8fb191c-fbd5-fe39-85d2-93be82a9a902 Address:e8fb191c-fbd5-fe39-85d2-93be82a9a902}]" 2026-04-14T14:15:00.602Z [INFO] server-aac4549e-be13-f83f-021f-4487274648c9: entering follower state: follower="Node at aac4549e-be13-f83f-021f-4487274648c9 [Follower]" leader-address= leader-id= 2026-04-14T14:15:00.603Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88e97c2c-215a-bafc-f0f4-1aa7545bd4b7 Address:88e97c2c-215a-bafc-f0f4-1aa7545bd4b7} {Suffrage:Voter ID:server-aac4549e-be13-f83f-021f-4487274648c9 Address:aac4549e-be13-f83f-021f-4487274648c9} {Suffrage:Voter ID:server-e8fb191c-fbd5-fe39-85d2-93be82a9a902 Address:e8fb191c-fbd5-fe39-85d2-93be82a9a902}]" 2026-04-14T14:15:00.603Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:00.603Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: entering follower state: follower="Node at e8fb191c-fbd5-fe39-85d2-93be82a9a902 [Follower]" leader-address= leader-id= 2026-04-14T14:15:01.242Z [WARN] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:01.242Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: entering candidate state: node="Node at e8fb191c-fbd5-fe39-85d2-93be82a9a902 [Candidate]" term=2 2026-04-14T14:15:01.242Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: election won: term=2 tally=2 2026-04-14T14:15:01.242Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: entering leader state: leader="Node at e8fb191c-fbd5-fe39-85d2-93be82a9a902 [Leader]" 2026-04-14T14:15:01.242Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: added peer, starting replication: peer=server-88e97c2c-215a-bafc-f0f4-1aa7545bd4b7 2026-04-14T14:15:01.242Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: added peer, starting replication: peer=server-aac4549e-be13-f83f-021f-4487274648c9 2026-04-14T14:15:01.242Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: pipelining replication: peer="{Voter server-aac4549e-be13-f83f-021f-4487274648c9 aac4549e-be13-f83f-021f-4487274648c9}" 2026-04-14T14:15:01.242Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: pipelining replication: peer="{Voter server-88e97c2c-215a-bafc-f0f4-1aa7545bd4b7 88e97c2c-215a-bafc-f0f4-1aa7545bd4b7}" 2026-04-14T14:15:02.248Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:01.242827654 +0000 UTC m=+32.715013422 (1 nodes), 639.801177ms from start of poll, 640.087568ms from cluster start. Timeout at 2026-04-14 14:15:02.247830502 +0000 UTC m=+33.720016271, 1.005002849s after stability 2026-04-14T14:15:02.249Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: starting snapshot up to: index=12 2026-04-14T14:15:02.249Z [INFO] creating new snapshot: path=/tmp/raft1052155925/snapshots/2-12-1776176102249.tmp 2026-04-14T14:15:02.249Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: snapshot complete up to: index=12 2026-04-14T14:15:02.250Z [INFO] creating new snapshot: path=/tmp/raft1052155925/snapshots/2-23-1776176102250.tmp 2026-04-14T14:15:02.250Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: copied to local snapshot: bytes=71 2026-04-14T14:15:02.251Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: snapshot restore progress: id=2-23-1776176102250 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:02.251Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: removing all old logs from log store 2026-04-14T14:15:02.251Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: compacting logs: from=1 to=22 2026-04-14T14:15:02.251Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: restored user snapshot: index=23 2026-04-14T14:15:02.251Z [ERROR] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: failed to get log: index=22 error="log not found" 2026-04-14T14:15:02.251Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: aborting pipeline replication: peer="{Voter server-aac4549e-be13-f83f-021f-4487274648c9 aac4549e-be13-f83f-021f-4487274648c9}" 2026-04-14T14:15:02.251Z [ERROR] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: failed to get log: index=22 error="log not found" 2026-04-14T14:15:02.251Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: aborting pipeline replication: peer="{Voter server-88e97c2c-215a-bafc-f0f4-1aa7545bd4b7 88e97c2c-215a-bafc-f0f4-1aa7545bd4b7}" 2026-04-14T14:15:02.258Z [ERROR] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: failed to get log: index=22 error="log not found" 2026-04-14T14:15:02.259Z [INFO] creating new snapshot: path=/tmp/raft4222968866/snapshots/2-23-1776176102259.tmp 2026-04-14T14:15:02.259Z [INFO] server-aac4549e-be13-f83f-021f-4487274648c9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:02.260Z [INFO] server-aac4549e-be13-f83f-021f-4487274648c9: copied to local snapshot: bytes=71 2026-04-14T14:15:02.260Z [INFO] server-aac4549e-be13-f83f-021f-4487274648c9: snapshot restore progress: id=2-23-1776176102259 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:02.260Z [INFO] server-aac4549e-be13-f83f-021f-4487274648c9: removing all old logs from log store 2026-04-14T14:15:02.260Z [INFO] server-aac4549e-be13-f83f-021f-4487274648c9: compacting logs: from=1 to=22 2026-04-14T14:15:02.260Z [INFO] server-aac4549e-be13-f83f-021f-4487274648c9: Installed remote snapshot 2026-04-14T14:15:02.260Z [WARN] server-aac4549e-be13-f83f-021f-4487274648c9: skipping application of old log: index=22 2026-04-14T14:15:02.260Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: pipelining replication: peer="{Voter server-aac4549e-be13-f83f-021f-4487274648c9 aac4549e-be13-f83f-021f-4487274648c9}" 2026-04-14T14:15:02.260Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:15:02.261Z [ERROR] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: failed to get log: index=22 error="log not found" 2026-04-14T14:15:02.262Z [INFO] creating new snapshot: path=/tmp/raft898091569/snapshots/2-23-1776176102262.tmp 2026-04-14T14:15:02.262Z [INFO] server-88e97c2c-215a-bafc-f0f4-1aa7545bd4b7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:02.262Z [INFO] server-88e97c2c-215a-bafc-f0f4-1aa7545bd4b7: copied to local snapshot: bytes=71 2026-04-14T14:15:02.262Z [INFO] server-88e97c2c-215a-bafc-f0f4-1aa7545bd4b7: snapshot restore progress: id=2-23-1776176102262 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:02.262Z [INFO] server-88e97c2c-215a-bafc-f0f4-1aa7545bd4b7: removing all old logs from log store 2026-04-14T14:15:02.262Z [INFO] server-88e97c2c-215a-bafc-f0f4-1aa7545bd4b7: compacting logs: from=1 to=22 2026-04-14T14:15:02.262Z [INFO] server-88e97c2c-215a-bafc-f0f4-1aa7545bd4b7: Installed remote snapshot 2026-04-14T14:15:02.262Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: pipelining replication: peer="{Voter server-88e97c2c-215a-bafc-f0f4-1aa7545bd4b7 88e97c2c-215a-bafc-f0f4-1aa7545bd4b7}" 2026-04-14T14:15:03.266Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:15:02.260329667 +0000 UTC m=+33.732515433 (2 nodes), 659ns from start of poll, 1.657589579s from cluster start. Timeout at 2026-04-14 14:15:03.265338757 +0000 UTC m=+34.737524523, 1.00500909s after stability 2026-04-14T14:15:03.278Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: aborting pipeline replication: peer="{Voter server-aac4549e-be13-f83f-021f-4487274648c9 aac4549e-be13-f83f-021f-4487274648c9}" 2026-04-14T14:15:03.278Z [INFO] server-e8fb191c-fbd5-fe39-85d2-93be82a9a902: aborting pipeline replication: peer="{Voter server-88e97c2c-215a-bafc-f0f4-1aa7545bd4b7 88e97c2c-215a-bafc-f0f4-1aa7545bd4b7}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true 2026-04-14T14:15:03.279Z [INFO] server-73b1338d-3342-45a6-8299-c97cc93d6f61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73b1338d-3342-45a6-8299-c97cc93d6f61 Address:73b1338d-3342-45a6-8299-c97cc93d6f61} {Suffrage:Voter ID:server-136b5543-de73-3d48-5cca-e7b84acf6f87 Address:136b5543-de73-3d48-5cca-e7b84acf6f87} {Suffrage:Voter ID:server-44e7dccf-beb1-f358-11c2-dd02ec50aac4 Address:44e7dccf-beb1-f358-11c2-dd02ec50aac4}]" 2026-04-14T14:15:03.279Z [INFO] server-73b1338d-3342-45a6-8299-c97cc93d6f61: entering follower state: follower="Node at 73b1338d-3342-45a6-8299-c97cc93d6f61 [Follower]" leader-address= leader-id= 2026-04-14T14:15:03.279Z [INFO] server-136b5543-de73-3d48-5cca-e7b84acf6f87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73b1338d-3342-45a6-8299-c97cc93d6f61 Address:73b1338d-3342-45a6-8299-c97cc93d6f61} {Suffrage:Voter ID:server-136b5543-de73-3d48-5cca-e7b84acf6f87 Address:136b5543-de73-3d48-5cca-e7b84acf6f87} {Suffrage:Voter ID:server-44e7dccf-beb1-f358-11c2-dd02ec50aac4 Address:44e7dccf-beb1-f358-11c2-dd02ec50aac4}]" 2026-04-14T14:15:03.279Z [INFO] server-44e7dccf-beb1-f358-11c2-dd02ec50aac4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73b1338d-3342-45a6-8299-c97cc93d6f61 Address:73b1338d-3342-45a6-8299-c97cc93d6f61} {Suffrage:Voter ID:server-136b5543-de73-3d48-5cca-e7b84acf6f87 Address:136b5543-de73-3d48-5cca-e7b84acf6f87} {Suffrage:Voter ID:server-44e7dccf-beb1-f358-11c2-dd02ec50aac4 Address:44e7dccf-beb1-f358-11c2-dd02ec50aac4}]" 2026-04-14T14:15:03.279Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:03.279Z [INFO] server-136b5543-de73-3d48-5cca-e7b84acf6f87: entering follower state: follower="Node at 136b5543-de73-3d48-5cca-e7b84acf6f87 [Follower]" leader-address= leader-id= 2026-04-14T14:15:03.279Z [INFO] server-44e7dccf-beb1-f358-11c2-dd02ec50aac4: entering follower state: follower="Node at 44e7dccf-beb1-f358-11c2-dd02ec50aac4 [Follower]" leader-address= leader-id= 2026-04-14T14:15:03.897Z [WARN] server-136b5543-de73-3d48-5cca-e7b84acf6f87: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:03.897Z [INFO] server-136b5543-de73-3d48-5cca-e7b84acf6f87: entering candidate state: node="Node at 136b5543-de73-3d48-5cca-e7b84acf6f87 [Candidate]" term=2 2026-04-14T14:15:03.897Z [INFO] server-136b5543-de73-3d48-5cca-e7b84acf6f87: election won: term=2 tally=2 2026-04-14T14:15:03.897Z [INFO] server-136b5543-de73-3d48-5cca-e7b84acf6f87: entering leader state: leader="Node at 136b5543-de73-3d48-5cca-e7b84acf6f87 [Leader]" 2026-04-14T14:15:03.897Z [INFO] server-136b5543-de73-3d48-5cca-e7b84acf6f87: added peer, starting replication: peer=server-73b1338d-3342-45a6-8299-c97cc93d6f61 2026-04-14T14:15:03.897Z [INFO] server-136b5543-de73-3d48-5cca-e7b84acf6f87: added peer, starting replication: peer=server-44e7dccf-beb1-f358-11c2-dd02ec50aac4 2026-04-14T14:15:03.897Z [INFO] server-136b5543-de73-3d48-5cca-e7b84acf6f87: pipelining replication: peer="{Voter server-44e7dccf-beb1-f358-11c2-dd02ec50aac4 44e7dccf-beb1-f358-11c2-dd02ec50aac4}" 2026-04-14T14:15:03.897Z [INFO] server-136b5543-de73-3d48-5cca-e7b84acf6f87: pipelining replication: peer="{Voter server-73b1338d-3342-45a6-8299-c97cc93d6f61 73b1338d-3342-45a6-8299-c97cc93d6f61}" 2026-04-14T14:15:04.903Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:03.897718736 +0000 UTC m=+35.369904521 (1 nodes), 617.777498ms from start of poll, 618.062564ms from cluster start. Timeout at 2026-04-14 14:15:04.902721259 +0000 UTC m=+36.374907096, 1.005002575s after stability 2026-04-14T14:15:04.905Z [INFO] server-136b5543-de73-3d48-5cca-e7b84acf6f87: starting snapshot up to: index=12 2026-04-14T14:15:04.905Z [INFO] creating new snapshot: path=/tmp/raft2748129587/snapshots/2-12-1776176104905.tmp 2026-04-14T14:15:05.015Z [INFO] server-136b5543-de73-3d48-5cca-e7b84acf6f87: snapshot complete up to: index=12 2026-04-14T14:15:05.016Z [INFO] server-bfb4e5d8-8ec4-12e3-556e-a50969472546: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfb4e5d8-8ec4-12e3-556e-a50969472546 Address:bfb4e5d8-8ec4-12e3-556e-a50969472546} {Suffrage:Voter ID:server-8b1850ac-0f35-168a-904d-ffbe23560f66 Address:8b1850ac-0f35-168a-904d-ffbe23560f66} {Suffrage:Voter ID:server-d394e46c-410e-c43a-08d1-28a6184d66d3 Address:d394e46c-410e-c43a-08d1-28a6184d66d3}]" 2026-04-14T14:15:05.016Z [INFO] server-bfb4e5d8-8ec4-12e3-556e-a50969472546: entering follower state: follower="Node at bfb4e5d8-8ec4-12e3-556e-a50969472546 [Follower]" leader-address= leader-id= 2026-04-14T14:15:05.016Z [INFO] server-8b1850ac-0f35-168a-904d-ffbe23560f66: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfb4e5d8-8ec4-12e3-556e-a50969472546 Address:bfb4e5d8-8ec4-12e3-556e-a50969472546} {Suffrage:Voter ID:server-8b1850ac-0f35-168a-904d-ffbe23560f66 Address:8b1850ac-0f35-168a-904d-ffbe23560f66} {Suffrage:Voter ID:server-d394e46c-410e-c43a-08d1-28a6184d66d3 Address:d394e46c-410e-c43a-08d1-28a6184d66d3}]" 2026-04-14T14:15:05.016Z [INFO] server-8b1850ac-0f35-168a-904d-ffbe23560f66: entering follower state: follower="Node at 8b1850ac-0f35-168a-904d-ffbe23560f66 [Follower]" leader-address= leader-id= 2026-04-14T14:15:05.016Z [INFO] server-d394e46c-410e-c43a-08d1-28a6184d66d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfb4e5d8-8ec4-12e3-556e-a50969472546 Address:bfb4e5d8-8ec4-12e3-556e-a50969472546} {Suffrage:Voter ID:server-8b1850ac-0f35-168a-904d-ffbe23560f66 Address:8b1850ac-0f35-168a-904d-ffbe23560f66} {Suffrage:Voter ID:server-d394e46c-410e-c43a-08d1-28a6184d66d3 Address:d394e46c-410e-c43a-08d1-28a6184d66d3}]" 2026-04-14T14:15:05.016Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:05.016Z [INFO] server-d394e46c-410e-c43a-08d1-28a6184d66d3: entering follower state: follower="Node at d394e46c-410e-c43a-08d1-28a6184d66d3 [Follower]" leader-address= leader-id= 2026-04-14T14:15:05.691Z [WARN] server-8b1850ac-0f35-168a-904d-ffbe23560f66: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:05.691Z [INFO] server-8b1850ac-0f35-168a-904d-ffbe23560f66: entering candidate state: node="Node at 8b1850ac-0f35-168a-904d-ffbe23560f66 [Candidate]" term=2 2026-04-14T14:15:05.691Z [INFO] server-8b1850ac-0f35-168a-904d-ffbe23560f66: election won: term=2 tally=2 2026-04-14T14:15:05.691Z [INFO] server-8b1850ac-0f35-168a-904d-ffbe23560f66: entering leader state: leader="Node at 8b1850ac-0f35-168a-904d-ffbe23560f66 [Leader]" 2026-04-14T14:15:05.691Z [INFO] server-8b1850ac-0f35-168a-904d-ffbe23560f66: added peer, starting replication: peer=server-bfb4e5d8-8ec4-12e3-556e-a50969472546 2026-04-14T14:15:05.692Z [INFO] server-8b1850ac-0f35-168a-904d-ffbe23560f66: added peer, starting replication: peer=server-d394e46c-410e-c43a-08d1-28a6184d66d3 2026-04-14T14:15:05.692Z [INFO] server-8b1850ac-0f35-168a-904d-ffbe23560f66: pipelining replication: peer="{Voter server-bfb4e5d8-8ec4-12e3-556e-a50969472546 bfb4e5d8-8ec4-12e3-556e-a50969472546}" 2026-04-14T14:15:05.693Z [INFO] server-8b1850ac-0f35-168a-904d-ffbe23560f66: pipelining replication: peer="{Voter server-d394e46c-410e-c43a-08d1-28a6184d66d3 d394e46c-410e-c43a-08d1-28a6184d66d3}" 2026-04-14T14:15:06.697Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:05.691993249 +0000 UTC m=+37.164179016 (1 nodes), 675.737771ms from start of poll, 676.064584ms from cluster start. Timeout at 2026-04-14 14:15:06.696998287 +0000 UTC m=+38.169184055, 1.005005039s after stability 2026-04-14T14:15:06.698Z [INFO] creating new snapshot: path=/tmp/raft1406761216/snapshots/2-14-1776176106698.tmp 2026-04-14T14:15:06.792Z [INFO] server-8b1850ac-0f35-168a-904d-ffbe23560f66: copied to local snapshot: bytes=71 2026-04-14T14:15:06.792Z [INFO] server-8b1850ac-0f35-168a-904d-ffbe23560f66: snapshot restore progress: id=2-14-1776176106698 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:06.792Z [INFO] server-8b1850ac-0f35-168a-904d-ffbe23560f66: restored user snapshot: index=14 2026-04-14T14:15:06.792Z [ERROR] server-8b1850ac-0f35-168a-904d-ffbe23560f66: failed to get log: index=3 error="log not found" 2026-04-14T14:15:06.792Z [INFO] server-8b1850ac-0f35-168a-904d-ffbe23560f66: aborting pipeline replication: peer="{Voter server-d394e46c-410e-c43a-08d1-28a6184d66d3 d394e46c-410e-c43a-08d1-28a6184d66d3}" 2026-04-14T14:15:06.792Z [ERROR] server-8b1850ac-0f35-168a-904d-ffbe23560f66: failed to get log: index=3 error="log not found" 2026-04-14T14:15:06.792Z [INFO] server-8b1850ac-0f35-168a-904d-ffbe23560f66: aborting pipeline replication: peer="{Voter server-bfb4e5d8-8ec4-12e3-556e-a50969472546 bfb4e5d8-8ec4-12e3-556e-a50969472546}" 2026-04-14T14:15:06.800Z [ERROR] server-8b1850ac-0f35-168a-904d-ffbe23560f66: failed to get log: index=3 error="log not found" 2026-04-14T14:15:06.801Z [ERROR] server-8b1850ac-0f35-168a-904d-ffbe23560f66: failed to get log: index=3 error="log not found" 2026-04-14T14:15:06.802Z [INFO] creating new snapshot: path=/tmp/raft1880514056/snapshots/2-14-1776176106802.tmp 2026-04-14T14:15:06.879Z [INFO] server-bfb4e5d8-8ec4-12e3-556e-a50969472546: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:06.879Z [INFO] creating new snapshot: path=/tmp/raft1281423790/snapshots/2-14-1776176106879.tmp 2026-04-14T14:15:06.879Z [INFO] server-bfb4e5d8-8ec4-12e3-556e-a50969472546: copied to local snapshot: bytes=71 2026-04-14T14:15:06.879Z [INFO] server-d394e46c-410e-c43a-08d1-28a6184d66d3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:06.879Z [INFO] server-bfb4e5d8-8ec4-12e3-556e-a50969472546: snapshot restore progress: id=2-14-1776176106802 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:06.879Z [INFO] server-bfb4e5d8-8ec4-12e3-556e-a50969472546: Installed remote snapshot 2026-04-14T14:15:06.879Z [INFO] server-d394e46c-410e-c43a-08d1-28a6184d66d3: copied to local snapshot: bytes=71 2026-04-14T14:15:06.880Z [INFO] server-8b1850ac-0f35-168a-904d-ffbe23560f66: pipelining replication: peer="{Voter server-bfb4e5d8-8ec4-12e3-556e-a50969472546 bfb4e5d8-8ec4-12e3-556e-a50969472546}" 2026-04-14T14:15:06.880Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:15:06.880Z [INFO] server-d394e46c-410e-c43a-08d1-28a6184d66d3: snapshot restore progress: id=2-14-1776176106879 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:06.880Z [INFO] server-d394e46c-410e-c43a-08d1-28a6184d66d3: Installed remote snapshot 2026-04-14T14:15:06.880Z [INFO] server-8b1850ac-0f35-168a-904d-ffbe23560f66: pipelining replication: peer="{Voter server-d394e46c-410e-c43a-08d1-28a6184d66d3 d394e46c-410e-c43a-08d1-28a6184d66d3}" 2026-04-14T14:15:07.886Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:15:06.880157201 +0000 UTC m=+38.352342976 (2 nodes), 839ns from start of poll, 1.864228544s from cluster start. Timeout at 2026-04-14 14:15:07.88516335 +0000 UTC m=+39.357349115, 1.005006139s after stability 2026-04-14T14:15:07.897Z [INFO] server-136b5543-de73-3d48-5cca-e7b84acf6f87: aborting pipeline replication: peer="{Voter server-44e7dccf-beb1-f358-11c2-dd02ec50aac4 44e7dccf-beb1-f358-11c2-dd02ec50aac4}" 2026-04-14T14:15:07.897Z [INFO] server-136b5543-de73-3d48-5cca-e7b84acf6f87: aborting pipeline replication: peer="{Voter server-73b1338d-3342-45a6-8299-c97cc93d6f61 73b1338d-3342-45a6-8299-c97cc93d6f61}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true 2026-04-14T14:15:07.944Z [INFO] server-d52cd430-ed26-3963-754d-e9284f0b67a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d52cd430-ed26-3963-754d-e9284f0b67a7 Address:d52cd430-ed26-3963-754d-e9284f0b67a7} {Suffrage:Voter ID:server-845ea58d-ce30-015a-05c8-577ddb1730ba Address:845ea58d-ce30-015a-05c8-577ddb1730ba} {Suffrage:Voter ID:server-4c19f85d-2501-8ebf-cca7-cb57a8fe0399 Address:4c19f85d-2501-8ebf-cca7-cb57a8fe0399}]" 2026-04-14T14:15:07.945Z [INFO] server-d52cd430-ed26-3963-754d-e9284f0b67a7: entering follower state: follower="Node at d52cd430-ed26-3963-754d-e9284f0b67a7 [Follower]" leader-address= leader-id= 2026-04-14T14:15:07.945Z [INFO] server-845ea58d-ce30-015a-05c8-577ddb1730ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d52cd430-ed26-3963-754d-e9284f0b67a7 Address:d52cd430-ed26-3963-754d-e9284f0b67a7} {Suffrage:Voter ID:server-845ea58d-ce30-015a-05c8-577ddb1730ba Address:845ea58d-ce30-015a-05c8-577ddb1730ba} {Suffrage:Voter ID:server-4c19f85d-2501-8ebf-cca7-cb57a8fe0399 Address:4c19f85d-2501-8ebf-cca7-cb57a8fe0399}]" 2026-04-14T14:15:07.945Z [INFO] server-4c19f85d-2501-8ebf-cca7-cb57a8fe0399: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d52cd430-ed26-3963-754d-e9284f0b67a7 Address:d52cd430-ed26-3963-754d-e9284f0b67a7} {Suffrage:Voter ID:server-845ea58d-ce30-015a-05c8-577ddb1730ba Address:845ea58d-ce30-015a-05c8-577ddb1730ba} {Suffrage:Voter ID:server-4c19f85d-2501-8ebf-cca7-cb57a8fe0399 Address:4c19f85d-2501-8ebf-cca7-cb57a8fe0399}]" 2026-04-14T14:15:07.945Z [INFO] server-845ea58d-ce30-015a-05c8-577ddb1730ba: entering follower state: follower="Node at 845ea58d-ce30-015a-05c8-577ddb1730ba [Follower]" leader-address= leader-id= 2026-04-14T14:15:07.945Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:07.945Z [INFO] server-4c19f85d-2501-8ebf-cca7-cb57a8fe0399: entering follower state: follower="Node at 4c19f85d-2501-8ebf-cca7-cb57a8fe0399 [Follower]" leader-address= leader-id= 2026-04-14T14:15:08.459Z [WARN] server-4c19f85d-2501-8ebf-cca7-cb57a8fe0399: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:08.459Z [INFO] server-4c19f85d-2501-8ebf-cca7-cb57a8fe0399: entering candidate state: node="Node at 4c19f85d-2501-8ebf-cca7-cb57a8fe0399 [Candidate]" term=2 2026-04-14T14:15:08.459Z [INFO] server-4c19f85d-2501-8ebf-cca7-cb57a8fe0399: election won: term=2 tally=2 2026-04-14T14:15:08.459Z [INFO] server-4c19f85d-2501-8ebf-cca7-cb57a8fe0399: entering leader state: leader="Node at 4c19f85d-2501-8ebf-cca7-cb57a8fe0399 [Leader]" 2026-04-14T14:15:08.459Z [INFO] server-4c19f85d-2501-8ebf-cca7-cb57a8fe0399: added peer, starting replication: peer=server-d52cd430-ed26-3963-754d-e9284f0b67a7 2026-04-14T14:15:08.459Z [INFO] server-4c19f85d-2501-8ebf-cca7-cb57a8fe0399: added peer, starting replication: peer=server-845ea58d-ce30-015a-05c8-577ddb1730ba 2026-04-14T14:15:08.459Z [INFO] server-4c19f85d-2501-8ebf-cca7-cb57a8fe0399: pipelining replication: peer="{Voter server-845ea58d-ce30-015a-05c8-577ddb1730ba 845ea58d-ce30-015a-05c8-577ddb1730ba}" 2026-04-14T14:15:08.459Z [INFO] server-4c19f85d-2501-8ebf-cca7-cb57a8fe0399: pipelining replication: peer="{Voter server-d52cd430-ed26-3963-754d-e9284f0b67a7 d52cd430-ed26-3963-754d-e9284f0b67a7}" 2026-04-14T14:15:09.464Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:08.459497589 +0000 UTC m=+39.931683353 (1 nodes), 514.33605ms from start of poll, 514.660911ms from cluster start. Timeout at 2026-04-14 14:15:09.464498242 +0000 UTC m=+40.936684005, 1.005000652s after stability 2026-04-14T14:15:09.465Z [INFO] server-4c19f85d-2501-8ebf-cca7-cb57a8fe0399: starting snapshot up to: index=12 2026-04-14T14:15:09.465Z [INFO] creating new snapshot: path=/tmp/raft3483629486/snapshots/2-12-1776176109465.tmp 2026-04-14T14:15:09.470Z [INFO] server-4c19f85d-2501-8ebf-cca7-cb57a8fe0399: snapshot complete up to: index=12 2026-04-14T14:15:09.471Z [INFO] server-e9644354-1722-ef51-2ca4-3d9045ae7d9e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9644354-1722-ef51-2ca4-3d9045ae7d9e Address:e9644354-1722-ef51-2ca4-3d9045ae7d9e} {Suffrage:Voter ID:server-28ac69d0-f5e0-9460-c220-c8f4e15d0787 Address:28ac69d0-f5e0-9460-c220-c8f4e15d0787} {Suffrage:Voter ID:server-3792d005-cebd-4897-506f-7b7c2b2380fa Address:3792d005-cebd-4897-506f-7b7c2b2380fa}]" 2026-04-14T14:15:09.471Z [INFO] server-e9644354-1722-ef51-2ca4-3d9045ae7d9e: entering follower state: follower="Node at e9644354-1722-ef51-2ca4-3d9045ae7d9e [Follower]" leader-address= leader-id= 2026-04-14T14:15:09.471Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9644354-1722-ef51-2ca4-3d9045ae7d9e Address:e9644354-1722-ef51-2ca4-3d9045ae7d9e} {Suffrage:Voter ID:server-28ac69d0-f5e0-9460-c220-c8f4e15d0787 Address:28ac69d0-f5e0-9460-c220-c8f4e15d0787} {Suffrage:Voter ID:server-3792d005-cebd-4897-506f-7b7c2b2380fa Address:3792d005-cebd-4897-506f-7b7c2b2380fa}]" 2026-04-14T14:15:09.471Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: entering follower state: follower="Node at 28ac69d0-f5e0-9460-c220-c8f4e15d0787 [Follower]" leader-address= leader-id= 2026-04-14T14:15:09.471Z [INFO] server-3792d005-cebd-4897-506f-7b7c2b2380fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9644354-1722-ef51-2ca4-3d9045ae7d9e Address:e9644354-1722-ef51-2ca4-3d9045ae7d9e} {Suffrage:Voter ID:server-28ac69d0-f5e0-9460-c220-c8f4e15d0787 Address:28ac69d0-f5e0-9460-c220-c8f4e15d0787} {Suffrage:Voter ID:server-3792d005-cebd-4897-506f-7b7c2b2380fa Address:3792d005-cebd-4897-506f-7b7c2b2380fa}]" 2026-04-14T14:15:09.471Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:09.471Z [INFO] server-3792d005-cebd-4897-506f-7b7c2b2380fa: entering follower state: follower="Node at 3792d005-cebd-4897-506f-7b7c2b2380fa [Follower]" leader-address= leader-id= 2026-04-14T14:15:10.060Z [WARN] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:10.060Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: entering candidate state: node="Node at 28ac69d0-f5e0-9460-c220-c8f4e15d0787 [Candidate]" term=2 2026-04-14T14:15:10.060Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: election won: term=2 tally=2 2026-04-14T14:15:10.060Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: entering leader state: leader="Node at 28ac69d0-f5e0-9460-c220-c8f4e15d0787 [Leader]" 2026-04-14T14:15:10.060Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: added peer, starting replication: peer=server-e9644354-1722-ef51-2ca4-3d9045ae7d9e 2026-04-14T14:15:10.060Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: added peer, starting replication: peer=server-3792d005-cebd-4897-506f-7b7c2b2380fa 2026-04-14T14:15:10.060Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: pipelining replication: peer="{Voter server-e9644354-1722-ef51-2ca4-3d9045ae7d9e e9644354-1722-ef51-2ca4-3d9045ae7d9e}" 2026-04-14T14:15:10.060Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: pipelining replication: peer="{Voter server-3792d005-cebd-4897-506f-7b7c2b2380fa 3792d005-cebd-4897-506f-7b7c2b2380fa}" 2026-04-14T14:15:11.066Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:10.060581528 +0000 UTC m=+41.532767514 (1 nodes), 588.791661ms from start of poll, 589.138109ms from cluster start. Timeout at 2026-04-14 14:15:11.065586024 +0000 UTC m=+42.537771811, 1.005004297s after stability 2026-04-14T14:15:11.066Z [INFO] creating new snapshot: path=/tmp/raft1976938298/snapshots/2-14-1776176111066.tmp 2026-04-14T14:15:11.067Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: copied to local snapshot: bytes=71 2026-04-14T14:15:11.067Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: snapshot restore progress: id=2-14-1776176111066 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:11.067Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: removing all old logs from log store 2026-04-14T14:15:11.067Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: compacting logs: from=1 to=2 2026-04-14T14:15:11.067Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: restored user snapshot: index=14 2026-04-14T14:15:11.067Z [ERROR] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: failed to get log: index=2 error="log not found" 2026-04-14T14:15:11.067Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: aborting pipeline replication: peer="{Voter server-3792d005-cebd-4897-506f-7b7c2b2380fa 3792d005-cebd-4897-506f-7b7c2b2380fa}" 2026-04-14T14:15:11.067Z [ERROR] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: failed to get log: index=2 error="log not found" 2026-04-14T14:15:11.067Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: aborting pipeline replication: peer="{Voter server-e9644354-1722-ef51-2ca4-3d9045ae7d9e e9644354-1722-ef51-2ca4-3d9045ae7d9e}" 2026-04-14T14:15:11.074Z [ERROR] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: failed to get log: index=2 error="log not found" 2026-04-14T14:15:11.074Z [INFO] creating new snapshot: path=/tmp/raft3713937642/snapshots/2-14-1776176111074.tmp 2026-04-14T14:15:11.074Z [INFO] server-e9644354-1722-ef51-2ca4-3d9045ae7d9e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:11.075Z [INFO] server-e9644354-1722-ef51-2ca4-3d9045ae7d9e: copied to local snapshot: bytes=71 2026-04-14T14:15:11.075Z [ERROR] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: failed to get log: index=2 error="log not found" 2026-04-14T14:15:11.075Z [INFO] server-e9644354-1722-ef51-2ca4-3d9045ae7d9e: snapshot restore progress: id=2-14-1776176111074 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:11.075Z [INFO] server-e9644354-1722-ef51-2ca4-3d9045ae7d9e: removing all old logs from log store 2026-04-14T14:15:11.075Z [INFO] server-e9644354-1722-ef51-2ca4-3d9045ae7d9e: compacting logs: from=1 to=2 2026-04-14T14:15:11.075Z [INFO] server-e9644354-1722-ef51-2ca4-3d9045ae7d9e: Installed remote snapshot 2026-04-14T14:15:11.075Z [INFO] creating new snapshot: path=/tmp/raft1653641623/snapshots/2-14-1776176111075.tmp 2026-04-14T14:15:11.075Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: pipelining replication: peer="{Voter server-e9644354-1722-ef51-2ca4-3d9045ae7d9e e9644354-1722-ef51-2ca4-3d9045ae7d9e}" 2026-04-14T14:15:11.075Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:15:11.075Z [INFO] server-3792d005-cebd-4897-506f-7b7c2b2380fa: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:11.075Z [INFO] server-3792d005-cebd-4897-506f-7b7c2b2380fa: copied to local snapshot: bytes=71 2026-04-14T14:15:11.076Z [INFO] server-3792d005-cebd-4897-506f-7b7c2b2380fa: snapshot restore progress: id=2-14-1776176111075 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:11.076Z [INFO] server-3792d005-cebd-4897-506f-7b7c2b2380fa: removing all old logs from log store 2026-04-14T14:15:11.076Z [INFO] server-3792d005-cebd-4897-506f-7b7c2b2380fa: compacting logs: from=1 to=2 2026-04-14T14:15:11.076Z [INFO] server-3792d005-cebd-4897-506f-7b7c2b2380fa: Installed remote snapshot 2026-04-14T14:15:11.076Z [INFO] server-28ac69d0-f5e0-9460-c220-c8f4e15d0787: pipelining replication: peer="{Voter server-3792d005-cebd-4897-506f-7b7c2b2380fa 3792d005-cebd-4897-506f-7b7c2b2380fa}" 2026-04-14T14:15:12.081Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:15:11.075543543 +0000 UTC m=+42.547729327 (2 nodes), 729ns from start of poll, 1.604099922s from cluster start. Timeout at 2026-04-14 14:15:12.080553561 +0000 UTC m=+43.552739329, 1.005010002s after stability 2026-04-14T14:15:12.092Z [INFO] server-4c19f85d-2501-8ebf-cca7-cb57a8fe0399: aborting pipeline replication: peer="{Voter server-845ea58d-ce30-015a-05c8-577ddb1730ba 845ea58d-ce30-015a-05c8-577ddb1730ba}" 2026-04-14T14:15:12.092Z [INFO] server-4c19f85d-2501-8ebf-cca7-cb57a8fe0399: aborting pipeline replication: peer="{Voter server-d52cd430-ed26-3963-754d-e9284f0b67a7 d52cd430-ed26-3963-754d-e9284f0b67a7}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false 2026-04-14T14:15:12.093Z [INFO] server-0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7 Address:0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7} {Suffrage:Voter ID:server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5 Address:57bcd3c6-c4e6-a65b-a52e-b91942e09bc5} {Suffrage:Voter ID:server-803a0ccb-e3c3-5ea7-bde8-41970100a627 Address:803a0ccb-e3c3-5ea7-bde8-41970100a627}]" 2026-04-14T14:15:12.093Z [INFO] server-0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7: entering follower state: follower="Node at 0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7 [Follower]" leader-address= leader-id= 2026-04-14T14:15:12.094Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7 Address:0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7} {Suffrage:Voter ID:server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5 Address:57bcd3c6-c4e6-a65b-a52e-b91942e09bc5} {Suffrage:Voter ID:server-803a0ccb-e3c3-5ea7-bde8-41970100a627 Address:803a0ccb-e3c3-5ea7-bde8-41970100a627}]" 2026-04-14T14:15:12.094Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: entering follower state: follower="Node at 57bcd3c6-c4e6-a65b-a52e-b91942e09bc5 [Follower]" leader-address= leader-id= 2026-04-14T14:15:12.094Z [INFO] server-803a0ccb-e3c3-5ea7-bde8-41970100a627: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7 Address:0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7} {Suffrage:Voter ID:server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5 Address:57bcd3c6-c4e6-a65b-a52e-b91942e09bc5} {Suffrage:Voter ID:server-803a0ccb-e3c3-5ea7-bde8-41970100a627 Address:803a0ccb-e3c3-5ea7-bde8-41970100a627}]" 2026-04-14T14:15:12.094Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:12.094Z [INFO] server-803a0ccb-e3c3-5ea7-bde8-41970100a627: entering follower state: follower="Node at 803a0ccb-e3c3-5ea7-bde8-41970100a627 [Follower]" leader-address= leader-id= 2026-04-14T14:15:12.637Z [WARN] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:12.637Z [WARN] server-803a0ccb-e3c3-5ea7-bde8-41970100a627: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:12.637Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: entering candidate state: node="Node at 57bcd3c6-c4e6-a65b-a52e-b91942e09bc5 [Candidate]" term=2 2026-04-14T14:15:12.637Z [INFO] server-803a0ccb-e3c3-5ea7-bde8-41970100a627: entering candidate state: node="Node at 803a0ccb-e3c3-5ea7-bde8-41970100a627 [Candidate]" term=2 2026-04-14T14:15:12.637Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: duplicate requestVote for same term: term=2 2026-04-14T14:15:12.637Z [INFO] server-803a0ccb-e3c3-5ea7-bde8-41970100a627: duplicate requestVote for same term: term=2 2026-04-14T14:15:12.637Z [INFO] server-0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7: duplicate requestVote for same term: term=2 2026-04-14T14:15:12.637Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: election won: term=2 tally=2 2026-04-14T14:15:12.637Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: entering leader state: leader="Node at 57bcd3c6-c4e6-a65b-a52e-b91942e09bc5 [Leader]" 2026-04-14T14:15:12.637Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: added peer, starting replication: peer=server-0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7 2026-04-14T14:15:12.637Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: added peer, starting replication: peer=server-803a0ccb-e3c3-5ea7-bde8-41970100a627 2026-04-14T14:15:12.637Z [INFO] server-803a0ccb-e3c3-5ea7-bde8-41970100a627: entering follower state: follower="Node at 803a0ccb-e3c3-5ea7-bde8-41970100a627 [Follower]" leader-address=57bcd3c6-c4e6-a65b-a52e-b91942e09bc5 leader-id=server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5 2026-04-14T14:15:12.637Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: pipelining replication: peer="{Voter server-803a0ccb-e3c3-5ea7-bde8-41970100a627 803a0ccb-e3c3-5ea7-bde8-41970100a627}" 2026-04-14T14:15:12.637Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: pipelining replication: peer="{Voter server-0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7 0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7}" 2026-04-14T14:15:13.643Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:12.637748228 +0000 UTC m=+44.109934012 (1 nodes), 543.632877ms from start of poll, 543.945978ms from cluster start. Timeout at 2026-04-14 14:15:13.642749254 +0000 UTC m=+45.114935023, 1.005001011s after stability 2026-04-14T14:15:13.643Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: starting snapshot up to: index=12 2026-04-14T14:15:13.643Z [INFO] creating new snapshot: path=/tmp/raft1221575930/snapshots/2-12-1776176113643.tmp 2026-04-14T14:15:13.679Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: snapshot complete up to: index=12 2026-04-14T14:15:13.680Z [INFO] creating new snapshot: path=/tmp/raft1221575930/snapshots/2-23-1776176113680.tmp 2026-04-14T14:15:13.680Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: copied to local snapshot: bytes=71 2026-04-14T14:15:13.681Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: snapshot restore progress: id=2-23-1776176113680 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:13.681Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: restored user snapshot: index=23 2026-04-14T14:15:13.681Z [ERROR] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: failed to get log: index=23 error="log not found" 2026-04-14T14:15:13.681Z [ERROR] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: failed to get log: index=23 error="log not found" 2026-04-14T14:15:13.681Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: aborting pipeline replication: peer="{Voter server-803a0ccb-e3c3-5ea7-bde8-41970100a627 803a0ccb-e3c3-5ea7-bde8-41970100a627}" 2026-04-14T14:15:13.681Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: aborting pipeline replication: peer="{Voter server-0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7 0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7}" 2026-04-14T14:15:13.687Z [ERROR] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: failed to get log: index=23 error="log not found" 2026-04-14T14:15:13.687Z [INFO] creating new snapshot: path=/tmp/raft4051985078/snapshots/2-23-1776176113687.tmp 2026-04-14T14:15:13.687Z [INFO] server-0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:13.690Z [ERROR] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: failed to get log: index=23 error="log not found" 2026-04-14T14:15:13.691Z [INFO] creating new snapshot: path=/tmp/raft606177043/snapshots/2-23-1776176113691.tmp 2026-04-14T14:15:13.958Z [INFO] server-0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7: copied to local snapshot: bytes=71 2026-04-14T14:15:13.958Z [INFO] server-803a0ccb-e3c3-5ea7-bde8-41970100a627: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:13.958Z [INFO] server-0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7: snapshot restore progress: id=2-23-1776176113687 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:13.958Z [INFO] server-0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7: Installed remote snapshot 2026-04-14T14:15:13.958Z [INFO] server-803a0ccb-e3c3-5ea7-bde8-41970100a627: copied to local snapshot: bytes=71 2026-04-14T14:15:13.958Z [WARN] server-0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7: skipping application of old log: index=23 2026-04-14T14:15:13.958Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: pipelining replication: peer="{Voter server-0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7 0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7}" 2026-04-14T14:15:13.958Z [INFO] server-803a0ccb-e3c3-5ea7-bde8-41970100a627: snapshot restore progress: id=2-23-1776176113691 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:13.958Z [INFO] server-803a0ccb-e3c3-5ea7-bde8-41970100a627: Installed remote snapshot 2026-04-14T14:15:13.958Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:15:13.958Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: pipelining replication: peer="{Voter server-803a0ccb-e3c3-5ea7-bde8-41970100a627 803a0ccb-e3c3-5ea7-bde8-41970100a627}" 2026-04-14T14:15:14.964Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:15:13.958708249 +0000 UTC m=+45.430894017 (2 nodes), 731ns from start of poll, 1.864905983s from cluster start. Timeout at 2026-04-14 14:15:14.963728376 +0000 UTC m=+46.435914144, 1.005020127s after stability 2026-04-14T14:15:14.975Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: aborting pipeline replication: peer="{Voter server-803a0ccb-e3c3-5ea7-bde8-41970100a627 803a0ccb-e3c3-5ea7-bde8-41970100a627}" 2026-04-14T14:15:14.975Z [INFO] server-57bcd3c6-c4e6-a65b-a52e-b91942e09bc5: aborting pipeline replication: peer="{Voter server-0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7 0e236e6f-2d35-24f5-c231-d9ea0b1fdfc7}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false 2026-04-14T14:15:14.978Z [INFO] server-bb047f61-9782-102a-97a9-1c630d08edb3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb047f61-9782-102a-97a9-1c630d08edb3 Address:bb047f61-9782-102a-97a9-1c630d08edb3} {Suffrage:Voter ID:server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15 Address:26a5e906-f6d6-08e0-1c0f-bba3abb86e15} {Suffrage:Voter ID:server-221ee348-4359-3f94-1ac9-4d3ee57bdae0 Address:221ee348-4359-3f94-1ac9-4d3ee57bdae0}]" 2026-04-14T14:15:14.978Z [INFO] server-bb047f61-9782-102a-97a9-1c630d08edb3: entering follower state: follower="Node at bb047f61-9782-102a-97a9-1c630d08edb3 [Follower]" leader-address= leader-id= 2026-04-14T14:15:14.978Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb047f61-9782-102a-97a9-1c630d08edb3 Address:bb047f61-9782-102a-97a9-1c630d08edb3} {Suffrage:Voter ID:server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15 Address:26a5e906-f6d6-08e0-1c0f-bba3abb86e15} {Suffrage:Voter ID:server-221ee348-4359-3f94-1ac9-4d3ee57bdae0 Address:221ee348-4359-3f94-1ac9-4d3ee57bdae0}]" 2026-04-14T14:15:14.978Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: entering follower state: follower="Node at 26a5e906-f6d6-08e0-1c0f-bba3abb86e15 [Follower]" leader-address= leader-id= 2026-04-14T14:15:14.978Z [INFO] server-221ee348-4359-3f94-1ac9-4d3ee57bdae0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb047f61-9782-102a-97a9-1c630d08edb3 Address:bb047f61-9782-102a-97a9-1c630d08edb3} {Suffrage:Voter ID:server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15 Address:26a5e906-f6d6-08e0-1c0f-bba3abb86e15} {Suffrage:Voter ID:server-221ee348-4359-3f94-1ac9-4d3ee57bdae0 Address:221ee348-4359-3f94-1ac9-4d3ee57bdae0}]" 2026-04-14T14:15:14.978Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:14.978Z [INFO] server-221ee348-4359-3f94-1ac9-4d3ee57bdae0: entering follower state: follower="Node at 221ee348-4359-3f94-1ac9-4d3ee57bdae0 [Follower]" leader-address= leader-id= 2026-04-14T14:15:15.554Z [WARN] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:15.554Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: entering candidate state: node="Node at 26a5e906-f6d6-08e0-1c0f-bba3abb86e15 [Candidate]" term=2 2026-04-14T14:15:15.555Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: election won: term=2 tally=2 2026-04-14T14:15:15.555Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: entering leader state: leader="Node at 26a5e906-f6d6-08e0-1c0f-bba3abb86e15 [Leader]" 2026-04-14T14:15:15.555Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: added peer, starting replication: peer=server-bb047f61-9782-102a-97a9-1c630d08edb3 2026-04-14T14:15:15.555Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: added peer, starting replication: peer=server-221ee348-4359-3f94-1ac9-4d3ee57bdae0 2026-04-14T14:15:15.555Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: pipelining replication: peer="{Voter server-221ee348-4359-3f94-1ac9-4d3ee57bdae0 221ee348-4359-3f94-1ac9-4d3ee57bdae0}" 2026-04-14T14:15:15.555Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: pipelining replication: peer="{Voter server-bb047f61-9782-102a-97a9-1c630d08edb3 bb047f61-9782-102a-97a9-1c630d08edb3}" 2026-04-14T14:15:16.560Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:15.555077102 +0000 UTC m=+47.027262874 (1 nodes), 576.372284ms from start of poll, 576.69829ms from cluster start. Timeout at 2026-04-14 14:15:16.560082349 +0000 UTC m=+48.032268112, 1.005005238s after stability 2026-04-14T14:15:16.561Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: starting snapshot up to: index=12 2026-04-14T14:15:16.561Z [INFO] creating new snapshot: path=/tmp/raft3657430703/snapshots/2-12-1776176116561.tmp 2026-04-14T14:15:16.622Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: snapshot complete up to: index=12 2026-04-14T14:15:16.623Z [INFO] creating new snapshot: path=/tmp/raft3657430703/snapshots/2-23-1776176116623.tmp 2026-04-14T14:15:16.624Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: copied to local snapshot: bytes=71 2026-04-14T14:15:16.624Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: snapshot restore progress: id=2-23-1776176116623 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:16.624Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: removing all old logs from log store 2026-04-14T14:15:16.624Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: compacting logs: from=1 to=22 2026-04-14T14:15:16.624Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: restored user snapshot: index=23 2026-04-14T14:15:16.624Z [ERROR] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: failed to get log: index=22 error="log not found" 2026-04-14T14:15:16.624Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: aborting pipeline replication: peer="{Voter server-221ee348-4359-3f94-1ac9-4d3ee57bdae0 221ee348-4359-3f94-1ac9-4d3ee57bdae0}" 2026-04-14T14:15:16.624Z [ERROR] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: failed to get log: index=22 error="log not found" 2026-04-14T14:15:16.624Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: aborting pipeline replication: peer="{Voter server-bb047f61-9782-102a-97a9-1c630d08edb3 bb047f61-9782-102a-97a9-1c630d08edb3}" 2026-04-14T14:15:16.631Z [ERROR] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: failed to get log: index=22 error="log not found" 2026-04-14T14:15:16.631Z [INFO] creating new snapshot: path=/tmp/raft3599680503/snapshots/2-23-1776176116631.tmp 2026-04-14T14:15:16.632Z [INFO] server-221ee348-4359-3f94-1ac9-4d3ee57bdae0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:16.633Z [ERROR] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: failed to get log: index=22 error="log not found" 2026-04-14T14:15:16.634Z [INFO] creating new snapshot: path=/tmp/raft3746503692/snapshots/2-23-1776176116634.tmp 2026-04-14T14:15:16.634Z [INFO] server-bb047f61-9782-102a-97a9-1c630d08edb3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:16.807Z [INFO] server-221ee348-4359-3f94-1ac9-4d3ee57bdae0: copied to local snapshot: bytes=71 2026-04-14T14:15:16.807Z [INFO] server-bb047f61-9782-102a-97a9-1c630d08edb3: copied to local snapshot: bytes=71 2026-04-14T14:15:16.807Z [INFO] server-221ee348-4359-3f94-1ac9-4d3ee57bdae0: snapshot restore progress: id=2-23-1776176116631 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:16.807Z [INFO] server-221ee348-4359-3f94-1ac9-4d3ee57bdae0: removing all old logs from log store 2026-04-14T14:15:16.807Z [INFO] server-bb047f61-9782-102a-97a9-1c630d08edb3: snapshot restore progress: id=2-23-1776176116634 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:16.807Z [INFO] server-221ee348-4359-3f94-1ac9-4d3ee57bdae0: compacting logs: from=1 to=22 2026-04-14T14:15:16.807Z [INFO] server-bb047f61-9782-102a-97a9-1c630d08edb3: removing all old logs from log store 2026-04-14T14:15:16.807Z [INFO] server-221ee348-4359-3f94-1ac9-4d3ee57bdae0: Installed remote snapshot 2026-04-14T14:15:16.807Z [INFO] server-bb047f61-9782-102a-97a9-1c630d08edb3: compacting logs: from=1 to=22 2026-04-14T14:15:16.807Z [INFO] server-bb047f61-9782-102a-97a9-1c630d08edb3: Installed remote snapshot 2026-04-14T14:15:16.807Z [WARN] server-221ee348-4359-3f94-1ac9-4d3ee57bdae0: skipping application of old log: index=22 2026-04-14T14:15:16.807Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: pipelining replication: peer="{Voter server-221ee348-4359-3f94-1ac9-4d3ee57bdae0 221ee348-4359-3f94-1ac9-4d3ee57bdae0}" 2026-04-14T14:15:16.807Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:15:16.807Z [WARN] server-bb047f61-9782-102a-97a9-1c630d08edb3: skipping application of old log: index=23 2026-04-14T14:15:16.807Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: pipelining replication: peer="{Voter server-bb047f61-9782-102a-97a9-1c630d08edb3 bb047f61-9782-102a-97a9-1c630d08edb3}" 2026-04-14T14:15:17.813Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:15:16.807798459 +0000 UTC m=+48.279984223 (2 nodes), 709ns from start of poll, 1.829419639s from cluster start. Timeout at 2026-04-14 14:15:17.812806086 +0000 UTC m=+49.284991850, 1.005007627s after stability 2026-04-14T14:15:17.825Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: aborting pipeline replication: peer="{Voter server-221ee348-4359-3f94-1ac9-4d3ee57bdae0 221ee348-4359-3f94-1ac9-4d3ee57bdae0}" 2026-04-14T14:15:17.825Z [INFO] server-26a5e906-f6d6-08e0-1c0f-bba3abb86e15: aborting pipeline replication: peer="{Voter server-bb047f61-9782-102a-97a9-1c630d08edb3 bb047f61-9782-102a-97a9-1c630d08edb3}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true 2026-04-14T14:15:18.051Z [INFO] server-854e3dbb-8cde-333a-de66-1d0a6cb5ed7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-854e3dbb-8cde-333a-de66-1d0a6cb5ed7f Address:854e3dbb-8cde-333a-de66-1d0a6cb5ed7f} {Suffrage:Voter ID:server-d2744bb0-1d83-5c6c-ac9a-959b2c314cf7 Address:d2744bb0-1d83-5c6c-ac9a-959b2c314cf7} {Suffrage:Voter ID:server-80800e1f-da4b-8b3c-d47e-ef05b3a77789 Address:80800e1f-da4b-8b3c-d47e-ef05b3a77789}]" 2026-04-14T14:15:18.052Z [INFO] server-854e3dbb-8cde-333a-de66-1d0a6cb5ed7f: entering follower state: follower="Node at 854e3dbb-8cde-333a-de66-1d0a6cb5ed7f [Follower]" leader-address= leader-id= 2026-04-14T14:15:18.052Z [INFO] server-d2744bb0-1d83-5c6c-ac9a-959b2c314cf7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-854e3dbb-8cde-333a-de66-1d0a6cb5ed7f Address:854e3dbb-8cde-333a-de66-1d0a6cb5ed7f} {Suffrage:Voter ID:server-d2744bb0-1d83-5c6c-ac9a-959b2c314cf7 Address:d2744bb0-1d83-5c6c-ac9a-959b2c314cf7} {Suffrage:Voter ID:server-80800e1f-da4b-8b3c-d47e-ef05b3a77789 Address:80800e1f-da4b-8b3c-d47e-ef05b3a77789}]" 2026-04-14T14:15:18.052Z [INFO] server-80800e1f-da4b-8b3c-d47e-ef05b3a77789: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-854e3dbb-8cde-333a-de66-1d0a6cb5ed7f Address:854e3dbb-8cde-333a-de66-1d0a6cb5ed7f} {Suffrage:Voter ID:server-d2744bb0-1d83-5c6c-ac9a-959b2c314cf7 Address:d2744bb0-1d83-5c6c-ac9a-959b2c314cf7} {Suffrage:Voter ID:server-80800e1f-da4b-8b3c-d47e-ef05b3a77789 Address:80800e1f-da4b-8b3c-d47e-ef05b3a77789}]" 2026-04-14T14:15:18.052Z [INFO] server-d2744bb0-1d83-5c6c-ac9a-959b2c314cf7: entering follower state: follower="Node at d2744bb0-1d83-5c6c-ac9a-959b2c314cf7 [Follower]" leader-address= leader-id= 2026-04-14T14:15:18.052Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:18.052Z [INFO] server-80800e1f-da4b-8b3c-d47e-ef05b3a77789: entering follower state: follower="Node at 80800e1f-da4b-8b3c-d47e-ef05b3a77789 [Follower]" leader-address= leader-id= 2026-04-14T14:15:18.587Z [WARN] server-854e3dbb-8cde-333a-de66-1d0a6cb5ed7f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:18.587Z [INFO] server-854e3dbb-8cde-333a-de66-1d0a6cb5ed7f: entering candidate state: node="Node at 854e3dbb-8cde-333a-de66-1d0a6cb5ed7f [Candidate]" term=2 2026-04-14T14:15:18.587Z [INFO] server-854e3dbb-8cde-333a-de66-1d0a6cb5ed7f: election won: term=2 tally=2 2026-04-14T14:15:18.587Z [INFO] server-854e3dbb-8cde-333a-de66-1d0a6cb5ed7f: entering leader state: leader="Node at 854e3dbb-8cde-333a-de66-1d0a6cb5ed7f [Leader]" 2026-04-14T14:15:18.587Z [INFO] server-854e3dbb-8cde-333a-de66-1d0a6cb5ed7f: added peer, starting replication: peer=server-d2744bb0-1d83-5c6c-ac9a-959b2c314cf7 2026-04-14T14:15:18.587Z [INFO] server-854e3dbb-8cde-333a-de66-1d0a6cb5ed7f: added peer, starting replication: peer=server-80800e1f-da4b-8b3c-d47e-ef05b3a77789 2026-04-14T14:15:18.587Z [INFO] server-854e3dbb-8cde-333a-de66-1d0a6cb5ed7f: pipelining replication: peer="{Voter server-d2744bb0-1d83-5c6c-ac9a-959b2c314cf7 d2744bb0-1d83-5c6c-ac9a-959b2c314cf7}" 2026-04-14T14:15:18.587Z [INFO] server-854e3dbb-8cde-333a-de66-1d0a6cb5ed7f: pipelining replication: peer="{Voter server-80800e1f-da4b-8b3c-d47e-ef05b3a77789 80800e1f-da4b-8b3c-d47e-ef05b3a77789}" 2026-04-14T14:15:19.593Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:18.587462978 +0000 UTC m=+50.059648760 (1 nodes), 535.27638ms from start of poll, 535.653692ms from cluster start. Timeout at 2026-04-14 14:15:19.592467408 +0000 UTC m=+51.064653175, 1.005004415s after stability 2026-04-14T14:15:19.593Z [INFO] server-854e3dbb-8cde-333a-de66-1d0a6cb5ed7f: starting snapshot up to: index=12 2026-04-14T14:15:19.593Z [INFO] creating new snapshot: path=/tmp/raft352334413/snapshots/2-12-1776176119593.tmp 2026-04-14T14:15:19.594Z [INFO] server-854e3dbb-8cde-333a-de66-1d0a6cb5ed7f: snapshot complete up to: index=12 2026-04-14T14:15:19.595Z [INFO] server-ed30009e-b8fb-f562-783c-a3df1168b703: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed30009e-b8fb-f562-783c-a3df1168b703 Address:ed30009e-b8fb-f562-783c-a3df1168b703} {Suffrage:Voter ID:server-03723536-dbbe-2edc-1ae7-919ae3ad9b91 Address:03723536-dbbe-2edc-1ae7-919ae3ad9b91} {Suffrage:Voter ID:server-a1fbb328-199b-c29d-b57c-bdfc14b32371 Address:a1fbb328-199b-c29d-b57c-bdfc14b32371}]" 2026-04-14T14:15:19.595Z [INFO] server-ed30009e-b8fb-f562-783c-a3df1168b703: entering follower state: follower="Node at ed30009e-b8fb-f562-783c-a3df1168b703 [Follower]" leader-address= leader-id= 2026-04-14T14:15:19.595Z [INFO] server-03723536-dbbe-2edc-1ae7-919ae3ad9b91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed30009e-b8fb-f562-783c-a3df1168b703 Address:ed30009e-b8fb-f562-783c-a3df1168b703} {Suffrage:Voter ID:server-03723536-dbbe-2edc-1ae7-919ae3ad9b91 Address:03723536-dbbe-2edc-1ae7-919ae3ad9b91} {Suffrage:Voter ID:server-a1fbb328-199b-c29d-b57c-bdfc14b32371 Address:a1fbb328-199b-c29d-b57c-bdfc14b32371}]" 2026-04-14T14:15:19.595Z [INFO] server-a1fbb328-199b-c29d-b57c-bdfc14b32371: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed30009e-b8fb-f562-783c-a3df1168b703 Address:ed30009e-b8fb-f562-783c-a3df1168b703} {Suffrage:Voter ID:server-03723536-dbbe-2edc-1ae7-919ae3ad9b91 Address:03723536-dbbe-2edc-1ae7-919ae3ad9b91} {Suffrage:Voter ID:server-a1fbb328-199b-c29d-b57c-bdfc14b32371 Address:a1fbb328-199b-c29d-b57c-bdfc14b32371}]" 2026-04-14T14:15:19.595Z [INFO] server-03723536-dbbe-2edc-1ae7-919ae3ad9b91: entering follower state: follower="Node at 03723536-dbbe-2edc-1ae7-919ae3ad9b91 [Follower]" leader-address= leader-id= 2026-04-14T14:15:19.595Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:19.595Z [INFO] server-a1fbb328-199b-c29d-b57c-bdfc14b32371: entering follower state: follower="Node at a1fbb328-199b-c29d-b57c-bdfc14b32371 [Follower]" leader-address= leader-id= 2026-04-14T14:15:20.211Z [WARN] server-ed30009e-b8fb-f562-783c-a3df1168b703: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:20.211Z [INFO] server-ed30009e-b8fb-f562-783c-a3df1168b703: entering candidate state: node="Node at ed30009e-b8fb-f562-783c-a3df1168b703 [Candidate]" term=2 2026-04-14T14:15:20.211Z [INFO] server-ed30009e-b8fb-f562-783c-a3df1168b703: election won: term=2 tally=2 2026-04-14T14:15:20.211Z [INFO] server-ed30009e-b8fb-f562-783c-a3df1168b703: entering leader state: leader="Node at ed30009e-b8fb-f562-783c-a3df1168b703 [Leader]" 2026-04-14T14:15:20.211Z [INFO] server-ed30009e-b8fb-f562-783c-a3df1168b703: added peer, starting replication: peer=server-03723536-dbbe-2edc-1ae7-919ae3ad9b91 2026-04-14T14:15:20.211Z [INFO] server-ed30009e-b8fb-f562-783c-a3df1168b703: added peer, starting replication: peer=server-a1fbb328-199b-c29d-b57c-bdfc14b32371 2026-04-14T14:15:20.211Z [INFO] server-ed30009e-b8fb-f562-783c-a3df1168b703: pipelining replication: peer="{Voter server-03723536-dbbe-2edc-1ae7-919ae3ad9b91 03723536-dbbe-2edc-1ae7-919ae3ad9b91}" 2026-04-14T14:15:20.211Z [INFO] server-ed30009e-b8fb-f562-783c-a3df1168b703: pipelining replication: peer="{Voter server-a1fbb328-199b-c29d-b57c-bdfc14b32371 a1fbb328-199b-c29d-b57c-bdfc14b32371}" 2026-04-14T14:15:21.217Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:20.211888934 +0000 UTC m=+51.684074703 (1 nodes), 616.204559ms from start of poll, 616.586924ms from cluster start. Timeout at 2026-04-14 14:15:21.216902811 +0000 UTC m=+52.689088579, 1.005013876s after stability 2026-04-14T14:15:21.217Z [INFO] creating new snapshot: path=/tmp/raft3884805066/snapshots/2-15-1776176121217.tmp 2026-04-14T14:15:21.489Z [INFO] server-ed30009e-b8fb-f562-783c-a3df1168b703: copied to local snapshot: bytes=71 2026-04-14T14:15:21.489Z [INFO] server-ed30009e-b8fb-f562-783c-a3df1168b703: snapshot restore progress: id=2-15-1776176121217 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:21.489Z [INFO] server-ed30009e-b8fb-f562-783c-a3df1168b703: restored user snapshot: index=15 2026-04-14T14:15:21.489Z [ERROR] server-ed30009e-b8fb-f562-783c-a3df1168b703: failed to get log: index=3 error="log not found" 2026-04-14T14:15:21.489Z [ERROR] server-ed30009e-b8fb-f562-783c-a3df1168b703: failed to get log: index=3 error="log not found" 2026-04-14T14:15:21.489Z [INFO] server-ed30009e-b8fb-f562-783c-a3df1168b703: aborting pipeline replication: peer="{Voter server-03723536-dbbe-2edc-1ae7-919ae3ad9b91 03723536-dbbe-2edc-1ae7-919ae3ad9b91}" 2026-04-14T14:15:21.489Z [INFO] server-ed30009e-b8fb-f562-783c-a3df1168b703: aborting pipeline replication: peer="{Voter server-a1fbb328-199b-c29d-b57c-bdfc14b32371 a1fbb328-199b-c29d-b57c-bdfc14b32371}" 2026-04-14T14:15:21.495Z [ERROR] server-ed30009e-b8fb-f562-783c-a3df1168b703: failed to get log: index=3 error="log not found" 2026-04-14T14:15:21.496Z [ERROR] server-ed30009e-b8fb-f562-783c-a3df1168b703: failed to get log: index=3 error="log not found" 2026-04-14T14:15:21.630Z [INFO] creating new snapshot: path=/tmp/raft1783845619/snapshots/2-15-1776176121630.tmp 2026-04-14T14:15:21.630Z [INFO] creating new snapshot: path=/tmp/raft1811222072/snapshots/2-15-1776176121630.tmp 2026-04-14T14:15:21.630Z [INFO] server-a1fbb328-199b-c29d-b57c-bdfc14b32371: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:21.630Z [INFO] server-03723536-dbbe-2edc-1ae7-919ae3ad9b91: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:21.630Z [INFO] server-a1fbb328-199b-c29d-b57c-bdfc14b32371: copied to local snapshot: bytes=71 2026-04-14T14:15:21.630Z [INFO] server-03723536-dbbe-2edc-1ae7-919ae3ad9b91: copied to local snapshot: bytes=71 2026-04-14T14:15:21.630Z [INFO] server-03723536-dbbe-2edc-1ae7-919ae3ad9b91: snapshot restore progress: id=2-15-1776176121630 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:21.630Z [INFO] server-03723536-dbbe-2edc-1ae7-919ae3ad9b91: Installed remote snapshot 2026-04-14T14:15:21.630Z [INFO] server-a1fbb328-199b-c29d-b57c-bdfc14b32371: snapshot restore progress: id=2-15-1776176121630 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:21.630Z [INFO] server-a1fbb328-199b-c29d-b57c-bdfc14b32371: Installed remote snapshot 2026-04-14T14:15:21.630Z [INFO] server-ed30009e-b8fb-f562-783c-a3df1168b703: pipelining replication: peer="{Voter server-03723536-dbbe-2edc-1ae7-919ae3ad9b91 03723536-dbbe-2edc-1ae7-919ae3ad9b91}" 2026-04-14T14:15:21.630Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:15:21.630Z [INFO] server-ed30009e-b8fb-f562-783c-a3df1168b703: pipelining replication: peer="{Voter server-a1fbb328-199b-c29d-b57c-bdfc14b32371 a1fbb328-199b-c29d-b57c-bdfc14b32371}" 2026-04-14T14:15:22.636Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:15:21.630987144 +0000 UTC m=+53.103172908 (2 nodes), 691ns from start of poll, 2.035685129s from cluster start. Timeout at 2026-04-14 14:15:22.636007264 +0000 UTC m=+54.108193029, 1.005020121s after stability 2026-04-14T14:15:22.647Z [INFO] server-854e3dbb-8cde-333a-de66-1d0a6cb5ed7f: aborting pipeline replication: peer="{Voter server-80800e1f-da4b-8b3c-d47e-ef05b3a77789 80800e1f-da4b-8b3c-d47e-ef05b3a77789}" 2026-04-14T14:15:22.647Z [INFO] server-854e3dbb-8cde-333a-de66-1d0a6cb5ed7f: aborting pipeline replication: peer="{Voter server-d2744bb0-1d83-5c6c-ac9a-959b2c314cf7 d2744bb0-1d83-5c6c-ac9a-959b2c314cf7}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true 2026-04-14T14:15:22.648Z [INFO] server-6d92cf48-5b90-c46f-40f9-88e9f3adf387: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d92cf48-5b90-c46f-40f9-88e9f3adf387 Address:6d92cf48-5b90-c46f-40f9-88e9f3adf387} {Suffrage:Voter ID:server-feaf3843-c7a6-9e5b-11a5-22ee502dd35c Address:feaf3843-c7a6-9e5b-11a5-22ee502dd35c} {Suffrage:Voter ID:server-cf0faa11-808b-1c48-ae19-e68d46dc0889 Address:cf0faa11-808b-1c48-ae19-e68d46dc0889}]" 2026-04-14T14:15:22.648Z [INFO] server-feaf3843-c7a6-9e5b-11a5-22ee502dd35c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d92cf48-5b90-c46f-40f9-88e9f3adf387 Address:6d92cf48-5b90-c46f-40f9-88e9f3adf387} {Suffrage:Voter ID:server-feaf3843-c7a6-9e5b-11a5-22ee502dd35c Address:feaf3843-c7a6-9e5b-11a5-22ee502dd35c} {Suffrage:Voter ID:server-cf0faa11-808b-1c48-ae19-e68d46dc0889 Address:cf0faa11-808b-1c48-ae19-e68d46dc0889}]" 2026-04-14T14:15:22.648Z [INFO] server-6d92cf48-5b90-c46f-40f9-88e9f3adf387: entering follower state: follower="Node at 6d92cf48-5b90-c46f-40f9-88e9f3adf387 [Follower]" leader-address= leader-id= 2026-04-14T14:15:22.649Z [INFO] server-cf0faa11-808b-1c48-ae19-e68d46dc0889: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d92cf48-5b90-c46f-40f9-88e9f3adf387 Address:6d92cf48-5b90-c46f-40f9-88e9f3adf387} {Suffrage:Voter ID:server-feaf3843-c7a6-9e5b-11a5-22ee502dd35c Address:feaf3843-c7a6-9e5b-11a5-22ee502dd35c} {Suffrage:Voter ID:server-cf0faa11-808b-1c48-ae19-e68d46dc0889 Address:cf0faa11-808b-1c48-ae19-e68d46dc0889}]" 2026-04-14T14:15:22.649Z [INFO] server-feaf3843-c7a6-9e5b-11a5-22ee502dd35c: entering follower state: follower="Node at feaf3843-c7a6-9e5b-11a5-22ee502dd35c [Follower]" leader-address= leader-id= 2026-04-14T14:15:22.649Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:22.649Z [INFO] server-cf0faa11-808b-1c48-ae19-e68d46dc0889: entering follower state: follower="Node at cf0faa11-808b-1c48-ae19-e68d46dc0889 [Follower]" leader-address= leader-id= 2026-04-14T14:15:23.216Z [WARN] server-cf0faa11-808b-1c48-ae19-e68d46dc0889: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:23.216Z [INFO] server-cf0faa11-808b-1c48-ae19-e68d46dc0889: entering candidate state: node="Node at cf0faa11-808b-1c48-ae19-e68d46dc0889 [Candidate]" term=2 2026-04-14T14:15:23.216Z [INFO] server-cf0faa11-808b-1c48-ae19-e68d46dc0889: election won: term=2 tally=2 2026-04-14T14:15:23.216Z [INFO] server-cf0faa11-808b-1c48-ae19-e68d46dc0889: entering leader state: leader="Node at cf0faa11-808b-1c48-ae19-e68d46dc0889 [Leader]" 2026-04-14T14:15:23.216Z [INFO] server-cf0faa11-808b-1c48-ae19-e68d46dc0889: added peer, starting replication: peer=server-6d92cf48-5b90-c46f-40f9-88e9f3adf387 2026-04-14T14:15:23.216Z [INFO] server-cf0faa11-808b-1c48-ae19-e68d46dc0889: added peer, starting replication: peer=server-feaf3843-c7a6-9e5b-11a5-22ee502dd35c 2026-04-14T14:15:23.217Z [INFO] server-cf0faa11-808b-1c48-ae19-e68d46dc0889: pipelining replication: peer="{Voter server-6d92cf48-5b90-c46f-40f9-88e9f3adf387 6d92cf48-5b90-c46f-40f9-88e9f3adf387}" 2026-04-14T14:15:23.217Z [INFO] server-cf0faa11-808b-1c48-ae19-e68d46dc0889: pipelining replication: peer="{Voter server-feaf3843-c7a6-9e5b-11a5-22ee502dd35c feaf3843-c7a6-9e5b-11a5-22ee502dd35c}" 2026-04-14T14:15:24.221Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:23.216942055 +0000 UTC m=+54.689127824 (1 nodes), 567.855795ms from start of poll, 568.262512ms from cluster start. Timeout at 2026-04-14 14:15:24.221942579 +0000 UTC m=+55.694128369, 1.005000545s after stability 2026-04-14T14:15:24.222Z [INFO] server-cf0faa11-808b-1c48-ae19-e68d46dc0889: starting snapshot up to: index=12 2026-04-14T14:15:24.222Z [INFO] creating new snapshot: path=/tmp/raft1300384498/snapshots/2-12-1776176124222.tmp 2026-04-14T14:15:24.223Z [INFO] server-cf0faa11-808b-1c48-ae19-e68d46dc0889: snapshot complete up to: index=12 2026-04-14T14:15:24.224Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7 Address:56186f58-ec64-df9f-cfa3-677bbd3d0ed7} {Suffrage:Voter ID:server-b90f157d-f822-3d05-e042-1538802b23a6 Address:b90f157d-f822-3d05-e042-1538802b23a6} {Suffrage:Voter ID:server-a6795317-b6c4-5574-59e3-24ba171e3f66 Address:a6795317-b6c4-5574-59e3-24ba171e3f66}]" 2026-04-14T14:15:24.224Z [INFO] server-b90f157d-f822-3d05-e042-1538802b23a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7 Address:56186f58-ec64-df9f-cfa3-677bbd3d0ed7} {Suffrage:Voter ID:server-b90f157d-f822-3d05-e042-1538802b23a6 Address:b90f157d-f822-3d05-e042-1538802b23a6} {Suffrage:Voter ID:server-a6795317-b6c4-5574-59e3-24ba171e3f66 Address:a6795317-b6c4-5574-59e3-24ba171e3f66}]" 2026-04-14T14:15:24.224Z [INFO] server-a6795317-b6c4-5574-59e3-24ba171e3f66: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7 Address:56186f58-ec64-df9f-cfa3-677bbd3d0ed7} {Suffrage:Voter ID:server-b90f157d-f822-3d05-e042-1538802b23a6 Address:b90f157d-f822-3d05-e042-1538802b23a6} {Suffrage:Voter ID:server-a6795317-b6c4-5574-59e3-24ba171e3f66 Address:a6795317-b6c4-5574-59e3-24ba171e3f66}]" 2026-04-14T14:15:24.224Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:24.224Z [INFO] server-b90f157d-f822-3d05-e042-1538802b23a6: entering follower state: follower="Node at b90f157d-f822-3d05-e042-1538802b23a6 [Follower]" leader-address= leader-id= 2026-04-14T14:15:24.225Z [INFO] server-a6795317-b6c4-5574-59e3-24ba171e3f66: entering follower state: follower="Node at a6795317-b6c4-5574-59e3-24ba171e3f66 [Follower]" leader-address= leader-id= 2026-04-14T14:15:24.224Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: entering follower state: follower="Node at 56186f58-ec64-df9f-cfa3-677bbd3d0ed7 [Follower]" leader-address= leader-id= 2026-04-14T14:15:24.994Z [WARN] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:24.994Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: entering candidate state: node="Node at 56186f58-ec64-df9f-cfa3-677bbd3d0ed7 [Candidate]" term=2 2026-04-14T14:15:24.994Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: election won: term=2 tally=2 2026-04-14T14:15:24.994Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: entering leader state: leader="Node at 56186f58-ec64-df9f-cfa3-677bbd3d0ed7 [Leader]" 2026-04-14T14:15:24.994Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: added peer, starting replication: peer=server-b90f157d-f822-3d05-e042-1538802b23a6 2026-04-14T14:15:24.994Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: added peer, starting replication: peer=server-a6795317-b6c4-5574-59e3-24ba171e3f66 2026-04-14T14:15:24.994Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: pipelining replication: peer="{Voter server-a6795317-b6c4-5574-59e3-24ba171e3f66 a6795317-b6c4-5574-59e3-24ba171e3f66}" 2026-04-14T14:15:24.994Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: pipelining replication: peer="{Voter server-b90f157d-f822-3d05-e042-1538802b23a6 b90f157d-f822-3d05-e042-1538802b23a6}" 2026-04-14T14:15:26.000Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:24.994798245 +0000 UTC m=+56.466984030 (1 nodes), 769.832404ms from start of poll, 770.239201ms from cluster start. Timeout at 2026-04-14 14:15:25.999799934 +0000 UTC m=+57.471985746, 1.005001716s after stability 2026-04-14T14:15:26.001Z [INFO] creating new snapshot: path=/tmp/raft2829427488/snapshots/2-15-1776176126001.tmp 2026-04-14T14:15:26.002Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: copied to local snapshot: bytes=71 2026-04-14T14:15:26.002Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: snapshot restore progress: id=2-15-1776176126001 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:26.002Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: removing all old logs from log store 2026-04-14T14:15:26.002Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: compacting logs: from=1 to=2 2026-04-14T14:15:26.002Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: restored user snapshot: index=15 2026-04-14T14:15:26.003Z [ERROR] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: failed to get log: index=2 error="log not found" 2026-04-14T14:15:26.003Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: aborting pipeline replication: peer="{Voter server-a6795317-b6c4-5574-59e3-24ba171e3f66 a6795317-b6c4-5574-59e3-24ba171e3f66}" 2026-04-14T14:15:26.003Z [ERROR] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: failed to get log: index=2 error="log not found" 2026-04-14T14:15:26.003Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: aborting pipeline replication: peer="{Voter server-b90f157d-f822-3d05-e042-1538802b23a6 b90f157d-f822-3d05-e042-1538802b23a6}" 2026-04-14T14:15:26.010Z [ERROR] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: failed to get log: index=2 error="log not found" 2026-04-14T14:15:26.010Z [INFO] creating new snapshot: path=/tmp/raft1547499712/snapshots/2-15-1776176126010.tmp 2026-04-14T14:15:26.010Z [INFO] server-a6795317-b6c4-5574-59e3-24ba171e3f66: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:26.010Z [ERROR] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: failed to get log: index=2 error="log not found" 2026-04-14T14:15:26.011Z [INFO] server-a6795317-b6c4-5574-59e3-24ba171e3f66: copied to local snapshot: bytes=71 2026-04-14T14:15:26.011Z [INFO] creating new snapshot: path=/tmp/raft2895168950/snapshots/2-15-1776176126011.tmp 2026-04-14T14:15:26.011Z [INFO] server-a6795317-b6c4-5574-59e3-24ba171e3f66: snapshot restore progress: id=2-15-1776176126010 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:26.011Z [INFO] server-a6795317-b6c4-5574-59e3-24ba171e3f66: removing all old logs from log store 2026-04-14T14:15:26.011Z [INFO] server-a6795317-b6c4-5574-59e3-24ba171e3f66: compacting logs: from=1 to=2 2026-04-14T14:15:26.011Z [INFO] server-a6795317-b6c4-5574-59e3-24ba171e3f66: Installed remote snapshot 2026-04-14T14:15:26.011Z [INFO] server-b90f157d-f822-3d05-e042-1538802b23a6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:26.011Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: pipelining replication: peer="{Voter server-a6795317-b6c4-5574-59e3-24ba171e3f66 a6795317-b6c4-5574-59e3-24ba171e3f66}" 2026-04-14T14:15:26.011Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:15:26.012Z [INFO] server-b90f157d-f822-3d05-e042-1538802b23a6: copied to local snapshot: bytes=71 2026-04-14T14:15:26.012Z [INFO] server-b90f157d-f822-3d05-e042-1538802b23a6: snapshot restore progress: id=2-15-1776176126011 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:26.012Z [INFO] server-b90f157d-f822-3d05-e042-1538802b23a6: removing all old logs from log store 2026-04-14T14:15:26.012Z [INFO] server-b90f157d-f822-3d05-e042-1538802b23a6: compacting logs: from=1 to=2 2026-04-14T14:15:26.012Z [INFO] server-b90f157d-f822-3d05-e042-1538802b23a6: Installed remote snapshot 2026-04-14T14:15:26.012Z [INFO] server-56186f58-ec64-df9f-cfa3-677bbd3d0ed7: pipelining replication: peer="{Voter server-b90f157d-f822-3d05-e042-1538802b23a6 b90f157d-f822-3d05-e042-1538802b23a6}" 2026-04-14T14:15:27.017Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:15:26.011925169 +0000 UTC m=+57.484110950 (2 nodes), 1.138µs from start of poll, 1.787366121s from cluster start. Timeout at 2026-04-14 14:15:27.016942039 +0000 UTC m=+58.489127838, 1.005016888s after stability 2026-04-14T14:15:27.029Z [INFO] server-cf0faa11-808b-1c48-ae19-e68d46dc0889: aborting pipeline replication: peer="{Voter server-feaf3843-c7a6-9e5b-11a5-22ee502dd35c feaf3843-c7a6-9e5b-11a5-22ee502dd35c}" 2026-04-14T14:15:27.029Z [INFO] server-cf0faa11-808b-1c48-ae19-e68d46dc0889: aborting pipeline replication: peer="{Voter server-6d92cf48-5b90-c46f-40f9-88e9f3adf387 6d92cf48-5b90-c46f-40f9-88e9f3adf387}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false 2026-04-14T14:15:27.031Z [INFO] server-cfc6017b-c6c7-648d-5636-bd75446ab644: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cfc6017b-c6c7-648d-5636-bd75446ab644 Address:cfc6017b-c6c7-648d-5636-bd75446ab644} {Suffrage:Voter ID:server-3a84d5e7-566d-3a03-cc3c-393357b58ab0 Address:3a84d5e7-566d-3a03-cc3c-393357b58ab0} {Suffrage:Voter ID:server-8b8dc4d1-36ff-baad-0eae-96ce8b627de3 Address:8b8dc4d1-36ff-baad-0eae-96ce8b627de3}]" 2026-04-14T14:15:27.031Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cfc6017b-c6c7-648d-5636-bd75446ab644 Address:cfc6017b-c6c7-648d-5636-bd75446ab644} {Suffrage:Voter ID:server-3a84d5e7-566d-3a03-cc3c-393357b58ab0 Address:3a84d5e7-566d-3a03-cc3c-393357b58ab0} {Suffrage:Voter ID:server-8b8dc4d1-36ff-baad-0eae-96ce8b627de3 Address:8b8dc4d1-36ff-baad-0eae-96ce8b627de3}]" 2026-04-14T14:15:27.031Z [INFO] server-cfc6017b-c6c7-648d-5636-bd75446ab644: entering follower state: follower="Node at cfc6017b-c6c7-648d-5636-bd75446ab644 [Follower]" leader-address= leader-id= 2026-04-14T14:15:27.031Z [INFO] server-8b8dc4d1-36ff-baad-0eae-96ce8b627de3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cfc6017b-c6c7-648d-5636-bd75446ab644 Address:cfc6017b-c6c7-648d-5636-bd75446ab644} {Suffrage:Voter ID:server-3a84d5e7-566d-3a03-cc3c-393357b58ab0 Address:3a84d5e7-566d-3a03-cc3c-393357b58ab0} {Suffrage:Voter ID:server-8b8dc4d1-36ff-baad-0eae-96ce8b627de3 Address:8b8dc4d1-36ff-baad-0eae-96ce8b627de3}]" 2026-04-14T14:15:27.031Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:27.031Z [INFO] server-8b8dc4d1-36ff-baad-0eae-96ce8b627de3: entering follower state: follower="Node at 8b8dc4d1-36ff-baad-0eae-96ce8b627de3 [Follower]" leader-address= leader-id= 2026-04-14T14:15:27.031Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: entering follower state: follower="Node at 3a84d5e7-566d-3a03-cc3c-393357b58ab0 [Follower]" leader-address= leader-id= 2026-04-14T14:15:27.689Z [WARN] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:27.689Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: entering candidate state: node="Node at 3a84d5e7-566d-3a03-cc3c-393357b58ab0 [Candidate]" term=2 2026-04-14T14:15:27.689Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: election won: term=2 tally=2 2026-04-14T14:15:27.689Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: entering leader state: leader="Node at 3a84d5e7-566d-3a03-cc3c-393357b58ab0 [Leader]" 2026-04-14T14:15:27.689Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: added peer, starting replication: peer=server-cfc6017b-c6c7-648d-5636-bd75446ab644 2026-04-14T14:15:27.689Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: added peer, starting replication: peer=server-8b8dc4d1-36ff-baad-0eae-96ce8b627de3 2026-04-14T14:15:27.690Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: pipelining replication: peer="{Voter server-8b8dc4d1-36ff-baad-0eae-96ce8b627de3 8b8dc4d1-36ff-baad-0eae-96ce8b627de3}" 2026-04-14T14:15:27.690Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: pipelining replication: peer="{Voter server-cfc6017b-c6c7-648d-5636-bd75446ab644 cfc6017b-c6c7-648d-5636-bd75446ab644}" 2026-04-14T14:15:28.975Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:27.689918086 +0000 UTC m=+59.162103874 (1 nodes), 658.618014ms from start of poll, 658.992078ms from cluster start. Timeout at 2026-04-14 14:15:28.69495047 +0000 UTC m=+60.167136239, 1.005032365s after stability 2026-04-14T14:15:28.976Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: starting snapshot up to: index=12 2026-04-14T14:15:28.976Z [INFO] creating new snapshot: path=/tmp/raft2552639970/snapshots/2-12-1776176128976.tmp 2026-04-14T14:15:28.979Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: snapshot complete up to: index=12 2026-04-14T14:15:28.980Z [INFO] creating new snapshot: path=/tmp/raft2552639970/snapshots/2-113-1776176128980.tmp 2026-04-14T14:15:28.982Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: copied to local snapshot: bytes=71 2026-04-14T14:15:28.982Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: snapshot restore progress: id=2-113-1776176128980 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:28.982Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: restored user snapshot: index=113 2026-04-14T14:15:28.982Z [ERROR] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: failed to get log: index=23 error="log not found" 2026-04-14T14:15:28.982Z [ERROR] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: failed to get log: index=23 error="log not found" 2026-04-14T14:15:28.982Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: aborting pipeline replication: peer="{Voter server-8b8dc4d1-36ff-baad-0eae-96ce8b627de3 8b8dc4d1-36ff-baad-0eae-96ce8b627de3}" 2026-04-14T14:15:28.982Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: aborting pipeline replication: peer="{Voter server-cfc6017b-c6c7-648d-5636-bd75446ab644 cfc6017b-c6c7-648d-5636-bd75446ab644}" 2026-04-14T14:15:28.989Z [ERROR] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: failed to get log: index=23 error="log not found" 2026-04-14T14:15:28.991Z [ERROR] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: failed to get log: index=23 error="log not found" 2026-04-14T14:15:28.991Z [INFO] creating new snapshot: path=/tmp/raft514128170/snapshots/2-113-1776176128991.tmp 2026-04-14T14:15:28.993Z [INFO] creating new snapshot: path=/tmp/raft2539596369/snapshots/2-113-1776176128993.tmp 2026-04-14T14:15:28.993Z [INFO] server-8b8dc4d1-36ff-baad-0eae-96ce8b627de3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:29.002Z [INFO] server-cfc6017b-c6c7-648d-5636-bd75446ab644: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:29.002Z [INFO] server-cfc6017b-c6c7-648d-5636-bd75446ab644: copied to local snapshot: bytes=71 2026-04-14T14:15:29.002Z [INFO] server-8b8dc4d1-36ff-baad-0eae-96ce8b627de3: copied to local snapshot: bytes=71 2026-04-14T14:15:29.003Z [INFO] server-cfc6017b-c6c7-648d-5636-bd75446ab644: snapshot restore progress: id=2-113-1776176128993 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:29.003Z [INFO] server-cfc6017b-c6c7-648d-5636-bd75446ab644: Installed remote snapshot 2026-04-14T14:15:29.003Z [WARN] server-cfc6017b-c6c7-648d-5636-bd75446ab644: skipping application of old log: index=22 2026-04-14T14:15:29.003Z [INFO] server-8b8dc4d1-36ff-baad-0eae-96ce8b627de3: snapshot restore progress: id=2-113-1776176128991 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:29.003Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: pipelining replication: peer="{Voter server-cfc6017b-c6c7-648d-5636-bd75446ab644 cfc6017b-c6c7-648d-5636-bd75446ab644}" 2026-04-14T14:15:29.003Z [INFO] server-8b8dc4d1-36ff-baad-0eae-96ce8b627de3: Installed remote snapshot 2026-04-14T14:15:29.003Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:15:29.003Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: pipelining replication: peer="{Voter server-8b8dc4d1-36ff-baad-0eae-96ce8b627de3 8b8dc4d1-36ff-baad-0eae-96ce8b627de3}" 2026-04-14T14:15:30.008Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:15:29.003305237 +0000 UTC m=+60.475491026 (2 nodes), 734ns from start of poll, 1.97237923s from cluster start. Timeout at 2026-04-14 14:15:30.008326746 +0000 UTC m=+61.480512555, 1.005021529s after stability 2026-04-14T14:15:30.020Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: aborting pipeline replication: peer="{Voter server-cfc6017b-c6c7-648d-5636-bd75446ab644 cfc6017b-c6c7-648d-5636-bd75446ab644}" 2026-04-14T14:15:30.020Z [INFO] server-3a84d5e7-566d-3a03-cc3c-393357b58ab0: aborting pipeline replication: peer="{Voter server-8b8dc4d1-36ff-baad-0eae-96ce8b627de3 8b8dc4d1-36ff-baad-0eae-96ce8b627de3}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false 2026-04-14T14:15:30.022Z [INFO] server-cd5e7e63-e891-8791-0f59-6daa7bdc88b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd5e7e63-e891-8791-0f59-6daa7bdc88b8 Address:cd5e7e63-e891-8791-0f59-6daa7bdc88b8} {Suffrage:Voter ID:server-b1535f20-8c4a-2ce1-97de-288fd9dec015 Address:b1535f20-8c4a-2ce1-97de-288fd9dec015} {Suffrage:Voter ID:server-edc2b967-7970-dfed-f38a-10a3f86f1c68 Address:edc2b967-7970-dfed-f38a-10a3f86f1c68}]" 2026-04-14T14:15:30.022Z [INFO] server-cd5e7e63-e891-8791-0f59-6daa7bdc88b8: entering follower state: follower="Node at cd5e7e63-e891-8791-0f59-6daa7bdc88b8 [Follower]" leader-address= leader-id= 2026-04-14T14:15:30.022Z [INFO] server-b1535f20-8c4a-2ce1-97de-288fd9dec015: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd5e7e63-e891-8791-0f59-6daa7bdc88b8 Address:cd5e7e63-e891-8791-0f59-6daa7bdc88b8} {Suffrage:Voter ID:server-b1535f20-8c4a-2ce1-97de-288fd9dec015 Address:b1535f20-8c4a-2ce1-97de-288fd9dec015} {Suffrage:Voter ID:server-edc2b967-7970-dfed-f38a-10a3f86f1c68 Address:edc2b967-7970-dfed-f38a-10a3f86f1c68}]" 2026-04-14T14:15:30.022Z [INFO] server-b1535f20-8c4a-2ce1-97de-288fd9dec015: entering follower state: follower="Node at b1535f20-8c4a-2ce1-97de-288fd9dec015 [Follower]" leader-address= leader-id= 2026-04-14T14:15:30.022Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd5e7e63-e891-8791-0f59-6daa7bdc88b8 Address:cd5e7e63-e891-8791-0f59-6daa7bdc88b8} {Suffrage:Voter ID:server-b1535f20-8c4a-2ce1-97de-288fd9dec015 Address:b1535f20-8c4a-2ce1-97de-288fd9dec015} {Suffrage:Voter ID:server-edc2b967-7970-dfed-f38a-10a3f86f1c68 Address:edc2b967-7970-dfed-f38a-10a3f86f1c68}]" 2026-04-14T14:15:30.022Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:30.022Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: entering follower state: follower="Node at edc2b967-7970-dfed-f38a-10a3f86f1c68 [Follower]" leader-address= leader-id= 2026-04-14T14:15:30.542Z [WARN] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:30.542Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: entering candidate state: node="Node at edc2b967-7970-dfed-f38a-10a3f86f1c68 [Candidate]" term=2 2026-04-14T14:15:30.542Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: election won: term=2 tally=2 2026-04-14T14:15:30.542Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: entering leader state: leader="Node at edc2b967-7970-dfed-f38a-10a3f86f1c68 [Leader]" 2026-04-14T14:15:30.542Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: added peer, starting replication: peer=server-cd5e7e63-e891-8791-0f59-6daa7bdc88b8 2026-04-14T14:15:30.542Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: added peer, starting replication: peer=server-b1535f20-8c4a-2ce1-97de-288fd9dec015 2026-04-14T14:15:30.542Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: pipelining replication: peer="{Voter server-b1535f20-8c4a-2ce1-97de-288fd9dec015 b1535f20-8c4a-2ce1-97de-288fd9dec015}" 2026-04-14T14:15:30.542Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: pipelining replication: peer="{Voter server-cd5e7e63-e891-8791-0f59-6daa7bdc88b8 cd5e7e63-e891-8791-0f59-6daa7bdc88b8}" 2026-04-14T14:15:31.547Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:30.542284074 +0000 UTC m=+62.014469838 (1 nodes), 519.40871ms from start of poll, 519.826102ms from cluster start. Timeout at 2026-04-14 14:15:31.547295764 +0000 UTC m=+63.019481549, 1.005011711s after stability 2026-04-14T14:15:31.548Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: starting snapshot up to: index=12 2026-04-14T14:15:31.548Z [INFO] creating new snapshot: path=/tmp/raft3840516845/snapshots/2-12-1776176131548.tmp 2026-04-14T14:15:31.642Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: snapshot complete up to: index=12 2026-04-14T14:15:31.642Z [INFO] creating new snapshot: path=/tmp/raft3840516845/snapshots/2-113-1776176131642.tmp 2026-04-14T14:15:31.643Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: copied to local snapshot: bytes=71 2026-04-14T14:15:31.643Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: snapshot restore progress: id=2-113-1776176131642 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:31.643Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: removing all old logs from log store 2026-04-14T14:15:31.643Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: compacting logs: from=1 to=22 2026-04-14T14:15:31.643Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: restored user snapshot: index=113 2026-04-14T14:15:31.643Z [ERROR] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: failed to get log: index=22 error="log not found" 2026-04-14T14:15:31.643Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: aborting pipeline replication: peer="{Voter server-b1535f20-8c4a-2ce1-97de-288fd9dec015 b1535f20-8c4a-2ce1-97de-288fd9dec015}" 2026-04-14T14:15:31.643Z [ERROR] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: failed to get log: index=22 error="log not found" 2026-04-14T14:15:31.643Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: aborting pipeline replication: peer="{Voter server-cd5e7e63-e891-8791-0f59-6daa7bdc88b8 cd5e7e63-e891-8791-0f59-6daa7bdc88b8}" 2026-04-14T14:15:31.649Z [ERROR] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: failed to get log: index=22 error="log not found" 2026-04-14T14:15:31.649Z [INFO] creating new snapshot: path=/tmp/raft3847672812/snapshots/2-113-1776176131649.tmp 2026-04-14T14:15:31.649Z [INFO] server-cd5e7e63-e891-8791-0f59-6daa7bdc88b8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:31.652Z [ERROR] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: failed to get log: index=22 error="log not found" 2026-04-14T14:15:31.653Z [INFO] creating new snapshot: path=/tmp/raft3870030939/snapshots/2-113-1776176131653.tmp 2026-04-14T14:15:31.653Z [INFO] server-b1535f20-8c4a-2ce1-97de-288fd9dec015: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:31.898Z [INFO] server-cd5e7e63-e891-8791-0f59-6daa7bdc88b8: copied to local snapshot: bytes=71 2026-04-14T14:15:31.898Z [INFO] server-b1535f20-8c4a-2ce1-97de-288fd9dec015: copied to local snapshot: bytes=71 2026-04-14T14:15:31.899Z [INFO] server-cd5e7e63-e891-8791-0f59-6daa7bdc88b8: snapshot restore progress: id=2-113-1776176131649 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:31.899Z [INFO] server-cd5e7e63-e891-8791-0f59-6daa7bdc88b8: removing all old logs from log store 2026-04-14T14:15:31.899Z [INFO] server-cd5e7e63-e891-8791-0f59-6daa7bdc88b8: compacting logs: from=1 to=22 2026-04-14T14:15:31.899Z [INFO] server-cd5e7e63-e891-8791-0f59-6daa7bdc88b8: Installed remote snapshot 2026-04-14T14:15:31.899Z [WARN] server-cd5e7e63-e891-8791-0f59-6daa7bdc88b8: skipping application of old log: index=22 2026-04-14T14:15:31.899Z [INFO] server-b1535f20-8c4a-2ce1-97de-288fd9dec015: snapshot restore progress: id=2-113-1776176131653 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:31.899Z [INFO] server-b1535f20-8c4a-2ce1-97de-288fd9dec015: removing all old logs from log store 2026-04-14T14:15:31.899Z [INFO] server-b1535f20-8c4a-2ce1-97de-288fd9dec015: compacting logs: from=1 to=22 2026-04-14T14:15:31.899Z [INFO] server-b1535f20-8c4a-2ce1-97de-288fd9dec015: Installed remote snapshot 2026-04-14T14:15:31.899Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:15:31.899Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: pipelining replication: peer="{Voter server-cd5e7e63-e891-8791-0f59-6daa7bdc88b8 cd5e7e63-e891-8791-0f59-6daa7bdc88b8}" 2026-04-14T14:15:31.899Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: pipelining replication: peer="{Voter server-b1535f20-8c4a-2ce1-97de-288fd9dec015 b1535f20-8c4a-2ce1-97de-288fd9dec015}" 2026-04-14T14:15:32.904Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:15:31.899317936 +0000 UTC m=+63.371503948 (2 nodes), 1.274µs from start of poll, 1.876860212s from cluster start. Timeout at 2026-04-14 14:15:32.904326887 +0000 UTC m=+64.376512680, 1.005008732s after stability 2026-04-14T14:15:32.916Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: aborting pipeline replication: peer="{Voter server-b1535f20-8c4a-2ce1-97de-288fd9dec015 b1535f20-8c4a-2ce1-97de-288fd9dec015}" 2026-04-14T14:15:32.916Z [INFO] server-edc2b967-7970-dfed-f38a-10a3f86f1c68: aborting pipeline replication: peer="{Voter server-cd5e7e63-e891-8791-0f59-6daa7bdc88b8 cd5e7e63-e891-8791-0f59-6daa7bdc88b8}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true 2026-04-14T14:15:32.918Z [INFO] server-141f7a22-5b67-f055-bfae-f957e4879fd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-141f7a22-5b67-f055-bfae-f957e4879fd9 Address:141f7a22-5b67-f055-bfae-f957e4879fd9} {Suffrage:Voter ID:server-c9b952c6-4837-b307-b458-d0313a7efdc5 Address:c9b952c6-4837-b307-b458-d0313a7efdc5} {Suffrage:Voter ID:server-776a5dd6-ba08-0f27-9b4d-253d32b8922f Address:776a5dd6-ba08-0f27-9b4d-253d32b8922f}]" 2026-04-14T14:15:32.918Z [INFO] server-141f7a22-5b67-f055-bfae-f957e4879fd9: entering follower state: follower="Node at 141f7a22-5b67-f055-bfae-f957e4879fd9 [Follower]" leader-address= leader-id= 2026-04-14T14:15:32.918Z [INFO] server-c9b952c6-4837-b307-b458-d0313a7efdc5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-141f7a22-5b67-f055-bfae-f957e4879fd9 Address:141f7a22-5b67-f055-bfae-f957e4879fd9} {Suffrage:Voter ID:server-c9b952c6-4837-b307-b458-d0313a7efdc5 Address:c9b952c6-4837-b307-b458-d0313a7efdc5} {Suffrage:Voter ID:server-776a5dd6-ba08-0f27-9b4d-253d32b8922f Address:776a5dd6-ba08-0f27-9b4d-253d32b8922f}]" 2026-04-14T14:15:32.919Z [INFO] server-c9b952c6-4837-b307-b458-d0313a7efdc5: entering follower state: follower="Node at c9b952c6-4837-b307-b458-d0313a7efdc5 [Follower]" leader-address= leader-id= 2026-04-14T14:15:32.919Z [INFO] server-776a5dd6-ba08-0f27-9b4d-253d32b8922f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-141f7a22-5b67-f055-bfae-f957e4879fd9 Address:141f7a22-5b67-f055-bfae-f957e4879fd9} {Suffrage:Voter ID:server-c9b952c6-4837-b307-b458-d0313a7efdc5 Address:c9b952c6-4837-b307-b458-d0313a7efdc5} {Suffrage:Voter ID:server-776a5dd6-ba08-0f27-9b4d-253d32b8922f Address:776a5dd6-ba08-0f27-9b4d-253d32b8922f}]" 2026-04-14T14:15:32.919Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:32.919Z [INFO] server-776a5dd6-ba08-0f27-9b4d-253d32b8922f: entering follower state: follower="Node at 776a5dd6-ba08-0f27-9b4d-253d32b8922f [Follower]" leader-address= leader-id= 2026-04-14T14:15:33.484Z [WARN] server-141f7a22-5b67-f055-bfae-f957e4879fd9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:33.484Z [INFO] server-141f7a22-5b67-f055-bfae-f957e4879fd9: entering candidate state: node="Node at 141f7a22-5b67-f055-bfae-f957e4879fd9 [Candidate]" term=2 2026-04-14T14:15:33.484Z [INFO] server-141f7a22-5b67-f055-bfae-f957e4879fd9: election won: term=2 tally=2 2026-04-14T14:15:33.484Z [INFO] server-141f7a22-5b67-f055-bfae-f957e4879fd9: entering leader state: leader="Node at 141f7a22-5b67-f055-bfae-f957e4879fd9 [Leader]" 2026-04-14T14:15:33.484Z [INFO] server-141f7a22-5b67-f055-bfae-f957e4879fd9: added peer, starting replication: peer=server-c9b952c6-4837-b307-b458-d0313a7efdc5 2026-04-14T14:15:33.484Z [INFO] server-141f7a22-5b67-f055-bfae-f957e4879fd9: added peer, starting replication: peer=server-776a5dd6-ba08-0f27-9b4d-253d32b8922f 2026-04-14T14:15:33.484Z [INFO] server-141f7a22-5b67-f055-bfae-f957e4879fd9: pipelining replication: peer="{Voter server-776a5dd6-ba08-0f27-9b4d-253d32b8922f 776a5dd6-ba08-0f27-9b4d-253d32b8922f}" 2026-04-14T14:15:33.484Z [INFO] server-141f7a22-5b67-f055-bfae-f957e4879fd9: pipelining replication: peer="{Voter server-c9b952c6-4837-b307-b458-d0313a7efdc5 c9b952c6-4837-b307-b458-d0313a7efdc5}" 2026-04-14T14:15:34.490Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:33.484657322 +0000 UTC m=+64.956843104 (1 nodes), 565.309092ms from start of poll, 566.006514ms from cluster start. Timeout at 2026-04-14 14:15:34.489661025 +0000 UTC m=+65.961846789, 1.005003685s after stability 2026-04-14T14:15:34.491Z [INFO] server-141f7a22-5b67-f055-bfae-f957e4879fd9: starting snapshot up to: index=12 2026-04-14T14:15:34.491Z [INFO] creating new snapshot: path=/tmp/raft827179461/snapshots/2-12-1776176134491.tmp 2026-04-14T14:15:34.491Z [INFO] server-141f7a22-5b67-f055-bfae-f957e4879fd9: snapshot complete up to: index=12 2026-04-14T14:15:34.493Z [INFO] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1 Address:c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1} {Suffrage:Voter ID:server-e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d Address:e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d} {Suffrage:Voter ID:server-b3216cfd-34ba-7964-1cbb-a4103715f485 Address:b3216cfd-34ba-7964-1cbb-a4103715f485}]" 2026-04-14T14:15:34.493Z [INFO] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: entering follower state: follower="Node at c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1 [Follower]" leader-address= leader-id= 2026-04-14T14:15:34.493Z [INFO] server-e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1 Address:c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1} {Suffrage:Voter ID:server-e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d Address:e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d} {Suffrage:Voter ID:server-b3216cfd-34ba-7964-1cbb-a4103715f485 Address:b3216cfd-34ba-7964-1cbb-a4103715f485}]" 2026-04-14T14:15:34.493Z [INFO] server-b3216cfd-34ba-7964-1cbb-a4103715f485: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1 Address:c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1} {Suffrage:Voter ID:server-e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d Address:e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d} {Suffrage:Voter ID:server-b3216cfd-34ba-7964-1cbb-a4103715f485 Address:b3216cfd-34ba-7964-1cbb-a4103715f485}]" 2026-04-14T14:15:34.493Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:34.493Z [INFO] server-e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d: entering follower state: follower="Node at e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d [Follower]" leader-address= leader-id= 2026-04-14T14:15:34.493Z [INFO] server-b3216cfd-34ba-7964-1cbb-a4103715f485: entering follower state: follower="Node at b3216cfd-34ba-7964-1cbb-a4103715f485 [Follower]" leader-address= leader-id= 2026-04-14T14:15:35.160Z [WARN] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:35.160Z [INFO] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: entering candidate state: node="Node at c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1 [Candidate]" term=2 2026-04-14T14:15:35.160Z [INFO] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: election won: term=2 tally=2 2026-04-14T14:15:35.161Z [INFO] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: entering leader state: leader="Node at c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1 [Leader]" 2026-04-14T14:15:35.161Z [INFO] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: added peer, starting replication: peer=server-e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d 2026-04-14T14:15:35.161Z [INFO] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: added peer, starting replication: peer=server-b3216cfd-34ba-7964-1cbb-a4103715f485 2026-04-14T14:15:35.161Z [INFO] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: pipelining replication: peer="{Voter server-b3216cfd-34ba-7964-1cbb-a4103715f485 b3216cfd-34ba-7964-1cbb-a4103715f485}" 2026-04-14T14:15:35.161Z [INFO] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: pipelining replication: peer="{Voter server-e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d}" 2026-04-14T14:15:36.166Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:35.161062274 +0000 UTC m=+66.633248054 (1 nodes), 667.224ms from start of poll, 667.609302ms from cluster start. Timeout at 2026-04-14 14:15:36.16606314 +0000 UTC m=+67.638248905, 1.005000851s after stability 2026-04-14T14:15:36.167Z [INFO] creating new snapshot: path=/tmp/raft3883786470/snapshots/2-113-1776176136167.tmp 2026-04-14T14:15:36.167Z [INFO] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: copied to local snapshot: bytes=71 2026-04-14T14:15:36.167Z [INFO] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: snapshot restore progress: id=2-113-1776176136167 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:36.167Z [INFO] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: restored user snapshot: index=113 2026-04-14T14:15:36.167Z [ERROR] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: failed to get log: index=3 error="log not found" 2026-04-14T14:15:36.167Z [INFO] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: aborting pipeline replication: peer="{Voter server-b3216cfd-34ba-7964-1cbb-a4103715f485 b3216cfd-34ba-7964-1cbb-a4103715f485}" 2026-04-14T14:15:36.167Z [ERROR] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: failed to get log: index=3 error="log not found" 2026-04-14T14:15:36.168Z [INFO] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: aborting pipeline replication: peer="{Voter server-e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d}" 2026-04-14T14:15:36.174Z [ERROR] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: failed to get log: index=3 error="log not found" 2026-04-14T14:15:36.175Z [INFO] creating new snapshot: path=/tmp/raft586186073/snapshots/2-113-1776176136175.tmp 2026-04-14T14:15:36.175Z [INFO] server-b3216cfd-34ba-7964-1cbb-a4103715f485: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:36.176Z [ERROR] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: failed to get log: index=3 error="log not found" 2026-04-14T14:15:36.176Z [INFO] server-b3216cfd-34ba-7964-1cbb-a4103715f485: copied to local snapshot: bytes=71 2026-04-14T14:15:36.176Z [INFO] server-b3216cfd-34ba-7964-1cbb-a4103715f485: snapshot restore progress: id=2-113-1776176136175 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:36.176Z [INFO] server-b3216cfd-34ba-7964-1cbb-a4103715f485: Installed remote snapshot 2026-04-14T14:15:36.176Z [INFO] creating new snapshot: path=/tmp/raft1900513895/snapshots/2-113-1776176136176.tmp 2026-04-14T14:15:36.176Z [INFO] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: pipelining replication: peer="{Voter server-b3216cfd-34ba-7964-1cbb-a4103715f485 b3216cfd-34ba-7964-1cbb-a4103715f485}" 2026-04-14T14:15:36.176Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:15:36.176Z [INFO] server-e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:36.177Z [INFO] server-e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d: copied to local snapshot: bytes=71 2026-04-14T14:15:36.177Z [INFO] server-e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d: snapshot restore progress: id=2-113-1776176136176 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:36.177Z [INFO] server-e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d: Installed remote snapshot 2026-04-14T14:15:36.177Z [INFO] server-c32a0a9c-19b6-a6bf-b3d8-73bc398dfaf1: pipelining replication: peer="{Voter server-e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d e2b6a9ec-af65-a87a-cc40-bda6ac9aef1d}" 2026-04-14T14:15:37.181Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:15:36.176761542 +0000 UTC m=+67.648947311 (2 nodes), 1.802µs from start of poll, 1.683308559s from cluster start. Timeout at 2026-04-14 14:15:37.181789395 +0000 UTC m=+68.653975176, 1.005027865s after stability 2026-04-14T14:15:37.193Z [INFO] server-141f7a22-5b67-f055-bfae-f957e4879fd9: aborting pipeline replication: peer="{Voter server-776a5dd6-ba08-0f27-9b4d-253d32b8922f 776a5dd6-ba08-0f27-9b4d-253d32b8922f}" 2026-04-14T14:15:37.193Z [INFO] server-141f7a22-5b67-f055-bfae-f957e4879fd9: aborting pipeline replication: peer="{Voter server-c9b952c6-4837-b307-b458-d0313a7efdc5 c9b952c6-4837-b307-b458-d0313a7efdc5}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true 2026-04-14T14:15:37.601Z [INFO] server-6b09effd-de6c-99f6-c6de-b3f7676a1839: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b09effd-de6c-99f6-c6de-b3f7676a1839 Address:6b09effd-de6c-99f6-c6de-b3f7676a1839} {Suffrage:Voter ID:server-103c9309-dd42-0352-0aa3-9b3aace340e7 Address:103c9309-dd42-0352-0aa3-9b3aace340e7} {Suffrage:Voter ID:server-1f97bd0b-f2ca-6f14-be9a-5ee47ab779f1 Address:1f97bd0b-f2ca-6f14-be9a-5ee47ab779f1}]" 2026-04-14T14:15:37.602Z [INFO] server-103c9309-dd42-0352-0aa3-9b3aace340e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b09effd-de6c-99f6-c6de-b3f7676a1839 Address:6b09effd-de6c-99f6-c6de-b3f7676a1839} {Suffrage:Voter ID:server-103c9309-dd42-0352-0aa3-9b3aace340e7 Address:103c9309-dd42-0352-0aa3-9b3aace340e7} {Suffrage:Voter ID:server-1f97bd0b-f2ca-6f14-be9a-5ee47ab779f1 Address:1f97bd0b-f2ca-6f14-be9a-5ee47ab779f1}]" 2026-04-14T14:15:37.602Z [INFO] server-6b09effd-de6c-99f6-c6de-b3f7676a1839: entering follower state: follower="Node at 6b09effd-de6c-99f6-c6de-b3f7676a1839 [Follower]" leader-address= leader-id= 2026-04-14T14:15:37.602Z [INFO] server-103c9309-dd42-0352-0aa3-9b3aace340e7: entering follower state: follower="Node at 103c9309-dd42-0352-0aa3-9b3aace340e7 [Follower]" leader-address= leader-id= 2026-04-14T14:15:37.602Z [INFO] server-1f97bd0b-f2ca-6f14-be9a-5ee47ab779f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b09effd-de6c-99f6-c6de-b3f7676a1839 Address:6b09effd-de6c-99f6-c6de-b3f7676a1839} {Suffrage:Voter ID:server-103c9309-dd42-0352-0aa3-9b3aace340e7 Address:103c9309-dd42-0352-0aa3-9b3aace340e7} {Suffrage:Voter ID:server-1f97bd0b-f2ca-6f14-be9a-5ee47ab779f1 Address:1f97bd0b-f2ca-6f14-be9a-5ee47ab779f1}]" 2026-04-14T14:15:37.602Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:37.602Z [INFO] server-1f97bd0b-f2ca-6f14-be9a-5ee47ab779f1: entering follower state: follower="Node at 1f97bd0b-f2ca-6f14-be9a-5ee47ab779f1 [Follower]" leader-address= leader-id= 2026-04-14T14:15:38.134Z [WARN] server-6b09effd-de6c-99f6-c6de-b3f7676a1839: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:38.134Z [INFO] server-6b09effd-de6c-99f6-c6de-b3f7676a1839: entering candidate state: node="Node at 6b09effd-de6c-99f6-c6de-b3f7676a1839 [Candidate]" term=2 2026-04-14T14:15:38.134Z [INFO] server-6b09effd-de6c-99f6-c6de-b3f7676a1839: election won: term=2 tally=2 2026-04-14T14:15:38.134Z [INFO] server-6b09effd-de6c-99f6-c6de-b3f7676a1839: entering leader state: leader="Node at 6b09effd-de6c-99f6-c6de-b3f7676a1839 [Leader]" 2026-04-14T14:15:38.134Z [INFO] server-6b09effd-de6c-99f6-c6de-b3f7676a1839: added peer, starting replication: peer=server-103c9309-dd42-0352-0aa3-9b3aace340e7 2026-04-14T14:15:38.134Z [INFO] server-6b09effd-de6c-99f6-c6de-b3f7676a1839: added peer, starting replication: peer=server-1f97bd0b-f2ca-6f14-be9a-5ee47ab779f1 2026-04-14T14:15:38.135Z [INFO] server-6b09effd-de6c-99f6-c6de-b3f7676a1839: pipelining replication: peer="{Voter server-1f97bd0b-f2ca-6f14-be9a-5ee47ab779f1 1f97bd0b-f2ca-6f14-be9a-5ee47ab779f1}" 2026-04-14T14:15:38.135Z [INFO] server-6b09effd-de6c-99f6-c6de-b3f7676a1839: pipelining replication: peer="{Voter server-103c9309-dd42-0352-0aa3-9b3aace340e7 103c9309-dd42-0352-0aa3-9b3aace340e7}" 2026-04-14T14:15:39.140Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:38.134968301 +0000 UTC m=+69.607154064 (1 nodes), 532.773967ms from start of poll, 533.140011ms from cluster start. Timeout at 2026-04-14 14:15:39.139972024 +0000 UTC m=+70.612157791, 1.005003727s after stability 2026-04-14T14:15:39.141Z [INFO] server-6b09effd-de6c-99f6-c6de-b3f7676a1839: starting snapshot up to: index=12 2026-04-14T14:15:39.141Z [INFO] creating new snapshot: path=/tmp/raft1051600905/snapshots/2-12-1776176139141.tmp 2026-04-14T14:15:39.277Z [INFO] server-6b09effd-de6c-99f6-c6de-b3f7676a1839: snapshot complete up to: index=12 2026-04-14T14:15:39.278Z [INFO] server-dc421e4c-8c8f-c79b-bbf2-fc23db8351a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc421e4c-8c8f-c79b-bbf2-fc23db8351a5 Address:dc421e4c-8c8f-c79b-bbf2-fc23db8351a5} {Suffrage:Voter ID:server-8bd52eef-dfb7-7dcb-03ec-a7970af94d2c Address:8bd52eef-dfb7-7dcb-03ec-a7970af94d2c} {Suffrage:Voter ID:server-0dac722c-5bdb-62f0-9ee7-97a139c02592 Address:0dac722c-5bdb-62f0-9ee7-97a139c02592}]" 2026-04-14T14:15:39.278Z [INFO] server-8bd52eef-dfb7-7dcb-03ec-a7970af94d2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc421e4c-8c8f-c79b-bbf2-fc23db8351a5 Address:dc421e4c-8c8f-c79b-bbf2-fc23db8351a5} {Suffrage:Voter ID:server-8bd52eef-dfb7-7dcb-03ec-a7970af94d2c Address:8bd52eef-dfb7-7dcb-03ec-a7970af94d2c} {Suffrage:Voter ID:server-0dac722c-5bdb-62f0-9ee7-97a139c02592 Address:0dac722c-5bdb-62f0-9ee7-97a139c02592}]" 2026-04-14T14:15:39.278Z [INFO] server-dc421e4c-8c8f-c79b-bbf2-fc23db8351a5: entering follower state: follower="Node at dc421e4c-8c8f-c79b-bbf2-fc23db8351a5 [Follower]" leader-address= leader-id= 2026-04-14T14:15:39.278Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc421e4c-8c8f-c79b-bbf2-fc23db8351a5 Address:dc421e4c-8c8f-c79b-bbf2-fc23db8351a5} {Suffrage:Voter ID:server-8bd52eef-dfb7-7dcb-03ec-a7970af94d2c Address:8bd52eef-dfb7-7dcb-03ec-a7970af94d2c} {Suffrage:Voter ID:server-0dac722c-5bdb-62f0-9ee7-97a139c02592 Address:0dac722c-5bdb-62f0-9ee7-97a139c02592}]" 2026-04-14T14:15:39.278Z [INFO] server-8bd52eef-dfb7-7dcb-03ec-a7970af94d2c: entering follower state: follower="Node at 8bd52eef-dfb7-7dcb-03ec-a7970af94d2c [Follower]" leader-address= leader-id= 2026-04-14T14:15:39.278Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:39.278Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: entering follower state: follower="Node at 0dac722c-5bdb-62f0-9ee7-97a139c02592 [Follower]" leader-address= leader-id= 2026-04-14T14:15:39.887Z [WARN] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:39.887Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: entering candidate state: node="Node at 0dac722c-5bdb-62f0-9ee7-97a139c02592 [Candidate]" term=2 2026-04-14T14:15:39.887Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: election won: term=2 tally=2 2026-04-14T14:15:39.887Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: entering leader state: leader="Node at 0dac722c-5bdb-62f0-9ee7-97a139c02592 [Leader]" 2026-04-14T14:15:39.887Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: added peer, starting replication: peer=server-dc421e4c-8c8f-c79b-bbf2-fc23db8351a5 2026-04-14T14:15:39.887Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: added peer, starting replication: peer=server-8bd52eef-dfb7-7dcb-03ec-a7970af94d2c 2026-04-14T14:15:39.887Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: pipelining replication: peer="{Voter server-8bd52eef-dfb7-7dcb-03ec-a7970af94d2c 8bd52eef-dfb7-7dcb-03ec-a7970af94d2c}" 2026-04-14T14:15:39.887Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: pipelining replication: peer="{Voter server-dc421e4c-8c8f-c79b-bbf2-fc23db8351a5 dc421e4c-8c8f-c79b-bbf2-fc23db8351a5}" 2026-04-14T14:15:40.892Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:39.887680676 +0000 UTC m=+71.359866537 (1 nodes), 608.943237ms from start of poll, 609.198936ms from cluster start. Timeout at 2026-04-14 14:15:40.892683672 +0000 UTC m=+72.364869437, 1.0050029s after stability 2026-04-14T14:15:40.892Z [INFO] creating new snapshot: path=/tmp/raft1574618812/snapshots/2-113-1776176140892.tmp 2026-04-14T14:15:41.173Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: copied to local snapshot: bytes=71 2026-04-14T14:15:41.173Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: snapshot restore progress: id=2-113-1776176140892 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:41.173Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: removing all old logs from log store 2026-04-14T14:15:41.173Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: compacting logs: from=1 to=2 2026-04-14T14:15:41.173Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: restored user snapshot: index=113 2026-04-14T14:15:41.173Z [ERROR] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: failed to get log: index=2 error="log not found" 2026-04-14T14:15:41.173Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: aborting pipeline replication: peer="{Voter server-8bd52eef-dfb7-7dcb-03ec-a7970af94d2c 8bd52eef-dfb7-7dcb-03ec-a7970af94d2c}" 2026-04-14T14:15:41.173Z [ERROR] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: failed to get log: index=2 error="log not found" 2026-04-14T14:15:41.173Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: aborting pipeline replication: peer="{Voter server-dc421e4c-8c8f-c79b-bbf2-fc23db8351a5 dc421e4c-8c8f-c79b-bbf2-fc23db8351a5}" 2026-04-14T14:15:41.180Z [ERROR] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: failed to get log: index=2 error="log not found" 2026-04-14T14:15:41.180Z [ERROR] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: failed to get log: index=2 error="log not found" 2026-04-14T14:15:41.180Z [INFO] creating new snapshot: path=/tmp/raft3833180304/snapshots/2-113-1776176141180.tmp 2026-04-14T14:15:41.367Z [INFO] server-8bd52eef-dfb7-7dcb-03ec-a7970af94d2c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:41.367Z [INFO] creating new snapshot: path=/tmp/raft1079355382/snapshots/2-113-1776176141367.tmp 2026-04-14T14:15:41.367Z [INFO] server-dc421e4c-8c8f-c79b-bbf2-fc23db8351a5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:41.367Z [INFO] server-8bd52eef-dfb7-7dcb-03ec-a7970af94d2c: copied to local snapshot: bytes=71 2026-04-14T14:15:41.367Z [INFO] server-8bd52eef-dfb7-7dcb-03ec-a7970af94d2c: snapshot restore progress: id=2-113-1776176141180 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:41.367Z [INFO] server-8bd52eef-dfb7-7dcb-03ec-a7970af94d2c: removing all old logs from log store 2026-04-14T14:15:41.367Z [INFO] server-8bd52eef-dfb7-7dcb-03ec-a7970af94d2c: compacting logs: from=1 to=2 2026-04-14T14:15:41.367Z [INFO] server-8bd52eef-dfb7-7dcb-03ec-a7970af94d2c: Installed remote snapshot 2026-04-14T14:15:41.367Z [INFO] server-dc421e4c-8c8f-c79b-bbf2-fc23db8351a5: copied to local snapshot: bytes=71 2026-04-14T14:15:41.367Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: pipelining replication: peer="{Voter server-8bd52eef-dfb7-7dcb-03ec-a7970af94d2c 8bd52eef-dfb7-7dcb-03ec-a7970af94d2c}" 2026-04-14T14:15:41.367Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:15:41.367Z [INFO] server-dc421e4c-8c8f-c79b-bbf2-fc23db8351a5: snapshot restore progress: id=2-113-1776176141367 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:41.367Z [INFO] server-dc421e4c-8c8f-c79b-bbf2-fc23db8351a5: removing all old logs from log store 2026-04-14T14:15:41.367Z [INFO] server-dc421e4c-8c8f-c79b-bbf2-fc23db8351a5: compacting logs: from=1 to=2 2026-04-14T14:15:41.367Z [INFO] server-dc421e4c-8c8f-c79b-bbf2-fc23db8351a5: Installed remote snapshot 2026-04-14T14:15:41.367Z [INFO] server-0dac722c-5bdb-62f0-9ee7-97a139c02592: pipelining replication: peer="{Voter server-dc421e4c-8c8f-c79b-bbf2-fc23db8351a5 dc421e4c-8c8f-c79b-bbf2-fc23db8351a5}" 2026-04-14T14:15:42.373Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:15:41.367802755 +0000 UTC m=+72.839988530 (2 nodes), 736ns from start of poll, 2.089320929s from cluster start. Timeout at 2026-04-14 14:15:42.372810061 +0000 UTC m=+73.844995824, 1.005007294s after stability 2026-04-14T14:15:42.386Z [INFO] server-6b09effd-de6c-99f6-c6de-b3f7676a1839: aborting pipeline replication: peer="{Voter server-103c9309-dd42-0352-0aa3-9b3aace340e7 103c9309-dd42-0352-0aa3-9b3aace340e7}" 2026-04-14T14:15:42.386Z [INFO] server-6b09effd-de6c-99f6-c6de-b3f7676a1839: aborting pipeline replication: peer="{Voter server-1f97bd0b-f2ca-6f14-be9a-5ee47ab779f1 1f97bd0b-f2ca-6f14-be9a-5ee47ab779f1}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false 2026-04-14T14:15:42.555Z [INFO] server-04f0db91-0198-679c-ae49-78635e29e567: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04f0db91-0198-679c-ae49-78635e29e567 Address:04f0db91-0198-679c-ae49-78635e29e567} {Suffrage:Voter ID:server-3dcba3b0-66d9-7fd2-6520-04086ba11881 Address:3dcba3b0-66d9-7fd2-6520-04086ba11881} {Suffrage:Voter ID:server-80c4d40a-f507-f8ea-f2f6-dd89e57664fb Address:80c4d40a-f507-f8ea-f2f6-dd89e57664fb}]" 2026-04-14T14:15:42.555Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04f0db91-0198-679c-ae49-78635e29e567 Address:04f0db91-0198-679c-ae49-78635e29e567} {Suffrage:Voter ID:server-3dcba3b0-66d9-7fd2-6520-04086ba11881 Address:3dcba3b0-66d9-7fd2-6520-04086ba11881} {Suffrage:Voter ID:server-80c4d40a-f507-f8ea-f2f6-dd89e57664fb Address:80c4d40a-f507-f8ea-f2f6-dd89e57664fb}]" 2026-04-14T14:15:42.555Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: entering follower state: follower="Node at 3dcba3b0-66d9-7fd2-6520-04086ba11881 [Follower]" leader-address= leader-id= 2026-04-14T14:15:42.555Z [INFO] server-80c4d40a-f507-f8ea-f2f6-dd89e57664fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04f0db91-0198-679c-ae49-78635e29e567 Address:04f0db91-0198-679c-ae49-78635e29e567} {Suffrage:Voter ID:server-3dcba3b0-66d9-7fd2-6520-04086ba11881 Address:3dcba3b0-66d9-7fd2-6520-04086ba11881} {Suffrage:Voter ID:server-80c4d40a-f507-f8ea-f2f6-dd89e57664fb Address:80c4d40a-f507-f8ea-f2f6-dd89e57664fb}]" 2026-04-14T14:15:42.555Z [INFO] server-04f0db91-0198-679c-ae49-78635e29e567: entering follower state: follower="Node at 04f0db91-0198-679c-ae49-78635e29e567 [Follower]" leader-address= leader-id= 2026-04-14T14:15:42.555Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:42.555Z [INFO] server-80c4d40a-f507-f8ea-f2f6-dd89e57664fb: entering follower state: follower="Node at 80c4d40a-f507-f8ea-f2f6-dd89e57664fb [Follower]" leader-address= leader-id= 2026-04-14T14:15:43.218Z [WARN] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:43.218Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: entering candidate state: node="Node at 3dcba3b0-66d9-7fd2-6520-04086ba11881 [Candidate]" term=2 2026-04-14T14:15:43.218Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: election won: term=2 tally=2 2026-04-14T14:15:43.218Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: entering leader state: leader="Node at 3dcba3b0-66d9-7fd2-6520-04086ba11881 [Leader]" 2026-04-14T14:15:43.218Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: added peer, starting replication: peer=server-04f0db91-0198-679c-ae49-78635e29e567 2026-04-14T14:15:43.218Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: added peer, starting replication: peer=server-80c4d40a-f507-f8ea-f2f6-dd89e57664fb 2026-04-14T14:15:43.218Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: pipelining replication: peer="{Voter server-80c4d40a-f507-f8ea-f2f6-dd89e57664fb 80c4d40a-f507-f8ea-f2f6-dd89e57664fb}" 2026-04-14T14:15:43.218Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: pipelining replication: peer="{Voter server-04f0db91-0198-679c-ae49-78635e29e567 04f0db91-0198-679c-ae49-78635e29e567}" 2026-04-14T14:15:44.223Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:43.218413411 +0000 UTC m=+74.690599175 (1 nodes), 662.918311ms from start of poll, 663.299872ms from cluster start. Timeout at 2026-04-14 14:15:44.223416786 +0000 UTC m=+75.695602549, 1.005003374s after stability 2026-04-14T14:15:44.224Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: starting snapshot up to: index=12 2026-04-14T14:15:44.224Z [INFO] creating new snapshot: path=/tmp/raft334065571/snapshots/2-12-1776176144224.tmp 2026-04-14T14:15:44.224Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: snapshot complete up to: index=12 2026-04-14T14:15:44.225Z [INFO] creating new snapshot: path=/tmp/raft334065571/snapshots/2-1013-1776176144225.tmp 2026-04-14T14:15:44.225Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: copied to local snapshot: bytes=71 2026-04-14T14:15:44.225Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: snapshot restore progress: id=2-1013-1776176144225 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:44.225Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: restored user snapshot: index=1013 2026-04-14T14:15:44.225Z [ERROR] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: failed to get log: index=23 error="log not found" 2026-04-14T14:15:44.225Z [ERROR] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: failed to get log: index=23 error="log not found" 2026-04-14T14:15:44.225Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: aborting pipeline replication: peer="{Voter server-04f0db91-0198-679c-ae49-78635e29e567 04f0db91-0198-679c-ae49-78635e29e567}" 2026-04-14T14:15:44.225Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: aborting pipeline replication: peer="{Voter server-80c4d40a-f507-f8ea-f2f6-dd89e57664fb 80c4d40a-f507-f8ea-f2f6-dd89e57664fb}" 2026-04-14T14:15:44.235Z [ERROR] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: failed to get log: index=23 error="log not found" 2026-04-14T14:15:44.235Z [ERROR] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: failed to get log: index=23 error="log not found" 2026-04-14T14:15:44.235Z [INFO] creating new snapshot: path=/tmp/raft147347801/snapshots/2-1013-1776176144235.tmp 2026-04-14T14:15:44.235Z [INFO] creating new snapshot: path=/tmp/raft1132508466/snapshots/2-1013-1776176144235.tmp 2026-04-14T14:15:44.236Z [INFO] server-04f0db91-0198-679c-ae49-78635e29e567: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:44.236Z [INFO] server-80c4d40a-f507-f8ea-f2f6-dd89e57664fb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:44.236Z [INFO] server-04f0db91-0198-679c-ae49-78635e29e567: copied to local snapshot: bytes=71 2026-04-14T14:15:44.236Z [INFO] server-80c4d40a-f507-f8ea-f2f6-dd89e57664fb: copied to local snapshot: bytes=71 2026-04-14T14:15:44.236Z [INFO] server-04f0db91-0198-679c-ae49-78635e29e567: snapshot restore progress: id=2-1013-1776176144235 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:44.236Z [INFO] server-04f0db91-0198-679c-ae49-78635e29e567: Installed remote snapshot 2026-04-14T14:15:44.236Z [WARN] server-04f0db91-0198-679c-ae49-78635e29e567: skipping application of old log: index=22 2026-04-14T14:15:44.236Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: pipelining replication: peer="{Voter server-04f0db91-0198-679c-ae49-78635e29e567 04f0db91-0198-679c-ae49-78635e29e567}" 2026-04-14T14:15:44.236Z [INFO] server-80c4d40a-f507-f8ea-f2f6-dd89e57664fb: snapshot restore progress: id=2-1013-1776176144235 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:44.236Z [INFO] server-80c4d40a-f507-f8ea-f2f6-dd89e57664fb: Installed remote snapshot 2026-04-14T14:15:44.236Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:15:44.236Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: pipelining replication: peer="{Voter server-80c4d40a-f507-f8ea-f2f6-dd89e57664fb 80c4d40a-f507-f8ea-f2f6-dd89e57664fb}" 2026-04-14T14:15:45.242Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:15:44.236653072 +0000 UTC m=+75.708838839 (2 nodes), 1.924µs from start of poll, 1.681539536s from cluster start. Timeout at 2026-04-14 14:15:45.241672242 +0000 UTC m=+76.713858028, 1.005019189s after stability 2026-04-14T14:15:45.254Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: aborting pipeline replication: peer="{Voter server-80c4d40a-f507-f8ea-f2f6-dd89e57664fb 80c4d40a-f507-f8ea-f2f6-dd89e57664fb}" 2026-04-14T14:15:45.254Z [INFO] server-3dcba3b0-66d9-7fd2-6520-04086ba11881: aborting pipeline replication: peer="{Voter server-04f0db91-0198-679c-ae49-78635e29e567 04f0db91-0198-679c-ae49-78635e29e567}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false 2026-04-14T14:15:45.257Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7872a119-22eb-af62-92c3-a4d71270d515 Address:7872a119-22eb-af62-92c3-a4d71270d515} {Suffrage:Voter ID:server-b2896666-5877-cb28-98d4-825432dcb12a Address:b2896666-5877-cb28-98d4-825432dcb12a} {Suffrage:Voter ID:server-5f475ecb-917a-fde8-14cc-abf0186f36bf Address:5f475ecb-917a-fde8-14cc-abf0186f36bf}]" 2026-04-14T14:15:45.257Z [INFO] server-b2896666-5877-cb28-98d4-825432dcb12a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7872a119-22eb-af62-92c3-a4d71270d515 Address:7872a119-22eb-af62-92c3-a4d71270d515} {Suffrage:Voter ID:server-b2896666-5877-cb28-98d4-825432dcb12a Address:b2896666-5877-cb28-98d4-825432dcb12a} {Suffrage:Voter ID:server-5f475ecb-917a-fde8-14cc-abf0186f36bf Address:5f475ecb-917a-fde8-14cc-abf0186f36bf}]" 2026-04-14T14:15:45.257Z [INFO] server-5f475ecb-917a-fde8-14cc-abf0186f36bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7872a119-22eb-af62-92c3-a4d71270d515 Address:7872a119-22eb-af62-92c3-a4d71270d515} {Suffrage:Voter ID:server-b2896666-5877-cb28-98d4-825432dcb12a Address:b2896666-5877-cb28-98d4-825432dcb12a} {Suffrage:Voter ID:server-5f475ecb-917a-fde8-14cc-abf0186f36bf Address:5f475ecb-917a-fde8-14cc-abf0186f36bf}]" 2026-04-14T14:15:45.257Z [INFO] server-b2896666-5877-cb28-98d4-825432dcb12a: entering follower state: follower="Node at b2896666-5877-cb28-98d4-825432dcb12a [Follower]" leader-address= leader-id= 2026-04-14T14:15:45.257Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:45.257Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: entering follower state: follower="Node at 7872a119-22eb-af62-92c3-a4d71270d515 [Follower]" leader-address= leader-id= 2026-04-14T14:15:45.257Z [INFO] server-5f475ecb-917a-fde8-14cc-abf0186f36bf: entering follower state: follower="Node at 5f475ecb-917a-fde8-14cc-abf0186f36bf [Follower]" leader-address= leader-id= 2026-04-14T14:15:45.882Z [WARN] server-7872a119-22eb-af62-92c3-a4d71270d515: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:45.882Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: entering candidate state: node="Node at 7872a119-22eb-af62-92c3-a4d71270d515 [Candidate]" term=2 2026-04-14T14:15:45.882Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: election won: term=2 tally=2 2026-04-14T14:15:45.882Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: entering leader state: leader="Node at 7872a119-22eb-af62-92c3-a4d71270d515 [Leader]" 2026-04-14T14:15:45.882Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: added peer, starting replication: peer=server-b2896666-5877-cb28-98d4-825432dcb12a 2026-04-14T14:15:45.882Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: added peer, starting replication: peer=server-5f475ecb-917a-fde8-14cc-abf0186f36bf 2026-04-14T14:15:45.882Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: pipelining replication: peer="{Voter server-b2896666-5877-cb28-98d4-825432dcb12a b2896666-5877-cb28-98d4-825432dcb12a}" 2026-04-14T14:15:45.882Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: pipelining replication: peer="{Voter server-5f475ecb-917a-fde8-14cc-abf0186f36bf 5f475ecb-917a-fde8-14cc-abf0186f36bf}" 2026-04-14T14:15:46.888Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:45.88280168 +0000 UTC m=+77.354987522 (1 nodes), 625.20446ms from start of poll, 625.578092ms from cluster start. Timeout at 2026-04-14 14:15:46.887805178 +0000 UTC m=+78.359990946, 1.005003424s after stability 2026-04-14T14:15:46.889Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: starting snapshot up to: index=12 2026-04-14T14:15:46.889Z [INFO] creating new snapshot: path=/tmp/raft3901074462/snapshots/2-12-1776176146889.tmp 2026-04-14T14:15:46.889Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: snapshot complete up to: index=12 2026-04-14T14:15:46.890Z [INFO] creating new snapshot: path=/tmp/raft3901074462/snapshots/2-1013-1776176146890.tmp 2026-04-14T14:15:46.891Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: copied to local snapshot: bytes=71 2026-04-14T14:15:46.892Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: snapshot restore progress: id=2-1013-1776176146890 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:46.893Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: removing all old logs from log store 2026-04-14T14:15:46.893Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: compacting logs: from=1 to=22 2026-04-14T14:15:46.893Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: restored user snapshot: index=1013 2026-04-14T14:15:46.893Z [ERROR] server-7872a119-22eb-af62-92c3-a4d71270d515: failed to get log: index=22 error="log not found" 2026-04-14T14:15:46.893Z [ERROR] server-7872a119-22eb-af62-92c3-a4d71270d515: failed to get log: index=22 error="log not found" 2026-04-14T14:15:46.893Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: aborting pipeline replication: peer="{Voter server-b2896666-5877-cb28-98d4-825432dcb12a b2896666-5877-cb28-98d4-825432dcb12a}" 2026-04-14T14:15:46.893Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: aborting pipeline replication: peer="{Voter server-5f475ecb-917a-fde8-14cc-abf0186f36bf 5f475ecb-917a-fde8-14cc-abf0186f36bf}" 2026-04-14T14:15:46.901Z [ERROR] server-7872a119-22eb-af62-92c3-a4d71270d515: failed to get log: index=22 error="log not found" 2026-04-14T14:15:46.901Z [ERROR] server-7872a119-22eb-af62-92c3-a4d71270d515: failed to get log: index=22 error="log not found" 2026-04-14T14:15:46.901Z [INFO] creating new snapshot: path=/tmp/raft2070284097/snapshots/2-1013-1776176146901.tmp 2026-04-14T14:15:46.901Z [INFO] creating new snapshot: path=/tmp/raft3488556104/snapshots/2-1013-1776176146901.tmp 2026-04-14T14:15:46.901Z [INFO] server-5f475ecb-917a-fde8-14cc-abf0186f36bf: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:46.902Z [INFO] server-b2896666-5877-cb28-98d4-825432dcb12a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:46.902Z [INFO] server-5f475ecb-917a-fde8-14cc-abf0186f36bf: copied to local snapshot: bytes=71 2026-04-14T14:15:46.902Z [INFO] server-b2896666-5877-cb28-98d4-825432dcb12a: copied to local snapshot: bytes=71 2026-04-14T14:15:46.902Z [INFO] server-5f475ecb-917a-fde8-14cc-abf0186f36bf: snapshot restore progress: id=2-1013-1776176146901 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:46.902Z [INFO] server-5f475ecb-917a-fde8-14cc-abf0186f36bf: removing all old logs from log store 2026-04-14T14:15:46.902Z [INFO] server-5f475ecb-917a-fde8-14cc-abf0186f36bf: compacting logs: from=1 to=22 2026-04-14T14:15:46.902Z [INFO] server-5f475ecb-917a-fde8-14cc-abf0186f36bf: Installed remote snapshot 2026-04-14T14:15:46.902Z [INFO] server-b2896666-5877-cb28-98d4-825432dcb12a: snapshot restore progress: id=2-1013-1776176146901 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:46.902Z [WARN] server-5f475ecb-917a-fde8-14cc-abf0186f36bf: skipping application of old log: index=22 2026-04-14T14:15:46.902Z [INFO] server-b2896666-5877-cb28-98d4-825432dcb12a: removing all old logs from log store 2026-04-14T14:15:46.902Z [INFO] server-b2896666-5877-cb28-98d4-825432dcb12a: compacting logs: from=1 to=22 2026-04-14T14:15:46.902Z [INFO] server-b2896666-5877-cb28-98d4-825432dcb12a: Installed remote snapshot 2026-04-14T14:15:46.902Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: pipelining replication: peer="{Voter server-5f475ecb-917a-fde8-14cc-abf0186f36bf 5f475ecb-917a-fde8-14cc-abf0186f36bf}" 2026-04-14T14:15:46.902Z [WARN] server-b2896666-5877-cb28-98d4-825432dcb12a: skipping application of old log: index=1013 2026-04-14T14:15:46.902Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:15:46.902Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: pipelining replication: peer="{Voter server-b2896666-5877-cb28-98d4-825432dcb12a b2896666-5877-cb28-98d4-825432dcb12a}" 2026-04-14T14:15:47.907Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:15:46.902692324 +0000 UTC m=+78.374878121 (2 nodes), 762ns from start of poll, 1.645468691s from cluster start. Timeout at 2026-04-14 14:15:47.907708868 +0000 UTC m=+79.379894636, 1.005016515s after stability 2026-04-14T14:15:47.919Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: aborting pipeline replication: peer="{Voter server-5f475ecb-917a-fde8-14cc-abf0186f36bf 5f475ecb-917a-fde8-14cc-abf0186f36bf}" 2026-04-14T14:15:47.919Z [INFO] server-7872a119-22eb-af62-92c3-a4d71270d515: aborting pipeline replication: peer="{Voter server-b2896666-5877-cb28-98d4-825432dcb12a b2896666-5877-cb28-98d4-825432dcb12a}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true 2026-04-14T14:15:48.092Z [INFO] server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8 Address:aecdcb3f-3d60-a580-7b3d-55ad5127d7d8} {Suffrage:Voter ID:server-27c818ca-3f8f-383b-e358-911ad843d499 Address:27c818ca-3f8f-383b-e358-911ad843d499} {Suffrage:Voter ID:server-db4fd913-cb9d-d59c-7cfb-eeb527f9d7d5 Address:db4fd913-cb9d-d59c-7cfb-eeb527f9d7d5}]" 2026-04-14T14:15:48.092Z [INFO] server-27c818ca-3f8f-383b-e358-911ad843d499: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8 Address:aecdcb3f-3d60-a580-7b3d-55ad5127d7d8} {Suffrage:Voter ID:server-27c818ca-3f8f-383b-e358-911ad843d499 Address:27c818ca-3f8f-383b-e358-911ad843d499} {Suffrage:Voter ID:server-db4fd913-cb9d-d59c-7cfb-eeb527f9d7d5 Address:db4fd913-cb9d-d59c-7cfb-eeb527f9d7d5}]" 2026-04-14T14:15:48.092Z [INFO] server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8: entering follower state: follower="Node at aecdcb3f-3d60-a580-7b3d-55ad5127d7d8 [Follower]" leader-address= leader-id= 2026-04-14T14:15:48.092Z [INFO] server-27c818ca-3f8f-383b-e358-911ad843d499: entering follower state: follower="Node at 27c818ca-3f8f-383b-e358-911ad843d499 [Follower]" leader-address= leader-id= 2026-04-14T14:15:48.092Z [INFO] server-db4fd913-cb9d-d59c-7cfb-eeb527f9d7d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8 Address:aecdcb3f-3d60-a580-7b3d-55ad5127d7d8} {Suffrage:Voter ID:server-27c818ca-3f8f-383b-e358-911ad843d499 Address:27c818ca-3f8f-383b-e358-911ad843d499} {Suffrage:Voter ID:server-db4fd913-cb9d-d59c-7cfb-eeb527f9d7d5 Address:db4fd913-cb9d-d59c-7cfb-eeb527f9d7d5}]" 2026-04-14T14:15:48.092Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:48.092Z [INFO] server-db4fd913-cb9d-d59c-7cfb-eeb527f9d7d5: entering follower state: follower="Node at db4fd913-cb9d-d59c-7cfb-eeb527f9d7d5 [Follower]" leader-address= leader-id= 2026-04-14T14:15:48.599Z [WARN] server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:48.599Z [INFO] server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8: entering candidate state: node="Node at aecdcb3f-3d60-a580-7b3d-55ad5127d7d8 [Candidate]" term=2 2026-04-14T14:15:48.599Z [INFO] server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8: election won: term=2 tally=2 2026-04-14T14:15:48.599Z [INFO] server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8: entering leader state: leader="Node at aecdcb3f-3d60-a580-7b3d-55ad5127d7d8 [Leader]" 2026-04-14T14:15:48.599Z [INFO] server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8: added peer, starting replication: peer=server-27c818ca-3f8f-383b-e358-911ad843d499 2026-04-14T14:15:48.599Z [INFO] server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8: added peer, starting replication: peer=server-db4fd913-cb9d-d59c-7cfb-eeb527f9d7d5 2026-04-14T14:15:48.599Z [INFO] server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8: pipelining replication: peer="{Voter server-27c818ca-3f8f-383b-e358-911ad843d499 27c818ca-3f8f-383b-e358-911ad843d499}" 2026-04-14T14:15:48.599Z [INFO] server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8: pipelining replication: peer="{Voter server-db4fd913-cb9d-d59c-7cfb-eeb527f9d7d5 db4fd913-cb9d-d59c-7cfb-eeb527f9d7d5}" 2026-04-14T14:15:49.605Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:48.599684323 +0000 UTC m=+80.071870102 (1 nodes), 507.314873ms from start of poll, 507.568553ms from cluster start. Timeout at 2026-04-14 14:15:49.604685261 +0000 UTC m=+81.076871031, 1.005000929s after stability 2026-04-14T14:15:49.605Z [INFO] server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8: starting snapshot up to: index=12 2026-04-14T14:15:49.605Z [INFO] creating new snapshot: path=/tmp/raft488835977/snapshots/2-12-1776176149605.tmp 2026-04-14T14:15:49.606Z [INFO] server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8: snapshot complete up to: index=12 2026-04-14T14:15:49.607Z [INFO] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-344fff22-e8f1-b0ac-b77c-e81b414dc922 Address:344fff22-e8f1-b0ac-b77c-e81b414dc922} {Suffrage:Voter ID:server-1c6d7084-b9cf-2a69-5308-da283e66bc5e Address:1c6d7084-b9cf-2a69-5308-da283e66bc5e} {Suffrage:Voter ID:server-9fecc72a-714c-6722-673b-6f93ee19a567 Address:9fecc72a-714c-6722-673b-6f93ee19a567}]" 2026-04-14T14:15:49.607Z [INFO] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: entering follower state: follower="Node at 344fff22-e8f1-b0ac-b77c-e81b414dc922 [Follower]" leader-address= leader-id= 2026-04-14T14:15:49.607Z [INFO] server-1c6d7084-b9cf-2a69-5308-da283e66bc5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-344fff22-e8f1-b0ac-b77c-e81b414dc922 Address:344fff22-e8f1-b0ac-b77c-e81b414dc922} {Suffrage:Voter ID:server-1c6d7084-b9cf-2a69-5308-da283e66bc5e Address:1c6d7084-b9cf-2a69-5308-da283e66bc5e} {Suffrage:Voter ID:server-9fecc72a-714c-6722-673b-6f93ee19a567 Address:9fecc72a-714c-6722-673b-6f93ee19a567}]" 2026-04-14T14:15:49.607Z [INFO] server-9fecc72a-714c-6722-673b-6f93ee19a567: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-344fff22-e8f1-b0ac-b77c-e81b414dc922 Address:344fff22-e8f1-b0ac-b77c-e81b414dc922} {Suffrage:Voter ID:server-1c6d7084-b9cf-2a69-5308-da283e66bc5e Address:1c6d7084-b9cf-2a69-5308-da283e66bc5e} {Suffrage:Voter ID:server-9fecc72a-714c-6722-673b-6f93ee19a567 Address:9fecc72a-714c-6722-673b-6f93ee19a567}]" 2026-04-14T14:15:49.607Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:49.607Z [INFO] server-1c6d7084-b9cf-2a69-5308-da283e66bc5e: entering follower state: follower="Node at 1c6d7084-b9cf-2a69-5308-da283e66bc5e [Follower]" leader-address= leader-id= 2026-04-14T14:15:49.607Z [INFO] server-9fecc72a-714c-6722-673b-6f93ee19a567: entering follower state: follower="Node at 9fecc72a-714c-6722-673b-6f93ee19a567 [Follower]" leader-address= leader-id= 2026-04-14T14:15:50.130Z [WARN] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:50.130Z [INFO] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: entering candidate state: node="Node at 344fff22-e8f1-b0ac-b77c-e81b414dc922 [Candidate]" term=2 2026-04-14T14:15:50.130Z [INFO] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: election won: term=2 tally=2 2026-04-14T14:15:50.130Z [INFO] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: entering leader state: leader="Node at 344fff22-e8f1-b0ac-b77c-e81b414dc922 [Leader]" 2026-04-14T14:15:50.130Z [INFO] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: added peer, starting replication: peer=server-1c6d7084-b9cf-2a69-5308-da283e66bc5e 2026-04-14T14:15:50.130Z [INFO] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: added peer, starting replication: peer=server-9fecc72a-714c-6722-673b-6f93ee19a567 2026-04-14T14:15:50.131Z [INFO] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: pipelining replication: peer="{Voter server-9fecc72a-714c-6722-673b-6f93ee19a567 9fecc72a-714c-6722-673b-6f93ee19a567}" 2026-04-14T14:15:50.131Z [INFO] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: pipelining replication: peer="{Voter server-1c6d7084-b9cf-2a69-5308-da283e66bc5e 1c6d7084-b9cf-2a69-5308-da283e66bc5e}" 2026-04-14T14:15:51.136Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:50.13113598 +0000 UTC m=+81.603321755 (1 nodes), 523.81054ms from start of poll, 524.093393ms from cluster start. Timeout at 2026-04-14 14:15:51.136143327 +0000 UTC m=+82.608329104, 1.005007349s after stability 2026-04-14T14:15:51.137Z [INFO] creating new snapshot: path=/tmp/raft108657998/snapshots/2-1013-1776176151137.tmp 2026-04-14T14:15:51.224Z [INFO] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: copied to local snapshot: bytes=71 2026-04-14T14:15:51.224Z [INFO] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: snapshot restore progress: id=2-1013-1776176151137 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:51.224Z [INFO] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: restored user snapshot: index=1013 2026-04-14T14:15:51.224Z [ERROR] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: failed to get log: index=3 error="log not found" 2026-04-14T14:15:51.224Z [INFO] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: aborting pipeline replication: peer="{Voter server-9fecc72a-714c-6722-673b-6f93ee19a567 9fecc72a-714c-6722-673b-6f93ee19a567}" 2026-04-14T14:15:51.224Z [ERROR] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: failed to get log: index=3 error="log not found" 2026-04-14T14:15:51.224Z [INFO] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: aborting pipeline replication: peer="{Voter server-1c6d7084-b9cf-2a69-5308-da283e66bc5e 1c6d7084-b9cf-2a69-5308-da283e66bc5e}" 2026-04-14T14:15:51.231Z [ERROR] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: failed to get log: index=3 error="log not found" 2026-04-14T14:15:51.231Z [ERROR] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: failed to get log: index=3 error="log not found" 2026-04-14T14:15:51.231Z [INFO] creating new snapshot: path=/tmp/raft3260441433/snapshots/2-1013-1776176151231.tmp 2026-04-14T14:15:51.232Z [INFO] creating new snapshot: path=/tmp/raft1089850020/snapshots/2-1013-1776176151232.tmp 2026-04-14T14:15:51.233Z [INFO] server-9fecc72a-714c-6722-673b-6f93ee19a567: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:51.233Z [INFO] server-1c6d7084-b9cf-2a69-5308-da283e66bc5e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:51.257Z [INFO] server-1c6d7084-b9cf-2a69-5308-da283e66bc5e: copied to local snapshot: bytes=71 2026-04-14T14:15:51.257Z [INFO] server-9fecc72a-714c-6722-673b-6f93ee19a567: copied to local snapshot: bytes=71 2026-04-14T14:15:51.257Z [INFO] server-1c6d7084-b9cf-2a69-5308-da283e66bc5e: snapshot restore progress: id=2-1013-1776176151231 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:51.257Z [INFO] server-1c6d7084-b9cf-2a69-5308-da283e66bc5e: Installed remote snapshot 2026-04-14T14:15:51.257Z [INFO] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: pipelining replication: peer="{Voter server-1c6d7084-b9cf-2a69-5308-da283e66bc5e 1c6d7084-b9cf-2a69-5308-da283e66bc5e}" 2026-04-14T14:15:51.257Z [INFO] server-9fecc72a-714c-6722-673b-6f93ee19a567: snapshot restore progress: id=2-1013-1776176151232 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:51.257Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:15:51.257Z [INFO] server-9fecc72a-714c-6722-673b-6f93ee19a567: Installed remote snapshot 2026-04-14T14:15:51.257Z [INFO] server-344fff22-e8f1-b0ac-b77c-e81b414dc922: pipelining replication: peer="{Voter server-9fecc72a-714c-6722-673b-6f93ee19a567 9fecc72a-714c-6722-673b-6f93ee19a567}" 2026-04-14T14:15:52.263Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:15:51.257787756 +0000 UTC m=+82.729973525 (2 nodes), 663ns from start of poll, 1.650745163s from cluster start. Timeout at 2026-04-14 14:15:52.262790744 +0000 UTC m=+83.734976513, 1.005002988s after stability 2026-04-14T14:15:52.275Z [INFO] server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8: aborting pipeline replication: peer="{Voter server-db4fd913-cb9d-d59c-7cfb-eeb527f9d7d5 db4fd913-cb9d-d59c-7cfb-eeb527f9d7d5}" 2026-04-14T14:15:52.275Z [INFO] server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8: aborting pipeline replication: peer="{Voter server-27c818ca-3f8f-383b-e358-911ad843d499 27c818ca-3f8f-383b-e358-911ad843d499}" 2026-04-14T14:15:52.775Z [ERROR] server-aecdcb3f-3d60-a580-7b3d-55ad5127d7d8: failed to heartbeat to: peer=db4fd913-cb9d-d59c-7cfb-eeb527f9d7d5 backoff time=10ms error="command timed out" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true 2026-04-14T14:15:52.974Z [INFO] server-367d873c-a02d-6b25-c9da-2b23d46cd3e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-367d873c-a02d-6b25-c9da-2b23d46cd3e8 Address:367d873c-a02d-6b25-c9da-2b23d46cd3e8} {Suffrage:Voter ID:server-c68b50b0-33d1-7be0-1daa-74e31bde763c Address:c68b50b0-33d1-7be0-1daa-74e31bde763c} {Suffrage:Voter ID:server-053852b6-5f45-57b7-5f23-a6638254faac Address:053852b6-5f45-57b7-5f23-a6638254faac}]" 2026-04-14T14:15:52.974Z [INFO] server-c68b50b0-33d1-7be0-1daa-74e31bde763c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-367d873c-a02d-6b25-c9da-2b23d46cd3e8 Address:367d873c-a02d-6b25-c9da-2b23d46cd3e8} {Suffrage:Voter ID:server-c68b50b0-33d1-7be0-1daa-74e31bde763c Address:c68b50b0-33d1-7be0-1daa-74e31bde763c} {Suffrage:Voter ID:server-053852b6-5f45-57b7-5f23-a6638254faac Address:053852b6-5f45-57b7-5f23-a6638254faac}]" 2026-04-14T14:15:52.974Z [INFO] server-367d873c-a02d-6b25-c9da-2b23d46cd3e8: entering follower state: follower="Node at 367d873c-a02d-6b25-c9da-2b23d46cd3e8 [Follower]" leader-address= leader-id= 2026-04-14T14:15:52.974Z [INFO] server-c68b50b0-33d1-7be0-1daa-74e31bde763c: entering follower state: follower="Node at c68b50b0-33d1-7be0-1daa-74e31bde763c [Follower]" leader-address= leader-id= 2026-04-14T14:15:52.974Z [INFO] server-053852b6-5f45-57b7-5f23-a6638254faac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-367d873c-a02d-6b25-c9da-2b23d46cd3e8 Address:367d873c-a02d-6b25-c9da-2b23d46cd3e8} {Suffrage:Voter ID:server-c68b50b0-33d1-7be0-1daa-74e31bde763c Address:c68b50b0-33d1-7be0-1daa-74e31bde763c} {Suffrage:Voter ID:server-053852b6-5f45-57b7-5f23-a6638254faac Address:053852b6-5f45-57b7-5f23-a6638254faac}]" 2026-04-14T14:15:52.974Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:52.974Z [INFO] server-053852b6-5f45-57b7-5f23-a6638254faac: entering follower state: follower="Node at 053852b6-5f45-57b7-5f23-a6638254faac [Follower]" leader-address= leader-id= 2026-04-14T14:15:53.594Z [WARN] server-c68b50b0-33d1-7be0-1daa-74e31bde763c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:53.594Z [INFO] server-c68b50b0-33d1-7be0-1daa-74e31bde763c: entering candidate state: node="Node at c68b50b0-33d1-7be0-1daa-74e31bde763c [Candidate]" term=2 2026-04-14T14:15:53.594Z [INFO] server-c68b50b0-33d1-7be0-1daa-74e31bde763c: election won: term=2 tally=2 2026-04-14T14:15:53.594Z [INFO] server-c68b50b0-33d1-7be0-1daa-74e31bde763c: entering leader state: leader="Node at c68b50b0-33d1-7be0-1daa-74e31bde763c [Leader]" 2026-04-14T14:15:53.594Z [INFO] server-c68b50b0-33d1-7be0-1daa-74e31bde763c: added peer, starting replication: peer=server-367d873c-a02d-6b25-c9da-2b23d46cd3e8 2026-04-14T14:15:53.594Z [INFO] server-c68b50b0-33d1-7be0-1daa-74e31bde763c: added peer, starting replication: peer=server-053852b6-5f45-57b7-5f23-a6638254faac 2026-04-14T14:15:53.594Z [INFO] server-c68b50b0-33d1-7be0-1daa-74e31bde763c: pipelining replication: peer="{Voter server-367d873c-a02d-6b25-c9da-2b23d46cd3e8 367d873c-a02d-6b25-c9da-2b23d46cd3e8}" 2026-04-14T14:15:53.594Z [INFO] server-c68b50b0-33d1-7be0-1daa-74e31bde763c: pipelining replication: peer="{Voter server-053852b6-5f45-57b7-5f23-a6638254faac 053852b6-5f45-57b7-5f23-a6638254faac}" 2026-04-14T14:15:54.599Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:53.594683253 +0000 UTC m=+85.066869022 (1 nodes), 619.785261ms from start of poll, 620.093809ms from cluster start. Timeout at 2026-04-14 14:15:54.599687308 +0000 UTC m=+86.071873071, 1.005004049s after stability 2026-04-14T14:15:54.600Z [INFO] server-c68b50b0-33d1-7be0-1daa-74e31bde763c: starting snapshot up to: index=12 2026-04-14T14:15:54.600Z [INFO] creating new snapshot: path=/tmp/raft236203635/snapshots/2-12-1776176154600.tmp 2026-04-14T14:15:54.825Z [INFO] server-c68b50b0-33d1-7be0-1daa-74e31bde763c: snapshot complete up to: index=12 2026-04-14T14:15:54.826Z [INFO] server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf Address:8b811ef1-79e7-bc6f-27da-5f07bb405fdf} {Suffrage:Voter ID:server-428d5645-4941-5654-5c90-f93a4f1ef3df Address:428d5645-4941-5654-5c90-f93a4f1ef3df} {Suffrage:Voter ID:server-5e183b11-b897-9bfb-b91b-c74b266eff01 Address:5e183b11-b897-9bfb-b91b-c74b266eff01}]" 2026-04-14T14:15:54.826Z [INFO] server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf: entering follower state: follower="Node at 8b811ef1-79e7-bc6f-27da-5f07bb405fdf [Follower]" leader-address= leader-id= 2026-04-14T14:15:54.826Z [INFO] server-428d5645-4941-5654-5c90-f93a4f1ef3df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf Address:8b811ef1-79e7-bc6f-27da-5f07bb405fdf} {Suffrage:Voter ID:server-428d5645-4941-5654-5c90-f93a4f1ef3df Address:428d5645-4941-5654-5c90-f93a4f1ef3df} {Suffrage:Voter ID:server-5e183b11-b897-9bfb-b91b-c74b266eff01 Address:5e183b11-b897-9bfb-b91b-c74b266eff01}]" 2026-04-14T14:15:54.826Z [INFO] server-428d5645-4941-5654-5c90-f93a4f1ef3df: entering follower state: follower="Node at 428d5645-4941-5654-5c90-f93a4f1ef3df [Follower]" leader-address= leader-id= 2026-04-14T14:15:54.826Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf Address:8b811ef1-79e7-bc6f-27da-5f07bb405fdf} {Suffrage:Voter ID:server-428d5645-4941-5654-5c90-f93a4f1ef3df Address:428d5645-4941-5654-5c90-f93a4f1ef3df} {Suffrage:Voter ID:server-5e183b11-b897-9bfb-b91b-c74b266eff01 Address:5e183b11-b897-9bfb-b91b-c74b266eff01}]" 2026-04-14T14:15:54.826Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: entering follower state: follower="Node at 5e183b11-b897-9bfb-b91b-c74b266eff01 [Follower]" leader-address= leader-id= 2026-04-14T14:15:54.826Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:55.360Z [WARN] server-5e183b11-b897-9bfb-b91b-c74b266eff01: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:55.360Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: entering candidate state: node="Node at 5e183b11-b897-9bfb-b91b-c74b266eff01 [Candidate]" term=2 2026-04-14T14:15:55.360Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: election won: term=2 tally=2 2026-04-14T14:15:55.360Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: entering leader state: leader="Node at 5e183b11-b897-9bfb-b91b-c74b266eff01 [Leader]" 2026-04-14T14:15:55.360Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: added peer, starting replication: peer=server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf 2026-04-14T14:15:55.360Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: added peer, starting replication: peer=server-428d5645-4941-5654-5c90-f93a4f1ef3df 2026-04-14T14:15:55.360Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: pipelining replication: peer="{Voter server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf 8b811ef1-79e7-bc6f-27da-5f07bb405fdf}" 2026-04-14T14:15:55.360Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: pipelining replication: peer="{Voter server-428d5645-4941-5654-5c90-f93a4f1ef3df 428d5645-4941-5654-5c90-f93a4f1ef3df}" 2026-04-14T14:15:56.366Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:55.360855884 +0000 UTC m=+86.833041652 (1 nodes), 534.257766ms from start of poll, 534.642392ms from cluster start. Timeout at 2026-04-14 14:15:56.365861806 +0000 UTC m=+87.838047575, 1.005005923s after stability 2026-04-14T14:15:56.367Z [INFO] creating new snapshot: path=/tmp/raft1414433529/snapshots/2-1013-1776176156367.tmp 2026-04-14T14:15:56.373Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: copied to local snapshot: bytes=71 2026-04-14T14:15:56.373Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: snapshot restore progress: id=2-1013-1776176156367 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:56.373Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: removing all old logs from log store 2026-04-14T14:15:56.373Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: compacting logs: from=1 to=2 2026-04-14T14:15:56.373Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: restored user snapshot: index=1013 2026-04-14T14:15:56.373Z [ERROR] server-5e183b11-b897-9bfb-b91b-c74b266eff01: failed to get log: index=2 error="log not found" 2026-04-14T14:15:56.373Z [ERROR] server-5e183b11-b897-9bfb-b91b-c74b266eff01: failed to get log: index=2 error="log not found" 2026-04-14T14:15:56.373Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: aborting pipeline replication: peer="{Voter server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf 8b811ef1-79e7-bc6f-27da-5f07bb405fdf}" 2026-04-14T14:15:56.373Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: aborting pipeline replication: peer="{Voter server-428d5645-4941-5654-5c90-f93a4f1ef3df 428d5645-4941-5654-5c90-f93a4f1ef3df}" 2026-04-14T14:15:56.381Z [ERROR] server-5e183b11-b897-9bfb-b91b-c74b266eff01: failed to get log: index=2 error="log not found" 2026-04-14T14:15:56.381Z [INFO] creating new snapshot: path=/tmp/raft3672420048/snapshots/2-1013-1776176156381.tmp 2026-04-14T14:15:56.381Z [INFO] server-428d5645-4941-5654-5c90-f93a4f1ef3df: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:56.383Z [ERROR] server-5e183b11-b897-9bfb-b91b-c74b266eff01: failed to get log: index=2 error="log not found" 2026-04-14T14:15:56.383Z [INFO] creating new snapshot: path=/tmp/raft3728562781/snapshots/2-1013-1776176156383.tmp 2026-04-14T14:15:56.868Z [WARN] server-5e183b11-b897-9bfb-b91b-c74b266eff01: failed to contact: server-id=server-428d5645-4941-5654-5c90-f93a4f1ef3df time=500.819816ms 2026-04-14T14:15:56.879Z [WARN] server-5e183b11-b897-9bfb-b91b-c74b266eff01: failed to contact: server-id=server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf time=506.58097ms 2026-04-14T14:15:56.879Z [WARN] server-5e183b11-b897-9bfb-b91b-c74b266eff01: failed to contact: server-id=server-428d5645-4941-5654-5c90-f93a4f1ef3df time=511.312489ms 2026-04-14T14:15:56.879Z [WARN] server-5e183b11-b897-9bfb-b91b-c74b266eff01: failed to contact quorum of nodes, stepping down 2026-04-14T14:15:56.879Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: entering follower state: follower="Node at 5e183b11-b897-9bfb-b91b-c74b266eff01 [Follower]" leader-address= leader-id= raft_test.go:1598: Restore failed: leadership lost while committing log 2026-04-14T14:15:56.879Z [INFO] server-c68b50b0-33d1-7be0-1daa-74e31bde763c: aborting pipeline replication: peer="{Voter server-053852b6-5f45-57b7-5f23-a6638254faac 053852b6-5f45-57b7-5f23-a6638254faac}" 2026-04-14T14:15:56.879Z [INFO] server-c68b50b0-33d1-7be0-1daa-74e31bde763c: aborting pipeline replication: peer="{Voter server-367d873c-a02d-6b25-c9da-2b23d46cd3e8 367d873c-a02d-6b25-c9da-2b23d46cd3e8}" 2026-04-14T14:15:56.883Z [INFO] server-428d5645-4941-5654-5c90-f93a4f1ef3df: copied to local snapshot: bytes=71 2026-04-14T14:15:56.883Z [INFO] server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:56.884Z [INFO] server-428d5645-4941-5654-5c90-f93a4f1ef3df: snapshot restore progress: id=2-1013-1776176156381 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:56.884Z [INFO] server-428d5645-4941-5654-5c90-f93a4f1ef3df: removing all old logs from log store 2026-04-14T14:15:56.884Z [INFO] server-428d5645-4941-5654-5c90-f93a4f1ef3df: compacting logs: from=1 to=2 2026-04-14T14:15:56.884Z [INFO] server-428d5645-4941-5654-5c90-f93a4f1ef3df: Installed remote snapshot 2026-04-14T14:15:56.884Z [INFO] server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf: copied to local snapshot: bytes=71 === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false 2026-04-14T14:15:56.884Z [INFO] server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf: snapshot restore progress: id=2-1013-1776176156383 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:56.884Z [INFO] server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf: removing all old logs from log store 2026-04-14T14:15:56.884Z [INFO] server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf: compacting logs: from=1 to=2 2026-04-14T14:15:56.884Z [INFO] server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf: Installed remote snapshot 2026-04-14T14:15:56.885Z [INFO] server-377910a6-054c-dd4d-9f57-0501141c5026: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-377910a6-054c-dd4d-9f57-0501141c5026 Address:377910a6-054c-dd4d-9f57-0501141c5026} {Suffrage:Voter ID:server-7da0cf03-5112-a4ad-3a76-54c8eaa64641 Address:7da0cf03-5112-a4ad-3a76-54c8eaa64641} {Suffrage:Voter ID:server-c328a319-14cf-004f-46d5-10e593fb55b3 Address:c328a319-14cf-004f-46d5-10e593fb55b3}]" 2026-04-14T14:15:56.885Z [INFO] server-7da0cf03-5112-a4ad-3a76-54c8eaa64641: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-377910a6-054c-dd4d-9f57-0501141c5026 Address:377910a6-054c-dd4d-9f57-0501141c5026} {Suffrage:Voter ID:server-7da0cf03-5112-a4ad-3a76-54c8eaa64641 Address:7da0cf03-5112-a4ad-3a76-54c8eaa64641} {Suffrage:Voter ID:server-c328a319-14cf-004f-46d5-10e593fb55b3 Address:c328a319-14cf-004f-46d5-10e593fb55b3}]" 2026-04-14T14:15:56.885Z [INFO] server-377910a6-054c-dd4d-9f57-0501141c5026: entering follower state: follower="Node at 377910a6-054c-dd4d-9f57-0501141c5026 [Follower]" leader-address= leader-id= 2026-04-14T14:15:56.885Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-377910a6-054c-dd4d-9f57-0501141c5026 Address:377910a6-054c-dd4d-9f57-0501141c5026} {Suffrage:Voter ID:server-7da0cf03-5112-a4ad-3a76-54c8eaa64641 Address:7da0cf03-5112-a4ad-3a76-54c8eaa64641} {Suffrage:Voter ID:server-c328a319-14cf-004f-46d5-10e593fb55b3 Address:c328a319-14cf-004f-46d5-10e593fb55b3}]" 2026-04-14T14:15:56.885Z [INFO] server-7da0cf03-5112-a4ad-3a76-54c8eaa64641: entering follower state: follower="Node at 7da0cf03-5112-a4ad-3a76-54c8eaa64641 [Follower]" leader-address= leader-id= 2026-04-14T14:15:56.885Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:15:56.885Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: entering follower state: follower="Node at c328a319-14cf-004f-46d5-10e593fb55b3 [Follower]" leader-address= leader-id= 2026-04-14T14:15:57.515Z [WARN] server-5e183b11-b897-9bfb-b91b-c74b266eff01: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:57.515Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: entering candidate state: node="Node at 5e183b11-b897-9bfb-b91b-c74b266eff01 [Candidate]" term=3 2026-04-14T14:15:57.515Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: election won: term=3 tally=2 2026-04-14T14:15:57.515Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: entering leader state: leader="Node at 5e183b11-b897-9bfb-b91b-c74b266eff01 [Leader]" 2026-04-14T14:15:57.515Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: added peer, starting replication: peer=server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf 2026-04-14T14:15:57.515Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: added peer, starting replication: peer=server-428d5645-4941-5654-5c90-f93a4f1ef3df 2026-04-14T14:15:57.515Z [WARN] server-428d5645-4941-5654-5c90-f93a4f1ef3df: failed to get previous log: previous-index=1014 last-index=1013 error="log not found" 2026-04-14T14:15:57.515Z [WARN] server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf: failed to get previous log: previous-index=1014 last-index=1013 error="log not found" 2026-04-14T14:15:57.515Z [WARN] server-5e183b11-b897-9bfb-b91b-c74b266eff01: appendEntries rejected, sending older logs: peer="{Voter server-428d5645-4941-5654-5c90-f93a4f1ef3df 428d5645-4941-5654-5c90-f93a4f1ef3df}" next=1014 2026-04-14T14:15:57.515Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: pipelining replication: peer="{Voter server-428d5645-4941-5654-5c90-f93a4f1ef3df 428d5645-4941-5654-5c90-f93a4f1ef3df}" 2026-04-14T14:15:57.515Z [WARN] server-5e183b11-b897-9bfb-b91b-c74b266eff01: appendEntries rejected, sending older logs: peer="{Voter server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf 8b811ef1-79e7-bc6f-27da-5f07bb405fdf}" next=1014 2026-04-14T14:15:57.515Z [INFO] server-5e183b11-b897-9bfb-b91b-c74b266eff01: pipelining replication: peer="{Voter server-8b811ef1-79e7-bc6f-27da-5f07bb405fdf 8b811ef1-79e7-bc6f-27da-5f07bb405fdf}" 2026-04-14T14:15:57.566Z [WARN] server-c328a319-14cf-004f-46d5-10e593fb55b3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:15:57.566Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: entering candidate state: node="Node at c328a319-14cf-004f-46d5-10e593fb55b3 [Candidate]" term=2 2026-04-14T14:15:57.566Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: election won: term=2 tally=2 2026-04-14T14:15:57.566Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: entering leader state: leader="Node at c328a319-14cf-004f-46d5-10e593fb55b3 [Leader]" 2026-04-14T14:15:57.566Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: added peer, starting replication: peer=server-377910a6-054c-dd4d-9f57-0501141c5026 2026-04-14T14:15:57.566Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: added peer, starting replication: peer=server-7da0cf03-5112-a4ad-3a76-54c8eaa64641 2026-04-14T14:15:57.566Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: pipelining replication: peer="{Voter server-377910a6-054c-dd4d-9f57-0501141c5026 377910a6-054c-dd4d-9f57-0501141c5026}" 2026-04-14T14:15:57.566Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: pipelining replication: peer="{Voter server-7da0cf03-5112-a4ad-3a76-54c8eaa64641 7da0cf03-5112-a4ad-3a76-54c8eaa64641}" 2026-04-14T14:15:58.572Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:15:57.566485596 +0000 UTC m=+89.038671361 (1 nodes), 681.176319ms from start of poll, 681.511191ms from cluster start. Timeout at 2026-04-14 14:15:58.571552541 +0000 UTC m=+90.043738309, 1.005066948s after stability 2026-04-14T14:15:58.572Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: starting snapshot up to: index=12 2026-04-14T14:15:58.572Z [INFO] creating new snapshot: path=/tmp/raft566898725/snapshots/2-12-1776176158572.tmp 2026-04-14T14:15:58.601Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: snapshot complete up to: index=12 2026-04-14T14:15:58.601Z [INFO] creating new snapshot: path=/tmp/raft566898725/snapshots/2-10013-1776176158601.tmp 2026-04-14T14:15:58.684Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: copied to local snapshot: bytes=71 2026-04-14T14:15:58.684Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: snapshot restore progress: id=2-10013-1776176158601 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:58.684Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: restored user snapshot: index=10013 2026-04-14T14:15:58.684Z [ERROR] server-c328a319-14cf-004f-46d5-10e593fb55b3: failed to get log: index=23 error="log not found" 2026-04-14T14:15:58.684Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: aborting pipeline replication: peer="{Voter server-7da0cf03-5112-a4ad-3a76-54c8eaa64641 7da0cf03-5112-a4ad-3a76-54c8eaa64641}" 2026-04-14T14:15:58.684Z [ERROR] server-c328a319-14cf-004f-46d5-10e593fb55b3: failed to get log: index=23 error="log not found" 2026-04-14T14:15:58.684Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: aborting pipeline replication: peer="{Voter server-377910a6-054c-dd4d-9f57-0501141c5026 377910a6-054c-dd4d-9f57-0501141c5026}" 2026-04-14T14:15:58.691Z [ERROR] server-c328a319-14cf-004f-46d5-10e593fb55b3: failed to get log: index=23 error="log not found" 2026-04-14T14:15:58.694Z [ERROR] server-c328a319-14cf-004f-46d5-10e593fb55b3: failed to get log: index=23 error="log not found" 2026-04-14T14:15:58.993Z [INFO] creating new snapshot: path=/tmp/raft765976460/snapshots/2-10013-1776176158993.tmp 2026-04-14T14:15:58.993Z [INFO] creating new snapshot: path=/tmp/raft3900704422/snapshots/2-10013-1776176158993.tmp 2026-04-14T14:15:58.993Z [INFO] server-7da0cf03-5112-a4ad-3a76-54c8eaa64641: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:58.994Z [INFO] server-377910a6-054c-dd4d-9f57-0501141c5026: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:58.994Z [INFO] server-7da0cf03-5112-a4ad-3a76-54c8eaa64641: copied to local snapshot: bytes=71 2026-04-14T14:15:58.994Z [INFO] server-377910a6-054c-dd4d-9f57-0501141c5026: copied to local snapshot: bytes=71 2026-04-14T14:15:58.994Z [INFO] server-7da0cf03-5112-a4ad-3a76-54c8eaa64641: snapshot restore progress: id=2-10013-1776176158993 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:58.994Z [INFO] server-7da0cf03-5112-a4ad-3a76-54c8eaa64641: Installed remote snapshot 2026-04-14T14:15:58.994Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: pipelining replication: peer="{Voter server-7da0cf03-5112-a4ad-3a76-54c8eaa64641 7da0cf03-5112-a4ad-3a76-54c8eaa64641}" 2026-04-14T14:15:58.994Z [INFO] server-377910a6-054c-dd4d-9f57-0501141c5026: snapshot restore progress: id=2-10013-1776176158993 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:15:58.994Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:15:58.994Z [INFO] server-377910a6-054c-dd4d-9f57-0501141c5026: Installed remote snapshot 2026-04-14T14:15:58.995Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: pipelining replication: peer="{Voter server-377910a6-054c-dd4d-9f57-0501141c5026 377910a6-054c-dd4d-9f57-0501141c5026}" 2026-04-14T14:16:00.001Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:15:58.994879013 +0000 UTC m=+90.467064804 (2 nodes), 1.366µs from start of poll, 2.109904634s from cluster start. Timeout at 2026-04-14 14:15:59.999898407 +0000 UTC m=+91.472084175, 1.005019371s after stability 2026-04-14T14:16:00.012Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: aborting pipeline replication: peer="{Voter server-7da0cf03-5112-a4ad-3a76-54c8eaa64641 7da0cf03-5112-a4ad-3a76-54c8eaa64641}" 2026-04-14T14:16:00.012Z [INFO] server-c328a319-14cf-004f-46d5-10e593fb55b3: aborting pipeline replication: peer="{Voter server-377910a6-054c-dd4d-9f57-0501141c5026 377910a6-054c-dd4d-9f57-0501141c5026}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false 2026-04-14T14:16:00.013Z [INFO] server-f9c08b02-2c16-b154-4ec1-29cd9da4cb9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f9c08b02-2c16-b154-4ec1-29cd9da4cb9f Address:f9c08b02-2c16-b154-4ec1-29cd9da4cb9f} {Suffrage:Voter ID:server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb Address:f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb} {Suffrage:Voter ID:server-19686fde-8987-9d7d-87be-9348d957dc9b Address:19686fde-8987-9d7d-87be-9348d957dc9b}]" 2026-04-14T14:16:00.013Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f9c08b02-2c16-b154-4ec1-29cd9da4cb9f Address:f9c08b02-2c16-b154-4ec1-29cd9da4cb9f} {Suffrage:Voter ID:server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb Address:f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb} {Suffrage:Voter ID:server-19686fde-8987-9d7d-87be-9348d957dc9b Address:19686fde-8987-9d7d-87be-9348d957dc9b}]" 2026-04-14T14:16:00.013Z [INFO] server-f9c08b02-2c16-b154-4ec1-29cd9da4cb9f: entering follower state: follower="Node at f9c08b02-2c16-b154-4ec1-29cd9da4cb9f [Follower]" leader-address= leader-id= 2026-04-14T14:16:00.013Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: entering follower state: follower="Node at f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb [Follower]" leader-address= leader-id= 2026-04-14T14:16:00.013Z [INFO] server-19686fde-8987-9d7d-87be-9348d957dc9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f9c08b02-2c16-b154-4ec1-29cd9da4cb9f Address:f9c08b02-2c16-b154-4ec1-29cd9da4cb9f} {Suffrage:Voter ID:server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb Address:f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb} {Suffrage:Voter ID:server-19686fde-8987-9d7d-87be-9348d957dc9b Address:19686fde-8987-9d7d-87be-9348d957dc9b}]" 2026-04-14T14:16:00.013Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:00.013Z [INFO] server-19686fde-8987-9d7d-87be-9348d957dc9b: entering follower state: follower="Node at 19686fde-8987-9d7d-87be-9348d957dc9b [Follower]" leader-address= leader-id= 2026-04-14T14:16:00.819Z [WARN] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:00.819Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: entering candidate state: node="Node at f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb [Candidate]" term=2 2026-04-14T14:16:00.819Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: election won: term=2 tally=2 2026-04-14T14:16:00.819Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: entering leader state: leader="Node at f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb [Leader]" 2026-04-14T14:16:00.819Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: added peer, starting replication: peer=server-f9c08b02-2c16-b154-4ec1-29cd9da4cb9f 2026-04-14T14:16:00.819Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: added peer, starting replication: peer=server-19686fde-8987-9d7d-87be-9348d957dc9b 2026-04-14T14:16:00.819Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: pipelining replication: peer="{Voter server-f9c08b02-2c16-b154-4ec1-29cd9da4cb9f f9c08b02-2c16-b154-4ec1-29cd9da4cb9f}" 2026-04-14T14:16:00.819Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: pipelining replication: peer="{Voter server-19686fde-8987-9d7d-87be-9348d957dc9b 19686fde-8987-9d7d-87be-9348d957dc9b}" 2026-04-14T14:16:01.825Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:00.819554481 +0000 UTC m=+92.291740254 (1 nodes), 805.910331ms from start of poll, 806.215273ms from cluster start. Timeout at 2026-04-14 14:16:01.824558289 +0000 UTC m=+93.296744056, 1.005003802s after stability 2026-04-14T14:16:01.825Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: starting snapshot up to: index=12 2026-04-14T14:16:01.825Z [INFO] creating new snapshot: path=/tmp/raft1066370199/snapshots/2-12-1776176161825.tmp 2026-04-14T14:16:02.046Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: snapshot complete up to: index=12 2026-04-14T14:16:02.046Z [INFO] creating new snapshot: path=/tmp/raft1066370199/snapshots/2-10013-1776176162046.tmp 2026-04-14T14:16:02.048Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: copied to local snapshot: bytes=71 2026-04-14T14:16:02.048Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: snapshot restore progress: id=2-10013-1776176162046 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:16:02.048Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: removing all old logs from log store 2026-04-14T14:16:02.048Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: compacting logs: from=1 to=22 2026-04-14T14:16:02.048Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: restored user snapshot: index=10013 2026-04-14T14:16:02.048Z [ERROR] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: failed to get log: index=22 error="log not found" 2026-04-14T14:16:02.048Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: aborting pipeline replication: peer="{Voter server-19686fde-8987-9d7d-87be-9348d957dc9b 19686fde-8987-9d7d-87be-9348d957dc9b}" 2026-04-14T14:16:02.048Z [ERROR] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: failed to get log: index=22 error="log not found" 2026-04-14T14:16:02.048Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: aborting pipeline replication: peer="{Voter server-f9c08b02-2c16-b154-4ec1-29cd9da4cb9f f9c08b02-2c16-b154-4ec1-29cd9da4cb9f}" 2026-04-14T14:16:02.055Z [ERROR] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: failed to get log: index=22 error="log not found" 2026-04-14T14:16:02.055Z [INFO] creating new snapshot: path=/tmp/raft3078412005/snapshots/2-10013-1776176162055.tmp 2026-04-14T14:16:02.056Z [INFO] server-f9c08b02-2c16-b154-4ec1-29cd9da4cb9f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:16:02.056Z [INFO] server-f9c08b02-2c16-b154-4ec1-29cd9da4cb9f: copied to local snapshot: bytes=71 2026-04-14T14:16:02.056Z [INFO] server-f9c08b02-2c16-b154-4ec1-29cd9da4cb9f: snapshot restore progress: id=2-10013-1776176162055 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:16:02.056Z [INFO] server-f9c08b02-2c16-b154-4ec1-29cd9da4cb9f: removing all old logs from log store 2026-04-14T14:16:02.056Z [INFO] server-f9c08b02-2c16-b154-4ec1-29cd9da4cb9f: compacting logs: from=1 to=22 2026-04-14T14:16:02.056Z [INFO] server-f9c08b02-2c16-b154-4ec1-29cd9da4cb9f: Installed remote snapshot 2026-04-14T14:16:02.056Z [WARN] server-f9c08b02-2c16-b154-4ec1-29cd9da4cb9f: skipping application of old log: index=22 2026-04-14T14:16:02.056Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: pipelining replication: peer="{Voter server-f9c08b02-2c16-b154-4ec1-29cd9da4cb9f f9c08b02-2c16-b154-4ec1-29cd9da4cb9f}" 2026-04-14T14:16:02.056Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:02.059Z [ERROR] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: failed to get log: index=22 error="log not found" 2026-04-14T14:16:02.059Z [INFO] creating new snapshot: path=/tmp/raft3607177976/snapshots/2-10013-1776176162059.tmp 2026-04-14T14:16:02.059Z [INFO] server-19686fde-8987-9d7d-87be-9348d957dc9b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:16:02.059Z [INFO] server-19686fde-8987-9d7d-87be-9348d957dc9b: copied to local snapshot: bytes=71 2026-04-14T14:16:02.059Z [INFO] server-19686fde-8987-9d7d-87be-9348d957dc9b: snapshot restore progress: id=2-10013-1776176162059 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:16:02.059Z [INFO] server-19686fde-8987-9d7d-87be-9348d957dc9b: removing all old logs from log store 2026-04-14T14:16:02.059Z [INFO] server-19686fde-8987-9d7d-87be-9348d957dc9b: compacting logs: from=1 to=22 2026-04-14T14:16:02.059Z [INFO] server-19686fde-8987-9d7d-87be-9348d957dc9b: Installed remote snapshot 2026-04-14T14:16:02.060Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: pipelining replication: peer="{Voter server-19686fde-8987-9d7d-87be-9348d957dc9b 19686fde-8987-9d7d-87be-9348d957dc9b}" 2026-04-14T14:16:03.062Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:02.056961527 +0000 UTC m=+93.529147295 (2 nodes), 850ns from start of poll, 2.043622314s from cluster start. Timeout at 2026-04-14 14:16:03.061970762 +0000 UTC m=+94.534156527, 1.005009232s after stability 2026-04-14T14:16:03.074Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: aborting pipeline replication: peer="{Voter server-19686fde-8987-9d7d-87be-9348d957dc9b 19686fde-8987-9d7d-87be-9348d957dc9b}" 2026-04-14T14:16:03.074Z [INFO] server-f02ef0a7-13eb-3eb7-0a7c-f8ec5c104ceb: aborting pipeline replication: peer="{Voter server-f9c08b02-2c16-b154-4ec1-29cd9da4cb9f f9c08b02-2c16-b154-4ec1-29cd9da4cb9f}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true 2026-04-14T14:16:03.075Z [INFO] server-caf2a85a-703d-5057-a930-57981335518f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-caf2a85a-703d-5057-a930-57981335518f Address:caf2a85a-703d-5057-a930-57981335518f} {Suffrage:Voter ID:server-b42bf1b1-a3e3-dca2-9eec-78b767fcf41a Address:b42bf1b1-a3e3-dca2-9eec-78b767fcf41a} {Suffrage:Voter ID:server-77073cbb-b09b-5203-3a05-0fd2b2c3fc6d Address:77073cbb-b09b-5203-3a05-0fd2b2c3fc6d}]" 2026-04-14T14:16:03.075Z [INFO] server-caf2a85a-703d-5057-a930-57981335518f: entering follower state: follower="Node at caf2a85a-703d-5057-a930-57981335518f [Follower]" leader-address= leader-id= 2026-04-14T14:16:03.075Z [INFO] server-b42bf1b1-a3e3-dca2-9eec-78b767fcf41a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-caf2a85a-703d-5057-a930-57981335518f Address:caf2a85a-703d-5057-a930-57981335518f} {Suffrage:Voter ID:server-b42bf1b1-a3e3-dca2-9eec-78b767fcf41a Address:b42bf1b1-a3e3-dca2-9eec-78b767fcf41a} {Suffrage:Voter ID:server-77073cbb-b09b-5203-3a05-0fd2b2c3fc6d Address:77073cbb-b09b-5203-3a05-0fd2b2c3fc6d}]" 2026-04-14T14:16:03.075Z [INFO] server-77073cbb-b09b-5203-3a05-0fd2b2c3fc6d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-caf2a85a-703d-5057-a930-57981335518f Address:caf2a85a-703d-5057-a930-57981335518f} {Suffrage:Voter ID:server-b42bf1b1-a3e3-dca2-9eec-78b767fcf41a Address:b42bf1b1-a3e3-dca2-9eec-78b767fcf41a} {Suffrage:Voter ID:server-77073cbb-b09b-5203-3a05-0fd2b2c3fc6d Address:77073cbb-b09b-5203-3a05-0fd2b2c3fc6d}]" 2026-04-14T14:16:03.075Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:03.075Z [INFO] server-b42bf1b1-a3e3-dca2-9eec-78b767fcf41a: entering follower state: follower="Node at b42bf1b1-a3e3-dca2-9eec-78b767fcf41a [Follower]" leader-address= leader-id= 2026-04-14T14:16:03.075Z [INFO] server-77073cbb-b09b-5203-3a05-0fd2b2c3fc6d: entering follower state: follower="Node at 77073cbb-b09b-5203-3a05-0fd2b2c3fc6d [Follower]" leader-address= leader-id= 2026-04-14T14:16:03.624Z [WARN] server-77073cbb-b09b-5203-3a05-0fd2b2c3fc6d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:03.624Z [INFO] server-77073cbb-b09b-5203-3a05-0fd2b2c3fc6d: entering candidate state: node="Node at 77073cbb-b09b-5203-3a05-0fd2b2c3fc6d [Candidate]" term=2 2026-04-14T14:16:03.624Z [INFO] server-77073cbb-b09b-5203-3a05-0fd2b2c3fc6d: election won: term=2 tally=2 2026-04-14T14:16:03.624Z [INFO] server-77073cbb-b09b-5203-3a05-0fd2b2c3fc6d: entering leader state: leader="Node at 77073cbb-b09b-5203-3a05-0fd2b2c3fc6d [Leader]" 2026-04-14T14:16:03.624Z [INFO] server-77073cbb-b09b-5203-3a05-0fd2b2c3fc6d: added peer, starting replication: peer=server-caf2a85a-703d-5057-a930-57981335518f 2026-04-14T14:16:03.624Z [INFO] server-77073cbb-b09b-5203-3a05-0fd2b2c3fc6d: added peer, starting replication: peer=server-b42bf1b1-a3e3-dca2-9eec-78b767fcf41a 2026-04-14T14:16:03.624Z [INFO] server-77073cbb-b09b-5203-3a05-0fd2b2c3fc6d: pipelining replication: peer="{Voter server-caf2a85a-703d-5057-a930-57981335518f caf2a85a-703d-5057-a930-57981335518f}" 2026-04-14T14:16:03.624Z [INFO] server-77073cbb-b09b-5203-3a05-0fd2b2c3fc6d: pipelining replication: peer="{Voter server-b42bf1b1-a3e3-dca2-9eec-78b767fcf41a b42bf1b1-a3e3-dca2-9eec-78b767fcf41a}" 2026-04-14T14:16:04.629Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:03.624659743 +0000 UTC m=+95.096845526 (1 nodes), 548.992098ms from start of poll, 549.292871ms from cluster start. Timeout at 2026-04-14 14:16:04.629662462 +0000 UTC m=+96.101848227, 1.005002701s after stability 2026-04-14T14:16:04.630Z [INFO] server-77073cbb-b09b-5203-3a05-0fd2b2c3fc6d: starting snapshot up to: index=12 2026-04-14T14:16:04.630Z [INFO] creating new snapshot: path=/tmp/raft3210964170/snapshots/2-12-1776176164630.tmp 2026-04-14T14:16:04.630Z [INFO] server-77073cbb-b09b-5203-3a05-0fd2b2c3fc6d: snapshot complete up to: index=12 2026-04-14T14:16:04.631Z [INFO] server-da096b5c-91e7-e24d-97ae-cf29fb8facb9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da096b5c-91e7-e24d-97ae-cf29fb8facb9 Address:da096b5c-91e7-e24d-97ae-cf29fb8facb9} {Suffrage:Voter ID:server-25f3c358-3ffd-93e8-549b-d607185c0f0f Address:25f3c358-3ffd-93e8-549b-d607185c0f0f} {Suffrage:Voter ID:server-e0a67d8a-3841-5334-1588-2d049039ad25 Address:e0a67d8a-3841-5334-1588-2d049039ad25}]" 2026-04-14T14:16:04.631Z [INFO] server-da096b5c-91e7-e24d-97ae-cf29fb8facb9: entering follower state: follower="Node at da096b5c-91e7-e24d-97ae-cf29fb8facb9 [Follower]" leader-address= leader-id= 2026-04-14T14:16:04.631Z [INFO] server-25f3c358-3ffd-93e8-549b-d607185c0f0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da096b5c-91e7-e24d-97ae-cf29fb8facb9 Address:da096b5c-91e7-e24d-97ae-cf29fb8facb9} {Suffrage:Voter ID:server-25f3c358-3ffd-93e8-549b-d607185c0f0f Address:25f3c358-3ffd-93e8-549b-d607185c0f0f} {Suffrage:Voter ID:server-e0a67d8a-3841-5334-1588-2d049039ad25 Address:e0a67d8a-3841-5334-1588-2d049039ad25}]" 2026-04-14T14:16:04.631Z [INFO] server-25f3c358-3ffd-93e8-549b-d607185c0f0f: entering follower state: follower="Node at 25f3c358-3ffd-93e8-549b-d607185c0f0f [Follower]" leader-address= leader-id= 2026-04-14T14:16:04.631Z [INFO] server-e0a67d8a-3841-5334-1588-2d049039ad25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da096b5c-91e7-e24d-97ae-cf29fb8facb9 Address:da096b5c-91e7-e24d-97ae-cf29fb8facb9} {Suffrage:Voter ID:server-25f3c358-3ffd-93e8-549b-d607185c0f0f Address:25f3c358-3ffd-93e8-549b-d607185c0f0f} {Suffrage:Voter ID:server-e0a67d8a-3841-5334-1588-2d049039ad25 Address:e0a67d8a-3841-5334-1588-2d049039ad25}]" 2026-04-14T14:16:04.631Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:04.631Z [INFO] server-e0a67d8a-3841-5334-1588-2d049039ad25: entering follower state: follower="Node at e0a67d8a-3841-5334-1588-2d049039ad25 [Follower]" leader-address= leader-id= 2026-04-14T14:16:05.208Z [WARN] server-e0a67d8a-3841-5334-1588-2d049039ad25: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:05.208Z [INFO] server-e0a67d8a-3841-5334-1588-2d049039ad25: entering candidate state: node="Node at e0a67d8a-3841-5334-1588-2d049039ad25 [Candidate]" term=2 2026-04-14T14:16:05.208Z [INFO] server-e0a67d8a-3841-5334-1588-2d049039ad25: election won: term=2 tally=2 2026-04-14T14:16:05.208Z [INFO] server-e0a67d8a-3841-5334-1588-2d049039ad25: entering leader state: leader="Node at e0a67d8a-3841-5334-1588-2d049039ad25 [Leader]" 2026-04-14T14:16:05.208Z [INFO] server-e0a67d8a-3841-5334-1588-2d049039ad25: added peer, starting replication: peer=server-da096b5c-91e7-e24d-97ae-cf29fb8facb9 2026-04-14T14:16:05.208Z [INFO] server-e0a67d8a-3841-5334-1588-2d049039ad25: added peer, starting replication: peer=server-25f3c358-3ffd-93e8-549b-d607185c0f0f 2026-04-14T14:16:05.209Z [INFO] server-e0a67d8a-3841-5334-1588-2d049039ad25: pipelining replication: peer="{Voter server-da096b5c-91e7-e24d-97ae-cf29fb8facb9 da096b5c-91e7-e24d-97ae-cf29fb8facb9}" 2026-04-14T14:16:05.209Z [INFO] server-e0a67d8a-3841-5334-1588-2d049039ad25: pipelining replication: peer="{Voter server-25f3c358-3ffd-93e8-549b-d607185c0f0f 25f3c358-3ffd-93e8-549b-d607185c0f0f}" 2026-04-14T14:16:06.214Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:05.208926792 +0000 UTC m=+96.681112561 (1 nodes), 577.184538ms from start of poll, 577.488795ms from cluster start. Timeout at 2026-04-14 14:16:06.213927732 +0000 UTC m=+97.686113499, 1.005000938s after stability 2026-04-14T14:16:06.214Z [INFO] creating new snapshot: path=/tmp/raft3303473477/snapshots/2-10013-1776176166214.tmp 2026-04-14T14:16:06.215Z [INFO] server-e0a67d8a-3841-5334-1588-2d049039ad25: copied to local snapshot: bytes=71 2026-04-14T14:16:06.215Z [INFO] server-e0a67d8a-3841-5334-1588-2d049039ad25: snapshot restore progress: id=2-10013-1776176166214 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:16:06.215Z [INFO] server-e0a67d8a-3841-5334-1588-2d049039ad25: restored user snapshot: index=10013 2026-04-14T14:16:06.215Z [ERROR] server-e0a67d8a-3841-5334-1588-2d049039ad25: failed to get log: index=3 error="log not found" 2026-04-14T14:16:06.215Z [INFO] server-e0a67d8a-3841-5334-1588-2d049039ad25: aborting pipeline replication: peer="{Voter server-25f3c358-3ffd-93e8-549b-d607185c0f0f 25f3c358-3ffd-93e8-549b-d607185c0f0f}" 2026-04-14T14:16:06.215Z [ERROR] server-e0a67d8a-3841-5334-1588-2d049039ad25: failed to get log: index=3 error="log not found" 2026-04-14T14:16:06.215Z [INFO] server-e0a67d8a-3841-5334-1588-2d049039ad25: aborting pipeline replication: peer="{Voter server-da096b5c-91e7-e24d-97ae-cf29fb8facb9 da096b5c-91e7-e24d-97ae-cf29fb8facb9}" 2026-04-14T14:16:06.222Z [ERROR] server-e0a67d8a-3841-5334-1588-2d049039ad25: failed to get log: index=3 error="log not found" 2026-04-14T14:16:06.222Z [INFO] creating new snapshot: path=/tmp/raft4255267005/snapshots/2-10013-1776176166222.tmp 2026-04-14T14:16:06.223Z [INFO] server-da096b5c-91e7-e24d-97ae-cf29fb8facb9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:16:06.223Z [INFO] server-da096b5c-91e7-e24d-97ae-cf29fb8facb9: copied to local snapshot: bytes=71 2026-04-14T14:16:06.223Z [INFO] server-da096b5c-91e7-e24d-97ae-cf29fb8facb9: snapshot restore progress: id=2-10013-1776176166222 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:16:06.223Z [INFO] server-da096b5c-91e7-e24d-97ae-cf29fb8facb9: Installed remote snapshot 2026-04-14T14:16:06.223Z [INFO] server-e0a67d8a-3841-5334-1588-2d049039ad25: pipelining replication: peer="{Voter server-da096b5c-91e7-e24d-97ae-cf29fb8facb9 da096b5c-91e7-e24d-97ae-cf29fb8facb9}" 2026-04-14T14:16:06.223Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:06.223Z [ERROR] server-e0a67d8a-3841-5334-1588-2d049039ad25: failed to get log: index=3 error="log not found" 2026-04-14T14:16:06.224Z [INFO] creating new snapshot: path=/tmp/raft1173848996/snapshots/2-10013-1776176166224.tmp 2026-04-14T14:16:06.224Z [INFO] server-25f3c358-3ffd-93e8-549b-d607185c0f0f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:16:06.224Z [INFO] server-25f3c358-3ffd-93e8-549b-d607185c0f0f: copied to local snapshot: bytes=71 2026-04-14T14:16:06.224Z [INFO] server-25f3c358-3ffd-93e8-549b-d607185c0f0f: snapshot restore progress: id=2-10013-1776176166224 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:16:06.224Z [INFO] server-25f3c358-3ffd-93e8-549b-d607185c0f0f: Installed remote snapshot 2026-04-14T14:16:06.224Z [INFO] server-e0a67d8a-3841-5334-1588-2d049039ad25: pipelining replication: peer="{Voter server-25f3c358-3ffd-93e8-549b-d607185c0f0f 25f3c358-3ffd-93e8-549b-d607185c0f0f}" 2026-04-14T14:16:07.229Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:06.223772323 +0000 UTC m=+97.695958092 (2 nodes), 552ns from start of poll, 1.592334326s from cluster start. Timeout at 2026-04-14 14:16:07.228801653 +0000 UTC m=+98.700987447, 1.005029355s after stability 2026-04-14T14:16:07.241Z [INFO] server-77073cbb-b09b-5203-3a05-0fd2b2c3fc6d: aborting pipeline replication: peer="{Voter server-caf2a85a-703d-5057-a930-57981335518f caf2a85a-703d-5057-a930-57981335518f}" 2026-04-14T14:16:07.241Z [INFO] server-77073cbb-b09b-5203-3a05-0fd2b2c3fc6d: aborting pipeline replication: peer="{Voter server-b42bf1b1-a3e3-dca2-9eec-78b767fcf41a b42bf1b1-a3e3-dca2-9eec-78b767fcf41a}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true 2026-04-14T14:16:07.242Z [INFO] server-8687d7ea-6789-f5c2-9acc-563edf3b039f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8687d7ea-6789-f5c2-9acc-563edf3b039f Address:8687d7ea-6789-f5c2-9acc-563edf3b039f} {Suffrage:Voter ID:server-ab0810d9-33db-86d1-e06f-fdd9b3810cb9 Address:ab0810d9-33db-86d1-e06f-fdd9b3810cb9} {Suffrage:Voter ID:server-79e26e89-d3f2-3c81-be17-b7b3e047fa99 Address:79e26e89-d3f2-3c81-be17-b7b3e047fa99}]" 2026-04-14T14:16:07.242Z [INFO] server-8687d7ea-6789-f5c2-9acc-563edf3b039f: entering follower state: follower="Node at 8687d7ea-6789-f5c2-9acc-563edf3b039f [Follower]" leader-address= leader-id= 2026-04-14T14:16:07.242Z [INFO] server-ab0810d9-33db-86d1-e06f-fdd9b3810cb9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8687d7ea-6789-f5c2-9acc-563edf3b039f Address:8687d7ea-6789-f5c2-9acc-563edf3b039f} {Suffrage:Voter ID:server-ab0810d9-33db-86d1-e06f-fdd9b3810cb9 Address:ab0810d9-33db-86d1-e06f-fdd9b3810cb9} {Suffrage:Voter ID:server-79e26e89-d3f2-3c81-be17-b7b3e047fa99 Address:79e26e89-d3f2-3c81-be17-b7b3e047fa99}]" 2026-04-14T14:16:07.242Z [INFO] server-79e26e89-d3f2-3c81-be17-b7b3e047fa99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8687d7ea-6789-f5c2-9acc-563edf3b039f Address:8687d7ea-6789-f5c2-9acc-563edf3b039f} {Suffrage:Voter ID:server-ab0810d9-33db-86d1-e06f-fdd9b3810cb9 Address:ab0810d9-33db-86d1-e06f-fdd9b3810cb9} {Suffrage:Voter ID:server-79e26e89-d3f2-3c81-be17-b7b3e047fa99 Address:79e26e89-d3f2-3c81-be17-b7b3e047fa99}]" 2026-04-14T14:16:07.242Z [INFO] server-ab0810d9-33db-86d1-e06f-fdd9b3810cb9: entering follower state: follower="Node at ab0810d9-33db-86d1-e06f-fdd9b3810cb9 [Follower]" leader-address= leader-id= 2026-04-14T14:16:07.242Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:07.242Z [INFO] server-79e26e89-d3f2-3c81-be17-b7b3e047fa99: entering follower state: follower="Node at 79e26e89-d3f2-3c81-be17-b7b3e047fa99 [Follower]" leader-address= leader-id= 2026-04-14T14:16:08.050Z [WARN] server-8687d7ea-6789-f5c2-9acc-563edf3b039f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:08.050Z [INFO] server-8687d7ea-6789-f5c2-9acc-563edf3b039f: entering candidate state: node="Node at 8687d7ea-6789-f5c2-9acc-563edf3b039f [Candidate]" term=2 2026-04-14T14:16:08.051Z [INFO] server-8687d7ea-6789-f5c2-9acc-563edf3b039f: election won: term=2 tally=2 2026-04-14T14:16:08.051Z [INFO] server-8687d7ea-6789-f5c2-9acc-563edf3b039f: entering leader state: leader="Node at 8687d7ea-6789-f5c2-9acc-563edf3b039f [Leader]" 2026-04-14T14:16:08.051Z [INFO] server-8687d7ea-6789-f5c2-9acc-563edf3b039f: added peer, starting replication: peer=server-ab0810d9-33db-86d1-e06f-fdd9b3810cb9 2026-04-14T14:16:08.051Z [INFO] server-8687d7ea-6789-f5c2-9acc-563edf3b039f: added peer, starting replication: peer=server-79e26e89-d3f2-3c81-be17-b7b3e047fa99 2026-04-14T14:16:08.051Z [INFO] server-8687d7ea-6789-f5c2-9acc-563edf3b039f: pipelining replication: peer="{Voter server-ab0810d9-33db-86d1-e06f-fdd9b3810cb9 ab0810d9-33db-86d1-e06f-fdd9b3810cb9}" 2026-04-14T14:16:08.051Z [INFO] server-8687d7ea-6789-f5c2-9acc-563edf3b039f: pipelining replication: peer="{Voter server-79e26e89-d3f2-3c81-be17-b7b3e047fa99 79e26e89-d3f2-3c81-be17-b7b3e047fa99}" 2026-04-14T14:16:09.056Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:08.051060237 +0000 UTC m=+99.523246035 (1 nodes), 808.321096ms from start of poll, 808.614569ms from cluster start. Timeout at 2026-04-14 14:16:09.056064027 +0000 UTC m=+100.528249794, 1.005003759s after stability 2026-04-14T14:16:09.058Z [INFO] server-8687d7ea-6789-f5c2-9acc-563edf3b039f: starting snapshot up to: index=12 2026-04-14T14:16:09.058Z [INFO] creating new snapshot: path=/tmp/raft2424837263/snapshots/2-12-1776176169058.tmp 2026-04-14T14:16:09.059Z [INFO] server-8687d7ea-6789-f5c2-9acc-563edf3b039f: snapshot complete up to: index=12 2026-04-14T14:16:09.060Z [INFO] server-9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e Address:9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e} {Suffrage:Voter ID:server-02f20f67-4229-0c38-2178-51103e972274 Address:02f20f67-4229-0c38-2178-51103e972274} {Suffrage:Voter ID:server-359d70b4-a48a-7164-b4ec-1c6fb45604ef Address:359d70b4-a48a-7164-b4ec-1c6fb45604ef}]" 2026-04-14T14:16:09.060Z [INFO] server-9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e: entering follower state: follower="Node at 9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e [Follower]" leader-address= leader-id= 2026-04-14T14:16:09.060Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e Address:9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e} {Suffrage:Voter ID:server-02f20f67-4229-0c38-2178-51103e972274 Address:02f20f67-4229-0c38-2178-51103e972274} {Suffrage:Voter ID:server-359d70b4-a48a-7164-b4ec-1c6fb45604ef Address:359d70b4-a48a-7164-b4ec-1c6fb45604ef}]" 2026-04-14T14:16:09.061Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: entering follower state: follower="Node at 02f20f67-4229-0c38-2178-51103e972274 [Follower]" leader-address= leader-id= 2026-04-14T14:16:09.061Z [INFO] server-359d70b4-a48a-7164-b4ec-1c6fb45604ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e Address:9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e} {Suffrage:Voter ID:server-02f20f67-4229-0c38-2178-51103e972274 Address:02f20f67-4229-0c38-2178-51103e972274} {Suffrage:Voter ID:server-359d70b4-a48a-7164-b4ec-1c6fb45604ef Address:359d70b4-a48a-7164-b4ec-1c6fb45604ef}]" 2026-04-14T14:16:09.061Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:09.061Z [INFO] server-359d70b4-a48a-7164-b4ec-1c6fb45604ef: entering follower state: follower="Node at 359d70b4-a48a-7164-b4ec-1c6fb45604ef [Follower]" leader-address= leader-id= 2026-04-14T14:16:09.616Z [WARN] server-02f20f67-4229-0c38-2178-51103e972274: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:09.616Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: entering candidate state: node="Node at 02f20f67-4229-0c38-2178-51103e972274 [Candidate]" term=2 2026-04-14T14:16:09.616Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: election won: term=2 tally=2 2026-04-14T14:16:09.616Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: entering leader state: leader="Node at 02f20f67-4229-0c38-2178-51103e972274 [Leader]" 2026-04-14T14:16:09.616Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: added peer, starting replication: peer=server-9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e 2026-04-14T14:16:09.616Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: added peer, starting replication: peer=server-359d70b4-a48a-7164-b4ec-1c6fb45604ef 2026-04-14T14:16:09.616Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: pipelining replication: peer="{Voter server-359d70b4-a48a-7164-b4ec-1c6fb45604ef 359d70b4-a48a-7164-b4ec-1c6fb45604ef}" 2026-04-14T14:16:09.616Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: pipelining replication: peer="{Voter server-9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e 9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e}" 2026-04-14T14:16:10.622Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:09.616316222 +0000 UTC m=+101.088502001 (1 nodes), 555.182258ms from start of poll, 555.737982ms from cluster start. Timeout at 2026-04-14 14:16:10.621321675 +0000 UTC m=+102.093507439, 1.005005438s after stability 2026-04-14T14:16:10.622Z [INFO] creating new snapshot: path=/tmp/raft2129431547/snapshots/2-10013-1776176170622.tmp 2026-04-14T14:16:10.622Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: copied to local snapshot: bytes=71 2026-04-14T14:16:10.623Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: snapshot restore progress: id=2-10013-1776176170622 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:16:10.623Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: removing all old logs from log store 2026-04-14T14:16:10.623Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: compacting logs: from=1 to=2 2026-04-14T14:16:10.623Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: restored user snapshot: index=10013 2026-04-14T14:16:10.623Z [ERROR] server-02f20f67-4229-0c38-2178-51103e972274: failed to get log: index=2 error="log not found" 2026-04-14T14:16:10.623Z [ERROR] server-02f20f67-4229-0c38-2178-51103e972274: failed to get log: index=2 error="log not found" 2026-04-14T14:16:10.623Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: aborting pipeline replication: peer="{Voter server-9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e 9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e}" 2026-04-14T14:16:10.623Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: aborting pipeline replication: peer="{Voter server-359d70b4-a48a-7164-b4ec-1c6fb45604ef 359d70b4-a48a-7164-b4ec-1c6fb45604ef}" 2026-04-14T14:16:10.631Z [ERROR] server-02f20f67-4229-0c38-2178-51103e972274: failed to get log: index=2 error="log not found" 2026-04-14T14:16:10.631Z [INFO] creating new snapshot: path=/tmp/raft1121806740/snapshots/2-10013-1776176170631.tmp 2026-04-14T14:16:10.631Z [ERROR] server-02f20f67-4229-0c38-2178-51103e972274: failed to get log: index=2 error="log not found" 2026-04-14T14:16:10.632Z [INFO] creating new snapshot: path=/tmp/raft1731422624/snapshots/2-10013-1776176170632.tmp 2026-04-14T14:16:10.632Z [INFO] server-9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:16:10.632Z [INFO] server-359d70b4-a48a-7164-b4ec-1c6fb45604ef: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-14T14:16:10.632Z [INFO] server-9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e: copied to local snapshot: bytes=71 2026-04-14T14:16:10.632Z [INFO] server-9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e: snapshot restore progress: id=2-10013-1776176170631 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:16:10.632Z [INFO] server-9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e: removing all old logs from log store 2026-04-14T14:16:10.632Z [INFO] server-9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e: compacting logs: from=1 to=2 2026-04-14T14:16:10.632Z [INFO] server-9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e: Installed remote snapshot 2026-04-14T14:16:10.632Z [INFO] server-359d70b4-a48a-7164-b4ec-1c6fb45604ef: copied to local snapshot: bytes=71 2026-04-14T14:16:10.632Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: pipelining replication: peer="{Voter server-9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e 9781cdd4-16fc-bfb7-4a40-f9b3a6165e0e}" 2026-04-14T14:16:10.632Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:10.633Z [INFO] server-359d70b4-a48a-7164-b4ec-1c6fb45604ef: snapshot restore progress: id=2-10013-1776176170632 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-14T14:16:10.633Z [INFO] server-359d70b4-a48a-7164-b4ec-1c6fb45604ef: removing all old logs from log store 2026-04-14T14:16:10.633Z [INFO] server-359d70b4-a48a-7164-b4ec-1c6fb45604ef: compacting logs: from=1 to=2 2026-04-14T14:16:10.633Z [INFO] server-359d70b4-a48a-7164-b4ec-1c6fb45604ef: Installed remote snapshot 2026-04-14T14:16:10.633Z [INFO] server-02f20f67-4229-0c38-2178-51103e972274: pipelining replication: peer="{Voter server-359d70b4-a48a-7164-b4ec-1c6fb45604ef 359d70b4-a48a-7164-b4ec-1c6fb45604ef}" 2026-04-14T14:16:11.638Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:10.633011231 +0000 UTC m=+102.105197013 (2 nodes), 963ns from start of poll, 1.572432994s from cluster start. Timeout at 2026-04-14 14:16:11.638112478 +0000 UTC m=+103.110298274, 1.005101261s after stability 2026-04-14T14:16:11.649Z [INFO] server-8687d7ea-6789-f5c2-9acc-563edf3b039f: aborting pipeline replication: peer="{Voter server-79e26e89-d3f2-3c81-be17-b7b3e047fa99 79e26e89-d3f2-3c81-be17-b7b3e047fa99}" 2026-04-14T14:16:11.649Z [INFO] server-8687d7ea-6789-f5c2-9acc-563edf3b039f: aborting pipeline replication: peer="{Voter server-ab0810d9-33db-86d1-e06f-fdd9b3810cb9 ab0810d9-33db-86d1-e06f-fdd9b3810cb9}" --- FAIL: TestRaft_UserRestore (87.97s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.66s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.70s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.86s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.10s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.61s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.68s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.67s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.15s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.88s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.99s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.68s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.38s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.99s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.90s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.68s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.80s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.85s) --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.84s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.88s) --- FAIL: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (3.91s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (3.13s) --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (3.06s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.17s) --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.41s) === RUN TestRaft_SendSnapshotFollower 2026-04-14T14:16:11.650Z [INFO] server-dc38c547-09e9-6151-04ce-5baa76f8a144: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc38c547-09e9-6151-04ce-5baa76f8a144 Address:dc38c547-09e9-6151-04ce-5baa76f8a144} {Suffrage:Voter ID:server-f74fdfe4-7d08-a88f-32c7-c4033b2f5f0c Address:f74fdfe4-7d08-a88f-32c7-c4033b2f5f0c} {Suffrage:Voter ID:server-122628a1-1d32-978e-da37-c6a3998ddb8b Address:122628a1-1d32-978e-da37-c6a3998ddb8b}]" 2026-04-14T14:16:11.650Z [INFO] server-dc38c547-09e9-6151-04ce-5baa76f8a144: entering follower state: follower="Node at dc38c547-09e9-6151-04ce-5baa76f8a144 [Follower]" leader-address= leader-id= 2026-04-14T14:16:11.650Z [INFO] server-f74fdfe4-7d08-a88f-32c7-c4033b2f5f0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc38c547-09e9-6151-04ce-5baa76f8a144 Address:dc38c547-09e9-6151-04ce-5baa76f8a144} {Suffrage:Voter ID:server-f74fdfe4-7d08-a88f-32c7-c4033b2f5f0c Address:f74fdfe4-7d08-a88f-32c7-c4033b2f5f0c} {Suffrage:Voter ID:server-122628a1-1d32-978e-da37-c6a3998ddb8b Address:122628a1-1d32-978e-da37-c6a3998ddb8b}]" 2026-04-14T14:16:11.650Z [INFO] server-f74fdfe4-7d08-a88f-32c7-c4033b2f5f0c: entering follower state: follower="Node at f74fdfe4-7d08-a88f-32c7-c4033b2f5f0c [Follower]" leader-address= leader-id= 2026-04-14T14:16:11.650Z [INFO] server-122628a1-1d32-978e-da37-c6a3998ddb8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc38c547-09e9-6151-04ce-5baa76f8a144 Address:dc38c547-09e9-6151-04ce-5baa76f8a144} {Suffrage:Voter ID:server-f74fdfe4-7d08-a88f-32c7-c4033b2f5f0c Address:f74fdfe4-7d08-a88f-32c7-c4033b2f5f0c} {Suffrage:Voter ID:server-122628a1-1d32-978e-da37-c6a3998ddb8b Address:122628a1-1d32-978e-da37-c6a3998ddb8b}]" 2026-04-14T14:16:11.651Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:11.651Z [INFO] server-122628a1-1d32-978e-da37-c6a3998ddb8b: entering follower state: follower="Node at 122628a1-1d32-978e-da37-c6a3998ddb8b [Follower]" leader-address= leader-id= 2026-04-14T14:16:11.702Z [WARN] server-122628a1-1d32-978e-da37-c6a3998ddb8b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:11.703Z [INFO] server-122628a1-1d32-978e-da37-c6a3998ddb8b: entering candidate state: node="Node at 122628a1-1d32-978e-da37-c6a3998ddb8b [Candidate]" term=2 2026-04-14T14:16:11.703Z [INFO] server-122628a1-1d32-978e-da37-c6a3998ddb8b: election won: term=2 tally=2 2026-04-14T14:16:11.703Z [INFO] server-122628a1-1d32-978e-da37-c6a3998ddb8b: entering leader state: leader="Node at 122628a1-1d32-978e-da37-c6a3998ddb8b [Leader]" 2026-04-14T14:16:11.703Z [INFO] server-122628a1-1d32-978e-da37-c6a3998ddb8b: added peer, starting replication: peer=server-dc38c547-09e9-6151-04ce-5baa76f8a144 2026-04-14T14:16:11.703Z [INFO] server-122628a1-1d32-978e-da37-c6a3998ddb8b: added peer, starting replication: peer=server-f74fdfe4-7d08-a88f-32c7-c4033b2f5f0c 2026-04-14T14:16:11.703Z [INFO] server-122628a1-1d32-978e-da37-c6a3998ddb8b: pipelining replication: peer="{Voter server-f74fdfe4-7d08-a88f-32c7-c4033b2f5f0c f74fdfe4-7d08-a88f-32c7-c4033b2f5f0c}" 2026-04-14T14:16:11.703Z [INFO] server-122628a1-1d32-978e-da37-c6a3998ddb8b: pipelining replication: peer="{Voter server-dc38c547-09e9-6151-04ce-5baa76f8a144 dc38c547-09e9-6151-04ce-5baa76f8a144}" 2026-04-14T14:16:11.808Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:11.703161644 +0000 UTC m=+103.175347427 (2 nodes), 51.711419ms from start of poll, 52.467836ms from cluster start. Timeout at 2026-04-14 14:16:11.808174571 +0000 UTC m=+103.280360344, 105.012917ms after stability 2026-04-14T14:16:11.808Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:11.915Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:11.808979831 +0000 UTC m=+103.281165600 (1 nodes), 833ns from start of poll, 158.286009ms from cluster start. Timeout at 2026-04-14 14:16:11.913987688 +0000 UTC m=+103.386173479, 105.007879ms after stability 2026-04-14T14:16:11.915Z [ERROR] server-122628a1-1d32-978e-da37-c6a3998ddb8b: failed to pipeline appendEntries: peer="{Voter server-dc38c547-09e9-6151-04ce-5baa76f8a144 dc38c547-09e9-6151-04ce-5baa76f8a144}" error="append pipeline closed" 2026-04-14T14:16:11.915Z [INFO] server-122628a1-1d32-978e-da37-c6a3998ddb8b: aborting pipeline replication: peer="{Voter server-dc38c547-09e9-6151-04ce-5baa76f8a144 dc38c547-09e9-6151-04ce-5baa76f8a144}" 2026-04-14T14:16:11.915Z [ERROR] server-122628a1-1d32-978e-da37-c6a3998ddb8b: failed to appendEntries to: peer="{Voter server-dc38c547-09e9-6151-04ce-5baa76f8a144 dc38c547-09e9-6151-04ce-5baa76f8a144}" error="failed to connect to peer: dc38c547-09e9-6151-04ce-5baa76f8a144" raft_test.go:1627: [INFO] Finished apply without behind follower 2026-04-14T14:16:11.917Z [ERROR] server-dc38c547-09e9-6151-04ce-5baa76f8a144: failed to take snapshot: error="nothing new to snapshot" 2026-04-14T14:16:11.917Z [INFO] server-f74fdfe4-7d08-a88f-32c7-c4033b2f5f0c: starting snapshot up to: index=95 2026-04-14T14:16:11.917Z [INFO] creating new snapshot: path=/tmp/raft380058075/snapshots/2-95-1776176171917.tmp 2026-04-14T14:16:11.917Z [INFO] server-f74fdfe4-7d08-a88f-32c7-c4033b2f5f0c: compacting logs: from=1 to=92 2026-04-14T14:16:11.917Z [INFO] server-f74fdfe4-7d08-a88f-32c7-c4033b2f5f0c: snapshot complete up to: index=95 2026-04-14T14:16:11.917Z [INFO] server-122628a1-1d32-978e-da37-c6a3998ddb8b: starting snapshot up to: index=102 2026-04-14T14:16:11.917Z [INFO] creating new snapshot: path=/tmp/raft2217704372/snapshots/2-102-1776176171917.tmp 2026-04-14T14:16:11.918Z [INFO] server-122628a1-1d32-978e-da37-c6a3998ddb8b: compacting logs: from=1 to=92 2026-04-14T14:16:11.918Z [INFO] server-122628a1-1d32-978e-da37-c6a3998ddb8b: snapshot complete up to: index=102 2026-04-14T14:16:11.925Z [ERROR] server-122628a1-1d32-978e-da37-c6a3998ddb8b: failed to get log: index=2 error="log not found" 2026-04-14T14:16:11.926Z [INFO] creating new snapshot: path=/tmp/raft2613558276/snapshots/2-102-1776176171926.tmp 2026-04-14T14:16:11.926Z [INFO] server-dc38c547-09e9-6151-04ce-5baa76f8a144: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2026-04-14T14:16:11.926Z [INFO] server-dc38c547-09e9-6151-04ce-5baa76f8a144: copied to local snapshot: bytes=693 2026-04-14T14:16:11.927Z [INFO] server-dc38c547-09e9-6151-04ce-5baa76f8a144: snapshot restore progress: id=2-102-1776176171926 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2026-04-14T14:16:11.927Z [INFO] server-dc38c547-09e9-6151-04ce-5baa76f8a144: Installed remote snapshot 2026-04-14T14:16:11.927Z [WARN] server-dc38c547-09e9-6151-04ce-5baa76f8a144: skipping application of old log: index=102 2026-04-14T14:16:11.927Z [INFO] server-122628a1-1d32-978e-da37-c6a3998ddb8b: pipelining replication: peer="{Voter server-dc38c547-09e9-6151-04ce-5baa76f8a144 dc38c547-09e9-6151-04ce-5baa76f8a144}" 2026-04-14T14:16:11.929Z [INFO] server-122628a1-1d32-978e-da37-c6a3998ddb8b: aborting pipeline replication: peer="{Voter server-f74fdfe4-7d08-a88f-32c7-c4033b2f5f0c f74fdfe4-7d08-a88f-32c7-c4033b2f5f0c}" 2026-04-14T14:16:11.929Z [INFO] server-122628a1-1d32-978e-da37-c6a3998ddb8b: aborting pipeline replication: peer="{Voter server-dc38c547-09e9-6151-04ce-5baa76f8a144 dc38c547-09e9-6151-04ce-5baa76f8a144}" --- PASS: TestRaft_SendSnapshotFollower (0.28s) === RUN TestRaft_SendSnapshotAndLogsFollower 2026-04-14T14:16:11.930Z [INFO] server-24133d62-dfb9-8b57-8380-1d53141774c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24133d62-dfb9-8b57-8380-1d53141774c5 Address:24133d62-dfb9-8b57-8380-1d53141774c5} {Suffrage:Voter ID:server-55990392-dbf1-b109-1a9c-0b7cbd5a723b Address:55990392-dbf1-b109-1a9c-0b7cbd5a723b} {Suffrage:Voter ID:server-9e463ab5-b6e5-bdf6-bee4-6922e79d627c Address:9e463ab5-b6e5-bdf6-bee4-6922e79d627c}]" 2026-04-14T14:16:11.930Z [INFO] server-24133d62-dfb9-8b57-8380-1d53141774c5: entering follower state: follower="Node at 24133d62-dfb9-8b57-8380-1d53141774c5 [Follower]" leader-address= leader-id= 2026-04-14T14:16:11.930Z [INFO] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24133d62-dfb9-8b57-8380-1d53141774c5 Address:24133d62-dfb9-8b57-8380-1d53141774c5} {Suffrage:Voter ID:server-55990392-dbf1-b109-1a9c-0b7cbd5a723b Address:55990392-dbf1-b109-1a9c-0b7cbd5a723b} {Suffrage:Voter ID:server-9e463ab5-b6e5-bdf6-bee4-6922e79d627c Address:9e463ab5-b6e5-bdf6-bee4-6922e79d627c}]" 2026-04-14T14:16:11.930Z [INFO] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: entering follower state: follower="Node at 55990392-dbf1-b109-1a9c-0b7cbd5a723b [Follower]" leader-address= leader-id= 2026-04-14T14:16:11.930Z [INFO] server-9e463ab5-b6e5-bdf6-bee4-6922e79d627c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24133d62-dfb9-8b57-8380-1d53141774c5 Address:24133d62-dfb9-8b57-8380-1d53141774c5} {Suffrage:Voter ID:server-55990392-dbf1-b109-1a9c-0b7cbd5a723b Address:55990392-dbf1-b109-1a9c-0b7cbd5a723b} {Suffrage:Voter ID:server-9e463ab5-b6e5-bdf6-bee4-6922e79d627c Address:9e463ab5-b6e5-bdf6-bee4-6922e79d627c}]" 2026-04-14T14:16:11.930Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:11.930Z [INFO] server-9e463ab5-b6e5-bdf6-bee4-6922e79d627c: entering follower state: follower="Node at 9e463ab5-b6e5-bdf6-bee4-6922e79d627c [Follower]" leader-address= leader-id= 2026-04-14T14:16:11.984Z [WARN] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:11.984Z [INFO] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: entering candidate state: node="Node at 55990392-dbf1-b109-1a9c-0b7cbd5a723b [Candidate]" term=2 2026-04-14T14:16:11.984Z [INFO] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: election won: term=2 tally=2 2026-04-14T14:16:11.984Z [INFO] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: entering leader state: leader="Node at 55990392-dbf1-b109-1a9c-0b7cbd5a723b [Leader]" 2026-04-14T14:16:11.984Z [INFO] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: added peer, starting replication: peer=server-24133d62-dfb9-8b57-8380-1d53141774c5 2026-04-14T14:16:11.984Z [INFO] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: added peer, starting replication: peer=server-9e463ab5-b6e5-bdf6-bee4-6922e79d627c 2026-04-14T14:16:11.984Z [INFO] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: pipelining replication: peer="{Voter server-9e463ab5-b6e5-bdf6-bee4-6922e79d627c 9e463ab5-b6e5-bdf6-bee4-6922e79d627c}" 2026-04-14T14:16:11.984Z [INFO] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: pipelining replication: peer="{Voter server-24133d62-dfb9-8b57-8380-1d53141774c5 24133d62-dfb9-8b57-8380-1d53141774c5}" 2026-04-14T14:16:12.090Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:11.984666309 +0000 UTC m=+103.456852091 (2 nodes), 53.838138ms from start of poll, 54.144361ms from cluster start. Timeout at 2026-04-14 14:16:12.089676785 +0000 UTC m=+103.561862568, 105.010477ms after stability 2026-04-14T14:16:12.090Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:12.196Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:12.090282437 +0000 UTC m=+103.562468205 (1 nodes), 1.971µs from start of poll, 159.760475ms from cluster start. Timeout at 2026-04-14 14:16:12.195312059 +0000 UTC m=+103.667497837, 105.029632ms after stability 2026-04-14T14:16:12.196Z [ERROR] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: failed to pipeline appendEntries: peer="{Voter server-24133d62-dfb9-8b57-8380-1d53141774c5 24133d62-dfb9-8b57-8380-1d53141774c5}" error="append pipeline closed" 2026-04-14T14:16:12.196Z [INFO] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: aborting pipeline replication: peer="{Voter server-24133d62-dfb9-8b57-8380-1d53141774c5 24133d62-dfb9-8b57-8380-1d53141774c5}" 2026-04-14T14:16:12.196Z [ERROR] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: failed to appendEntries to: peer="{Voter server-24133d62-dfb9-8b57-8380-1d53141774c5 24133d62-dfb9-8b57-8380-1d53141774c5}" error="failed to connect to peer: 24133d62-dfb9-8b57-8380-1d53141774c5" raft_test.go:1669: [INFO] Finished apply without behind follower 2026-04-14T14:16:12.200Z [ERROR] server-24133d62-dfb9-8b57-8380-1d53141774c5: failed to take snapshot: error="nothing new to snapshot" 2026-04-14T14:16:12.200Z [INFO] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: starting snapshot up to: index=102 2026-04-14T14:16:12.200Z [INFO] creating new snapshot: path=/tmp/raft1048209324/snapshots/2-102-1776176172200.tmp 2026-04-14T14:16:12.201Z [INFO] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: compacting logs: from=1 to=92 2026-04-14T14:16:12.201Z [INFO] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: snapshot complete up to: index=102 2026-04-14T14:16:12.201Z [INFO] server-9e463ab5-b6e5-bdf6-bee4-6922e79d627c: starting snapshot up to: index=97 2026-04-14T14:16:12.201Z [INFO] creating new snapshot: path=/tmp/raft2991854135/snapshots/2-97-1776176172201.tmp 2026-04-14T14:16:12.201Z [INFO] server-9e463ab5-b6e5-bdf6-bee4-6922e79d627c: compacting logs: from=1 to=92 2026-04-14T14:16:12.201Z [INFO] server-9e463ab5-b6e5-bdf6-bee4-6922e79d627c: snapshot complete up to: index=97 2026-04-14T14:16:12.202Z [ERROR] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: failed to heartbeat to: peer=24133d62-dfb9-8b57-8380-1d53141774c5 backoff time=10ms error="failed to connect to peer: 24133d62-dfb9-8b57-8380-1d53141774c5" raft_test.go:1690: [INFO] Finished apply without behind follower 2026-04-14T14:16:12.207Z [ERROR] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: failed to get log: index=2 error="log not found" 2026-04-14T14:16:12.207Z [INFO] creating new snapshot: path=/tmp/raft203914989/snapshots/2-102-1776176172207.tmp 2026-04-14T14:16:12.207Z [INFO] server-24133d62-dfb9-8b57-8380-1d53141774c5: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2026-04-14T14:16:12.207Z [INFO] server-24133d62-dfb9-8b57-8380-1d53141774c5: copied to local snapshot: bytes=693 2026-04-14T14:16:12.207Z [INFO] server-24133d62-dfb9-8b57-8380-1d53141774c5: snapshot restore progress: id=2-102-1776176172207 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2026-04-14T14:16:12.207Z [INFO] server-24133d62-dfb9-8b57-8380-1d53141774c5: Installed remote snapshot 2026-04-14T14:16:12.208Z [INFO] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: pipelining replication: peer="{Voter server-24133d62-dfb9-8b57-8380-1d53141774c5 24133d62-dfb9-8b57-8380-1d53141774c5}" 2026-04-14T14:16:12.215Z [INFO] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: aborting pipeline replication: peer="{Voter server-9e463ab5-b6e5-bdf6-bee4-6922e79d627c 9e463ab5-b6e5-bdf6-bee4-6922e79d627c}" 2026-04-14T14:16:12.215Z [INFO] server-55990392-dbf1-b109-1a9c-0b7cbd5a723b: aborting pipeline replication: peer="{Voter server-24133d62-dfb9-8b57-8380-1d53141774c5 24133d62-dfb9-8b57-8380-1d53141774c5}" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.29s) === RUN TestRaft_ReJoinFollower 2026-04-14T14:16:12.217Z [INFO] server-2e107498-086b-a8b3-5ab7-4971b8cd31e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e107498-086b-a8b3-5ab7-4971b8cd31e1 Address:2e107498-086b-a8b3-5ab7-4971b8cd31e1} {Suffrage:Voter ID:server-afe06084-4ef7-77a5-8d65-376a515bab66 Address:afe06084-4ef7-77a5-8d65-376a515bab66} {Suffrage:Voter ID:server-2b2afab4-4f0a-421f-b915-5f6e40bf3987 Address:2b2afab4-4f0a-421f-b915-5f6e40bf3987}]" 2026-04-14T14:16:12.217Z [INFO] server-afe06084-4ef7-77a5-8d65-376a515bab66: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e107498-086b-a8b3-5ab7-4971b8cd31e1 Address:2e107498-086b-a8b3-5ab7-4971b8cd31e1} {Suffrage:Voter ID:server-afe06084-4ef7-77a5-8d65-376a515bab66 Address:afe06084-4ef7-77a5-8d65-376a515bab66} {Suffrage:Voter ID:server-2b2afab4-4f0a-421f-b915-5f6e40bf3987 Address:2b2afab4-4f0a-421f-b915-5f6e40bf3987}]" 2026-04-14T14:16:12.217Z [INFO] server-2e107498-086b-a8b3-5ab7-4971b8cd31e1: entering follower state: follower="Node at 2e107498-086b-a8b3-5ab7-4971b8cd31e1 [Follower]" leader-address= leader-id= 2026-04-14T14:16:12.217Z [INFO] server-2b2afab4-4f0a-421f-b915-5f6e40bf3987: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e107498-086b-a8b3-5ab7-4971b8cd31e1 Address:2e107498-086b-a8b3-5ab7-4971b8cd31e1} {Suffrage:Voter ID:server-afe06084-4ef7-77a5-8d65-376a515bab66 Address:afe06084-4ef7-77a5-8d65-376a515bab66} {Suffrage:Voter ID:server-2b2afab4-4f0a-421f-b915-5f6e40bf3987 Address:2b2afab4-4f0a-421f-b915-5f6e40bf3987}]" 2026-04-14T14:16:12.217Z [INFO] server-afe06084-4ef7-77a5-8d65-376a515bab66: entering follower state: follower="Node at afe06084-4ef7-77a5-8d65-376a515bab66 [Follower]" leader-address= leader-id= 2026-04-14T14:16:12.217Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:12.217Z [INFO] server-2b2afab4-4f0a-421f-b915-5f6e40bf3987: entering follower state: follower="Node at 2b2afab4-4f0a-421f-b915-5f6e40bf3987 [Follower]" leader-address= leader-id= 2026-04-14T14:16:12.290Z [WARN] server-afe06084-4ef7-77a5-8d65-376a515bab66: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:12.290Z [INFO] server-afe06084-4ef7-77a5-8d65-376a515bab66: entering candidate state: node="Node at afe06084-4ef7-77a5-8d65-376a515bab66 [Candidate]" term=2 2026-04-14T14:16:12.290Z [INFO] server-afe06084-4ef7-77a5-8d65-376a515bab66: election won: term=2 tally=2 2026-04-14T14:16:12.290Z [INFO] server-afe06084-4ef7-77a5-8d65-376a515bab66: entering leader state: leader="Node at afe06084-4ef7-77a5-8d65-376a515bab66 [Leader]" 2026-04-14T14:16:12.290Z [INFO] server-afe06084-4ef7-77a5-8d65-376a515bab66: added peer, starting replication: peer=server-2e107498-086b-a8b3-5ab7-4971b8cd31e1 2026-04-14T14:16:12.290Z [INFO] server-afe06084-4ef7-77a5-8d65-376a515bab66: added peer, starting replication: peer=server-2b2afab4-4f0a-421f-b915-5f6e40bf3987 2026-04-14T14:16:12.290Z [INFO] server-afe06084-4ef7-77a5-8d65-376a515bab66: pipelining replication: peer="{Voter server-2b2afab4-4f0a-421f-b915-5f6e40bf3987 2b2afab4-4f0a-421f-b915-5f6e40bf3987}" 2026-04-14T14:16:12.290Z [INFO] server-afe06084-4ef7-77a5-8d65-376a515bab66: pipelining replication: peer="{Voter server-2e107498-086b-a8b3-5ab7-4971b8cd31e1 2e107498-086b-a8b3-5ab7-4971b8cd31e1}" 2026-04-14T14:16:12.396Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:12.290505094 +0000 UTC m=+103.762690873 (1 nodes), 72.831997ms from start of poll, 73.100168ms from cluster start. Timeout at 2026-04-14 14:16:12.395514319 +0000 UTC m=+103.867700095, 105.009222ms after stability 2026-04-14T14:16:12.401Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:12.507Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:12.40179002 +0000 UTC m=+103.873975798 (2 nodes), 2.961µs from start of poll, 184.385093ms from cluster start. Timeout at 2026-04-14 14:16:12.506817184 +0000 UTC m=+103.979002953, 105.027155ms after stability 2026-04-14T14:16:12.507Z [INFO] server-afe06084-4ef7-77a5-8d65-376a515bab66: updating configuration: command=RemoveServer server-id=server-2e107498-086b-a8b3-5ab7-4971b8cd31e1 server-addr= servers="[{Suffrage:Voter ID:server-afe06084-4ef7-77a5-8d65-376a515bab66 Address:afe06084-4ef7-77a5-8d65-376a515bab66} {Suffrage:Voter ID:server-2b2afab4-4f0a-421f-b915-5f6e40bf3987 Address:2b2afab4-4f0a-421f-b915-5f6e40bf3987}]" 2026-04-14T14:16:12.507Z [INFO] server-afe06084-4ef7-77a5-8d65-376a515bab66: removed peer, stopping replication: peer=server-2e107498-086b-a8b3-5ab7-4971b8cd31e1 last-index=3 2026-04-14T14:16:12.507Z [INFO] server-afe06084-4ef7-77a5-8d65-376a515bab66: aborting pipeline replication: peer="{Voter server-2e107498-086b-a8b3-5ab7-4971b8cd31e1 2e107498-086b-a8b3-5ab7-4971b8cd31e1}" 2026-04-14T14:16:12.586Z [WARN] server-2e107498-086b-a8b3-5ab7-4971b8cd31e1: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2026-04-14T14:16:12.613Z [INFO] server-afe06084-4ef7-77a5-8d65-376a515bab66: updating configuration: command=AddVoter server-id=server-2e107498-086b-a8b3-5ab7-4971b8cd31e1 server-addr=2e107498-086b-a8b3-5ab7-4971b8cd31e1 servers="[{Suffrage:Voter ID:server-afe06084-4ef7-77a5-8d65-376a515bab66 Address:afe06084-4ef7-77a5-8d65-376a515bab66} {Suffrage:Voter ID:server-2b2afab4-4f0a-421f-b915-5f6e40bf3987 Address:2b2afab4-4f0a-421f-b915-5f6e40bf3987} {Suffrage:Voter ID:server-2e107498-086b-a8b3-5ab7-4971b8cd31e1 Address:2e107498-086b-a8b3-5ab7-4971b8cd31e1}]" 2026-04-14T14:16:12.613Z [INFO] server-afe06084-4ef7-77a5-8d65-376a515bab66: added peer, starting replication: peer=server-2e107498-086b-a8b3-5ab7-4971b8cd31e1 2026-04-14T14:16:12.613Z [WARN] server-2e107498-086b-a8b3-5ab7-4971b8cd31e1: failed to get previous log: previous-index=4 last-index=3 error="log not found" 2026-04-14T14:16:12.614Z [WARN] server-afe06084-4ef7-77a5-8d65-376a515bab66: appendEntries rejected, sending older logs: peer="{Voter server-2e107498-086b-a8b3-5ab7-4971b8cd31e1 2e107498-086b-a8b3-5ab7-4971b8cd31e1}" next=4 2026-04-14T14:16:12.614Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:12.614Z [INFO] server-afe06084-4ef7-77a5-8d65-376a515bab66: pipelining replication: peer="{Voter server-2e107498-086b-a8b3-5ab7-4971b8cd31e1 2e107498-086b-a8b3-5ab7-4971b8cd31e1}" 2026-04-14T14:16:12.719Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:12.614071684 +0000 UTC m=+104.086257464 (1 nodes), 1.6µs from start of poll, 396.666759ms from cluster start. Timeout at 2026-04-14 14:16:12.719108197 +0000 UTC m=+104.191293962, 105.036498ms after stability 2026-04-14T14:16:12.719Z [INFO] server-afe06084-4ef7-77a5-8d65-376a515bab66: aborting pipeline replication: peer="{Voter server-2b2afab4-4f0a-421f-b915-5f6e40bf3987 2b2afab4-4f0a-421f-b915-5f6e40bf3987}" 2026-04-14T14:16:12.719Z [INFO] server-afe06084-4ef7-77a5-8d65-376a515bab66: aborting pipeline replication: peer="{Voter server-2e107498-086b-a8b3-5ab7-4971b8cd31e1 2e107498-086b-a8b3-5ab7-4971b8cd31e1}" --- PASS: TestRaft_ReJoinFollower (0.50s) === RUN TestRaft_LeaderLeaseExpire 2026-04-14T14:16:12.720Z [INFO] server-f16e6ede-7041-9366-52ac-dd1c7cd2785a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f16e6ede-7041-9366-52ac-dd1c7cd2785a Address:f16e6ede-7041-9366-52ac-dd1c7cd2785a} {Suffrage:Voter ID:server-cab2f5f8-185b-616f-faa0-0cd350a04f53 Address:cab2f5f8-185b-616f-faa0-0cd350a04f53}]" 2026-04-14T14:16:12.720Z [INFO] server-f16e6ede-7041-9366-52ac-dd1c7cd2785a: entering follower state: follower="Node at f16e6ede-7041-9366-52ac-dd1c7cd2785a [Follower]" leader-address= leader-id= 2026-04-14T14:16:12.720Z [INFO] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f16e6ede-7041-9366-52ac-dd1c7cd2785a Address:f16e6ede-7041-9366-52ac-dd1c7cd2785a} {Suffrage:Voter ID:server-cab2f5f8-185b-616f-faa0-0cd350a04f53 Address:cab2f5f8-185b-616f-faa0-0cd350a04f53}]" 2026-04-14T14:16:12.720Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:12.720Z [INFO] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: entering follower state: follower="Node at cab2f5f8-185b-616f-faa0-0cd350a04f53 [Follower]" leader-address= leader-id= 2026-04-14T14:16:12.780Z [WARN] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:12.780Z [INFO] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: entering candidate state: node="Node at cab2f5f8-185b-616f-faa0-0cd350a04f53 [Candidate]" term=2 2026-04-14T14:16:12.780Z [INFO] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: election won: term=2 tally=2 2026-04-14T14:16:12.780Z [INFO] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: entering leader state: leader="Node at cab2f5f8-185b-616f-faa0-0cd350a04f53 [Leader]" 2026-04-14T14:16:12.780Z [INFO] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: added peer, starting replication: peer=server-f16e6ede-7041-9366-52ac-dd1c7cd2785a 2026-04-14T14:16:12.781Z [INFO] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: pipelining replication: peer="{Voter server-f16e6ede-7041-9366-52ac-dd1c7cd2785a f16e6ede-7041-9366-52ac-dd1c7cd2785a}" 2026-04-14T14:16:12.886Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:12.78096968 +0000 UTC m=+104.253155445 (1 nodes), 60.340252ms from start of poll, 60.631901ms from cluster start. Timeout at 2026-04-14 14:16:12.885974562 +0000 UTC m=+104.358160344, 105.004899ms after stability 2026-04-14T14:16:12.892Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:12.997Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:12.892494869 +0000 UTC m=+104.364680633 (1 nodes), 2.302µs from start of poll, 172.157089ms from cluster start. Timeout at 2026-04-14 14:16:12.997518422 +0000 UTC m=+104.469704215, 105.023582ms after stability raft_test.go:1799: [INFO] Disconnecting Node at f16e6ede-7041-9366-52ac-dd1c7cd2785a [Follower] 2026-04-14T14:16:13.004Z [ERROR] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: failed to pipeline appendEntries: peer="{Voter server-f16e6ede-7041-9366-52ac-dd1c7cd2785a f16e6ede-7041-9366-52ac-dd1c7cd2785a}" error="append pipeline closed" 2026-04-14T14:16:13.004Z [INFO] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: aborting pipeline replication: peer="{Voter server-f16e6ede-7041-9366-52ac-dd1c7cd2785a f16e6ede-7041-9366-52ac-dd1c7cd2785a}" 2026-04-14T14:16:13.006Z [ERROR] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: failed to heartbeat to: peer=f16e6ede-7041-9366-52ac-dd1c7cd2785a backoff time=10ms error="failed to connect to peer: f16e6ede-7041-9366-52ac-dd1c7cd2785a" 2026-04-14T14:16:13.012Z [ERROR] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: failed to appendEntries to: peer="{Voter server-f16e6ede-7041-9366-52ac-dd1c7cd2785a f16e6ede-7041-9366-52ac-dd1c7cd2785a}" error="failed to connect to peer: f16e6ede-7041-9366-52ac-dd1c7cd2785a" 2026-04-14T14:16:13.026Z [ERROR] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: failed to heartbeat to: peer=f16e6ede-7041-9366-52ac-dd1c7cd2785a backoff time=10ms error="failed to connect to peer: f16e6ede-7041-9366-52ac-dd1c7cd2785a" 2026-04-14T14:16:13.032Z [ERROR] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: failed to appendEntries to: peer="{Voter server-f16e6ede-7041-9366-52ac-dd1c7cd2785a f16e6ede-7041-9366-52ac-dd1c7cd2785a}" error="failed to connect to peer: f16e6ede-7041-9366-52ac-dd1c7cd2785a" 2026-04-14T14:16:13.047Z [ERROR] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: failed to heartbeat to: peer=f16e6ede-7041-9366-52ac-dd1c7cd2785a backoff time=10ms error="failed to connect to peer: f16e6ede-7041-9366-52ac-dd1c7cd2785a" 2026-04-14T14:16:13.047Z [WARN] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: failed to contact: server-id=server-f16e6ede-7041-9366-52ac-dd1c7cd2785a time=50.237399ms 2026-04-14T14:16:13.047Z [WARN] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: failed to contact quorum of nodes, stepping down 2026-04-14T14:16:13.047Z [INFO] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: entering follower state: follower="Node at cab2f5f8-185b-616f-faa0-0cd350a04f53 [Follower]" leader-address= leader-id= 2026-04-14T14:16:13.047Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:13.048Z [ERROR] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: failed to appendEntries to: peer="{Voter server-f16e6ede-7041-9366-52ac-dd1c7cd2785a f16e6ede-7041-9366-52ac-dd1c7cd2785a}" error="failed to connect to peer: f16e6ede-7041-9366-52ac-dd1c7cd2785a" 2026-04-14T14:16:13.097Z [WARN] server-f16e6ede-7041-9366-52ac-dd1c7cd2785a: heartbeat timeout reached, starting election: last-leader-addr=cab2f5f8-185b-616f-faa0-0cd350a04f53 last-leader-id=server-cab2f5f8-185b-616f-faa0-0cd350a04f53 2026-04-14T14:16:13.097Z [INFO] server-f16e6ede-7041-9366-52ac-dd1c7cd2785a: entering candidate state: node="Node at f16e6ede-7041-9366-52ac-dd1c7cd2785a [Candidate]" term=3 2026-04-14T14:16:13.097Z [ERROR] server-f16e6ede-7041-9366-52ac-dd1c7cd2785a: failed to make requestVote RPC: target="{Voter server-cab2f5f8-185b-616f-faa0-0cd350a04f53 cab2f5f8-185b-616f-faa0-0cd350a04f53}" error="failed to connect to peer: cab2f5f8-185b-616f-faa0-0cd350a04f53" term=3 2026-04-14T14:16:13.131Z [WARN] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:13.131Z [INFO] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: entering candidate state: node="Node at cab2f5f8-185b-616f-faa0-0cd350a04f53 [Candidate]" term=3 2026-04-14T14:16:13.131Z [ERROR] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: failed to make requestVote RPC: target="{Voter server-f16e6ede-7041-9366-52ac-dd1c7cd2785a f16e6ede-7041-9366-52ac-dd1c7cd2785a}" error="failed to connect to peer: f16e6ede-7041-9366-52ac-dd1c7cd2785a" term=3 2026-04-14T14:16:13.175Z [WARN] server-f16e6ede-7041-9366-52ac-dd1c7cd2785a: Election timeout reached, restarting election 2026-04-14T14:16:13.175Z [INFO] server-f16e6ede-7041-9366-52ac-dd1c7cd2785a: entering candidate state: node="Node at f16e6ede-7041-9366-52ac-dd1c7cd2785a [Candidate]" term=4 2026-04-14T14:16:13.176Z [ERROR] server-f16e6ede-7041-9366-52ac-dd1c7cd2785a: failed to make requestVote RPC: target="{Voter server-cab2f5f8-185b-616f-faa0-0cd350a04f53 cab2f5f8-185b-616f-faa0-0cd350a04f53}" error="failed to connect to peer: cab2f5f8-185b-616f-faa0-0cd350a04f53" term=4 2026-04-14T14:16:13.207Z [WARN] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: Election timeout reached, restarting election 2026-04-14T14:16:13.207Z [INFO] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: entering candidate state: node="Node at cab2f5f8-185b-616f-faa0-0cd350a04f53 [Candidate]" term=4 2026-04-14T14:16:13.207Z [ERROR] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: failed to make requestVote RPC: target="{Voter server-f16e6ede-7041-9366-52ac-dd1c7cd2785a f16e6ede-7041-9366-52ac-dd1c7cd2785a}" error="failed to connect to peer: f16e6ede-7041-9366-52ac-dd1c7cd2785a" term=4 2026-04-14T14:16:13.237Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:13.1319748 +0000 UTC m=+104.604160563 (0 nodes), 84.34456ms from start of poll, 411.637019ms from cluster start. Timeout at 2026-04-14 14:16:13.236978671 +0000 UTC m=+104.709164436, 105.003873ms after stability 2026-04-14T14:16:13.263Z [WARN] server-f16e6ede-7041-9366-52ac-dd1c7cd2785a: Election timeout reached, restarting election 2026-04-14T14:16:13.263Z [INFO] server-f16e6ede-7041-9366-52ac-dd1c7cd2785a: entering candidate state: node="Node at f16e6ede-7041-9366-52ac-dd1c7cd2785a [Candidate]" term=5 2026-04-14T14:16:13.263Z [ERROR] server-f16e6ede-7041-9366-52ac-dd1c7cd2785a: failed to make requestVote RPC: target="{Voter server-cab2f5f8-185b-616f-faa0-0cd350a04f53 cab2f5f8-185b-616f-faa0-0cd350a04f53}" error="failed to connect to peer: cab2f5f8-185b-616f-faa0-0cd350a04f53" term=5 2026-04-14T14:16:13.280Z [WARN] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: Election timeout reached, restarting election 2026-04-14T14:16:13.280Z [INFO] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: entering candidate state: node="Node at cab2f5f8-185b-616f-faa0-0cd350a04f53 [Candidate]" term=5 2026-04-14T14:16:13.281Z [ERROR] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: failed to make requestVote RPC: target="{Voter server-f16e6ede-7041-9366-52ac-dd1c7cd2785a f16e6ede-7041-9366-52ac-dd1c7cd2785a}" error="failed to connect to peer: f16e6ede-7041-9366-52ac-dd1c7cd2785a" term=5 2026-04-14T14:16:13.338Z [WARN] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: Election timeout reached, restarting election 2026-04-14T14:16:13.338Z [INFO] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: entering candidate state: node="Node at cab2f5f8-185b-616f-faa0-0cd350a04f53 [Candidate]" term=6 2026-04-14T14:16:13.338Z [ERROR] server-cab2f5f8-185b-616f-faa0-0cd350a04f53: failed to make requestVote RPC: target="{Voter server-f16e6ede-7041-9366-52ac-dd1c7cd2785a f16e6ede-7041-9366-52ac-dd1c7cd2785a}" error="failed to connect to peer: f16e6ede-7041-9366-52ac-dd1c7cd2785a" term=6 --- PASS: TestRaft_LeaderLeaseExpire (0.62s) === RUN TestRaft_Barrier 2026-04-14T14:16:13.344Z [INFO] server-5a5ded6a-167f-36f4-efdb-4cdb961c0bfa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a5ded6a-167f-36f4-efdb-4cdb961c0bfa Address:5a5ded6a-167f-36f4-efdb-4cdb961c0bfa} {Suffrage:Voter ID:server-21285a5c-2988-5b1e-5dec-a253aa862cb3 Address:21285a5c-2988-5b1e-5dec-a253aa862cb3} {Suffrage:Voter ID:server-43c8733b-5128-a31d-3651-cca29f1ba669 Address:43c8733b-5128-a31d-3651-cca29f1ba669}]" 2026-04-14T14:16:13.344Z [INFO] server-21285a5c-2988-5b1e-5dec-a253aa862cb3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a5ded6a-167f-36f4-efdb-4cdb961c0bfa Address:5a5ded6a-167f-36f4-efdb-4cdb961c0bfa} {Suffrage:Voter ID:server-21285a5c-2988-5b1e-5dec-a253aa862cb3 Address:21285a5c-2988-5b1e-5dec-a253aa862cb3} {Suffrage:Voter ID:server-43c8733b-5128-a31d-3651-cca29f1ba669 Address:43c8733b-5128-a31d-3651-cca29f1ba669}]" 2026-04-14T14:16:13.344Z [INFO] server-5a5ded6a-167f-36f4-efdb-4cdb961c0bfa: entering follower state: follower="Node at 5a5ded6a-167f-36f4-efdb-4cdb961c0bfa [Follower]" leader-address= leader-id= 2026-04-14T14:16:13.344Z [INFO] server-21285a5c-2988-5b1e-5dec-a253aa862cb3: entering follower state: follower="Node at 21285a5c-2988-5b1e-5dec-a253aa862cb3 [Follower]" leader-address= leader-id= 2026-04-14T14:16:13.344Z [INFO] server-43c8733b-5128-a31d-3651-cca29f1ba669: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a5ded6a-167f-36f4-efdb-4cdb961c0bfa Address:5a5ded6a-167f-36f4-efdb-4cdb961c0bfa} {Suffrage:Voter ID:server-21285a5c-2988-5b1e-5dec-a253aa862cb3 Address:21285a5c-2988-5b1e-5dec-a253aa862cb3} {Suffrage:Voter ID:server-43c8733b-5128-a31d-3651-cca29f1ba669 Address:43c8733b-5128-a31d-3651-cca29f1ba669}]" 2026-04-14T14:16:13.344Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:13.344Z [INFO] server-43c8733b-5128-a31d-3651-cca29f1ba669: entering follower state: follower="Node at 43c8733b-5128-a31d-3651-cca29f1ba669 [Follower]" leader-address= leader-id= 2026-04-14T14:16:13.395Z [WARN] server-5a5ded6a-167f-36f4-efdb-4cdb961c0bfa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:13.395Z [INFO] server-5a5ded6a-167f-36f4-efdb-4cdb961c0bfa: entering candidate state: node="Node at 5a5ded6a-167f-36f4-efdb-4cdb961c0bfa [Candidate]" term=2 2026-04-14T14:16:13.395Z [INFO] server-5a5ded6a-167f-36f4-efdb-4cdb961c0bfa: election won: term=2 tally=2 2026-04-14T14:16:13.395Z [INFO] server-5a5ded6a-167f-36f4-efdb-4cdb961c0bfa: entering leader state: leader="Node at 5a5ded6a-167f-36f4-efdb-4cdb961c0bfa [Leader]" 2026-04-14T14:16:13.395Z [INFO] server-5a5ded6a-167f-36f4-efdb-4cdb961c0bfa: added peer, starting replication: peer=server-21285a5c-2988-5b1e-5dec-a253aa862cb3 2026-04-14T14:16:13.395Z [INFO] server-5a5ded6a-167f-36f4-efdb-4cdb961c0bfa: added peer, starting replication: peer=server-43c8733b-5128-a31d-3651-cca29f1ba669 2026-04-14T14:16:13.395Z [INFO] server-5a5ded6a-167f-36f4-efdb-4cdb961c0bfa: pipelining replication: peer="{Voter server-21285a5c-2988-5b1e-5dec-a253aa862cb3 21285a5c-2988-5b1e-5dec-a253aa862cb3}" 2026-04-14T14:16:13.395Z [INFO] server-5a5ded6a-167f-36f4-efdb-4cdb961c0bfa: pipelining replication: peer="{Voter server-43c8733b-5128-a31d-3651-cca29f1ba669 43c8733b-5128-a31d-3651-cca29f1ba669}" 2026-04-14T14:16:13.500Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:13.395514346 +0000 UTC m=+104.867700123 (1 nodes), 51.265191ms from start of poll, 51.5652ms from cluster start. Timeout at 2026-04-14 14:16:13.500515509 +0000 UTC m=+104.972701273, 105.00115ms after stability 2026-04-14T14:16:13.519Z [INFO] server-5a5ded6a-167f-36f4-efdb-4cdb961c0bfa: aborting pipeline replication: peer="{Voter server-43c8733b-5128-a31d-3651-cca29f1ba669 43c8733b-5128-a31d-3651-cca29f1ba669}" 2026-04-14T14:16:13.519Z [INFO] server-5a5ded6a-167f-36f4-efdb-4cdb961c0bfa: aborting pipeline replication: peer="{Voter server-21285a5c-2988-5b1e-5dec-a253aa862cb3 21285a5c-2988-5b1e-5dec-a253aa862cb3}" --- PASS: TestRaft_Barrier (0.18s) === RUN TestRaft_VerifyLeader 2026-04-14T14:16:13.520Z [INFO] server-48d1e2f8-4631-aa0e-606a-a954d9ebe97d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48d1e2f8-4631-aa0e-606a-a954d9ebe97d Address:48d1e2f8-4631-aa0e-606a-a954d9ebe97d} {Suffrage:Voter ID:server-625f9f11-e512-4168-56fe-5ccd8dffbd3c Address:625f9f11-e512-4168-56fe-5ccd8dffbd3c} {Suffrage:Voter ID:server-68024fcc-cac8-230c-a79d-bda2277cea4c Address:68024fcc-cac8-230c-a79d-bda2277cea4c}]" 2026-04-14T14:16:13.520Z [INFO] server-625f9f11-e512-4168-56fe-5ccd8dffbd3c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48d1e2f8-4631-aa0e-606a-a954d9ebe97d Address:48d1e2f8-4631-aa0e-606a-a954d9ebe97d} {Suffrage:Voter ID:server-625f9f11-e512-4168-56fe-5ccd8dffbd3c Address:625f9f11-e512-4168-56fe-5ccd8dffbd3c} {Suffrage:Voter ID:server-68024fcc-cac8-230c-a79d-bda2277cea4c Address:68024fcc-cac8-230c-a79d-bda2277cea4c}]" 2026-04-14T14:16:13.520Z [INFO] server-48d1e2f8-4631-aa0e-606a-a954d9ebe97d: entering follower state: follower="Node at 48d1e2f8-4631-aa0e-606a-a954d9ebe97d [Follower]" leader-address= leader-id= 2026-04-14T14:16:13.520Z [INFO] server-625f9f11-e512-4168-56fe-5ccd8dffbd3c: entering follower state: follower="Node at 625f9f11-e512-4168-56fe-5ccd8dffbd3c [Follower]" leader-address= leader-id= 2026-04-14T14:16:13.520Z [INFO] server-68024fcc-cac8-230c-a79d-bda2277cea4c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48d1e2f8-4631-aa0e-606a-a954d9ebe97d Address:48d1e2f8-4631-aa0e-606a-a954d9ebe97d} {Suffrage:Voter ID:server-625f9f11-e512-4168-56fe-5ccd8dffbd3c Address:625f9f11-e512-4168-56fe-5ccd8dffbd3c} {Suffrage:Voter ID:server-68024fcc-cac8-230c-a79d-bda2277cea4c Address:68024fcc-cac8-230c-a79d-bda2277cea4c}]" 2026-04-14T14:16:13.520Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:13.520Z [INFO] server-68024fcc-cac8-230c-a79d-bda2277cea4c: entering follower state: follower="Node at 68024fcc-cac8-230c-a79d-bda2277cea4c [Follower]" leader-address= leader-id= 2026-04-14T14:16:13.576Z [WARN] server-48d1e2f8-4631-aa0e-606a-a954d9ebe97d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:13.576Z [INFO] server-48d1e2f8-4631-aa0e-606a-a954d9ebe97d: entering candidate state: node="Node at 48d1e2f8-4631-aa0e-606a-a954d9ebe97d [Candidate]" term=2 2026-04-14T14:16:13.576Z [INFO] server-48d1e2f8-4631-aa0e-606a-a954d9ebe97d: election won: term=2 tally=2 2026-04-14T14:16:13.576Z [INFO] server-48d1e2f8-4631-aa0e-606a-a954d9ebe97d: entering leader state: leader="Node at 48d1e2f8-4631-aa0e-606a-a954d9ebe97d [Leader]" 2026-04-14T14:16:13.576Z [INFO] server-48d1e2f8-4631-aa0e-606a-a954d9ebe97d: added peer, starting replication: peer=server-625f9f11-e512-4168-56fe-5ccd8dffbd3c 2026-04-14T14:16:13.576Z [INFO] server-48d1e2f8-4631-aa0e-606a-a954d9ebe97d: added peer, starting replication: peer=server-68024fcc-cac8-230c-a79d-bda2277cea4c 2026-04-14T14:16:13.576Z [INFO] server-48d1e2f8-4631-aa0e-606a-a954d9ebe97d: pipelining replication: peer="{Voter server-625f9f11-e512-4168-56fe-5ccd8dffbd3c 625f9f11-e512-4168-56fe-5ccd8dffbd3c}" 2026-04-14T14:16:13.576Z [INFO] server-48d1e2f8-4631-aa0e-606a-a954d9ebe97d: pipelining replication: peer="{Voter server-68024fcc-cac8-230c-a79d-bda2277cea4c 68024fcc-cac8-230c-a79d-bda2277cea4c}" 2026-04-14T14:16:13.681Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:13.576445441 +0000 UTC m=+105.048631204 (1 nodes), 55.906621ms from start of poll, 56.215773ms from cluster start. Timeout at 2026-04-14 14:16:13.681453245 +0000 UTC m=+105.153639014, 105.00781ms after stability 2026-04-14T14:16:13.681Z [INFO] server-48d1e2f8-4631-aa0e-606a-a954d9ebe97d: aborting pipeline replication: peer="{Voter server-68024fcc-cac8-230c-a79d-bda2277cea4c 68024fcc-cac8-230c-a79d-bda2277cea4c}" 2026-04-14T14:16:13.681Z [INFO] server-48d1e2f8-4631-aa0e-606a-a954d9ebe97d: aborting pipeline replication: peer="{Voter server-625f9f11-e512-4168-56fe-5ccd8dffbd3c 625f9f11-e512-4168-56fe-5ccd8dffbd3c}" 2026-04-14T14:16:14.182Z [ERROR] server-48d1e2f8-4631-aa0e-606a-a954d9ebe97d: failed to heartbeat to: peer=625f9f11-e512-4168-56fe-5ccd8dffbd3c backoff time=10ms error="command timed out" --- PASS: TestRaft_VerifyLeader (0.66s) === RUN TestRaft_VerifyLeader_Single 2026-04-14T14:16:14.182Z [INFO] server-6bd23a4e-1ae5-6a8a-5fa9-e0066f75a0d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6bd23a4e-1ae5-6a8a-5fa9-e0066f75a0d6 Address:6bd23a4e-1ae5-6a8a-5fa9-e0066f75a0d6}]" 2026-04-14T14:16:14.183Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:14.183Z [INFO] server-6bd23a4e-1ae5-6a8a-5fa9-e0066f75a0d6: entering follower state: follower="Node at 6bd23a4e-1ae5-6a8a-5fa9-e0066f75a0d6 [Follower]" leader-address= leader-id= 2026-04-14T14:16:14.246Z [WARN] server-6bd23a4e-1ae5-6a8a-5fa9-e0066f75a0d6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:14.246Z [INFO] server-6bd23a4e-1ae5-6a8a-5fa9-e0066f75a0d6: entering candidate state: node="Node at 6bd23a4e-1ae5-6a8a-5fa9-e0066f75a0d6 [Candidate]" term=2 2026-04-14T14:16:14.246Z [INFO] server-6bd23a4e-1ae5-6a8a-5fa9-e0066f75a0d6: election won: term=2 tally=1 2026-04-14T14:16:14.246Z [INFO] server-6bd23a4e-1ae5-6a8a-5fa9-e0066f75a0d6: entering leader state: leader="Node at 6bd23a4e-1ae5-6a8a-5fa9-e0066f75a0d6 [Leader]" 2026-04-14T14:16:14.351Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:14.246575256 +0000 UTC m=+105.718761025 (1 nodes), 63.559945ms from start of poll, 63.696978ms from cluster start. Timeout at 2026-04-14 14:16:14.351578537 +0000 UTC m=+105.823764301, 105.003276ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.17s) === RUN TestRaft_VerifyLeader_Fail 2026-04-14T14:16:14.352Z [INFO] server-b521906c-f046-ab36-55da-e98a1d40fa3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b521906c-f046-ab36-55da-e98a1d40fa3b Address:b521906c-f046-ab36-55da-e98a1d40fa3b} {Suffrage:Voter ID:server-85146384-ef8e-f8e6-02b4-4f5013770234 Address:85146384-ef8e-f8e6-02b4-4f5013770234}]" 2026-04-14T14:16:14.352Z [INFO] server-85146384-ef8e-f8e6-02b4-4f5013770234: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b521906c-f046-ab36-55da-e98a1d40fa3b Address:b521906c-f046-ab36-55da-e98a1d40fa3b} {Suffrage:Voter ID:server-85146384-ef8e-f8e6-02b4-4f5013770234 Address:85146384-ef8e-f8e6-02b4-4f5013770234}]" 2026-04-14T14:16:14.352Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:14.352Z [INFO] server-b521906c-f046-ab36-55da-e98a1d40fa3b: entering follower state: follower="Node at b521906c-f046-ab36-55da-e98a1d40fa3b [Follower]" leader-address= leader-id= 2026-04-14T14:16:14.352Z [INFO] server-85146384-ef8e-f8e6-02b4-4f5013770234: entering follower state: follower="Node at 85146384-ef8e-f8e6-02b4-4f5013770234 [Follower]" leader-address= leader-id= 2026-04-14T14:16:14.420Z [WARN] server-85146384-ef8e-f8e6-02b4-4f5013770234: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:14.420Z [INFO] server-85146384-ef8e-f8e6-02b4-4f5013770234: entering candidate state: node="Node at 85146384-ef8e-f8e6-02b4-4f5013770234 [Candidate]" term=2 2026-04-14T14:16:14.421Z [INFO] server-85146384-ef8e-f8e6-02b4-4f5013770234: election won: term=2 tally=2 2026-04-14T14:16:14.421Z [INFO] server-85146384-ef8e-f8e6-02b4-4f5013770234: entering leader state: leader="Node at 85146384-ef8e-f8e6-02b4-4f5013770234 [Leader]" 2026-04-14T14:16:14.421Z [INFO] server-85146384-ef8e-f8e6-02b4-4f5013770234: added peer, starting replication: peer=server-b521906c-f046-ab36-55da-e98a1d40fa3b 2026-04-14T14:16:14.421Z [INFO] server-85146384-ef8e-f8e6-02b4-4f5013770234: pipelining replication: peer="{Voter server-b521906c-f046-ab36-55da-e98a1d40fa3b b521906c-f046-ab36-55da-e98a1d40fa3b}" 2026-04-14T14:16:14.527Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:14.421140007 +0000 UTC m=+105.893325775 (1 nodes), 68.428609ms from start of poll, 68.627164ms from cluster start. Timeout at 2026-04-14 14:16:14.526146304 +0000 UTC m=+105.998332132, 105.006357ms after stability 2026-04-14T14:16:14.527Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:14.633Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:14.527528759 +0000 UTC m=+105.999714524 (1 nodes), 828ns from start of poll, 175.015913ms from cluster start. Timeout at 2026-04-14 14:16:14.632539137 +0000 UTC m=+106.104724916, 105.010392ms after stability 2026-04-14T14:16:14.634Z [ERROR] server-85146384-ef8e-f8e6-02b4-4f5013770234: peer has newer term, stopping replication: peer="{Voter server-b521906c-f046-ab36-55da-e98a1d40fa3b b521906c-f046-ab36-55da-e98a1d40fa3b}" 2026-04-14T14:16:14.634Z [INFO] server-85146384-ef8e-f8e6-02b4-4f5013770234: aborting pipeline replication: peer="{Voter server-b521906c-f046-ab36-55da-e98a1d40fa3b b521906c-f046-ab36-55da-e98a1d40fa3b}" 2026-04-14T14:16:14.634Z [INFO] server-85146384-ef8e-f8e6-02b4-4f5013770234: entering follower state: follower="Node at 85146384-ef8e-f8e6-02b4-4f5013770234 [Follower]" leader-address= leader-id= --- PASS: TestRaft_VerifyLeader_Fail (0.28s) === RUN TestRaft_VerifyLeader_PartialConnect 2026-04-14T14:16:14.636Z [INFO] server-3a1471b8-3de5-c536-478a-7e1b6427dade: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a1471b8-3de5-c536-478a-7e1b6427dade Address:3a1471b8-3de5-c536-478a-7e1b6427dade} {Suffrage:Voter ID:server-1566d30b-b90d-d540-1225-0d1133e3a720 Address:1566d30b-b90d-d540-1225-0d1133e3a720} {Suffrage:Voter ID:server-ac45b547-65bb-11c9-ef73-36009c7620e1 Address:ac45b547-65bb-11c9-ef73-36009c7620e1}]" 2026-04-14T14:16:14.636Z [INFO] server-3a1471b8-3de5-c536-478a-7e1b6427dade: entering follower state: follower="Node at 3a1471b8-3de5-c536-478a-7e1b6427dade [Follower]" leader-address= leader-id= 2026-04-14T14:16:14.636Z [INFO] server-1566d30b-b90d-d540-1225-0d1133e3a720: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a1471b8-3de5-c536-478a-7e1b6427dade Address:3a1471b8-3de5-c536-478a-7e1b6427dade} {Suffrage:Voter ID:server-1566d30b-b90d-d540-1225-0d1133e3a720 Address:1566d30b-b90d-d540-1225-0d1133e3a720} {Suffrage:Voter ID:server-ac45b547-65bb-11c9-ef73-36009c7620e1 Address:ac45b547-65bb-11c9-ef73-36009c7620e1}]" 2026-04-14T14:16:14.636Z [INFO] server-1566d30b-b90d-d540-1225-0d1133e3a720: entering follower state: follower="Node at 1566d30b-b90d-d540-1225-0d1133e3a720 [Follower]" leader-address= leader-id= 2026-04-14T14:16:14.636Z [INFO] server-ac45b547-65bb-11c9-ef73-36009c7620e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a1471b8-3de5-c536-478a-7e1b6427dade Address:3a1471b8-3de5-c536-478a-7e1b6427dade} {Suffrage:Voter ID:server-1566d30b-b90d-d540-1225-0d1133e3a720 Address:1566d30b-b90d-d540-1225-0d1133e3a720} {Suffrage:Voter ID:server-ac45b547-65bb-11c9-ef73-36009c7620e1 Address:ac45b547-65bb-11c9-ef73-36009c7620e1}]" 2026-04-14T14:16:14.636Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:14.636Z [INFO] server-ac45b547-65bb-11c9-ef73-36009c7620e1: entering follower state: follower="Node at ac45b547-65bb-11c9-ef73-36009c7620e1 [Follower]" leader-address= leader-id= 2026-04-14T14:16:14.688Z [WARN] server-1566d30b-b90d-d540-1225-0d1133e3a720: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:14.688Z [INFO] server-1566d30b-b90d-d540-1225-0d1133e3a720: entering candidate state: node="Node at 1566d30b-b90d-d540-1225-0d1133e3a720 [Candidate]" term=2 2026-04-14T14:16:14.688Z [INFO] server-1566d30b-b90d-d540-1225-0d1133e3a720: election won: term=2 tally=2 2026-04-14T14:16:14.688Z [INFO] server-1566d30b-b90d-d540-1225-0d1133e3a720: entering leader state: leader="Node at 1566d30b-b90d-d540-1225-0d1133e3a720 [Leader]" 2026-04-14T14:16:14.688Z [INFO] server-1566d30b-b90d-d540-1225-0d1133e3a720: added peer, starting replication: peer=server-3a1471b8-3de5-c536-478a-7e1b6427dade 2026-04-14T14:16:14.688Z [INFO] server-1566d30b-b90d-d540-1225-0d1133e3a720: added peer, starting replication: peer=server-ac45b547-65bb-11c9-ef73-36009c7620e1 2026-04-14T14:16:14.688Z [INFO] server-1566d30b-b90d-d540-1225-0d1133e3a720: pipelining replication: peer="{Voter server-ac45b547-65bb-11c9-ef73-36009c7620e1 ac45b547-65bb-11c9-ef73-36009c7620e1}" 2026-04-14T14:16:14.688Z [INFO] server-1566d30b-b90d-d540-1225-0d1133e3a720: pipelining replication: peer="{Voter server-3a1471b8-3de5-c536-478a-7e1b6427dade 3a1471b8-3de5-c536-478a-7e1b6427dade}" 2026-04-14T14:16:14.794Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:14.688813323 +0000 UTC m=+106.160999090 (1 nodes), 52.442074ms from start of poll, 52.875638ms from cluster start. Timeout at 2026-04-14 14:16:14.793825454 +0000 UTC m=+106.266011221, 105.012131ms after stability 2026-04-14T14:16:14.800Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:14.906Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:14.800924269 +0000 UTC m=+106.273110040 (2 nodes), 2.926µs from start of poll, 164.986588ms from cluster start. Timeout at 2026-04-14 14:16:14.905950235 +0000 UTC m=+106.378136004, 105.025964ms after stability raft_test.go:1968: [INFO] Disconnecting Node at 3a1471b8-3de5-c536-478a-7e1b6427dade [Follower] 2026-04-14T14:16:14.906Z [ERROR] server-1566d30b-b90d-d540-1225-0d1133e3a720: failed to heartbeat to: peer=3a1471b8-3de5-c536-478a-7e1b6427dade backoff time=10ms error="failed to connect to peer: 3a1471b8-3de5-c536-478a-7e1b6427dade" 2026-04-14T14:16:14.906Z [INFO] server-1566d30b-b90d-d540-1225-0d1133e3a720: aborting pipeline replication: peer="{Voter server-3a1471b8-3de5-c536-478a-7e1b6427dade 3a1471b8-3de5-c536-478a-7e1b6427dade}" 2026-04-14T14:16:14.906Z [INFO] server-1566d30b-b90d-d540-1225-0d1133e3a720: aborting pipeline replication: peer="{Voter server-ac45b547-65bb-11c9-ef73-36009c7620e1 ac45b547-65bb-11c9-ef73-36009c7620e1}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.27s) === RUN TestRaft_NotifyCh 2026-04-14T14:16:14.907Z [INFO] server-db63f7d1-5bde-3f42-bb0e-09f2ce7c3873: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db63f7d1-5bde-3f42-bb0e-09f2ce7c3873 Address:db63f7d1-5bde-3f42-bb0e-09f2ce7c3873}]" 2026-04-14T14:16:14.907Z [INFO] server-db63f7d1-5bde-3f42-bb0e-09f2ce7c3873: entering follower state: follower="Node at db63f7d1-5bde-3f42-bb0e-09f2ce7c3873 [Follower]" leader-address= leader-id= 2026-04-14T14:16:15.003Z [WARN] server-db63f7d1-5bde-3f42-bb0e-09f2ce7c3873: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:15.003Z [INFO] server-db63f7d1-5bde-3f42-bb0e-09f2ce7c3873: entering candidate state: node="Node at db63f7d1-5bde-3f42-bb0e-09f2ce7c3873 [Candidate]" term=2 2026-04-14T14:16:15.003Z [INFO] server-db63f7d1-5bde-3f42-bb0e-09f2ce7c3873: election won: term=2 tally=1 2026-04-14T14:16:15.003Z [INFO] server-db63f7d1-5bde-3f42-bb0e-09f2ce7c3873: entering leader state: leader="Node at db63f7d1-5bde-3f42-bb0e-09f2ce7c3873 [Leader]" --- PASS: TestRaft_NotifyCh (0.10s) === RUN TestRaft_AppendEntry 2026-04-14T14:16:15.004Z [INFO] server-92c252e3-feec-9b3f-d279-20071c2b966f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92c252e3-feec-9b3f-d279-20071c2b966f Address:92c252e3-feec-9b3f-d279-20071c2b966f} {Suffrage:Voter ID:server-6ad4c63d-e988-805d-ad1e-8b5cf7bdaa1f Address:6ad4c63d-e988-805d-ad1e-8b5cf7bdaa1f} {Suffrage:Voter ID:server-283b234b-3169-2b3b-ef10-adc856d85ed7 Address:283b234b-3169-2b3b-ef10-adc856d85ed7}]" 2026-04-14T14:16:15.004Z [INFO] server-92c252e3-feec-9b3f-d279-20071c2b966f: entering follower state: follower="Node at 92c252e3-feec-9b3f-d279-20071c2b966f [Follower]" leader-address= leader-id= 2026-04-14T14:16:15.004Z [INFO] server-6ad4c63d-e988-805d-ad1e-8b5cf7bdaa1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92c252e3-feec-9b3f-d279-20071c2b966f Address:92c252e3-feec-9b3f-d279-20071c2b966f} {Suffrage:Voter ID:server-6ad4c63d-e988-805d-ad1e-8b5cf7bdaa1f Address:6ad4c63d-e988-805d-ad1e-8b5cf7bdaa1f} {Suffrage:Voter ID:server-283b234b-3169-2b3b-ef10-adc856d85ed7 Address:283b234b-3169-2b3b-ef10-adc856d85ed7}]" 2026-04-14T14:16:15.004Z [INFO] server-6ad4c63d-e988-805d-ad1e-8b5cf7bdaa1f: entering follower state: follower="Node at 6ad4c63d-e988-805d-ad1e-8b5cf7bdaa1f [Follower]" leader-address= leader-id= 2026-04-14T14:16:15.004Z [INFO] server-283b234b-3169-2b3b-ef10-adc856d85ed7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92c252e3-feec-9b3f-d279-20071c2b966f Address:92c252e3-feec-9b3f-d279-20071c2b966f} {Suffrage:Voter ID:server-6ad4c63d-e988-805d-ad1e-8b5cf7bdaa1f Address:6ad4c63d-e988-805d-ad1e-8b5cf7bdaa1f} {Suffrage:Voter ID:server-283b234b-3169-2b3b-ef10-adc856d85ed7 Address:283b234b-3169-2b3b-ef10-adc856d85ed7}]" 2026-04-14T14:16:15.004Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:15.004Z [INFO] server-283b234b-3169-2b3b-ef10-adc856d85ed7: entering follower state: follower="Node at 283b234b-3169-2b3b-ef10-adc856d85ed7 [Follower]" leader-address= leader-id= 2026-04-14T14:16:15.062Z [WARN] server-92c252e3-feec-9b3f-d279-20071c2b966f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:15.062Z [INFO] server-92c252e3-feec-9b3f-d279-20071c2b966f: entering candidate state: node="Node at 92c252e3-feec-9b3f-d279-20071c2b966f [Candidate]" term=2 2026-04-14T14:16:15.062Z [INFO] server-92c252e3-feec-9b3f-d279-20071c2b966f: election won: term=2 tally=2 2026-04-14T14:16:15.062Z [INFO] server-92c252e3-feec-9b3f-d279-20071c2b966f: entering leader state: leader="Node at 92c252e3-feec-9b3f-d279-20071c2b966f [Leader]" 2026-04-14T14:16:15.062Z [INFO] server-92c252e3-feec-9b3f-d279-20071c2b966f: added peer, starting replication: peer=server-6ad4c63d-e988-805d-ad1e-8b5cf7bdaa1f 2026-04-14T14:16:15.062Z [INFO] server-92c252e3-feec-9b3f-d279-20071c2b966f: added peer, starting replication: peer=server-283b234b-3169-2b3b-ef10-adc856d85ed7 2026-04-14T14:16:15.062Z [INFO] server-92c252e3-feec-9b3f-d279-20071c2b966f: pipelining replication: peer="{Voter server-6ad4c63d-e988-805d-ad1e-8b5cf7bdaa1f 6ad4c63d-e988-805d-ad1e-8b5cf7bdaa1f}" 2026-04-14T14:16:15.062Z [INFO] server-92c252e3-feec-9b3f-d279-20071c2b966f: pipelining replication: peer="{Voter server-283b234b-3169-2b3b-ef10-adc856d85ed7 283b234b-3169-2b3b-ef10-adc856d85ed7}" 2026-04-14T14:16:15.168Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:15.062792037 +0000 UTC m=+106.534977801 (2 nodes), 58.294662ms from start of poll, 58.611247ms from cluster start. Timeout at 2026-04-14 14:16:15.167797394 +0000 UTC m=+106.639983163, 105.005362ms after stability 2026-04-14T14:16:15.168Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:15.274Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:15.168240484 +0000 UTC m=+106.640426251 (1 nodes), 1.677µs from start of poll, 164.059697ms from cluster start. Timeout at 2026-04-14 14:16:15.273335136 +0000 UTC m=+106.745520922, 105.094671ms after stability 2026-04-14T14:16:15.274Z [WARN] server-6ad4c63d-e988-805d-ad1e-8b5cf7bdaa1f: clearing log suffix: from=1 to=2 2026-04-14T14:16:15.274Z [WARN] server-6ad4c63d-e988-805d-ad1e-8b5cf7bdaa1f: skipping application of old log: index=1 2026-04-14T14:16:15.274Z [WARN] server-6ad4c63d-e988-805d-ad1e-8b5cf7bdaa1f: skipping application of old log: index=1 2026-04-14T14:16:15.274Z [INFO] server-92c252e3-feec-9b3f-d279-20071c2b966f: aborting pipeline replication: peer="{Voter server-283b234b-3169-2b3b-ef10-adc856d85ed7 283b234b-3169-2b3b-ef10-adc856d85ed7}" 2026-04-14T14:16:15.274Z [INFO] server-92c252e3-feec-9b3f-d279-20071c2b966f: aborting pipeline replication: peer="{Voter server-6ad4c63d-e988-805d-ad1e-8b5cf7bdaa1f 6ad4c63d-e988-805d-ad1e-8b5cf7bdaa1f}" --- PASS: TestRaft_AppendEntry (0.27s) === RUN TestRaft_VotingGrant_WhenLeaderAvailable 2026-04-14T14:16:15.275Z [INFO] server-3ed09d40-3936-d810-71db-da9397cb3e5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ed09d40-3936-d810-71db-da9397cb3e5e Address:3ed09d40-3936-d810-71db-da9397cb3e5e} {Suffrage:Voter ID:server-6d11d981-034d-cd26-70ce-924db7f7caa8 Address:6d11d981-034d-cd26-70ce-924db7f7caa8} {Suffrage:Voter ID:server-ea468d8a-1e25-2f0d-9dfa-f4dae403c899 Address:ea468d8a-1e25-2f0d-9dfa-f4dae403c899}]" 2026-04-14T14:16:15.275Z [INFO] server-6d11d981-034d-cd26-70ce-924db7f7caa8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ed09d40-3936-d810-71db-da9397cb3e5e Address:3ed09d40-3936-d810-71db-da9397cb3e5e} {Suffrage:Voter ID:server-6d11d981-034d-cd26-70ce-924db7f7caa8 Address:6d11d981-034d-cd26-70ce-924db7f7caa8} {Suffrage:Voter ID:server-ea468d8a-1e25-2f0d-9dfa-f4dae403c899 Address:ea468d8a-1e25-2f0d-9dfa-f4dae403c899}]" 2026-04-14T14:16:15.275Z [INFO] server-3ed09d40-3936-d810-71db-da9397cb3e5e: entering follower state: follower="Node at 3ed09d40-3936-d810-71db-da9397cb3e5e [Follower]" leader-address= leader-id= 2026-04-14T14:16:15.276Z [INFO] server-ea468d8a-1e25-2f0d-9dfa-f4dae403c899: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ed09d40-3936-d810-71db-da9397cb3e5e Address:3ed09d40-3936-d810-71db-da9397cb3e5e} {Suffrage:Voter ID:server-6d11d981-034d-cd26-70ce-924db7f7caa8 Address:6d11d981-034d-cd26-70ce-924db7f7caa8} {Suffrage:Voter ID:server-ea468d8a-1e25-2f0d-9dfa-f4dae403c899 Address:ea468d8a-1e25-2f0d-9dfa-f4dae403c899}]" 2026-04-14T14:16:15.276Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:15.276Z [INFO] server-6d11d981-034d-cd26-70ce-924db7f7caa8: entering follower state: follower="Node at 6d11d981-034d-cd26-70ce-924db7f7caa8 [Follower]" leader-address= leader-id= 2026-04-14T14:16:15.276Z [INFO] server-ea468d8a-1e25-2f0d-9dfa-f4dae403c899: entering follower state: follower="Node at ea468d8a-1e25-2f0d-9dfa-f4dae403c899 [Follower]" leader-address= leader-id= 2026-04-14T14:16:15.334Z [WARN] server-6d11d981-034d-cd26-70ce-924db7f7caa8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:15.334Z [INFO] server-6d11d981-034d-cd26-70ce-924db7f7caa8: entering candidate state: node="Node at 6d11d981-034d-cd26-70ce-924db7f7caa8 [Candidate]" term=2 2026-04-14T14:16:15.334Z [INFO] server-6d11d981-034d-cd26-70ce-924db7f7caa8: election won: term=2 tally=2 2026-04-14T14:16:15.334Z [INFO] server-6d11d981-034d-cd26-70ce-924db7f7caa8: entering leader state: leader="Node at 6d11d981-034d-cd26-70ce-924db7f7caa8 [Leader]" 2026-04-14T14:16:15.334Z [INFO] server-6d11d981-034d-cd26-70ce-924db7f7caa8: added peer, starting replication: peer=server-3ed09d40-3936-d810-71db-da9397cb3e5e 2026-04-14T14:16:15.334Z [INFO] server-6d11d981-034d-cd26-70ce-924db7f7caa8: added peer, starting replication: peer=server-ea468d8a-1e25-2f0d-9dfa-f4dae403c899 2026-04-14T14:16:15.335Z [INFO] server-6d11d981-034d-cd26-70ce-924db7f7caa8: pipelining replication: peer="{Voter server-3ed09d40-3936-d810-71db-da9397cb3e5e 3ed09d40-3936-d810-71db-da9397cb3e5e}" 2026-04-14T14:16:15.335Z [INFO] server-6d11d981-034d-cd26-70ce-924db7f7caa8: pipelining replication: peer="{Voter server-ea468d8a-1e25-2f0d-9dfa-f4dae403c899 ea468d8a-1e25-2f0d-9dfa-f4dae403c899}" 2026-04-14T14:16:15.440Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:15.334702175 +0000 UTC m=+106.806887939 (2 nodes), 58.656538ms from start of poll, 59.001966ms from cluster start. Timeout at 2026-04-14 14:16:15.439706639 +0000 UTC m=+106.911892404, 105.004465ms after stability 2026-04-14T14:16:15.440Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:15.545Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:15.440524181 +0000 UTC m=+106.912709949 (1 nodes), 2.299µs from start of poll, 164.823976ms from cluster start. Timeout at 2026-04-14 14:16:15.545536279 +0000 UTC m=+107.017722066, 105.012117ms after stability 2026-04-14T14:16:15.545Z [WARN] server-ea468d8a-1e25-2f0d-9dfa-f4dae403c899: rejecting vote request since we have a leader: from=3ed09d40-3936-d810-71db-da9397cb3e5e leader=6d11d981-034d-cd26-70ce-924db7f7caa8 leader-id=server-6d11d981-034d-cd26-70ce-924db7f7caa8 2026-04-14T14:16:15.545Z [INFO] server-6d11d981-034d-cd26-70ce-924db7f7caa8: aborting pipeline replication: peer="{Voter server-ea468d8a-1e25-2f0d-9dfa-f4dae403c899 ea468d8a-1e25-2f0d-9dfa-f4dae403c899}" 2026-04-14T14:16:15.545Z [INFO] server-6d11d981-034d-cd26-70ce-924db7f7caa8: aborting pipeline replication: peer="{Voter server-3ed09d40-3936-d810-71db-da9397cb3e5e 3ed09d40-3936-d810-71db-da9397cb3e5e}" --- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.27s) === RUN TestRaft_ProtocolVersion_RejectRPC 2026-04-14T14:16:15.546Z [INFO] server-b786c605-e97b-2c59-d4a0-b3dffb18ad6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b786c605-e97b-2c59-d4a0-b3dffb18ad6f Address:b786c605-e97b-2c59-d4a0-b3dffb18ad6f} {Suffrage:Voter ID:server-0ade1ba5-c544-a569-9c6d-8c749043e0f0 Address:0ade1ba5-c544-a569-9c6d-8c749043e0f0} {Suffrage:Voter ID:server-22866309-85d7-17ee-7891-7f248a5cfdd1 Address:22866309-85d7-17ee-7891-7f248a5cfdd1}]" 2026-04-14T14:16:15.546Z [INFO] server-0ade1ba5-c544-a569-9c6d-8c749043e0f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b786c605-e97b-2c59-d4a0-b3dffb18ad6f Address:b786c605-e97b-2c59-d4a0-b3dffb18ad6f} {Suffrage:Voter ID:server-0ade1ba5-c544-a569-9c6d-8c749043e0f0 Address:0ade1ba5-c544-a569-9c6d-8c749043e0f0} {Suffrage:Voter ID:server-22866309-85d7-17ee-7891-7f248a5cfdd1 Address:22866309-85d7-17ee-7891-7f248a5cfdd1}]" 2026-04-14T14:16:15.546Z [INFO] server-b786c605-e97b-2c59-d4a0-b3dffb18ad6f: entering follower state: follower="Node at b786c605-e97b-2c59-d4a0-b3dffb18ad6f [Follower]" leader-address= leader-id= 2026-04-14T14:16:15.546Z [INFO] server-0ade1ba5-c544-a569-9c6d-8c749043e0f0: entering follower state: follower="Node at 0ade1ba5-c544-a569-9c6d-8c749043e0f0 [Follower]" leader-address= leader-id= 2026-04-14T14:16:15.546Z [INFO] server-22866309-85d7-17ee-7891-7f248a5cfdd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b786c605-e97b-2c59-d4a0-b3dffb18ad6f Address:b786c605-e97b-2c59-d4a0-b3dffb18ad6f} {Suffrage:Voter ID:server-0ade1ba5-c544-a569-9c6d-8c749043e0f0 Address:0ade1ba5-c544-a569-9c6d-8c749043e0f0} {Suffrage:Voter ID:server-22866309-85d7-17ee-7891-7f248a5cfdd1 Address:22866309-85d7-17ee-7891-7f248a5cfdd1}]" 2026-04-14T14:16:15.546Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:15.546Z [INFO] server-22866309-85d7-17ee-7891-7f248a5cfdd1: entering follower state: follower="Node at 22866309-85d7-17ee-7891-7f248a5cfdd1 [Follower]" leader-address= leader-id= 2026-04-14T14:16:15.601Z [WARN] server-22866309-85d7-17ee-7891-7f248a5cfdd1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:15.601Z [INFO] server-22866309-85d7-17ee-7891-7f248a5cfdd1: entering candidate state: node="Node at 22866309-85d7-17ee-7891-7f248a5cfdd1 [Candidate]" term=2 2026-04-14T14:16:15.601Z [INFO] server-22866309-85d7-17ee-7891-7f248a5cfdd1: election won: term=2 tally=2 2026-04-14T14:16:15.601Z [INFO] server-22866309-85d7-17ee-7891-7f248a5cfdd1: entering leader state: leader="Node at 22866309-85d7-17ee-7891-7f248a5cfdd1 [Leader]" 2026-04-14T14:16:15.601Z [INFO] server-22866309-85d7-17ee-7891-7f248a5cfdd1: added peer, starting replication: peer=server-b786c605-e97b-2c59-d4a0-b3dffb18ad6f 2026-04-14T14:16:15.601Z [INFO] server-22866309-85d7-17ee-7891-7f248a5cfdd1: added peer, starting replication: peer=server-0ade1ba5-c544-a569-9c6d-8c749043e0f0 2026-04-14T14:16:15.601Z [INFO] server-22866309-85d7-17ee-7891-7f248a5cfdd1: pipelining replication: peer="{Voter server-0ade1ba5-c544-a569-9c6d-8c749043e0f0 0ade1ba5-c544-a569-9c6d-8c749043e0f0}" 2026-04-14T14:16:15.602Z [INFO] server-22866309-85d7-17ee-7891-7f248a5cfdd1: pipelining replication: peer="{Voter server-b786c605-e97b-2c59-d4a0-b3dffb18ad6f b786c605-e97b-2c59-d4a0-b3dffb18ad6f}" 2026-04-14T14:16:15.708Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:15.601926377 +0000 UTC m=+107.074112140 (2 nodes), 55.019956ms from start of poll, 55.317463ms from cluster start. Timeout at 2026-04-14 14:16:15.706928767 +0000 UTC m=+107.179114535, 105.002395ms after stability 2026-04-14T14:16:15.708Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:15.813Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:15.70807638 +0000 UTC m=+107.180262147 (1 nodes), 880ns from start of poll, 161.46747ms from cluster start. Timeout at 2026-04-14 14:16:15.813090251 +0000 UTC m=+107.285276020, 105.013873ms after stability 2026-04-14T14:16:15.813Z [INFO] server-22866309-85d7-17ee-7891-7f248a5cfdd1: aborting pipeline replication: peer="{Voter server-0ade1ba5-c544-a569-9c6d-8c749043e0f0 0ade1ba5-c544-a569-9c6d-8c749043e0f0}" 2026-04-14T14:16:15.813Z [INFO] server-22866309-85d7-17ee-7891-7f248a5cfdd1: aborting pipeline replication: peer="{Voter server-b786c605-e97b-2c59-d4a0-b3dffb18ad6f b786c605-e97b-2c59-d4a0-b3dffb18ad6f}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 raft_test.go:2151: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 raft_test.go:2196: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s) === RUN TestRaft_LeaderID_Propagated 2026-04-14T14:16:15.815Z [INFO] server-89c62147-80a6-d980-6ace-d69c9cef5136: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89c62147-80a6-d980-6ace-d69c9cef5136 Address:89c62147-80a6-d980-6ace-d69c9cef5136} {Suffrage:Voter ID:server-ac62651f-0723-61ae-803e-e6ceef97bd2c Address:ac62651f-0723-61ae-803e-e6ceef97bd2c} {Suffrage:Voter ID:server-c707de26-a234-122a-dc3a-5f7c9a973b5c Address:c707de26-a234-122a-dc3a-5f7c9a973b5c}]" 2026-04-14T14:16:15.815Z [INFO] server-ac62651f-0723-61ae-803e-e6ceef97bd2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89c62147-80a6-d980-6ace-d69c9cef5136 Address:89c62147-80a6-d980-6ace-d69c9cef5136} {Suffrage:Voter ID:server-ac62651f-0723-61ae-803e-e6ceef97bd2c Address:ac62651f-0723-61ae-803e-e6ceef97bd2c} {Suffrage:Voter ID:server-c707de26-a234-122a-dc3a-5f7c9a973b5c Address:c707de26-a234-122a-dc3a-5f7c9a973b5c}]" 2026-04-14T14:16:15.815Z [INFO] server-89c62147-80a6-d980-6ace-d69c9cef5136: entering follower state: follower="Node at 89c62147-80a6-d980-6ace-d69c9cef5136 [Follower]" leader-address= leader-id= 2026-04-14T14:16:15.815Z [INFO] server-ac62651f-0723-61ae-803e-e6ceef97bd2c: entering follower state: follower="Node at ac62651f-0723-61ae-803e-e6ceef97bd2c [Follower]" leader-address= leader-id= 2026-04-14T14:16:15.815Z [INFO] server-c707de26-a234-122a-dc3a-5f7c9a973b5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89c62147-80a6-d980-6ace-d69c9cef5136 Address:89c62147-80a6-d980-6ace-d69c9cef5136} {Suffrage:Voter ID:server-ac62651f-0723-61ae-803e-e6ceef97bd2c Address:ac62651f-0723-61ae-803e-e6ceef97bd2c} {Suffrage:Voter ID:server-c707de26-a234-122a-dc3a-5f7c9a973b5c Address:c707de26-a234-122a-dc3a-5f7c9a973b5c}]" 2026-04-14T14:16:15.815Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:15.815Z [INFO] server-c707de26-a234-122a-dc3a-5f7c9a973b5c: entering follower state: follower="Node at c707de26-a234-122a-dc3a-5f7c9a973b5c [Follower]" leader-address= leader-id= 2026-04-14T14:16:15.876Z [WARN] server-89c62147-80a6-d980-6ace-d69c9cef5136: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:15.876Z [INFO] server-89c62147-80a6-d980-6ace-d69c9cef5136: entering candidate state: node="Node at 89c62147-80a6-d980-6ace-d69c9cef5136 [Candidate]" term=2 2026-04-14T14:16:15.876Z [INFO] server-89c62147-80a6-d980-6ace-d69c9cef5136: election won: term=2 tally=2 2026-04-14T14:16:15.876Z [INFO] server-89c62147-80a6-d980-6ace-d69c9cef5136: entering leader state: leader="Node at 89c62147-80a6-d980-6ace-d69c9cef5136 [Leader]" 2026-04-14T14:16:15.876Z [INFO] server-89c62147-80a6-d980-6ace-d69c9cef5136: added peer, starting replication: peer=server-ac62651f-0723-61ae-803e-e6ceef97bd2c 2026-04-14T14:16:15.876Z [INFO] server-89c62147-80a6-d980-6ace-d69c9cef5136: added peer, starting replication: peer=server-c707de26-a234-122a-dc3a-5f7c9a973b5c 2026-04-14T14:16:15.876Z [INFO] server-89c62147-80a6-d980-6ace-d69c9cef5136: pipelining replication: peer="{Voter server-c707de26-a234-122a-dc3a-5f7c9a973b5c c707de26-a234-122a-dc3a-5f7c9a973b5c}" 2026-04-14T14:16:15.876Z [INFO] server-89c62147-80a6-d980-6ace-d69c9cef5136: pipelining replication: peer="{Voter server-ac62651f-0723-61ae-803e-e6ceef97bd2c ac62651f-0723-61ae-803e-e6ceef97bd2c}" 2026-04-14T14:16:15.982Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:15.876705191 +0000 UTC m=+107.348890956 (1 nodes), 61.500047ms from start of poll, 61.795913ms from cluster start. Timeout at 2026-04-14 14:16:15.981710731 +0000 UTC m=+107.453896496, 105.00554ms after stability 2026-04-14T14:16:15.982Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:16.087Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:15.982263965 +0000 UTC m=+107.454449729 (1 nodes), 1.608µs from start of poll, 167.354686ms from cluster start. Timeout at 2026-04-14 14:16:16.087301873 +0000 UTC m=+107.559487641, 105.037912ms after stability 2026-04-14T14:16:16.088Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:16.194Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:16.088198642 +0000 UTC m=+107.560384409 (1 nodes), 794ns from start of poll, 273.289366ms from cluster start. Timeout at 2026-04-14 14:16:16.193219849 +0000 UTC m=+107.665405618, 105.021209ms after stability 2026-04-14T14:16:16.194Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:16.299Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:16.19418975 +0000 UTC m=+107.666375516 (1 nodes), 634ns from start of poll, 379.280473ms from cluster start. Timeout at 2026-04-14 14:16:16.29921616 +0000 UTC m=+107.771401925, 105.026409ms after stability 2026-04-14T14:16:16.299Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:16.405Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:16.299707147 +0000 UTC m=+107.771892921 (1 nodes), 688ns from start of poll, 484.797878ms from cluster start. Timeout at 2026-04-14 14:16:16.404725893 +0000 UTC m=+107.876911680, 105.018759ms after stability 2026-04-14T14:16:16.405Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:16.512Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:16.405938011 +0000 UTC m=+107.878123787 (1 nodes), 618ns from start of poll, 591.028744ms from cluster start. Timeout at 2026-04-14 14:16:16.510969757 +0000 UTC m=+107.983155566, 105.031779ms after stability 2026-04-14T14:16:16.617Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:16.723Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:16.618021553 +0000 UTC m=+108.090207324 (1 nodes), 2.307µs from start of poll, 803.112281ms from cluster start. Timeout at 2026-04-14 14:16:16.723067223 +0000 UTC m=+108.195253004, 105.04568ms after stability 2026-04-14T14:16:16.724Z [INFO] server-89c62147-80a6-d980-6ace-d69c9cef5136: aborting pipeline replication: peer="{Voter server-c707de26-a234-122a-dc3a-5f7c9a973b5c c707de26-a234-122a-dc3a-5f7c9a973b5c}" 2026-04-14T14:16:16.724Z [INFO] server-89c62147-80a6-d980-6ace-d69c9cef5136: aborting pipeline replication: peer="{Voter server-ac62651f-0723-61ae-803e-e6ceef97bd2c ac62651f-0723-61ae-803e-e6ceef97bd2c}" --- PASS: TestRaft_LeaderID_Propagated (0.91s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer 2026-04-14T14:16:16.725Z [INFO] server-ac6f824c-8db2-5809-ee53-dc3739686f21: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac6f824c-8db2-5809-ee53-dc3739686f21 Address:ac6f824c-8db2-5809-ee53-dc3739686f21} {Suffrage:Voter ID:server-526ad209-a28d-53bd-b325-f569f7514961 Address:526ad209-a28d-53bd-b325-f569f7514961} {Suffrage:Voter ID:server-6889661a-c762-4a13-4edb-54788ad7e2bb Address:6889661a-c762-4a13-4edb-54788ad7e2bb}]" 2026-04-14T14:16:16.725Z [INFO] server-ac6f824c-8db2-5809-ee53-dc3739686f21: entering follower state: follower="Node at ac6f824c-8db2-5809-ee53-dc3739686f21 [Follower]" leader-address= leader-id= 2026-04-14T14:16:16.725Z [INFO] server-526ad209-a28d-53bd-b325-f569f7514961: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac6f824c-8db2-5809-ee53-dc3739686f21 Address:ac6f824c-8db2-5809-ee53-dc3739686f21} {Suffrage:Voter ID:server-526ad209-a28d-53bd-b325-f569f7514961 Address:526ad209-a28d-53bd-b325-f569f7514961} {Suffrage:Voter ID:server-6889661a-c762-4a13-4edb-54788ad7e2bb Address:6889661a-c762-4a13-4edb-54788ad7e2bb}]" 2026-04-14T14:16:16.725Z [INFO] server-526ad209-a28d-53bd-b325-f569f7514961: entering follower state: follower="Node at 526ad209-a28d-53bd-b325-f569f7514961 [Follower]" leader-address= leader-id= 2026-04-14T14:16:16.726Z [INFO] server-6889661a-c762-4a13-4edb-54788ad7e2bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac6f824c-8db2-5809-ee53-dc3739686f21 Address:ac6f824c-8db2-5809-ee53-dc3739686f21} {Suffrage:Voter ID:server-526ad209-a28d-53bd-b325-f569f7514961 Address:526ad209-a28d-53bd-b325-f569f7514961} {Suffrage:Voter ID:server-6889661a-c762-4a13-4edb-54788ad7e2bb Address:6889661a-c762-4a13-4edb-54788ad7e2bb}]" 2026-04-14T14:16:16.726Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:16.726Z [INFO] server-6889661a-c762-4a13-4edb-54788ad7e2bb: entering follower state: follower="Node at 6889661a-c762-4a13-4edb-54788ad7e2bb [Follower]" leader-address= leader-id= 2026-04-14T14:16:16.785Z [WARN] server-ac6f824c-8db2-5809-ee53-dc3739686f21: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:16.786Z [INFO] server-ac6f824c-8db2-5809-ee53-dc3739686f21: entering candidate state: node="Node at ac6f824c-8db2-5809-ee53-dc3739686f21 [Candidate]" term=2 2026-04-14T14:16:16.786Z [INFO] server-ac6f824c-8db2-5809-ee53-dc3739686f21: election won: term=2 tally=2 2026-04-14T14:16:16.786Z [INFO] server-ac6f824c-8db2-5809-ee53-dc3739686f21: entering leader state: leader="Node at ac6f824c-8db2-5809-ee53-dc3739686f21 [Leader]" 2026-04-14T14:16:16.786Z [INFO] server-ac6f824c-8db2-5809-ee53-dc3739686f21: added peer, starting replication: peer=server-526ad209-a28d-53bd-b325-f569f7514961 2026-04-14T14:16:16.786Z [INFO] server-ac6f824c-8db2-5809-ee53-dc3739686f21: added peer, starting replication: peer=server-6889661a-c762-4a13-4edb-54788ad7e2bb 2026-04-14T14:16:16.786Z [INFO] server-ac6f824c-8db2-5809-ee53-dc3739686f21: pipelining replication: peer="{Voter server-6889661a-c762-4a13-4edb-54788ad7e2bb 6889661a-c762-4a13-4edb-54788ad7e2bb}" 2026-04-14T14:16:16.786Z [INFO] server-ac6f824c-8db2-5809-ee53-dc3739686f21: pipelining replication: peer="{Voter server-526ad209-a28d-53bd-b325-f569f7514961 526ad209-a28d-53bd-b325-f569f7514961}" 2026-04-14T14:16:16.891Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:16.786135275 +0000 UTC m=+108.258321062 (1 nodes), 60.031821ms from start of poll, 60.575334ms from cluster start. Timeout at 2026-04-14 14:16:16.891138571 +0000 UTC m=+108.363324339, 105.003277ms after stability 2026-04-14T14:16:16.891Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:16.997Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:16.891942321 +0000 UTC m=+108.364128103 (1 nodes), 2.354µs from start of poll, 166.382375ms from cluster start. Timeout at 2026-04-14 14:16:16.996958473 +0000 UTC m=+108.469144261, 105.016158ms after stability 2026-04-14T14:16:16.997Z [INFO] server-526ad209-a28d-53bd-b325-f569f7514961: entering candidate state: node="Node at 526ad209-a28d-53bd-b325-f569f7514961 [Candidate]" term=3 2026-04-14T14:16:16.997Z [INFO] server-526ad209-a28d-53bd-b325-f569f7514961: election won: term=3 tally=2 2026-04-14T14:16:16.997Z [INFO] server-526ad209-a28d-53bd-b325-f569f7514961: entering leader state: leader="Node at 526ad209-a28d-53bd-b325-f569f7514961 [Leader]" 2026-04-14T14:16:16.997Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:16.997Z [INFO] server-ac6f824c-8db2-5809-ee53-dc3739686f21: entering follower state: follower="Node at ac6f824c-8db2-5809-ee53-dc3739686f21 [Follower]" leader-address= leader-id= 2026-04-14T14:16:16.997Z [INFO] server-526ad209-a28d-53bd-b325-f569f7514961: added peer, starting replication: peer=server-ac6f824c-8db2-5809-ee53-dc3739686f21 2026-04-14T14:16:16.997Z [INFO] server-526ad209-a28d-53bd-b325-f569f7514961: added peer, starting replication: peer=server-6889661a-c762-4a13-4edb-54788ad7e2bb 2026-04-14T14:16:16.997Z [INFO] server-ac6f824c-8db2-5809-ee53-dc3739686f21: aborting pipeline replication: peer="{Voter server-526ad209-a28d-53bd-b325-f569f7514961 526ad209-a28d-53bd-b325-f569f7514961}" 2026-04-14T14:16:16.997Z [INFO] server-526ad209-a28d-53bd-b325-f569f7514961: pipelining replication: peer="{Voter server-6889661a-c762-4a13-4edb-54788ad7e2bb 6889661a-c762-4a13-4edb-54788ad7e2bb}" 2026-04-14T14:16:16.997Z [INFO] server-ac6f824c-8db2-5809-ee53-dc3739686f21: aborting pipeline replication: peer="{Voter server-6889661a-c762-4a13-4edb-54788ad7e2bb 6889661a-c762-4a13-4edb-54788ad7e2bb}" 2026-04-14T14:16:16.997Z [INFO] server-526ad209-a28d-53bd-b325-f569f7514961: pipelining replication: peer="{Voter server-ac6f824c-8db2-5809-ee53-dc3739686f21 ac6f824c-8db2-5809-ee53-dc3739686f21}" 2026-04-14T14:16:17.103Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:16.997714536 +0000 UTC m=+108.469900369 (1 nodes), 21.05µs from start of poll, 272.154641ms from cluster start. Timeout at 2026-04-14 14:16:17.10278739 +0000 UTC m=+108.574973160, 105.072791ms after stability 2026-04-14T14:16:17.103Z [INFO] server-526ad209-a28d-53bd-b325-f569f7514961: aborting pipeline replication: peer="{Voter server-6889661a-c762-4a13-4edb-54788ad7e2bb 6889661a-c762-4a13-4edb-54788ad7e2bb}" 2026-04-14T14:16:17.103Z [INFO] server-526ad209-a28d-53bd-b325-f569f7514961: aborting pipeline replication: peer="{Voter server-ac6f824c-8db2-5809-ee53-dc3739686f21 ac6f824c-8db2-5809-ee53-dc3739686f21}" --- PASS: TestRaft_LeadershipTransfer (0.38s) === RUN TestRaft_LeadershipTransferWithOneNode 2026-04-14T14:16:17.104Z [INFO] server-d8d80571-c564-48f1-a298-7a4b6629907d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8d80571-c564-48f1-a298-7a4b6629907d Address:d8d80571-c564-48f1-a298-7a4b6629907d}]" 2026-04-14T14:16:17.104Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:17.104Z [INFO] server-d8d80571-c564-48f1-a298-7a4b6629907d: entering follower state: follower="Node at d8d80571-c564-48f1-a298-7a4b6629907d [Follower]" leader-address= leader-id= 2026-04-14T14:16:17.156Z [WARN] server-d8d80571-c564-48f1-a298-7a4b6629907d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:17.156Z [INFO] server-d8d80571-c564-48f1-a298-7a4b6629907d: entering candidate state: node="Node at d8d80571-c564-48f1-a298-7a4b6629907d [Candidate]" term=2 2026-04-14T14:16:17.156Z [INFO] server-d8d80571-c564-48f1-a298-7a4b6629907d: election won: term=2 tally=1 2026-04-14T14:16:17.156Z [INFO] server-d8d80571-c564-48f1-a298-7a4b6629907d: entering leader state: leader="Node at d8d80571-c564-48f1-a298-7a4b6629907d [Leader]" 2026-04-14T14:16:17.262Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:17.156898523 +0000 UTC m=+108.629084286 (1 nodes), 52.606916ms from start of poll, 52.745784ms from cluster start. Timeout at 2026-04-14 14:16:17.261902021 +0000 UTC m=+108.734087789, 105.003503ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.16s) === RUN TestRaft_LeadershipTransferWithSevenNodes 2026-04-14T14:16:17.264Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8391f441-4f7c-1084-0744-977323de82e4 Address:8391f441-4f7c-1084-0744-977323de82e4} {Suffrage:Voter ID:server-e903b126-4746-e594-b380-7b72761bc8eb Address:e903b126-4746-e594-b380-7b72761bc8eb} {Suffrage:Voter ID:server-02547553-b9c4-6944-57c3-33ddee58d9e3 Address:02547553-b9c4-6944-57c3-33ddee58d9e3} {Suffrage:Voter ID:server-99470da0-c6e5-af52-aa77-6009f8b8f87b Address:99470da0-c6e5-af52-aa77-6009f8b8f87b} {Suffrage:Voter ID:server-6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe Address:6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe} {Suffrage:Voter ID:server-44c1fb1a-7082-bc4d-394d-45f49ea7f966 Address:44c1fb1a-7082-bc4d-394d-45f49ea7f966} {Suffrage:Voter ID:server-cbdec43f-757f-da89-a232-9421dac38bb7 Address:cbdec43f-757f-da89-a232-9421dac38bb7}]" 2026-04-14T14:16:17.264Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: entering follower state: follower="Node at 8391f441-4f7c-1084-0744-977323de82e4 [Follower]" leader-address= leader-id= 2026-04-14T14:16:17.264Z [INFO] server-e903b126-4746-e594-b380-7b72761bc8eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8391f441-4f7c-1084-0744-977323de82e4 Address:8391f441-4f7c-1084-0744-977323de82e4} {Suffrage:Voter ID:server-e903b126-4746-e594-b380-7b72761bc8eb Address:e903b126-4746-e594-b380-7b72761bc8eb} {Suffrage:Voter ID:server-02547553-b9c4-6944-57c3-33ddee58d9e3 Address:02547553-b9c4-6944-57c3-33ddee58d9e3} {Suffrage:Voter ID:server-99470da0-c6e5-af52-aa77-6009f8b8f87b Address:99470da0-c6e5-af52-aa77-6009f8b8f87b} {Suffrage:Voter ID:server-6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe Address:6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe} {Suffrage:Voter ID:server-44c1fb1a-7082-bc4d-394d-45f49ea7f966 Address:44c1fb1a-7082-bc4d-394d-45f49ea7f966} {Suffrage:Voter ID:server-cbdec43f-757f-da89-a232-9421dac38bb7 Address:cbdec43f-757f-da89-a232-9421dac38bb7}]" 2026-04-14T14:16:17.264Z [INFO] server-e903b126-4746-e594-b380-7b72761bc8eb: entering follower state: follower="Node at e903b126-4746-e594-b380-7b72761bc8eb [Follower]" leader-address= leader-id= 2026-04-14T14:16:17.264Z [INFO] server-02547553-b9c4-6944-57c3-33ddee58d9e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8391f441-4f7c-1084-0744-977323de82e4 Address:8391f441-4f7c-1084-0744-977323de82e4} {Suffrage:Voter ID:server-e903b126-4746-e594-b380-7b72761bc8eb Address:e903b126-4746-e594-b380-7b72761bc8eb} {Suffrage:Voter ID:server-02547553-b9c4-6944-57c3-33ddee58d9e3 Address:02547553-b9c4-6944-57c3-33ddee58d9e3} {Suffrage:Voter ID:server-99470da0-c6e5-af52-aa77-6009f8b8f87b Address:99470da0-c6e5-af52-aa77-6009f8b8f87b} {Suffrage:Voter ID:server-6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe Address:6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe} {Suffrage:Voter ID:server-44c1fb1a-7082-bc4d-394d-45f49ea7f966 Address:44c1fb1a-7082-bc4d-394d-45f49ea7f966} {Suffrage:Voter ID:server-cbdec43f-757f-da89-a232-9421dac38bb7 Address:cbdec43f-757f-da89-a232-9421dac38bb7}]" 2026-04-14T14:16:17.264Z [INFO] server-02547553-b9c4-6944-57c3-33ddee58d9e3: entering follower state: follower="Node at 02547553-b9c4-6944-57c3-33ddee58d9e3 [Follower]" leader-address= leader-id= 2026-04-14T14:16:17.264Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8391f441-4f7c-1084-0744-977323de82e4 Address:8391f441-4f7c-1084-0744-977323de82e4} {Suffrage:Voter ID:server-e903b126-4746-e594-b380-7b72761bc8eb Address:e903b126-4746-e594-b380-7b72761bc8eb} {Suffrage:Voter ID:server-02547553-b9c4-6944-57c3-33ddee58d9e3 Address:02547553-b9c4-6944-57c3-33ddee58d9e3} {Suffrage:Voter ID:server-99470da0-c6e5-af52-aa77-6009f8b8f87b Address:99470da0-c6e5-af52-aa77-6009f8b8f87b} {Suffrage:Voter ID:server-6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe Address:6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe} {Suffrage:Voter ID:server-44c1fb1a-7082-bc4d-394d-45f49ea7f966 Address:44c1fb1a-7082-bc4d-394d-45f49ea7f966} {Suffrage:Voter ID:server-cbdec43f-757f-da89-a232-9421dac38bb7 Address:cbdec43f-757f-da89-a232-9421dac38bb7}]" 2026-04-14T14:16:17.265Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: entering follower state: follower="Node at 99470da0-c6e5-af52-aa77-6009f8b8f87b [Follower]" leader-address= leader-id= 2026-04-14T14:16:17.265Z [INFO] server-6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8391f441-4f7c-1084-0744-977323de82e4 Address:8391f441-4f7c-1084-0744-977323de82e4} {Suffrage:Voter ID:server-e903b126-4746-e594-b380-7b72761bc8eb Address:e903b126-4746-e594-b380-7b72761bc8eb} {Suffrage:Voter ID:server-02547553-b9c4-6944-57c3-33ddee58d9e3 Address:02547553-b9c4-6944-57c3-33ddee58d9e3} {Suffrage:Voter ID:server-99470da0-c6e5-af52-aa77-6009f8b8f87b Address:99470da0-c6e5-af52-aa77-6009f8b8f87b} {Suffrage:Voter ID:server-6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe Address:6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe} {Suffrage:Voter ID:server-44c1fb1a-7082-bc4d-394d-45f49ea7f966 Address:44c1fb1a-7082-bc4d-394d-45f49ea7f966} {Suffrage:Voter ID:server-cbdec43f-757f-da89-a232-9421dac38bb7 Address:cbdec43f-757f-da89-a232-9421dac38bb7}]" 2026-04-14T14:16:17.265Z [INFO] server-6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe: entering follower state: follower="Node at 6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe [Follower]" leader-address= leader-id= 2026-04-14T14:16:17.265Z [INFO] server-44c1fb1a-7082-bc4d-394d-45f49ea7f966: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8391f441-4f7c-1084-0744-977323de82e4 Address:8391f441-4f7c-1084-0744-977323de82e4} {Suffrage:Voter ID:server-e903b126-4746-e594-b380-7b72761bc8eb Address:e903b126-4746-e594-b380-7b72761bc8eb} {Suffrage:Voter ID:server-02547553-b9c4-6944-57c3-33ddee58d9e3 Address:02547553-b9c4-6944-57c3-33ddee58d9e3} {Suffrage:Voter ID:server-99470da0-c6e5-af52-aa77-6009f8b8f87b Address:99470da0-c6e5-af52-aa77-6009f8b8f87b} {Suffrage:Voter ID:server-6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe Address:6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe} {Suffrage:Voter ID:server-44c1fb1a-7082-bc4d-394d-45f49ea7f966 Address:44c1fb1a-7082-bc4d-394d-45f49ea7f966} {Suffrage:Voter ID:server-cbdec43f-757f-da89-a232-9421dac38bb7 Address:cbdec43f-757f-da89-a232-9421dac38bb7}]" 2026-04-14T14:16:17.265Z [INFO] server-44c1fb1a-7082-bc4d-394d-45f49ea7f966: entering follower state: follower="Node at 44c1fb1a-7082-bc4d-394d-45f49ea7f966 [Follower]" leader-address= leader-id= 2026-04-14T14:16:17.265Z [INFO] server-cbdec43f-757f-da89-a232-9421dac38bb7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8391f441-4f7c-1084-0744-977323de82e4 Address:8391f441-4f7c-1084-0744-977323de82e4} {Suffrage:Voter ID:server-e903b126-4746-e594-b380-7b72761bc8eb Address:e903b126-4746-e594-b380-7b72761bc8eb} {Suffrage:Voter ID:server-02547553-b9c4-6944-57c3-33ddee58d9e3 Address:02547553-b9c4-6944-57c3-33ddee58d9e3} {Suffrage:Voter ID:server-99470da0-c6e5-af52-aa77-6009f8b8f87b Address:99470da0-c6e5-af52-aa77-6009f8b8f87b} {Suffrage:Voter ID:server-6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe Address:6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe} {Suffrage:Voter ID:server-44c1fb1a-7082-bc4d-394d-45f49ea7f966 Address:44c1fb1a-7082-bc4d-394d-45f49ea7f966} {Suffrage:Voter ID:server-cbdec43f-757f-da89-a232-9421dac38bb7 Address:cbdec43f-757f-da89-a232-9421dac38bb7}]" 2026-04-14T14:16:17.265Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:17.265Z [INFO] server-cbdec43f-757f-da89-a232-9421dac38bb7: entering follower state: follower="Node at cbdec43f-757f-da89-a232-9421dac38bb7 [Follower]" leader-address= leader-id= 2026-04-14T14:16:17.316Z [WARN] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:17.316Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: entering candidate state: node="Node at 99470da0-c6e5-af52-aa77-6009f8b8f87b [Candidate]" term=2 2026-04-14T14:16:17.317Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: election won: term=2 tally=4 2026-04-14T14:16:17.317Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: entering leader state: leader="Node at 99470da0-c6e5-af52-aa77-6009f8b8f87b [Leader]" 2026-04-14T14:16:17.317Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: added peer, starting replication: peer=server-8391f441-4f7c-1084-0744-977323de82e4 2026-04-14T14:16:17.317Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: added peer, starting replication: peer=server-e903b126-4746-e594-b380-7b72761bc8eb 2026-04-14T14:16:17.317Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: added peer, starting replication: peer=server-02547553-b9c4-6944-57c3-33ddee58d9e3 2026-04-14T14:16:17.317Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: added peer, starting replication: peer=server-6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe 2026-04-14T14:16:17.317Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: added peer, starting replication: peer=server-44c1fb1a-7082-bc4d-394d-45f49ea7f966 2026-04-14T14:16:17.317Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: added peer, starting replication: peer=server-cbdec43f-757f-da89-a232-9421dac38bb7 2026-04-14T14:16:17.317Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: pipelining replication: peer="{Voter server-e903b126-4746-e594-b380-7b72761bc8eb e903b126-4746-e594-b380-7b72761bc8eb}" 2026-04-14T14:16:17.317Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: pipelining replication: peer="{Voter server-cbdec43f-757f-da89-a232-9421dac38bb7 cbdec43f-757f-da89-a232-9421dac38bb7}" 2026-04-14T14:16:17.317Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: pipelining replication: peer="{Voter server-44c1fb1a-7082-bc4d-394d-45f49ea7f966 44c1fb1a-7082-bc4d-394d-45f49ea7f966}" 2026-04-14T14:16:17.317Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: pipelining replication: peer="{Voter server-8391f441-4f7c-1084-0744-977323de82e4 8391f441-4f7c-1084-0744-977323de82e4}" 2026-04-14T14:16:17.317Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: pipelining replication: peer="{Voter server-6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe 6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe}" 2026-04-14T14:16:17.317Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: pipelining replication: peer="{Voter server-02547553-b9c4-6944-57c3-33ddee58d9e3 02547553-b9c4-6944-57c3-33ddee58d9e3}" 2026-04-14T14:16:17.422Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:17.317142597 +0000 UTC m=+108.789328380 (1 nodes), 51.798685ms from start of poll, 53.020124ms from cluster start. Timeout at 2026-04-14 14:16:17.422147157 +0000 UTC m=+108.894332926, 105.004546ms after stability 2026-04-14T14:16:17.422Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:17.531Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:17.422766486 +0000 UTC m=+108.894952270 (6 nodes), 3.503µs from start of poll, 158.644014ms from cluster start. Timeout at 2026-04-14 14:16:17.527825468 +0000 UTC m=+109.000011245, 105.058975ms after stability 2026-04-14T14:16:17.532Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:17.638Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:17.532027973 +0000 UTC m=+109.004213747 (1 nodes), 1.421µs from start of poll, 267.905491ms from cluster start. Timeout at 2026-04-14 14:16:17.637049604 +0000 UTC m=+109.109235390, 105.021643ms after stability 2026-04-14T14:16:17.638Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: entering candidate state: node="Node at 8391f441-4f7c-1084-0744-977323de82e4 [Candidate]" term=3 2026-04-14T14:16:17.638Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: election won: term=3 tally=4 2026-04-14T14:16:17.638Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: entering leader state: leader="Node at 8391f441-4f7c-1084-0744-977323de82e4 [Leader]" 2026-04-14T14:16:17.638Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: added peer, starting replication: peer=server-e903b126-4746-e594-b380-7b72761bc8eb 2026-04-14T14:16:17.638Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: added peer, starting replication: peer=server-02547553-b9c4-6944-57c3-33ddee58d9e3 2026-04-14T14:16:17.638Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: entering follower state: follower="Node at 99470da0-c6e5-af52-aa77-6009f8b8f87b [Follower]" leader-address= leader-id= 2026-04-14T14:16:17.638Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: aborting pipeline replication: peer="{Voter server-e903b126-4746-e594-b380-7b72761bc8eb e903b126-4746-e594-b380-7b72761bc8eb}" 2026-04-14T14:16:17.638Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:17.638Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: added peer, starting replication: peer=server-99470da0-c6e5-af52-aa77-6009f8b8f87b 2026-04-14T14:16:17.638Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: aborting pipeline replication: peer="{Voter server-6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe 6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe}" 2026-04-14T14:16:17.638Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: aborting pipeline replication: peer="{Voter server-8391f441-4f7c-1084-0744-977323de82e4 8391f441-4f7c-1084-0744-977323de82e4}" 2026-04-14T14:16:17.638Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: pipelining replication: peer="{Voter server-02547553-b9c4-6944-57c3-33ddee58d9e3 02547553-b9c4-6944-57c3-33ddee58d9e3}" 2026-04-14T14:16:17.638Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: aborting pipeline replication: peer="{Voter server-cbdec43f-757f-da89-a232-9421dac38bb7 cbdec43f-757f-da89-a232-9421dac38bb7}" 2026-04-14T14:16:17.638Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: aborting pipeline replication: peer="{Voter server-02547553-b9c4-6944-57c3-33ddee58d9e3 02547553-b9c4-6944-57c3-33ddee58d9e3}" 2026-04-14T14:16:17.638Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: pipelining replication: peer="{Voter server-e903b126-4746-e594-b380-7b72761bc8eb e903b126-4746-e594-b380-7b72761bc8eb}" 2026-04-14T14:16:17.638Z [INFO] server-99470da0-c6e5-af52-aa77-6009f8b8f87b: aborting pipeline replication: peer="{Voter server-44c1fb1a-7082-bc4d-394d-45f49ea7f966 44c1fb1a-7082-bc4d-394d-45f49ea7f966}" 2026-04-14T14:16:17.638Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: added peer, starting replication: peer=server-6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe 2026-04-14T14:16:17.638Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: added peer, starting replication: peer=server-44c1fb1a-7082-bc4d-394d-45f49ea7f966 2026-04-14T14:16:17.638Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: pipelining replication: peer="{Voter server-99470da0-c6e5-af52-aa77-6009f8b8f87b 99470da0-c6e5-af52-aa77-6009f8b8f87b}" 2026-04-14T14:16:17.638Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: pipelining replication: peer="{Voter server-6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe 6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe}" 2026-04-14T14:16:17.638Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: added peer, starting replication: peer=server-cbdec43f-757f-da89-a232-9421dac38bb7 2026-04-14T14:16:17.638Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: pipelining replication: peer="{Voter server-44c1fb1a-7082-bc4d-394d-45f49ea7f966 44c1fb1a-7082-bc4d-394d-45f49ea7f966}" 2026-04-14T14:16:17.638Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: pipelining replication: peer="{Voter server-cbdec43f-757f-da89-a232-9421dac38bb7 cbdec43f-757f-da89-a232-9421dac38bb7}" 2026-04-14T14:16:17.743Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:17.63847083 +0000 UTC m=+109.110656614 (1 nodes), 43.47µs from start of poll, 374.348358ms from cluster start. Timeout at 2026-04-14 14:16:17.743473051 +0000 UTC m=+109.215658816, 105.002202ms after stability 2026-04-14T14:16:17.743Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: aborting pipeline replication: peer="{Voter server-cbdec43f-757f-da89-a232-9421dac38bb7 cbdec43f-757f-da89-a232-9421dac38bb7}" 2026-04-14T14:16:17.743Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: aborting pipeline replication: peer="{Voter server-99470da0-c6e5-af52-aa77-6009f8b8f87b 99470da0-c6e5-af52-aa77-6009f8b8f87b}" 2026-04-14T14:16:17.743Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: aborting pipeline replication: peer="{Voter server-02547553-b9c4-6944-57c3-33ddee58d9e3 02547553-b9c4-6944-57c3-33ddee58d9e3}" 2026-04-14T14:16:17.743Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: aborting pipeline replication: peer="{Voter server-e903b126-4746-e594-b380-7b72761bc8eb e903b126-4746-e594-b380-7b72761bc8eb}" 2026-04-14T14:16:17.743Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: aborting pipeline replication: peer="{Voter server-44c1fb1a-7082-bc4d-394d-45f49ea7f966 44c1fb1a-7082-bc4d-394d-45f49ea7f966}" 2026-04-14T14:16:17.743Z [INFO] server-8391f441-4f7c-1084-0744-977323de82e4: aborting pipeline replication: peer="{Voter server-6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe 6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe}" 2026-04-14T14:16:18.243Z [ERROR] server-8391f441-4f7c-1084-0744-977323de82e4: failed to heartbeat to: peer=6c1ef5eb-54f8-01bc-5cd9-df7286a82fbe backoff time=10ms error="command timed out" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.98s) === RUN TestRaft_LeadershipTransferToInvalidID 2026-04-14T14:16:18.245Z [INFO] server-63a3cdb7-77bd-cc17-247b-87d87731bc34: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63a3cdb7-77bd-cc17-247b-87d87731bc34 Address:63a3cdb7-77bd-cc17-247b-87d87731bc34} {Suffrage:Voter ID:server-fa3ee8e7-3743-c2e2-1be8-3eede79d504d Address:fa3ee8e7-3743-c2e2-1be8-3eede79d504d} {Suffrage:Voter ID:server-64be1c31-1910-7250-6d63-1da80ad10663 Address:64be1c31-1910-7250-6d63-1da80ad10663}]" 2026-04-14T14:16:18.245Z [INFO] server-fa3ee8e7-3743-c2e2-1be8-3eede79d504d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63a3cdb7-77bd-cc17-247b-87d87731bc34 Address:63a3cdb7-77bd-cc17-247b-87d87731bc34} {Suffrage:Voter ID:server-fa3ee8e7-3743-c2e2-1be8-3eede79d504d Address:fa3ee8e7-3743-c2e2-1be8-3eede79d504d} {Suffrage:Voter ID:server-64be1c31-1910-7250-6d63-1da80ad10663 Address:64be1c31-1910-7250-6d63-1da80ad10663}]" 2026-04-14T14:16:18.245Z [INFO] server-63a3cdb7-77bd-cc17-247b-87d87731bc34: entering follower state: follower="Node at 63a3cdb7-77bd-cc17-247b-87d87731bc34 [Follower]" leader-address= leader-id= 2026-04-14T14:16:18.245Z [INFO] server-fa3ee8e7-3743-c2e2-1be8-3eede79d504d: entering follower state: follower="Node at fa3ee8e7-3743-c2e2-1be8-3eede79d504d [Follower]" leader-address= leader-id= 2026-04-14T14:16:18.245Z [INFO] server-64be1c31-1910-7250-6d63-1da80ad10663: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63a3cdb7-77bd-cc17-247b-87d87731bc34 Address:63a3cdb7-77bd-cc17-247b-87d87731bc34} {Suffrage:Voter ID:server-fa3ee8e7-3743-c2e2-1be8-3eede79d504d Address:fa3ee8e7-3743-c2e2-1be8-3eede79d504d} {Suffrage:Voter ID:server-64be1c31-1910-7250-6d63-1da80ad10663 Address:64be1c31-1910-7250-6d63-1da80ad10663}]" 2026-04-14T14:16:18.245Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:18.245Z [INFO] server-64be1c31-1910-7250-6d63-1da80ad10663: entering follower state: follower="Node at 64be1c31-1910-7250-6d63-1da80ad10663 [Follower]" leader-address= leader-id= 2026-04-14T14:16:18.313Z [WARN] server-fa3ee8e7-3743-c2e2-1be8-3eede79d504d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:18.313Z [INFO] server-fa3ee8e7-3743-c2e2-1be8-3eede79d504d: entering candidate state: node="Node at fa3ee8e7-3743-c2e2-1be8-3eede79d504d [Candidate]" term=2 2026-04-14T14:16:18.313Z [INFO] server-fa3ee8e7-3743-c2e2-1be8-3eede79d504d: election won: term=2 tally=2 2026-04-14T14:16:18.313Z [INFO] server-fa3ee8e7-3743-c2e2-1be8-3eede79d504d: entering leader state: leader="Node at fa3ee8e7-3743-c2e2-1be8-3eede79d504d [Leader]" 2026-04-14T14:16:18.313Z [INFO] server-fa3ee8e7-3743-c2e2-1be8-3eede79d504d: added peer, starting replication: peer=server-63a3cdb7-77bd-cc17-247b-87d87731bc34 2026-04-14T14:16:18.313Z [INFO] server-fa3ee8e7-3743-c2e2-1be8-3eede79d504d: added peer, starting replication: peer=server-64be1c31-1910-7250-6d63-1da80ad10663 2026-04-14T14:16:18.314Z [INFO] server-fa3ee8e7-3743-c2e2-1be8-3eede79d504d: pipelining replication: peer="{Voter server-63a3cdb7-77bd-cc17-247b-87d87731bc34 63a3cdb7-77bd-cc17-247b-87d87731bc34}" 2026-04-14T14:16:18.314Z [INFO] server-fa3ee8e7-3743-c2e2-1be8-3eede79d504d: pipelining replication: peer="{Voter server-64be1c31-1910-7250-6d63-1da80ad10663 64be1c31-1910-7250-6d63-1da80ad10663}" 2026-04-14T14:16:18.419Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:18.313971163 +0000 UTC m=+109.786156931 (1 nodes), 68.433602ms from start of poll, 68.764486ms from cluster start. Timeout at 2026-04-14 14:16:18.418974218 +0000 UTC m=+109.891159986, 105.003055ms after stability 2026-04-14T14:16:18.419Z [INFO] server-fa3ee8e7-3743-c2e2-1be8-3eede79d504d: aborting pipeline replication: peer="{Voter server-64be1c31-1910-7250-6d63-1da80ad10663 64be1c31-1910-7250-6d63-1da80ad10663}" 2026-04-14T14:16:18.419Z [INFO] server-fa3ee8e7-3743-c2e2-1be8-3eede79d504d: aborting pipeline replication: peer="{Voter server-63a3cdb7-77bd-cc17-247b-87d87731bc34 63a3cdb7-77bd-cc17-247b-87d87731bc34}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.18s) === RUN TestRaft_LeadershipTransferToInvalidAddress 2026-04-14T14:16:18.420Z [INFO] server-a7f772b8-22f0-9679-e9fd-3edacecee44d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a7f772b8-22f0-9679-e9fd-3edacecee44d Address:a7f772b8-22f0-9679-e9fd-3edacecee44d} {Suffrage:Voter ID:server-cd9d137a-79ac-14f4-e3c5-434250f795bd Address:cd9d137a-79ac-14f4-e3c5-434250f795bd} {Suffrage:Voter ID:server-e343b7f1-e75e-5539-3ae0-1bf79a7dade4 Address:e343b7f1-e75e-5539-3ae0-1bf79a7dade4}]" 2026-04-14T14:16:18.420Z [INFO] server-a7f772b8-22f0-9679-e9fd-3edacecee44d: entering follower state: follower="Node at a7f772b8-22f0-9679-e9fd-3edacecee44d [Follower]" leader-address= leader-id= 2026-04-14T14:16:18.420Z [INFO] server-cd9d137a-79ac-14f4-e3c5-434250f795bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a7f772b8-22f0-9679-e9fd-3edacecee44d Address:a7f772b8-22f0-9679-e9fd-3edacecee44d} {Suffrage:Voter ID:server-cd9d137a-79ac-14f4-e3c5-434250f795bd Address:cd9d137a-79ac-14f4-e3c5-434250f795bd} {Suffrage:Voter ID:server-e343b7f1-e75e-5539-3ae0-1bf79a7dade4 Address:e343b7f1-e75e-5539-3ae0-1bf79a7dade4}]" 2026-04-14T14:16:18.420Z [INFO] server-cd9d137a-79ac-14f4-e3c5-434250f795bd: entering follower state: follower="Node at cd9d137a-79ac-14f4-e3c5-434250f795bd [Follower]" leader-address= leader-id= 2026-04-14T14:16:18.420Z [INFO] server-e343b7f1-e75e-5539-3ae0-1bf79a7dade4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a7f772b8-22f0-9679-e9fd-3edacecee44d Address:a7f772b8-22f0-9679-e9fd-3edacecee44d} {Suffrage:Voter ID:server-cd9d137a-79ac-14f4-e3c5-434250f795bd Address:cd9d137a-79ac-14f4-e3c5-434250f795bd} {Suffrage:Voter ID:server-e343b7f1-e75e-5539-3ae0-1bf79a7dade4 Address:e343b7f1-e75e-5539-3ae0-1bf79a7dade4}]" 2026-04-14T14:16:18.420Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:18.420Z [INFO] server-e343b7f1-e75e-5539-3ae0-1bf79a7dade4: entering follower state: follower="Node at e343b7f1-e75e-5539-3ae0-1bf79a7dade4 [Follower]" leader-address= leader-id= 2026-04-14T14:16:18.504Z [WARN] server-cd9d137a-79ac-14f4-e3c5-434250f795bd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:18.504Z [INFO] server-cd9d137a-79ac-14f4-e3c5-434250f795bd: entering candidate state: node="Node at cd9d137a-79ac-14f4-e3c5-434250f795bd [Candidate]" term=2 2026-04-14T14:16:18.504Z [INFO] server-cd9d137a-79ac-14f4-e3c5-434250f795bd: election won: term=2 tally=2 2026-04-14T14:16:18.504Z [INFO] server-cd9d137a-79ac-14f4-e3c5-434250f795bd: entering leader state: leader="Node at cd9d137a-79ac-14f4-e3c5-434250f795bd [Leader]" 2026-04-14T14:16:18.504Z [INFO] server-cd9d137a-79ac-14f4-e3c5-434250f795bd: added peer, starting replication: peer=server-a7f772b8-22f0-9679-e9fd-3edacecee44d 2026-04-14T14:16:18.504Z [INFO] server-cd9d137a-79ac-14f4-e3c5-434250f795bd: added peer, starting replication: peer=server-e343b7f1-e75e-5539-3ae0-1bf79a7dade4 2026-04-14T14:16:18.504Z [INFO] server-cd9d137a-79ac-14f4-e3c5-434250f795bd: pipelining replication: peer="{Voter server-a7f772b8-22f0-9679-e9fd-3edacecee44d a7f772b8-22f0-9679-e9fd-3edacecee44d}" 2026-04-14T14:16:18.505Z [INFO] server-cd9d137a-79ac-14f4-e3c5-434250f795bd: pipelining replication: peer="{Voter server-e343b7f1-e75e-5539-3ae0-1bf79a7dade4 e343b7f1-e75e-5539-3ae0-1bf79a7dade4}" 2026-04-14T14:16:18.610Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:18.504775716 +0000 UTC m=+109.976961547 (2 nodes), 84.096613ms from start of poll, 84.383899ms from cluster start. Timeout at 2026-04-14 14:16:18.609777806 +0000 UTC m=+110.081963574, 105.002027ms after stability 2026-04-14T14:16:18.610Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:18.715Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:18.610429919 +0000 UTC m=+110.082615705 (1 nodes), 2.526µs from start of poll, 190.038057ms from cluster start. Timeout at 2026-04-14 14:16:18.715456353 +0000 UTC m=+110.187642118, 105.026413ms after stability 2026-04-14T14:16:18.715Z [INFO] server-cd9d137a-79ac-14f4-e3c5-434250f795bd: aborting pipeline replication: peer="{Voter server-e343b7f1-e75e-5539-3ae0-1bf79a7dade4 e343b7f1-e75e-5539-3ae0-1bf79a7dade4}" 2026-04-14T14:16:18.715Z [INFO] server-cd9d137a-79ac-14f4-e3c5-434250f795bd: aborting pipeline replication: peer="{Voter server-a7f772b8-22f0-9679-e9fd-3edacecee44d a7f772b8-22f0-9679-e9fd-3edacecee44d}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.30s) === RUN TestRaft_LeadershipTransferToBehindServer 2026-04-14T14:16:18.716Z [INFO] server-dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2 Address:dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2} {Suffrage:Voter ID:server-a997ae89-ff3a-b876-2dee-736bf3a0cc97 Address:a997ae89-ff3a-b876-2dee-736bf3a0cc97} {Suffrage:Voter ID:server-e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0 Address:e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0}]" 2026-04-14T14:16:18.716Z [INFO] server-a997ae89-ff3a-b876-2dee-736bf3a0cc97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2 Address:dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2} {Suffrage:Voter ID:server-a997ae89-ff3a-b876-2dee-736bf3a0cc97 Address:a997ae89-ff3a-b876-2dee-736bf3a0cc97} {Suffrage:Voter ID:server-e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0 Address:e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0}]" 2026-04-14T14:16:18.716Z [INFO] server-dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2: entering follower state: follower="Node at dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2 [Follower]" leader-address= leader-id= 2026-04-14T14:16:18.716Z [INFO] server-a997ae89-ff3a-b876-2dee-736bf3a0cc97: entering follower state: follower="Node at a997ae89-ff3a-b876-2dee-736bf3a0cc97 [Follower]" leader-address= leader-id= 2026-04-14T14:16:18.716Z [INFO] server-e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2 Address:dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2} {Suffrage:Voter ID:server-a997ae89-ff3a-b876-2dee-736bf3a0cc97 Address:a997ae89-ff3a-b876-2dee-736bf3a0cc97} {Suffrage:Voter ID:server-e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0 Address:e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0}]" 2026-04-14T14:16:18.716Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:18.716Z [INFO] server-e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0: entering follower state: follower="Node at e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0 [Follower]" leader-address= leader-id= 2026-04-14T14:16:18.778Z [WARN] server-a997ae89-ff3a-b876-2dee-736bf3a0cc97: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:18.778Z [INFO] server-a997ae89-ff3a-b876-2dee-736bf3a0cc97: entering candidate state: node="Node at a997ae89-ff3a-b876-2dee-736bf3a0cc97 [Candidate]" term=2 2026-04-14T14:16:18.779Z [INFO] server-a997ae89-ff3a-b876-2dee-736bf3a0cc97: election won: term=2 tally=2 2026-04-14T14:16:18.779Z [INFO] server-a997ae89-ff3a-b876-2dee-736bf3a0cc97: entering leader state: leader="Node at a997ae89-ff3a-b876-2dee-736bf3a0cc97 [Leader]" 2026-04-14T14:16:18.779Z [INFO] server-a997ae89-ff3a-b876-2dee-736bf3a0cc97: added peer, starting replication: peer=server-dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2 2026-04-14T14:16:18.779Z [INFO] server-a997ae89-ff3a-b876-2dee-736bf3a0cc97: added peer, starting replication: peer=server-e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0 2026-04-14T14:16:18.779Z [INFO] server-a997ae89-ff3a-b876-2dee-736bf3a0cc97: pipelining replication: peer="{Voter server-e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0 e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0}" 2026-04-14T14:16:18.779Z [INFO] server-a997ae89-ff3a-b876-2dee-736bf3a0cc97: pipelining replication: peer="{Voter server-dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2 dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2}" 2026-04-14T14:16:18.884Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:18.779061393 +0000 UTC m=+110.251247163 (1 nodes), 62.164691ms from start of poll, 62.499414ms from cluster start. Timeout at 2026-04-14 14:16:18.884061994 +0000 UTC m=+110.356247759, 105.000596ms after stability 2026-04-14T14:16:18.884Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:18.990Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:18.884131533 +0000 UTC m=+110.356317309 (2 nodes), 1.351µs from start of poll, 167.56956ms from cluster start. Timeout at 2026-04-14 14:16:18.989154211 +0000 UTC m=+110.461339980, 105.022671ms after stability 2026-04-14T14:16:19.017Z [INFO] server-dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2: entering candidate state: node="Node at dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2 [Candidate]" term=3 2026-04-14T14:16:19.017Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:19.017Z [INFO] server-dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2: election won: term=3 tally=2 2026-04-14T14:16:19.017Z [INFO] server-a997ae89-ff3a-b876-2dee-736bf3a0cc97: entering follower state: follower="Node at a997ae89-ff3a-b876-2dee-736bf3a0cc97 [Follower]" leader-address= leader-id= 2026-04-14T14:16:19.017Z [INFO] server-dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2: entering leader state: leader="Node at dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2 [Leader]" 2026-04-14T14:16:19.017Z [INFO] server-a997ae89-ff3a-b876-2dee-736bf3a0cc97: aborting pipeline replication: peer="{Voter server-dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2 dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2}" 2026-04-14T14:16:19.017Z [INFO] server-dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2: added peer, starting replication: peer=server-a997ae89-ff3a-b876-2dee-736bf3a0cc97 2026-04-14T14:16:19.017Z [INFO] server-dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2: added peer, starting replication: peer=server-e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0 2026-04-14T14:16:19.017Z [INFO] server-a997ae89-ff3a-b876-2dee-736bf3a0cc97: aborting pipeline replication: peer="{Voter server-e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0 e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0}" 2026-04-14T14:16:19.017Z [INFO] server-dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2: pipelining replication: peer="{Voter server-a997ae89-ff3a-b876-2dee-736bf3a0cc97 a997ae89-ff3a-b876-2dee-736bf3a0cc97}" 2026-04-14T14:16:19.017Z [INFO] server-dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2: pipelining replication: peer="{Voter server-e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0 e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0}" 2026-04-14T14:16:19.123Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:19.01723557 +0000 UTC m=+110.489421335 (1 nodes), 38.83µs from start of poll, 300.673586ms from cluster start. Timeout at 2026-04-14 14:16:19.122274686 +0000 UTC m=+110.594460454, 105.039119ms after stability 2026-04-14T14:16:19.123Z [INFO] server-dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2: aborting pipeline replication: peer="{Voter server-e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0 e2d96bc0-17bc-fa56-4d21-e45bd54bbdd0}" 2026-04-14T14:16:19.123Z [INFO] server-dfaaeb24-625d-52b5-f3b0-f7ff47a78fe2: aborting pipeline replication: peer="{Voter server-a997ae89-ff3a-b876-2dee-736bf3a0cc97 a997ae89-ff3a-b876-2dee-736bf3a0cc97}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.41s) === RUN TestRaft_LeadershipTransferToItself 2026-04-14T14:16:19.125Z [INFO] server-1ec888a8-5e68-db9f-e575-b417a515f7ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ec888a8-5e68-db9f-e575-b417a515f7ff Address:1ec888a8-5e68-db9f-e575-b417a515f7ff} {Suffrage:Voter ID:server-8e7044d7-9d30-d758-62d1-a294fce17d2c Address:8e7044d7-9d30-d758-62d1-a294fce17d2c} {Suffrage:Voter ID:server-2b5f3670-c1dd-f951-9ee7-44fe3cccf22e Address:2b5f3670-c1dd-f951-9ee7-44fe3cccf22e}]" 2026-04-14T14:16:19.125Z [INFO] server-8e7044d7-9d30-d758-62d1-a294fce17d2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ec888a8-5e68-db9f-e575-b417a515f7ff Address:1ec888a8-5e68-db9f-e575-b417a515f7ff} {Suffrage:Voter ID:server-8e7044d7-9d30-d758-62d1-a294fce17d2c Address:8e7044d7-9d30-d758-62d1-a294fce17d2c} {Suffrage:Voter ID:server-2b5f3670-c1dd-f951-9ee7-44fe3cccf22e Address:2b5f3670-c1dd-f951-9ee7-44fe3cccf22e}]" 2026-04-14T14:16:19.125Z [INFO] server-1ec888a8-5e68-db9f-e575-b417a515f7ff: entering follower state: follower="Node at 1ec888a8-5e68-db9f-e575-b417a515f7ff [Follower]" leader-address= leader-id= 2026-04-14T14:16:19.125Z [INFO] server-8e7044d7-9d30-d758-62d1-a294fce17d2c: entering follower state: follower="Node at 8e7044d7-9d30-d758-62d1-a294fce17d2c [Follower]" leader-address= leader-id= 2026-04-14T14:16:19.125Z [INFO] server-2b5f3670-c1dd-f951-9ee7-44fe3cccf22e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ec888a8-5e68-db9f-e575-b417a515f7ff Address:1ec888a8-5e68-db9f-e575-b417a515f7ff} {Suffrage:Voter ID:server-8e7044d7-9d30-d758-62d1-a294fce17d2c Address:8e7044d7-9d30-d758-62d1-a294fce17d2c} {Suffrage:Voter ID:server-2b5f3670-c1dd-f951-9ee7-44fe3cccf22e Address:2b5f3670-c1dd-f951-9ee7-44fe3cccf22e}]" 2026-04-14T14:16:19.125Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:19.125Z [INFO] server-2b5f3670-c1dd-f951-9ee7-44fe3cccf22e: entering follower state: follower="Node at 2b5f3670-c1dd-f951-9ee7-44fe3cccf22e [Follower]" leader-address= leader-id= 2026-04-14T14:16:19.180Z [WARN] server-2b5f3670-c1dd-f951-9ee7-44fe3cccf22e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:19.180Z [INFO] server-2b5f3670-c1dd-f951-9ee7-44fe3cccf22e: entering candidate state: node="Node at 2b5f3670-c1dd-f951-9ee7-44fe3cccf22e [Candidate]" term=2 2026-04-14T14:16:19.180Z [INFO] server-2b5f3670-c1dd-f951-9ee7-44fe3cccf22e: election won: term=2 tally=2 2026-04-14T14:16:19.180Z [INFO] server-2b5f3670-c1dd-f951-9ee7-44fe3cccf22e: entering leader state: leader="Node at 2b5f3670-c1dd-f951-9ee7-44fe3cccf22e [Leader]" 2026-04-14T14:16:19.180Z [INFO] server-2b5f3670-c1dd-f951-9ee7-44fe3cccf22e: added peer, starting replication: peer=server-1ec888a8-5e68-db9f-e575-b417a515f7ff 2026-04-14T14:16:19.180Z [INFO] server-2b5f3670-c1dd-f951-9ee7-44fe3cccf22e: added peer, starting replication: peer=server-8e7044d7-9d30-d758-62d1-a294fce17d2c 2026-04-14T14:16:19.180Z [INFO] server-2b5f3670-c1dd-f951-9ee7-44fe3cccf22e: pipelining replication: peer="{Voter server-1ec888a8-5e68-db9f-e575-b417a515f7ff 1ec888a8-5e68-db9f-e575-b417a515f7ff}" 2026-04-14T14:16:19.180Z [INFO] server-2b5f3670-c1dd-f951-9ee7-44fe3cccf22e: pipelining replication: peer="{Voter server-8e7044d7-9d30-d758-62d1-a294fce17d2c 8e7044d7-9d30-d758-62d1-a294fce17d2c}" 2026-04-14T14:16:19.286Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:19.18080614 +0000 UTC m=+110.652991916 (1 nodes), 54.794332ms from start of poll, 55.548251ms from cluster start. Timeout at 2026-04-14 14:16:19.28580766 +0000 UTC m=+110.757993423, 105.001507ms after stability 2026-04-14T14:16:19.286Z [INFO] server-2b5f3670-c1dd-f951-9ee7-44fe3cccf22e: cannot transfer leadership to itself 2026-04-14T14:16:19.286Z [INFO] server-2b5f3670-c1dd-f951-9ee7-44fe3cccf22e: aborting pipeline replication: peer="{Voter server-8e7044d7-9d30-d758-62d1-a294fce17d2c 8e7044d7-9d30-d758-62d1-a294fce17d2c}" 2026-04-14T14:16:19.286Z [INFO] server-2b5f3670-c1dd-f951-9ee7-44fe3cccf22e: aborting pipeline replication: peer="{Voter server-1ec888a8-5e68-db9f-e575-b417a515f7ff 1ec888a8-5e68-db9f-e575-b417a515f7ff}" --- PASS: TestRaft_LeadershipTransferToItself (0.16s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests 2026-04-14T14:16:19.287Z [INFO] server-8bc7a246-15b4-9c7a-1f73-9f3417319f0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8bc7a246-15b4-9c7a-1f73-9f3417319f0c Address:8bc7a246-15b4-9c7a-1f73-9f3417319f0c} {Suffrage:Voter ID:server-238a19ee-1897-8316-6390-55af2e51c360 Address:238a19ee-1897-8316-6390-55af2e51c360} {Suffrage:Voter ID:server-ed12823c-09da-a930-81c1-147d3c160e33 Address:ed12823c-09da-a930-81c1-147d3c160e33}]" 2026-04-14T14:16:19.287Z [INFO] server-8bc7a246-15b4-9c7a-1f73-9f3417319f0c: entering follower state: follower="Node at 8bc7a246-15b4-9c7a-1f73-9f3417319f0c [Follower]" leader-address= leader-id= 2026-04-14T14:16:19.287Z [INFO] server-238a19ee-1897-8316-6390-55af2e51c360: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8bc7a246-15b4-9c7a-1f73-9f3417319f0c Address:8bc7a246-15b4-9c7a-1f73-9f3417319f0c} {Suffrage:Voter ID:server-238a19ee-1897-8316-6390-55af2e51c360 Address:238a19ee-1897-8316-6390-55af2e51c360} {Suffrage:Voter ID:server-ed12823c-09da-a930-81c1-147d3c160e33 Address:ed12823c-09da-a930-81c1-147d3c160e33}]" 2026-04-14T14:16:19.287Z [INFO] server-238a19ee-1897-8316-6390-55af2e51c360: entering follower state: follower="Node at 238a19ee-1897-8316-6390-55af2e51c360 [Follower]" leader-address= leader-id= 2026-04-14T14:16:19.287Z [INFO] server-ed12823c-09da-a930-81c1-147d3c160e33: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8bc7a246-15b4-9c7a-1f73-9f3417319f0c Address:8bc7a246-15b4-9c7a-1f73-9f3417319f0c} {Suffrage:Voter ID:server-238a19ee-1897-8316-6390-55af2e51c360 Address:238a19ee-1897-8316-6390-55af2e51c360} {Suffrage:Voter ID:server-ed12823c-09da-a930-81c1-147d3c160e33 Address:ed12823c-09da-a930-81c1-147d3c160e33}]" 2026-04-14T14:16:19.287Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:19.287Z [INFO] server-ed12823c-09da-a930-81c1-147d3c160e33: entering follower state: follower="Node at ed12823c-09da-a930-81c1-147d3c160e33 [Follower]" leader-address= leader-id= 2026-04-14T14:16:19.357Z [WARN] server-ed12823c-09da-a930-81c1-147d3c160e33: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:19.357Z [INFO] server-ed12823c-09da-a930-81c1-147d3c160e33: entering candidate state: node="Node at ed12823c-09da-a930-81c1-147d3c160e33 [Candidate]" term=2 2026-04-14T14:16:19.357Z [INFO] server-ed12823c-09da-a930-81c1-147d3c160e33: election won: term=2 tally=2 2026-04-14T14:16:19.357Z [INFO] server-ed12823c-09da-a930-81c1-147d3c160e33: entering leader state: leader="Node at ed12823c-09da-a930-81c1-147d3c160e33 [Leader]" 2026-04-14T14:16:19.357Z [INFO] server-ed12823c-09da-a930-81c1-147d3c160e33: added peer, starting replication: peer=server-8bc7a246-15b4-9c7a-1f73-9f3417319f0c 2026-04-14T14:16:19.357Z [INFO] server-ed12823c-09da-a930-81c1-147d3c160e33: added peer, starting replication: peer=server-238a19ee-1897-8316-6390-55af2e51c360 2026-04-14T14:16:19.357Z [INFO] server-ed12823c-09da-a930-81c1-147d3c160e33: pipelining replication: peer="{Voter server-8bc7a246-15b4-9c7a-1f73-9f3417319f0c 8bc7a246-15b4-9c7a-1f73-9f3417319f0c}" 2026-04-14T14:16:19.358Z [INFO] server-ed12823c-09da-a930-81c1-147d3c160e33: pipelining replication: peer="{Voter server-238a19ee-1897-8316-6390-55af2e51c360 238a19ee-1897-8316-6390-55af2e51c360}" 2026-04-14T14:16:19.462Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:19.357888813 +0000 UTC m=+110.830074578 (1 nodes), 69.996995ms from start of poll, 70.36429ms from cluster start. Timeout at 2026-04-14 14:16:19.462891747 +0000 UTC m=+110.935077511, 105.002933ms after stability raft_test.go:2457: waiting on future 0 raft_test.go:2457: waiting on future 1 raft_test.go:2457: waiting on future 2 raft_test.go:2457: waiting on future 3 raft_test.go:2457: waiting on future 4 raft_test.go:2457: waiting on future 5 raft_test.go:2457: waiting on future 6 2026-04-14T14:16:19.463Z [INFO] server-ed12823c-09da-a930-81c1-147d3c160e33: aborting pipeline replication: peer="{Voter server-8bc7a246-15b4-9c7a-1f73-9f3417319f0c 8bc7a246-15b4-9c7a-1f73-9f3417319f0c}" 2026-04-14T14:16:19.463Z [INFO] server-ed12823c-09da-a930-81c1-147d3c160e33: aborting pipeline replication: peer="{Voter server-238a19ee-1897-8316-6390-55af2e51c360 238a19ee-1897-8316-6390-55af2e51c360}" --- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.18s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout 2026-04-14T14:16:19.464Z [INFO] server-8ff6c772-28ab-e98c-e312-cf85e6d209a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ff6c772-28ab-e98c-e312-cf85e6d209a9 Address:8ff6c772-28ab-e98c-e312-cf85e6d209a9} {Suffrage:Voter ID:server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198 Address:51f3a200-7af1-7edc-ca4f-e8f60fd3a198} {Suffrage:Voter ID:server-d42b99f7-0b81-e875-302a-9210b379aed3 Address:d42b99f7-0b81-e875-302a-9210b379aed3}]" 2026-04-14T14:16:19.464Z [INFO] server-8ff6c772-28ab-e98c-e312-cf85e6d209a9: entering follower state: follower="Node at 8ff6c772-28ab-e98c-e312-cf85e6d209a9 [Follower]" leader-address= leader-id= 2026-04-14T14:16:19.464Z [INFO] server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ff6c772-28ab-e98c-e312-cf85e6d209a9 Address:8ff6c772-28ab-e98c-e312-cf85e6d209a9} {Suffrage:Voter ID:server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198 Address:51f3a200-7af1-7edc-ca4f-e8f60fd3a198} {Suffrage:Voter ID:server-d42b99f7-0b81-e875-302a-9210b379aed3 Address:d42b99f7-0b81-e875-302a-9210b379aed3}]" 2026-04-14T14:16:19.464Z [INFO] server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198: entering follower state: follower="Node at 51f3a200-7af1-7edc-ca4f-e8f60fd3a198 [Follower]" leader-address= leader-id= 2026-04-14T14:16:19.464Z [INFO] server-d42b99f7-0b81-e875-302a-9210b379aed3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ff6c772-28ab-e98c-e312-cf85e6d209a9 Address:8ff6c772-28ab-e98c-e312-cf85e6d209a9} {Suffrage:Voter ID:server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198 Address:51f3a200-7af1-7edc-ca4f-e8f60fd3a198} {Suffrage:Voter ID:server-d42b99f7-0b81-e875-302a-9210b379aed3 Address:d42b99f7-0b81-e875-302a-9210b379aed3}]" 2026-04-14T14:16:19.464Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:19.464Z [INFO] server-d42b99f7-0b81-e875-302a-9210b379aed3: entering follower state: follower="Node at d42b99f7-0b81-e875-302a-9210b379aed3 [Follower]" leader-address= leader-id= 2026-04-14T14:16:19.519Z [WARN] server-d42b99f7-0b81-e875-302a-9210b379aed3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:19.519Z [INFO] server-d42b99f7-0b81-e875-302a-9210b379aed3: entering candidate state: node="Node at d42b99f7-0b81-e875-302a-9210b379aed3 [Candidate]" term=2 2026-04-14T14:16:19.519Z [INFO] server-d42b99f7-0b81-e875-302a-9210b379aed3: election won: term=2 tally=2 2026-04-14T14:16:19.519Z [INFO] server-d42b99f7-0b81-e875-302a-9210b379aed3: entering leader state: leader="Node at d42b99f7-0b81-e875-302a-9210b379aed3 [Leader]" 2026-04-14T14:16:19.519Z [INFO] server-d42b99f7-0b81-e875-302a-9210b379aed3: added peer, starting replication: peer=server-8ff6c772-28ab-e98c-e312-cf85e6d209a9 2026-04-14T14:16:19.519Z [INFO] server-d42b99f7-0b81-e875-302a-9210b379aed3: added peer, starting replication: peer=server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198 2026-04-14T14:16:19.519Z [INFO] server-d42b99f7-0b81-e875-302a-9210b379aed3: pipelining replication: peer="{Voter server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198 51f3a200-7af1-7edc-ca4f-e8f60fd3a198}" 2026-04-14T14:16:19.519Z [INFO] server-d42b99f7-0b81-e875-302a-9210b379aed3: pipelining replication: peer="{Voter server-8ff6c772-28ab-e98c-e312-cf85e6d209a9 8ff6c772-28ab-e98c-e312-cf85e6d209a9}" 2026-04-14T14:16:19.726Z [WARN] server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198: heartbeat timeout reached, starting election: last-leader-addr=d42b99f7-0b81-e875-302a-9210b379aed3 last-leader-id=server-d42b99f7-0b81-e875-302a-9210b379aed3 2026-04-14T14:16:19.726Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:19.519858302 +0000 UTC m=+110.992044103 (1 nodes), 55.527812ms from start of poll, 55.827773ms from cluster start. Timeout at 2026-04-14 14:16:19.624866364 +0000 UTC m=+111.097052131, 105.008028ms after stability 2026-04-14T14:16:19.726Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:19.726Z [INFO] server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198: entering candidate state: node="Node at 51f3a200-7af1-7edc-ca4f-e8f60fd3a198 [Candidate]" term=3 2026-04-14T14:16:19.727Z [WARN] server-8ff6c772-28ab-e98c-e312-cf85e6d209a9: rejecting vote request since we have a leader: from=51f3a200-7af1-7edc-ca4f-e8f60fd3a198 leader=d42b99f7-0b81-e875-302a-9210b379aed3 leader-id=server-d42b99f7-0b81-e875-302a-9210b379aed3 2026-04-14T14:16:19.727Z [WARN] server-d42b99f7-0b81-e875-302a-9210b379aed3: rejecting vote request since we have a leader: from=51f3a200-7af1-7edc-ca4f-e8f60fd3a198 leader=d42b99f7-0b81-e875-302a-9210b379aed3 leader-id=server-d42b99f7-0b81-e875-302a-9210b379aed3 2026-04-14T14:16:19.735Z [ERROR] server-d42b99f7-0b81-e875-302a-9210b379aed3: peer has newer term, stopping replication: peer="{Voter server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198 51f3a200-7af1-7edc-ca4f-e8f60fd3a198}" 2026-04-14T14:16:19.735Z [INFO] server-d42b99f7-0b81-e875-302a-9210b379aed3: aborting pipeline replication: peer="{Voter server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198 51f3a200-7af1-7edc-ca4f-e8f60fd3a198}" 2026-04-14T14:16:19.735Z [INFO] server-d42b99f7-0b81-e875-302a-9210b379aed3: entering follower state: follower="Node at d42b99f7-0b81-e875-302a-9210b379aed3 [Follower]" leader-address= leader-id= 2026-04-14T14:16:19.735Z [INFO] server-d42b99f7-0b81-e875-302a-9210b379aed3: aborting pipeline replication: peer="{Voter server-8ff6c772-28ab-e98c-e312-cf85e6d209a9 8ff6c772-28ab-e98c-e312-cf85e6d209a9}" 2026-04-14T14:16:19.790Z [WARN] server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198: Election timeout reached, restarting election 2026-04-14T14:16:19.790Z [INFO] server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198: entering candidate state: node="Node at 51f3a200-7af1-7edc-ca4f-e8f60fd3a198 [Candidate]" term=4 2026-04-14T14:16:19.790Z [WARN] server-8ff6c772-28ab-e98c-e312-cf85e6d209a9: rejecting vote request since we have a leader: from=51f3a200-7af1-7edc-ca4f-e8f60fd3a198 leader=d42b99f7-0b81-e875-302a-9210b379aed3 leader-id=server-d42b99f7-0b81-e875-302a-9210b379aed3 2026-04-14T14:16:19.790Z [INFO] server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198: election won: term=4 tally=2 2026-04-14T14:16:19.790Z [INFO] server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198: entering leader state: leader="Node at 51f3a200-7af1-7edc-ca4f-e8f60fd3a198 [Leader]" 2026-04-14T14:16:19.790Z [INFO] server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198: added peer, starting replication: peer=server-8ff6c772-28ab-e98c-e312-cf85e6d209a9 2026-04-14T14:16:19.790Z [INFO] server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198: added peer, starting replication: peer=server-d42b99f7-0b81-e875-302a-9210b379aed3 2026-04-14T14:16:19.790Z [INFO] server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198: pipelining replication: peer="{Voter server-8ff6c772-28ab-e98c-e312-cf85e6d209a9 8ff6c772-28ab-e98c-e312-cf85e6d209a9}" 2026-04-14T14:16:19.790Z [INFO] server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198: pipelining replication: peer="{Voter server-d42b99f7-0b81-e875-302a-9210b379aed3 d42b99f7-0b81-e875-302a-9210b379aed3}" 2026-04-14T14:16:19.896Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:19.790621385 +0000 UTC m=+111.262807154 (2 nodes), 64.017876ms from start of poll, 326.590824ms from cluster start. Timeout at 2026-04-14 14:16:19.895623996 +0000 UTC m=+111.367809762, 105.002608ms after stability raft_test.go:2494: leadership transfer should err with: leadership transfer timeout 2026-04-14T14:16:19.941Z [INFO] server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198: aborting pipeline replication: peer="{Voter server-d42b99f7-0b81-e875-302a-9210b379aed3 d42b99f7-0b81-e875-302a-9210b379aed3}" 2026-04-14T14:16:19.941Z [INFO] server-51f3a200-7af1-7edc-ca4f-e8f60fd3a198: aborting pipeline replication: peer="{Voter server-8ff6c772-28ab-e98c-e312-cf85e6d209a9 8ff6c772-28ab-e98c-e312-cf85e6d209a9}" --- FAIL: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.48s) === RUN TestRaft_LeadershipTransferIgnoresNonvoters 2026-04-14T14:16:19.942Z [INFO] server-b391082c-9b88-c28a-c790-d1b63145448c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b391082c-9b88-c28a-c790-d1b63145448c Address:b391082c-9b88-c28a-c790-d1b63145448c} {Suffrage:Voter ID:server-97f3f0b3-a8e4-e7b7-f3e3-310f212c18d9 Address:97f3f0b3-a8e4-e7b7-f3e3-310f212c18d9}]" 2026-04-14T14:16:19.942Z [INFO] server-b391082c-9b88-c28a-c790-d1b63145448c: entering follower state: follower="Node at b391082c-9b88-c28a-c790-d1b63145448c [Follower]" leader-address= leader-id= 2026-04-14T14:16:19.942Z [INFO] server-97f3f0b3-a8e4-e7b7-f3e3-310f212c18d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b391082c-9b88-c28a-c790-d1b63145448c Address:b391082c-9b88-c28a-c790-d1b63145448c} {Suffrage:Voter ID:server-97f3f0b3-a8e4-e7b7-f3e3-310f212c18d9 Address:97f3f0b3-a8e4-e7b7-f3e3-310f212c18d9}]" 2026-04-14T14:16:19.942Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:19.942Z [INFO] server-97f3f0b3-a8e4-e7b7-f3e3-310f212c18d9: entering follower state: follower="Node at 97f3f0b3-a8e4-e7b7-f3e3-310f212c18d9 [Follower]" leader-address= leader-id= 2026-04-14T14:16:20.004Z [WARN] server-b391082c-9b88-c28a-c790-d1b63145448c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:20.004Z [INFO] server-b391082c-9b88-c28a-c790-d1b63145448c: entering candidate state: node="Node at b391082c-9b88-c28a-c790-d1b63145448c [Candidate]" term=2 2026-04-14T14:16:20.004Z [INFO] server-b391082c-9b88-c28a-c790-d1b63145448c: election won: term=2 tally=2 2026-04-14T14:16:20.004Z [INFO] server-b391082c-9b88-c28a-c790-d1b63145448c: entering leader state: leader="Node at b391082c-9b88-c28a-c790-d1b63145448c [Leader]" 2026-04-14T14:16:20.004Z [INFO] server-b391082c-9b88-c28a-c790-d1b63145448c: added peer, starting replication: peer=server-97f3f0b3-a8e4-e7b7-f3e3-310f212c18d9 2026-04-14T14:16:20.004Z [INFO] server-b391082c-9b88-c28a-c790-d1b63145448c: pipelining replication: peer="{Voter server-97f3f0b3-a8e4-e7b7-f3e3-310f212c18d9 97f3f0b3-a8e4-e7b7-f3e3-310f212c18d9}" 2026-04-14T14:16:20.109Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:20.004695839 +0000 UTC m=+111.476881608 (1 nodes), 61.99268ms from start of poll, 62.218716ms from cluster start. Timeout at 2026-04-14 14:16:20.109703682 +0000 UTC m=+111.581889445, 105.007837ms after stability 2026-04-14T14:16:20.109Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:20.215Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:20.109841982 +0000 UTC m=+111.582027747 (1 nodes), 1.817µs from start of poll, 167.364855ms from cluster start. Timeout at 2026-04-14 14:16:20.214866156 +0000 UTC m=+111.687051955, 105.024208ms after stability 2026-04-14T14:16:20.215Z [INFO] server-b391082c-9b88-c28a-c790-d1b63145448c: updating configuration: command=DemoteVoter server-id=server-97f3f0b3-a8e4-e7b7-f3e3-310f212c18d9 server-addr= servers="[{Suffrage:Voter ID:server-b391082c-9b88-c28a-c790-d1b63145448c Address:b391082c-9b88-c28a-c790-d1b63145448c} {Suffrage:Nonvoter ID:server-97f3f0b3-a8e4-e7b7-f3e3-310f212c18d9 Address:97f3f0b3-a8e4-e7b7-f3e3-310f212c18d9}]" 2026-04-14T14:16:20.215Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:20.321Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:20.215866877 +0000 UTC m=+111.688052645 (1 nodes), 678ns from start of poll, 273.389753ms from cluster start. Timeout at 2026-04-14 14:16:20.320885036 +0000 UTC m=+111.793070824, 105.018179ms after stability 2026-04-14T14:16:20.321Z [INFO] server-b391082c-9b88-c28a-c790-d1b63145448c: aborting pipeline replication: peer="{Voter server-97f3f0b3-a8e4-e7b7-f3e3-310f212c18d9 97f3f0b3-a8e4-e7b7-f3e3-310f212c18d9}" --- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.38s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap 2026-04-14T14:16:20.322Z [INFO] server-b2cbd3bb-db98-b486-7348-a7388ab83fd4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2cbd3bb-db98-b486-7348-a7388ab83fd4 Address:b2cbd3bb-db98-b486-7348-a7388ab83fd4} {Suffrage:Voter ID:server-dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21 Address:dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21}]" 2026-04-14T14:16:20.322Z [INFO] server-dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2cbd3bb-db98-b486-7348-a7388ab83fd4 Address:b2cbd3bb-db98-b486-7348-a7388ab83fd4} {Suffrage:Voter ID:server-dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21 Address:dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21}]" 2026-04-14T14:16:20.322Z [INFO] server-b2cbd3bb-db98-b486-7348-a7388ab83fd4: entering follower state: follower="Node at b2cbd3bb-db98-b486-7348-a7388ab83fd4 [Follower]" leader-address= leader-id= 2026-04-14T14:16:20.322Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:20.322Z [INFO] server-dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21: entering follower state: follower="Node at dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21 [Follower]" leader-address= leader-id= 2026-04-14T14:16:20.383Z [WARN] server-dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:20.383Z [INFO] server-dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21: entering candidate state: node="Node at dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21 [Candidate]" term=2 2026-04-14T14:16:20.383Z [INFO] server-dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21: election won: term=2 tally=2 2026-04-14T14:16:20.383Z [INFO] server-dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21: entering leader state: leader="Node at dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21 [Leader]" 2026-04-14T14:16:20.383Z [INFO] server-dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21: added peer, starting replication: peer=server-b2cbd3bb-db98-b486-7348-a7388ab83fd4 2026-04-14T14:16:20.383Z [INFO] server-dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21: pipelining replication: peer="{Voter server-b2cbd3bb-db98-b486-7348-a7388ab83fd4 b2cbd3bb-db98-b486-7348-a7388ab83fd4}" 2026-04-14T14:16:20.488Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:20.38318651 +0000 UTC m=+111.855372299 (1 nodes), 60.678178ms from start of poll, 60.874722ms from cluster start. Timeout at 2026-04-14 14:16:20.488199164 +0000 UTC m=+111.960384947, 105.012648ms after stability 2026-04-14T14:16:20.488Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:20.593Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:20.48852065 +0000 UTC m=+111.960706417 (1 nodes), 1.447µs from start of poll, 166.20884ms from cluster start. Timeout at 2026-04-14 14:16:20.593543556 +0000 UTC m=+112.065729342, 105.022925ms after stability 2026-04-14T14:16:20.605Z [INFO] server-dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2cbd3bb-db98-b486-7348-a7388ab83fd4 Address:b2cbd3bb-db98-b486-7348-a7388ab83fd4} {Suffrage:Voter ID:server-dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21 Address:dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21}]" 2026-04-14T14:16:20.605Z [INFO] server-dd7e4f67-bc3a-57c3-6b9c-c1e7835b4d21: aborting pipeline replication: peer="{Voter server-b2cbd3bb-db98-b486-7348-a7388ab83fd4 b2cbd3bb-db98-b486-7348-a7388ab83fd4}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.28s) === RUN TestRaft_LogStoreIsMonotonic 2026-04-14T14:16:20.606Z [INFO] server-723824cc-86a0-9936-7e16-f771376adf70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-723824cc-86a0-9936-7e16-f771376adf70 Address:723824cc-86a0-9936-7e16-f771376adf70}]" 2026-04-14T14:16:20.606Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:20.606Z [INFO] server-723824cc-86a0-9936-7e16-f771376adf70: entering follower state: follower="Node at 723824cc-86a0-9936-7e16-f771376adf70 [Follower]" leader-address= leader-id= 2026-04-14T14:16:20.706Z [WARN] server-723824cc-86a0-9936-7e16-f771376adf70: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:20.706Z [INFO] server-723824cc-86a0-9936-7e16-f771376adf70: entering candidate state: node="Node at 723824cc-86a0-9936-7e16-f771376adf70 [Candidate]" term=2 2026-04-14T14:16:20.706Z [INFO] server-723824cc-86a0-9936-7e16-f771376adf70: election won: term=2 tally=1 2026-04-14T14:16:20.706Z [INFO] server-723824cc-86a0-9936-7e16-f771376adf70: entering leader state: leader="Node at 723824cc-86a0-9936-7e16-f771376adf70 [Leader]" 2026-04-14T14:16:20.811Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:20.706129124 +0000 UTC m=+112.178314900 (1 nodes), 99.600907ms from start of poll, 99.726249ms from cluster start. Timeout at 2026-04-14 14:16:20.811130959 +0000 UTC m=+112.283316728, 105.001828ms after stability --- PASS: TestRaft_LogStoreIsMonotonic (0.21s) === RUN TestRaft_CacheLogWithStoreError 2026-04-14T14:16:20.812Z [INFO] server-b0f96c71-eb5b-4c28-7bd0-aec37ee85996: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b0f96c71-eb5b-4c28-7bd0-aec37ee85996 Address:b0f96c71-eb5b-4c28-7bd0-aec37ee85996} {Suffrage:Voter ID:server-6c034acc-8f0d-02b2-4f67-db6ec87ab4a8 Address:6c034acc-8f0d-02b2-4f67-db6ec87ab4a8}]" 2026-04-14T14:16:20.812Z [INFO] server-6c034acc-8f0d-02b2-4f67-db6ec87ab4a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b0f96c71-eb5b-4c28-7bd0-aec37ee85996 Address:b0f96c71-eb5b-4c28-7bd0-aec37ee85996} {Suffrage:Voter ID:server-6c034acc-8f0d-02b2-4f67-db6ec87ab4a8 Address:6c034acc-8f0d-02b2-4f67-db6ec87ab4a8}]" 2026-04-14T14:16:20.812Z [INFO] server-b0f96c71-eb5b-4c28-7bd0-aec37ee85996: entering follower state: follower="Node at b0f96c71-eb5b-4c28-7bd0-aec37ee85996 [Follower]" leader-address= leader-id= 2026-04-14T14:16:20.812Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:20.812Z [INFO] server-6c034acc-8f0d-02b2-4f67-db6ec87ab4a8: entering follower state: follower="Node at 6c034acc-8f0d-02b2-4f67-db6ec87ab4a8 [Follower]" leader-address= leader-id= 2026-04-14T14:16:20.891Z [WARN] server-b0f96c71-eb5b-4c28-7bd0-aec37ee85996: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:20.891Z [INFO] server-b0f96c71-eb5b-4c28-7bd0-aec37ee85996: entering candidate state: node="Node at b0f96c71-eb5b-4c28-7bd0-aec37ee85996 [Candidate]" term=2 2026-04-14T14:16:20.891Z [INFO] server-b0f96c71-eb5b-4c28-7bd0-aec37ee85996: election won: term=2 tally=2 2026-04-14T14:16:20.891Z [INFO] server-b0f96c71-eb5b-4c28-7bd0-aec37ee85996: entering leader state: leader="Node at b0f96c71-eb5b-4c28-7bd0-aec37ee85996 [Leader]" 2026-04-14T14:16:20.891Z [INFO] server-b0f96c71-eb5b-4c28-7bd0-aec37ee85996: added peer, starting replication: peer=server-6c034acc-8f0d-02b2-4f67-db6ec87ab4a8 2026-04-14T14:16:20.891Z [INFO] server-b0f96c71-eb5b-4c28-7bd0-aec37ee85996: pipelining replication: peer="{Voter server-6c034acc-8f0d-02b2-4f67-db6ec87ab4a8 6c034acc-8f0d-02b2-4f67-db6ec87ab4a8}" 2026-04-14T14:16:20.997Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:20.891908178 +0000 UTC m=+112.364093945 (1 nodes), 79.596072ms from start of poll, 79.812232ms from cluster start. Timeout at 2026-04-14 14:16:20.9969242 +0000 UTC m=+112.469109969, 105.016024ms after stability 2026-04-14T14:16:20.997Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:21.103Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:20.997727949 +0000 UTC m=+112.469913717 (1 nodes), 1.163µs from start of poll, 185.632004ms from cluster start. Timeout at 2026-04-14 14:16:21.102737294 +0000 UTC m=+112.574923065, 105.009348ms after stability 2026-04-14T14:16:21.103Z [ERROR] server-6c034acc-8f0d-02b2-4f67-db6ec87ab4a8: failed to append to logs: error="unable to store logs within log store, err: "some error"" 2026-04-14T14:16:21.104Z [WARN] server-6c034acc-8f0d-02b2-4f67-db6ec87ab4a8: failed to get previous log: previous-index=9 last-index=7 error="log not found" 2026-04-14T14:16:21.104Z [INFO] server-b0f96c71-eb5b-4c28-7bd0-aec37ee85996: aborting pipeline replication: peer="{Voter server-6c034acc-8f0d-02b2-4f67-db6ec87ab4a8 6c034acc-8f0d-02b2-4f67-db6ec87ab4a8}" 2026-04-14T14:16:21.113Z [INFO] server-b0f96c71-eb5b-4c28-7bd0-aec37ee85996: pipelining replication: peer="{Voter server-6c034acc-8f0d-02b2-4f67-db6ec87ab4a8 6c034acc-8f0d-02b2-4f67-db6ec87ab4a8}" 2026-04-14T14:16:21.114Z [INFO] server-6c034acc-8f0d-02b2-4f67-db6ec87ab4a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b0f96c71-eb5b-4c28-7bd0-aec37ee85996 Address:b0f96c71-eb5b-4c28-7bd0-aec37ee85996} {Suffrage:Voter ID:server-6c034acc-8f0d-02b2-4f67-db6ec87ab4a8 Address:6c034acc-8f0d-02b2-4f67-db6ec87ab4a8}]" 2026-04-14T14:16:21.114Z [INFO] server-b0f96c71-eb5b-4c28-7bd0-aec37ee85996: aborting pipeline replication: peer="{Voter server-6c034acc-8f0d-02b2-4f67-db6ec87ab4a8 6c034acc-8f0d-02b2-4f67-db6ec87ab4a8}" --- PASS: TestRaft_CacheLogWithStoreError (0.30s) === RUN TestRaft_ReloadConfig 2026-04-14T14:16:21.114Z [INFO] server-c2ec5f48-c2fb-4252-3e58-1db112d2b582: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2ec5f48-c2fb-4252-3e58-1db112d2b582 Address:c2ec5f48-c2fb-4252-3e58-1db112d2b582}]" 2026-04-14T14:16:21.114Z [INFO] server-c2ec5f48-c2fb-4252-3e58-1db112d2b582: entering follower state: follower="Node at c2ec5f48-c2fb-4252-3e58-1db112d2b582 [Shutdown]" leader-address= leader-id= --- PASS: TestRaft_ReloadConfig (0.00s) === RUN TestRaft_ReloadConfigValidates 2026-04-14T14:16:21.115Z [INFO] server-38800a06-1bfa-aa5d-b7df-1fe0605d0c4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38800a06-1bfa-aa5d-b7df-1fe0605d0c4b Address:38800a06-1bfa-aa5d-b7df-1fe0605d0c4b}]" 2026-04-14T14:16:21.115Z [INFO] server-38800a06-1bfa-aa5d-b7df-1fe0605d0c4b: entering follower state: follower="Node at 38800a06-1bfa-aa5d-b7df-1fe0605d0c4b [Shutdown]" leader-address= leader-id= --- PASS: TestRaft_ReloadConfigValidates (0.00s) === RUN TestRaft_InstallSnapshot_InvalidPeers 2026-04-14T14:16:21.115Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)" --- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s) === RUN TestRaft_VoteNotGranted_WhenNodeNotInCluster 2026-04-14T14:16:21.116Z [INFO] server-fc845d78-36be-3ac3-3069-13796f2a91b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc845d78-36be-3ac3-3069-13796f2a91b9 Address:fc845d78-36be-3ac3-3069-13796f2a91b9} {Suffrage:Voter ID:server-0534de4a-f46d-23bd-ee68-6738734bb65c Address:0534de4a-f46d-23bd-ee68-6738734bb65c} {Suffrage:Voter ID:server-2a9d5ce9-0394-1c81-79e4-55e22b654b62 Address:2a9d5ce9-0394-1c81-79e4-55e22b654b62}]" 2026-04-14T14:16:21.116Z [INFO] server-0534de4a-f46d-23bd-ee68-6738734bb65c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc845d78-36be-3ac3-3069-13796f2a91b9 Address:fc845d78-36be-3ac3-3069-13796f2a91b9} {Suffrage:Voter ID:server-0534de4a-f46d-23bd-ee68-6738734bb65c Address:0534de4a-f46d-23bd-ee68-6738734bb65c} {Suffrage:Voter ID:server-2a9d5ce9-0394-1c81-79e4-55e22b654b62 Address:2a9d5ce9-0394-1c81-79e4-55e22b654b62}]" 2026-04-14T14:16:21.116Z [INFO] server-fc845d78-36be-3ac3-3069-13796f2a91b9: entering follower state: follower="Node at fc845d78-36be-3ac3-3069-13796f2a91b9 [Follower]" leader-address= leader-id= 2026-04-14T14:16:21.116Z [INFO] server-0534de4a-f46d-23bd-ee68-6738734bb65c: entering follower state: follower="Node at 0534de4a-f46d-23bd-ee68-6738734bb65c [Follower]" leader-address= leader-id= 2026-04-14T14:16:21.116Z [INFO] server-2a9d5ce9-0394-1c81-79e4-55e22b654b62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc845d78-36be-3ac3-3069-13796f2a91b9 Address:fc845d78-36be-3ac3-3069-13796f2a91b9} {Suffrage:Voter ID:server-0534de4a-f46d-23bd-ee68-6738734bb65c Address:0534de4a-f46d-23bd-ee68-6738734bb65c} {Suffrage:Voter ID:server-2a9d5ce9-0394-1c81-79e4-55e22b654b62 Address:2a9d5ce9-0394-1c81-79e4-55e22b654b62}]" 2026-04-14T14:16:21.116Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:21.116Z [INFO] server-2a9d5ce9-0394-1c81-79e4-55e22b654b62: entering follower state: follower="Node at 2a9d5ce9-0394-1c81-79e4-55e22b654b62 [Follower]" leader-address= leader-id= 2026-04-14T14:16:21.192Z [WARN] server-fc845d78-36be-3ac3-3069-13796f2a91b9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:21.192Z [INFO] server-fc845d78-36be-3ac3-3069-13796f2a91b9: entering candidate state: node="Node at fc845d78-36be-3ac3-3069-13796f2a91b9 [Candidate]" term=2 2026-04-14T14:16:21.192Z [INFO] server-fc845d78-36be-3ac3-3069-13796f2a91b9: election won: term=2 tally=2 2026-04-14T14:16:21.192Z [INFO] server-fc845d78-36be-3ac3-3069-13796f2a91b9: entering leader state: leader="Node at fc845d78-36be-3ac3-3069-13796f2a91b9 [Leader]" 2026-04-14T14:16:21.192Z [INFO] server-fc845d78-36be-3ac3-3069-13796f2a91b9: added peer, starting replication: peer=server-0534de4a-f46d-23bd-ee68-6738734bb65c 2026-04-14T14:16:21.192Z [INFO] server-fc845d78-36be-3ac3-3069-13796f2a91b9: added peer, starting replication: peer=server-2a9d5ce9-0394-1c81-79e4-55e22b654b62 2026-04-14T14:16:21.192Z [INFO] server-fc845d78-36be-3ac3-3069-13796f2a91b9: pipelining replication: peer="{Voter server-2a9d5ce9-0394-1c81-79e4-55e22b654b62 2a9d5ce9-0394-1c81-79e4-55e22b654b62}" 2026-04-14T14:16:21.192Z [INFO] server-fc845d78-36be-3ac3-3069-13796f2a91b9: pipelining replication: peer="{Voter server-0534de4a-f46d-23bd-ee68-6738734bb65c 0534de4a-f46d-23bd-ee68-6738734bb65c}" 2026-04-14T14:16:21.298Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:21.192255837 +0000 UTC m=+112.664441618 (1 nodes), 75.842312ms from start of poll, 76.097805ms from cluster start. Timeout at 2026-04-14 14:16:21.297258933 +0000 UTC m=+112.769444701, 105.003083ms after stability 2026-04-14T14:16:21.303Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:21.409Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:21.303966551 +0000 UTC m=+112.776152316 (2 nodes), 1.067µs from start of poll, 187.808503ms from cluster start. Timeout at 2026-04-14 14:16:21.408986812 +0000 UTC m=+112.881172607, 105.020291ms after stability 2026-04-14T14:16:21.409Z [INFO] server-fc845d78-36be-3ac3-3069-13796f2a91b9: updating configuration: command=RemoveServer server-id=server-0534de4a-f46d-23bd-ee68-6738734bb65c server-addr= servers="[{Suffrage:Voter ID:server-fc845d78-36be-3ac3-3069-13796f2a91b9 Address:fc845d78-36be-3ac3-3069-13796f2a91b9} {Suffrage:Voter ID:server-2a9d5ce9-0394-1c81-79e4-55e22b654b62 Address:2a9d5ce9-0394-1c81-79e4-55e22b654b62}]" 2026-04-14T14:16:21.409Z [INFO] server-fc845d78-36be-3ac3-3069-13796f2a91b9: removed peer, stopping replication: peer=server-0534de4a-f46d-23bd-ee68-6738734bb65c last-index=3 2026-04-14T14:16:21.409Z [INFO] server-fc845d78-36be-3ac3-3069-13796f2a91b9: aborting pipeline replication: peer="{Voter server-0534de4a-f46d-23bd-ee68-6738734bb65c 0534de4a-f46d-23bd-ee68-6738734bb65c}" 2026-04-14T14:16:21.464Z [WARN] server-0534de4a-f46d-23bd-ee68-6738734bb65c: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2026-04-14T14:16:21.516Z [ERROR] server-fc845d78-36be-3ac3-3069-13796f2a91b9: failed to pipeline appendEntries: peer="{Voter server-2a9d5ce9-0394-1c81-79e4-55e22b654b62 2a9d5ce9-0394-1c81-79e4-55e22b654b62}" error="append pipeline closed" 2026-04-14T14:16:21.516Z [INFO] server-fc845d78-36be-3ac3-3069-13796f2a91b9: aborting pipeline replication: peer="{Voter server-2a9d5ce9-0394-1c81-79e4-55e22b654b62 2a9d5ce9-0394-1c81-79e4-55e22b654b62}" 2026-04-14T14:16:21.521Z [ERROR] server-fc845d78-36be-3ac3-3069-13796f2a91b9: failed to heartbeat to: peer=2a9d5ce9-0394-1c81-79e4-55e22b654b62 backoff time=10ms error="failed to connect to peer: 2a9d5ce9-0394-1c81-79e4-55e22b654b62" 2026-04-14T14:16:21.526Z [ERROR] server-fc845d78-36be-3ac3-3069-13796f2a91b9: failed to appendEntries to: peer="{Voter server-2a9d5ce9-0394-1c81-79e4-55e22b654b62 2a9d5ce9-0394-1c81-79e4-55e22b654b62}" error="failed to connect to peer: 2a9d5ce9-0394-1c81-79e4-55e22b654b62" 2026-04-14T14:16:21.537Z [ERROR] server-fc845d78-36be-3ac3-3069-13796f2a91b9: failed to heartbeat to: peer=2a9d5ce9-0394-1c81-79e4-55e22b654b62 backoff time=10ms error="failed to connect to peer: 2a9d5ce9-0394-1c81-79e4-55e22b654b62" 2026-04-14T14:16:21.543Z [ERROR] server-fc845d78-36be-3ac3-3069-13796f2a91b9: failed to appendEntries to: peer="{Voter server-2a9d5ce9-0394-1c81-79e4-55e22b654b62 2a9d5ce9-0394-1c81-79e4-55e22b654b62}" error="failed to connect to peer: 2a9d5ce9-0394-1c81-79e4-55e22b654b62" 2026-04-14T14:16:21.558Z [ERROR] server-fc845d78-36be-3ac3-3069-13796f2a91b9: failed to heartbeat to: peer=2a9d5ce9-0394-1c81-79e4-55e22b654b62 backoff time=10ms error="failed to connect to peer: 2a9d5ce9-0394-1c81-79e4-55e22b654b62" 2026-04-14T14:16:21.560Z [WARN] server-fc845d78-36be-3ac3-3069-13796f2a91b9: failed to contact: server-id=server-2a9d5ce9-0394-1c81-79e4-55e22b654b62 time=50.224016ms 2026-04-14T14:16:21.560Z [WARN] server-fc845d78-36be-3ac3-3069-13796f2a91b9: failed to contact quorum of nodes, stepping down 2026-04-14T14:16:21.560Z [INFO] server-fc845d78-36be-3ac3-3069-13796f2a91b9: entering follower state: follower="Node at fc845d78-36be-3ac3-3069-13796f2a91b9 [Follower]" leader-address= leader-id= 2026-04-14T14:16:21.563Z [ERROR] server-fc845d78-36be-3ac3-3069-13796f2a91b9: failed to appendEntries to: peer="{Voter server-2a9d5ce9-0394-1c81-79e4-55e22b654b62 2a9d5ce9-0394-1c81-79e4-55e22b654b62}" error="failed to connect to peer: 2a9d5ce9-0394-1c81-79e4-55e22b654b62" 2026-04-14T14:16:21.636Z [WARN] server-fc845d78-36be-3ac3-3069-13796f2a91b9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:21.636Z [INFO] server-fc845d78-36be-3ac3-3069-13796f2a91b9: entering candidate state: node="Node at fc845d78-36be-3ac3-3069-13796f2a91b9 [Candidate]" term=3 2026-04-14T14:16:21.637Z [ERROR] server-fc845d78-36be-3ac3-3069-13796f2a91b9: failed to make requestVote RPC: target="{Voter server-2a9d5ce9-0394-1c81-79e4-55e22b654b62 2a9d5ce9-0394-1c81-79e4-55e22b654b62}" error="failed to connect to peer: 2a9d5ce9-0394-1c81-79e4-55e22b654b62" term=3 2026-04-14T14:16:21.654Z [WARN] server-2a9d5ce9-0394-1c81-79e4-55e22b654b62: heartbeat timeout reached, starting election: last-leader-addr=fc845d78-36be-3ac3-3069-13796f2a91b9 last-leader-id=server-fc845d78-36be-3ac3-3069-13796f2a91b9 2026-04-14T14:16:21.654Z [INFO] server-2a9d5ce9-0394-1c81-79e4-55e22b654b62: entering candidate state: node="Node at 2a9d5ce9-0394-1c81-79e4-55e22b654b62 [Candidate]" term=3 2026-04-14T14:16:21.654Z [ERROR] server-2a9d5ce9-0394-1c81-79e4-55e22b654b62: failed to make requestVote RPC: target="{Voter server-fc845d78-36be-3ac3-3069-13796f2a91b9 fc845d78-36be-3ac3-3069-13796f2a91b9}" error="failed to connect to peer: fc845d78-36be-3ac3-3069-13796f2a91b9" term=3 2026-04-14T14:16:21.655Z [WARN] server-2a9d5ce9-0394-1c81-79e4-55e22b654b62: rejecting vote request since node is not in configuration: from=0534de4a-f46d-23bd-ee68-6738734bb65c --- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.54s) === RUN TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin 2026-04-14T14:16:21.656Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 Address:575c82a1-2244-4f0e-5b81-d4fc4ceb4e19} {Suffrage:Voter ID:server-e163f3c8-2108-22f6-24ba-8cd58306f916 Address:e163f3c8-2108-22f6-24ba-8cd58306f916} {Suffrage:Voter ID:server-900eb1fa-94c6-b810-5d13-851d0cacc467 Address:900eb1fa-94c6-b810-5d13-851d0cacc467}]" 2026-04-14T14:16:21.656Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: entering follower state: follower="Node at 575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 [Follower]" leader-address= leader-id= 2026-04-14T14:16:21.656Z [INFO] server-e163f3c8-2108-22f6-24ba-8cd58306f916: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 Address:575c82a1-2244-4f0e-5b81-d4fc4ceb4e19} {Suffrage:Voter ID:server-e163f3c8-2108-22f6-24ba-8cd58306f916 Address:e163f3c8-2108-22f6-24ba-8cd58306f916} {Suffrage:Voter ID:server-900eb1fa-94c6-b810-5d13-851d0cacc467 Address:900eb1fa-94c6-b810-5d13-851d0cacc467}]" 2026-04-14T14:16:21.656Z [INFO] server-900eb1fa-94c6-b810-5d13-851d0cacc467: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 Address:575c82a1-2244-4f0e-5b81-d4fc4ceb4e19} {Suffrage:Voter ID:server-e163f3c8-2108-22f6-24ba-8cd58306f916 Address:e163f3c8-2108-22f6-24ba-8cd58306f916} {Suffrage:Voter ID:server-900eb1fa-94c6-b810-5d13-851d0cacc467 Address:900eb1fa-94c6-b810-5d13-851d0cacc467}]" 2026-04-14T14:16:21.656Z [INFO] server-e163f3c8-2108-22f6-24ba-8cd58306f916: entering follower state: follower="Node at e163f3c8-2108-22f6-24ba-8cd58306f916 [Follower]" leader-address= leader-id= 2026-04-14T14:16:21.656Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:21.656Z [INFO] server-900eb1fa-94c6-b810-5d13-851d0cacc467: entering follower state: follower="Node at 900eb1fa-94c6-b810-5d13-851d0cacc467 [Follower]" leader-address= leader-id= 2026-04-14T14:16:21.708Z [WARN] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:21.708Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: entering candidate state: node="Node at 575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 [Candidate]" term=2 2026-04-14T14:16:21.708Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: election won: term=2 tally=2 2026-04-14T14:16:21.708Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: entering leader state: leader="Node at 575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 [Leader]" 2026-04-14T14:16:21.708Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: added peer, starting replication: peer=server-e163f3c8-2108-22f6-24ba-8cd58306f916 2026-04-14T14:16:21.708Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: added peer, starting replication: peer=server-900eb1fa-94c6-b810-5d13-851d0cacc467 2026-04-14T14:16:21.708Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: pipelining replication: peer="{Voter server-e163f3c8-2108-22f6-24ba-8cd58306f916 e163f3c8-2108-22f6-24ba-8cd58306f916}" 2026-04-14T14:16:21.708Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: pipelining replication: peer="{Voter server-900eb1fa-94c6-b810-5d13-851d0cacc467 900eb1fa-94c6-b810-5d13-851d0cacc467}" 2026-04-14T14:16:21.813Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:21.708541124 +0000 UTC m=+113.180726889 (1 nodes), 51.819512ms from start of poll, 52.084647ms from cluster start. Timeout at 2026-04-14 14:16:21.813542943 +0000 UTC m=+113.285728708, 105.001819ms after stability 2026-04-14T14:16:21.819Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:21.925Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:21.819818491 +0000 UTC m=+113.292004256 (2 nodes), 1.385µs from start of poll, 163.362014ms from cluster start. Timeout at 2026-04-14 14:16:21.924863052 +0000 UTC m=+113.397048828, 105.044572ms after stability 2026-04-14T14:16:21.929Z [ERROR] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: failed to pipeline appendEntries: peer="{Voter server-e163f3c8-2108-22f6-24ba-8cd58306f916 e163f3c8-2108-22f6-24ba-8cd58306f916}" error="append pipeline closed" 2026-04-14T14:16:21.929Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: aborting pipeline replication: peer="{Voter server-e163f3c8-2108-22f6-24ba-8cd58306f916 e163f3c8-2108-22f6-24ba-8cd58306f916}" 2026-04-14T14:16:21.931Z [ERROR] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: failed to heartbeat to: peer=e163f3c8-2108-22f6-24ba-8cd58306f916 backoff time=10ms error="failed to connect to peer: e163f3c8-2108-22f6-24ba-8cd58306f916" 2026-04-14T14:16:21.938Z [ERROR] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: failed to appendEntries to: peer="{Voter server-e163f3c8-2108-22f6-24ba-8cd58306f916 e163f3c8-2108-22f6-24ba-8cd58306f916}" error="failed to connect to peer: e163f3c8-2108-22f6-24ba-8cd58306f916" 2026-04-14T14:16:21.948Z [ERROR] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: failed to heartbeat to: peer=e163f3c8-2108-22f6-24ba-8cd58306f916 backoff time=10ms error="failed to connect to peer: e163f3c8-2108-22f6-24ba-8cd58306f916" 2026-04-14T14:16:21.957Z [ERROR] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: failed to appendEntries to: peer="{Voter server-e163f3c8-2108-22f6-24ba-8cd58306f916 e163f3c8-2108-22f6-24ba-8cd58306f916}" error="failed to connect to peer: e163f3c8-2108-22f6-24ba-8cd58306f916" 2026-04-14T14:16:21.965Z [ERROR] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: failed to heartbeat to: peer=e163f3c8-2108-22f6-24ba-8cd58306f916 backoff time=10ms error="failed to connect to peer: e163f3c8-2108-22f6-24ba-8cd58306f916" 2026-04-14T14:16:21.973Z [WARN] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: failed to contact: server-id=server-e163f3c8-2108-22f6-24ba-8cd58306f916 time=50.838117ms 2026-04-14T14:16:21.975Z [ERROR] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: failed to appendEntries to: peer="{Voter server-e163f3c8-2108-22f6-24ba-8cd58306f916 e163f3c8-2108-22f6-24ba-8cd58306f916}" error="failed to connect to peer: e163f3c8-2108-22f6-24ba-8cd58306f916" 2026-04-14T14:16:21.983Z [ERROR] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: failed to heartbeat to: peer=e163f3c8-2108-22f6-24ba-8cd58306f916 backoff time=20ms error="failed to connect to peer: e163f3c8-2108-22f6-24ba-8cd58306f916" 2026-04-14T14:16:22.004Z [ERROR] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: failed to appendEntries to: peer="{Voter server-e163f3c8-2108-22f6-24ba-8cd58306f916 e163f3c8-2108-22f6-24ba-8cd58306f916}" error="failed to connect to peer: e163f3c8-2108-22f6-24ba-8cd58306f916" 2026-04-14T14:16:22.008Z [WARN] server-e163f3c8-2108-22f6-24ba-8cd58306f916: heartbeat timeout reached, starting election: last-leader-addr=575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 last-leader-id=server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 2026-04-14T14:16:22.008Z [INFO] server-e163f3c8-2108-22f6-24ba-8cd58306f916: entering candidate state: node="Node at e163f3c8-2108-22f6-24ba-8cd58306f916 [Candidate]" term=3 2026-04-14T14:16:22.008Z [ERROR] server-e163f3c8-2108-22f6-24ba-8cd58306f916: failed to make requestVote RPC: target="{Voter server-900eb1fa-94c6-b810-5d13-851d0cacc467 900eb1fa-94c6-b810-5d13-851d0cacc467}" error="failed to connect to peer: 900eb1fa-94c6-b810-5d13-851d0cacc467" term=3 2026-04-14T14:16:22.008Z [ERROR] server-e163f3c8-2108-22f6-24ba-8cd58306f916: failed to make requestVote RPC: target="{Voter server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 575c82a1-2244-4f0e-5b81-d4fc4ceb4e19}" error="failed to connect to peer: 575c82a1-2244-4f0e-5b81-d4fc4ceb4e19" term=3 2026-04-14T14:16:22.009Z [ERROR] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: failed to heartbeat to: peer=e163f3c8-2108-22f6-24ba-8cd58306f916 backoff time=25ms error="failed to connect to peer: e163f3c8-2108-22f6-24ba-8cd58306f916" 2026-04-14T14:16:22.023Z [WARN] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: failed to contact: server-id=server-e163f3c8-2108-22f6-24ba-8cd58306f916 time=100.790526ms 2026-04-14T14:16:22.031Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: updating configuration: command=RemoveServer server-id=server-e163f3c8-2108-22f6-24ba-8cd58306f916 server-addr= servers="[{Suffrage:Voter ID:server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 Address:575c82a1-2244-4f0e-5b81-d4fc4ceb4e19} {Suffrage:Voter ID:server-900eb1fa-94c6-b810-5d13-851d0cacc467 Address:900eb1fa-94c6-b810-5d13-851d0cacc467}]" 2026-04-14T14:16:22.031Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: removed peer, stopping replication: peer=server-e163f3c8-2108-22f6-24ba-8cd58306f916 last-index=3 2026-04-14T14:16:22.031Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: updating configuration: command=AddNonvoter server-id=server-e163f3c8-2108-22f6-24ba-8cd58306f916 server-addr=e163f3c8-2108-22f6-24ba-8cd58306f916 servers="[{Suffrage:Voter ID:server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 Address:575c82a1-2244-4f0e-5b81-d4fc4ceb4e19} {Suffrage:Voter ID:server-900eb1fa-94c6-b810-5d13-851d0cacc467 Address:900eb1fa-94c6-b810-5d13-851d0cacc467} {Suffrage:Nonvoter ID:server-e163f3c8-2108-22f6-24ba-8cd58306f916 Address:e163f3c8-2108-22f6-24ba-8cd58306f916}]" 2026-04-14T14:16:22.031Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: added peer, starting replication: peer=server-e163f3c8-2108-22f6-24ba-8cd58306f916 2026-04-14T14:16:22.031Z [ERROR] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: failed to appendEntries to: peer="{Nonvoter server-e163f3c8-2108-22f6-24ba-8cd58306f916 e163f3c8-2108-22f6-24ba-8cd58306f916}" error="failed to connect to peer: e163f3c8-2108-22f6-24ba-8cd58306f916" 2026-04-14T14:16:22.047Z [ERROR] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: peer has newer term, stopping replication: peer="{Nonvoter server-e163f3c8-2108-22f6-24ba-8cd58306f916 e163f3c8-2108-22f6-24ba-8cd58306f916}" 2026-04-14T14:16:22.047Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: entering follower state: follower="Node at 575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 [Follower]" leader-address= leader-id= 2026-04-14T14:16:22.047Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: aborting pipeline replication: peer="{Voter server-900eb1fa-94c6-b810-5d13-851d0cacc467 900eb1fa-94c6-b810-5d13-851d0cacc467}" 2026-04-14T14:16:22.053Z [ERROR] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: peer has newer term, stopping replication: peer="{Voter server-e163f3c8-2108-22f6-24ba-8cd58306f916 e163f3c8-2108-22f6-24ba-8cd58306f916}" 2026-04-14T14:16:22.094Z [WARN] server-e163f3c8-2108-22f6-24ba-8cd58306f916: Election timeout reached, restarting election 2026-04-14T14:16:22.094Z [INFO] server-e163f3c8-2108-22f6-24ba-8cd58306f916: entering candidate state: node="Node at e163f3c8-2108-22f6-24ba-8cd58306f916 [Candidate]" term=23 2026-04-14T14:16:22.094Z [WARN] server-900eb1fa-94c6-b810-5d13-851d0cacc467: rejecting vote request since we have a leader: from=e163f3c8-2108-22f6-24ba-8cd58306f916 leader=575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 leader-id=server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 2026-04-14T14:16:22.094Z [WARN] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: rejecting vote request since node is not a voter: from=e163f3c8-2108-22f6-24ba-8cd58306f916 2026-04-14T14:16:22.110Z [WARN] server-900eb1fa-94c6-b810-5d13-851d0cacc467: heartbeat timeout reached, starting election: last-leader-addr=575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 last-leader-id=server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 2026-04-14T14:16:22.110Z [INFO] server-900eb1fa-94c6-b810-5d13-851d0cacc467: entering candidate state: node="Node at 900eb1fa-94c6-b810-5d13-851d0cacc467 [Candidate]" term=3 2026-04-14T14:16:22.110Z [INFO] server-900eb1fa-94c6-b810-5d13-851d0cacc467: entering follower state: follower="Node at 900eb1fa-94c6-b810-5d13-851d0cacc467 [Follower]" leader-address= leader-id= 2026-04-14T14:16:22.111Z [WARN] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:22.111Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: entering candidate state: node="Node at 575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 [Candidate]" term=24 2026-04-14T14:16:22.111Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: election won: term=24 tally=2 2026-04-14T14:16:22.111Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: entering leader state: leader="Node at 575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 [Leader]" 2026-04-14T14:16:22.111Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: added peer, starting replication: peer=server-900eb1fa-94c6-b810-5d13-851d0cacc467 2026-04-14T14:16:22.111Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: added peer, starting replication: peer=server-e163f3c8-2108-22f6-24ba-8cd58306f916 2026-04-14T14:16:22.111Z [WARN] server-e163f3c8-2108-22f6-24ba-8cd58306f916: failed to get previous log: previous-index=4 last-index=2 error="log not found" 2026-04-14T14:16:22.111Z [INFO] server-e163f3c8-2108-22f6-24ba-8cd58306f916: entering follower state: follower="Node at e163f3c8-2108-22f6-24ba-8cd58306f916 [Follower]" leader-address=575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 leader-id=server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 2026-04-14T14:16:22.111Z [WARN] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: appendEntries rejected, sending older logs: peer="{Nonvoter server-e163f3c8-2108-22f6-24ba-8cd58306f916 e163f3c8-2108-22f6-24ba-8cd58306f916}" next=3 2026-04-14T14:16:22.111Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: pipelining replication: peer="{Voter server-900eb1fa-94c6-b810-5d13-851d0cacc467 900eb1fa-94c6-b810-5d13-851d0cacc467}" 2026-04-14T14:16:22.111Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: pipelining replication: peer="{Nonvoter server-e163f3c8-2108-22f6-24ba-8cd58306f916 e163f3c8-2108-22f6-24ba-8cd58306f916}" 2026-04-14T14:16:22.137Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:22.243Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:22.137714263 +0000 UTC m=+113.609900031 (1 nodes), 43.16µs from start of poll, 481.257789ms from cluster start. Timeout at 2026-04-14 14:16:22.242755592 +0000 UTC m=+113.714941371, 105.04134ms after stability 2026-04-14T14:16:22.266Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: updating configuration: command=RemoveServer server-id=server-e163f3c8-2108-22f6-24ba-8cd58306f916 server-addr= servers="[{Suffrage:Voter ID:server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 Address:575c82a1-2244-4f0e-5b81-d4fc4ceb4e19} {Suffrage:Voter ID:server-900eb1fa-94c6-b810-5d13-851d0cacc467 Address:900eb1fa-94c6-b810-5d13-851d0cacc467}]" 2026-04-14T14:16:22.266Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: removed peer, stopping replication: peer=server-e163f3c8-2108-22f6-24ba-8cd58306f916 last-index=106 2026-04-14T14:16:22.267Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: aborting pipeline replication: peer="{Nonvoter server-e163f3c8-2108-22f6-24ba-8cd58306f916 e163f3c8-2108-22f6-24ba-8cd58306f916}" 2026-04-14T14:16:22.267Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: updating configuration: command=AddVoter server-id=server-e163f3c8-2108-22f6-24ba-8cd58306f916 server-addr=e163f3c8-2108-22f6-24ba-8cd58306f916 servers="[{Suffrage:Voter ID:server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19 Address:575c82a1-2244-4f0e-5b81-d4fc4ceb4e19} {Suffrage:Voter ID:server-900eb1fa-94c6-b810-5d13-851d0cacc467 Address:900eb1fa-94c6-b810-5d13-851d0cacc467} {Suffrage:Voter ID:server-e163f3c8-2108-22f6-24ba-8cd58306f916 Address:e163f3c8-2108-22f6-24ba-8cd58306f916}]" 2026-04-14T14:16:22.267Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: added peer, starting replication: peer=server-e163f3c8-2108-22f6-24ba-8cd58306f916 2026-04-14T14:16:22.267Z [WARN] server-e163f3c8-2108-22f6-24ba-8cd58306f916: failed to get previous log: previous-index=107 last-index=106 error="log not found" 2026-04-14T14:16:22.267Z [WARN] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: appendEntries rejected, sending older logs: peer="{Voter server-e163f3c8-2108-22f6-24ba-8cd58306f916 e163f3c8-2108-22f6-24ba-8cd58306f916}" next=107 2026-04-14T14:16:22.267Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: pipelining replication: peer="{Voter server-e163f3c8-2108-22f6-24ba-8cd58306f916 e163f3c8-2108-22f6-24ba-8cd58306f916}" 2026-04-14T14:16:23.336Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:23.441Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:23.336179561 +0000 UTC m=+114.808365325 (1 nodes), 1.085µs from start of poll, 1.679723083s from cluster start. Timeout at 2026-04-14 14:16:23.441210206 +0000 UTC m=+114.913395974, 105.030649ms after stability 2026-04-14T14:16:23.441Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: aborting pipeline replication: peer="{Voter server-e163f3c8-2108-22f6-24ba-8cd58306f916 e163f3c8-2108-22f6-24ba-8cd58306f916}" 2026-04-14T14:16:23.441Z [INFO] server-575c82a1-2244-4f0e-5b81-d4fc4ceb4e19: aborting pipeline replication: peer="{Voter server-900eb1fa-94c6-b810-5d13-851d0cacc467 900eb1fa-94c6-b810-5d13-851d0cacc467}" --- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.79s) === RUN TestRaft_FollowerRemovalNoElection 2026-04-14T14:16:23.445Z [INFO] server-1f41c186-9b67-5df6-5aea-46ff2eafbef6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f41c186-9b67-5df6-5aea-46ff2eafbef6 Address:1f41c186-9b67-5df6-5aea-46ff2eafbef6} {Suffrage:Voter ID:server-748442d6-a1f4-827c-30f7-6539dbb40e3b Address:748442d6-a1f4-827c-30f7-6539dbb40e3b} {Suffrage:Voter ID:server-b64f7613-b8c5-810d-b32c-4302c08efe58 Address:b64f7613-b8c5-810d-b32c-4302c08efe58}]" 2026-04-14T14:16:23.445Z [INFO] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f41c186-9b67-5df6-5aea-46ff2eafbef6 Address:1f41c186-9b67-5df6-5aea-46ff2eafbef6} {Suffrage:Voter ID:server-748442d6-a1f4-827c-30f7-6539dbb40e3b Address:748442d6-a1f4-827c-30f7-6539dbb40e3b} {Suffrage:Voter ID:server-b64f7613-b8c5-810d-b32c-4302c08efe58 Address:b64f7613-b8c5-810d-b32c-4302c08efe58}]" 2026-04-14T14:16:23.445Z [INFO] server-1f41c186-9b67-5df6-5aea-46ff2eafbef6: entering follower state: follower="Node at 1f41c186-9b67-5df6-5aea-46ff2eafbef6 [Follower]" leader-address= leader-id= 2026-04-14T14:16:23.445Z [INFO] server-b64f7613-b8c5-810d-b32c-4302c08efe58: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f41c186-9b67-5df6-5aea-46ff2eafbef6 Address:1f41c186-9b67-5df6-5aea-46ff2eafbef6} {Suffrage:Voter ID:server-748442d6-a1f4-827c-30f7-6539dbb40e3b Address:748442d6-a1f4-827c-30f7-6539dbb40e3b} {Suffrage:Voter ID:server-b64f7613-b8c5-810d-b32c-4302c08efe58 Address:b64f7613-b8c5-810d-b32c-4302c08efe58}]" 2026-04-14T14:16:23.445Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:23.445Z [INFO] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: entering follower state: follower="Node at 748442d6-a1f4-827c-30f7-6539dbb40e3b [Follower]" leader-address= leader-id= 2026-04-14T14:16:23.445Z [INFO] server-b64f7613-b8c5-810d-b32c-4302c08efe58: entering follower state: follower="Node at b64f7613-b8c5-810d-b32c-4302c08efe58 [Follower]" leader-address= leader-id= 2026-04-14T14:16:23.550Z [WARN] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:23.550Z [INFO] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: entering candidate state: node="Node at 748442d6-a1f4-827c-30f7-6539dbb40e3b [Candidate]" term=2 2026-04-14T14:16:23.550Z [INFO] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: election won: term=2 tally=2 2026-04-14T14:16:23.550Z [INFO] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: entering leader state: leader="Node at 748442d6-a1f4-827c-30f7-6539dbb40e3b [Leader]" 2026-04-14T14:16:23.550Z [INFO] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: added peer, starting replication: peer=server-1f41c186-9b67-5df6-5aea-46ff2eafbef6 2026-04-14T14:16:23.550Z [INFO] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: added peer, starting replication: peer=server-b64f7613-b8c5-810d-b32c-4302c08efe58 2026-04-14T14:16:23.550Z [INFO] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: pipelining replication: peer="{Voter server-b64f7613-b8c5-810d-b32c-4302c08efe58 b64f7613-b8c5-810d-b32c-4302c08efe58}" 2026-04-14T14:16:23.550Z [INFO] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: pipelining replication: peer="{Voter server-1f41c186-9b67-5df6-5aea-46ff2eafbef6 1f41c186-9b67-5df6-5aea-46ff2eafbef6}" 2026-04-14T14:16:23.756Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:23.550813568 +0000 UTC m=+115.022999337 (1 nodes), 105.264769ms from start of poll, 105.534739ms from cluster start. Timeout at 2026-04-14 14:16:23.755817015 +0000 UTC m=+115.228002784, 205.003447ms after stability 2026-04-14T14:16:23.756Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:23.961Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:23.756572145 +0000 UTC m=+115.228757912 (1 nodes), 1.965µs from start of poll, 311.293314ms from cluster start. Timeout at 2026-04-14 14:16:23.961589724 +0000 UTC m=+115.433775530, 205.017618ms after stability 2026-04-14T14:16:23.967Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:24.173Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:23.967760017 +0000 UTC m=+115.439945781 (2 nodes), 2.442µs from start of poll, 522.481183ms from cluster start. Timeout at 2026-04-14 14:16:24.17278871 +0000 UTC m=+115.644974479, 205.028698ms after stability 2026-04-14T14:16:24.173Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:24.378Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:24.173062484 +0000 UTC m=+115.645248251 (1 nodes), 1.396µs from start of poll, 727.783653ms from cluster start. Timeout at 2026-04-14 14:16:24.3780891 +0000 UTC m=+115.850274886, 205.026635ms after stability raft_test.go:2961: [INFO] restarting Node at 1f41c186-9b67-5df6-5aea-46ff2eafbef6 [Follower] 2026-04-14T14:16:24.424Z [WARN] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: failed to contact: server-id=server-1f41c186-9b67-5df6-5aea-46ff2eafbef6 time=50.02849ms 2026-04-14T14:16:24.472Z [WARN] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: failed to contact: server-id=server-1f41c186-9b67-5df6-5aea-46ff2eafbef6 time=97.967723ms 2026-04-14T14:16:24.484Z [INFO] server-1f41c186-9b67-5df6-5aea-46ff2eafbef6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f41c186-9b67-5df6-5aea-46ff2eafbef6 Address:1f41c186-9b67-5df6-5aea-46ff2eafbef6} {Suffrage:Voter ID:server-748442d6-a1f4-827c-30f7-6539dbb40e3b Address:748442d6-a1f4-827c-30f7-6539dbb40e3b} {Suffrage:Voter ID:server-b64f7613-b8c5-810d-b32c-4302c08efe58 Address:b64f7613-b8c5-810d-b32c-4302c08efe58}]" 2026-04-14T14:16:24.484Z [INFO] server-1f41c186-9b67-5df6-5aea-46ff2eafbef6: entering follower state: follower="Node at 1f41c186-9b67-5df6-5aea-46ff2eafbef6 [Follower]" leader-address= leader-id= 2026-04-14T14:16:24.513Z [WARN] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: failed to contact: server-id=server-1f41c186-9b67-5df6-5aea-46ff2eafbef6 time=138.923995ms 2026-04-14T14:16:24.586Z [WARN] server-1f41c186-9b67-5df6-5aea-46ff2eafbef6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:24.587Z [INFO] server-1f41c186-9b67-5df6-5aea-46ff2eafbef6: entering candidate state: node="Node at 1f41c186-9b67-5df6-5aea-46ff2eafbef6 [Candidate]" term=3 2026-04-14T14:16:24.587Z [WARN] server-b64f7613-b8c5-810d-b32c-4302c08efe58: rejecting vote request since we have a leader: from=1f41c186-9b67-5df6-5aea-46ff2eafbef6 leader=748442d6-a1f4-827c-30f7-6539dbb40e3b leader-id=server-748442d6-a1f4-827c-30f7-6539dbb40e3b 2026-04-14T14:16:24.587Z [WARN] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: rejecting vote request since we have a leader: from=1f41c186-9b67-5df6-5aea-46ff2eafbef6 leader=748442d6-a1f4-827c-30f7-6539dbb40e3b leader-id=server-748442d6-a1f4-827c-30f7-6539dbb40e3b 2026-04-14T14:16:24.700Z [WARN] server-1f41c186-9b67-5df6-5aea-46ff2eafbef6: Election timeout reached, restarting election 2026-04-14T14:16:24.700Z [INFO] server-1f41c186-9b67-5df6-5aea-46ff2eafbef6: entering candidate state: node="Node at 1f41c186-9b67-5df6-5aea-46ff2eafbef6 [Candidate]" term=4 2026-04-14T14:16:24.700Z [WARN] server-b64f7613-b8c5-810d-b32c-4302c08efe58: rejecting vote request since we have a leader: from=1f41c186-9b67-5df6-5aea-46ff2eafbef6 leader=748442d6-a1f4-827c-30f7-6539dbb40e3b leader-id=server-748442d6-a1f4-827c-30f7-6539dbb40e3b 2026-04-14T14:16:24.700Z [WARN] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: rejecting vote request since we have a leader: from=1f41c186-9b67-5df6-5aea-46ff2eafbef6 leader=748442d6-a1f4-827c-30f7-6539dbb40e3b leader-id=server-748442d6-a1f4-827c-30f7-6539dbb40e3b 2026-04-14T14:16:24.735Z [ERROR] cluster: peer sees incorrect leader: peer="Node at 1f41c186-9b67-5df6-5aea-46ff2eafbef6 [Candidate]" leader=[none] expected-leader=748442d6-a1f4-827c-30f7-6539dbb40e3b testing.go:604: at least one peer has the wrong notion of leader 2026-04-14T14:16:24.735Z [INFO] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: aborting pipeline replication: peer="{Voter server-b64f7613-b8c5-810d-b32c-4302c08efe58 b64f7613-b8c5-810d-b32c-4302c08efe58}" 2026-04-14T14:16:24.888Z [ERROR] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: failed to heartbeat to: peer=1f41c186-9b67-5df6-5aea-46ff2eafbef6 backoff time=10ms error="command timed out" 2026-04-14T14:16:25.008Z [ERROR] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: failed to pipeline appendEntries: peer="{Voter server-1f41c186-9b67-5df6-5aea-46ff2eafbef6 1f41c186-9b67-5df6-5aea-46ff2eafbef6}" error="command enqueue timeout" 2026-04-14T14:16:25.008Z [INFO] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: aborting pipeline replication: peer="{Voter server-1f41c186-9b67-5df6-5aea-46ff2eafbef6 1f41c186-9b67-5df6-5aea-46ff2eafbef6}" 2026-04-14T14:16:25.416Z [ERROR] server-748442d6-a1f4-827c-30f7-6539dbb40e3b: failed to heartbeat to: peer=1f41c186-9b67-5df6-5aea-46ff2eafbef6 backoff time=10ms error="command timed out" --- FAIL: TestRaft_FollowerRemovalNoElection (2.12s) === RUN TestRaft_VoteWithNoIDNoAddr 2026-04-14T14:16:25.565Z [INFO] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e Address:7a6fd403-8d66-0760-a86e-dd5dddd1cf7e} {Suffrage:Voter ID:server-b0373472-129a-4a7f-b638-75a85d7d9af7 Address:b0373472-129a-4a7f-b638-75a85d7d9af7} {Suffrage:Voter ID:server-3d2d8172-3bfb-bd0b-a6b7-59689dd3de47 Address:3d2d8172-3bfb-bd0b-a6b7-59689dd3de47}]" 2026-04-14T14:16:25.565Z [INFO] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: entering follower state: follower="Node at 7a6fd403-8d66-0760-a86e-dd5dddd1cf7e [Follower]" leader-address= leader-id= 2026-04-14T14:16:25.565Z [INFO] server-b0373472-129a-4a7f-b638-75a85d7d9af7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e Address:7a6fd403-8d66-0760-a86e-dd5dddd1cf7e} {Suffrage:Voter ID:server-b0373472-129a-4a7f-b638-75a85d7d9af7 Address:b0373472-129a-4a7f-b638-75a85d7d9af7} {Suffrage:Voter ID:server-3d2d8172-3bfb-bd0b-a6b7-59689dd3de47 Address:3d2d8172-3bfb-bd0b-a6b7-59689dd3de47}]" 2026-04-14T14:16:25.565Z [INFO] server-3d2d8172-3bfb-bd0b-a6b7-59689dd3de47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e Address:7a6fd403-8d66-0760-a86e-dd5dddd1cf7e} {Suffrage:Voter ID:server-b0373472-129a-4a7f-b638-75a85d7d9af7 Address:b0373472-129a-4a7f-b638-75a85d7d9af7} {Suffrage:Voter ID:server-3d2d8172-3bfb-bd0b-a6b7-59689dd3de47 Address:3d2d8172-3bfb-bd0b-a6b7-59689dd3de47}]" 2026-04-14T14:16:25.565Z [INFO] server-b0373472-129a-4a7f-b638-75a85d7d9af7: entering follower state: follower="Node at b0373472-129a-4a7f-b638-75a85d7d9af7 [Follower]" leader-address= leader-id= 2026-04-14T14:16:25.565Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:25.565Z [INFO] server-3d2d8172-3bfb-bd0b-a6b7-59689dd3de47: entering follower state: follower="Node at 3d2d8172-3bfb-bd0b-a6b7-59689dd3de47 [Follower]" leader-address= leader-id= 2026-04-14T14:16:25.617Z [WARN] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-14T14:16:25.617Z [INFO] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: entering candidate state: node="Node at 7a6fd403-8d66-0760-a86e-dd5dddd1cf7e [Candidate]" term=2 2026-04-14T14:16:25.617Z [INFO] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: election won: term=2 tally=2 2026-04-14T14:16:25.618Z [INFO] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: entering leader state: leader="Node at 7a6fd403-8d66-0760-a86e-dd5dddd1cf7e [Leader]" 2026-04-14T14:16:25.618Z [INFO] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: added peer, starting replication: peer=server-b0373472-129a-4a7f-b638-75a85d7d9af7 2026-04-14T14:16:25.618Z [INFO] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: added peer, starting replication: peer=server-3d2d8172-3bfb-bd0b-a6b7-59689dd3de47 2026-04-14T14:16:25.618Z [INFO] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: pipelining replication: peer="{Voter server-b0373472-129a-4a7f-b638-75a85d7d9af7 b0373472-129a-4a7f-b638-75a85d7d9af7}" 2026-04-14T14:16:25.618Z [INFO] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: pipelining replication: peer="{Voter server-3d2d8172-3bfb-bd0b-a6b7-59689dd3de47 3d2d8172-3bfb-bd0b-a6b7-59689dd3de47}" 2026-04-14T14:16:25.723Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:25.618033582 +0000 UTC m=+117.090219350 (1 nodes), 52.216897ms from start of poll, 52.502775ms from cluster start. Timeout at 2026-04-14 14:16:25.723034879 +0000 UTC m=+117.195220651, 105.001301ms after stability 2026-04-14T14:16:25.723Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-14T14:16:25.829Z [INFO] cluster: stable state for Leader reached at 2026-04-14 14:16:25.723964026 +0000 UTC m=+117.196149799 (1 nodes), 2.213µs from start of poll, 158.433224ms from cluster start. Timeout at 2026-04-14 14:16:25.828991903 +0000 UTC m=+117.301177667, 105.027868ms after stability 2026-04-14T14:16:25.835Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-14T14:16:25.941Z [INFO] cluster: stable state for Follower reached at 2026-04-14 14:16:25.835646689 +0000 UTC m=+117.307832465 (2 nodes), 3.406µs from start of poll, 270.11589ms from cluster start. Timeout at 2026-04-14 14:16:25.94066412 +0000 UTC m=+117.412849889, 105.017424ms after stability 2026-04-14T14:16:25.942Z [ERROR] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: failed to heartbeat to: peer=3d2d8172-3bfb-bd0b-a6b7-59689dd3de47 backoff time=10ms error="failed to connect to peer: 3d2d8172-3bfb-bd0b-a6b7-59689dd3de47" 2026-04-14T14:16:25.945Z [ERROR] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: failed to pipeline appendEntries: peer="{Voter server-3d2d8172-3bfb-bd0b-a6b7-59689dd3de47 3d2d8172-3bfb-bd0b-a6b7-59689dd3de47}" error="append pipeline closed" 2026-04-14T14:16:25.945Z [INFO] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: aborting pipeline replication: peer="{Voter server-3d2d8172-3bfb-bd0b-a6b7-59689dd3de47 3d2d8172-3bfb-bd0b-a6b7-59689dd3de47}" 2026-04-14T14:16:25.945Z [ERROR] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: failed to heartbeat to: peer=b0373472-129a-4a7f-b638-75a85d7d9af7 backoff time=10ms error="failed to connect to peer: b0373472-129a-4a7f-b638-75a85d7d9af7" 2026-04-14T14:16:25.948Z [ERROR] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: failed to pipeline appendEntries: peer="{Voter server-b0373472-129a-4a7f-b638-75a85d7d9af7 b0373472-129a-4a7f-b638-75a85d7d9af7}" error="append pipeline closed" 2026-04-14T14:16:25.948Z [INFO] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: aborting pipeline replication: peer="{Voter server-b0373472-129a-4a7f-b638-75a85d7d9af7 b0373472-129a-4a7f-b638-75a85d7d9af7}" 2026-04-14T14:16:25.953Z [ERROR] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: failed to appendEntries to: peer="{Voter server-3d2d8172-3bfb-bd0b-a6b7-59689dd3de47 3d2d8172-3bfb-bd0b-a6b7-59689dd3de47}" error="failed to connect to peer: 3d2d8172-3bfb-bd0b-a6b7-59689dd3de47" 2026-04-14T14:16:25.956Z [ERROR] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: failed to appendEntries to: peer="{Voter server-b0373472-129a-4a7f-b638-75a85d7d9af7 b0373472-129a-4a7f-b638-75a85d7d9af7}" error="failed to connect to peer: b0373472-129a-4a7f-b638-75a85d7d9af7" 2026-04-14T14:16:25.960Z [ERROR] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: failed to heartbeat to: peer=3d2d8172-3bfb-bd0b-a6b7-59689dd3de47 backoff time=10ms error="failed to connect to peer: 3d2d8172-3bfb-bd0b-a6b7-59689dd3de47" 2026-04-14T14:16:25.963Z [ERROR] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: failed to heartbeat to: peer=b0373472-129a-4a7f-b638-75a85d7d9af7 backoff time=10ms error="failed to connect to peer: b0373472-129a-4a7f-b638-75a85d7d9af7" 2026-04-14T14:16:25.970Z [ERROR] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: failed to appendEntries to: peer="{Voter server-3d2d8172-3bfb-bd0b-a6b7-59689dd3de47 3d2d8172-3bfb-bd0b-a6b7-59689dd3de47}" error="failed to connect to peer: 3d2d8172-3bfb-bd0b-a6b7-59689dd3de47" 2026-04-14T14:16:25.975Z [ERROR] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: failed to appendEntries to: peer="{Voter server-b0373472-129a-4a7f-b638-75a85d7d9af7 b0373472-129a-4a7f-b638-75a85d7d9af7}" error="failed to connect to peer: b0373472-129a-4a7f-b638-75a85d7d9af7" 2026-04-14T14:16:25.979Z [ERROR] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: failed to heartbeat to: peer=3d2d8172-3bfb-bd0b-a6b7-59689dd3de47 backoff time=10ms error="failed to connect to peer: 3d2d8172-3bfb-bd0b-a6b7-59689dd3de47" 2026-04-14T14:16:25.983Z [ERROR] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: failed to heartbeat to: peer=b0373472-129a-4a7f-b638-75a85d7d9af7 backoff time=10ms error="failed to connect to peer: b0373472-129a-4a7f-b638-75a85d7d9af7" 2026-04-14T14:16:25.990Z [WARN] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: failed to contact: server-id=server-b0373472-129a-4a7f-b638-75a85d7d9af7 time=50.937789ms 2026-04-14T14:16:25.990Z [WARN] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: failed to contact: server-id=server-3d2d8172-3bfb-bd0b-a6b7-59689dd3de47 time=50.978923ms 2026-04-14T14:16:25.990Z [WARN] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: failed to contact quorum of nodes, stepping down 2026-04-14T14:16:25.990Z [INFO] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: entering follower state: follower="Node at 7a6fd403-8d66-0760-a86e-dd5dddd1cf7e [Follower]" leader-address= leader-id= 2026-04-14T14:16:25.990Z [ERROR] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: failed to appendEntries to: peer="{Voter server-3d2d8172-3bfb-bd0b-a6b7-59689dd3de47 3d2d8172-3bfb-bd0b-a6b7-59689dd3de47}" error="failed to connect to peer: 3d2d8172-3bfb-bd0b-a6b7-59689dd3de47" 2026-04-14T14:16:25.991Z [WARN] server-b0373472-129a-4a7f-b638-75a85d7d9af7: heartbeat timeout reached, starting election: last-leader-addr=7a6fd403-8d66-0760-a86e-dd5dddd1cf7e last-leader-id=server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e 2026-04-14T14:16:25.991Z [INFO] server-b0373472-129a-4a7f-b638-75a85d7d9af7: entering candidate state: node="Node at b0373472-129a-4a7f-b638-75a85d7d9af7 [Candidate]" term=3 2026-04-14T14:16:25.991Z [WARN] server-3d2d8172-3bfb-bd0b-a6b7-59689dd3de47: rejecting vote request since we have a leader: from=b0373472-129a-4a7f-b638-75a85d7d9af7 leader=7a6fd403-8d66-0760-a86e-dd5dddd1cf7e leader-id=server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e 2026-04-14T14:16:25.991Z [ERROR] server-b0373472-129a-4a7f-b638-75a85d7d9af7: failed to make requestVote RPC: target="{Voter server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e 7a6fd403-8d66-0760-a86e-dd5dddd1cf7e}" error="failed to connect to peer: 7a6fd403-8d66-0760-a86e-dd5dddd1cf7e" term=3 2026-04-14T14:16:25.995Z [ERROR] server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e: failed to appendEntries to: peer="{Voter server-b0373472-129a-4a7f-b638-75a85d7d9af7 b0373472-129a-4a7f-b638-75a85d7d9af7}" error="failed to connect to peer: b0373472-129a-4a7f-b638-75a85d7d9af7" 2026-04-14T14:16:26.039Z [WARN] server-3d2d8172-3bfb-bd0b-a6b7-59689dd3de47: heartbeat timeout reached, starting election: last-leader-addr=7a6fd403-8d66-0760-a86e-dd5dddd1cf7e last-leader-id=server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e 2026-04-14T14:16:26.039Z [INFO] server-3d2d8172-3bfb-bd0b-a6b7-59689dd3de47: entering candidate state: node="Node at 3d2d8172-3bfb-bd0b-a6b7-59689dd3de47 [Candidate]" term=3 2026-04-14T14:16:26.039Z [ERROR] server-3d2d8172-3bfb-bd0b-a6b7-59689dd3de47: failed to make requestVote RPC: target="{Voter server-7a6fd403-8d66-0760-a86e-dd5dddd1cf7e 7a6fd403-8d66-0760-a86e-dd5dddd1cf7e}" error="failed to connect to peer: 7a6fd403-8d66-0760-a86e-dd5dddd1cf7e" term=3 2026-04-14T14:16:26.039Z [INFO] server-b0373472-129a-4a7f-b638-75a85d7d9af7: duplicate requestVote for same term: term=3 2026-04-14T14:16:26.047Z [INFO] server-b0373472-129a-4a7f-b638-75a85d7d9af7: entering follower state: follower="Node at b0373472-129a-4a7f-b638-75a85d7d9af7 [Follower]" leader-address= leader-id= --- PASS: TestRaft_VoteWithNoIDNoAddr (0.58s) === RUN TestRaft_runFollower_State_Transition === RUN TestRaft_runFollower_State_Transition/NonVoter 2026-04-14T14:16:26.142Z [INFO] 127.0.0.1:39103 :: starting node: addr=127.0.0.1:39103 2026-04-14T14:16:26.143Z [INFO] 127.0.0.1:39103 :: initial configuration: index=0 servers=[] 2026-04-14T14:16:26.143Z [INFO] 127.0.0.1:39103 :: entering follower state: follower="Node at 127.0.0.1:39103 [Follower]" leader-address= leader-id= 2026-04-14T14:16:26.214Z [WARN] 127.0.0.1:39103 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election === RUN TestRaft_runFollower_State_Transition/Voter 2026-04-14T14:16:26.448Z [INFO] 127.0.0.1:37315 :: starting node: addr=127.0.0.1:37315 2026-04-14T14:16:26.448Z [INFO] 127.0.0.1:37315 :: initial configuration: index=0 servers=[] 2026-04-14T14:16:26.448Z [INFO] 127.0.0.1:37315 :: entering follower state: follower="Node at 127.0.0.1:37315 [Follower]" leader-address= leader-id= 2026-04-14T14:16:26.499Z [WARN] 127.0.0.1:37315 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= === RUN TestRaft_runFollower_State_Transition/Not_in_Config 2026-04-14T14:16:26.599Z [INFO] 127.0.0.1:42585 :: starting node: addr=127.0.0.1:42585 2026-04-14T14:16:26.600Z [INFO] 127.0.0.1:42585 :: initial configuration: index=0 servers=[] 2026-04-14T14:16:26.600Z [INFO] 127.0.0.1:42585 :: entering follower state: follower="Node at 127.0.0.1:42585 [Follower]" leader-address= leader-id= 2026-04-14T14:16:26.672Z [WARN] 127.0.0.1:42585 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election --- PASS: TestRaft_runFollower_State_Transition (0.61s) --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Voter (0.31s) --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s) === RUN TestRaft_runFollower_ReloadTimeoutConfigs 2026-04-14T14:16:26.751Z [INFO] 127.0.0.1:43123 :: starting node: addr=127.0.0.1:43123 2026-04-14T14:16:26.751Z [INFO] 127.0.0.1:43123 :: initial configuration: index=0 servers=[] 2026-04-14T14:16:26.751Z [INFO] 127.0.0.1:43123 :: entering follower state: follower="Node at 127.0.0.1:43123 [Follower]" leader-address= leader-id= 2026-04-14T14:16:26.751Z [WARN] 127.0.0.1:43123 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= --- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s) === RUN TestSaturationMetric === RUN TestSaturationMetric/without_smoothing --- PASS: TestSaturationMetric (0.00s) --- PASS: TestSaturationMetric/without_smoothing (0.00s) === RUN TestSaturationMetric_IncorrectUsage === RUN TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_first --- PASS: TestSaturationMetric_IncorrectUsage (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_EmptyAddr --- PASS: TestTCPTransport_EmptyAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) === RUN TestOverrideNotifyBool --- PASS: TestOverrideNotifyBool (0.00s) FAIL FAIL github.com/hashicorp/raft 118.386s ? github.com/hashicorp/raft/bench [no test files] FAIL dh_auto_test: error: cd obj-x86_64-linux-gnu && go test -vet=off -v -p 42 github.com/hashicorp/raft github.com/hashicorp/raft/bench returned exit code 1 make[1]: *** [debian/rules:17: override_dh_auto_test] Error 25 make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' make: *** [debian/rules:14: binary] Error 2 dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2 I: copying local configuration E: Failed autobuilding of package I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/946523 and its subdirectories Wed Mar 12 07:53:34 UTC 2025 W: No second build log, what happened?