I: pbuilder: network access will be disabled during build I: Current time: Sun Sep 14 09:45:40 -12 2025 I: pbuilder-time-stamp: 1757886340 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/forky-reproducible-base.tgz] I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: Copying source file I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc] I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz] I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz] I: Extracting source dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: no acceptable signature found dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying 0001-Skip-unreliable-tests.patch I: Not using root during the build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/3199738/tmp/hooks/D02_print_environment starting I: set BUILDDIR='/build/reproducible-path' BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' BUILDUSERNAME='pbuilder1' BUILD_ARCH='amd64' DEBIAN_FRONTEND='noninteractive' DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=40 ' DISTRIBUTION='forky' HOME='/root' HOST_ARCH='amd64' IFS=' ' INVOCATION_ID='e362ddfb13e4462baeea94aab7fb5a30' LANG='C' LANGUAGE='en_US:en' LC_ALL='C' MAIL='/var/mail/root' OPTIND='1' PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' PBCURRENTCOMMANDLINEOPERATION='build' PBUILDER_OPERATION='build' PBUILDER_PKGDATADIR='/usr/share/pbuilder' PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' PBUILDER_SYSCONFDIR='/etc' PPID='3199738' PS1='# ' PS2='> ' PS4='+ ' PWD='/' SHELL='/bin/bash' SHLVL='2' SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.S3h9JXH8/pbuilderrc_ViBi --distribution forky --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/forky-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.S3h9JXH8/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc' SUDO_GID='110' SUDO_HOME='/var/lib/jenkins' SUDO_UID='105' SUDO_USER='jenkins' TERM='unknown' TZ='/usr/share/zoneinfo/Etc/GMT+12' USER='root' _='/usr/bin/systemd-run' http_proxy='http://46.16.76.132:3128' I: uname -a Linux ionos1-amd64 6.12.43+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.43-1 (2025-08-27) x86_64 GNU/Linux I: ls -l /bin lrwxrwxrwx 1 root root 7 Aug 10 12:30 /bin -> usr/bin I: user script /srv/workspace/pbuilder/3199738/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: amd64 Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 19917 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however: Package debhelper-compat is not installed. pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however: Package dh-sequence-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however: Package golang-github-stretchr-testify-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.24-go{a} golang-1.24-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-v2-dev{a} golang-github-aws-smithy-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-prometheus-sigv4-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} golang-uber-goleak-dev{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libunistring5{a} libxml2-16{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 0 packages upgraded, 108 newly installed, 0 to remove and 0 not upgraded. Need to get 126 MB of archives. After unpacking 856 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian forky/main amd64 sensible-utils all 0.0.26 [27.0 kB] Get: 2 http://deb.debian.org/debian forky/main amd64 libmagic-mgc amd64 1:5.46-5 [338 kB] Get: 3 http://deb.debian.org/debian forky/main amd64 libmagic1t64 amd64 1:5.46-5 [109 kB] Get: 4 http://deb.debian.org/debian forky/main amd64 file amd64 1:5.46-5 [43.6 kB] Get: 5 http://deb.debian.org/debian forky/main amd64 gettext-base amd64 0.23.1-2+b1 [244 kB] Get: 6 http://deb.debian.org/debian forky/main amd64 libuchardet0 amd64 0.0.8-2 [68.5 kB] Get: 7 http://deb.debian.org/debian forky/main amd64 groff-base amd64 1.23.0-9 [1187 kB] Get: 8 http://deb.debian.org/debian forky/main amd64 bsdextrautils amd64 2.41.1-2 [95.0 kB] Get: 9 http://deb.debian.org/debian forky/main amd64 libpipeline1 amd64 1.5.8-1 [42.0 kB] Get: 10 http://deb.debian.org/debian forky/main amd64 man-db amd64 2.13.1-1 [1469 kB] Get: 11 http://deb.debian.org/debian forky/main amd64 m4 amd64 1.4.20-1 [324 kB] Get: 12 http://deb.debian.org/debian forky/main amd64 autoconf all 2.72-3.1 [494 kB] Get: 13 http://deb.debian.org/debian forky/main amd64 autotools-dev all 20240727.1 [60.2 kB] Get: 14 http://deb.debian.org/debian forky/main amd64 automake all 1:1.17-4 [862 kB] Get: 15 http://deb.debian.org/debian forky/main amd64 autopoint all 0.23.1-2 [770 kB] Get: 16 http://deb.debian.org/debian forky/main amd64 libdebhelper-perl all 13.26 [91.8 kB] Get: 17 http://deb.debian.org/debian forky/main amd64 libtool all 2.5.4-5 [540 kB] Get: 18 http://deb.debian.org/debian forky/main amd64 dh-autoreconf all 20 [17.1 kB] Get: 19 http://deb.debian.org/debian forky/main amd64 libarchive-zip-perl all 1.68-1 [104 kB] Get: 20 http://deb.debian.org/debian forky/main amd64 libfile-stripnondeterminism-perl all 1.15.0-1 [19.9 kB] Get: 21 http://deb.debian.org/debian forky/main amd64 dh-strip-nondeterminism all 1.15.0-1 [8812 B] Get: 22 http://deb.debian.org/debian forky/main amd64 libelf1t64 amd64 0.193-3 [192 kB] Get: 23 http://deb.debian.org/debian forky/main amd64 dwz amd64 0.16-2 [108 kB] Get: 24 http://deb.debian.org/debian forky/main amd64 libunistring5 amd64 1.3-2 [477 kB] Get: 25 http://deb.debian.org/debian forky/main amd64 libxml2-16 amd64 2.14.5+dfsg-0.2 [638 kB] Get: 26 http://deb.debian.org/debian forky/main amd64 gettext amd64 0.23.1-2+b1 [1680 kB] Get: 27 http://deb.debian.org/debian forky/main amd64 intltool-debian all 0.35.0+20060710.6 [22.9 kB] Get: 28 http://deb.debian.org/debian forky/main amd64 po-debconf all 1.0.21+nmu1 [248 kB] Get: 29 http://deb.debian.org/debian forky/main amd64 debhelper all 13.26 [939 kB] Get: 30 http://deb.debian.org/debian forky/main amd64 dh-golang all 1.63 [24.4 kB] Get: 31 http://deb.debian.org/debian forky/main amd64 golang-1.24-src all 1.24.4-1 [21.2 MB] Get: 32 http://deb.debian.org/debian forky/main amd64 golang-1.24-go amd64 1.24.4-1 [28.7 MB] Get: 33 http://deb.debian.org/debian forky/main amd64 golang-src all 2:1.24~2 [5136 B] Get: 34 http://deb.debian.org/debian forky/main amd64 golang-go amd64 2:1.24~2 [44.3 kB] Get: 35 http://deb.debian.org/debian forky/main amd64 golang-any amd64 2:1.24~2 [5216 B] Get: 36 http://deb.debian.org/debian forky/main amd64 golang-github-alecthomas-units-dev all 0.0~git20240927.0f3dac3-1 [8880 B] Get: 37 http://deb.debian.org/debian forky/main amd64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 38 http://deb.debian.org/debian forky/main amd64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 39 http://deb.debian.org/debian forky/main amd64 golang-golang-x-sys-dev all 0.36.0-1 [640 kB] Get: 40 http://deb.debian.org/debian forky/main amd64 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B] Get: 41 http://deb.debian.org/debian forky/main amd64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB] Get: 42 http://deb.debian.org/debian forky/main amd64 golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB] Get: 43 http://deb.debian.org/debian forky/main amd64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB] Get: 44 http://deb.debian.org/debian forky/main amd64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 45 http://deb.debian.org/debian forky/main amd64 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB] Get: 46 http://deb.debian.org/debian forky/main amd64 golang-gopkg-yaml.v3-dev all 3.0.4-1 [76.0 kB] Get: 47 http://deb.debian.org/debian forky/main amd64 golang-github-stretchr-testify-dev all 1.10.0-1 [78.4 kB] Get: 48 http://deb.debian.org/debian forky/main amd64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 49 http://deb.debian.org/debian forky/main amd64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB] Get: 50 http://deb.debian.org/debian forky/main amd64 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB] Get: 51 http://deb.debian.org/debian forky/main amd64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B] Get: 52 http://deb.debian.org/debian forky/main amd64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB] Get: 53 http://deb.debian.org/debian forky/main amd64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 54 http://deb.debian.org/debian forky/main amd64 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB] Get: 55 http://deb.debian.org/debian forky/main amd64 golang-google-protobuf-dev all 1.36.7-1 [736 kB] Get: 56 http://deb.debian.org/debian forky/main amd64 golang-github-golang-protobuf-1-5-dev all 1.5.4-2 [137 kB] Get: 57 http://deb.debian.org/debian forky/main amd64 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B] Get: 58 http://deb.debian.org/debian forky/main amd64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 59 http://deb.debian.org/debian forky/main amd64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB] Get: 60 http://deb.debian.org/debian forky/main amd64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB] Get: 61 http://deb.debian.org/debian forky/main amd64 golang-github-beorn7-perks-dev all 1.0.1-3 [12.2 kB] Get: 62 http://deb.debian.org/debian forky/main amd64 golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB] Get: 63 http://deb.debian.org/debian forky/main amd64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 64 http://deb.debian.org/debian forky/main amd64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB] Get: 65 http://deb.debian.org/debian forky/main amd64 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB] Get: 66 http://deb.debian.org/debian forky/main amd64 golang-github-klauspost-compress-dev all 1.18.0+ds1-1 [33.3 MB] Get: 67 http://deb.debian.org/debian forky/main amd64 golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB] Get: 68 http://deb.debian.org/debian forky/main amd64 golang-github-prometheus-client-model-dev all 0.6.2-1 [11.1 kB] Get: 69 http://deb.debian.org/debian forky/main amd64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 70 http://deb.debian.org/debian forky/main amd64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB] Get: 71 http://deb.debian.org/debian forky/main amd64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB] Get: 72 http://deb.debian.org/debian forky/main amd64 golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B] Get: 73 http://deb.debian.org/debian forky/main amd64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 74 http://deb.debian.org/debian forky/main amd64 golang-golang-x-term-dev all 0.35.0-1 [16.1 kB] Get: 75 http://deb.debian.org/debian forky/main amd64 golang-golang-x-text-dev all 0.22.0-1 [3995 kB] Get: 76 http://deb.debian.org/debian forky/main amd64 golang-golang-x-crypto-dev all 1:0.25.0-1 [1682 kB] Get: 77 http://deb.debian.org/debian forky/main amd64 golang-golang-x-net-dev all 1:0.27.0-2 [904 kB] Get: 78 http://deb.debian.org/debian forky/main amd64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-4 [14.6 kB] Get: 79 http://deb.debian.org/debian forky/main amd64 golang-github-aws-smithy-go-dev all 1.20.3-1 [125 kB] Get: 80 http://deb.debian.org/debian forky/main amd64 golang-gopkg-yaml.v2-dev all 2.4.2-1 [60.8 kB] Get: 81 http://deb.debian.org/debian forky/main amd64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB] Get: 82 http://deb.debian.org/debian forky/main amd64 golang-github-aws-aws-sdk-go-v2-dev all 1.30.3-1 [15.8 MB] Get: 83 http://deb.debian.org/debian forky/main amd64 golang-github-prometheus-sigv4-dev all 0.2.1-1 [9724 B] Get: 84 http://deb.debian.org/debian forky/main amd64 golang-golang-x-oauth2-dev all 0.27.0-1 [37.2 kB] Get: 85 http://deb.debian.org/debian forky/main amd64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB] Get: 86 http://deb.debian.org/debian forky/main amd64 golang-github-prometheus-common-dev all 0.65.0-1 [120 kB] Get: 87 http://deb.debian.org/debian forky/main amd64 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB] Get: 88 http://deb.debian.org/debian forky/main amd64 golang-github-prometheus-procfs-dev all 0.17.0-1 [199 kB] Get: 89 http://deb.debian.org/debian forky/main amd64 golang-uber-goleak-dev all 1.3.0-1 [18.1 kB] Get: 90 http://deb.debian.org/debian forky/main amd64 golang-github-prometheus-client-golang-dev all 1.23.0-1 [219 kB] Get: 91 http://deb.debian.org/debian forky/main amd64 golang-github-armon-go-metrics-dev all 0.4.1-2 [35.2 kB] Get: 92 http://deb.debian.org/debian forky/main amd64 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB] Get: 93 http://deb.debian.org/debian forky/main amd64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B] Get: 94 http://deb.debian.org/debian forky/main amd64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 95 http://deb.debian.org/debian forky/main amd64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 96 http://deb.debian.org/debian forky/main amd64 libsasl2-modules-db amd64 2.1.28+dfsg1-9 [19.8 kB] Get: 97 http://deb.debian.org/debian forky/main amd64 libsasl2-2 amd64 2.1.28+dfsg1-9 [57.5 kB] Get: 98 http://deb.debian.org/debian forky/main amd64 libssl-dev amd64 3.5.2-1 [2957 kB] Get: 99 http://deb.debian.org/debian forky/main amd64 libsasl2-dev amd64 2.1.28+dfsg1-9 [234 kB] Get: 100 http://deb.debian.org/debian forky/main amd64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB] Get: 101 http://deb.debian.org/debian forky/main amd64 golang-golang-x-mod-dev all 0.19.0-1 [104 kB] Get: 102 http://deb.debian.org/debian forky/main amd64 golang-github-yuin-goldmark-dev all 1.7.4-1 [113 kB] Get: 103 http://deb.debian.org/debian forky/main amd64 golang-golang-x-tools-dev all 1:0.25.0+ds-1 [1608 kB] Get: 104 http://deb.debian.org/debian forky/main amd64 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB] Get: 105 http://deb.debian.org/debian forky/main amd64 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB] Get: 106 http://deb.debian.org/debian forky/main amd64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-2 [31.2 kB] Get: 107 http://deb.debian.org/debian forky/main amd64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 108 http://deb.debian.org/debian forky/main amd64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 126 MB in 1s (91.9 MB/s) Preconfiguring packages ... Selecting previously unselected package sensible-utils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19917 files and directories currently installed.) Preparing to unpack .../000-sensible-utils_0.0.26_all.deb ... Unpacking sensible-utils (0.0.26) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../001-libmagic-mgc_1%3a5.46-5_amd64.deb ... Unpacking libmagic-mgc (1:5.46-5) ... Selecting previously unselected package libmagic1t64:amd64. Preparing to unpack .../002-libmagic1t64_1%3a5.46-5_amd64.deb ... Unpacking libmagic1t64:amd64 (1:5.46-5) ... Selecting previously unselected package file. Preparing to unpack .../003-file_1%3a5.46-5_amd64.deb ... Unpacking file (1:5.46-5) ... Selecting previously unselected package gettext-base. Preparing to unpack .../004-gettext-base_0.23.1-2+b1_amd64.deb ... Unpacking gettext-base (0.23.1-2+b1) ... Selecting previously unselected package libuchardet0:amd64. Preparing to unpack .../005-libuchardet0_0.0.8-2_amd64.deb ... Unpacking libuchardet0:amd64 (0.0.8-2) ... Selecting previously unselected package groff-base. Preparing to unpack .../006-groff-base_1.23.0-9_amd64.deb ... Unpacking groff-base (1.23.0-9) ... Selecting previously unselected package bsdextrautils. Preparing to unpack .../007-bsdextrautils_2.41.1-2_amd64.deb ... Unpacking bsdextrautils (2.41.1-2) ... Selecting previously unselected package libpipeline1:amd64. Preparing to unpack .../008-libpipeline1_1.5.8-1_amd64.deb ... Unpacking libpipeline1:amd64 (1.5.8-1) ... Selecting previously unselected package man-db. Preparing to unpack .../009-man-db_2.13.1-1_amd64.deb ... Unpacking man-db (2.13.1-1) ... Selecting previously unselected package m4. Preparing to unpack .../010-m4_1.4.20-1_amd64.deb ... Unpacking m4 (1.4.20-1) ... Selecting previously unselected package autoconf. Preparing to unpack .../011-autoconf_2.72-3.1_all.deb ... Unpacking autoconf (2.72-3.1) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../012-autotools-dev_20240727.1_all.deb ... Unpacking autotools-dev (20240727.1) ... Selecting previously unselected package automake. Preparing to unpack .../013-automake_1%3a1.17-4_all.deb ... Unpacking automake (1:1.17-4) ... Selecting previously unselected package autopoint. Preparing to unpack .../014-autopoint_0.23.1-2_all.deb ... Unpacking autopoint (0.23.1-2) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../015-libdebhelper-perl_13.26_all.deb ... Unpacking libdebhelper-perl (13.26) ... Selecting previously unselected package libtool. Preparing to unpack .../016-libtool_2.5.4-5_all.deb ... Unpacking libtool (2.5.4-5) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../017-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.15.0-1_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.15.0-1) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../020-dh-strip-nondeterminism_1.15.0-1_all.deb ... Unpacking dh-strip-nondeterminism (1.15.0-1) ... Selecting previously unselected package libelf1t64:amd64. Preparing to unpack .../021-libelf1t64_0.193-3_amd64.deb ... Unpacking libelf1t64:amd64 (0.193-3) ... Selecting previously unselected package dwz. Preparing to unpack .../022-dwz_0.16-2_amd64.deb ... Unpacking dwz (0.16-2) ... Selecting previously unselected package libunistring5:amd64. Preparing to unpack .../023-libunistring5_1.3-2_amd64.deb ... Unpacking libunistring5:amd64 (1.3-2) ... Selecting previously unselected package libxml2-16:amd64. Preparing to unpack .../024-libxml2-16_2.14.5+dfsg-0.2_amd64.deb ... Unpacking libxml2-16:amd64 (2.14.5+dfsg-0.2) ... Selecting previously unselected package gettext. Preparing to unpack .../025-gettext_0.23.1-2+b1_amd64.deb ... Unpacking gettext (0.23.1-2+b1) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../026-intltool-debian_0.35.0+20060710.6_all.deb ... Unpacking intltool-debian (0.35.0+20060710.6) ... Selecting previously unselected package po-debconf. Preparing to unpack .../027-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../028-debhelper_13.26_all.deb ... Unpacking debhelper (13.26) ... Selecting previously unselected package dh-golang. Preparing to unpack .../029-dh-golang_1.63_all.deb ... Unpacking dh-golang (1.63) ... Selecting previously unselected package golang-1.24-src. Preparing to unpack .../030-golang-1.24-src_1.24.4-1_all.deb ... Unpacking golang-1.24-src (1.24.4-1) ... Selecting previously unselected package golang-1.24-go. Preparing to unpack .../031-golang-1.24-go_1.24.4-1_amd64.deb ... Unpacking golang-1.24-go (1.24.4-1) ... Selecting previously unselected package golang-src. Preparing to unpack .../032-golang-src_2%3a1.24~2_all.deb ... Unpacking golang-src (2:1.24~2) ... Selecting previously unselected package golang-go:amd64. Preparing to unpack .../033-golang-go_2%3a1.24~2_amd64.deb ... Unpacking golang-go:amd64 (2:1.24~2) ... Selecting previously unselected package golang-any:amd64. Preparing to unpack .../034-golang-any_2%3a1.24~2_amd64.deb ... Unpacking golang-any:amd64 (2:1.24~2) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../035-golang-github-alecthomas-units-dev_0.0~git20240927.0f3dac3-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20240927.0f3dac3-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../036-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../037-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../038-golang-golang-x-sys-dev_0.36.0-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.36.0-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../039-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../040-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../041-golang-github-fatih-color-dev_1.17.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.17.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../042-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../043-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../044-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../045-golang-gopkg-yaml.v3-dev_3.0.4-1_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.4-1) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../046-golang-github-stretchr-testify-dev_1.10.0-1_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.10.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../047-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../048-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../049-golang-github-pkg-errors-dev_0.9.1-3_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-3) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../050-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../051-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../052-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-google-go-cmp-dev. Preparing to unpack .../053-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ... Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ... Selecting previously unselected package golang-google-protobuf-dev. Preparing to unpack .../054-golang-google-protobuf-dev_1.36.7-1_all.deb ... Unpacking golang-google-protobuf-dev (1.36.7-1) ... Selecting previously unselected package golang-github-golang-protobuf-1-5-dev. Preparing to unpack .../055-golang-github-golang-protobuf-1-5-dev_1.5.4-2_all.deb ... Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../056-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../057-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../058-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../059-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../060-golang-github-beorn7-perks-dev_1.0.1-3_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-3) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../061-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../062-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../063-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../064-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ... Selecting previously unselected package golang-github-klauspost-compress-dev. Preparing to unpack .../065-golang-github-klauspost-compress-dev_1.18.0+ds1-1_all.deb ... Unpacking golang-github-klauspost-compress-dev (1.18.0+ds1-1) ... Selecting previously unselected package golang-github-kylelemons-godebug-dev. Preparing to unpack .../066-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ... Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../067-golang-github-prometheus-client-model-dev_0.6.2-1_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.6.2-1) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../068-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-kit-log-dev. Preparing to unpack .../069-golang-github-go-kit-log-dev_0.2.1-1_all.deb ... Unpacking golang-github-go-kit-log-dev (0.2.1-1) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../070-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Selecting previously unselected package golang-github-munnerz-goautoneg-dev. Preparing to unpack .../071-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ... Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../072-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-golang-x-term-dev. Preparing to unpack .../073-golang-golang-x-term-dev_0.35.0-1_all.deb ... Unpacking golang-golang-x-term-dev (0.35.0-1) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../074-golang-golang-x-text-dev_0.22.0-1_all.deb ... Unpacking golang-golang-x-text-dev (0.22.0-1) ... Selecting previously unselected package golang-golang-x-crypto-dev. Preparing to unpack .../075-golang-golang-x-crypto-dev_1%3a0.25.0-1_all.deb ... Unpacking golang-golang-x-crypto-dev (1:0.25.0-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../076-golang-golang-x-net-dev_1%3a0.27.0-2_all.deb ... Unpacking golang-golang-x-net-dev (1:0.27.0-2) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../077-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-4_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ... Selecting previously unselected package golang-github-aws-smithy-go-dev. Preparing to unpack .../078-golang-github-aws-smithy-go-dev_1.20.3-1_all.deb ... Unpacking golang-github-aws-smithy-go-dev (1.20.3-1) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../079-golang-gopkg-yaml.v2-dev_2.4.2-1_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.2-1) ... Selecting previously unselected package golang-github-jmespath-go-jmespath-dev. Preparing to unpack .../080-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ... Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Selecting previously unselected package golang-github-aws-aws-sdk-go-v2-dev. Preparing to unpack .../081-golang-github-aws-aws-sdk-go-v2-dev_1.30.3-1_all.deb ... Unpacking golang-github-aws-aws-sdk-go-v2-dev (1.30.3-1) ... Selecting previously unselected package golang-github-prometheus-sigv4-dev. Preparing to unpack .../082-golang-github-prometheus-sigv4-dev_0.2.1-1_all.deb ... Unpacking golang-github-prometheus-sigv4-dev (0.2.1-1) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../083-golang-golang-x-oauth2-dev_0.27.0-1_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.27.0-1) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../084-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../085-golang-github-prometheus-common-dev_0.65.0-1_all.deb ... Unpacking golang-github-prometheus-common-dev (0.65.0-1) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../086-golang-golang-x-sync-dev_0.7.0-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.7.0-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../087-golang-github-prometheus-procfs-dev_0.17.0-1_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.17.0-1) ... Selecting previously unselected package golang-uber-goleak-dev. Preparing to unpack .../088-golang-uber-goleak-dev_1.3.0-1_all.deb ... Unpacking golang-uber-goleak-dev (1.3.0-1) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../089-golang-github-prometheus-client-golang-dev_1.23.0-1_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.23.0-1) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../090-golang-github-armon-go-metrics-dev_0.4.1-2_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.4.1-2) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../091-golang-github-creack-pty-dev_1.1.21-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.21-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../092-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../093-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../094-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:amd64. Preparing to unpack .../095-libsasl2-modules-db_2.1.28+dfsg1-9_amd64.deb ... Unpacking libsasl2-modules-db:amd64 (2.1.28+dfsg1-9) ... Selecting previously unselected package libsasl2-2:amd64. Preparing to unpack .../096-libsasl2-2_2.1.28+dfsg1-9_amd64.deb ... Unpacking libsasl2-2:amd64 (2.1.28+dfsg1-9) ... Selecting previously unselected package libssl-dev:amd64. Preparing to unpack .../097-libssl-dev_3.5.2-1_amd64.deb ... Unpacking libssl-dev:amd64 (3.5.2-1) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../098-libsasl2-dev_2.1.28+dfsg1-9_amd64.deb ... Unpacking libsasl2-dev (2.1.28+dfsg1-9) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../099-golang-github-kr-text-dev_0.2.0-2_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-2) ... Selecting previously unselected package golang-golang-x-mod-dev. Preparing to unpack .../100-golang-golang-x-mod-dev_0.19.0-1_all.deb ... Unpacking golang-golang-x-mod-dev (0.19.0-1) ... Selecting previously unselected package golang-github-yuin-goldmark-dev. Preparing to unpack .../101-golang-github-yuin-goldmark-dev_1.7.4-1_all.deb ... Unpacking golang-github-yuin-goldmark-dev (1.7.4-1) ... Selecting previously unselected package golang-golang-x-tools-dev. Preparing to unpack .../102-golang-golang-x-tools-dev_1%3a0.25.0+ds-1_all.deb ... Unpacking golang-golang-x-tools-dev (1:0.25.0+ds-1) ... Selecting previously unselected package golang-github-rogpeppe-go-internal-dev. Preparing to unpack .../103-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ... Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../104-golang-github-kr-pretty-dev_0.3.1-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.3.1-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../105-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-2_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../106-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../107-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up golang-1.24-src (1.24.4-1) ... Setting up libpipeline1:amd64 (1.5.8-1) ... Setting up golang-github-google-go-cmp-dev (0.6.0-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.2-1) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Setting up bsdextrautils (2.41.1-2) ... Setting up golang-github-creack-pty-dev (1.1.21-1) ... Setting up golang-github-stretchr-objx-dev (0.5.2-1) ... Setting up libmagic-mgc (1:5.46-5) ... Setting up golang-github-pkg-errors-dev (0.9.1-3) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-github-aws-smithy-go-dev (1.20.3-1) ... Setting up libxml2-16:amd64 (2.14.5+dfsg-0.2) ... Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ... Setting up golang-gopkg-yaml.v3-dev (3.0.4-1) ... Setting up libdebhelper-perl (13.26) ... Setting up golang-golang-x-sys-dev (0.36.0-1) ... Setting up libmagic1t64:amd64 (1:5.46-5) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.23.1-2+b1) ... Setting up m4 (1.4.20-1) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up golang-golang-x-term-dev (0.35.0-1) ... Setting up file (1:5.46-5) ... Setting up golang-github-beorn7-perks-dev (1.0.1-3) ... Setting up libelf1t64:amd64 (0.193-3) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:amd64 (2.1.28+dfsg1-9) ... Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ... Setting up autotools-dev (20240727.1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-google-protobuf-dev (1.36.7-1) ... Setting up golang-github-kr-text-dev (0.2.0-2) ... Setting up golang-github-stretchr-testify-dev (1.10.0-1) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Setting up golang-github-yuin-goldmark-dev (1.7.4-1) ... Setting up libunistring5:amd64 (1.3-2) ... Setting up libssl-dev:amd64 (3.5.2-1) ... Setting up golang-golang-x-sync-dev (0.7.0-1) ... Setting up golang-golang-x-mod-dev (0.19.0-1) ... Setting up autopoint (0.23.1-2) ... Setting up golang-1.24-go (1.24.4-1) ... Setting up golang-github-klauspost-compress-dev (1.18.0+ds1-1) ... Setting up libsasl2-2:amd64 (2.1.28+dfsg1-9) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Setting up autoconf (2.72-3.1) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Setting up dwz (0.16-2) ... Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ... Setting up sensible-utils (0.0.26) ... Setting up golang-golang-x-text-dev (0.22.0-1) ... Setting up libuchardet0:amd64 (0.0.8-2) ... Setting up golang-github-prometheus-procfs-dev (0.17.0-1) ... Setting up golang-github-alecthomas-units-dev (0.0~git20240927.0f3dac3-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ... Setting up golang-uber-goleak-dev (1.3.0-1) ... Setting up golang-golang-x-crypto-dev (1:0.25.0-1) ... Setting up golang-github-prometheus-client-model-dev (0.6.2-1) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-golang-x-oauth2-dev (0.27.0-1) ... Setting up golang-src (2:1.24~2) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Setting up automake (1:1.17-4) ... update-alternatives: using /usr/bin/automake-1.17 to provide /usr/bin/automake (automake) in auto mode Setting up libfile-stripnondeterminism-perl (1.15.0-1) ... Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ... Setting up golang-github-go-kit-log-dev (0.2.1-1) ... Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-2) ... Setting up gettext (0.23.1-2+b1) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Setting up libtool (2.5.4-5) ... Setting up golang-go:amd64 (2:1.24~2) ... Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ... Setting up golang-github-json-iterator-go-dev (1.1.12-2) ... Setting up golang-any:amd64 (2:1.24~2) ... Setting up intltool-debian (0.35.0+20060710.6) ... Setting up dh-autoreconf (20) ... Setting up libsasl2-dev (2.1.28+dfsg1-9) ... Setting up golang-golang-x-net-dev (1:0.27.0-2) ... Setting up dh-strip-nondeterminism (1.15.0-1) ... Setting up groff-base (1.23.0-9) ... Setting up golang-github-fatih-color-dev (1.17.0-1) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up golang-golang-x-tools-dev (1:0.25.0+ds-1) ... Setting up man-db (2.13.1-1) ... Not building database; man-db/auto-update is not 'true'. Setting up golang-github-aws-aws-sdk-go-v2-dev (1.30.3-1) ... Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up golang-github-kr-pretty-dev (0.3.1-1) ... Setting up debhelper (13.26) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Setting up dh-golang (1.63) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ... Setting up golang-github-prometheus-sigv4-dev (0.2.1-1) ... Setting up golang-github-prometheus-common-dev (0.65.0-1) ... Setting up golang-github-prometheus-client-golang-dev (1.23.0-1) ... Setting up golang-github-armon-go-metrics-dev (0.4.1-2) ... Processing triggers for libc-bin (2.41-12) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps I: Building the package I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.5.0-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.5.0-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Daniel Swarbrick dpkg-source --before-build . dpkg-buildpackage: info: host architecture amd64 debian/rules clean dh clean --buildsystem=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules binary dh binary --buildsystem=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-x86_64-linux-gnu && go install -trimpath -v -p 40 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/itoa encoding cmp unicode/utf16 internal/msan crypto/internal/boring/sig internal/asan internal/profilerecord vendor/golang.org/x/crypto/internal/alias unicode/utf8 container/list internal/unsafeheader internal/coverage/rtcov log/internal internal/byteorder internal/godebugs internal/goarch math/bits internal/runtime/atomic internal/nettrace internal/runtime/syscall vendor/golang.org/x/crypto/cryptobyte/asn1 sync/atomic crypto/internal/fips140/alias unicode internal/goos internal/goexperiment internal/cpu internal/runtime/math internal/abi internal/runtime/sys crypto/internal/fips140/subtle crypto/internal/fips140deps/byteorder internal/chacha8rand internal/runtime/exithook crypto/internal/fips140deps/cpu internal/bytealg math internal/stringslite internal/race internal/sync internal/runtime/maps runtime internal/reflectlite crypto/subtle iter weak sync maps slices internal/singleflight internal/testlog unique internal/bisect errors sort runtime/cgo internal/oserror internal/godebug io path strconv github.com/hashicorp/golang-lru/simplelru math/rand/v2 vendor/golang.org/x/net/dns/dnsmessage syscall bytes hash internal/saferio strings hash/crc64 hash/crc32 crypto/internal/fips140deps/godebug math/rand crypto/internal/randutil crypto net/netip reflect vendor/golang.org/x/text/transform crypto/internal/impl bufio net/http/internal/ascii crypto/internal/fips140 github.com/hashicorp/go-immutable-radix html regexp/syntax crypto/tls/internal/fips140tls crypto/internal/fips140/sha512 crypto/internal/fips140/sha256 crypto/internal/fips140/sha3 crypto/internal/fips140/hmac crypto/sha3 crypto/internal/fips140/check crypto/internal/fips140hash crypto/internal/fips140/hkdf crypto/internal/fips140/tls12 crypto/internal/fips140/edwards25519/field crypto/internal/fips140/bigmod crypto/internal/fips140/aes crypto/internal/fips140/nistec/fiat crypto/internal/fips140/tls13 internal/syscall/execenv time internal/syscall/unix regexp crypto/internal/fips140/edwards25519 context io/fs internal/poll crypto/internal/fips140/nistec internal/filepathlite os internal/fmtsort encoding/binary encoding/base64 vendor/golang.org/x/crypto/internal/poly1305 golang.org/x/sys/unix encoding/pem crypto/internal/sysrand io/ioutil path/filepath internal/sysinfo fmt os/signal vendor/golang.org/x/sys/cpu net crypto/internal/entropy crypto/internal/fips140/drbg crypto/internal/fips140only crypto/internal/fips140/ecdh crypto/internal/fips140/ed25519 crypto/internal/fips140/mlkem crypto/internal/fips140/ecdsa crypto/internal/fips140/rsa crypto/internal/fips140/aes/gcm crypto/rc4 crypto/md5 crypto/cipher log encoding/hex net/url net/http/internal mime/quotedprintable mime flag text/template/parse runtime/trace compress/flate vendor/golang.org/x/net/http2/hpack encoding/gob runtime/debug math/big go/token vendor/golang.org/x/text/unicode/norm encoding/json crypto/internal/boring crypto/des vendor/golang.org/x/crypto/chacha20 crypto/sha512 crypto/ecdh crypto/hmac crypto/sha1 crypto/sha256 crypto/aes vendor/golang.org/x/crypto/chacha20poly1305 vendor/golang.org/x/text/unicode/bidi testing compress/gzip text/template vendor/golang.org/x/text/secure/bidirule vendor/golang.org/x/net/idna github.com/mattn/go-isatty github.com/mattn/go-colorable crypto/internal/boring/bbig crypto/rand crypto/elliptic crypto/dsa encoding/asn1 github.com/fatih/color crypto/ed25519 crypto/internal/hpke crypto/rsa github.com/hashicorp/go-hclog html/template crypto/x509/pkix vendor/golang.org/x/crypto/cryptobyte crypto/ecdsa vendor/golang.org/x/net/http/httpproxy net/textproto crypto/x509 vendor/golang.org/x/net/http/httpguts mime/multipart crypto/tls net/http/httptrace net/http net/rpc github.com/armon/go-metrics github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench debian/rules override_dh_auto_test make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' dh_auto_test cd obj-x86_64-linux-gnu && go test -vet=off -v -p 40 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir 2025-09-14T21:47:09.434Z [INFO] creating new snapshot: path=/tmp/raft728516800/raft2866414255/snapshots/3-10-1757886429434.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s) === RUN TestFileSS_CreateSnapshot 2025-09-14T21:47:09.438Z [INFO] creating new snapshot: path=/tmp/raft975389611/snapshots/3-10-1757886429438.tmp 2025-09-14T21:47:09.440Z [WARN] found temporary snapshot: name=3-10-1757886429438.tmp --- PASS: TestFileSS_CreateSnapshot (0.01s) === RUN TestFileSS_CancelSnapshot 2025-09-14T21:47:09.446Z [INFO] creating new snapshot: path=/tmp/raft4150517409/snapshots/3-10-1757886429446.tmp --- PASS: TestFileSS_CancelSnapshot (0.00s) === RUN TestFileSS_Retention 2025-09-14T21:47:09.449Z [INFO] creating new snapshot: path=/tmp/raft1515935710/snapshots/3-10-1757886429449.tmp 2025-09-14T21:47:09.455Z [INFO] creating new snapshot: path=/tmp/raft1515935710/snapshots/3-11-1757886429455.tmp 2025-09-14T21:47:09.465Z [INFO] creating new snapshot: path=/tmp/raft1515935710/snapshots/3-12-1757886429465.tmp 2025-09-14T21:47:09.471Z [INFO] reaping snapshot: path=/tmp/raft1515935710/snapshots/3-10-1757886429449 2025-09-14T21:47:09.471Z [INFO] creating new snapshot: path=/tmp/raft1515935710/snapshots/3-13-1757886429471.tmp 2025-09-14T21:47:09.477Z [INFO] reaping snapshot: path=/tmp/raft1515935710/snapshots/3-11-1757886429455 2025-09-14T21:47:09.477Z [INFO] creating new snapshot: path=/tmp/raft1515935710/snapshots/3-14-1757886429477.tmp 2025-09-14T21:47:09.483Z [INFO] reaping snapshot: path=/tmp/raft1515935710/snapshots/3-12-1757886429465 --- PASS: TestFileSS_Retention (0.03s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering 2025-09-14T21:47:09.484Z [INFO] creating new snapshot: path=/tmp/raft2784509578/snapshots/5-130350-1757886429484.tmp 2025-09-14T21:47:09.490Z [INFO] creating new snapshot: path=/tmp/raft2784509578/snapshots/36-204917-1757886429490.tmp --- PASS: TestFileSS_Ordering (0.01s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestRaft_RestartFollower_LongInitialHeartbeat --- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestLogCacheWithBackendStoreError --- PASS: TestLogCacheWithBackendStoreError (0.00s) === RUN TestOldestLog === RUN TestOldestLog/empty_logs === RUN TestOldestLog/simple_case --- PASS: TestOldestLog (0.00s) --- PASS: TestOldestLog/empty_logs (0.00s) --- PASS: TestOldestLog/simple_case (0.00s) === RUN TestEmitsLogStoreMetrics --- PASS: TestEmitsLogStoreMetrics (0.01s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.01s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams --- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.02s) === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 2025-09-14T21:47:09.591Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 2025-09-14T21:47:09.643Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 2025-09-14T21:47:09.695Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.15s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s) === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:839: countingWriter: 2025-09-14T21:47:09.701Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-09-14T21:47:09.706Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-09-14T21:47:09.716Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-09-14T21:47:09.737Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-09-14T21:47:09.778Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-09-14T21:47:09.858Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-09-14T21:47:10.019Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-09-14T21:47:10.340Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-09-14T21:47:10.981Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-09-14T21:47:11.983Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-09-14T21:47:12.985Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop 2025-09-14T21:47:13.703Z [INFO] server-178ed993-5499-8099-dad6-4e1b8ca00b80: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-178ed993-5499-8099-dad6-4e1b8ca00b80 Address:178ed993-5499-8099-dad6-4e1b8ca00b80}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown 2025-09-14T21:47:13.704Z [INFO] server-414cd7b3-010f-345b-15e9-d324b3e46787: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-414cd7b3-010f-345b-15e9-d324b3e46787 Address:414cd7b3-010f-345b-15e9-d324b3e46787}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap 2025-09-14T21:47:13.704Z [INFO] server-e07562a0-07de-f28c-f4ef-e46994a8bcd3: initial configuration: index=0 servers=[] 2025-09-14T21:47:13.704Z [INFO] server-ab0d4973-80bd-e71d-a6e1-f85282260fc4: initial configuration: index=0 servers=[] 2025-09-14T21:47:13.705Z [INFO] server-6d14e43f-b39e-6df8-faf4-54e1b139abc9: initial configuration: index=0 servers=[] 2025-09-14T21:47:13.705Z [INFO] server-ab0d4973-80bd-e71d-a6e1-f85282260fc4: entering follower state: follower="Node at ab0d4973-80bd-e71d-a6e1-f85282260fc4 [Follower]" leader-address= leader-id= 2025-09-14T21:47:13.705Z [INFO] server-e07562a0-07de-f28c-f4ef-e46994a8bcd3: entering follower state: follower="Node at e07562a0-07de-f28c-f4ef-e46994a8bcd3 [Follower]" leader-address= leader-id= 2025-09-14T21:47:13.705Z [INFO] server-6d14e43f-b39e-6df8-faf4-54e1b139abc9: entering follower state: follower="Node at 6d14e43f-b39e-6df8-faf4-54e1b139abc9 [Follower]" leader-address= leader-id= 2025-09-14T21:47:13.705Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:13.756Z [WARN] server-e07562a0-07de-f28c-f4ef-e46994a8bcd3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:13.756Z [INFO] server-e07562a0-07de-f28c-f4ef-e46994a8bcd3: entering candidate state: node="Node at e07562a0-07de-f28c-f4ef-e46994a8bcd3 [Candidate]" term=2 2025-09-14T21:47:13.756Z [INFO] server-e07562a0-07de-f28c-f4ef-e46994a8bcd3: election won: term=2 tally=2 2025-09-14T21:47:13.756Z [INFO] server-e07562a0-07de-f28c-f4ef-e46994a8bcd3: entering leader state: leader="Node at e07562a0-07de-f28c-f4ef-e46994a8bcd3 [Leader]" 2025-09-14T21:47:13.756Z [INFO] server-e07562a0-07de-f28c-f4ef-e46994a8bcd3: added peer, starting replication: peer=server-ab0d4973-80bd-e71d-a6e1-f85282260fc4 2025-09-14T21:47:13.756Z [INFO] server-e07562a0-07de-f28c-f4ef-e46994a8bcd3: added peer, starting replication: peer=server-6d14e43f-b39e-6df8-faf4-54e1b139abc9 2025-09-14T21:47:13.756Z [WARN] server-6d14e43f-b39e-6df8-faf4-54e1b139abc9: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2025-09-14T21:47:13.756Z [WARN] server-ab0d4973-80bd-e71d-a6e1-f85282260fc4: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2025-09-14T21:47:13.756Z [WARN] server-e07562a0-07de-f28c-f4ef-e46994a8bcd3: appendEntries rejected, sending older logs: peer="{Voter server-6d14e43f-b39e-6df8-faf4-54e1b139abc9 6d14e43f-b39e-6df8-faf4-54e1b139abc9}" next=1 2025-09-14T21:47:13.756Z [WARN] server-e07562a0-07de-f28c-f4ef-e46994a8bcd3: appendEntries rejected, sending older logs: peer="{Voter server-ab0d4973-80bd-e71d-a6e1-f85282260fc4 ab0d4973-80bd-e71d-a6e1-f85282260fc4}" next=1 2025-09-14T21:47:13.757Z [INFO] server-e07562a0-07de-f28c-f4ef-e46994a8bcd3: pipelining replication: peer="{Voter server-6d14e43f-b39e-6df8-faf4-54e1b139abc9 6d14e43f-b39e-6df8-faf4-54e1b139abc9}" 2025-09-14T21:47:13.757Z [INFO] server-e07562a0-07de-f28c-f4ef-e46994a8bcd3: pipelining replication: peer="{Voter server-ab0d4973-80bd-e71d-a6e1-f85282260fc4 ab0d4973-80bd-e71d-a6e1-f85282260fc4}" 2025-09-14T21:47:13.862Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:13.756773943 +0000 UTC m=+4.324652224 (2 nodes), 51.469499ms from start of poll, 51.878934ms from cluster start. Timeout at 2025-09-14 21:47:13.861777216 +0000 UTC m=+4.429655491, 105.003267ms after stability 2025-09-14T21:47:13.862Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:13.967Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:13.862507522 +0000 UTC m=+4.430385797 (1 nodes), 1.094µs from start of poll, 157.612507ms from cluster start. Timeout at 2025-09-14 21:47:13.967510939 +0000 UTC m=+4.535389202, 105.003405ms after stability 2025-09-14T21:47:13.979Z [INFO] server-e07562a0-07de-f28c-f4ef-e46994a8bcd3: aborting pipeline replication: peer="{Voter server-6d14e43f-b39e-6df8-faf4-54e1b139abc9 6d14e43f-b39e-6df8-faf4-54e1b139abc9}" 2025-09-14T21:47:13.979Z [INFO] server-e07562a0-07de-f28c-f4ef-e46994a8bcd3: aborting pipeline replication: peer="{Voter server-ab0d4973-80bd-e71d-a6e1-f85282260fc4 ab0d4973-80bd-e71d-a6e1-f85282260fc4}" --- PASS: TestRaft_LiveBootstrap (0.28s) === RUN TestRaft_LiveBootstrap_From_NonVoter 2025-09-14T21:47:13.980Z [INFO] server-de9e1523-c1fe-91f8-6a91-ecfbcbd109e6: initial configuration: index=0 servers=[] 2025-09-14T21:47:13.980Z [INFO] server-010ab5e3-5c9d-0c0a-4b56-999c9cf87ab8: initial configuration: index=0 servers=[] 2025-09-14T21:47:13.980Z [INFO] server-de9e1523-c1fe-91f8-6a91-ecfbcbd109e6: entering follower state: follower="Node at de9e1523-c1fe-91f8-6a91-ecfbcbd109e6 [Follower]" leader-address= leader-id= 2025-09-14T21:47:13.980Z [INFO] server-010ab5e3-5c9d-0c0a-4b56-999c9cf87ab8: entering follower state: follower="Node at 010ab5e3-5c9d-0c0a-4b56-999c9cf87ab8 [Follower]" leader-address= leader-id= --- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s) === RUN TestRaft_RecoverCluster_NoState 2025-09-14T21:47:13.980Z [INFO] server-dca61606-27e1-5ba2-40be-873e0390e117: initial configuration: index=0 servers=[] 2025-09-14T21:47:13.980Z [INFO] server-dca61606-27e1-5ba2-40be-873e0390e117: entering follower state: follower="Node at dca61606-27e1-5ba2-40be-873e0390e117 [Shutdown]" leader-address= leader-id= --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster === RUN TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs 2025-09-14T21:47:13.982Z [INFO] server-a6b06da8-0bee-632f-238f-d5b041579a7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6b06da8-0bee-632f-238f-d5b041579a7c Address:a6b06da8-0bee-632f-238f-d5b041579a7c} {Suffrage:Voter ID:server-e213828d-de47-6ab8-e652-9cb88582cd8d Address:e213828d-de47-6ab8-e652-9cb88582cd8d} {Suffrage:Voter ID:server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9 Address:b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9}]" 2025-09-14T21:47:13.982Z [INFO] server-a6b06da8-0bee-632f-238f-d5b041579a7c: entering follower state: follower="Node at a6b06da8-0bee-632f-238f-d5b041579a7c [Follower]" leader-address= leader-id= 2025-09-14T21:47:13.982Z [INFO] server-e213828d-de47-6ab8-e652-9cb88582cd8d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6b06da8-0bee-632f-238f-d5b041579a7c Address:a6b06da8-0bee-632f-238f-d5b041579a7c} {Suffrage:Voter ID:server-e213828d-de47-6ab8-e652-9cb88582cd8d Address:e213828d-de47-6ab8-e652-9cb88582cd8d} {Suffrage:Voter ID:server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9 Address:b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9}]" 2025-09-14T21:47:13.982Z [INFO] server-e213828d-de47-6ab8-e652-9cb88582cd8d: entering follower state: follower="Node at e213828d-de47-6ab8-e652-9cb88582cd8d [Follower]" leader-address= leader-id= 2025-09-14T21:47:13.982Z [INFO] server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6b06da8-0bee-632f-238f-d5b041579a7c Address:a6b06da8-0bee-632f-238f-d5b041579a7c} {Suffrage:Voter ID:server-e213828d-de47-6ab8-e652-9cb88582cd8d Address:e213828d-de47-6ab8-e652-9cb88582cd8d} {Suffrage:Voter ID:server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9 Address:b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9}]" 2025-09-14T21:47:13.982Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:13.983Z [INFO] server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9: entering follower state: follower="Node at b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9 [Follower]" leader-address= leader-id= 2025-09-14T21:47:14.058Z [WARN] server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:14.058Z [INFO] server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9: entering candidate state: node="Node at b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9 [Candidate]" term=2 2025-09-14T21:47:14.058Z [INFO] server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9: election won: term=2 tally=2 2025-09-14T21:47:14.058Z [INFO] server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9: entering leader state: leader="Node at b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9 [Leader]" 2025-09-14T21:47:14.058Z [INFO] server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9: added peer, starting replication: peer=server-a6b06da8-0bee-632f-238f-d5b041579a7c 2025-09-14T21:47:14.058Z [INFO] server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9: added peer, starting replication: peer=server-e213828d-de47-6ab8-e652-9cb88582cd8d 2025-09-14T21:47:14.058Z [INFO] server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9: pipelining replication: peer="{Voter server-e213828d-de47-6ab8-e652-9cb88582cd8d e213828d-de47-6ab8-e652-9cb88582cd8d}" 2025-09-14T21:47:14.058Z [INFO] server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9: pipelining replication: peer="{Voter server-a6b06da8-0bee-632f-238f-d5b041579a7c a6b06da8-0bee-632f-238f-d5b041579a7c}" 2025-09-14T21:47:14.164Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:14.058717612 +0000 UTC m=+4.626595878 (1 nodes), 75.718754ms from start of poll, 76.215819ms from cluster start. Timeout at 2025-09-14 21:47:14.163720723 +0000 UTC m=+4.731598999, 105.003121ms after stability 2025-09-14T21:47:14.164Z [INFO] server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9: aborting pipeline replication: peer="{Voter server-a6b06da8-0bee-632f-238f-d5b041579a7c a6b06da8-0bee-632f-238f-d5b041579a7c}" 2025-09-14T21:47:14.164Z [INFO] server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9: aborting pipeline replication: peer="{Voter server-e213828d-de47-6ab8-e652-9cb88582cd8d e213828d-de47-6ab8-e652-9cb88582cd8d}" 2025-09-14T21:47:14.164Z [INFO] creating new snapshot: path=/tmp/raft3954091077/snapshots/2-2-1757886434164.tmp 2025-09-14T21:47:14.165Z [INFO] server-a6b06da8-0bee-632f-238f-d5b041579a7c: starting restore from snapshot: id=2-2-1757886434164 last-index=2 last-term=2 size-in-bytes=1 2025-09-14T21:47:14.165Z [INFO] server-a6b06da8-0bee-632f-238f-d5b041579a7c: snapshot restore progress: id=2-2-1757886434164 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2025-09-14T21:47:14.165Z [INFO] server-a6b06da8-0bee-632f-238f-d5b041579a7c: restored from snapshot: id=2-2-1757886434164 last-index=2 last-term=2 size-in-bytes=1 2025-09-14T21:47:14.165Z [INFO] server-a6b06da8-0bee-632f-238f-d5b041579a7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6b06da8-0bee-632f-238f-d5b041579a7c Address:a6b06da8-0bee-632f-238f-d5b041579a7c} {Suffrage:Voter ID:server-e213828d-de47-6ab8-e652-9cb88582cd8d Address:e213828d-de47-6ab8-e652-9cb88582cd8d} {Suffrage:Voter ID:server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9 Address:b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9}]" 2025-09-14T21:47:14.165Z [INFO] creating new snapshot: path=/tmp/raft2709868940/snapshots/2-2-1757886434165.tmp 2025-09-14T21:47:14.165Z [INFO] server-a6b06da8-0bee-632f-238f-d5b041579a7c: entering follower state: follower="Node at a6b06da8-0bee-632f-238f-d5b041579a7c [Follower]" leader-address= leader-id= 2025-09-14T21:47:14.166Z [INFO] server-e213828d-de47-6ab8-e652-9cb88582cd8d: starting restore from snapshot: id=2-2-1757886434165 last-index=2 last-term=2 size-in-bytes=1 2025-09-14T21:47:14.166Z [INFO] server-e213828d-de47-6ab8-e652-9cb88582cd8d: snapshot restore progress: id=2-2-1757886434165 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2025-09-14T21:47:14.166Z [INFO] server-e213828d-de47-6ab8-e652-9cb88582cd8d: restored from snapshot: id=2-2-1757886434165 last-index=2 last-term=2 size-in-bytes=1 2025-09-14T21:47:14.166Z [INFO] server-e213828d-de47-6ab8-e652-9cb88582cd8d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6b06da8-0bee-632f-238f-d5b041579a7c Address:a6b06da8-0bee-632f-238f-d5b041579a7c} {Suffrage:Voter ID:server-e213828d-de47-6ab8-e652-9cb88582cd8d Address:e213828d-de47-6ab8-e652-9cb88582cd8d} {Suffrage:Voter ID:server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9 Address:b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9}]" 2025-09-14T21:47:14.166Z [INFO] server-e213828d-de47-6ab8-e652-9cb88582cd8d: entering follower state: follower="Node at e213828d-de47-6ab8-e652-9cb88582cd8d [Follower]" leader-address= leader-id= 2025-09-14T21:47:14.166Z [INFO] creating new snapshot: path=/tmp/raft3889863724/snapshots/2-2-1757886434166.tmp 2025-09-14T21:47:14.166Z [INFO] server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9: starting restore from snapshot: id=2-2-1757886434166 last-index=2 last-term=2 size-in-bytes=1 2025-09-14T21:47:14.167Z [INFO] server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9: snapshot restore progress: id=2-2-1757886434166 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2025-09-14T21:47:14.167Z [INFO] server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9: restored from snapshot: id=2-2-1757886434166 last-index=2 last-term=2 size-in-bytes=1 2025-09-14T21:47:14.167Z [INFO] server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6b06da8-0bee-632f-238f-d5b041579a7c Address:a6b06da8-0bee-632f-238f-d5b041579a7c} {Suffrage:Voter ID:server-e213828d-de47-6ab8-e652-9cb88582cd8d Address:e213828d-de47-6ab8-e652-9cb88582cd8d} {Suffrage:Voter ID:server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9 Address:b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9}]" 2025-09-14T21:47:14.167Z [INFO] server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9: entering follower state: follower="Node at b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9 [Follower]" leader-address= leader-id= 2025-09-14T21:47:14.233Z [WARN] server-e213828d-de47-6ab8-e652-9cb88582cd8d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:14.233Z [INFO] server-e213828d-de47-6ab8-e652-9cb88582cd8d: entering candidate state: node="Node at e213828d-de47-6ab8-e652-9cb88582cd8d [Candidate]" term=3 2025-09-14T21:47:14.233Z [INFO] server-e213828d-de47-6ab8-e652-9cb88582cd8d: election won: term=3 tally=2 2025-09-14T21:47:14.233Z [INFO] server-e213828d-de47-6ab8-e652-9cb88582cd8d: entering leader state: leader="Node at e213828d-de47-6ab8-e652-9cb88582cd8d [Leader]" 2025-09-14T21:47:14.233Z [INFO] server-e213828d-de47-6ab8-e652-9cb88582cd8d: added peer, starting replication: peer=server-a6b06da8-0bee-632f-238f-d5b041579a7c 2025-09-14T21:47:14.233Z [INFO] server-e213828d-de47-6ab8-e652-9cb88582cd8d: added peer, starting replication: peer=server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9 2025-09-14T21:47:14.233Z [INFO] server-e213828d-de47-6ab8-e652-9cb88582cd8d: pipelining replication: peer="{Voter server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9 b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9}" 2025-09-14T21:47:14.233Z [INFO] server-e213828d-de47-6ab8-e652-9cb88582cd8d: pipelining replication: peer="{Voter server-a6b06da8-0bee-632f-238f-d5b041579a7c a6b06da8-0bee-632f-238f-d5b041579a7c}" 2025-09-14T21:47:14.482Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:14.588Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:14.482783748 +0000 UTC m=+5.050662023 (1 nodes), 36.215µs from start of poll, 500.281964ms from cluster start. Timeout at 2025-09-14 21:47:14.587789854 +0000 UTC m=+5.155668112, 105.006089ms after stability 2025-09-14T21:47:14.588Z [INFO] server-e213828d-de47-6ab8-e652-9cb88582cd8d: aborting pipeline replication: peer="{Voter server-b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9 b0c89cdc-2e6f-919e-5ec3-d00ff3f461c9}" 2025-09-14T21:47:14.588Z [INFO] server-e213828d-de47-6ab8-e652-9cb88582cd8d: aborting pipeline replication: peer="{Voter server-a6b06da8-0bee-632f-238f-d5b041579a7c a6b06da8-0bee-632f-238f-d5b041579a7c}" === RUN TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs 2025-09-14T21:47:14.589Z [INFO] server-ecf4e414-25bb-579f-a7c1-8868f93e1939: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecf4e414-25bb-579f-a7c1-8868f93e1939 Address:ecf4e414-25bb-579f-a7c1-8868f93e1939} {Suffrage:Voter ID:server-5abfc239-f902-81f8-852a-94949335f35c Address:5abfc239-f902-81f8-852a-94949335f35c} {Suffrage:Voter ID:server-17a02552-ee36-bfc8-4b24-0d1514701775 Address:17a02552-ee36-bfc8-4b24-0d1514701775}]" 2025-09-14T21:47:14.589Z [INFO] server-ecf4e414-25bb-579f-a7c1-8868f93e1939: entering follower state: follower="Node at ecf4e414-25bb-579f-a7c1-8868f93e1939 [Follower]" leader-address= leader-id= 2025-09-14T21:47:14.590Z [INFO] server-5abfc239-f902-81f8-852a-94949335f35c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecf4e414-25bb-579f-a7c1-8868f93e1939 Address:ecf4e414-25bb-579f-a7c1-8868f93e1939} {Suffrage:Voter ID:server-5abfc239-f902-81f8-852a-94949335f35c Address:5abfc239-f902-81f8-852a-94949335f35c} {Suffrage:Voter ID:server-17a02552-ee36-bfc8-4b24-0d1514701775 Address:17a02552-ee36-bfc8-4b24-0d1514701775}]" 2025-09-14T21:47:14.590Z [INFO] server-5abfc239-f902-81f8-852a-94949335f35c: entering follower state: follower="Node at 5abfc239-f902-81f8-852a-94949335f35c [Follower]" leader-address= leader-id= 2025-09-14T21:47:14.590Z [INFO] server-17a02552-ee36-bfc8-4b24-0d1514701775: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecf4e414-25bb-579f-a7c1-8868f93e1939 Address:ecf4e414-25bb-579f-a7c1-8868f93e1939} {Suffrage:Voter ID:server-5abfc239-f902-81f8-852a-94949335f35c Address:5abfc239-f902-81f8-852a-94949335f35c} {Suffrage:Voter ID:server-17a02552-ee36-bfc8-4b24-0d1514701775 Address:17a02552-ee36-bfc8-4b24-0d1514701775}]" 2025-09-14T21:47:14.590Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:14.590Z [INFO] server-17a02552-ee36-bfc8-4b24-0d1514701775: entering follower state: follower="Node at 17a02552-ee36-bfc8-4b24-0d1514701775 [Follower]" leader-address= leader-id= 2025-09-14T21:47:14.641Z [WARN] server-5abfc239-f902-81f8-852a-94949335f35c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:14.641Z [INFO] server-5abfc239-f902-81f8-852a-94949335f35c: entering candidate state: node="Node at 5abfc239-f902-81f8-852a-94949335f35c [Candidate]" term=2 2025-09-14T21:47:14.641Z [INFO] server-5abfc239-f902-81f8-852a-94949335f35c: election won: term=2 tally=2 2025-09-14T21:47:14.641Z [INFO] server-5abfc239-f902-81f8-852a-94949335f35c: entering leader state: leader="Node at 5abfc239-f902-81f8-852a-94949335f35c [Leader]" 2025-09-14T21:47:14.641Z [INFO] server-5abfc239-f902-81f8-852a-94949335f35c: added peer, starting replication: peer=server-ecf4e414-25bb-579f-a7c1-8868f93e1939 2025-09-14T21:47:14.641Z [INFO] server-5abfc239-f902-81f8-852a-94949335f35c: added peer, starting replication: peer=server-17a02552-ee36-bfc8-4b24-0d1514701775 2025-09-14T21:47:14.641Z [INFO] server-5abfc239-f902-81f8-852a-94949335f35c: pipelining replication: peer="{Voter server-17a02552-ee36-bfc8-4b24-0d1514701775 17a02552-ee36-bfc8-4b24-0d1514701775}" 2025-09-14T21:47:14.642Z [INFO] server-5abfc239-f902-81f8-852a-94949335f35c: pipelining replication: peer="{Voter server-ecf4e414-25bb-579f-a7c1-8868f93e1939 ecf4e414-25bb-579f-a7c1-8868f93e1939}" 2025-09-14T21:47:14.747Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:14.641912619 +0000 UTC m=+5.209790878 (1 nodes), 51.796829ms from start of poll, 52.093153ms from cluster start. Timeout at 2025-09-14 21:47:14.746917592 +0000 UTC m=+5.314795851, 105.004973ms after stability 2025-09-14T21:47:14.748Z [INFO] server-5abfc239-f902-81f8-852a-94949335f35c: aborting pipeline replication: peer="{Voter server-17a02552-ee36-bfc8-4b24-0d1514701775 17a02552-ee36-bfc8-4b24-0d1514701775}" 2025-09-14T21:47:14.748Z [INFO] server-5abfc239-f902-81f8-852a-94949335f35c: aborting pipeline replication: peer="{Voter server-ecf4e414-25bb-579f-a7c1-8868f93e1939 ecf4e414-25bb-579f-a7c1-8868f93e1939}" 2025-09-14T21:47:14.748Z [INFO] creating new snapshot: path=/tmp/raft2845246429/snapshots/2-6-1757886434748.tmp 2025-09-14T21:47:14.749Z [INFO] server-ecf4e414-25bb-579f-a7c1-8868f93e1939: starting restore from snapshot: id=2-6-1757886434748 last-index=6 last-term=2 size-in-bytes=25 2025-09-14T21:47:14.749Z [INFO] server-ecf4e414-25bb-579f-a7c1-8868f93e1939: snapshot restore progress: id=2-6-1757886434748 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2025-09-14T21:47:14.749Z [INFO] server-ecf4e414-25bb-579f-a7c1-8868f93e1939: restored from snapshot: id=2-6-1757886434748 last-index=6 last-term=2 size-in-bytes=25 2025-09-14T21:47:14.749Z [INFO] server-ecf4e414-25bb-579f-a7c1-8868f93e1939: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecf4e414-25bb-579f-a7c1-8868f93e1939 Address:ecf4e414-25bb-579f-a7c1-8868f93e1939} {Suffrage:Voter ID:server-5abfc239-f902-81f8-852a-94949335f35c Address:5abfc239-f902-81f8-852a-94949335f35c} {Suffrage:Voter ID:server-17a02552-ee36-bfc8-4b24-0d1514701775 Address:17a02552-ee36-bfc8-4b24-0d1514701775}]" 2025-09-14T21:47:14.749Z [INFO] server-ecf4e414-25bb-579f-a7c1-8868f93e1939: entering follower state: follower="Node at ecf4e414-25bb-579f-a7c1-8868f93e1939 [Follower]" leader-address= leader-id= 2025-09-14T21:47:14.749Z [INFO] creating new snapshot: path=/tmp/raft984182829/snapshots/2-6-1757886434749.tmp 2025-09-14T21:47:14.749Z [INFO] server-5abfc239-f902-81f8-852a-94949335f35c: starting restore from snapshot: id=2-6-1757886434749 last-index=6 last-term=2 size-in-bytes=25 2025-09-14T21:47:14.750Z [INFO] server-5abfc239-f902-81f8-852a-94949335f35c: snapshot restore progress: id=2-6-1757886434749 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2025-09-14T21:47:14.750Z [INFO] server-5abfc239-f902-81f8-852a-94949335f35c: restored from snapshot: id=2-6-1757886434749 last-index=6 last-term=2 size-in-bytes=25 2025-09-14T21:47:14.750Z [INFO] server-5abfc239-f902-81f8-852a-94949335f35c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecf4e414-25bb-579f-a7c1-8868f93e1939 Address:ecf4e414-25bb-579f-a7c1-8868f93e1939} {Suffrage:Voter ID:server-5abfc239-f902-81f8-852a-94949335f35c Address:5abfc239-f902-81f8-852a-94949335f35c} {Suffrage:Voter ID:server-17a02552-ee36-bfc8-4b24-0d1514701775 Address:17a02552-ee36-bfc8-4b24-0d1514701775}]" 2025-09-14T21:47:14.751Z [INFO] server-5abfc239-f902-81f8-852a-94949335f35c: entering follower state: follower="Node at 5abfc239-f902-81f8-852a-94949335f35c [Follower]" leader-address= leader-id= 2025-09-14T21:47:14.751Z [INFO] creating new snapshot: path=/tmp/raft3330325067/snapshots/2-6-1757886434751.tmp 2025-09-14T21:47:14.751Z [INFO] server-17a02552-ee36-bfc8-4b24-0d1514701775: starting restore from snapshot: id=2-6-1757886434751 last-index=6 last-term=2 size-in-bytes=25 2025-09-14T21:47:14.751Z [INFO] server-17a02552-ee36-bfc8-4b24-0d1514701775: snapshot restore progress: id=2-6-1757886434751 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2025-09-14T21:47:14.751Z [INFO] server-17a02552-ee36-bfc8-4b24-0d1514701775: restored from snapshot: id=2-6-1757886434751 last-index=6 last-term=2 size-in-bytes=25 2025-09-14T21:47:14.751Z [INFO] server-17a02552-ee36-bfc8-4b24-0d1514701775: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecf4e414-25bb-579f-a7c1-8868f93e1939 Address:ecf4e414-25bb-579f-a7c1-8868f93e1939} {Suffrage:Voter ID:server-5abfc239-f902-81f8-852a-94949335f35c Address:5abfc239-f902-81f8-852a-94949335f35c} {Suffrage:Voter ID:server-17a02552-ee36-bfc8-4b24-0d1514701775 Address:17a02552-ee36-bfc8-4b24-0d1514701775}]" 2025-09-14T21:47:14.751Z [INFO] server-17a02552-ee36-bfc8-4b24-0d1514701775: entering follower state: follower="Node at 17a02552-ee36-bfc8-4b24-0d1514701775 [Follower]" leader-address= leader-id= 2025-09-14T21:47:14.824Z [WARN] server-ecf4e414-25bb-579f-a7c1-8868f93e1939: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:14.824Z [INFO] server-ecf4e414-25bb-579f-a7c1-8868f93e1939: entering candidate state: node="Node at ecf4e414-25bb-579f-a7c1-8868f93e1939 [Candidate]" term=3 2025-09-14T21:47:14.824Z [INFO] server-ecf4e414-25bb-579f-a7c1-8868f93e1939: election won: term=3 tally=2 2025-09-14T21:47:14.824Z [INFO] server-ecf4e414-25bb-579f-a7c1-8868f93e1939: entering leader state: leader="Node at ecf4e414-25bb-579f-a7c1-8868f93e1939 [Leader]" 2025-09-14T21:47:14.824Z [INFO] server-ecf4e414-25bb-579f-a7c1-8868f93e1939: added peer, starting replication: peer=server-5abfc239-f902-81f8-852a-94949335f35c 2025-09-14T21:47:14.824Z [INFO] server-ecf4e414-25bb-579f-a7c1-8868f93e1939: added peer, starting replication: peer=server-17a02552-ee36-bfc8-4b24-0d1514701775 2025-09-14T21:47:14.824Z [INFO] server-ecf4e414-25bb-579f-a7c1-8868f93e1939: pipelining replication: peer="{Voter server-5abfc239-f902-81f8-852a-94949335f35c 5abfc239-f902-81f8-852a-94949335f35c}" 2025-09-14T21:47:14.824Z [INFO] server-ecf4e414-25bb-579f-a7c1-8868f93e1939: pipelining replication: peer="{Voter server-17a02552-ee36-bfc8-4b24-0d1514701775 17a02552-ee36-bfc8-4b24-0d1514701775}" 2025-09-14T21:47:15.067Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:15.172Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:15.067839651 +0000 UTC m=+5.635717924 (1 nodes), 37.025µs from start of poll, 478.020199ms from cluster start. Timeout at 2025-09-14 21:47:15.172842533 +0000 UTC m=+5.740720814, 105.00289ms after stability 2025-09-14T21:47:15.172Z [INFO] server-ecf4e414-25bb-579f-a7c1-8868f93e1939: aborting pipeline replication: peer="{Voter server-17a02552-ee36-bfc8-4b24-0d1514701775 17a02552-ee36-bfc8-4b24-0d1514701775}" 2025-09-14T21:47:15.173Z [INFO] server-ecf4e414-25bb-579f-a7c1-8868f93e1939: aborting pipeline replication: peer="{Voter server-5abfc239-f902-81f8-852a-94949335f35c 5abfc239-f902-81f8-852a-94949335f35c}" === RUN TestRaft_RecoverCluster/snapshot,_with_trailing_logs 2025-09-14T21:47:15.174Z [INFO] server-45a945a2-c431-35d7-757f-247ae77536f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45a945a2-c431-35d7-757f-247ae77536f5 Address:45a945a2-c431-35d7-757f-247ae77536f5} {Suffrage:Voter ID:server-da1064af-9360-2c52-6c2b-246e2e072a46 Address:da1064af-9360-2c52-6c2b-246e2e072a46} {Suffrage:Voter ID:server-362735e9-0fe0-ac05-3922-9621eaabf162 Address:362735e9-0fe0-ac05-3922-9621eaabf162}]" 2025-09-14T21:47:15.174Z [INFO] server-45a945a2-c431-35d7-757f-247ae77536f5: entering follower state: follower="Node at 45a945a2-c431-35d7-757f-247ae77536f5 [Follower]" leader-address= leader-id= 2025-09-14T21:47:15.174Z [INFO] server-da1064af-9360-2c52-6c2b-246e2e072a46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45a945a2-c431-35d7-757f-247ae77536f5 Address:45a945a2-c431-35d7-757f-247ae77536f5} {Suffrage:Voter ID:server-da1064af-9360-2c52-6c2b-246e2e072a46 Address:da1064af-9360-2c52-6c2b-246e2e072a46} {Suffrage:Voter ID:server-362735e9-0fe0-ac05-3922-9621eaabf162 Address:362735e9-0fe0-ac05-3922-9621eaabf162}]" 2025-09-14T21:47:15.174Z [INFO] server-da1064af-9360-2c52-6c2b-246e2e072a46: entering follower state: follower="Node at da1064af-9360-2c52-6c2b-246e2e072a46 [Follower]" leader-address= leader-id= 2025-09-14T21:47:15.175Z [INFO] server-362735e9-0fe0-ac05-3922-9621eaabf162: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45a945a2-c431-35d7-757f-247ae77536f5 Address:45a945a2-c431-35d7-757f-247ae77536f5} {Suffrage:Voter ID:server-da1064af-9360-2c52-6c2b-246e2e072a46 Address:da1064af-9360-2c52-6c2b-246e2e072a46} {Suffrage:Voter ID:server-362735e9-0fe0-ac05-3922-9621eaabf162 Address:362735e9-0fe0-ac05-3922-9621eaabf162}]" 2025-09-14T21:47:15.175Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:15.175Z [INFO] server-362735e9-0fe0-ac05-3922-9621eaabf162: entering follower state: follower="Node at 362735e9-0fe0-ac05-3922-9621eaabf162 [Follower]" leader-address= leader-id= 2025-09-14T21:47:15.248Z [WARN] server-362735e9-0fe0-ac05-3922-9621eaabf162: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:15.248Z [INFO] server-362735e9-0fe0-ac05-3922-9621eaabf162: entering candidate state: node="Node at 362735e9-0fe0-ac05-3922-9621eaabf162 [Candidate]" term=2 2025-09-14T21:47:15.248Z [INFO] server-362735e9-0fe0-ac05-3922-9621eaabf162: election won: term=2 tally=2 2025-09-14T21:47:15.248Z [INFO] server-362735e9-0fe0-ac05-3922-9621eaabf162: entering leader state: leader="Node at 362735e9-0fe0-ac05-3922-9621eaabf162 [Leader]" 2025-09-14T21:47:15.248Z [INFO] server-362735e9-0fe0-ac05-3922-9621eaabf162: added peer, starting replication: peer=server-45a945a2-c431-35d7-757f-247ae77536f5 2025-09-14T21:47:15.248Z [INFO] server-362735e9-0fe0-ac05-3922-9621eaabf162: added peer, starting replication: peer=server-da1064af-9360-2c52-6c2b-246e2e072a46 2025-09-14T21:47:15.248Z [INFO] server-362735e9-0fe0-ac05-3922-9621eaabf162: pipelining replication: peer="{Voter server-da1064af-9360-2c52-6c2b-246e2e072a46 da1064af-9360-2c52-6c2b-246e2e072a46}" 2025-09-14T21:47:15.248Z [INFO] server-362735e9-0fe0-ac05-3922-9621eaabf162: pipelining replication: peer="{Voter server-45a945a2-c431-35d7-757f-247ae77536f5 45a945a2-c431-35d7-757f-247ae77536f5}" 2025-09-14T21:47:15.353Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:15.248334583 +0000 UTC m=+5.816212858 (1 nodes), 72.490722ms from start of poll, 73.807516ms from cluster start. Timeout at 2025-09-14 21:47:15.353335041 +0000 UTC m=+5.921213303, 105.000445ms after stability 2025-09-14T21:47:15.355Z [INFO] server-362735e9-0fe0-ac05-3922-9621eaabf162: aborting pipeline replication: peer="{Voter server-da1064af-9360-2c52-6c2b-246e2e072a46 da1064af-9360-2c52-6c2b-246e2e072a46}" 2025-09-14T21:47:15.355Z [INFO] server-362735e9-0fe0-ac05-3922-9621eaabf162: aborting pipeline replication: peer="{Voter server-45a945a2-c431-35d7-757f-247ae77536f5 45a945a2-c431-35d7-757f-247ae77536f5}" 2025-09-14T21:47:15.355Z [INFO] creating new snapshot: path=/tmp/raft764529137/snapshots/2-27-1757886435355.tmp 2025-09-14T21:47:15.355Z [INFO] server-45a945a2-c431-35d7-757f-247ae77536f5: starting restore from snapshot: id=2-27-1757886435355 last-index=27 last-term=2 size-in-bytes=168 2025-09-14T21:47:15.355Z [INFO] server-45a945a2-c431-35d7-757f-247ae77536f5: snapshot restore progress: id=2-27-1757886435355 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2025-09-14T21:47:15.356Z [INFO] server-45a945a2-c431-35d7-757f-247ae77536f5: restored from snapshot: id=2-27-1757886435355 last-index=27 last-term=2 size-in-bytes=168 2025-09-14T21:47:15.356Z [INFO] server-45a945a2-c431-35d7-757f-247ae77536f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45a945a2-c431-35d7-757f-247ae77536f5 Address:45a945a2-c431-35d7-757f-247ae77536f5} {Suffrage:Voter ID:server-da1064af-9360-2c52-6c2b-246e2e072a46 Address:da1064af-9360-2c52-6c2b-246e2e072a46} {Suffrage:Voter ID:server-362735e9-0fe0-ac05-3922-9621eaabf162 Address:362735e9-0fe0-ac05-3922-9621eaabf162}]" 2025-09-14T21:47:15.356Z [INFO] creating new snapshot: path=/tmp/raft2613420928/snapshots/2-27-1757886435356.tmp 2025-09-14T21:47:15.356Z [INFO] server-45a945a2-c431-35d7-757f-247ae77536f5: entering follower state: follower="Node at 45a945a2-c431-35d7-757f-247ae77536f5 [Follower]" leader-address= leader-id= 2025-09-14T21:47:15.357Z [INFO] server-da1064af-9360-2c52-6c2b-246e2e072a46: starting restore from snapshot: id=2-27-1757886435356 last-index=27 last-term=2 size-in-bytes=168 2025-09-14T21:47:15.357Z [INFO] server-da1064af-9360-2c52-6c2b-246e2e072a46: snapshot restore progress: id=2-27-1757886435356 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2025-09-14T21:47:15.357Z [INFO] server-da1064af-9360-2c52-6c2b-246e2e072a46: restored from snapshot: id=2-27-1757886435356 last-index=27 last-term=2 size-in-bytes=168 2025-09-14T21:47:15.357Z [INFO] server-da1064af-9360-2c52-6c2b-246e2e072a46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45a945a2-c431-35d7-757f-247ae77536f5 Address:45a945a2-c431-35d7-757f-247ae77536f5} {Suffrage:Voter ID:server-da1064af-9360-2c52-6c2b-246e2e072a46 Address:da1064af-9360-2c52-6c2b-246e2e072a46} {Suffrage:Voter ID:server-362735e9-0fe0-ac05-3922-9621eaabf162 Address:362735e9-0fe0-ac05-3922-9621eaabf162}]" 2025-09-14T21:47:15.357Z [INFO] creating new snapshot: path=/tmp/raft49880398/snapshots/2-27-1757886435357.tmp 2025-09-14T21:47:15.357Z [INFO] server-da1064af-9360-2c52-6c2b-246e2e072a46: entering follower state: follower="Node at da1064af-9360-2c52-6c2b-246e2e072a46 [Follower]" leader-address= leader-id= 2025-09-14T21:47:15.358Z [INFO] server-362735e9-0fe0-ac05-3922-9621eaabf162: starting restore from snapshot: id=2-27-1757886435357 last-index=27 last-term=2 size-in-bytes=168 2025-09-14T21:47:15.358Z [INFO] server-362735e9-0fe0-ac05-3922-9621eaabf162: snapshot restore progress: id=2-27-1757886435357 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2025-09-14T21:47:15.358Z [INFO] server-362735e9-0fe0-ac05-3922-9621eaabf162: restored from snapshot: id=2-27-1757886435357 last-index=27 last-term=2 size-in-bytes=168 2025-09-14T21:47:15.358Z [INFO] server-362735e9-0fe0-ac05-3922-9621eaabf162: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45a945a2-c431-35d7-757f-247ae77536f5 Address:45a945a2-c431-35d7-757f-247ae77536f5} {Suffrage:Voter ID:server-da1064af-9360-2c52-6c2b-246e2e072a46 Address:da1064af-9360-2c52-6c2b-246e2e072a46} {Suffrage:Voter ID:server-362735e9-0fe0-ac05-3922-9621eaabf162 Address:362735e9-0fe0-ac05-3922-9621eaabf162}]" 2025-09-14T21:47:15.358Z [INFO] server-362735e9-0fe0-ac05-3922-9621eaabf162: entering follower state: follower="Node at 362735e9-0fe0-ac05-3922-9621eaabf162 [Follower]" leader-address= leader-id= 2025-09-14T21:47:15.438Z [WARN] server-da1064af-9360-2c52-6c2b-246e2e072a46: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:15.438Z [INFO] server-da1064af-9360-2c52-6c2b-246e2e072a46: entering candidate state: node="Node at da1064af-9360-2c52-6c2b-246e2e072a46 [Candidate]" term=3 2025-09-14T21:47:15.438Z [INFO] server-da1064af-9360-2c52-6c2b-246e2e072a46: election won: term=3 tally=2 2025-09-14T21:47:15.438Z [INFO] server-da1064af-9360-2c52-6c2b-246e2e072a46: entering leader state: leader="Node at da1064af-9360-2c52-6c2b-246e2e072a46 [Leader]" 2025-09-14T21:47:15.438Z [INFO] server-da1064af-9360-2c52-6c2b-246e2e072a46: added peer, starting replication: peer=server-45a945a2-c431-35d7-757f-247ae77536f5 2025-09-14T21:47:15.438Z [INFO] server-da1064af-9360-2c52-6c2b-246e2e072a46: added peer, starting replication: peer=server-362735e9-0fe0-ac05-3922-9621eaabf162 2025-09-14T21:47:15.438Z [INFO] server-da1064af-9360-2c52-6c2b-246e2e072a46: pipelining replication: peer="{Voter server-362735e9-0fe0-ac05-3922-9621eaabf162 362735e9-0fe0-ac05-3922-9621eaabf162}" 2025-09-14T21:47:15.439Z [INFO] server-da1064af-9360-2c52-6c2b-246e2e072a46: pipelining replication: peer="{Voter server-45a945a2-c431-35d7-757f-247ae77536f5 45a945a2-c431-35d7-757f-247ae77536f5}" 2025-09-14T21:47:15.673Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:15.779Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:15.673918388 +0000 UTC m=+6.241796662 (1 nodes), 28.954µs from start of poll, 499.39132ms from cluster start. Timeout at 2025-09-14 21:47:15.778924856 +0000 UTC m=+6.346803133, 105.006471ms after stability 2025-09-14T21:47:15.779Z [INFO] server-da1064af-9360-2c52-6c2b-246e2e072a46: aborting pipeline replication: peer="{Voter server-45a945a2-c431-35d7-757f-247ae77536f5 45a945a2-c431-35d7-757f-247ae77536f5}" 2025-09-14T21:47:15.779Z [INFO] server-da1064af-9360-2c52-6c2b-246e2e072a46: aborting pipeline replication: peer="{Voter server-362735e9-0fe0-ac05-3922-9621eaabf162 362735e9-0fe0-ac05-3922-9621eaabf162}" --- PASS: TestRaft_RecoverCluster (1.80s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.61s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.58s) --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.61s) === RUN TestRaft_HasExistingState raft_test.go:258: Skipping unreliable test --- SKIP: TestRaft_HasExistingState (0.00s) === RUN TestRaft_SingleNode 2025-09-14T21:47:15.781Z [INFO] server-a6b511b8-4d48-d46e-8867-3130a2c8689e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6b511b8-4d48-d46e-8867-3130a2c8689e Address:a6b511b8-4d48-d46e-8867-3130a2c8689e}]" 2025-09-14T21:47:15.781Z [INFO] server-a6b511b8-4d48-d46e-8867-3130a2c8689e: entering follower state: follower="Node at a6b511b8-4d48-d46e-8867-3130a2c8689e [Follower]" leader-address= leader-id= 2025-09-14T21:47:15.844Z [WARN] server-a6b511b8-4d48-d46e-8867-3130a2c8689e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:15.844Z [INFO] server-a6b511b8-4d48-d46e-8867-3130a2c8689e: entering candidate state: node="Node at a6b511b8-4d48-d46e-8867-3130a2c8689e [Candidate]" term=2 2025-09-14T21:47:15.844Z [INFO] server-a6b511b8-4d48-d46e-8867-3130a2c8689e: election won: term=2 tally=1 2025-09-14T21:47:15.844Z [INFO] server-a6b511b8-4d48-d46e-8867-3130a2c8689e: entering leader state: leader="Node at a6b511b8-4d48-d46e-8867-3130a2c8689e [Leader]" --- PASS: TestRaft_SingleNode (0.06s) === RUN TestRaft_TripleNode 2025-09-14T21:47:15.845Z [INFO] server-9b5f1b33-ff7f-a294-f2ea-4e319633a3f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b5f1b33-ff7f-a294-f2ea-4e319633a3f5 Address:9b5f1b33-ff7f-a294-f2ea-4e319633a3f5} {Suffrage:Voter ID:server-2eda20e1-21ac-2e2d-ea21-f3f8e80ee402 Address:2eda20e1-21ac-2e2d-ea21-f3f8e80ee402} {Suffrage:Voter ID:server-155d9255-d0b9-dd93-9afb-6b45f027bac2 Address:155d9255-d0b9-dd93-9afb-6b45f027bac2}]" 2025-09-14T21:47:15.845Z [INFO] server-9b5f1b33-ff7f-a294-f2ea-4e319633a3f5: entering follower state: follower="Node at 9b5f1b33-ff7f-a294-f2ea-4e319633a3f5 [Follower]" leader-address= leader-id= 2025-09-14T21:47:15.845Z [INFO] server-2eda20e1-21ac-2e2d-ea21-f3f8e80ee402: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b5f1b33-ff7f-a294-f2ea-4e319633a3f5 Address:9b5f1b33-ff7f-a294-f2ea-4e319633a3f5} {Suffrage:Voter ID:server-2eda20e1-21ac-2e2d-ea21-f3f8e80ee402 Address:2eda20e1-21ac-2e2d-ea21-f3f8e80ee402} {Suffrage:Voter ID:server-155d9255-d0b9-dd93-9afb-6b45f027bac2 Address:155d9255-d0b9-dd93-9afb-6b45f027bac2}]" 2025-09-14T21:47:15.845Z [INFO] server-155d9255-d0b9-dd93-9afb-6b45f027bac2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b5f1b33-ff7f-a294-f2ea-4e319633a3f5 Address:9b5f1b33-ff7f-a294-f2ea-4e319633a3f5} {Suffrage:Voter ID:server-2eda20e1-21ac-2e2d-ea21-f3f8e80ee402 Address:2eda20e1-21ac-2e2d-ea21-f3f8e80ee402} {Suffrage:Voter ID:server-155d9255-d0b9-dd93-9afb-6b45f027bac2 Address:155d9255-d0b9-dd93-9afb-6b45f027bac2}]" 2025-09-14T21:47:15.845Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:15.845Z [INFO] server-2eda20e1-21ac-2e2d-ea21-f3f8e80ee402: entering follower state: follower="Node at 2eda20e1-21ac-2e2d-ea21-f3f8e80ee402 [Follower]" leader-address= leader-id= 2025-09-14T21:47:15.845Z [INFO] server-155d9255-d0b9-dd93-9afb-6b45f027bac2: entering follower state: follower="Node at 155d9255-d0b9-dd93-9afb-6b45f027bac2 [Follower]" leader-address= leader-id= 2025-09-14T21:47:15.912Z [WARN] server-9b5f1b33-ff7f-a294-f2ea-4e319633a3f5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:15.912Z [INFO] server-9b5f1b33-ff7f-a294-f2ea-4e319633a3f5: entering candidate state: node="Node at 9b5f1b33-ff7f-a294-f2ea-4e319633a3f5 [Candidate]" term=2 2025-09-14T21:47:15.912Z [INFO] server-9b5f1b33-ff7f-a294-f2ea-4e319633a3f5: election won: term=2 tally=2 2025-09-14T21:47:15.912Z [INFO] server-9b5f1b33-ff7f-a294-f2ea-4e319633a3f5: entering leader state: leader="Node at 9b5f1b33-ff7f-a294-f2ea-4e319633a3f5 [Leader]" 2025-09-14T21:47:15.912Z [INFO] server-9b5f1b33-ff7f-a294-f2ea-4e319633a3f5: added peer, starting replication: peer=server-2eda20e1-21ac-2e2d-ea21-f3f8e80ee402 2025-09-14T21:47:15.912Z [INFO] server-9b5f1b33-ff7f-a294-f2ea-4e319633a3f5: added peer, starting replication: peer=server-155d9255-d0b9-dd93-9afb-6b45f027bac2 2025-09-14T21:47:15.912Z [INFO] server-9b5f1b33-ff7f-a294-f2ea-4e319633a3f5: pipelining replication: peer="{Voter server-155d9255-d0b9-dd93-9afb-6b45f027bac2 155d9255-d0b9-dd93-9afb-6b45f027bac2}" 2025-09-14T21:47:15.913Z [INFO] server-9b5f1b33-ff7f-a294-f2ea-4e319633a3f5: pipelining replication: peer="{Voter server-2eda20e1-21ac-2e2d-ea21-f3f8e80ee402 2eda20e1-21ac-2e2d-ea21-f3f8e80ee402}" 2025-09-14T21:47:16.017Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:15.912421455 +0000 UTC m=+6.480299714 (2 nodes), 66.847614ms from start of poll, 67.184212ms from cluster start. Timeout at 2025-09-14 21:47:16.017424635 +0000 UTC m=+6.585302908, 105.003194ms after stability 2025-09-14T21:47:16.017Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:16.123Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:16.017845091 +0000 UTC m=+6.585723356 (1 nodes), 1.156µs from start of poll, 172.607854ms from cluster start. Timeout at 2025-09-14 21:47:16.122850912 +0000 UTC m=+6.690729171, 105.005815ms after stability 2025-09-14T21:47:16.134Z [INFO] server-9b5f1b33-ff7f-a294-f2ea-4e319633a3f5: aborting pipeline replication: peer="{Voter server-155d9255-d0b9-dd93-9afb-6b45f027bac2 155d9255-d0b9-dd93-9afb-6b45f027bac2}" 2025-09-14T21:47:16.134Z [INFO] server-9b5f1b33-ff7f-a294-f2ea-4e319633a3f5: aborting pipeline replication: peer="{Voter server-2eda20e1-21ac-2e2d-ea21-f3f8e80ee402 2eda20e1-21ac-2e2d-ea21-f3f8e80ee402}" --- PASS: TestRaft_TripleNode (0.29s) === RUN TestRaft_LeaderFail 2025-09-14T21:47:16.135Z [INFO] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79 Address:5ad9dad7-f73a-5bff-7081-5f98e7d3db79} {Suffrage:Voter ID:server-8ac88d7a-edb1-572a-cc98-450480183c73 Address:8ac88d7a-edb1-572a-cc98-450480183c73} {Suffrage:Voter ID:server-a45b6704-4289-c207-9b0c-e97f2bd51d08 Address:a45b6704-4289-c207-9b0c-e97f2bd51d08}]" 2025-09-14T21:47:16.135Z [INFO] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: entering follower state: follower="Node at 5ad9dad7-f73a-5bff-7081-5f98e7d3db79 [Follower]" leader-address= leader-id= 2025-09-14T21:47:16.135Z [INFO] server-8ac88d7a-edb1-572a-cc98-450480183c73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79 Address:5ad9dad7-f73a-5bff-7081-5f98e7d3db79} {Suffrage:Voter ID:server-8ac88d7a-edb1-572a-cc98-450480183c73 Address:8ac88d7a-edb1-572a-cc98-450480183c73} {Suffrage:Voter ID:server-a45b6704-4289-c207-9b0c-e97f2bd51d08 Address:a45b6704-4289-c207-9b0c-e97f2bd51d08}]" 2025-09-14T21:47:16.135Z [INFO] server-8ac88d7a-edb1-572a-cc98-450480183c73: entering follower state: follower="Node at 8ac88d7a-edb1-572a-cc98-450480183c73 [Follower]" leader-address= leader-id= 2025-09-14T21:47:16.135Z [INFO] server-a45b6704-4289-c207-9b0c-e97f2bd51d08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79 Address:5ad9dad7-f73a-5bff-7081-5f98e7d3db79} {Suffrage:Voter ID:server-8ac88d7a-edb1-572a-cc98-450480183c73 Address:8ac88d7a-edb1-572a-cc98-450480183c73} {Suffrage:Voter ID:server-a45b6704-4289-c207-9b0c-e97f2bd51d08 Address:a45b6704-4289-c207-9b0c-e97f2bd51d08}]" 2025-09-14T21:47:16.135Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:16.135Z [INFO] server-a45b6704-4289-c207-9b0c-e97f2bd51d08: entering follower state: follower="Node at a45b6704-4289-c207-9b0c-e97f2bd51d08 [Follower]" leader-address= leader-id= 2025-09-14T21:47:16.198Z [WARN] server-8ac88d7a-edb1-572a-cc98-450480183c73: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:16.198Z [INFO] server-8ac88d7a-edb1-572a-cc98-450480183c73: entering candidate state: node="Node at 8ac88d7a-edb1-572a-cc98-450480183c73 [Candidate]" term=2 2025-09-14T21:47:16.198Z [INFO] server-8ac88d7a-edb1-572a-cc98-450480183c73: election won: term=2 tally=2 2025-09-14T21:47:16.198Z [INFO] server-8ac88d7a-edb1-572a-cc98-450480183c73: entering leader state: leader="Node at 8ac88d7a-edb1-572a-cc98-450480183c73 [Leader]" 2025-09-14T21:47:16.198Z [INFO] server-8ac88d7a-edb1-572a-cc98-450480183c73: added peer, starting replication: peer=server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79 2025-09-14T21:47:16.198Z [INFO] server-8ac88d7a-edb1-572a-cc98-450480183c73: added peer, starting replication: peer=server-a45b6704-4289-c207-9b0c-e97f2bd51d08 2025-09-14T21:47:16.198Z [INFO] server-8ac88d7a-edb1-572a-cc98-450480183c73: pipelining replication: peer="{Voter server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79 5ad9dad7-f73a-5bff-7081-5f98e7d3db79}" 2025-09-14T21:47:16.198Z [INFO] server-8ac88d7a-edb1-572a-cc98-450480183c73: pipelining replication: peer="{Voter server-a45b6704-4289-c207-9b0c-e97f2bd51d08 a45b6704-4289-c207-9b0c-e97f2bd51d08}" 2025-09-14T21:47:16.304Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:16.19825383 +0000 UTC m=+6.766132093 (2 nodes), 62.527283ms from start of poll, 62.94091ms from cluster start. Timeout at 2025-09-14 21:47:16.303256164 +0000 UTC m=+6.871134424, 105.002331ms after stability 2025-09-14T21:47:16.304Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:16.409Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:16.304237039 +0000 UTC m=+6.872115316 (1 nodes), 593ns from start of poll, 168.924133ms from cluster start. Timeout at 2025-09-14 21:47:16.409243616 +0000 UTC m=+6.977121891, 105.006575ms after stability raft_test.go:378: [INFO] Disconnecting Node at 8ac88d7a-edb1-572a-cc98-450480183c73 [Leader] 2025-09-14T21:47:16.421Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to heartbeat to: peer=a45b6704-4289-c207-9b0c-e97f2bd51d08 backoff time=10ms error="failed to connect to peer: a45b6704-4289-c207-9b0c-e97f2bd51d08" 2025-09-14T21:47:16.421Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:16.422Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to heartbeat to: peer=5ad9dad7-f73a-5bff-7081-5f98e7d3db79 backoff time=10ms error="failed to connect to peer: 5ad9dad7-f73a-5bff-7081-5f98e7d3db79" 2025-09-14T21:47:16.427Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to pipeline appendEntries: peer="{Voter server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79 5ad9dad7-f73a-5bff-7081-5f98e7d3db79}" error="append pipeline closed" 2025-09-14T21:47:16.427Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to pipeline appendEntries: peer="{Voter server-a45b6704-4289-c207-9b0c-e97f2bd51d08 a45b6704-4289-c207-9b0c-e97f2bd51d08}" error="append pipeline closed" 2025-09-14T21:47:16.427Z [INFO] server-8ac88d7a-edb1-572a-cc98-450480183c73: aborting pipeline replication: peer="{Voter server-a45b6704-4289-c207-9b0c-e97f2bd51d08 a45b6704-4289-c207-9b0c-e97f2bd51d08}" 2025-09-14T21:47:16.427Z [INFO] server-8ac88d7a-edb1-572a-cc98-450480183c73: aborting pipeline replication: peer="{Voter server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79 5ad9dad7-f73a-5bff-7081-5f98e7d3db79}" 2025-09-14T21:47:16.432Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to appendEntries to: peer="{Voter server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79 5ad9dad7-f73a-5bff-7081-5f98e7d3db79}" error="failed to connect to peer: 5ad9dad7-f73a-5bff-7081-5f98e7d3db79" 2025-09-14T21:47:16.435Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to appendEntries to: peer="{Voter server-a45b6704-4289-c207-9b0c-e97f2bd51d08 a45b6704-4289-c207-9b0c-e97f2bd51d08}" error="failed to connect to peer: a45b6704-4289-c207-9b0c-e97f2bd51d08" 2025-09-14T21:47:16.439Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to heartbeat to: peer=5ad9dad7-f73a-5bff-7081-5f98e7d3db79 backoff time=10ms error="failed to connect to peer: 5ad9dad7-f73a-5bff-7081-5f98e7d3db79" 2025-09-14T21:47:16.441Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to heartbeat to: peer=a45b6704-4289-c207-9b0c-e97f2bd51d08 backoff time=10ms error="failed to connect to peer: a45b6704-4289-c207-9b0c-e97f2bd51d08" 2025-09-14T21:47:16.448Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to appendEntries to: peer="{Voter server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79 5ad9dad7-f73a-5bff-7081-5f98e7d3db79}" error="failed to connect to peer: 5ad9dad7-f73a-5bff-7081-5f98e7d3db79" 2025-09-14T21:47:16.455Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to appendEntries to: peer="{Voter server-a45b6704-4289-c207-9b0c-e97f2bd51d08 a45b6704-4289-c207-9b0c-e97f2bd51d08}" error="failed to connect to peer: a45b6704-4289-c207-9b0c-e97f2bd51d08" 2025-09-14T21:47:16.458Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to heartbeat to: peer=a45b6704-4289-c207-9b0c-e97f2bd51d08 backoff time=10ms error="failed to connect to peer: a45b6704-4289-c207-9b0c-e97f2bd51d08" 2025-09-14T21:47:16.460Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to heartbeat to: peer=5ad9dad7-f73a-5bff-7081-5f98e7d3db79 backoff time=10ms error="failed to connect to peer: 5ad9dad7-f73a-5bff-7081-5f98e7d3db79" 2025-09-14T21:47:16.466Z [WARN] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to contact: server-id=server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79 time=50.341286ms 2025-09-14T21:47:16.469Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to appendEntries to: peer="{Voter server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79 5ad9dad7-f73a-5bff-7081-5f98e7d3db79}" error="failed to connect to peer: 5ad9dad7-f73a-5bff-7081-5f98e7d3db79" 2025-09-14T21:47:16.474Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to appendEntries to: peer="{Voter server-a45b6704-4289-c207-9b0c-e97f2bd51d08 a45b6704-4289-c207-9b0c-e97f2bd51d08}" error="failed to connect to peer: a45b6704-4289-c207-9b0c-e97f2bd51d08" 2025-09-14T21:47:16.475Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to heartbeat to: peer=a45b6704-4289-c207-9b0c-e97f2bd51d08 backoff time=20ms error="failed to connect to peer: a45b6704-4289-c207-9b0c-e97f2bd51d08" 2025-09-14T21:47:16.475Z [WARN] server-a45b6704-4289-c207-9b0c-e97f2bd51d08: heartbeat timeout reached, starting election: last-leader-addr=8ac88d7a-edb1-572a-cc98-450480183c73 last-leader-id=server-8ac88d7a-edb1-572a-cc98-450480183c73 2025-09-14T21:47:16.475Z [INFO] server-a45b6704-4289-c207-9b0c-e97f2bd51d08: entering candidate state: node="Node at a45b6704-4289-c207-9b0c-e97f2bd51d08 [Candidate]" term=3 2025-09-14T21:47:16.476Z [WARN] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: rejecting vote request since we have a leader: from=a45b6704-4289-c207-9b0c-e97f2bd51d08 leader=8ac88d7a-edb1-572a-cc98-450480183c73 leader-id=server-8ac88d7a-edb1-572a-cc98-450480183c73 2025-09-14T21:47:16.476Z [ERROR] server-a45b6704-4289-c207-9b0c-e97f2bd51d08: failed to make requestVote RPC: target="{Voter server-8ac88d7a-edb1-572a-cc98-450480183c73 8ac88d7a-edb1-572a-cc98-450480183c73}" error="failed to connect to peer: 8ac88d7a-edb1-572a-cc98-450480183c73" term=3 2025-09-14T21:47:16.477Z [WARN] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to contact: server-id=server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79 time=60.838754ms 2025-09-14T21:47:16.477Z [WARN] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to contact: server-id=server-a45b6704-4289-c207-9b0c-e97f2bd51d08 time=59.767962ms 2025-09-14T21:47:16.477Z [WARN] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to contact quorum of nodes, stepping down 2025-09-14T21:47:16.477Z [INFO] server-8ac88d7a-edb1-572a-cc98-450480183c73: entering follower state: follower="Node at 8ac88d7a-edb1-572a-cc98-450480183c73 [Follower]" leader-address= leader-id= 2025-09-14T21:47:16.535Z [WARN] server-a45b6704-4289-c207-9b0c-e97f2bd51d08: Election timeout reached, restarting election 2025-09-14T21:47:16.535Z [INFO] server-a45b6704-4289-c207-9b0c-e97f2bd51d08: entering candidate state: node="Node at a45b6704-4289-c207-9b0c-e97f2bd51d08 [Candidate]" term=4 2025-09-14T21:47:16.535Z [ERROR] server-a45b6704-4289-c207-9b0c-e97f2bd51d08: failed to make requestVote RPC: target="{Voter server-8ac88d7a-edb1-572a-cc98-450480183c73 8ac88d7a-edb1-572a-cc98-450480183c73}" error="failed to connect to peer: 8ac88d7a-edb1-572a-cc98-450480183c73" term=4 2025-09-14T21:47:16.535Z [WARN] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: rejecting vote request since we have a leader: from=a45b6704-4289-c207-9b0c-e97f2bd51d08 leader=8ac88d7a-edb1-572a-cc98-450480183c73 leader-id=server-8ac88d7a-edb1-572a-cc98-450480183c73 2025-09-14T21:47:16.545Z [WARN] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: heartbeat timeout reached, starting election: last-leader-addr=8ac88d7a-edb1-572a-cc98-450480183c73 last-leader-id=server-8ac88d7a-edb1-572a-cc98-450480183c73 2025-09-14T21:47:16.545Z [INFO] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: entering candidate state: node="Node at 5ad9dad7-f73a-5bff-7081-5f98e7d3db79 [Candidate]" term=3 2025-09-14T21:47:16.545Z [ERROR] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: failed to make requestVote RPC: target="{Voter server-8ac88d7a-edb1-572a-cc98-450480183c73 8ac88d7a-edb1-572a-cc98-450480183c73}" error="failed to connect to peer: 8ac88d7a-edb1-572a-cc98-450480183c73" term=3 2025-09-14T21:47:16.545Z [INFO] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: entering follower state: follower="Node at 5ad9dad7-f73a-5bff-7081-5f98e7d3db79 [Follower]" leader-address= leader-id= 2025-09-14T21:47:16.572Z [WARN] server-8ac88d7a-edb1-572a-cc98-450480183c73: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:16.572Z [INFO] server-8ac88d7a-edb1-572a-cc98-450480183c73: entering candidate state: node="Node at 8ac88d7a-edb1-572a-cc98-450480183c73 [Candidate]" term=3 2025-09-14T21:47:16.572Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to make requestVote RPC: target="{Voter server-a45b6704-4289-c207-9b0c-e97f2bd51d08 a45b6704-4289-c207-9b0c-e97f2bd51d08}" error="failed to connect to peer: a45b6704-4289-c207-9b0c-e97f2bd51d08" term=3 2025-09-14T21:47:16.572Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to make requestVote RPC: target="{Voter server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79 5ad9dad7-f73a-5bff-7081-5f98e7d3db79}" error="failed to connect to peer: 5ad9dad7-f73a-5bff-7081-5f98e7d3db79" term=3 2025-09-14T21:47:16.622Z [WARN] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:16.622Z [INFO] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: entering candidate state: node="Node at 5ad9dad7-f73a-5bff-7081-5f98e7d3db79 [Candidate]" term=5 2025-09-14T21:47:16.622Z [ERROR] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: failed to make requestVote RPC: target="{Voter server-8ac88d7a-edb1-572a-cc98-450480183c73 8ac88d7a-edb1-572a-cc98-450480183c73}" error="failed to connect to peer: 8ac88d7a-edb1-572a-cc98-450480183c73" term=5 2025-09-14T21:47:16.622Z [INFO] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: election won: term=5 tally=2 2025-09-14T21:47:16.622Z [INFO] server-a45b6704-4289-c207-9b0c-e97f2bd51d08: entering follower state: follower="Node at a45b6704-4289-c207-9b0c-e97f2bd51d08 [Follower]" leader-address= leader-id= 2025-09-14T21:47:16.622Z [INFO] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: entering leader state: leader="Node at 5ad9dad7-f73a-5bff-7081-5f98e7d3db79 [Leader]" 2025-09-14T21:47:16.622Z [INFO] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: added peer, starting replication: peer=server-8ac88d7a-edb1-572a-cc98-450480183c73 2025-09-14T21:47:16.622Z [INFO] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: added peer, starting replication: peer=server-a45b6704-4289-c207-9b0c-e97f2bd51d08 2025-09-14T21:47:16.622Z [ERROR] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: failed to appendEntries to: peer="{Voter server-8ac88d7a-edb1-572a-cc98-450480183c73 8ac88d7a-edb1-572a-cc98-450480183c73}" error="failed to connect to peer: 8ac88d7a-edb1-572a-cc98-450480183c73" 2025-09-14T21:47:16.622Z [INFO] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: pipelining replication: peer="{Voter server-a45b6704-4289-c207-9b0c-e97f2bd51d08 a45b6704-4289-c207-9b0c-e97f2bd51d08}" 2025-09-14T21:47:16.630Z [WARN] server-8ac88d7a-edb1-572a-cc98-450480183c73: Election timeout reached, restarting election 2025-09-14T21:47:16.630Z [INFO] server-8ac88d7a-edb1-572a-cc98-450480183c73: entering candidate state: node="Node at 8ac88d7a-edb1-572a-cc98-450480183c73 [Candidate]" term=4 2025-09-14T21:47:16.630Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to make requestVote RPC: target="{Voter server-a45b6704-4289-c207-9b0c-e97f2bd51d08 a45b6704-4289-c207-9b0c-e97f2bd51d08}" error="failed to connect to peer: a45b6704-4289-c207-9b0c-e97f2bd51d08" term=4 2025-09-14T21:47:16.630Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to make requestVote RPC: target="{Voter server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79 5ad9dad7-f73a-5bff-7081-5f98e7d3db79}" error="failed to connect to peer: 5ad9dad7-f73a-5bff-7081-5f98e7d3db79" term=4 2025-09-14T21:47:16.632Z [ERROR] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: failed to heartbeat to: peer=8ac88d7a-edb1-572a-cc98-450480183c73 backoff time=10ms error="failed to connect to peer: 8ac88d7a-edb1-572a-cc98-450480183c73" 2025-09-14T21:47:16.639Z [ERROR] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: failed to appendEntries to: peer="{Voter server-8ac88d7a-edb1-572a-cc98-450480183c73 8ac88d7a-edb1-572a-cc98-450480183c73}" error="failed to connect to peer: 8ac88d7a-edb1-572a-cc98-450480183c73" 2025-09-14T21:47:16.653Z [ERROR] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: failed to heartbeat to: peer=8ac88d7a-edb1-572a-cc98-450480183c73 backoff time=10ms error="failed to connect to peer: 8ac88d7a-edb1-572a-cc98-450480183c73" 2025-09-14T21:47:16.658Z [ERROR] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: failed to appendEntries to: peer="{Voter server-8ac88d7a-edb1-572a-cc98-450480183c73 8ac88d7a-edb1-572a-cc98-450480183c73}" error="failed to connect to peer: 8ac88d7a-edb1-572a-cc98-450480183c73" 2025-09-14T21:47:16.672Z [ERROR] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: failed to heartbeat to: peer=8ac88d7a-edb1-572a-cc98-450480183c73 backoff time=10ms error="failed to connect to peer: 8ac88d7a-edb1-572a-cc98-450480183c73" 2025-09-14T21:47:16.673Z [WARN] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: failed to contact: server-id=server-8ac88d7a-edb1-572a-cc98-450480183c73 time=50.933823ms 2025-09-14T21:47:16.683Z [ERROR] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: failed to appendEntries to: peer="{Voter server-8ac88d7a-edb1-572a-cc98-450480183c73 8ac88d7a-edb1-572a-cc98-450480183c73}" error="failed to connect to peer: 8ac88d7a-edb1-572a-cc98-450480183c73" 2025-09-14T21:47:16.684Z [WARN] server-8ac88d7a-edb1-572a-cc98-450480183c73: Election timeout reached, restarting election 2025-09-14T21:47:16.684Z [INFO] server-8ac88d7a-edb1-572a-cc98-450480183c73: entering candidate state: node="Node at 8ac88d7a-edb1-572a-cc98-450480183c73 [Candidate]" term=5 2025-09-14T21:47:16.684Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to make requestVote RPC: target="{Voter server-a45b6704-4289-c207-9b0c-e97f2bd51d08 a45b6704-4289-c207-9b0c-e97f2bd51d08}" error="failed to connect to peer: a45b6704-4289-c207-9b0c-e97f2bd51d08" term=5 2025-09-14T21:47:16.684Z [ERROR] server-8ac88d7a-edb1-572a-cc98-450480183c73: failed to make requestVote RPC: target="{Voter server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79 5ad9dad7-f73a-5bff-7081-5f98e7d3db79}" error="failed to connect to peer: 5ad9dad7-f73a-5bff-7081-5f98e7d3db79" term=5 2025-09-14T21:47:16.692Z [ERROR] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: failed to heartbeat to: peer=8ac88d7a-edb1-572a-cc98-450480183c73 backoff time=20ms error="failed to connect to peer: 8ac88d7a-edb1-572a-cc98-450480183c73" 2025-09-14T21:47:16.721Z [ERROR] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: failed to heartbeat to: peer=8ac88d7a-edb1-572a-cc98-450480183c73 backoff time=25ms error="failed to connect to peer: 8ac88d7a-edb1-572a-cc98-450480183c73" 2025-09-14T21:47:16.722Z [WARN] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: failed to contact: server-id=server-8ac88d7a-edb1-572a-cc98-450480183c73 time=100.658929ms 2025-09-14T21:47:16.727Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:16.622167975 +0000 UTC m=+7.190046237 (1 nodes), 200.451715ms from start of poll, 486.855054ms from cluster start. Timeout at 2025-09-14 21:47:16.727171549 +0000 UTC m=+7.295049811, 105.003574ms after stability raft_test.go:413: [INFO] Reconnecting Node at 8ac88d7a-edb1-572a-cc98-450480183c73 [Candidate] 2025-09-14T21:47:16.731Z [INFO] server-8ac88d7a-edb1-572a-cc98-450480183c73: entering follower state: follower="Node at 8ac88d7a-edb1-572a-cc98-450480183c73 [Follower]" leader-address=5ad9dad7-f73a-5bff-7081-5f98e7d3db79 leader-id=server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79 2025-09-14T21:47:16.731Z [INFO] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: pipelining replication: peer="{Voter server-8ac88d7a-edb1-572a-cc98-450480183c73 8ac88d7a-edb1-572a-cc98-450480183c73}" 2025-09-14T21:47:16.736Z [INFO] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: aborting pipeline replication: peer="{Voter server-a45b6704-4289-c207-9b0c-e97f2bd51d08 a45b6704-4289-c207-9b0c-e97f2bd51d08}" 2025-09-14T21:47:16.736Z [INFO] server-5ad9dad7-f73a-5bff-7081-5f98e7d3db79: aborting pipeline replication: peer="{Voter server-8ac88d7a-edb1-572a-cc98-450480183c73 8ac88d7a-edb1-572a-cc98-450480183c73}" --- PASS: TestRaft_LeaderFail (0.60s) === RUN TestRaft_BehindFollower 2025-09-14T21:47:16.738Z [INFO] server-45787dcb-0833-3ce1-aee3-5bd75b683074: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45787dcb-0833-3ce1-aee3-5bd75b683074 Address:45787dcb-0833-3ce1-aee3-5bd75b683074} {Suffrage:Voter ID:server-33a00f48-5b2f-9cf1-4830-e433f135400d Address:33a00f48-5b2f-9cf1-4830-e433f135400d} {Suffrage:Voter ID:server-27c34f35-211c-393c-7c23-c244d0a2c99b Address:27c34f35-211c-393c-7c23-c244d0a2c99b}]" 2025-09-14T21:47:16.738Z [INFO] server-45787dcb-0833-3ce1-aee3-5bd75b683074: entering follower state: follower="Node at 45787dcb-0833-3ce1-aee3-5bd75b683074 [Follower]" leader-address= leader-id= 2025-09-14T21:47:16.738Z [INFO] server-33a00f48-5b2f-9cf1-4830-e433f135400d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45787dcb-0833-3ce1-aee3-5bd75b683074 Address:45787dcb-0833-3ce1-aee3-5bd75b683074} {Suffrage:Voter ID:server-33a00f48-5b2f-9cf1-4830-e433f135400d Address:33a00f48-5b2f-9cf1-4830-e433f135400d} {Suffrage:Voter ID:server-27c34f35-211c-393c-7c23-c244d0a2c99b Address:27c34f35-211c-393c-7c23-c244d0a2c99b}]" 2025-09-14T21:47:16.738Z [INFO] server-27c34f35-211c-393c-7c23-c244d0a2c99b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45787dcb-0833-3ce1-aee3-5bd75b683074 Address:45787dcb-0833-3ce1-aee3-5bd75b683074} {Suffrage:Voter ID:server-33a00f48-5b2f-9cf1-4830-e433f135400d Address:33a00f48-5b2f-9cf1-4830-e433f135400d} {Suffrage:Voter ID:server-27c34f35-211c-393c-7c23-c244d0a2c99b Address:27c34f35-211c-393c-7c23-c244d0a2c99b}]" 2025-09-14T21:47:16.738Z [INFO] server-33a00f48-5b2f-9cf1-4830-e433f135400d: entering follower state: follower="Node at 33a00f48-5b2f-9cf1-4830-e433f135400d [Follower]" leader-address= leader-id= 2025-09-14T21:47:16.738Z [INFO] server-27c34f35-211c-393c-7c23-c244d0a2c99b: entering follower state: follower="Node at 27c34f35-211c-393c-7c23-c244d0a2c99b [Follower]" leader-address= leader-id= 2025-09-14T21:47:16.738Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:16.789Z [WARN] server-33a00f48-5b2f-9cf1-4830-e433f135400d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:16.789Z [INFO] server-33a00f48-5b2f-9cf1-4830-e433f135400d: entering candidate state: node="Node at 33a00f48-5b2f-9cf1-4830-e433f135400d [Candidate]" term=2 2025-09-14T21:47:16.789Z [INFO] server-33a00f48-5b2f-9cf1-4830-e433f135400d: election won: term=2 tally=2 2025-09-14T21:47:16.789Z [INFO] server-33a00f48-5b2f-9cf1-4830-e433f135400d: entering leader state: leader="Node at 33a00f48-5b2f-9cf1-4830-e433f135400d [Leader]" 2025-09-14T21:47:16.789Z [INFO] server-33a00f48-5b2f-9cf1-4830-e433f135400d: added peer, starting replication: peer=server-45787dcb-0833-3ce1-aee3-5bd75b683074 2025-09-14T21:47:16.789Z [INFO] server-33a00f48-5b2f-9cf1-4830-e433f135400d: added peer, starting replication: peer=server-27c34f35-211c-393c-7c23-c244d0a2c99b 2025-09-14T21:47:16.789Z [INFO] server-33a00f48-5b2f-9cf1-4830-e433f135400d: pipelining replication: peer="{Voter server-27c34f35-211c-393c-7c23-c244d0a2c99b 27c34f35-211c-393c-7c23-c244d0a2c99b}" 2025-09-14T21:47:16.789Z [INFO] server-33a00f48-5b2f-9cf1-4830-e433f135400d: pipelining replication: peer="{Voter server-45787dcb-0833-3ce1-aee3-5bd75b683074 45787dcb-0833-3ce1-aee3-5bd75b683074}" 2025-09-14T21:47:16.895Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:16.789619532 +0000 UTC m=+7.357497791 (1 nodes), 51.237474ms from start of poll, 51.587934ms from cluster start. Timeout at 2025-09-14 21:47:16.894622473 +0000 UTC m=+7.462500732, 105.002941ms after stability 2025-09-14T21:47:16.895Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:17.000Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:16.89557572 +0000 UTC m=+7.463453995 (2 nodes), 1.318µs from start of poll, 157.544138ms from cluster start. Timeout at 2025-09-14 21:47:17.000580223 +0000 UTC m=+7.568458517, 105.004522ms after stability 2025-09-14T21:47:17.000Z [ERROR] server-33a00f48-5b2f-9cf1-4830-e433f135400d: failed to pipeline appendEntries: peer="{Voter server-45787dcb-0833-3ce1-aee3-5bd75b683074 45787dcb-0833-3ce1-aee3-5bd75b683074}" error="append pipeline closed" 2025-09-14T21:47:17.000Z [INFO] server-33a00f48-5b2f-9cf1-4830-e433f135400d: aborting pipeline replication: peer="{Voter server-45787dcb-0833-3ce1-aee3-5bd75b683074 45787dcb-0833-3ce1-aee3-5bd75b683074}" 2025-09-14T21:47:17.000Z [ERROR] server-33a00f48-5b2f-9cf1-4830-e433f135400d: failed to appendEntries to: peer="{Voter server-45787dcb-0833-3ce1-aee3-5bd75b683074 45787dcb-0833-3ce1-aee3-5bd75b683074}" error="failed to connect to peer: 45787dcb-0833-3ce1-aee3-5bd75b683074" raft_test.go:459: [INFO] Finished apply without behind follower 2025-09-14T21:47:17.011Z [INFO] server-33a00f48-5b2f-9cf1-4830-e433f135400d: pipelining replication: peer="{Voter server-45787dcb-0833-3ce1-aee3-5bd75b683074 45787dcb-0833-3ce1-aee3-5bd75b683074}" 2025-09-14T21:47:17.026Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:17.131Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:17.026258925 +0000 UTC m=+7.594137191 (1 nodes), 2.36µs from start of poll, 288.227334ms from cluster start. Timeout at 2025-09-14 21:47:17.131265616 +0000 UTC m=+7.699143892, 105.006701ms after stability 2025-09-14T21:47:17.131Z [INFO] server-33a00f48-5b2f-9cf1-4830-e433f135400d: aborting pipeline replication: peer="{Voter server-45787dcb-0833-3ce1-aee3-5bd75b683074 45787dcb-0833-3ce1-aee3-5bd75b683074}" 2025-09-14T21:47:17.131Z [INFO] server-33a00f48-5b2f-9cf1-4830-e433f135400d: aborting pipeline replication: peer="{Voter server-27c34f35-211c-393c-7c23-c244d0a2c99b 27c34f35-211c-393c-7c23-c244d0a2c99b}" --- PASS: TestRaft_BehindFollower (0.39s) === RUN TestRaft_ApplyNonLeader 2025-09-14T21:47:17.132Z [INFO] server-a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb Address:a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb} {Suffrage:Voter ID:server-194bd09c-e367-1893-bc9b-ee3528896292 Address:194bd09c-e367-1893-bc9b-ee3528896292} {Suffrage:Voter ID:server-3652cf41-8c2b-55ca-aeb9-4c97a658b3ae Address:3652cf41-8c2b-55ca-aeb9-4c97a658b3ae}]" 2025-09-14T21:47:17.132Z [INFO] server-a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb: entering follower state: follower="Node at a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb [Follower]" leader-address= leader-id= 2025-09-14T21:47:17.133Z [INFO] server-194bd09c-e367-1893-bc9b-ee3528896292: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb Address:a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb} {Suffrage:Voter ID:server-194bd09c-e367-1893-bc9b-ee3528896292 Address:194bd09c-e367-1893-bc9b-ee3528896292} {Suffrage:Voter ID:server-3652cf41-8c2b-55ca-aeb9-4c97a658b3ae Address:3652cf41-8c2b-55ca-aeb9-4c97a658b3ae}]" 2025-09-14T21:47:17.133Z [INFO] server-194bd09c-e367-1893-bc9b-ee3528896292: entering follower state: follower="Node at 194bd09c-e367-1893-bc9b-ee3528896292 [Follower]" leader-address= leader-id= 2025-09-14T21:47:17.133Z [INFO] server-3652cf41-8c2b-55ca-aeb9-4c97a658b3ae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb Address:a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb} {Suffrage:Voter ID:server-194bd09c-e367-1893-bc9b-ee3528896292 Address:194bd09c-e367-1893-bc9b-ee3528896292} {Suffrage:Voter ID:server-3652cf41-8c2b-55ca-aeb9-4c97a658b3ae Address:3652cf41-8c2b-55ca-aeb9-4c97a658b3ae}]" 2025-09-14T21:47:17.133Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:17.133Z [INFO] server-3652cf41-8c2b-55ca-aeb9-4c97a658b3ae: entering follower state: follower="Node at 3652cf41-8c2b-55ca-aeb9-4c97a658b3ae [Follower]" leader-address= leader-id= 2025-09-14T21:47:17.207Z [WARN] server-a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:17.207Z [INFO] server-a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb: entering candidate state: node="Node at a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb [Candidate]" term=2 2025-09-14T21:47:17.207Z [INFO] server-a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb: election won: term=2 tally=2 2025-09-14T21:47:17.207Z [INFO] server-a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb: entering leader state: leader="Node at a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb [Leader]" 2025-09-14T21:47:17.207Z [INFO] server-a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb: added peer, starting replication: peer=server-194bd09c-e367-1893-bc9b-ee3528896292 2025-09-14T21:47:17.207Z [INFO] server-a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb: added peer, starting replication: peer=server-3652cf41-8c2b-55ca-aeb9-4c97a658b3ae 2025-09-14T21:47:17.207Z [INFO] server-a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb: pipelining replication: peer="{Voter server-3652cf41-8c2b-55ca-aeb9-4c97a658b3ae 3652cf41-8c2b-55ca-aeb9-4c97a658b3ae}" 2025-09-14T21:47:17.207Z [INFO] server-a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb: pipelining replication: peer="{Voter server-194bd09c-e367-1893-bc9b-ee3528896292 194bd09c-e367-1893-bc9b-ee3528896292}" 2025-09-14T21:47:17.313Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:17.207532705 +0000 UTC m=+7.775410980 (1 nodes), 74.374013ms from start of poll, 74.756975ms from cluster start. Timeout at 2025-09-14 21:47:17.312534995 +0000 UTC m=+7.880413254, 105.002274ms after stability 2025-09-14T21:47:17.313Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:17.418Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:17.313125544 +0000 UTC m=+7.881003816 (2 nodes), 1.313µs from start of poll, 180.349811ms from cluster start. Timeout at 2025-09-14 21:47:17.418129702 +0000 UTC m=+7.986007966, 105.00415ms after stability 2025-09-14T21:47:17.418Z [INFO] server-a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb: aborting pipeline replication: peer="{Voter server-3652cf41-8c2b-55ca-aeb9-4c97a658b3ae 3652cf41-8c2b-55ca-aeb9-4c97a658b3ae}" 2025-09-14T21:47:17.418Z [INFO] server-a418b5e1-0a7d-0aba-3d06-f58a2f1d8dcb: aborting pipeline replication: peer="{Voter server-194bd09c-e367-1893-bc9b-ee3528896292 194bd09c-e367-1893-bc9b-ee3528896292}" --- PASS: TestRaft_ApplyNonLeader (0.29s) === RUN TestRaft_ApplyConcurrent 2025-09-14T21:47:17.420Z [INFO] server-9933f755-9d60-dc6f-f2a9-3cf40c8c7799: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9933f755-9d60-dc6f-f2a9-3cf40c8c7799 Address:9933f755-9d60-dc6f-f2a9-3cf40c8c7799} {Suffrage:Voter ID:server-fa7be72e-365d-670d-3cd8-c316c4fb5c89 Address:fa7be72e-365d-670d-3cd8-c316c4fb5c89} {Suffrage:Voter ID:server-74afb8f4-c838-8921-6580-068babf936ed Address:74afb8f4-c838-8921-6580-068babf936ed}]" 2025-09-14T21:47:17.420Z [INFO] server-9933f755-9d60-dc6f-f2a9-3cf40c8c7799: entering follower state: follower="Node at 9933f755-9d60-dc6f-f2a9-3cf40c8c7799 [Follower]" leader-address= leader-id= 2025-09-14T21:47:17.420Z [INFO] server-fa7be72e-365d-670d-3cd8-c316c4fb5c89: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9933f755-9d60-dc6f-f2a9-3cf40c8c7799 Address:9933f755-9d60-dc6f-f2a9-3cf40c8c7799} {Suffrage:Voter ID:server-fa7be72e-365d-670d-3cd8-c316c4fb5c89 Address:fa7be72e-365d-670d-3cd8-c316c4fb5c89} {Suffrage:Voter ID:server-74afb8f4-c838-8921-6580-068babf936ed Address:74afb8f4-c838-8921-6580-068babf936ed}]" 2025-09-14T21:47:17.420Z [INFO] server-fa7be72e-365d-670d-3cd8-c316c4fb5c89: entering follower state: follower="Node at fa7be72e-365d-670d-3cd8-c316c4fb5c89 [Follower]" leader-address= leader-id= 2025-09-14T21:47:17.420Z [INFO] server-74afb8f4-c838-8921-6580-068babf936ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9933f755-9d60-dc6f-f2a9-3cf40c8c7799 Address:9933f755-9d60-dc6f-f2a9-3cf40c8c7799} {Suffrage:Voter ID:server-fa7be72e-365d-670d-3cd8-c316c4fb5c89 Address:fa7be72e-365d-670d-3cd8-c316c4fb5c89} {Suffrage:Voter ID:server-74afb8f4-c838-8921-6580-068babf936ed Address:74afb8f4-c838-8921-6580-068babf936ed}]" 2025-09-14T21:47:17.420Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:17.420Z [INFO] server-74afb8f4-c838-8921-6580-068babf936ed: entering follower state: follower="Node at 74afb8f4-c838-8921-6580-068babf936ed [Follower]" leader-address= leader-id= 2025-09-14T21:47:17.575Z [WARN] server-fa7be72e-365d-670d-3cd8-c316c4fb5c89: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:17.575Z [INFO] server-fa7be72e-365d-670d-3cd8-c316c4fb5c89: entering candidate state: node="Node at fa7be72e-365d-670d-3cd8-c316c4fb5c89 [Candidate]" term=2 2025-09-14T21:47:17.575Z [INFO] server-fa7be72e-365d-670d-3cd8-c316c4fb5c89: election won: term=2 tally=2 2025-09-14T21:47:17.575Z [INFO] server-fa7be72e-365d-670d-3cd8-c316c4fb5c89: entering leader state: leader="Node at fa7be72e-365d-670d-3cd8-c316c4fb5c89 [Leader]" 2025-09-14T21:47:17.575Z [INFO] server-fa7be72e-365d-670d-3cd8-c316c4fb5c89: added peer, starting replication: peer=server-9933f755-9d60-dc6f-f2a9-3cf40c8c7799 2025-09-14T21:47:17.576Z [INFO] server-fa7be72e-365d-670d-3cd8-c316c4fb5c89: added peer, starting replication: peer=server-74afb8f4-c838-8921-6580-068babf936ed 2025-09-14T21:47:17.576Z [INFO] server-fa7be72e-365d-670d-3cd8-c316c4fb5c89: pipelining replication: peer="{Voter server-74afb8f4-c838-8921-6580-068babf936ed 74afb8f4-c838-8921-6580-068babf936ed}" 2025-09-14T21:47:17.576Z [INFO] server-fa7be72e-365d-670d-3cd8-c316c4fb5c89: pipelining replication: peer="{Voter server-9933f755-9d60-dc6f-f2a9-3cf40c8c7799 9933f755-9d60-dc6f-f2a9-3cf40c8c7799}" 2025-09-14T21:47:17.781Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:17.576030886 +0000 UTC m=+8.143909144 (1 nodes), 155.744383ms from start of poll, 156.109358ms from cluster start. Timeout at 2025-09-14 21:47:17.781031823 +0000 UTC m=+8.348910085, 205.000941ms after stability 2025-09-14T21:47:17.794Z [INFO] server-fa7be72e-365d-670d-3cd8-c316c4fb5c89: aborting pipeline replication: peer="{Voter server-74afb8f4-c838-8921-6580-068babf936ed 74afb8f4-c838-8921-6580-068babf936ed}" 2025-09-14T21:47:17.794Z [INFO] server-fa7be72e-365d-670d-3cd8-c316c4fb5c89: aborting pipeline replication: peer="{Voter server-9933f755-9d60-dc6f-f2a9-3cf40c8c7799 9933f755-9d60-dc6f-f2a9-3cf40c8c7799}" --- PASS: TestRaft_ApplyConcurrent (0.38s) === RUN TestRaft_ApplyConcurrent_Timeout 2025-09-14T21:47:17.795Z [INFO] server-33cddb51-b211-d8c2-4eb1-f8e99dd6f11e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33cddb51-b211-d8c2-4eb1-f8e99dd6f11e Address:33cddb51-b211-d8c2-4eb1-f8e99dd6f11e}]" 2025-09-14T21:47:17.795Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:17.795Z [INFO] server-33cddb51-b211-d8c2-4eb1-f8e99dd6f11e: entering follower state: follower="Node at 33cddb51-b211-d8c2-4eb1-f8e99dd6f11e [Follower]" leader-address= leader-id= 2025-09-14T21:47:17.946Z [WARN] server-33cddb51-b211-d8c2-4eb1-f8e99dd6f11e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:17.946Z [INFO] server-33cddb51-b211-d8c2-4eb1-f8e99dd6f11e: entering candidate state: node="Node at 33cddb51-b211-d8c2-4eb1-f8e99dd6f11e [Candidate]" term=2 2025-09-14T21:47:17.946Z [INFO] server-33cddb51-b211-d8c2-4eb1-f8e99dd6f11e: election won: term=2 tally=1 2025-09-14T21:47:17.946Z [INFO] server-33cddb51-b211-d8c2-4eb1-f8e99dd6f11e: entering leader state: leader="Node at 33cddb51-b211-d8c2-4eb1-f8e99dd6f11e [Leader]" 2025-09-14T21:47:18.148Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:17.946726776 +0000 UTC m=+8.514605044 (1 nodes), 151.06898ms from start of poll, 151.221558ms from cluster start. Timeout at 2025-09-14 21:47:18.147731865 +0000 UTC m=+8.715610128, 201.005084ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.35s) === RUN TestRaft_JoinNode 2025-09-14T21:47:18.150Z [INFO] server-d694c153-7c5d-e664-0044-75cab98f6b71: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d694c153-7c5d-e664-0044-75cab98f6b71 Address:d694c153-7c5d-e664-0044-75cab98f6b71} {Suffrage:Voter ID:server-08da9311-9a95-6f25-2003-d405740aa3d8 Address:08da9311-9a95-6f25-2003-d405740aa3d8}]" 2025-09-14T21:47:18.150Z [INFO] server-d694c153-7c5d-e664-0044-75cab98f6b71: entering follower state: follower="Node at d694c153-7c5d-e664-0044-75cab98f6b71 [Follower]" leader-address= leader-id= 2025-09-14T21:47:18.150Z [INFO] server-08da9311-9a95-6f25-2003-d405740aa3d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d694c153-7c5d-e664-0044-75cab98f6b71 Address:d694c153-7c5d-e664-0044-75cab98f6b71} {Suffrage:Voter ID:server-08da9311-9a95-6f25-2003-d405740aa3d8 Address:08da9311-9a95-6f25-2003-d405740aa3d8}]" 2025-09-14T21:47:18.150Z [INFO] server-08da9311-9a95-6f25-2003-d405740aa3d8: entering follower state: follower="Node at 08da9311-9a95-6f25-2003-d405740aa3d8 [Follower]" leader-address= leader-id= 2025-09-14T21:47:18.150Z [INFO] server-6008b188-0184-4c6a-2ff8-e7a4b020818c: initial configuration: index=0 servers=[] 2025-09-14T21:47:18.150Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:18.150Z [INFO] server-6008b188-0184-4c6a-2ff8-e7a4b020818c: entering follower state: follower="Node at 6008b188-0184-4c6a-2ff8-e7a4b020818c [Follower]" leader-address= leader-id= 2025-09-14T21:47:18.203Z [WARN] server-08da9311-9a95-6f25-2003-d405740aa3d8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:18.203Z [INFO] server-08da9311-9a95-6f25-2003-d405740aa3d8: entering candidate state: node="Node at 08da9311-9a95-6f25-2003-d405740aa3d8 [Candidate]" term=2 2025-09-14T21:47:18.203Z [INFO] server-08da9311-9a95-6f25-2003-d405740aa3d8: election won: term=2 tally=2 2025-09-14T21:47:18.203Z [INFO] server-08da9311-9a95-6f25-2003-d405740aa3d8: entering leader state: leader="Node at 08da9311-9a95-6f25-2003-d405740aa3d8 [Leader]" 2025-09-14T21:47:18.203Z [INFO] server-08da9311-9a95-6f25-2003-d405740aa3d8: added peer, starting replication: peer=server-d694c153-7c5d-e664-0044-75cab98f6b71 2025-09-14T21:47:18.203Z [INFO] server-08da9311-9a95-6f25-2003-d405740aa3d8: pipelining replication: peer="{Voter server-d694c153-7c5d-e664-0044-75cab98f6b71 d694c153-7c5d-e664-0044-75cab98f6b71}" 2025-09-14T21:47:18.237Z [WARN] server-6008b188-0184-4c6a-2ff8-e7a4b020818c: no known peers, aborting election 2025-09-14T21:47:18.308Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:18.203416223 +0000 UTC m=+8.771294485 (1 nodes), 52.643735ms from start of poll, 53.522828ms from cluster start. Timeout at 2025-09-14 21:47:18.308419281 +0000 UTC m=+8.876297540, 105.003055ms after stability 2025-09-14T21:47:18.308Z [INFO] server-08da9311-9a95-6f25-2003-d405740aa3d8: updating configuration: command=AddVoter server-id=server-6008b188-0184-4c6a-2ff8-e7a4b020818c server-addr=6008b188-0184-4c6a-2ff8-e7a4b020818c servers="[{Suffrage:Voter ID:server-d694c153-7c5d-e664-0044-75cab98f6b71 Address:d694c153-7c5d-e664-0044-75cab98f6b71} {Suffrage:Voter ID:server-08da9311-9a95-6f25-2003-d405740aa3d8 Address:08da9311-9a95-6f25-2003-d405740aa3d8} {Suffrage:Voter ID:server-6008b188-0184-4c6a-2ff8-e7a4b020818c Address:6008b188-0184-4c6a-2ff8-e7a4b020818c}]" 2025-09-14T21:47:18.308Z [INFO] server-08da9311-9a95-6f25-2003-d405740aa3d8: added peer, starting replication: peer=server-6008b188-0184-4c6a-2ff8-e7a4b020818c 2025-09-14T21:47:18.308Z [WARN] server-6008b188-0184-4c6a-2ff8-e7a4b020818c: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2025-09-14T21:47:18.308Z [WARN] server-08da9311-9a95-6f25-2003-d405740aa3d8: appendEntries rejected, sending older logs: peer="{Voter server-6008b188-0184-4c6a-2ff8-e7a4b020818c 6008b188-0184-4c6a-2ff8-e7a4b020818c}" next=1 2025-09-14T21:47:18.309Z [INFO] server-08da9311-9a95-6f25-2003-d405740aa3d8: pipelining replication: peer="{Voter server-6008b188-0184-4c6a-2ff8-e7a4b020818c 6008b188-0184-4c6a-2ff8-e7a4b020818c}" 2025-09-14T21:47:18.309Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:18.414Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:18.309052329 +0000 UTC m=+8.876930592 (1 nodes), 698ns from start of poll, 159.158935ms from cluster start. Timeout at 2025-09-14 21:47:18.414058007 +0000 UTC m=+8.981936290, 105.005698ms after stability 2025-09-14T21:47:18.414Z [INFO] server-08da9311-9a95-6f25-2003-d405740aa3d8: aborting pipeline replication: peer="{Voter server-6008b188-0184-4c6a-2ff8-e7a4b020818c 6008b188-0184-4c6a-2ff8-e7a4b020818c}" 2025-09-14T21:47:18.414Z [INFO] server-08da9311-9a95-6f25-2003-d405740aa3d8: aborting pipeline replication: peer="{Voter server-d694c153-7c5d-e664-0044-75cab98f6b71 d694c153-7c5d-e664-0044-75cab98f6b71}" --- PASS: TestRaft_JoinNode (0.27s) === RUN TestRaft_JoinNode_ConfigStore 2025-09-14T21:47:18.415Z [INFO] server-567f5064-4d11-7cb4-c169-96dd4723fccb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-567f5064-4d11-7cb4-c169-96dd4723fccb Address:567f5064-4d11-7cb4-c169-96dd4723fccb}]" 2025-09-14T21:47:18.415Z [INFO] server-567f5064-4d11-7cb4-c169-96dd4723fccb: entering follower state: follower="Node at 567f5064-4d11-7cb4-c169-96dd4723fccb [Follower]" leader-address= leader-id= 2025-09-14T21:47:18.415Z [INFO] server-a0267b40-1854-c832-4812-333260614fff: initial configuration: index=0 servers=[] 2025-09-14T21:47:18.415Z [INFO] server-a0267b40-1854-c832-4812-333260614fff: entering follower state: follower="Node at a0267b40-1854-c832-4812-333260614fff [Follower]" leader-address= leader-id= 2025-09-14T21:47:18.415Z [INFO] server-8bd2bc19-a819-9e0e-51f4-2872d33c8b05: initial configuration: index=0 servers=[] 2025-09-14T21:47:18.415Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:18.415Z [INFO] server-8bd2bc19-a819-9e0e-51f4-2872d33c8b05: entering follower state: follower="Node at 8bd2bc19-a819-9e0e-51f4-2872d33c8b05 [Follower]" leader-address= leader-id= 2025-09-14T21:47:18.471Z [WARN] server-567f5064-4d11-7cb4-c169-96dd4723fccb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:18.471Z [INFO] server-567f5064-4d11-7cb4-c169-96dd4723fccb: entering candidate state: node="Node at 567f5064-4d11-7cb4-c169-96dd4723fccb [Candidate]" term=2 2025-09-14T21:47:18.471Z [INFO] server-567f5064-4d11-7cb4-c169-96dd4723fccb: election won: term=2 tally=1 2025-09-14T21:47:18.471Z [INFO] server-567f5064-4d11-7cb4-c169-96dd4723fccb: entering leader state: leader="Node at 567f5064-4d11-7cb4-c169-96dd4723fccb [Leader]" 2025-09-14T21:47:18.482Z [WARN] server-8bd2bc19-a819-9e0e-51f4-2872d33c8b05: no known peers, aborting election 2025-09-14T21:47:18.495Z [WARN] server-a0267b40-1854-c832-4812-333260614fff: no known peers, aborting election 2025-09-14T21:47:18.577Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:18.471770269 +0000 UTC m=+9.039648539 (1 nodes), 55.791542ms from start of poll, 56.459065ms from cluster start. Timeout at 2025-09-14 21:47:18.576773469 +0000 UTC m=+9.144651732, 105.003193ms after stability 2025-09-14T21:47:18.577Z [INFO] server-567f5064-4d11-7cb4-c169-96dd4723fccb: updating configuration: command=AddVoter server-id=server-a0267b40-1854-c832-4812-333260614fff server-addr=a0267b40-1854-c832-4812-333260614fff servers="[{Suffrage:Voter ID:server-567f5064-4d11-7cb4-c169-96dd4723fccb Address:567f5064-4d11-7cb4-c169-96dd4723fccb} {Suffrage:Voter ID:server-a0267b40-1854-c832-4812-333260614fff Address:a0267b40-1854-c832-4812-333260614fff}]" 2025-09-14T21:47:18.577Z [INFO] server-567f5064-4d11-7cb4-c169-96dd4723fccb: added peer, starting replication: peer=server-a0267b40-1854-c832-4812-333260614fff 2025-09-14T21:47:18.577Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:18.578Z [WARN] server-a0267b40-1854-c832-4812-333260614fff: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2025-09-14T21:47:18.578Z [WARN] server-567f5064-4d11-7cb4-c169-96dd4723fccb: appendEntries rejected, sending older logs: peer="{Voter server-a0267b40-1854-c832-4812-333260614fff a0267b40-1854-c832-4812-333260614fff}" next=1 2025-09-14T21:47:18.578Z [INFO] server-567f5064-4d11-7cb4-c169-96dd4723fccb: pipelining replication: peer="{Voter server-a0267b40-1854-c832-4812-333260614fff a0267b40-1854-c832-4812-333260614fff}" 2025-09-14T21:47:18.683Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:18.577916206 +0000 UTC m=+9.145794474 (1 nodes), 924ns from start of poll, 162.605ms from cluster start. Timeout at 2025-09-14 21:47:18.682920965 +0000 UTC m=+9.250799224, 105.00475ms after stability 2025-09-14T21:47:18.683Z [INFO] server-567f5064-4d11-7cb4-c169-96dd4723fccb: updating configuration: command=AddVoter server-id=server-8bd2bc19-a819-9e0e-51f4-2872d33c8b05 server-addr=8bd2bc19-a819-9e0e-51f4-2872d33c8b05 servers="[{Suffrage:Voter ID:server-567f5064-4d11-7cb4-c169-96dd4723fccb Address:567f5064-4d11-7cb4-c169-96dd4723fccb} {Suffrage:Voter ID:server-a0267b40-1854-c832-4812-333260614fff Address:a0267b40-1854-c832-4812-333260614fff} {Suffrage:Voter ID:server-8bd2bc19-a819-9e0e-51f4-2872d33c8b05 Address:8bd2bc19-a819-9e0e-51f4-2872d33c8b05}]" 2025-09-14T21:47:18.683Z [INFO] server-567f5064-4d11-7cb4-c169-96dd4723fccb: added peer, starting replication: peer=server-8bd2bc19-a819-9e0e-51f4-2872d33c8b05 2025-09-14T21:47:18.683Z [WARN] server-8bd2bc19-a819-9e0e-51f4-2872d33c8b05: failed to get previous log: previous-index=4 last-index=0 error="log not found" 2025-09-14T21:47:18.684Z [WARN] server-567f5064-4d11-7cb4-c169-96dd4723fccb: appendEntries rejected, sending older logs: peer="{Voter server-8bd2bc19-a819-9e0e-51f4-2872d33c8b05 8bd2bc19-a819-9e0e-51f4-2872d33c8b05}" next=1 2025-09-14T21:47:18.684Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:18.684Z [INFO] server-567f5064-4d11-7cb4-c169-96dd4723fccb: pipelining replication: peer="{Voter server-8bd2bc19-a819-9e0e-51f4-2872d33c8b05 8bd2bc19-a819-9e0e-51f4-2872d33c8b05}" 2025-09-14T21:47:18.789Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:18.68414999 +0000 UTC m=+9.252028271 (1 nodes), 807ns from start of poll, 268.838797ms from cluster start. Timeout at 2025-09-14 21:47:18.789155033 +0000 UTC m=+9.357033308, 105.005037ms after stability 2025-09-14T21:47:18.789Z [INFO] server-567f5064-4d11-7cb4-c169-96dd4723fccb: aborting pipeline replication: peer="{Voter server-a0267b40-1854-c832-4812-333260614fff a0267b40-1854-c832-4812-333260614fff}" 2025-09-14T21:47:18.789Z [INFO] server-567f5064-4d11-7cb4-c169-96dd4723fccb: aborting pipeline replication: peer="{Voter server-8bd2bc19-a819-9e0e-51f4-2872d33c8b05 8bd2bc19-a819-9e0e-51f4-2872d33c8b05}" --- PASS: TestRaft_JoinNode_ConfigStore (0.38s) === RUN TestRaft_RemoveFollower 2025-09-14T21:47:18.791Z [INFO] server-5e83f3e5-7527-4a17-e825-cb7179d78783: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e83f3e5-7527-4a17-e825-cb7179d78783 Address:5e83f3e5-7527-4a17-e825-cb7179d78783} {Suffrage:Voter ID:server-96451686-fc28-65ee-2a3d-d1503c21433a Address:96451686-fc28-65ee-2a3d-d1503c21433a} {Suffrage:Voter ID:server-2541dbfe-d648-e821-3204-d53e9407f0a1 Address:2541dbfe-d648-e821-3204-d53e9407f0a1}]" 2025-09-14T21:47:18.791Z [INFO] server-5e83f3e5-7527-4a17-e825-cb7179d78783: entering follower state: follower="Node at 5e83f3e5-7527-4a17-e825-cb7179d78783 [Follower]" leader-address= leader-id= 2025-09-14T21:47:18.791Z [INFO] server-96451686-fc28-65ee-2a3d-d1503c21433a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e83f3e5-7527-4a17-e825-cb7179d78783 Address:5e83f3e5-7527-4a17-e825-cb7179d78783} {Suffrage:Voter ID:server-96451686-fc28-65ee-2a3d-d1503c21433a Address:96451686-fc28-65ee-2a3d-d1503c21433a} {Suffrage:Voter ID:server-2541dbfe-d648-e821-3204-d53e9407f0a1 Address:2541dbfe-d648-e821-3204-d53e9407f0a1}]" 2025-09-14T21:47:18.791Z [INFO] server-96451686-fc28-65ee-2a3d-d1503c21433a: entering follower state: follower="Node at 96451686-fc28-65ee-2a3d-d1503c21433a [Follower]" leader-address= leader-id= 2025-09-14T21:47:18.791Z [INFO] server-2541dbfe-d648-e821-3204-d53e9407f0a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e83f3e5-7527-4a17-e825-cb7179d78783 Address:5e83f3e5-7527-4a17-e825-cb7179d78783} {Suffrage:Voter ID:server-96451686-fc28-65ee-2a3d-d1503c21433a Address:96451686-fc28-65ee-2a3d-d1503c21433a} {Suffrage:Voter ID:server-2541dbfe-d648-e821-3204-d53e9407f0a1 Address:2541dbfe-d648-e821-3204-d53e9407f0a1}]" 2025-09-14T21:47:18.791Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:18.791Z [INFO] server-2541dbfe-d648-e821-3204-d53e9407f0a1: entering follower state: follower="Node at 2541dbfe-d648-e821-3204-d53e9407f0a1 [Follower]" leader-address= leader-id= 2025-09-14T21:47:18.852Z [WARN] server-2541dbfe-d648-e821-3204-d53e9407f0a1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:18.852Z [INFO] server-2541dbfe-d648-e821-3204-d53e9407f0a1: entering candidate state: node="Node at 2541dbfe-d648-e821-3204-d53e9407f0a1 [Candidate]" term=2 2025-09-14T21:47:18.852Z [INFO] server-2541dbfe-d648-e821-3204-d53e9407f0a1: election won: term=2 tally=2 2025-09-14T21:47:18.852Z [INFO] server-2541dbfe-d648-e821-3204-d53e9407f0a1: entering leader state: leader="Node at 2541dbfe-d648-e821-3204-d53e9407f0a1 [Leader]" 2025-09-14T21:47:18.852Z [INFO] server-2541dbfe-d648-e821-3204-d53e9407f0a1: added peer, starting replication: peer=server-5e83f3e5-7527-4a17-e825-cb7179d78783 2025-09-14T21:47:18.852Z [INFO] server-2541dbfe-d648-e821-3204-d53e9407f0a1: added peer, starting replication: peer=server-96451686-fc28-65ee-2a3d-d1503c21433a 2025-09-14T21:47:18.852Z [INFO] server-2541dbfe-d648-e821-3204-d53e9407f0a1: pipelining replication: peer="{Voter server-96451686-fc28-65ee-2a3d-d1503c21433a 96451686-fc28-65ee-2a3d-d1503c21433a}" 2025-09-14T21:47:18.852Z [INFO] server-2541dbfe-d648-e821-3204-d53e9407f0a1: pipelining replication: peer="{Voter server-5e83f3e5-7527-4a17-e825-cb7179d78783 5e83f3e5-7527-4a17-e825-cb7179d78783}" 2025-09-14T21:47:18.958Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:18.852779961 +0000 UTC m=+9.420658224 (1 nodes), 60.879425ms from start of poll, 61.221211ms from cluster start. Timeout at 2025-09-14 21:47:18.957783402 +0000 UTC m=+9.525661683, 105.003459ms after stability 2025-09-14T21:47:18.963Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:19.069Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:18.963776553 +0000 UTC m=+9.531654816 (2 nodes), 2.515µs from start of poll, 172.217803ms from cluster start. Timeout at 2025-09-14 21:47:19.068785156 +0000 UTC m=+9.636663454, 105.008638ms after stability 2025-09-14T21:47:19.069Z [INFO] server-2541dbfe-d648-e821-3204-d53e9407f0a1: updating configuration: command=RemoveServer server-id=server-5e83f3e5-7527-4a17-e825-cb7179d78783 server-addr= servers="[{Suffrage:Voter ID:server-96451686-fc28-65ee-2a3d-d1503c21433a Address:96451686-fc28-65ee-2a3d-d1503c21433a} {Suffrage:Voter ID:server-2541dbfe-d648-e821-3204-d53e9407f0a1 Address:2541dbfe-d648-e821-3204-d53e9407f0a1}]" 2025-09-14T21:47:19.069Z [INFO] server-2541dbfe-d648-e821-3204-d53e9407f0a1: removed peer, stopping replication: peer=server-5e83f3e5-7527-4a17-e825-cb7179d78783 last-index=3 2025-09-14T21:47:19.069Z [INFO] server-2541dbfe-d648-e821-3204-d53e9407f0a1: aborting pipeline replication: peer="{Voter server-5e83f3e5-7527-4a17-e825-cb7179d78783 5e83f3e5-7527-4a17-e825-cb7179d78783}" 2025-09-14T21:47:19.120Z [WARN] server-5e83f3e5-7527-4a17-e825-cb7179d78783: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2025-09-14T21:47:19.175Z [INFO] server-2541dbfe-d648-e821-3204-d53e9407f0a1: aborting pipeline replication: peer="{Voter server-96451686-fc28-65ee-2a3d-d1503c21433a 96451686-fc28-65ee-2a3d-d1503c21433a}" --- PASS: TestRaft_RemoveFollower (0.39s) === RUN TestRaft_RemoveLeader 2025-09-14T21:47:19.176Z [INFO] server-5f30675e-af9f-7406-2fe8-42a5926f0307: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f30675e-af9f-7406-2fe8-42a5926f0307 Address:5f30675e-af9f-7406-2fe8-42a5926f0307} {Suffrage:Voter ID:server-299e8461-13b6-6985-9dd1-d899d13a87bb Address:299e8461-13b6-6985-9dd1-d899d13a87bb} {Suffrage:Voter ID:server-09a5208b-14cb-7482-51dc-a2b232d2f6dd Address:09a5208b-14cb-7482-51dc-a2b232d2f6dd}]" 2025-09-14T21:47:19.176Z [INFO] server-299e8461-13b6-6985-9dd1-d899d13a87bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f30675e-af9f-7406-2fe8-42a5926f0307 Address:5f30675e-af9f-7406-2fe8-42a5926f0307} {Suffrage:Voter ID:server-299e8461-13b6-6985-9dd1-d899d13a87bb Address:299e8461-13b6-6985-9dd1-d899d13a87bb} {Suffrage:Voter ID:server-09a5208b-14cb-7482-51dc-a2b232d2f6dd Address:09a5208b-14cb-7482-51dc-a2b232d2f6dd}]" 2025-09-14T21:47:19.176Z [INFO] server-5f30675e-af9f-7406-2fe8-42a5926f0307: entering follower state: follower="Node at 5f30675e-af9f-7406-2fe8-42a5926f0307 [Follower]" leader-address= leader-id= 2025-09-14T21:47:19.176Z [INFO] server-299e8461-13b6-6985-9dd1-d899d13a87bb: entering follower state: follower="Node at 299e8461-13b6-6985-9dd1-d899d13a87bb [Follower]" leader-address= leader-id= 2025-09-14T21:47:19.176Z [INFO] server-09a5208b-14cb-7482-51dc-a2b232d2f6dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f30675e-af9f-7406-2fe8-42a5926f0307 Address:5f30675e-af9f-7406-2fe8-42a5926f0307} {Suffrage:Voter ID:server-299e8461-13b6-6985-9dd1-d899d13a87bb Address:299e8461-13b6-6985-9dd1-d899d13a87bb} {Suffrage:Voter ID:server-09a5208b-14cb-7482-51dc-a2b232d2f6dd Address:09a5208b-14cb-7482-51dc-a2b232d2f6dd}]" 2025-09-14T21:47:19.176Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:19.176Z [INFO] server-09a5208b-14cb-7482-51dc-a2b232d2f6dd: entering follower state: follower="Node at 09a5208b-14cb-7482-51dc-a2b232d2f6dd [Follower]" leader-address= leader-id= 2025-09-14T21:47:19.254Z [WARN] server-299e8461-13b6-6985-9dd1-d899d13a87bb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:19.254Z [INFO] server-299e8461-13b6-6985-9dd1-d899d13a87bb: entering candidate state: node="Node at 299e8461-13b6-6985-9dd1-d899d13a87bb [Candidate]" term=2 2025-09-14T21:47:19.254Z [INFO] server-299e8461-13b6-6985-9dd1-d899d13a87bb: election won: term=2 tally=2 2025-09-14T21:47:19.254Z [INFO] server-299e8461-13b6-6985-9dd1-d899d13a87bb: entering leader state: leader="Node at 299e8461-13b6-6985-9dd1-d899d13a87bb [Leader]" 2025-09-14T21:47:19.254Z [INFO] server-299e8461-13b6-6985-9dd1-d899d13a87bb: added peer, starting replication: peer=server-5f30675e-af9f-7406-2fe8-42a5926f0307 2025-09-14T21:47:19.254Z [INFO] server-299e8461-13b6-6985-9dd1-d899d13a87bb: added peer, starting replication: peer=server-09a5208b-14cb-7482-51dc-a2b232d2f6dd 2025-09-14T21:47:19.254Z [INFO] server-299e8461-13b6-6985-9dd1-d899d13a87bb: pipelining replication: peer="{Voter server-5f30675e-af9f-7406-2fe8-42a5926f0307 5f30675e-af9f-7406-2fe8-42a5926f0307}" 2025-09-14T21:47:19.254Z [INFO] server-299e8461-13b6-6985-9dd1-d899d13a87bb: pipelining replication: peer="{Voter server-09a5208b-14cb-7482-51dc-a2b232d2f6dd 09a5208b-14cb-7482-51dc-a2b232d2f6dd}" 2025-09-14T21:47:19.360Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:19.254864354 +0000 UTC m=+9.822742616 (1 nodes), 78.484002ms from start of poll, 78.783957ms from cluster start. Timeout at 2025-09-14 21:47:19.359865375 +0000 UTC m=+9.927743638, 105.001022ms after stability 2025-09-14T21:47:19.366Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:19.472Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:19.366298823 +0000 UTC m=+9.934177140 (2 nodes), 2.723µs from start of poll, 190.218481ms from cluster start. Timeout at 2025-09-14 21:47:19.471304721 +0000 UTC m=+10.039182996, 105.005856ms after stability 2025-09-14T21:47:19.472Z [INFO] server-299e8461-13b6-6985-9dd1-d899d13a87bb: updating configuration: command=RemoveServer server-id=server-299e8461-13b6-6985-9dd1-d899d13a87bb server-addr= servers="[{Suffrage:Voter ID:server-5f30675e-af9f-7406-2fe8-42a5926f0307 Address:5f30675e-af9f-7406-2fe8-42a5926f0307} {Suffrage:Voter ID:server-09a5208b-14cb-7482-51dc-a2b232d2f6dd Address:09a5208b-14cb-7482-51dc-a2b232d2f6dd}]" 2025-09-14T21:47:19.472Z [INFO] server-299e8461-13b6-6985-9dd1-d899d13a87bb: removed ourself, shutting down 2025-09-14T21:47:19.472Z [INFO] server-299e8461-13b6-6985-9dd1-d899d13a87bb: aborting pipeline replication: peer="{Voter server-09a5208b-14cb-7482-51dc-a2b232d2f6dd 09a5208b-14cb-7482-51dc-a2b232d2f6dd}" 2025-09-14T21:47:19.472Z [INFO] server-299e8461-13b6-6985-9dd1-d899d13a87bb: aborting pipeline replication: peer="{Voter server-5f30675e-af9f-7406-2fe8-42a5926f0307 5f30675e-af9f-7406-2fe8-42a5926f0307}" 2025-09-14T21:47:19.541Z [WARN] server-5f30675e-af9f-7406-2fe8-42a5926f0307: heartbeat timeout reached, starting election: last-leader-addr=299e8461-13b6-6985-9dd1-d899d13a87bb last-leader-id=server-299e8461-13b6-6985-9dd1-d899d13a87bb 2025-09-14T21:47:19.541Z [INFO] server-5f30675e-af9f-7406-2fe8-42a5926f0307: entering candidate state: node="Node at 5f30675e-af9f-7406-2fe8-42a5926f0307 [Candidate]" term=3 2025-09-14T21:47:19.541Z [WARN] server-09a5208b-14cb-7482-51dc-a2b232d2f6dd: rejecting vote request since we have a leader: from=5f30675e-af9f-7406-2fe8-42a5926f0307 leader=299e8461-13b6-6985-9dd1-d899d13a87bb leader-id=server-299e8461-13b6-6985-9dd1-d899d13a87bb 2025-09-14T21:47:19.559Z [WARN] server-09a5208b-14cb-7482-51dc-a2b232d2f6dd: heartbeat timeout reached, starting election: last-leader-addr=299e8461-13b6-6985-9dd1-d899d13a87bb last-leader-id=server-299e8461-13b6-6985-9dd1-d899d13a87bb 2025-09-14T21:47:19.559Z [INFO] server-09a5208b-14cb-7482-51dc-a2b232d2f6dd: entering candidate state: node="Node at 09a5208b-14cb-7482-51dc-a2b232d2f6dd [Candidate]" term=3 2025-09-14T21:47:19.559Z [INFO] server-5f30675e-af9f-7406-2fe8-42a5926f0307: duplicate requestVote for same term: term=3 2025-09-14T21:47:19.626Z [WARN] server-09a5208b-14cb-7482-51dc-a2b232d2f6dd: Election timeout reached, restarting election 2025-09-14T21:47:19.626Z [INFO] server-09a5208b-14cb-7482-51dc-a2b232d2f6dd: entering candidate state: node="Node at 09a5208b-14cb-7482-51dc-a2b232d2f6dd [Candidate]" term=4 2025-09-14T21:47:19.626Z [INFO] server-5f30675e-af9f-7406-2fe8-42a5926f0307: entering follower state: follower="Node at 5f30675e-af9f-7406-2fe8-42a5926f0307 [Follower]" leader-address= leader-id= 2025-09-14T21:47:19.626Z [INFO] server-09a5208b-14cb-7482-51dc-a2b232d2f6dd: election won: term=4 tally=2 2025-09-14T21:47:19.626Z [INFO] server-09a5208b-14cb-7482-51dc-a2b232d2f6dd: entering leader state: leader="Node at 09a5208b-14cb-7482-51dc-a2b232d2f6dd [Leader]" 2025-09-14T21:47:19.626Z [INFO] server-09a5208b-14cb-7482-51dc-a2b232d2f6dd: added peer, starting replication: peer=server-5f30675e-af9f-7406-2fe8-42a5926f0307 2025-09-14T21:47:19.626Z [INFO] server-09a5208b-14cb-7482-51dc-a2b232d2f6dd: pipelining replication: peer="{Voter server-5f30675e-af9f-7406-2fe8-42a5926f0307 5f30675e-af9f-7406-2fe8-42a5926f0307}" 2025-09-14T21:47:19.684Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:19.789Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:19.684352399 +0000 UTC m=+10.252230679 (1 nodes), 44.859µs from start of poll, 508.27202ms from cluster start. Timeout at 2025-09-14 21:47:19.789354725 +0000 UTC m=+10.357232988, 105.002309ms after stability 2025-09-14T21:47:19.789Z [INFO] server-09a5208b-14cb-7482-51dc-a2b232d2f6dd: aborting pipeline replication: peer="{Voter server-5f30675e-af9f-7406-2fe8-42a5926f0307 5f30675e-af9f-7406-2fe8-42a5926f0307}" --- PASS: TestRaft_RemoveLeader (0.61s) === RUN TestRaft_RemoveLeader_NoShutdown 2025-09-14T21:47:19.790Z [INFO] server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822 Address:ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822} {Suffrage:Voter ID:server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b Address:2483e3cc-1a10-4f9f-fdc9-24622aeab57b} {Suffrage:Voter ID:server-be859453-cb57-1140-c317-3884fe354eaa Address:be859453-cb57-1140-c317-3884fe354eaa}]" 2025-09-14T21:47:19.791Z [INFO] server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822 Address:ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822} {Suffrage:Voter ID:server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b Address:2483e3cc-1a10-4f9f-fdc9-24622aeab57b} {Suffrage:Voter ID:server-be859453-cb57-1140-c317-3884fe354eaa Address:be859453-cb57-1140-c317-3884fe354eaa}]" 2025-09-14T21:47:19.791Z [INFO] server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822: entering follower state: follower="Node at ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822 [Follower]" leader-address= leader-id= 2025-09-14T21:47:19.791Z [INFO] server-be859453-cb57-1140-c317-3884fe354eaa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822 Address:ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822} {Suffrage:Voter ID:server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b Address:2483e3cc-1a10-4f9f-fdc9-24622aeab57b} {Suffrage:Voter ID:server-be859453-cb57-1140-c317-3884fe354eaa Address:be859453-cb57-1140-c317-3884fe354eaa}]" 2025-09-14T21:47:19.791Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:19.791Z [INFO] server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b: entering follower state: follower="Node at 2483e3cc-1a10-4f9f-fdc9-24622aeab57b [Follower]" leader-address= leader-id= 2025-09-14T21:47:19.791Z [INFO] server-be859453-cb57-1140-c317-3884fe354eaa: entering follower state: follower="Node at be859453-cb57-1140-c317-3884fe354eaa [Follower]" leader-address= leader-id= 2025-09-14T21:47:19.847Z [WARN] server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:19.847Z [INFO] server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b: entering candidate state: node="Node at 2483e3cc-1a10-4f9f-fdc9-24622aeab57b [Candidate]" term=2 2025-09-14T21:47:19.847Z [INFO] server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b: election won: term=2 tally=2 2025-09-14T21:47:19.847Z [INFO] server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b: entering leader state: leader="Node at 2483e3cc-1a10-4f9f-fdc9-24622aeab57b [Leader]" 2025-09-14T21:47:19.847Z [INFO] server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b: added peer, starting replication: peer=server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822 2025-09-14T21:47:19.847Z [INFO] server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b: added peer, starting replication: peer=server-be859453-cb57-1140-c317-3884fe354eaa 2025-09-14T21:47:19.847Z [INFO] server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b: pipelining replication: peer="{Voter server-be859453-cb57-1140-c317-3884fe354eaa be859453-cb57-1140-c317-3884fe354eaa}" 2025-09-14T21:47:19.847Z [INFO] server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b: pipelining replication: peer="{Voter server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822 ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822}" 2025-09-14T21:47:19.953Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:19.84745831 +0000 UTC m=+10.415336596 (2 nodes), 56.240126ms from start of poll, 56.586094ms from cluster start. Timeout at 2025-09-14 21:47:19.952462176 +0000 UTC m=+10.520340435, 105.003839ms after stability 2025-09-14T21:47:19.953Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:20.058Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:19.95311803 +0000 UTC m=+10.520996293 (1 nodes), 1.551µs from start of poll, 162.245791ms from cluster start. Timeout at 2025-09-14 21:47:20.058123189 +0000 UTC m=+10.626001464, 105.005171ms after stability 2025-09-14T21:47:20.061Z [INFO] server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b: updating configuration: command=RemoveServer server-id=server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b server-addr= servers="[{Suffrage:Voter ID:server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822 Address:ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822} {Suffrage:Voter ID:server-be859453-cb57-1140-c317-3884fe354eaa Address:be859453-cb57-1140-c317-3884fe354eaa}]" 2025-09-14T21:47:20.061Z [INFO] server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b: removed ourself, transitioning to follower 2025-09-14T21:47:20.061Z [INFO] server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b: entering follower state: follower="Node at 2483e3cc-1a10-4f9f-fdc9-24622aeab57b [Follower]" leader-address= leader-id= 2025-09-14T21:47:20.061Z [INFO] server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b: aborting pipeline replication: peer="{Voter server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822 ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822}" 2025-09-14T21:47:20.061Z [INFO] server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b: aborting pipeline replication: peer="{Voter server-be859453-cb57-1140-c317-3884fe354eaa be859453-cb57-1140-c317-3884fe354eaa}" 2025-09-14T21:47:20.114Z [WARN] server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822: heartbeat timeout reached, starting election: last-leader-addr=2483e3cc-1a10-4f9f-fdc9-24622aeab57b last-leader-id=server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b 2025-09-14T21:47:20.114Z [INFO] server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822: entering candidate state: node="Node at ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822 [Candidate]" term=3 2025-09-14T21:47:20.114Z [WARN] server-be859453-cb57-1140-c317-3884fe354eaa: rejecting vote request since we have a leader: from=ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822 leader=2483e3cc-1a10-4f9f-fdc9-24622aeab57b leader-id=server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b 2025-09-14T21:47:20.132Z [WARN] server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b: not part of stable configuration, aborting election 2025-09-14T21:47:20.150Z [WARN] server-be859453-cb57-1140-c317-3884fe354eaa: heartbeat timeout reached, starting election: last-leader-addr=2483e3cc-1a10-4f9f-fdc9-24622aeab57b last-leader-id=server-2483e3cc-1a10-4f9f-fdc9-24622aeab57b 2025-09-14T21:47:20.150Z [INFO] server-be859453-cb57-1140-c317-3884fe354eaa: entering candidate state: node="Node at be859453-cb57-1140-c317-3884fe354eaa [Candidate]" term=3 2025-09-14T21:47:20.150Z [INFO] server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822: duplicate requestVote for same term: term=3 2025-09-14T21:47:20.167Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:20.213Z [WARN] server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822: Election timeout reached, restarting election 2025-09-14T21:47:20.213Z [INFO] server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822: entering candidate state: node="Node at ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822 [Candidate]" term=4 2025-09-14T21:47:20.213Z [INFO] server-be859453-cb57-1140-c317-3884fe354eaa: entering follower state: follower="Node at be859453-cb57-1140-c317-3884fe354eaa [Follower]" leader-address= leader-id= 2025-09-14T21:47:20.213Z [INFO] server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822: election won: term=4 tally=2 2025-09-14T21:47:20.213Z [INFO] server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822: entering leader state: leader="Node at ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822 [Leader]" 2025-09-14T21:47:20.213Z [INFO] server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822: added peer, starting replication: peer=server-be859453-cb57-1140-c317-3884fe354eaa 2025-09-14T21:47:20.213Z [INFO] server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822: pipelining replication: peer="{Voter server-be859453-cb57-1140-c317-3884fe354eaa be859453-cb57-1140-c317-3884fe354eaa}" 2025-09-14T21:47:20.318Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:20.213269655 +0000 UTC m=+10.781147921 (1 nodes), 45.720334ms from start of poll, 422.397419ms from cluster start. Timeout at 2025-09-14 21:47:20.318272752 +0000 UTC m=+10.886151029, 105.003108ms after stability 2025-09-14T21:47:20.424Z [INFO] server-ac9cbd92-0ccf-b31a-8b37-7b6bc22b0822: aborting pipeline replication: peer="{Voter server-be859453-cb57-1140-c317-3884fe354eaa be859453-cb57-1140-c317-3884fe354eaa}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.63s) === RUN TestRaft_RemoveFollower_SplitCluster 2025-09-14T21:47:20.425Z [INFO] server-faee056d-80cb-341f-c47a-c948c5b86cce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-faee056d-80cb-341f-c47a-c948c5b86cce Address:faee056d-80cb-341f-c47a-c948c5b86cce} {Suffrage:Voter ID:server-c78b1dfd-013d-ba19-c36e-94b70b2b5c72 Address:c78b1dfd-013d-ba19-c36e-94b70b2b5c72} {Suffrage:Voter ID:server-e186ddae-a9ed-c879-1dcb-17da1b5d0bb0 Address:e186ddae-a9ed-c879-1dcb-17da1b5d0bb0} {Suffrage:Voter ID:server-47798b0d-660f-6356-8524-816d28240b2f Address:47798b0d-660f-6356-8524-816d28240b2f}]" 2025-09-14T21:47:20.425Z [INFO] server-faee056d-80cb-341f-c47a-c948c5b86cce: entering follower state: follower="Node at faee056d-80cb-341f-c47a-c948c5b86cce [Follower]" leader-address= leader-id= 2025-09-14T21:47:20.425Z [INFO] server-c78b1dfd-013d-ba19-c36e-94b70b2b5c72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-faee056d-80cb-341f-c47a-c948c5b86cce Address:faee056d-80cb-341f-c47a-c948c5b86cce} {Suffrage:Voter ID:server-c78b1dfd-013d-ba19-c36e-94b70b2b5c72 Address:c78b1dfd-013d-ba19-c36e-94b70b2b5c72} {Suffrage:Voter ID:server-e186ddae-a9ed-c879-1dcb-17da1b5d0bb0 Address:e186ddae-a9ed-c879-1dcb-17da1b5d0bb0} {Suffrage:Voter ID:server-47798b0d-660f-6356-8524-816d28240b2f Address:47798b0d-660f-6356-8524-816d28240b2f}]" 2025-09-14T21:47:20.425Z [INFO] server-c78b1dfd-013d-ba19-c36e-94b70b2b5c72: entering follower state: follower="Node at c78b1dfd-013d-ba19-c36e-94b70b2b5c72 [Follower]" leader-address= leader-id= 2025-09-14T21:47:20.425Z [INFO] server-e186ddae-a9ed-c879-1dcb-17da1b5d0bb0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-faee056d-80cb-341f-c47a-c948c5b86cce Address:faee056d-80cb-341f-c47a-c948c5b86cce} {Suffrage:Voter ID:server-c78b1dfd-013d-ba19-c36e-94b70b2b5c72 Address:c78b1dfd-013d-ba19-c36e-94b70b2b5c72} {Suffrage:Voter ID:server-e186ddae-a9ed-c879-1dcb-17da1b5d0bb0 Address:e186ddae-a9ed-c879-1dcb-17da1b5d0bb0} {Suffrage:Voter ID:server-47798b0d-660f-6356-8524-816d28240b2f Address:47798b0d-660f-6356-8524-816d28240b2f}]" 2025-09-14T21:47:20.426Z [INFO] server-e186ddae-a9ed-c879-1dcb-17da1b5d0bb0: entering follower state: follower="Node at e186ddae-a9ed-c879-1dcb-17da1b5d0bb0 [Follower]" leader-address= leader-id= 2025-09-14T21:47:20.426Z [INFO] server-47798b0d-660f-6356-8524-816d28240b2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-faee056d-80cb-341f-c47a-c948c5b86cce Address:faee056d-80cb-341f-c47a-c948c5b86cce} {Suffrage:Voter ID:server-c78b1dfd-013d-ba19-c36e-94b70b2b5c72 Address:c78b1dfd-013d-ba19-c36e-94b70b2b5c72} {Suffrage:Voter ID:server-e186ddae-a9ed-c879-1dcb-17da1b5d0bb0 Address:e186ddae-a9ed-c879-1dcb-17da1b5d0bb0} {Suffrage:Voter ID:server-47798b0d-660f-6356-8524-816d28240b2f Address:47798b0d-660f-6356-8524-816d28240b2f}]" 2025-09-14T21:47:20.426Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:20.426Z [INFO] server-47798b0d-660f-6356-8524-816d28240b2f: entering follower state: follower="Node at 47798b0d-660f-6356-8524-816d28240b2f [Follower]" leader-address= leader-id= 2025-09-14T21:47:20.481Z [WARN] server-faee056d-80cb-341f-c47a-c948c5b86cce: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:20.481Z [INFO] server-faee056d-80cb-341f-c47a-c948c5b86cce: entering candidate state: node="Node at faee056d-80cb-341f-c47a-c948c5b86cce [Candidate]" term=2 2025-09-14T21:47:20.481Z [INFO] server-faee056d-80cb-341f-c47a-c948c5b86cce: election won: term=2 tally=3 2025-09-14T21:47:20.481Z [INFO] server-faee056d-80cb-341f-c47a-c948c5b86cce: entering leader state: leader="Node at faee056d-80cb-341f-c47a-c948c5b86cce [Leader]" 2025-09-14T21:47:20.481Z [INFO] server-faee056d-80cb-341f-c47a-c948c5b86cce: added peer, starting replication: peer=server-c78b1dfd-013d-ba19-c36e-94b70b2b5c72 2025-09-14T21:47:20.481Z [INFO] server-faee056d-80cb-341f-c47a-c948c5b86cce: added peer, starting replication: peer=server-e186ddae-a9ed-c879-1dcb-17da1b5d0bb0 2025-09-14T21:47:20.481Z [INFO] server-faee056d-80cb-341f-c47a-c948c5b86cce: added peer, starting replication: peer=server-47798b0d-660f-6356-8524-816d28240b2f 2025-09-14T21:47:20.481Z [INFO] server-faee056d-80cb-341f-c47a-c948c5b86cce: pipelining replication: peer="{Voter server-47798b0d-660f-6356-8524-816d28240b2f 47798b0d-660f-6356-8524-816d28240b2f}" 2025-09-14T21:47:20.481Z [INFO] server-faee056d-80cb-341f-c47a-c948c5b86cce: pipelining replication: peer="{Voter server-c78b1dfd-013d-ba19-c36e-94b70b2b5c72 c78b1dfd-013d-ba19-c36e-94b70b2b5c72}" 2025-09-14T21:47:20.481Z [INFO] server-faee056d-80cb-341f-c47a-c948c5b86cce: pipelining replication: peer="{Voter server-e186ddae-a9ed-c879-1dcb-17da1b5d0bb0 e186ddae-a9ed-c879-1dcb-17da1b5d0bb0}" 2025-09-14T21:47:20.587Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:20.481348678 +0000 UTC m=+11.049226939 (1 nodes), 55.215884ms from start of poll, 55.78176ms from cluster start. Timeout at 2025-09-14 21:47:20.586351433 +0000 UTC m=+11.154229692, 105.002753ms after stability 2025-09-14T21:47:20.693Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:20.798Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:20.693350225 +0000 UTC m=+11.261228500 (3 nodes), 1.873µs from start of poll, 267.783321ms from cluster start. Timeout at 2025-09-14 21:47:20.798356617 +0000 UTC m=+11.366234879, 105.006379ms after stability 2025-09-14T21:47:20.798Z [INFO] server-faee056d-80cb-341f-c47a-c948c5b86cce: updating configuration: command=RemoveServer server-id=server-47798b0d-660f-6356-8524-816d28240b2f server-addr= servers="[{Suffrage:Voter ID:server-faee056d-80cb-341f-c47a-c948c5b86cce Address:faee056d-80cb-341f-c47a-c948c5b86cce} {Suffrage:Voter ID:server-c78b1dfd-013d-ba19-c36e-94b70b2b5c72 Address:c78b1dfd-013d-ba19-c36e-94b70b2b5c72} {Suffrage:Voter ID:server-e186ddae-a9ed-c879-1dcb-17da1b5d0bb0 Address:e186ddae-a9ed-c879-1dcb-17da1b5d0bb0}]" 2025-09-14T21:47:20.798Z [INFO] server-faee056d-80cb-341f-c47a-c948c5b86cce: removed peer, stopping replication: peer=server-47798b0d-660f-6356-8524-816d28240b2f last-index=3 2025-09-14T21:47:20.798Z [ERROR] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to pipeline appendEntries: peer="{Voter server-c78b1dfd-013d-ba19-c36e-94b70b2b5c72 c78b1dfd-013d-ba19-c36e-94b70b2b5c72}" error="append pipeline closed" 2025-09-14T21:47:20.798Z [INFO] server-faee056d-80cb-341f-c47a-c948c5b86cce: aborting pipeline replication: peer="{Voter server-c78b1dfd-013d-ba19-c36e-94b70b2b5c72 c78b1dfd-013d-ba19-c36e-94b70b2b5c72}" 2025-09-14T21:47:20.798Z [ERROR] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to pipeline appendEntries: peer="{Voter server-e186ddae-a9ed-c879-1dcb-17da1b5d0bb0 e186ddae-a9ed-c879-1dcb-17da1b5d0bb0}" error="append pipeline closed" 2025-09-14T21:47:20.798Z [INFO] server-faee056d-80cb-341f-c47a-c948c5b86cce: aborting pipeline replication: peer="{Voter server-e186ddae-a9ed-c879-1dcb-17da1b5d0bb0 e186ddae-a9ed-c879-1dcb-17da1b5d0bb0}" 2025-09-14T21:47:20.798Z [INFO] server-faee056d-80cb-341f-c47a-c948c5b86cce: aborting pipeline replication: peer="{Voter server-47798b0d-660f-6356-8524-816d28240b2f 47798b0d-660f-6356-8524-816d28240b2f}" 2025-09-14T21:47:20.802Z [ERROR] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to heartbeat to: peer=e186ddae-a9ed-c879-1dcb-17da1b5d0bb0 backoff time=10ms error="failed to connect to peer: e186ddae-a9ed-c879-1dcb-17da1b5d0bb0" 2025-09-14T21:47:20.802Z [ERROR] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to heartbeat to: peer=c78b1dfd-013d-ba19-c36e-94b70b2b5c72 backoff time=10ms error="failed to connect to peer: c78b1dfd-013d-ba19-c36e-94b70b2b5c72" 2025-09-14T21:47:20.806Z [ERROR] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to appendEntries to: peer="{Voter server-c78b1dfd-013d-ba19-c36e-94b70b2b5c72 c78b1dfd-013d-ba19-c36e-94b70b2b5c72}" error="failed to connect to peer: c78b1dfd-013d-ba19-c36e-94b70b2b5c72" 2025-09-14T21:47:20.809Z [ERROR] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to appendEntries to: peer="{Voter server-e186ddae-a9ed-c879-1dcb-17da1b5d0bb0 e186ddae-a9ed-c879-1dcb-17da1b5d0bb0}" error="failed to connect to peer: e186ddae-a9ed-c879-1dcb-17da1b5d0bb0" 2025-09-14T21:47:20.819Z [ERROR] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to heartbeat to: peer=e186ddae-a9ed-c879-1dcb-17da1b5d0bb0 backoff time=10ms error="failed to connect to peer: e186ddae-a9ed-c879-1dcb-17da1b5d0bb0" 2025-09-14T21:47:20.819Z [ERROR] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to heartbeat to: peer=c78b1dfd-013d-ba19-c36e-94b70b2b5c72 backoff time=10ms error="failed to connect to peer: c78b1dfd-013d-ba19-c36e-94b70b2b5c72" 2025-09-14T21:47:20.822Z [ERROR] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to appendEntries to: peer="{Voter server-c78b1dfd-013d-ba19-c36e-94b70b2b5c72 c78b1dfd-013d-ba19-c36e-94b70b2b5c72}" error="failed to connect to peer: c78b1dfd-013d-ba19-c36e-94b70b2b5c72" 2025-09-14T21:47:20.826Z [ERROR] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to appendEntries to: peer="{Voter server-e186ddae-a9ed-c879-1dcb-17da1b5d0bb0 e186ddae-a9ed-c879-1dcb-17da1b5d0bb0}" error="failed to connect to peer: e186ddae-a9ed-c879-1dcb-17da1b5d0bb0" 2025-09-14T21:47:20.836Z [ERROR] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to heartbeat to: peer=e186ddae-a9ed-c879-1dcb-17da1b5d0bb0 backoff time=10ms error="failed to connect to peer: e186ddae-a9ed-c879-1dcb-17da1b5d0bb0" 2025-09-14T21:47:20.839Z [ERROR] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to heartbeat to: peer=c78b1dfd-013d-ba19-c36e-94b70b2b5c72 backoff time=10ms error="failed to connect to peer: c78b1dfd-013d-ba19-c36e-94b70b2b5c72" 2025-09-14T21:47:20.840Z [ERROR] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to appendEntries to: peer="{Voter server-c78b1dfd-013d-ba19-c36e-94b70b2b5c72 c78b1dfd-013d-ba19-c36e-94b70b2b5c72}" error="failed to connect to peer: c78b1dfd-013d-ba19-c36e-94b70b2b5c72" 2025-09-14T21:47:20.843Z [ERROR] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to appendEntries to: peer="{Voter server-e186ddae-a9ed-c879-1dcb-17da1b5d0bb0 e186ddae-a9ed-c879-1dcb-17da1b5d0bb0}" error="failed to connect to peer: e186ddae-a9ed-c879-1dcb-17da1b5d0bb0" 2025-09-14T21:47:20.846Z [WARN] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to contact: server-id=server-c78b1dfd-013d-ba19-c36e-94b70b2b5c72 time=50.318651ms 2025-09-14T21:47:20.846Z [WARN] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to contact: server-id=server-e186ddae-a9ed-c879-1dcb-17da1b5d0bb0 time=50.293635ms 2025-09-14T21:47:20.846Z [WARN] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to contact quorum of nodes, stepping down 2025-09-14T21:47:20.846Z [INFO] server-faee056d-80cb-341f-c47a-c948c5b86cce: entering follower state: follower="Node at faee056d-80cb-341f-c47a-c948c5b86cce [Follower]" leader-address= leader-id= 2025-09-14T21:47:20.846Z [ERROR] server-faee056d-80cb-341f-c47a-c948c5b86cce: failed to appendEntries to: peer="{Voter server-c78b1dfd-013d-ba19-c36e-94b70b2b5c72 c78b1dfd-013d-ba19-c36e-94b70b2b5c72}" error="failed to connect to peer: c78b1dfd-013d-ba19-c36e-94b70b2b5c72" --- PASS: TestRaft_RemoveFollower_SplitCluster (0.42s) === RUN TestRaft_AddKnownPeer 2025-09-14T21:47:20.848Z [INFO] server-c82cba46-a061-5621-d598-0400385b22ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c82cba46-a061-5621-d598-0400385b22ea Address:c82cba46-a061-5621-d598-0400385b22ea} {Suffrage:Voter ID:server-4d9821fa-7797-fb89-8d42-e532bed29da9 Address:4d9821fa-7797-fb89-8d42-e532bed29da9} {Suffrage:Voter ID:server-d963c799-dbf9-c003-1fe6-0142820883c9 Address:d963c799-dbf9-c003-1fe6-0142820883c9}]" 2025-09-14T21:47:20.848Z [INFO] server-4d9821fa-7797-fb89-8d42-e532bed29da9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c82cba46-a061-5621-d598-0400385b22ea Address:c82cba46-a061-5621-d598-0400385b22ea} {Suffrage:Voter ID:server-4d9821fa-7797-fb89-8d42-e532bed29da9 Address:4d9821fa-7797-fb89-8d42-e532bed29da9} {Suffrage:Voter ID:server-d963c799-dbf9-c003-1fe6-0142820883c9 Address:d963c799-dbf9-c003-1fe6-0142820883c9}]" 2025-09-14T21:47:20.848Z [INFO] server-c82cba46-a061-5621-d598-0400385b22ea: entering follower state: follower="Node at c82cba46-a061-5621-d598-0400385b22ea [Follower]" leader-address= leader-id= 2025-09-14T21:47:20.848Z [INFO] server-d963c799-dbf9-c003-1fe6-0142820883c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c82cba46-a061-5621-d598-0400385b22ea Address:c82cba46-a061-5621-d598-0400385b22ea} {Suffrage:Voter ID:server-4d9821fa-7797-fb89-8d42-e532bed29da9 Address:4d9821fa-7797-fb89-8d42-e532bed29da9} {Suffrage:Voter ID:server-d963c799-dbf9-c003-1fe6-0142820883c9 Address:d963c799-dbf9-c003-1fe6-0142820883c9}]" 2025-09-14T21:47:20.848Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:20.848Z [INFO] server-d963c799-dbf9-c003-1fe6-0142820883c9: entering follower state: follower="Node at d963c799-dbf9-c003-1fe6-0142820883c9 [Follower]" leader-address= leader-id= 2025-09-14T21:47:20.848Z [INFO] server-4d9821fa-7797-fb89-8d42-e532bed29da9: entering follower state: follower="Node at 4d9821fa-7797-fb89-8d42-e532bed29da9 [Follower]" leader-address= leader-id= 2025-09-14T21:47:20.906Z [WARN] server-c82cba46-a061-5621-d598-0400385b22ea: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:20.906Z [INFO] server-c82cba46-a061-5621-d598-0400385b22ea: entering candidate state: node="Node at c82cba46-a061-5621-d598-0400385b22ea [Candidate]" term=2 2025-09-14T21:47:20.906Z [INFO] server-c82cba46-a061-5621-d598-0400385b22ea: election won: term=2 tally=2 2025-09-14T21:47:20.906Z [INFO] server-c82cba46-a061-5621-d598-0400385b22ea: entering leader state: leader="Node at c82cba46-a061-5621-d598-0400385b22ea [Leader]" 2025-09-14T21:47:20.906Z [INFO] server-c82cba46-a061-5621-d598-0400385b22ea: added peer, starting replication: peer=server-4d9821fa-7797-fb89-8d42-e532bed29da9 2025-09-14T21:47:20.906Z [INFO] server-c82cba46-a061-5621-d598-0400385b22ea: added peer, starting replication: peer=server-d963c799-dbf9-c003-1fe6-0142820883c9 2025-09-14T21:47:20.906Z [INFO] server-c82cba46-a061-5621-d598-0400385b22ea: pipelining replication: peer="{Voter server-d963c799-dbf9-c003-1fe6-0142820883c9 d963c799-dbf9-c003-1fe6-0142820883c9}" 2025-09-14T21:47:20.906Z [INFO] server-c82cba46-a061-5621-d598-0400385b22ea: pipelining replication: peer="{Voter server-4d9821fa-7797-fb89-8d42-e532bed29da9 4d9821fa-7797-fb89-8d42-e532bed29da9}" 2025-09-14T21:47:21.012Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:20.906937458 +0000 UTC m=+11.474815733 (1 nodes), 58.212471ms from start of poll, 58.463163ms from cluster start. Timeout at 2025-09-14 21:47:21.011939787 +0000 UTC m=+11.579818046, 105.002313ms after stability 2025-09-14T21:47:21.012Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:21.117Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:21.012301379 +0000 UTC m=+11.580179641 (2 nodes), 1.192µs from start of poll, 163.827071ms from cluster start. Timeout at 2025-09-14 21:47:21.117307069 +0000 UTC m=+11.685185334, 105.005693ms after stability 2025-09-14T21:47:21.117Z [INFO] server-c82cba46-a061-5621-d598-0400385b22ea: updating configuration: command=AddVoter server-id=server-4d9821fa-7797-fb89-8d42-e532bed29da9 server-addr=4d9821fa-7797-fb89-8d42-e532bed29da9 servers="[{Suffrage:Voter ID:server-c82cba46-a061-5621-d598-0400385b22ea Address:c82cba46-a061-5621-d598-0400385b22ea} {Suffrage:Voter ID:server-4d9821fa-7797-fb89-8d42-e532bed29da9 Address:4d9821fa-7797-fb89-8d42-e532bed29da9} {Suffrage:Voter ID:server-d963c799-dbf9-c003-1fe6-0142820883c9 Address:d963c799-dbf9-c003-1fe6-0142820883c9}]" 2025-09-14T21:47:21.118Z [INFO] server-c82cba46-a061-5621-d598-0400385b22ea: aborting pipeline replication: peer="{Voter server-d963c799-dbf9-c003-1fe6-0142820883c9 d963c799-dbf9-c003-1fe6-0142820883c9}" 2025-09-14T21:47:21.118Z [INFO] server-c82cba46-a061-5621-d598-0400385b22ea: aborting pipeline replication: peer="{Voter server-4d9821fa-7797-fb89-8d42-e532bed29da9 4d9821fa-7797-fb89-8d42-e532bed29da9}" --- PASS: TestRaft_AddKnownPeer (0.27s) === RUN TestRaft_RemoveUnknownPeer 2025-09-14T21:47:21.119Z [INFO] server-bfa9268d-e6cd-dd8f-ec86-cb0226aca609: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfa9268d-e6cd-dd8f-ec86-cb0226aca609 Address:bfa9268d-e6cd-dd8f-ec86-cb0226aca609} {Suffrage:Voter ID:server-50f76965-ad70-ce53-9948-760cd726d161 Address:50f76965-ad70-ce53-9948-760cd726d161} {Suffrage:Voter ID:server-90affe23-3e1b-d826-2b40-2ee158107580 Address:90affe23-3e1b-d826-2b40-2ee158107580}]" 2025-09-14T21:47:21.119Z [INFO] server-50f76965-ad70-ce53-9948-760cd726d161: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfa9268d-e6cd-dd8f-ec86-cb0226aca609 Address:bfa9268d-e6cd-dd8f-ec86-cb0226aca609} {Suffrage:Voter ID:server-50f76965-ad70-ce53-9948-760cd726d161 Address:50f76965-ad70-ce53-9948-760cd726d161} {Suffrage:Voter ID:server-90affe23-3e1b-d826-2b40-2ee158107580 Address:90affe23-3e1b-d826-2b40-2ee158107580}]" 2025-09-14T21:47:21.119Z [INFO] server-bfa9268d-e6cd-dd8f-ec86-cb0226aca609: entering follower state: follower="Node at bfa9268d-e6cd-dd8f-ec86-cb0226aca609 [Follower]" leader-address= leader-id= 2025-09-14T21:47:21.119Z [INFO] server-50f76965-ad70-ce53-9948-760cd726d161: entering follower state: follower="Node at 50f76965-ad70-ce53-9948-760cd726d161 [Follower]" leader-address= leader-id= 2025-09-14T21:47:21.119Z [INFO] server-90affe23-3e1b-d826-2b40-2ee158107580: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfa9268d-e6cd-dd8f-ec86-cb0226aca609 Address:bfa9268d-e6cd-dd8f-ec86-cb0226aca609} {Suffrage:Voter ID:server-50f76965-ad70-ce53-9948-760cd726d161 Address:50f76965-ad70-ce53-9948-760cd726d161} {Suffrage:Voter ID:server-90affe23-3e1b-d826-2b40-2ee158107580 Address:90affe23-3e1b-d826-2b40-2ee158107580}]" 2025-09-14T21:47:21.119Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:21.119Z [INFO] server-90affe23-3e1b-d826-2b40-2ee158107580: entering follower state: follower="Node at 90affe23-3e1b-d826-2b40-2ee158107580 [Follower]" leader-address= leader-id= 2025-09-14T21:47:21.175Z [WARN] server-50f76965-ad70-ce53-9948-760cd726d161: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:21.175Z [INFO] server-50f76965-ad70-ce53-9948-760cd726d161: entering candidate state: node="Node at 50f76965-ad70-ce53-9948-760cd726d161 [Candidate]" term=2 2025-09-14T21:47:21.175Z [INFO] server-50f76965-ad70-ce53-9948-760cd726d161: election won: term=2 tally=2 2025-09-14T21:47:21.175Z [INFO] server-50f76965-ad70-ce53-9948-760cd726d161: entering leader state: leader="Node at 50f76965-ad70-ce53-9948-760cd726d161 [Leader]" 2025-09-14T21:47:21.175Z [INFO] server-50f76965-ad70-ce53-9948-760cd726d161: added peer, starting replication: peer=server-bfa9268d-e6cd-dd8f-ec86-cb0226aca609 2025-09-14T21:47:21.175Z [INFO] server-50f76965-ad70-ce53-9948-760cd726d161: added peer, starting replication: peer=server-90affe23-3e1b-d826-2b40-2ee158107580 2025-09-14T21:47:21.175Z [INFO] server-50f76965-ad70-ce53-9948-760cd726d161: pipelining replication: peer="{Voter server-90affe23-3e1b-d826-2b40-2ee158107580 90affe23-3e1b-d826-2b40-2ee158107580}" 2025-09-14T21:47:21.175Z [INFO] server-50f76965-ad70-ce53-9948-760cd726d161: pipelining replication: peer="{Voter server-bfa9268d-e6cd-dd8f-ec86-cb0226aca609 bfa9268d-e6cd-dd8f-ec86-cb0226aca609}" 2025-09-14T21:47:21.281Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:21.175364024 +0000 UTC m=+11.743242300 (1 nodes), 55.726927ms from start of poll, 56.092769ms from cluster start. Timeout at 2025-09-14 21:47:21.280370237 +0000 UTC m=+11.848248499, 105.006199ms after stability 2025-09-14T21:47:21.281Z [INFO] server-50f76965-ad70-ce53-9948-760cd726d161: updating configuration: command=RemoveServer server-id=797e5710-be0e-a803-38d8-3240ae35cae2 server-addr= servers="[{Suffrage:Voter ID:server-bfa9268d-e6cd-dd8f-ec86-cb0226aca609 Address:bfa9268d-e6cd-dd8f-ec86-cb0226aca609} {Suffrage:Voter ID:server-50f76965-ad70-ce53-9948-760cd726d161 Address:50f76965-ad70-ce53-9948-760cd726d161} {Suffrage:Voter ID:server-90affe23-3e1b-d826-2b40-2ee158107580 Address:90affe23-3e1b-d826-2b40-2ee158107580}]" 2025-09-14T21:47:21.281Z [INFO] server-50f76965-ad70-ce53-9948-760cd726d161: aborting pipeline replication: peer="{Voter server-90affe23-3e1b-d826-2b40-2ee158107580 90affe23-3e1b-d826-2b40-2ee158107580}" 2025-09-14T21:47:21.281Z [INFO] server-50f76965-ad70-ce53-9948-760cd726d161: aborting pipeline replication: peer="{Voter server-bfa9268d-e6cd-dd8f-ec86-cb0226aca609 bfa9268d-e6cd-dd8f-ec86-cb0226aca609}" --- PASS: TestRaft_RemoveUnknownPeer (0.16s) === RUN TestRaft_SnapshotRestore 2025-09-14T21:47:21.282Z [INFO] server-04bc18a1-d832-a513-49c4-6883344748f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04bc18a1-d832-a513-49c4-6883344748f1 Address:04bc18a1-d832-a513-49c4-6883344748f1}]" 2025-09-14T21:47:21.282Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:21.282Z [INFO] server-04bc18a1-d832-a513-49c4-6883344748f1: entering follower state: follower="Node at 04bc18a1-d832-a513-49c4-6883344748f1 [Follower]" leader-address= leader-id= 2025-09-14T21:47:21.353Z [WARN] server-04bc18a1-d832-a513-49c4-6883344748f1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:21.353Z [INFO] server-04bc18a1-d832-a513-49c4-6883344748f1: entering candidate state: node="Node at 04bc18a1-d832-a513-49c4-6883344748f1 [Candidate]" term=2 2025-09-14T21:47:21.353Z [INFO] server-04bc18a1-d832-a513-49c4-6883344748f1: election won: term=2 tally=1 2025-09-14T21:47:21.353Z [INFO] server-04bc18a1-d832-a513-49c4-6883344748f1: entering leader state: leader="Node at 04bc18a1-d832-a513-49c4-6883344748f1 [Leader]" 2025-09-14T21:47:21.459Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:21.353868573 +0000 UTC m=+11.921746848 (1 nodes), 71.720192ms from start of poll, 71.832287ms from cluster start. Timeout at 2025-09-14 21:47:21.458871103 +0000 UTC m=+12.026749365, 105.002517ms after stability 2025-09-14T21:47:21.460Z [INFO] server-04bc18a1-d832-a513-49c4-6883344748f1: starting snapshot up to: index=102 2025-09-14T21:47:21.460Z [INFO] creating new snapshot: path=/tmp/raft940930400/snapshots/2-102-1757886441460.tmp 2025-09-14T21:47:21.507Z [INFO] server-04bc18a1-d832-a513-49c4-6883344748f1: compacting logs: from=1 to=92 2025-09-14T21:47:21.507Z [INFO] server-04bc18a1-d832-a513-49c4-6883344748f1: snapshot complete up to: index=102 2025-09-14T21:47:21.507Z [INFO] server-04bc18a1-d832-a513-49c4-6883344748f1: starting restore from snapshot: id=2-102-1757886441460 last-index=102 last-term=2 size-in-bytes=693 2025-09-14T21:47:21.507Z [INFO] server-04bc18a1-d832-a513-49c4-6883344748f1: snapshot restore progress: id=2-102-1757886441460 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2025-09-14T21:47:21.507Z [INFO] server-04bc18a1-d832-a513-49c4-6883344748f1: restored from snapshot: id=2-102-1757886441460 last-index=102 last-term=2 size-in-bytes=693 2025-09-14T21:47:21.507Z [INFO] server-04bc18a1-d832-a513-49c4-6883344748f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04bc18a1-d832-a513-49c4-6883344748f1 Address:04bc18a1-d832-a513-49c4-6883344748f1}]" --- PASS: TestRaft_SnapshotRestore (0.23s) === RUN TestRaft_RestoreSnapshotOnStartup_Monotonic 2025-09-14T21:47:21.516Z [INFO] server-ee02570a-6c25-3531-2c07-506d43654330: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee02570a-6c25-3531-2c07-506d43654330 Address:ee02570a-6c25-3531-2c07-506d43654330}]" 2025-09-14T21:47:21.516Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:21.516Z [INFO] server-ee02570a-6c25-3531-2c07-506d43654330: entering follower state: follower="Node at ee02570a-6c25-3531-2c07-506d43654330 [Follower]" leader-address= leader-id= 2025-09-14T21:47:21.584Z [WARN] server-ee02570a-6c25-3531-2c07-506d43654330: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:21.584Z [INFO] server-ee02570a-6c25-3531-2c07-506d43654330: entering candidate state: node="Node at ee02570a-6c25-3531-2c07-506d43654330 [Candidate]" term=2 2025-09-14T21:47:21.584Z [INFO] server-ee02570a-6c25-3531-2c07-506d43654330: election won: term=2 tally=1 2025-09-14T21:47:21.584Z [INFO] server-ee02570a-6c25-3531-2c07-506d43654330: entering leader state: leader="Node at ee02570a-6c25-3531-2c07-506d43654330 [Leader]" 2025-09-14T21:47:21.690Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:21.584258989 +0000 UTC m=+12.152137248 (1 nodes), 67.46037ms from start of poll, 67.637971ms from cluster start. Timeout at 2025-09-14 21:47:21.68926095 +0000 UTC m=+12.257139212, 105.001964ms after stability 2025-09-14T21:47:21.692Z [INFO] server-ee02570a-6c25-3531-2c07-506d43654330: starting snapshot up to: index=102 2025-09-14T21:47:21.692Z [INFO] creating new snapshot: path=/tmp/raft4174084017/snapshots/2-102-1757886441692.tmp 2025-09-14T21:47:21.725Z [INFO] server-ee02570a-6c25-3531-2c07-506d43654330: compacting logs: from=1 to=92 2025-09-14T21:47:21.725Z [INFO] server-ee02570a-6c25-3531-2c07-506d43654330: snapshot complete up to: index=102 2025-09-14T21:47:21.725Z [INFO] server-ee02570a-6c25-3531-2c07-506d43654330: starting restore from snapshot: id=2-102-1757886441692 last-index=102 last-term=2 size-in-bytes=693 2025-09-14T21:47:21.726Z [INFO] server-ee02570a-6c25-3531-2c07-506d43654330: snapshot restore progress: id=2-102-1757886441692 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2025-09-14T21:47:21.726Z [INFO] server-ee02570a-6c25-3531-2c07-506d43654330: restored from snapshot: id=2-102-1757886441692 last-index=102 last-term=2 size-in-bytes=693 2025-09-14T21:47:21.726Z [INFO] server-ee02570a-6c25-3531-2c07-506d43654330: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee02570a-6c25-3531-2c07-506d43654330 Address:ee02570a-6c25-3531-2c07-506d43654330}]" --- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.22s) === RUN TestRaft_SnapshotRestore_Progress 2025-09-14T21:47:21.737Z [INFO] server-d39f7f7b-8b19-075a-38e0-5bd36f153d51: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d39f7f7b-8b19-075a-38e0-5bd36f153d51 Address:d39f7f7b-8b19-075a-38e0-5bd36f153d51}]" 2025-09-14T21:47:21.737Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:21.737Z [INFO] server-d39f7f7b-8b19-075a-38e0-5bd36f153d51: entering follower state: follower="Node at d39f7f7b-8b19-075a-38e0-5bd36f153d51 [Follower]" leader-address= leader-id= 2025-09-14T21:47:21.810Z [WARN] server-d39f7f7b-8b19-075a-38e0-5bd36f153d51: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:21.810Z [INFO] server-d39f7f7b-8b19-075a-38e0-5bd36f153d51: entering candidate state: node="Node at d39f7f7b-8b19-075a-38e0-5bd36f153d51 [Candidate]" term=2 2025-09-14T21:47:21.810Z [INFO] server-d39f7f7b-8b19-075a-38e0-5bd36f153d51: election won: term=2 tally=1 2025-09-14T21:47:21.810Z [INFO] server-d39f7f7b-8b19-075a-38e0-5bd36f153d51: entering leader state: leader="Node at d39f7f7b-8b19-075a-38e0-5bd36f153d51 [Leader]" 2025-09-14T21:47:21.915Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:21.810770607 +0000 UTC m=+12.378648891 (1 nodes), 73.216542ms from start of poll, 73.328238ms from cluster start. Timeout at 2025-09-14 21:47:21.915776267 +0000 UTC m=+12.483654531, 105.00564ms after stability 2025-09-14T21:47:21.917Z [INFO] server-d39f7f7b-8b19-075a-38e0-5bd36f153d51: starting snapshot up to: index=102 2025-09-14T21:47:21.917Z [INFO] creating new snapshot: path=/tmp/raft3116470536/snapshots/2-102-1757886441917.tmp 2025-09-14T21:47:21.978Z [INFO] server-d39f7f7b-8b19-075a-38e0-5bd36f153d51: compacting logs: from=1 to=92 2025-09-14T21:47:21.978Z [INFO] server-d39f7f7b-8b19-075a-38e0-5bd36f153d51: snapshot complete up to: index=102 --- PASS: TestRaft_SnapshotRestore_Progress (0.28s) === RUN TestRaft_NoRestoreOnStart 2025-09-14T21:47:22.014Z [INFO] server-42b2f213-93ea-0fb6-5b17-00e630674b70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42b2f213-93ea-0fb6-5b17-00e630674b70 Address:42b2f213-93ea-0fb6-5b17-00e630674b70}]" 2025-09-14T21:47:22.014Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:22.014Z [INFO] server-42b2f213-93ea-0fb6-5b17-00e630674b70: entering follower state: follower="Node at 42b2f213-93ea-0fb6-5b17-00e630674b70 [Follower]" leader-address= leader-id= 2025-09-14T21:47:22.075Z [WARN] server-42b2f213-93ea-0fb6-5b17-00e630674b70: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:22.075Z [INFO] server-42b2f213-93ea-0fb6-5b17-00e630674b70: entering candidate state: node="Node at 42b2f213-93ea-0fb6-5b17-00e630674b70 [Candidate]" term=2 2025-09-14T21:47:22.075Z [INFO] server-42b2f213-93ea-0fb6-5b17-00e630674b70: election won: term=2 tally=1 2025-09-14T21:47:22.075Z [INFO] server-42b2f213-93ea-0fb6-5b17-00e630674b70: entering leader state: leader="Node at 42b2f213-93ea-0fb6-5b17-00e630674b70 [Leader]" 2025-09-14T21:47:22.180Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:22.075464639 +0000 UTC m=+12.643342985 (1 nodes), 60.533102ms from start of poll, 60.750264ms from cluster start. Timeout at 2025-09-14 21:47:22.180467786 +0000 UTC m=+12.748346050, 105.003065ms after stability 2025-09-14T21:47:22.182Z [INFO] server-42b2f213-93ea-0fb6-5b17-00e630674b70: starting snapshot up to: index=102 2025-09-14T21:47:22.182Z [INFO] creating new snapshot: path=/tmp/raft1213642719/snapshots/2-102-1757886442182.tmp 2025-09-14T21:47:22.212Z [INFO] server-42b2f213-93ea-0fb6-5b17-00e630674b70: compacting logs: from=1 to=92 2025-09-14T21:47:22.212Z [INFO] server-42b2f213-93ea-0fb6-5b17-00e630674b70: snapshot complete up to: index=102 2025-09-14T21:47:22.212Z [INFO] server-42b2f213-93ea-0fb6-5b17-00e630674b70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42b2f213-93ea-0fb6-5b17-00e630674b70 Address:42b2f213-93ea-0fb6-5b17-00e630674b70}]" --- PASS: TestRaft_NoRestoreOnStart (0.20s) === RUN TestRaft_SnapshotRestore_PeerChange 2025-09-14T21:47:22.212Z [INFO] server-42b2f213-93ea-0fb6-5b17-00e630674b70: entering follower state: follower="Node at 42b2f213-93ea-0fb6-5b17-00e630674b70 [Follower]" leader-address= leader-id= raft_test.go:1266: Skipping unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot 2025-09-14T21:47:22.212Z [INFO] server-a7a1d667-f241-8de5-7ea3-06af293c3ea6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a7a1d667-f241-8de5-7ea3-06af293c3ea6 Address:a7a1d667-f241-8de5-7ea3-06af293c3ea6}]" 2025-09-14T21:47:22.213Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:22.213Z [INFO] server-a7a1d667-f241-8de5-7ea3-06af293c3ea6: entering follower state: follower="Node at a7a1d667-f241-8de5-7ea3-06af293c3ea6 [Follower]" leader-address= leader-id= 2025-09-14T21:47:22.270Z [WARN] server-42b2f213-93ea-0fb6-5b17-00e630674b70: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:22.270Z [INFO] server-42b2f213-93ea-0fb6-5b17-00e630674b70: entering candidate state: node="Node at 42b2f213-93ea-0fb6-5b17-00e630674b70 [Candidate]" term=3 2025-09-14T21:47:22.270Z [INFO] server-42b2f213-93ea-0fb6-5b17-00e630674b70: election won: term=3 tally=1 2025-09-14T21:47:22.270Z [INFO] server-42b2f213-93ea-0fb6-5b17-00e630674b70: entering leader state: leader="Node at 42b2f213-93ea-0fb6-5b17-00e630674b70 [Leader]" 2025-09-14T21:47:22.270Z [WARN] server-a7a1d667-f241-8de5-7ea3-06af293c3ea6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:22.270Z [INFO] server-a7a1d667-f241-8de5-7ea3-06af293c3ea6: entering candidate state: node="Node at a7a1d667-f241-8de5-7ea3-06af293c3ea6 [Candidate]" term=2 2025-09-14T21:47:22.270Z [INFO] server-a7a1d667-f241-8de5-7ea3-06af293c3ea6: election won: term=2 tally=1 2025-09-14T21:47:22.270Z [INFO] server-a7a1d667-f241-8de5-7ea3-06af293c3ea6: entering leader state: leader="Node at a7a1d667-f241-8de5-7ea3-06af293c3ea6 [Leader]" 2025-09-14T21:47:22.376Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:22.270977128 +0000 UTC m=+12.838855409 (1 nodes), 57.955209ms from start of poll, 58.122216ms from cluster start. Timeout at 2025-09-14 21:47:22.375980359 +0000 UTC m=+12.943858636, 105.003227ms after stability 2025-09-14T21:47:22.385Z [INFO] server-a7a1d667-f241-8de5-7ea3-06af293c3ea6: starting snapshot up to: index=102 2025-09-14T21:47:22.385Z [INFO] creating new snapshot: path=/tmp/raft400913938/snapshots/2-102-1757886442385.tmp 2025-09-14T21:47:22.394Z [INFO] server-a7a1d667-f241-8de5-7ea3-06af293c3ea6: compacting logs: from=1 to=92 2025-09-14T21:47:22.395Z [INFO] server-a7a1d667-f241-8de5-7ea3-06af293c3ea6: snapshot complete up to: index=102 --- PASS: TestRaft_AutoSnapshot (0.27s) === RUN TestRaft_UserSnapshot 2025-09-14T21:47:22.486Z [INFO] server-3b17a901-60e9-fd17-ecec-47d97ecd3435: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b17a901-60e9-fd17-ecec-47d97ecd3435 Address:3b17a901-60e9-fd17-ecec-47d97ecd3435}]" 2025-09-14T21:47:22.486Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:22.486Z [INFO] server-3b17a901-60e9-fd17-ecec-47d97ecd3435: entering follower state: follower="Node at 3b17a901-60e9-fd17-ecec-47d97ecd3435 [Follower]" leader-address= leader-id= 2025-09-14T21:47:22.548Z [WARN] server-3b17a901-60e9-fd17-ecec-47d97ecd3435: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:22.548Z [INFO] server-3b17a901-60e9-fd17-ecec-47d97ecd3435: entering candidate state: node="Node at 3b17a901-60e9-fd17-ecec-47d97ecd3435 [Candidate]" term=2 2025-09-14T21:47:22.548Z [INFO] server-3b17a901-60e9-fd17-ecec-47d97ecd3435: election won: term=2 tally=1 2025-09-14T21:47:22.548Z [INFO] server-3b17a901-60e9-fd17-ecec-47d97ecd3435: entering leader state: leader="Node at 3b17a901-60e9-fd17-ecec-47d97ecd3435 [Leader]" 2025-09-14T21:47:22.653Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:22.548496609 +0000 UTC m=+13.116374868 (1 nodes), 61.736524ms from start of poll, 62.727158ms from cluster start. Timeout at 2025-09-14 21:47:22.653500248 +0000 UTC m=+13.221378510, 105.003642ms after stability 2025-09-14T21:47:22.653Z [ERROR] server-3b17a901-60e9-fd17-ecec-47d97ecd3435: failed to take snapshot: error="nothing new to snapshot" 2025-09-14T21:47:22.654Z [INFO] server-3b17a901-60e9-fd17-ecec-47d97ecd3435: starting snapshot up to: index=12 2025-09-14T21:47:22.654Z [INFO] creating new snapshot: path=/tmp/raft1731270072/snapshots/2-12-1757886442654.tmp 2025-09-14T21:47:22.663Z [INFO] server-3b17a901-60e9-fd17-ecec-47d97ecd3435: compacting logs: from=1 to=2 2025-09-14T21:47:22.663Z [INFO] server-3b17a901-60e9-fd17-ecec-47d97ecd3435: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.22s) === RUN TestRaft_UserRestore === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false 2025-09-14T21:47:22.705Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dce6c1c-5c14-3e47-31dc-e1d92387d452 Address:2dce6c1c-5c14-3e47-31dc-e1d92387d452} {Suffrage:Voter ID:server-13d61be1-c7be-c458-d6df-014714c19818 Address:13d61be1-c7be-c458-d6df-014714c19818} {Suffrage:Voter ID:server-67c77815-89de-2b0a-37d9-f4c39b100a79 Address:67c77815-89de-2b0a-37d9-f4c39b100a79}]" 2025-09-14T21:47:22.705Z [INFO] server-13d61be1-c7be-c458-d6df-014714c19818: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dce6c1c-5c14-3e47-31dc-e1d92387d452 Address:2dce6c1c-5c14-3e47-31dc-e1d92387d452} {Suffrage:Voter ID:server-13d61be1-c7be-c458-d6df-014714c19818 Address:13d61be1-c7be-c458-d6df-014714c19818} {Suffrage:Voter ID:server-67c77815-89de-2b0a-37d9-f4c39b100a79 Address:67c77815-89de-2b0a-37d9-f4c39b100a79}]" 2025-09-14T21:47:22.705Z [INFO] server-13d61be1-c7be-c458-d6df-014714c19818: entering follower state: follower="Node at 13d61be1-c7be-c458-d6df-014714c19818 [Follower]" leader-address= leader-id= 2025-09-14T21:47:22.705Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: entering follower state: follower="Node at 2dce6c1c-5c14-3e47-31dc-e1d92387d452 [Follower]" leader-address= leader-id= 2025-09-14T21:47:22.706Z [INFO] server-67c77815-89de-2b0a-37d9-f4c39b100a79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dce6c1c-5c14-3e47-31dc-e1d92387d452 Address:2dce6c1c-5c14-3e47-31dc-e1d92387d452} {Suffrage:Voter ID:server-13d61be1-c7be-c458-d6df-014714c19818 Address:13d61be1-c7be-c458-d6df-014714c19818} {Suffrage:Voter ID:server-67c77815-89de-2b0a-37d9-f4c39b100a79 Address:67c77815-89de-2b0a-37d9-f4c39b100a79}]" 2025-09-14T21:47:22.706Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:22.706Z [INFO] server-67c77815-89de-2b0a-37d9-f4c39b100a79: entering follower state: follower="Node at 67c77815-89de-2b0a-37d9-f4c39b100a79 [Follower]" leader-address= leader-id= 2025-09-14T21:47:23.216Z [WARN] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:23.216Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: entering candidate state: node="Node at 2dce6c1c-5c14-3e47-31dc-e1d92387d452 [Candidate]" term=2 2025-09-14T21:47:23.216Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: election won: term=2 tally=2 2025-09-14T21:47:23.216Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: entering leader state: leader="Node at 2dce6c1c-5c14-3e47-31dc-e1d92387d452 [Leader]" 2025-09-14T21:47:23.216Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: added peer, starting replication: peer=server-13d61be1-c7be-c458-d6df-014714c19818 2025-09-14T21:47:23.216Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: added peer, starting replication: peer=server-67c77815-89de-2b0a-37d9-f4c39b100a79 2025-09-14T21:47:23.216Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: pipelining replication: peer="{Voter server-67c77815-89de-2b0a-37d9-f4c39b100a79 67c77815-89de-2b0a-37d9-f4c39b100a79}" 2025-09-14T21:47:23.216Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: pipelining replication: peer="{Voter server-13d61be1-c7be-c458-d6df-014714c19818 13d61be1-c7be-c458-d6df-014714c19818}" 2025-09-14T21:47:24.222Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:23.216744927 +0000 UTC m=+13.784623208 (1 nodes), 510.615587ms from start of poll, 511.074904ms from cluster start. Timeout at 2025-09-14 21:47:24.221746275 +0000 UTC m=+14.789624537, 1.005001329s after stability 2025-09-14T21:47:24.223Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: starting snapshot up to: index=12 2025-09-14T21:47:24.223Z [INFO] creating new snapshot: path=/tmp/raft3099076064/snapshots/2-12-1757886444223.tmp 2025-09-14T21:47:24.340Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: snapshot complete up to: index=12 2025-09-14T21:47:24.340Z [INFO] creating new snapshot: path=/tmp/raft3099076064/snapshots/2-23-1757886444340.tmp 2025-09-14T21:47:24.363Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: copied to local snapshot: bytes=71 2025-09-14T21:47:24.363Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: snapshot restore progress: id=2-23-1757886444340 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:24.363Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: restored user snapshot: index=23 2025-09-14T21:47:24.363Z [ERROR] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: failed to get log: index=23 error="log not found" 2025-09-14T21:47:24.363Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: aborting pipeline replication: peer="{Voter server-13d61be1-c7be-c458-d6df-014714c19818 13d61be1-c7be-c458-d6df-014714c19818}" 2025-09-14T21:47:24.363Z [ERROR] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: failed to get log: index=23 error="log not found" 2025-09-14T21:47:24.363Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: aborting pipeline replication: peer="{Voter server-67c77815-89de-2b0a-37d9-f4c39b100a79 67c77815-89de-2b0a-37d9-f4c39b100a79}" 2025-09-14T21:47:24.369Z [ERROR] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: failed to get log: index=23 error="log not found" 2025-09-14T21:47:24.369Z [INFO] creating new snapshot: path=/tmp/raft3326188979/snapshots/2-23-1757886444369.tmp 2025-09-14T21:47:24.370Z [INFO] server-13d61be1-c7be-c458-d6df-014714c19818: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:24.373Z [ERROR] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: failed to get log: index=23 error="log not found" 2025-09-14T21:47:24.374Z [INFO] creating new snapshot: path=/tmp/raft3453568729/snapshots/2-23-1757886444374.tmp 2025-09-14T21:47:24.374Z [INFO] server-67c77815-89de-2b0a-37d9-f4c39b100a79: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:24.385Z [INFO] server-67c77815-89de-2b0a-37d9-f4c39b100a79: copied to local snapshot: bytes=71 2025-09-14T21:47:24.385Z [INFO] server-13d61be1-c7be-c458-d6df-014714c19818: copied to local snapshot: bytes=71 2025-09-14T21:47:24.385Z [INFO] server-13d61be1-c7be-c458-d6df-014714c19818: snapshot restore progress: id=2-23-1757886444369 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:24.385Z [INFO] server-13d61be1-c7be-c458-d6df-014714c19818: Installed remote snapshot 2025-09-14T21:47:24.385Z [INFO] server-67c77815-89de-2b0a-37d9-f4c39b100a79: snapshot restore progress: id=2-23-1757886444374 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:24.385Z [INFO] server-67c77815-89de-2b0a-37d9-f4c39b100a79: Installed remote snapshot 2025-09-14T21:47:24.385Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: pipelining replication: peer="{Voter server-13d61be1-c7be-c458-d6df-014714c19818 13d61be1-c7be-c458-d6df-014714c19818}" 2025-09-14T21:47:24.385Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:24.385Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: pipelining replication: peer="{Voter server-67c77815-89de-2b0a-37d9-f4c39b100a79 67c77815-89de-2b0a-37d9-f4c39b100a79}" 2025-09-14T21:47:25.390Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:24.385719364 +0000 UTC m=+14.953597627 (2 nodes), 569ns from start of poll, 1.680049323s from cluster start. Timeout at 2025-09-14 21:47:25.390724307 +0000 UTC m=+15.958602587, 1.00500496s after stability 2025-09-14T21:47:25.407Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: aborting pipeline replication: peer="{Voter server-67c77815-89de-2b0a-37d9-f4c39b100a79 67c77815-89de-2b0a-37d9-f4c39b100a79}" 2025-09-14T21:47:25.407Z [INFO] server-2dce6c1c-5c14-3e47-31dc-e1d92387d452: aborting pipeline replication: peer="{Voter server-13d61be1-c7be-c458-d6df-014714c19818 13d61be1-c7be-c458-d6df-014714c19818}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false 2025-09-14T21:47:25.409Z [INFO] server-c7e902c0-b800-afe0-fcfc-178de6193623: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7e902c0-b800-afe0-fcfc-178de6193623 Address:c7e902c0-b800-afe0-fcfc-178de6193623} {Suffrage:Voter ID:server-c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d Address:c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d} {Suffrage:Voter ID:server-5e9bb107-d5ca-1a30-add0-165a429011e6 Address:5e9bb107-d5ca-1a30-add0-165a429011e6}]" 2025-09-14T21:47:25.409Z [INFO] server-c7e902c0-b800-afe0-fcfc-178de6193623: entering follower state: follower="Node at c7e902c0-b800-afe0-fcfc-178de6193623 [Follower]" leader-address= leader-id= 2025-09-14T21:47:25.409Z [INFO] server-c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7e902c0-b800-afe0-fcfc-178de6193623 Address:c7e902c0-b800-afe0-fcfc-178de6193623} {Suffrage:Voter ID:server-c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d Address:c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d} {Suffrage:Voter ID:server-5e9bb107-d5ca-1a30-add0-165a429011e6 Address:5e9bb107-d5ca-1a30-add0-165a429011e6}]" 2025-09-14T21:47:25.409Z [INFO] server-c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d: entering follower state: follower="Node at c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d [Follower]" leader-address= leader-id= 2025-09-14T21:47:25.410Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7e902c0-b800-afe0-fcfc-178de6193623 Address:c7e902c0-b800-afe0-fcfc-178de6193623} {Suffrage:Voter ID:server-c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d Address:c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d} {Suffrage:Voter ID:server-5e9bb107-d5ca-1a30-add0-165a429011e6 Address:5e9bb107-d5ca-1a30-add0-165a429011e6}]" 2025-09-14T21:47:25.410Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:25.410Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: entering follower state: follower="Node at 5e9bb107-d5ca-1a30-add0-165a429011e6 [Follower]" leader-address= leader-id= 2025-09-14T21:47:26.093Z [WARN] server-5e9bb107-d5ca-1a30-add0-165a429011e6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:26.093Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: entering candidate state: node="Node at 5e9bb107-d5ca-1a30-add0-165a429011e6 [Candidate]" term=2 2025-09-14T21:47:26.093Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: election won: term=2 tally=2 2025-09-14T21:47:26.093Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: entering leader state: leader="Node at 5e9bb107-d5ca-1a30-add0-165a429011e6 [Leader]" 2025-09-14T21:47:26.093Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: added peer, starting replication: peer=server-c7e902c0-b800-afe0-fcfc-178de6193623 2025-09-14T21:47:26.093Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: added peer, starting replication: peer=server-c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d 2025-09-14T21:47:26.093Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: pipelining replication: peer="{Voter server-c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d}" 2025-09-14T21:47:26.093Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: pipelining replication: peer="{Voter server-c7e902c0-b800-afe0-fcfc-178de6193623 c7e902c0-b800-afe0-fcfc-178de6193623}" 2025-09-14T21:47:27.099Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:26.093482474 +0000 UTC m=+16.661360738 (1 nodes), 683.36954ms from start of poll, 683.814541ms from cluster start. Timeout at 2025-09-14 21:47:27.098488176 +0000 UTC m=+17.666366435, 1.005005697s after stability 2025-09-14T21:47:27.100Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: starting snapshot up to: index=12 2025-09-14T21:47:27.100Z [INFO] creating new snapshot: path=/tmp/raft1709795829/snapshots/2-12-1757886447100.tmp 2025-09-14T21:47:27.101Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: snapshot complete up to: index=12 2025-09-14T21:47:27.101Z [INFO] creating new snapshot: path=/tmp/raft1709795829/snapshots/2-23-1757886447101.tmp 2025-09-14T21:47:27.102Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: copied to local snapshot: bytes=71 2025-09-14T21:47:27.102Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: snapshot restore progress: id=2-23-1757886447101 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:27.102Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: removing all old logs from log store 2025-09-14T21:47:27.102Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: compacting logs: from=1 to=22 2025-09-14T21:47:27.102Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: restored user snapshot: index=23 2025-09-14T21:47:27.102Z [ERROR] server-5e9bb107-d5ca-1a30-add0-165a429011e6: failed to get log: index=22 error="log not found" 2025-09-14T21:47:27.102Z [ERROR] server-5e9bb107-d5ca-1a30-add0-165a429011e6: failed to get log: index=22 error="log not found" 2025-09-14T21:47:27.102Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: aborting pipeline replication: peer="{Voter server-c7e902c0-b800-afe0-fcfc-178de6193623 c7e902c0-b800-afe0-fcfc-178de6193623}" 2025-09-14T21:47:27.102Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: aborting pipeline replication: peer="{Voter server-c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d}" 2025-09-14T21:47:27.109Z [ERROR] server-5e9bb107-d5ca-1a30-add0-165a429011e6: failed to get log: index=22 error="log not found" 2025-09-14T21:47:27.109Z [INFO] creating new snapshot: path=/tmp/raft1192198378/snapshots/2-23-1757886447109.tmp 2025-09-14T21:47:27.109Z [INFO] server-c7e902c0-b800-afe0-fcfc-178de6193623: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:27.110Z [INFO] server-c7e902c0-b800-afe0-fcfc-178de6193623: copied to local snapshot: bytes=71 2025-09-14T21:47:27.110Z [INFO] server-c7e902c0-b800-afe0-fcfc-178de6193623: snapshot restore progress: id=2-23-1757886447109 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:27.110Z [INFO] server-c7e902c0-b800-afe0-fcfc-178de6193623: removing all old logs from log store 2025-09-14T21:47:27.110Z [INFO] server-c7e902c0-b800-afe0-fcfc-178de6193623: compacting logs: from=1 to=22 2025-09-14T21:47:27.110Z [INFO] server-c7e902c0-b800-afe0-fcfc-178de6193623: Installed remote snapshot 2025-09-14T21:47:27.110Z [WARN] server-c7e902c0-b800-afe0-fcfc-178de6193623: skipping application of old log: index=22 2025-09-14T21:47:27.110Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: pipelining replication: peer="{Voter server-c7e902c0-b800-afe0-fcfc-178de6193623 c7e902c0-b800-afe0-fcfc-178de6193623}" 2025-09-14T21:47:27.110Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:27.111Z [ERROR] server-5e9bb107-d5ca-1a30-add0-165a429011e6: failed to get log: index=22 error="log not found" 2025-09-14T21:47:27.111Z [INFO] creating new snapshot: path=/tmp/raft4061918673/snapshots/2-23-1757886447111.tmp 2025-09-14T21:47:27.111Z [INFO] server-c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:27.112Z [INFO] server-c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d: copied to local snapshot: bytes=71 2025-09-14T21:47:27.112Z [INFO] server-c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d: snapshot restore progress: id=2-23-1757886447111 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:27.112Z [INFO] server-c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d: removing all old logs from log store 2025-09-14T21:47:27.112Z [INFO] server-c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d: compacting logs: from=1 to=22 2025-09-14T21:47:27.112Z [INFO] server-c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d: Installed remote snapshot 2025-09-14T21:47:27.112Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: pipelining replication: peer="{Voter server-c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d}" 2025-09-14T21:47:28.116Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:27.110476466 +0000 UTC m=+17.678354728 (2 nodes), 562ns from start of poll, 1.700808531s from cluster start. Timeout at 2025-09-14 21:47:28.115482285 +0000 UTC m=+18.683360562, 1.005005834s after stability 2025-09-14T21:47:28.128Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: aborting pipeline replication: peer="{Voter server-c7e902c0-b800-afe0-fcfc-178de6193623 c7e902c0-b800-afe0-fcfc-178de6193623}" 2025-09-14T21:47:28.128Z [INFO] server-5e9bb107-d5ca-1a30-add0-165a429011e6: aborting pipeline replication: peer="{Voter server-c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d c9109b82-2ec9-96b2-f35e-a20b3d9ffe9d}" === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true 2025-09-14T21:47:28.130Z [INFO] server-c98b0678-abce-352a-584f-bbd1537f5867: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c98b0678-abce-352a-584f-bbd1537f5867 Address:c98b0678-abce-352a-584f-bbd1537f5867} {Suffrage:Voter ID:server-ea5b127d-0f04-f482-07ea-5e81b484dcf4 Address:ea5b127d-0f04-f482-07ea-5e81b484dcf4} {Suffrage:Voter ID:server-975440a1-869c-5311-8030-8066df86d075 Address:975440a1-869c-5311-8030-8066df86d075}]" 2025-09-14T21:47:28.130Z [INFO] server-c98b0678-abce-352a-584f-bbd1537f5867: entering follower state: follower="Node at c98b0678-abce-352a-584f-bbd1537f5867 [Follower]" leader-address= leader-id= 2025-09-14T21:47:28.130Z [INFO] server-ea5b127d-0f04-f482-07ea-5e81b484dcf4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c98b0678-abce-352a-584f-bbd1537f5867 Address:c98b0678-abce-352a-584f-bbd1537f5867} {Suffrage:Voter ID:server-ea5b127d-0f04-f482-07ea-5e81b484dcf4 Address:ea5b127d-0f04-f482-07ea-5e81b484dcf4} {Suffrage:Voter ID:server-975440a1-869c-5311-8030-8066df86d075 Address:975440a1-869c-5311-8030-8066df86d075}]" 2025-09-14T21:47:28.130Z [INFO] server-ea5b127d-0f04-f482-07ea-5e81b484dcf4: entering follower state: follower="Node at ea5b127d-0f04-f482-07ea-5e81b484dcf4 [Follower]" leader-address= leader-id= 2025-09-14T21:47:28.130Z [INFO] server-975440a1-869c-5311-8030-8066df86d075: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c98b0678-abce-352a-584f-bbd1537f5867 Address:c98b0678-abce-352a-584f-bbd1537f5867} {Suffrage:Voter ID:server-ea5b127d-0f04-f482-07ea-5e81b484dcf4 Address:ea5b127d-0f04-f482-07ea-5e81b484dcf4} {Suffrage:Voter ID:server-975440a1-869c-5311-8030-8066df86d075 Address:975440a1-869c-5311-8030-8066df86d075}]" 2025-09-14T21:47:28.130Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:28.130Z [INFO] server-975440a1-869c-5311-8030-8066df86d075: entering follower state: follower="Node at 975440a1-869c-5311-8030-8066df86d075 [Follower]" leader-address= leader-id= 2025-09-14T21:47:28.820Z [WARN] server-c98b0678-abce-352a-584f-bbd1537f5867: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:28.820Z [INFO] server-c98b0678-abce-352a-584f-bbd1537f5867: entering candidate state: node="Node at c98b0678-abce-352a-584f-bbd1537f5867 [Candidate]" term=2 2025-09-14T21:47:28.820Z [INFO] server-c98b0678-abce-352a-584f-bbd1537f5867: election won: term=2 tally=2 2025-09-14T21:47:28.820Z [INFO] server-c98b0678-abce-352a-584f-bbd1537f5867: entering leader state: leader="Node at c98b0678-abce-352a-584f-bbd1537f5867 [Leader]" 2025-09-14T21:47:28.820Z [INFO] server-c98b0678-abce-352a-584f-bbd1537f5867: added peer, starting replication: peer=server-ea5b127d-0f04-f482-07ea-5e81b484dcf4 2025-09-14T21:47:28.820Z [INFO] server-c98b0678-abce-352a-584f-bbd1537f5867: added peer, starting replication: peer=server-975440a1-869c-5311-8030-8066df86d075 2025-09-14T21:47:28.820Z [INFO] server-c98b0678-abce-352a-584f-bbd1537f5867: pipelining replication: peer="{Voter server-975440a1-869c-5311-8030-8066df86d075 975440a1-869c-5311-8030-8066df86d075}" 2025-09-14T21:47:28.820Z [INFO] server-c98b0678-abce-352a-584f-bbd1537f5867: pipelining replication: peer="{Voter server-ea5b127d-0f04-f482-07ea-5e81b484dcf4 ea5b127d-0f04-f482-07ea-5e81b484dcf4}" 2025-09-14T21:47:29.826Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:28.820507585 +0000 UTC m=+19.388385859 (1 nodes), 690.127389ms from start of poll, 690.459514ms from cluster start. Timeout at 2025-09-14 21:47:29.825510797 +0000 UTC m=+20.393389057, 1.005003198s after stability 2025-09-14T21:47:29.826Z [INFO] server-c98b0678-abce-352a-584f-bbd1537f5867: starting snapshot up to: index=12 2025-09-14T21:47:29.826Z [INFO] creating new snapshot: path=/tmp/raft2945392987/snapshots/2-12-1757886449826.tmp 2025-09-14T21:47:29.827Z [INFO] server-c98b0678-abce-352a-584f-bbd1537f5867: snapshot complete up to: index=12 2025-09-14T21:47:29.828Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38971478-809c-95e2-3b17-c84e9880db93 Address:38971478-809c-95e2-3b17-c84e9880db93} {Suffrage:Voter ID:server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c Address:915e7e29-8f69-5b4f-b2a7-860c5804ae3c} {Suffrage:Voter ID:server-d3a15db6-442f-1c38-0601-eab09266c438 Address:d3a15db6-442f-1c38-0601-eab09266c438}]" 2025-09-14T21:47:29.828Z [INFO] server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38971478-809c-95e2-3b17-c84e9880db93 Address:38971478-809c-95e2-3b17-c84e9880db93} {Suffrage:Voter ID:server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c Address:915e7e29-8f69-5b4f-b2a7-860c5804ae3c} {Suffrage:Voter ID:server-d3a15db6-442f-1c38-0601-eab09266c438 Address:d3a15db6-442f-1c38-0601-eab09266c438}]" 2025-09-14T21:47:29.828Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: entering follower state: follower="Node at 38971478-809c-95e2-3b17-c84e9880db93 [Follower]" leader-address= leader-id= 2025-09-14T21:47:29.829Z [INFO] server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c: entering follower state: follower="Node at 915e7e29-8f69-5b4f-b2a7-860c5804ae3c [Follower]" leader-address= leader-id= 2025-09-14T21:47:29.829Z [INFO] server-d3a15db6-442f-1c38-0601-eab09266c438: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38971478-809c-95e2-3b17-c84e9880db93 Address:38971478-809c-95e2-3b17-c84e9880db93} {Suffrage:Voter ID:server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c Address:915e7e29-8f69-5b4f-b2a7-860c5804ae3c} {Suffrage:Voter ID:server-d3a15db6-442f-1c38-0601-eab09266c438 Address:d3a15db6-442f-1c38-0601-eab09266c438}]" 2025-09-14T21:47:29.829Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:29.829Z [INFO] server-d3a15db6-442f-1c38-0601-eab09266c438: entering follower state: follower="Node at d3a15db6-442f-1c38-0601-eab09266c438 [Follower]" leader-address= leader-id= 2025-09-14T21:47:30.385Z [WARN] server-38971478-809c-95e2-3b17-c84e9880db93: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:30.385Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: entering candidate state: node="Node at 38971478-809c-95e2-3b17-c84e9880db93 [Candidate]" term=2 2025-09-14T21:47:30.385Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: election won: term=2 tally=2 2025-09-14T21:47:30.385Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: entering leader state: leader="Node at 38971478-809c-95e2-3b17-c84e9880db93 [Leader]" 2025-09-14T21:47:30.385Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: added peer, starting replication: peer=server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c 2025-09-14T21:47:30.385Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: added peer, starting replication: peer=server-d3a15db6-442f-1c38-0601-eab09266c438 2025-09-14T21:47:30.385Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: pipelining replication: peer="{Voter server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c 915e7e29-8f69-5b4f-b2a7-860c5804ae3c}" 2025-09-14T21:47:30.386Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: pipelining replication: peer="{Voter server-d3a15db6-442f-1c38-0601-eab09266c438 d3a15db6-442f-1c38-0601-eab09266c438}" 2025-09-14T21:47:31.391Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:30.385842109 +0000 UTC m=+20.953720369 (1 nodes), 555.979357ms from start of poll, 557.516319ms from cluster start. Timeout at 2025-09-14 21:47:31.390845798 +0000 UTC m=+21.958724073, 1.005003704s after stability 2025-09-14T21:47:31.391Z [INFO] creating new snapshot: path=/tmp/raft4002210013/snapshots/2-13-1757886451391.tmp 2025-09-14T21:47:31.492Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: copied to local snapshot: bytes=71 2025-09-14T21:47:31.492Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: snapshot restore progress: id=2-13-1757886451391 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:31.492Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: restored user snapshot: index=13 2025-09-14T21:47:31.492Z [ERROR] server-38971478-809c-95e2-3b17-c84e9880db93: failed to get log: index=3 error="log not found" 2025-09-14T21:47:31.492Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: aborting pipeline replication: peer="{Voter server-d3a15db6-442f-1c38-0601-eab09266c438 d3a15db6-442f-1c38-0601-eab09266c438}" 2025-09-14T21:47:31.492Z [ERROR] server-38971478-809c-95e2-3b17-c84e9880db93: failed to get log: index=3 error="log not found" 2025-09-14T21:47:31.492Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: aborting pipeline replication: peer="{Voter server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c 915e7e29-8f69-5b4f-b2a7-860c5804ae3c}" 2025-09-14T21:47:31.499Z [ERROR] server-38971478-809c-95e2-3b17-c84e9880db93: failed to get log: index=3 error="log not found" 2025-09-14T21:47:31.499Z [INFO] creating new snapshot: path=/tmp/raft53706367/snapshots/2-13-1757886451499.tmp 2025-09-14T21:47:31.500Z [ERROR] server-38971478-809c-95e2-3b17-c84e9880db93: failed to get log: index=3 error="log not found" 2025-09-14T21:47:31.500Z [INFO] server-d3a15db6-442f-1c38-0601-eab09266c438: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:31.501Z [INFO] creating new snapshot: path=/tmp/raft160044099/snapshots/2-13-1757886451501.tmp 2025-09-14T21:47:31.501Z [INFO] server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:31.526Z [INFO] server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c: copied to local snapshot: bytes=71 2025-09-14T21:47:31.526Z [INFO] server-d3a15db6-442f-1c38-0601-eab09266c438: copied to local snapshot: bytes=71 2025-09-14T21:47:31.526Z [INFO] server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c: snapshot restore progress: id=2-13-1757886451501 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:31.526Z [INFO] server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c: Installed remote snapshot 2025-09-14T21:47:31.526Z [INFO] server-d3a15db6-442f-1c38-0601-eab09266c438: snapshot restore progress: id=2-13-1757886451499 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:31.526Z [INFO] server-d3a15db6-442f-1c38-0601-eab09266c438: Installed remote snapshot 2025-09-14T21:47:31.526Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: pipelining replication: peer="{Voter server-d3a15db6-442f-1c38-0601-eab09266c438 d3a15db6-442f-1c38-0601-eab09266c438}" 2025-09-14T21:47:31.526Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: pipelining replication: peer="{Voter server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c 915e7e29-8f69-5b4f-b2a7-860c5804ae3c}" 2025-09-14T21:47:31.526Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:32.532Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:31.526870359 +0000 UTC m=+22.094748623 (2 nodes), 2.276µs from start of poll, 1.698544573s from cluster start. Timeout at 2025-09-14 21:47:32.531875623 +0000 UTC m=+23.099753885, 1.005005262s after stability 2025-09-14T21:47:32.544Z [INFO] server-c98b0678-abce-352a-584f-bbd1537f5867: aborting pipeline replication: peer="{Voter server-975440a1-869c-5311-8030-8066df86d075 975440a1-869c-5311-8030-8066df86d075}" 2025-09-14T21:47:32.544Z [INFO] server-c98b0678-abce-352a-584f-bbd1537f5867: aborting pipeline replication: peer="{Voter server-ea5b127d-0f04-f482-07ea-5e81b484dcf4 ea5b127d-0f04-f482-07ea-5e81b484dcf4}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true 2025-09-14T21:47:32.548Z [INFO] server-cce5bb44-899a-164a-d35a-13db86b2ff5b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cce5bb44-899a-164a-d35a-13db86b2ff5b Address:cce5bb44-899a-164a-d35a-13db86b2ff5b} {Suffrage:Voter ID:server-27d8bc2d-17be-3583-e676-c197c7a81722 Address:27d8bc2d-17be-3583-e676-c197c7a81722} {Suffrage:Voter ID:server-fde746ae-973e-dd7a-f129-47cfd1708c15 Address:fde746ae-973e-dd7a-f129-47cfd1708c15}]" 2025-09-14T21:47:32.780Z [INFO] server-27d8bc2d-17be-3583-e676-c197c7a81722: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cce5bb44-899a-164a-d35a-13db86b2ff5b Address:cce5bb44-899a-164a-d35a-13db86b2ff5b} {Suffrage:Voter ID:server-27d8bc2d-17be-3583-e676-c197c7a81722 Address:27d8bc2d-17be-3583-e676-c197c7a81722} {Suffrage:Voter ID:server-fde746ae-973e-dd7a-f129-47cfd1708c15 Address:fde746ae-973e-dd7a-f129-47cfd1708c15}]" 2025-09-14T21:47:32.780Z [INFO] server-cce5bb44-899a-164a-d35a-13db86b2ff5b: entering follower state: follower="Node at cce5bb44-899a-164a-d35a-13db86b2ff5b [Follower]" leader-address= leader-id= 2025-09-14T21:47:32.781Z [INFO] server-fde746ae-973e-dd7a-f129-47cfd1708c15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cce5bb44-899a-164a-d35a-13db86b2ff5b Address:cce5bb44-899a-164a-d35a-13db86b2ff5b} {Suffrage:Voter ID:server-27d8bc2d-17be-3583-e676-c197c7a81722 Address:27d8bc2d-17be-3583-e676-c197c7a81722} {Suffrage:Voter ID:server-fde746ae-973e-dd7a-f129-47cfd1708c15 Address:fde746ae-973e-dd7a-f129-47cfd1708c15}]" 2025-09-14T21:47:32.781Z [INFO] server-27d8bc2d-17be-3583-e676-c197c7a81722: entering follower state: follower="Node at 27d8bc2d-17be-3583-e676-c197c7a81722 [Follower]" leader-address= leader-id= 2025-09-14T21:47:32.781Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:32.781Z [INFO] server-fde746ae-973e-dd7a-f129-47cfd1708c15: entering follower state: follower="Node at fde746ae-973e-dd7a-f129-47cfd1708c15 [Follower]" leader-address= leader-id= 2025-09-14T21:47:33.334Z [WARN] server-fde746ae-973e-dd7a-f129-47cfd1708c15: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:33.334Z [INFO] server-fde746ae-973e-dd7a-f129-47cfd1708c15: entering candidate state: node="Node at fde746ae-973e-dd7a-f129-47cfd1708c15 [Candidate]" term=2 2025-09-14T21:47:33.334Z [INFO] server-fde746ae-973e-dd7a-f129-47cfd1708c15: election won: term=2 tally=2 2025-09-14T21:47:33.334Z [INFO] server-fde746ae-973e-dd7a-f129-47cfd1708c15: entering leader state: leader="Node at fde746ae-973e-dd7a-f129-47cfd1708c15 [Leader]" 2025-09-14T21:47:33.334Z [INFO] server-fde746ae-973e-dd7a-f129-47cfd1708c15: added peer, starting replication: peer=server-cce5bb44-899a-164a-d35a-13db86b2ff5b 2025-09-14T21:47:33.334Z [INFO] server-fde746ae-973e-dd7a-f129-47cfd1708c15: added peer, starting replication: peer=server-27d8bc2d-17be-3583-e676-c197c7a81722 2025-09-14T21:47:33.334Z [INFO] server-fde746ae-973e-dd7a-f129-47cfd1708c15: pipelining replication: peer="{Voter server-27d8bc2d-17be-3583-e676-c197c7a81722 27d8bc2d-17be-3583-e676-c197c7a81722}" 2025-09-14T21:47:33.334Z [INFO] server-fde746ae-973e-dd7a-f129-47cfd1708c15: pipelining replication: peer="{Voter server-cce5bb44-899a-164a-d35a-13db86b2ff5b cce5bb44-899a-164a-d35a-13db86b2ff5b}" 2025-09-14T21:47:34.340Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:33.334579893 +0000 UTC m=+23.902458165 (1 nodes), 553.005988ms from start of poll, 786.940749ms from cluster start. Timeout at 2025-09-14 21:47:34.339586315 +0000 UTC m=+24.907464577, 1.005006412s after stability 2025-09-14T21:47:34.340Z [INFO] server-fde746ae-973e-dd7a-f129-47cfd1708c15: starting snapshot up to: index=12 2025-09-14T21:47:34.340Z [INFO] creating new snapshot: path=/tmp/raft3513523421/snapshots/2-12-1757886454340.tmp 2025-09-14T21:47:34.341Z [INFO] server-fde746ae-973e-dd7a-f129-47cfd1708c15: snapshot complete up to: index=12 2025-09-14T21:47:34.342Z [INFO] server-3570d3eb-7e1e-839b-cd55-50ebe56c552f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3570d3eb-7e1e-839b-cd55-50ebe56c552f Address:3570d3eb-7e1e-839b-cd55-50ebe56c552f} {Suffrage:Voter ID:server-87992c0d-4415-d675-93d7-339575a8d92f Address:87992c0d-4415-d675-93d7-339575a8d92f} {Suffrage:Voter ID:server-4e453e51-881c-c095-48e1-2e60fccd8f77 Address:4e453e51-881c-c095-48e1-2e60fccd8f77}]" 2025-09-14T21:47:34.342Z [INFO] server-3570d3eb-7e1e-839b-cd55-50ebe56c552f: entering follower state: follower="Node at 3570d3eb-7e1e-839b-cd55-50ebe56c552f [Follower]" leader-address= leader-id= 2025-09-14T21:47:34.342Z [INFO] server-87992c0d-4415-d675-93d7-339575a8d92f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3570d3eb-7e1e-839b-cd55-50ebe56c552f Address:3570d3eb-7e1e-839b-cd55-50ebe56c552f} {Suffrage:Voter ID:server-87992c0d-4415-d675-93d7-339575a8d92f Address:87992c0d-4415-d675-93d7-339575a8d92f} {Suffrage:Voter ID:server-4e453e51-881c-c095-48e1-2e60fccd8f77 Address:4e453e51-881c-c095-48e1-2e60fccd8f77}]" 2025-09-14T21:47:34.342Z [INFO] server-87992c0d-4415-d675-93d7-339575a8d92f: entering follower state: follower="Node at 87992c0d-4415-d675-93d7-339575a8d92f [Follower]" leader-address= leader-id= 2025-09-14T21:47:34.342Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3570d3eb-7e1e-839b-cd55-50ebe56c552f Address:3570d3eb-7e1e-839b-cd55-50ebe56c552f} {Suffrage:Voter ID:server-87992c0d-4415-d675-93d7-339575a8d92f Address:87992c0d-4415-d675-93d7-339575a8d92f} {Suffrage:Voter ID:server-4e453e51-881c-c095-48e1-2e60fccd8f77 Address:4e453e51-881c-c095-48e1-2e60fccd8f77}]" 2025-09-14T21:47:34.342Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:34.342Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: entering follower state: follower="Node at 4e453e51-881c-c095-48e1-2e60fccd8f77 [Follower]" leader-address= leader-id= 2025-09-14T21:47:34.847Z [WARN] server-4e453e51-881c-c095-48e1-2e60fccd8f77: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:34.847Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: entering candidate state: node="Node at 4e453e51-881c-c095-48e1-2e60fccd8f77 [Candidate]" term=2 2025-09-14T21:47:34.847Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: election won: term=2 tally=2 2025-09-14T21:47:34.847Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: entering leader state: leader="Node at 4e453e51-881c-c095-48e1-2e60fccd8f77 [Leader]" 2025-09-14T21:47:34.847Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: added peer, starting replication: peer=server-3570d3eb-7e1e-839b-cd55-50ebe56c552f 2025-09-14T21:47:34.847Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: added peer, starting replication: peer=server-87992c0d-4415-d675-93d7-339575a8d92f 2025-09-14T21:47:34.847Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: pipelining replication: peer="{Voter server-87992c0d-4415-d675-93d7-339575a8d92f 87992c0d-4415-d675-93d7-339575a8d92f}" 2025-09-14T21:47:34.847Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: pipelining replication: peer="{Voter server-3570d3eb-7e1e-839b-cd55-50ebe56c552f 3570d3eb-7e1e-839b-cd55-50ebe56c552f}" 2025-09-14T21:47:35.853Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:34.847317367 +0000 UTC m=+25.415195650 (1 nodes), 504.972697ms from start of poll, 505.34657ms from cluster start. Timeout at 2025-09-14 21:47:35.852320507 +0000 UTC m=+26.420198769, 1.005003119s after stability 2025-09-14T21:47:35.853Z [INFO] creating new snapshot: path=/tmp/raft682359691/snapshots/2-13-1757886455853.tmp 2025-09-14T21:47:35.853Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: copied to local snapshot: bytes=71 2025-09-14T21:47:35.854Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: snapshot restore progress: id=2-13-1757886455853 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:35.854Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: removing all old logs from log store 2025-09-14T21:47:35.854Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: compacting logs: from=1 to=2 2025-09-14T21:47:35.854Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: restored user snapshot: index=13 2025-09-14T21:47:35.854Z [ERROR] server-4e453e51-881c-c095-48e1-2e60fccd8f77: failed to get log: index=2 error="log not found" 2025-09-14T21:47:35.854Z [ERROR] server-4e453e51-881c-c095-48e1-2e60fccd8f77: failed to get log: index=2 error="log not found" 2025-09-14T21:47:35.854Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: aborting pipeline replication: peer="{Voter server-3570d3eb-7e1e-839b-cd55-50ebe56c552f 3570d3eb-7e1e-839b-cd55-50ebe56c552f}" 2025-09-14T21:47:35.854Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: aborting pipeline replication: peer="{Voter server-87992c0d-4415-d675-93d7-339575a8d92f 87992c0d-4415-d675-93d7-339575a8d92f}" 2025-09-14T21:47:35.862Z [ERROR] server-4e453e51-881c-c095-48e1-2e60fccd8f77: failed to get log: index=2 error="log not found" 2025-09-14T21:47:35.863Z [INFO] creating new snapshot: path=/tmp/raft2577800396/snapshots/2-13-1757886455863.tmp 2025-09-14T21:47:35.863Z [INFO] server-3570d3eb-7e1e-839b-cd55-50ebe56c552f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:35.863Z [INFO] server-3570d3eb-7e1e-839b-cd55-50ebe56c552f: copied to local snapshot: bytes=71 2025-09-14T21:47:35.863Z [ERROR] server-4e453e51-881c-c095-48e1-2e60fccd8f77: failed to get log: index=2 error="log not found" 2025-09-14T21:47:35.863Z [INFO] server-3570d3eb-7e1e-839b-cd55-50ebe56c552f: snapshot restore progress: id=2-13-1757886455863 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:35.863Z [INFO] server-3570d3eb-7e1e-839b-cd55-50ebe56c552f: removing all old logs from log store 2025-09-14T21:47:35.863Z [INFO] server-3570d3eb-7e1e-839b-cd55-50ebe56c552f: compacting logs: from=1 to=2 2025-09-14T21:47:35.863Z [INFO] server-3570d3eb-7e1e-839b-cd55-50ebe56c552f: Installed remote snapshot 2025-09-14T21:47:35.863Z [INFO] creating new snapshot: path=/tmp/raft3819354238/snapshots/2-13-1757886455863.tmp 2025-09-14T21:47:35.863Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: pipelining replication: peer="{Voter server-3570d3eb-7e1e-839b-cd55-50ebe56c552f 3570d3eb-7e1e-839b-cd55-50ebe56c552f}" 2025-09-14T21:47:35.863Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:35.864Z [INFO] server-87992c0d-4415-d675-93d7-339575a8d92f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:35.864Z [INFO] server-87992c0d-4415-d675-93d7-339575a8d92f: copied to local snapshot: bytes=71 2025-09-14T21:47:35.864Z [INFO] server-87992c0d-4415-d675-93d7-339575a8d92f: snapshot restore progress: id=2-13-1757886455863 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:35.864Z [INFO] server-87992c0d-4415-d675-93d7-339575a8d92f: removing all old logs from log store 2025-09-14T21:47:35.864Z [INFO] server-87992c0d-4415-d675-93d7-339575a8d92f: compacting logs: from=1 to=2 2025-09-14T21:47:35.864Z [INFO] server-87992c0d-4415-d675-93d7-339575a8d92f: Installed remote snapshot 2025-09-14T21:47:35.864Z [INFO] server-4e453e51-881c-c095-48e1-2e60fccd8f77: pipelining replication: peer="{Voter server-87992c0d-4415-d675-93d7-339575a8d92f 87992c0d-4415-d675-93d7-339575a8d92f}" 2025-09-14T21:47:36.869Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:35.863934466 +0000 UTC m=+26.431812724 (2 nodes), 344ns from start of poll, 1.521963644s from cluster start. Timeout at 2025-09-14 21:47:36.868943133 +0000 UTC m=+27.436821414, 1.00500869s after stability 2025-09-14T21:47:36.882Z [INFO] server-fde746ae-973e-dd7a-f129-47cfd1708c15: aborting pipeline replication: peer="{Voter server-27d8bc2d-17be-3583-e676-c197c7a81722 27d8bc2d-17be-3583-e676-c197c7a81722}" 2025-09-14T21:47:36.882Z [INFO] server-fde746ae-973e-dd7a-f129-47cfd1708c15: aborting pipeline replication: peer="{Voter server-cce5bb44-899a-164a-d35a-13db86b2ff5b cce5bb44-899a-164a-d35a-13db86b2ff5b}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false 2025-09-14T21:47:36.883Z [INFO] server-e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb Address:e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb} {Suffrage:Voter ID:server-8e7f5f04-ee19-2f67-2c20-6025624d3685 Address:8e7f5f04-ee19-2f67-2c20-6025624d3685} {Suffrage:Voter ID:server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3 Address:3ccc1f4b-47b1-4380-b618-9fd8acb803d3}]" 2025-09-14T21:47:36.883Z [INFO] server-e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb: entering follower state: follower="Node at e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb [Follower]" leader-address= leader-id= 2025-09-14T21:47:36.883Z [INFO] server-8e7f5f04-ee19-2f67-2c20-6025624d3685: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb Address:e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb} {Suffrage:Voter ID:server-8e7f5f04-ee19-2f67-2c20-6025624d3685 Address:8e7f5f04-ee19-2f67-2c20-6025624d3685} {Suffrage:Voter ID:server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3 Address:3ccc1f4b-47b1-4380-b618-9fd8acb803d3}]" 2025-09-14T21:47:36.883Z [INFO] server-8e7f5f04-ee19-2f67-2c20-6025624d3685: entering follower state: follower="Node at 8e7f5f04-ee19-2f67-2c20-6025624d3685 [Follower]" leader-address= leader-id= 2025-09-14T21:47:36.884Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb Address:e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb} {Suffrage:Voter ID:server-8e7f5f04-ee19-2f67-2c20-6025624d3685 Address:8e7f5f04-ee19-2f67-2c20-6025624d3685} {Suffrage:Voter ID:server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3 Address:3ccc1f4b-47b1-4380-b618-9fd8acb803d3}]" 2025-09-14T21:47:36.884Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:36.884Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: entering follower state: follower="Node at 3ccc1f4b-47b1-4380-b618-9fd8acb803d3 [Follower]" leader-address= leader-id= 2025-09-14T21:47:37.497Z [WARN] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:37.497Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: entering candidate state: node="Node at 3ccc1f4b-47b1-4380-b618-9fd8acb803d3 [Candidate]" term=2 2025-09-14T21:47:37.497Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: election won: term=2 tally=2 2025-09-14T21:47:37.497Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: entering leader state: leader="Node at 3ccc1f4b-47b1-4380-b618-9fd8acb803d3 [Leader]" 2025-09-14T21:47:37.497Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: added peer, starting replication: peer=server-e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb 2025-09-14T21:47:37.497Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: added peer, starting replication: peer=server-8e7f5f04-ee19-2f67-2c20-6025624d3685 2025-09-14T21:47:37.497Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: pipelining replication: peer="{Voter server-e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb}" 2025-09-14T21:47:37.497Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: pipelining replication: peer="{Voter server-8e7f5f04-ee19-2f67-2c20-6025624d3685 8e7f5f04-ee19-2f67-2c20-6025624d3685}" 2025-09-14T21:47:38.502Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:37.497523146 +0000 UTC m=+28.065401421 (1 nodes), 613.433516ms from start of poll, 613.801843ms from cluster start. Timeout at 2025-09-14 21:47:38.50252629 +0000 UTC m=+29.070404554, 1.005003133s after stability 2025-09-14T21:47:38.503Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: starting snapshot up to: index=12 2025-09-14T21:47:38.503Z [INFO] creating new snapshot: path=/tmp/raft3191540930/snapshots/2-12-1757886458503.tmp 2025-09-14T21:47:38.503Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: snapshot complete up to: index=12 2025-09-14T21:47:38.504Z [INFO] creating new snapshot: path=/tmp/raft3191540930/snapshots/2-23-1757886458504.tmp 2025-09-14T21:47:38.504Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: copied to local snapshot: bytes=71 2025-09-14T21:47:38.505Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: snapshot restore progress: id=2-23-1757886458504 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:38.505Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: restored user snapshot: index=23 2025-09-14T21:47:38.505Z [ERROR] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: failed to get log: index=23 error="log not found" 2025-09-14T21:47:38.505Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: aborting pipeline replication: peer="{Voter server-8e7f5f04-ee19-2f67-2c20-6025624d3685 8e7f5f04-ee19-2f67-2c20-6025624d3685}" 2025-09-14T21:47:38.505Z [ERROR] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: failed to get log: index=23 error="log not found" 2025-09-14T21:47:38.505Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: aborting pipeline replication: peer="{Voter server-e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb}" 2025-09-14T21:47:38.511Z [ERROR] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: failed to get log: index=23 error="log not found" 2025-09-14T21:47:38.512Z [INFO] creating new snapshot: path=/tmp/raft198132707/snapshots/2-23-1757886458512.tmp 2025-09-14T21:47:38.512Z [INFO] server-e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:38.512Z [INFO] server-e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb: copied to local snapshot: bytes=71 2025-09-14T21:47:38.513Z [ERROR] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: failed to get log: index=23 error="log not found" 2025-09-14T21:47:38.513Z [INFO] server-e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb: snapshot restore progress: id=2-23-1757886458512 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:38.513Z [INFO] server-e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb: Installed remote snapshot 2025-09-14T21:47:38.513Z [WARN] server-e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb: skipping application of old log: index=22 2025-09-14T21:47:38.513Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: pipelining replication: peer="{Voter server-e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb}" 2025-09-14T21:47:38.513Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:38.513Z [INFO] creating new snapshot: path=/tmp/raft768387708/snapshots/2-23-1757886458513.tmp 2025-09-14T21:47:38.513Z [INFO] server-8e7f5f04-ee19-2f67-2c20-6025624d3685: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:38.513Z [INFO] server-8e7f5f04-ee19-2f67-2c20-6025624d3685: copied to local snapshot: bytes=71 2025-09-14T21:47:38.514Z [INFO] server-8e7f5f04-ee19-2f67-2c20-6025624d3685: snapshot restore progress: id=2-23-1757886458513 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:38.514Z [INFO] server-8e7f5f04-ee19-2f67-2c20-6025624d3685: Installed remote snapshot 2025-09-14T21:47:38.514Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: pipelining replication: peer="{Voter server-8e7f5f04-ee19-2f67-2c20-6025624d3685 8e7f5f04-ee19-2f67-2c20-6025624d3685}" 2025-09-14T21:47:39.518Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:38.513266898 +0000 UTC m=+29.081145168 (2 nodes), 573ns from start of poll, 1.62954559s from cluster start. Timeout at 2025-09-14 21:47:39.51827688 +0000 UTC m=+30.086155148, 1.00500998s after stability 2025-09-14T21:47:39.535Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: aborting pipeline replication: peer="{Voter server-8e7f5f04-ee19-2f67-2c20-6025624d3685 8e7f5f04-ee19-2f67-2c20-6025624d3685}" 2025-09-14T21:47:39.535Z [INFO] server-3ccc1f4b-47b1-4380-b618-9fd8acb803d3: aborting pipeline replication: peer="{Voter server-e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb e3b1fee3-ac20-11d3-cdb7-a03e6334a8eb}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false 2025-09-14T21:47:39.537Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b49f1ce-5ddb-85a9-275a-818644bda5b5 Address:6b49f1ce-5ddb-85a9-275a-818644bda5b5} {Suffrage:Voter ID:server-d9221201-714a-ac7e-905c-582644bf1697 Address:d9221201-714a-ac7e-905c-582644bf1697} {Suffrage:Voter ID:server-73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c Address:73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c}]" 2025-09-14T21:47:39.537Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: entering follower state: follower="Node at 6b49f1ce-5ddb-85a9-275a-818644bda5b5 [Follower]" leader-address= leader-id= 2025-09-14T21:47:39.537Z [INFO] server-d9221201-714a-ac7e-905c-582644bf1697: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b49f1ce-5ddb-85a9-275a-818644bda5b5 Address:6b49f1ce-5ddb-85a9-275a-818644bda5b5} {Suffrage:Voter ID:server-d9221201-714a-ac7e-905c-582644bf1697 Address:d9221201-714a-ac7e-905c-582644bf1697} {Suffrage:Voter ID:server-73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c Address:73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c}]" 2025-09-14T21:47:39.537Z [INFO] server-d9221201-714a-ac7e-905c-582644bf1697: entering follower state: follower="Node at d9221201-714a-ac7e-905c-582644bf1697 [Follower]" leader-address= leader-id= 2025-09-14T21:47:39.537Z [INFO] server-73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b49f1ce-5ddb-85a9-275a-818644bda5b5 Address:6b49f1ce-5ddb-85a9-275a-818644bda5b5} {Suffrage:Voter ID:server-d9221201-714a-ac7e-905c-582644bf1697 Address:d9221201-714a-ac7e-905c-582644bf1697} {Suffrage:Voter ID:server-73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c Address:73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c}]" 2025-09-14T21:47:39.537Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:39.537Z [INFO] server-73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c: entering follower state: follower="Node at 73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c [Follower]" leader-address= leader-id= 2025-09-14T21:47:40.169Z [WARN] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:40.169Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: entering candidate state: node="Node at 6b49f1ce-5ddb-85a9-275a-818644bda5b5 [Candidate]" term=2 2025-09-14T21:47:40.169Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: election won: term=2 tally=2 2025-09-14T21:47:40.169Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: entering leader state: leader="Node at 6b49f1ce-5ddb-85a9-275a-818644bda5b5 [Leader]" 2025-09-14T21:47:40.169Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: added peer, starting replication: peer=server-d9221201-714a-ac7e-905c-582644bf1697 2025-09-14T21:47:40.169Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: added peer, starting replication: peer=server-73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c 2025-09-14T21:47:40.169Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: pipelining replication: peer="{Voter server-73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c 73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c}" 2025-09-14T21:47:40.169Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: pipelining replication: peer="{Voter server-d9221201-714a-ac7e-905c-582644bf1697 d9221201-714a-ac7e-905c-582644bf1697}" 2025-09-14T21:47:41.174Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:40.169290926 +0000 UTC m=+30.737169192 (1 nodes), 632.006057ms from start of poll, 632.338821ms from cluster start. Timeout at 2025-09-14 21:47:41.174292096 +0000 UTC m=+31.742170355, 1.005001163s after stability 2025-09-14T21:47:41.174Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: starting snapshot up to: index=12 2025-09-14T21:47:41.174Z [INFO] creating new snapshot: path=/tmp/raft231897056/snapshots/2-12-1757886461174.tmp 2025-09-14T21:47:41.175Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: snapshot complete up to: index=12 2025-09-14T21:47:41.175Z [INFO] creating new snapshot: path=/tmp/raft231897056/snapshots/2-23-1757886461175.tmp 2025-09-14T21:47:41.176Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: copied to local snapshot: bytes=71 2025-09-14T21:47:41.176Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: snapshot restore progress: id=2-23-1757886461175 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:41.176Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: removing all old logs from log store 2025-09-14T21:47:41.176Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: compacting logs: from=1 to=22 2025-09-14T21:47:41.176Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: restored user snapshot: index=23 2025-09-14T21:47:41.176Z [ERROR] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: failed to get log: index=22 error="log not found" 2025-09-14T21:47:41.176Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: aborting pipeline replication: peer="{Voter server-d9221201-714a-ac7e-905c-582644bf1697 d9221201-714a-ac7e-905c-582644bf1697}" 2025-09-14T21:47:41.176Z [ERROR] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: failed to get log: index=22 error="log not found" 2025-09-14T21:47:41.176Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: aborting pipeline replication: peer="{Voter server-73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c 73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c}" 2025-09-14T21:47:41.185Z [ERROR] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: failed to get log: index=22 error="log not found" 2025-09-14T21:47:41.186Z [INFO] creating new snapshot: path=/tmp/raft47809012/snapshots/2-23-1757886461186.tmp 2025-09-14T21:47:41.186Z [ERROR] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: failed to get log: index=22 error="log not found" 2025-09-14T21:47:41.186Z [INFO] server-73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:41.186Z [INFO] creating new snapshot: path=/tmp/raft3752722665/snapshots/2-23-1757886461186.tmp 2025-09-14T21:47:41.186Z [INFO] server-73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c: copied to local snapshot: bytes=71 2025-09-14T21:47:41.186Z [INFO] server-d9221201-714a-ac7e-905c-582644bf1697: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:41.186Z [INFO] server-73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c: snapshot restore progress: id=2-23-1757886461186 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:41.186Z [INFO] server-73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c: removing all old logs from log store 2025-09-14T21:47:41.186Z [INFO] server-73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c: compacting logs: from=1 to=22 2025-09-14T21:47:41.186Z [INFO] server-73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c: Installed remote snapshot 2025-09-14T21:47:41.186Z [WARN] server-73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c: skipping application of old log: index=22 2025-09-14T21:47:41.186Z [INFO] server-d9221201-714a-ac7e-905c-582644bf1697: copied to local snapshot: bytes=71 2025-09-14T21:47:41.186Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: pipelining replication: peer="{Voter server-73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c 73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c}" 2025-09-14T21:47:41.187Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:41.187Z [INFO] server-d9221201-714a-ac7e-905c-582644bf1697: snapshot restore progress: id=2-23-1757886461186 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:41.187Z [INFO] server-d9221201-714a-ac7e-905c-582644bf1697: removing all old logs from log store 2025-09-14T21:47:41.187Z [INFO] server-d9221201-714a-ac7e-905c-582644bf1697: compacting logs: from=1 to=22 2025-09-14T21:47:41.187Z [INFO] server-d9221201-714a-ac7e-905c-582644bf1697: Installed remote snapshot 2025-09-14T21:47:41.188Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: pipelining replication: peer="{Voter server-d9221201-714a-ac7e-905c-582644bf1697 d9221201-714a-ac7e-905c-582644bf1697}" 2025-09-14T21:47:42.192Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:41.187033641 +0000 UTC m=+31.754911916 (2 nodes), 695ns from start of poll, 1.650081545s from cluster start. Timeout at 2025-09-14 21:47:42.192036839 +0000 UTC m=+32.759915102, 1.005003186s after stability 2025-09-14T21:47:42.203Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: aborting pipeline replication: peer="{Voter server-73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c 73619e0d-2ef6-3dd5-da93-9c0a9cff4a6c}" 2025-09-14T21:47:42.203Z [INFO] server-6b49f1ce-5ddb-85a9-275a-818644bda5b5: aborting pipeline replication: peer="{Voter server-d9221201-714a-ac7e-905c-582644bf1697 d9221201-714a-ac7e-905c-582644bf1697}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true 2025-09-14T21:47:42.204Z [INFO] server-e58746ab-9a9b-c4bf-92ef-6a90e735dcf1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e58746ab-9a9b-c4bf-92ef-6a90e735dcf1 Address:e58746ab-9a9b-c4bf-92ef-6a90e735dcf1} {Suffrage:Voter ID:server-6aa923d8-e7fb-347e-7cb1-1d39f51f2093 Address:6aa923d8-e7fb-347e-7cb1-1d39f51f2093} {Suffrage:Voter ID:server-e7ee7458-1b96-f983-d94a-95eca36139f1 Address:e7ee7458-1b96-f983-d94a-95eca36139f1}]" 2025-09-14T21:47:42.204Z [INFO] server-6aa923d8-e7fb-347e-7cb1-1d39f51f2093: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e58746ab-9a9b-c4bf-92ef-6a90e735dcf1 Address:e58746ab-9a9b-c4bf-92ef-6a90e735dcf1} {Suffrage:Voter ID:server-6aa923d8-e7fb-347e-7cb1-1d39f51f2093 Address:6aa923d8-e7fb-347e-7cb1-1d39f51f2093} {Suffrage:Voter ID:server-e7ee7458-1b96-f983-d94a-95eca36139f1 Address:e7ee7458-1b96-f983-d94a-95eca36139f1}]" 2025-09-14T21:47:42.204Z [INFO] server-e58746ab-9a9b-c4bf-92ef-6a90e735dcf1: entering follower state: follower="Node at e58746ab-9a9b-c4bf-92ef-6a90e735dcf1 [Follower]" leader-address= leader-id= 2025-09-14T21:47:42.204Z [INFO] server-6aa923d8-e7fb-347e-7cb1-1d39f51f2093: entering follower state: follower="Node at 6aa923d8-e7fb-347e-7cb1-1d39f51f2093 [Follower]" leader-address= leader-id= 2025-09-14T21:47:42.204Z [INFO] server-e7ee7458-1b96-f983-d94a-95eca36139f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e58746ab-9a9b-c4bf-92ef-6a90e735dcf1 Address:e58746ab-9a9b-c4bf-92ef-6a90e735dcf1} {Suffrage:Voter ID:server-6aa923d8-e7fb-347e-7cb1-1d39f51f2093 Address:6aa923d8-e7fb-347e-7cb1-1d39f51f2093} {Suffrage:Voter ID:server-e7ee7458-1b96-f983-d94a-95eca36139f1 Address:e7ee7458-1b96-f983-d94a-95eca36139f1}]" 2025-09-14T21:47:42.204Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:42.204Z [INFO] server-e7ee7458-1b96-f983-d94a-95eca36139f1: entering follower state: follower="Node at e7ee7458-1b96-f983-d94a-95eca36139f1 [Follower]" leader-address= leader-id= 2025-09-14T21:47:42.785Z [WARN] server-6aa923d8-e7fb-347e-7cb1-1d39f51f2093: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:42.785Z [INFO] server-6aa923d8-e7fb-347e-7cb1-1d39f51f2093: entering candidate state: node="Node at 6aa923d8-e7fb-347e-7cb1-1d39f51f2093 [Candidate]" term=2 2025-09-14T21:47:42.785Z [INFO] server-6aa923d8-e7fb-347e-7cb1-1d39f51f2093: election won: term=2 tally=2 2025-09-14T21:47:42.785Z [INFO] server-6aa923d8-e7fb-347e-7cb1-1d39f51f2093: entering leader state: leader="Node at 6aa923d8-e7fb-347e-7cb1-1d39f51f2093 [Leader]" 2025-09-14T21:47:42.785Z [INFO] server-6aa923d8-e7fb-347e-7cb1-1d39f51f2093: added peer, starting replication: peer=server-e58746ab-9a9b-c4bf-92ef-6a90e735dcf1 2025-09-14T21:47:42.785Z [INFO] server-6aa923d8-e7fb-347e-7cb1-1d39f51f2093: added peer, starting replication: peer=server-e7ee7458-1b96-f983-d94a-95eca36139f1 2025-09-14T21:47:42.785Z [INFO] server-6aa923d8-e7fb-347e-7cb1-1d39f51f2093: pipelining replication: peer="{Voter server-e58746ab-9a9b-c4bf-92ef-6a90e735dcf1 e58746ab-9a9b-c4bf-92ef-6a90e735dcf1}" 2025-09-14T21:47:42.785Z [INFO] server-6aa923d8-e7fb-347e-7cb1-1d39f51f2093: pipelining replication: peer="{Voter server-e7ee7458-1b96-f983-d94a-95eca36139f1 e7ee7458-1b96-f983-d94a-95eca36139f1}" 2025-09-14T21:47:43.791Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:42.785546609 +0000 UTC m=+33.353424883 (1 nodes), 580.648661ms from start of poll, 581.018158ms from cluster start. Timeout at 2025-09-14 21:47:43.790547217 +0000 UTC m=+34.358425477, 1.005000594s after stability 2025-09-14T21:47:43.792Z [INFO] server-6aa923d8-e7fb-347e-7cb1-1d39f51f2093: starting snapshot up to: index=12 2025-09-14T21:47:43.792Z [INFO] creating new snapshot: path=/tmp/raft1418677721/snapshots/2-12-1757886463792.tmp 2025-09-14T21:47:43.793Z [INFO] server-6aa923d8-e7fb-347e-7cb1-1d39f51f2093: snapshot complete up to: index=12 2025-09-14T21:47:43.794Z [INFO] server-aa408c05-a067-7ab9-5279-c4bd1e97375b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa408c05-a067-7ab9-5279-c4bd1e97375b Address:aa408c05-a067-7ab9-5279-c4bd1e97375b} {Suffrage:Voter ID:server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c Address:1c6bc856-c7fe-49bd-cd37-4bd2af91b47c} {Suffrage:Voter ID:server-4eda85dc-1e00-5969-c42d-cfe089004319 Address:4eda85dc-1e00-5969-c42d-cfe089004319}]" 2025-09-14T21:47:43.794Z [INFO] server-aa408c05-a067-7ab9-5279-c4bd1e97375b: entering follower state: follower="Node at aa408c05-a067-7ab9-5279-c4bd1e97375b [Follower]" leader-address= leader-id= 2025-09-14T21:47:43.794Z [INFO] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa408c05-a067-7ab9-5279-c4bd1e97375b Address:aa408c05-a067-7ab9-5279-c4bd1e97375b} {Suffrage:Voter ID:server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c Address:1c6bc856-c7fe-49bd-cd37-4bd2af91b47c} {Suffrage:Voter ID:server-4eda85dc-1e00-5969-c42d-cfe089004319 Address:4eda85dc-1e00-5969-c42d-cfe089004319}]" 2025-09-14T21:47:43.794Z [INFO] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: entering follower state: follower="Node at 1c6bc856-c7fe-49bd-cd37-4bd2af91b47c [Follower]" leader-address= leader-id= 2025-09-14T21:47:43.794Z [INFO] server-4eda85dc-1e00-5969-c42d-cfe089004319: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa408c05-a067-7ab9-5279-c4bd1e97375b Address:aa408c05-a067-7ab9-5279-c4bd1e97375b} {Suffrage:Voter ID:server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c Address:1c6bc856-c7fe-49bd-cd37-4bd2af91b47c} {Suffrage:Voter ID:server-4eda85dc-1e00-5969-c42d-cfe089004319 Address:4eda85dc-1e00-5969-c42d-cfe089004319}]" 2025-09-14T21:47:43.794Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:43.795Z [INFO] server-4eda85dc-1e00-5969-c42d-cfe089004319: entering follower state: follower="Node at 4eda85dc-1e00-5969-c42d-cfe089004319 [Follower]" leader-address= leader-id= 2025-09-14T21:47:44.423Z [WARN] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:44.423Z [INFO] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: entering candidate state: node="Node at 1c6bc856-c7fe-49bd-cd37-4bd2af91b47c [Candidate]" term=2 2025-09-14T21:47:44.423Z [INFO] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: election won: term=2 tally=2 2025-09-14T21:47:44.423Z [INFO] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: entering leader state: leader="Node at 1c6bc856-c7fe-49bd-cd37-4bd2af91b47c [Leader]" 2025-09-14T21:47:44.423Z [INFO] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: added peer, starting replication: peer=server-aa408c05-a067-7ab9-5279-c4bd1e97375b 2025-09-14T21:47:44.423Z [INFO] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: added peer, starting replication: peer=server-4eda85dc-1e00-5969-c42d-cfe089004319 2025-09-14T21:47:44.423Z [INFO] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: pipelining replication: peer="{Voter server-aa408c05-a067-7ab9-5279-c4bd1e97375b aa408c05-a067-7ab9-5279-c4bd1e97375b}" 2025-09-14T21:47:44.423Z [INFO] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: pipelining replication: peer="{Voter server-4eda85dc-1e00-5969-c42d-cfe089004319 4eda85dc-1e00-5969-c42d-cfe089004319}" 2025-09-14T21:47:45.429Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:44.423784824 +0000 UTC m=+34.991663093 (1 nodes), 628.780871ms from start of poll, 629.275107ms from cluster start. Timeout at 2025-09-14 21:47:45.428787261 +0000 UTC m=+35.996665523, 1.00500243s after stability 2025-09-14T21:47:45.429Z [INFO] creating new snapshot: path=/tmp/raft2351699010/snapshots/2-14-1757886465429.tmp 2025-09-14T21:47:45.430Z [INFO] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: copied to local snapshot: bytes=71 2025-09-14T21:47:45.430Z [INFO] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: snapshot restore progress: id=2-14-1757886465429 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:45.430Z [INFO] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: restored user snapshot: index=14 2025-09-14T21:47:45.430Z [ERROR] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: failed to get log: index=3 error="log not found" 2025-09-14T21:47:45.430Z [INFO] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: aborting pipeline replication: peer="{Voter server-4eda85dc-1e00-5969-c42d-cfe089004319 4eda85dc-1e00-5969-c42d-cfe089004319}" 2025-09-14T21:47:45.430Z [ERROR] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: failed to get log: index=3 error="log not found" 2025-09-14T21:47:45.430Z [INFO] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: aborting pipeline replication: peer="{Voter server-aa408c05-a067-7ab9-5279-c4bd1e97375b aa408c05-a067-7ab9-5279-c4bd1e97375b}" 2025-09-14T21:47:45.437Z [ERROR] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: failed to get log: index=3 error="log not found" 2025-09-14T21:47:45.437Z [ERROR] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: failed to get log: index=3 error="log not found" 2025-09-14T21:47:45.437Z [INFO] creating new snapshot: path=/tmp/raft3182193939/snapshots/2-14-1757886465437.tmp 2025-09-14T21:47:45.437Z [INFO] creating new snapshot: path=/tmp/raft2205415420/snapshots/2-14-1757886465437.tmp 2025-09-14T21:47:45.437Z [INFO] server-4eda85dc-1e00-5969-c42d-cfe089004319: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:45.437Z [INFO] server-aa408c05-a067-7ab9-5279-c4bd1e97375b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:45.438Z [INFO] server-aa408c05-a067-7ab9-5279-c4bd1e97375b: copied to local snapshot: bytes=71 2025-09-14T21:47:45.438Z [INFO] server-4eda85dc-1e00-5969-c42d-cfe089004319: copied to local snapshot: bytes=71 2025-09-14T21:47:45.438Z [INFO] server-4eda85dc-1e00-5969-c42d-cfe089004319: snapshot restore progress: id=2-14-1757886465437 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:45.438Z [INFO] server-4eda85dc-1e00-5969-c42d-cfe089004319: Installed remote snapshot 2025-09-14T21:47:45.438Z [INFO] server-aa408c05-a067-7ab9-5279-c4bd1e97375b: snapshot restore progress: id=2-14-1757886465437 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:45.438Z [INFO] server-aa408c05-a067-7ab9-5279-c4bd1e97375b: Installed remote snapshot 2025-09-14T21:47:45.438Z [INFO] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: pipelining replication: peer="{Voter server-4eda85dc-1e00-5969-c42d-cfe089004319 4eda85dc-1e00-5969-c42d-cfe089004319}" 2025-09-14T21:47:45.438Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:45.438Z [INFO] server-1c6bc856-c7fe-49bd-cd37-4bd2af91b47c: pipelining replication: peer="{Voter server-aa408c05-a067-7ab9-5279-c4bd1e97375b aa408c05-a067-7ab9-5279-c4bd1e97375b}" 2025-09-14T21:47:46.444Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:45.438644312 +0000 UTC m=+36.006526863 (2 nodes), 5.182µs from start of poll, 1.644138877s from cluster start. Timeout at 2025-09-14 21:47:46.4436542 +0000 UTC m=+37.011532458, 1.005005595s after stability 2025-09-14T21:47:46.455Z [INFO] server-6aa923d8-e7fb-347e-7cb1-1d39f51f2093: aborting pipeline replication: peer="{Voter server-e7ee7458-1b96-f983-d94a-95eca36139f1 e7ee7458-1b96-f983-d94a-95eca36139f1}" 2025-09-14T21:47:46.455Z [INFO] server-6aa923d8-e7fb-347e-7cb1-1d39f51f2093: aborting pipeline replication: peer="{Voter server-e58746ab-9a9b-c4bf-92ef-6a90e735dcf1 e58746ab-9a9b-c4bf-92ef-6a90e735dcf1}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true 2025-09-14T21:47:46.456Z [INFO] server-88fcc125-0bf1-c090-397b-1f999cacbaa4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88fcc125-0bf1-c090-397b-1f999cacbaa4 Address:88fcc125-0bf1-c090-397b-1f999cacbaa4} {Suffrage:Voter ID:server-f1289554-89fb-002e-1a04-8c38b166dfd3 Address:f1289554-89fb-002e-1a04-8c38b166dfd3} {Suffrage:Voter ID:server-a140599f-22cc-5e1c-776f-7a7d6a002eee Address:a140599f-22cc-5e1c-776f-7a7d6a002eee}]" 2025-09-14T21:47:46.456Z [INFO] server-88fcc125-0bf1-c090-397b-1f999cacbaa4: entering follower state: follower="Node at 88fcc125-0bf1-c090-397b-1f999cacbaa4 [Follower]" leader-address= leader-id= 2025-09-14T21:47:46.456Z [INFO] server-f1289554-89fb-002e-1a04-8c38b166dfd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88fcc125-0bf1-c090-397b-1f999cacbaa4 Address:88fcc125-0bf1-c090-397b-1f999cacbaa4} {Suffrage:Voter ID:server-f1289554-89fb-002e-1a04-8c38b166dfd3 Address:f1289554-89fb-002e-1a04-8c38b166dfd3} {Suffrage:Voter ID:server-a140599f-22cc-5e1c-776f-7a7d6a002eee Address:a140599f-22cc-5e1c-776f-7a7d6a002eee}]" 2025-09-14T21:47:46.456Z [INFO] server-f1289554-89fb-002e-1a04-8c38b166dfd3: entering follower state: follower="Node at f1289554-89fb-002e-1a04-8c38b166dfd3 [Follower]" leader-address= leader-id= 2025-09-14T21:47:46.456Z [INFO] server-a140599f-22cc-5e1c-776f-7a7d6a002eee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88fcc125-0bf1-c090-397b-1f999cacbaa4 Address:88fcc125-0bf1-c090-397b-1f999cacbaa4} {Suffrage:Voter ID:server-f1289554-89fb-002e-1a04-8c38b166dfd3 Address:f1289554-89fb-002e-1a04-8c38b166dfd3} {Suffrage:Voter ID:server-a140599f-22cc-5e1c-776f-7a7d6a002eee Address:a140599f-22cc-5e1c-776f-7a7d6a002eee}]" 2025-09-14T21:47:46.456Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:46.456Z [INFO] server-a140599f-22cc-5e1c-776f-7a7d6a002eee: entering follower state: follower="Node at a140599f-22cc-5e1c-776f-7a7d6a002eee [Follower]" leader-address= leader-id= 2025-09-14T21:47:47.031Z [WARN] server-88fcc125-0bf1-c090-397b-1f999cacbaa4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:47.031Z [INFO] server-88fcc125-0bf1-c090-397b-1f999cacbaa4: entering candidate state: node="Node at 88fcc125-0bf1-c090-397b-1f999cacbaa4 [Candidate]" term=2 2025-09-14T21:47:47.031Z [INFO] server-88fcc125-0bf1-c090-397b-1f999cacbaa4: election won: term=2 tally=2 2025-09-14T21:47:47.031Z [INFO] server-88fcc125-0bf1-c090-397b-1f999cacbaa4: entering leader state: leader="Node at 88fcc125-0bf1-c090-397b-1f999cacbaa4 [Leader]" 2025-09-14T21:47:47.031Z [INFO] server-88fcc125-0bf1-c090-397b-1f999cacbaa4: added peer, starting replication: peer=server-f1289554-89fb-002e-1a04-8c38b166dfd3 2025-09-14T21:47:47.031Z [INFO] server-88fcc125-0bf1-c090-397b-1f999cacbaa4: added peer, starting replication: peer=server-a140599f-22cc-5e1c-776f-7a7d6a002eee 2025-09-14T21:47:47.032Z [INFO] server-88fcc125-0bf1-c090-397b-1f999cacbaa4: pipelining replication: peer="{Voter server-a140599f-22cc-5e1c-776f-7a7d6a002eee a140599f-22cc-5e1c-776f-7a7d6a002eee}" 2025-09-14T21:47:47.032Z [INFO] server-88fcc125-0bf1-c090-397b-1f999cacbaa4: pipelining replication: peer="{Voter server-f1289554-89fb-002e-1a04-8c38b166dfd3 f1289554-89fb-002e-1a04-8c38b166dfd3}" 2025-09-14T21:47:48.037Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:47.03204626 +0000 UTC m=+37.599924542 (1 nodes), 575.14222ms from start of poll, 575.489835ms from cluster start. Timeout at 2025-09-14 21:47:48.037049804 +0000 UTC m=+38.604928066, 1.005003524s after stability 2025-09-14T21:47:48.037Z [INFO] server-88fcc125-0bf1-c090-397b-1f999cacbaa4: starting snapshot up to: index=12 2025-09-14T21:47:48.038Z [INFO] creating new snapshot: path=/tmp/raft1582461060/snapshots/2-12-1757886468038.tmp 2025-09-14T21:47:48.038Z [INFO] server-88fcc125-0bf1-c090-397b-1f999cacbaa4: snapshot complete up to: index=12 2025-09-14T21:47:48.039Z [INFO] server-44db6509-c2cc-56cf-9d4b-a7b59633c136: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44db6509-c2cc-56cf-9d4b-a7b59633c136 Address:44db6509-c2cc-56cf-9d4b-a7b59633c136} {Suffrage:Voter ID:server-a45966cf-4970-6e3e-0762-a695f6366ab8 Address:a45966cf-4970-6e3e-0762-a695f6366ab8} {Suffrage:Voter ID:server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec Address:564c2386-ae8b-2d8e-9fe0-1a34ecb724ec}]" 2025-09-14T21:47:48.039Z [INFO] server-44db6509-c2cc-56cf-9d4b-a7b59633c136: entering follower state: follower="Node at 44db6509-c2cc-56cf-9d4b-a7b59633c136 [Follower]" leader-address= leader-id= 2025-09-14T21:47:48.039Z [INFO] server-a45966cf-4970-6e3e-0762-a695f6366ab8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44db6509-c2cc-56cf-9d4b-a7b59633c136 Address:44db6509-c2cc-56cf-9d4b-a7b59633c136} {Suffrage:Voter ID:server-a45966cf-4970-6e3e-0762-a695f6366ab8 Address:a45966cf-4970-6e3e-0762-a695f6366ab8} {Suffrage:Voter ID:server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec Address:564c2386-ae8b-2d8e-9fe0-1a34ecb724ec}]" 2025-09-14T21:47:48.039Z [INFO] server-a45966cf-4970-6e3e-0762-a695f6366ab8: entering follower state: follower="Node at a45966cf-4970-6e3e-0762-a695f6366ab8 [Follower]" leader-address= leader-id= 2025-09-14T21:47:48.039Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44db6509-c2cc-56cf-9d4b-a7b59633c136 Address:44db6509-c2cc-56cf-9d4b-a7b59633c136} {Suffrage:Voter ID:server-a45966cf-4970-6e3e-0762-a695f6366ab8 Address:a45966cf-4970-6e3e-0762-a695f6366ab8} {Suffrage:Voter ID:server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec Address:564c2386-ae8b-2d8e-9fe0-1a34ecb724ec}]" 2025-09-14T21:47:48.039Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:48.039Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: entering follower state: follower="Node at 564c2386-ae8b-2d8e-9fe0-1a34ecb724ec [Follower]" leader-address= leader-id= 2025-09-14T21:47:48.632Z [WARN] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:48.632Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: entering candidate state: node="Node at 564c2386-ae8b-2d8e-9fe0-1a34ecb724ec [Candidate]" term=2 2025-09-14T21:47:48.633Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: election won: term=2 tally=2 2025-09-14T21:47:48.633Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: entering leader state: leader="Node at 564c2386-ae8b-2d8e-9fe0-1a34ecb724ec [Leader]" 2025-09-14T21:47:48.633Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: added peer, starting replication: peer=server-44db6509-c2cc-56cf-9d4b-a7b59633c136 2025-09-14T21:47:48.633Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: added peer, starting replication: peer=server-a45966cf-4970-6e3e-0762-a695f6366ab8 2025-09-14T21:47:48.633Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: pipelining replication: peer="{Voter server-a45966cf-4970-6e3e-0762-a695f6366ab8 a45966cf-4970-6e3e-0762-a695f6366ab8}" 2025-09-14T21:47:48.633Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: pipelining replication: peer="{Voter server-44db6509-c2cc-56cf-9d4b-a7b59633c136 44db6509-c2cc-56cf-9d4b-a7b59633c136}" 2025-09-14T21:47:49.639Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:48.633183167 +0000 UTC m=+39.201061444 (1 nodes), 593.485298ms from start of poll, 593.78372ms from cluster start. Timeout at 2025-09-14 21:47:49.63818391 +0000 UTC m=+40.206062173, 1.005000729s after stability 2025-09-14T21:47:49.639Z [INFO] creating new snapshot: path=/tmp/raft1929690746/snapshots/2-14-1757886469639.tmp 2025-09-14T21:47:49.639Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: copied to local snapshot: bytes=71 2025-09-14T21:47:49.640Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: snapshot restore progress: id=2-14-1757886469639 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:49.640Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: removing all old logs from log store 2025-09-14T21:47:49.640Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: compacting logs: from=1 to=2 2025-09-14T21:47:49.640Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: restored user snapshot: index=14 2025-09-14T21:47:49.640Z [ERROR] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: failed to get log: index=2 error="log not found" 2025-09-14T21:47:49.640Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: aborting pipeline replication: peer="{Voter server-a45966cf-4970-6e3e-0762-a695f6366ab8 a45966cf-4970-6e3e-0762-a695f6366ab8}" 2025-09-14T21:47:49.640Z [ERROR] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: failed to get log: index=2 error="log not found" 2025-09-14T21:47:49.640Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: aborting pipeline replication: peer="{Voter server-44db6509-c2cc-56cf-9d4b-a7b59633c136 44db6509-c2cc-56cf-9d4b-a7b59633c136}" 2025-09-14T21:47:49.645Z [ERROR] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: failed to get log: index=2 error="log not found" 2025-09-14T21:47:49.646Z [INFO] creating new snapshot: path=/tmp/raft3076804189/snapshots/2-14-1757886469646.tmp 2025-09-14T21:47:49.646Z [INFO] server-44db6509-c2cc-56cf-9d4b-a7b59633c136: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:49.646Z [INFO] server-44db6509-c2cc-56cf-9d4b-a7b59633c136: copied to local snapshot: bytes=71 2025-09-14T21:47:49.646Z [INFO] server-44db6509-c2cc-56cf-9d4b-a7b59633c136: snapshot restore progress: id=2-14-1757886469646 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:49.646Z [INFO] server-44db6509-c2cc-56cf-9d4b-a7b59633c136: removing all old logs from log store 2025-09-14T21:47:49.646Z [INFO] server-44db6509-c2cc-56cf-9d4b-a7b59633c136: compacting logs: from=1 to=2 2025-09-14T21:47:49.646Z [INFO] server-44db6509-c2cc-56cf-9d4b-a7b59633c136: Installed remote snapshot 2025-09-14T21:47:49.646Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: pipelining replication: peer="{Voter server-44db6509-c2cc-56cf-9d4b-a7b59633c136 44db6509-c2cc-56cf-9d4b-a7b59633c136}" 2025-09-14T21:47:49.646Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:49.647Z [ERROR] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: failed to get log: index=2 error="log not found" 2025-09-14T21:47:49.648Z [INFO] creating new snapshot: path=/tmp/raft1524987130/snapshots/2-14-1757886469648.tmp 2025-09-14T21:47:49.648Z [INFO] server-a45966cf-4970-6e3e-0762-a695f6366ab8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:49.648Z [INFO] server-a45966cf-4970-6e3e-0762-a695f6366ab8: copied to local snapshot: bytes=71 2025-09-14T21:47:49.648Z [INFO] server-a45966cf-4970-6e3e-0762-a695f6366ab8: snapshot restore progress: id=2-14-1757886469648 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:49.648Z [INFO] server-a45966cf-4970-6e3e-0762-a695f6366ab8: removing all old logs from log store 2025-09-14T21:47:49.648Z [INFO] server-a45966cf-4970-6e3e-0762-a695f6366ab8: compacting logs: from=1 to=2 2025-09-14T21:47:49.648Z [INFO] server-a45966cf-4970-6e3e-0762-a695f6366ab8: Installed remote snapshot 2025-09-14T21:47:49.648Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: pipelining replication: peer="{Voter server-a45966cf-4970-6e3e-0762-a695f6366ab8 a45966cf-4970-6e3e-0762-a695f6366ab8}" 2025-09-14T21:47:50.652Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:49.646884064 +0000 UTC m=+40.214762323 (2 nodes), 704ns from start of poll, 1.607484599s from cluster start. Timeout at 2025-09-14 21:47:50.651891832 +0000 UTC m=+41.219770121, 1.005007798s after stability 2025-09-14T21:47:50.663Z [INFO] server-88fcc125-0bf1-c090-397b-1f999cacbaa4: aborting pipeline replication: peer="{Voter server-a140599f-22cc-5e1c-776f-7a7d6a002eee a140599f-22cc-5e1c-776f-7a7d6a002eee}" 2025-09-14T21:47:50.663Z [INFO] server-88fcc125-0bf1-c090-397b-1f999cacbaa4: aborting pipeline replication: peer="{Voter server-f1289554-89fb-002e-1a04-8c38b166dfd3 f1289554-89fb-002e-1a04-8c38b166dfd3}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false 2025-09-14T21:47:50.665Z [INFO] server-195f6f77-9758-58e4-6784-89911490a7f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-195f6f77-9758-58e4-6784-89911490a7f2 Address:195f6f77-9758-58e4-6784-89911490a7f2} {Suffrage:Voter ID:server-665075fb-3d36-0140-082a-2f1d1ad2f4bd Address:665075fb-3d36-0140-082a-2f1d1ad2f4bd} {Suffrage:Voter ID:server-4dfd80a3-ab29-459b-20d4-52af32963b57 Address:4dfd80a3-ab29-459b-20d4-52af32963b57}]" 2025-09-14T21:47:50.665Z [INFO] server-195f6f77-9758-58e4-6784-89911490a7f2: entering follower state: follower="Node at 195f6f77-9758-58e4-6784-89911490a7f2 [Follower]" leader-address= leader-id= 2025-09-14T21:47:50.665Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-195f6f77-9758-58e4-6784-89911490a7f2 Address:195f6f77-9758-58e4-6784-89911490a7f2} {Suffrage:Voter ID:server-665075fb-3d36-0140-082a-2f1d1ad2f4bd Address:665075fb-3d36-0140-082a-2f1d1ad2f4bd} {Suffrage:Voter ID:server-4dfd80a3-ab29-459b-20d4-52af32963b57 Address:4dfd80a3-ab29-459b-20d4-52af32963b57}]" 2025-09-14T21:47:50.665Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: entering follower state: follower="Node at 665075fb-3d36-0140-082a-2f1d1ad2f4bd [Follower]" leader-address= leader-id= 2025-09-14T21:47:50.665Z [INFO] server-4dfd80a3-ab29-459b-20d4-52af32963b57: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-195f6f77-9758-58e4-6784-89911490a7f2 Address:195f6f77-9758-58e4-6784-89911490a7f2} {Suffrage:Voter ID:server-665075fb-3d36-0140-082a-2f1d1ad2f4bd Address:665075fb-3d36-0140-082a-2f1d1ad2f4bd} {Suffrage:Voter ID:server-4dfd80a3-ab29-459b-20d4-52af32963b57 Address:4dfd80a3-ab29-459b-20d4-52af32963b57}]" 2025-09-14T21:47:50.665Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:50.665Z [INFO] server-4dfd80a3-ab29-459b-20d4-52af32963b57: entering follower state: follower="Node at 4dfd80a3-ab29-459b-20d4-52af32963b57 [Follower]" leader-address= leader-id= 2025-09-14T21:47:51.270Z [WARN] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:51.270Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: entering candidate state: node="Node at 665075fb-3d36-0140-082a-2f1d1ad2f4bd [Candidate]" term=2 2025-09-14T21:47:51.270Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: election won: term=2 tally=2 2025-09-14T21:47:51.270Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: entering leader state: leader="Node at 665075fb-3d36-0140-082a-2f1d1ad2f4bd [Leader]" 2025-09-14T21:47:51.270Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: added peer, starting replication: peer=server-195f6f77-9758-58e4-6784-89911490a7f2 2025-09-14T21:47:51.270Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: added peer, starting replication: peer=server-4dfd80a3-ab29-459b-20d4-52af32963b57 2025-09-14T21:47:51.270Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: pipelining replication: peer="{Voter server-195f6f77-9758-58e4-6784-89911490a7f2 195f6f77-9758-58e4-6784-89911490a7f2}" 2025-09-14T21:47:51.270Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: pipelining replication: peer="{Voter server-4dfd80a3-ab29-459b-20d4-52af32963b57 4dfd80a3-ab29-459b-20d4-52af32963b57}" 2025-09-14T21:47:52.275Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:51.270284931 +0000 UTC m=+41.838163191 (1 nodes), 604.820522ms from start of poll, 605.23076ms from cluster start. Timeout at 2025-09-14 21:47:52.275287295 +0000 UTC m=+42.843165557, 1.005002366s after stability 2025-09-14T21:47:52.275Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: starting snapshot up to: index=12 2025-09-14T21:47:52.275Z [INFO] creating new snapshot: path=/tmp/raft1329645071/snapshots/2-12-1757886472275.tmp 2025-09-14T21:47:52.276Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: snapshot complete up to: index=12 2025-09-14T21:47:52.276Z [INFO] creating new snapshot: path=/tmp/raft1329645071/snapshots/2-23-1757886472276.tmp 2025-09-14T21:47:52.277Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: copied to local snapshot: bytes=71 2025-09-14T21:47:52.277Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: snapshot restore progress: id=2-23-1757886472276 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:52.277Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: restored user snapshot: index=23 2025-09-14T21:47:52.277Z [ERROR] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: failed to get log: index=23 error="log not found" 2025-09-14T21:47:52.277Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: aborting pipeline replication: peer="{Voter server-4dfd80a3-ab29-459b-20d4-52af32963b57 4dfd80a3-ab29-459b-20d4-52af32963b57}" 2025-09-14T21:47:52.277Z [ERROR] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: failed to get log: index=23 error="log not found" 2025-09-14T21:47:52.277Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: aborting pipeline replication: peer="{Voter server-195f6f77-9758-58e4-6784-89911490a7f2 195f6f77-9758-58e4-6784-89911490a7f2}" 2025-09-14T21:47:52.284Z [ERROR] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: failed to get log: index=23 error="log not found" 2025-09-14T21:47:52.284Z [INFO] creating new snapshot: path=/tmp/raft1059971454/snapshots/2-23-1757886472284.tmp 2025-09-14T21:47:52.285Z [ERROR] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: failed to get log: index=23 error="log not found" 2025-09-14T21:47:52.285Z [INFO] server-4dfd80a3-ab29-459b-20d4-52af32963b57: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:52.285Z [INFO] server-4dfd80a3-ab29-459b-20d4-52af32963b57: copied to local snapshot: bytes=71 2025-09-14T21:47:52.285Z [INFO] creating new snapshot: path=/tmp/raft1145057729/snapshots/2-23-1757886472285.tmp 2025-09-14T21:47:52.285Z [INFO] server-4dfd80a3-ab29-459b-20d4-52af32963b57: snapshot restore progress: id=2-23-1757886472284 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:52.285Z [INFO] server-4dfd80a3-ab29-459b-20d4-52af32963b57: Installed remote snapshot 2025-09-14T21:47:52.285Z [WARN] server-4dfd80a3-ab29-459b-20d4-52af32963b57: skipping application of old log: index=22 2025-09-14T21:47:52.285Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: pipelining replication: peer="{Voter server-4dfd80a3-ab29-459b-20d4-52af32963b57 4dfd80a3-ab29-459b-20d4-52af32963b57}" 2025-09-14T21:47:52.285Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:52.285Z [INFO] server-195f6f77-9758-58e4-6784-89911490a7f2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:52.286Z [INFO] server-195f6f77-9758-58e4-6784-89911490a7f2: copied to local snapshot: bytes=71 2025-09-14T21:47:52.286Z [INFO] server-195f6f77-9758-58e4-6784-89911490a7f2: snapshot restore progress: id=2-23-1757886472285 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:52.286Z [INFO] server-195f6f77-9758-58e4-6784-89911490a7f2: Installed remote snapshot 2025-09-14T21:47:52.286Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: pipelining replication: peer="{Voter server-195f6f77-9758-58e4-6784-89911490a7f2 195f6f77-9758-58e4-6784-89911490a7f2}" 2025-09-14T21:47:53.291Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:52.285803138 +0000 UTC m=+42.853681405 (2 nodes), 926ns from start of poll, 1.620748974s from cluster start. Timeout at 2025-09-14 21:47:53.290814527 +0000 UTC m=+43.858692801, 1.005011396s after stability 2025-09-14T21:47:53.302Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: aborting pipeline replication: peer="{Voter server-4dfd80a3-ab29-459b-20d4-52af32963b57 4dfd80a3-ab29-459b-20d4-52af32963b57}" 2025-09-14T21:47:53.302Z [INFO] server-665075fb-3d36-0140-082a-2f1d1ad2f4bd: aborting pipeline replication: peer="{Voter server-195f6f77-9758-58e4-6784-89911490a7f2 195f6f77-9758-58e4-6784-89911490a7f2}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false 2025-09-14T21:47:53.304Z [INFO] server-6ff20800-cf66-b3fc-fc9b-28f8eebc708f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ff20800-cf66-b3fc-fc9b-28f8eebc708f Address:6ff20800-cf66-b3fc-fc9b-28f8eebc708f} {Suffrage:Voter ID:server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0 Address:5b1a278d-f4f8-0fe3-62b7-9f955204d2a0} {Suffrage:Voter ID:server-1fd94ef7-220a-9ca6-6122-344a2f54a697 Address:1fd94ef7-220a-9ca6-6122-344a2f54a697}]" 2025-09-14T21:47:53.304Z [INFO] server-6ff20800-cf66-b3fc-fc9b-28f8eebc708f: entering follower state: follower="Node at 6ff20800-cf66-b3fc-fc9b-28f8eebc708f [Follower]" leader-address= leader-id= 2025-09-14T21:47:53.304Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ff20800-cf66-b3fc-fc9b-28f8eebc708f Address:6ff20800-cf66-b3fc-fc9b-28f8eebc708f} {Suffrage:Voter ID:server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0 Address:5b1a278d-f4f8-0fe3-62b7-9f955204d2a0} {Suffrage:Voter ID:server-1fd94ef7-220a-9ca6-6122-344a2f54a697 Address:1fd94ef7-220a-9ca6-6122-344a2f54a697}]" 2025-09-14T21:47:53.304Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: entering follower state: follower="Node at 5b1a278d-f4f8-0fe3-62b7-9f955204d2a0 [Follower]" leader-address= leader-id= 2025-09-14T21:47:53.304Z [INFO] server-1fd94ef7-220a-9ca6-6122-344a2f54a697: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ff20800-cf66-b3fc-fc9b-28f8eebc708f Address:6ff20800-cf66-b3fc-fc9b-28f8eebc708f} {Suffrage:Voter ID:server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0 Address:5b1a278d-f4f8-0fe3-62b7-9f955204d2a0} {Suffrage:Voter ID:server-1fd94ef7-220a-9ca6-6122-344a2f54a697 Address:1fd94ef7-220a-9ca6-6122-344a2f54a697}]" 2025-09-14T21:47:53.304Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:53.305Z [INFO] server-1fd94ef7-220a-9ca6-6122-344a2f54a697: entering follower state: follower="Node at 1fd94ef7-220a-9ca6-6122-344a2f54a697 [Follower]" leader-address= leader-id= 2025-09-14T21:47:53.914Z [WARN] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:53.914Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: entering candidate state: node="Node at 5b1a278d-f4f8-0fe3-62b7-9f955204d2a0 [Candidate]" term=2 2025-09-14T21:47:53.914Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: election won: term=2 tally=2 2025-09-14T21:47:53.914Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: entering leader state: leader="Node at 5b1a278d-f4f8-0fe3-62b7-9f955204d2a0 [Leader]" 2025-09-14T21:47:53.914Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: added peer, starting replication: peer=server-6ff20800-cf66-b3fc-fc9b-28f8eebc708f 2025-09-14T21:47:53.914Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: added peer, starting replication: peer=server-1fd94ef7-220a-9ca6-6122-344a2f54a697 2025-09-14T21:47:53.914Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: pipelining replication: peer="{Voter server-1fd94ef7-220a-9ca6-6122-344a2f54a697 1fd94ef7-220a-9ca6-6122-344a2f54a697}" 2025-09-14T21:47:53.914Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: pipelining replication: peer="{Voter server-6ff20800-cf66-b3fc-fc9b-28f8eebc708f 6ff20800-cf66-b3fc-fc9b-28f8eebc708f}" 2025-09-14T21:47:54.920Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:53.914771681 +0000 UTC m=+44.482649944 (1 nodes), 609.76887ms from start of poll, 610.237967ms from cluster start. Timeout at 2025-09-14 21:47:54.919772732 +0000 UTC m=+45.487650990, 1.005001046s after stability 2025-09-14T21:47:54.920Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: starting snapshot up to: index=12 2025-09-14T21:47:54.920Z [INFO] creating new snapshot: path=/tmp/raft2216955410/snapshots/2-12-1757886474920.tmp 2025-09-14T21:47:54.943Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: snapshot complete up to: index=12 2025-09-14T21:47:54.944Z [INFO] creating new snapshot: path=/tmp/raft2216955410/snapshots/2-23-1757886474944.tmp 2025-09-14T21:47:55.029Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: copied to local snapshot: bytes=71 2025-09-14T21:47:55.030Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: snapshot restore progress: id=2-23-1757886474944 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:55.031Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: removing all old logs from log store 2025-09-14T21:47:55.031Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: compacting logs: from=1 to=22 2025-09-14T21:47:55.031Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: restored user snapshot: index=23 2025-09-14T21:47:55.031Z [ERROR] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: failed to get log: index=22 error="log not found" 2025-09-14T21:47:55.031Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: aborting pipeline replication: peer="{Voter server-6ff20800-cf66-b3fc-fc9b-28f8eebc708f 6ff20800-cf66-b3fc-fc9b-28f8eebc708f}" 2025-09-14T21:47:55.031Z [ERROR] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: failed to get log: index=22 error="log not found" 2025-09-14T21:47:55.031Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: aborting pipeline replication: peer="{Voter server-1fd94ef7-220a-9ca6-6122-344a2f54a697 1fd94ef7-220a-9ca6-6122-344a2f54a697}" 2025-09-14T21:47:55.040Z [ERROR] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: failed to get log: index=22 error="log not found" 2025-09-14T21:47:55.040Z [INFO] creating new snapshot: path=/tmp/raft4020023248/snapshots/2-23-1757886475040.tmp 2025-09-14T21:47:55.040Z [INFO] server-6ff20800-cf66-b3fc-fc9b-28f8eebc708f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:55.041Z [ERROR] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: failed to get log: index=22 error="log not found" 2025-09-14T21:47:55.041Z [INFO] creating new snapshot: path=/tmp/raft2462869972/snapshots/2-23-1757886475041.tmp 2025-09-14T21:47:55.041Z [INFO] server-1fd94ef7-220a-9ca6-6122-344a2f54a697: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:55.106Z [INFO] server-6ff20800-cf66-b3fc-fc9b-28f8eebc708f: copied to local snapshot: bytes=71 2025-09-14T21:47:55.106Z [INFO] server-1fd94ef7-220a-9ca6-6122-344a2f54a697: copied to local snapshot: bytes=71 2025-09-14T21:47:55.107Z [INFO] server-6ff20800-cf66-b3fc-fc9b-28f8eebc708f: snapshot restore progress: id=2-23-1757886475040 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:55.107Z [INFO] server-6ff20800-cf66-b3fc-fc9b-28f8eebc708f: removing all old logs from log store 2025-09-14T21:47:55.107Z [INFO] server-6ff20800-cf66-b3fc-fc9b-28f8eebc708f: compacting logs: from=1 to=22 2025-09-14T21:47:55.107Z [INFO] server-6ff20800-cf66-b3fc-fc9b-28f8eebc708f: Installed remote snapshot 2025-09-14T21:47:55.107Z [INFO] server-1fd94ef7-220a-9ca6-6122-344a2f54a697: snapshot restore progress: id=2-23-1757886475041 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:55.107Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: pipelining replication: peer="{Voter server-6ff20800-cf66-b3fc-fc9b-28f8eebc708f 6ff20800-cf66-b3fc-fc9b-28f8eebc708f}" 2025-09-14T21:47:55.107Z [INFO] server-1fd94ef7-220a-9ca6-6122-344a2f54a697: removing all old logs from log store 2025-09-14T21:47:55.107Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:47:55.107Z [INFO] server-1fd94ef7-220a-9ca6-6122-344a2f54a697: compacting logs: from=1 to=22 2025-09-14T21:47:55.107Z [INFO] server-1fd94ef7-220a-9ca6-6122-344a2f54a697: Installed remote snapshot 2025-09-14T21:47:55.107Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: pipelining replication: peer="{Voter server-1fd94ef7-220a-9ca6-6122-344a2f54a697 1fd94ef7-220a-9ca6-6122-344a2f54a697}" 2025-09-14T21:47:56.112Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:55.10724846 +0000 UTC m=+45.675126727 (2 nodes), 1.003µs from start of poll, 1.80271475s from cluster start. Timeout at 2025-09-14 21:47:56.112253616 +0000 UTC m=+46.680131890, 1.005005163s after stability 2025-09-14T21:47:56.124Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: aborting pipeline replication: peer="{Voter server-6ff20800-cf66-b3fc-fc9b-28f8eebc708f 6ff20800-cf66-b3fc-fc9b-28f8eebc708f}" 2025-09-14T21:47:56.124Z [INFO] server-5b1a278d-f4f8-0fe3-62b7-9f955204d2a0: aborting pipeline replication: peer="{Voter server-1fd94ef7-220a-9ca6-6122-344a2f54a697 1fd94ef7-220a-9ca6-6122-344a2f54a697}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true 2025-09-14T21:47:56.125Z [INFO] server-d5a7cccb-fd3b-7da8-8d29-6fe2f001b3a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5a7cccb-fd3b-7da8-8d29-6fe2f001b3a2 Address:d5a7cccb-fd3b-7da8-8d29-6fe2f001b3a2} {Suffrage:Voter ID:server-c204b0b2-f32f-9b44-7a96-765f4e2b8615 Address:c204b0b2-f32f-9b44-7a96-765f4e2b8615} {Suffrage:Voter ID:server-4b2a6508-8035-b4ce-17b2-e8554e09e0b0 Address:4b2a6508-8035-b4ce-17b2-e8554e09e0b0}]" 2025-09-14T21:47:56.125Z [INFO] server-d5a7cccb-fd3b-7da8-8d29-6fe2f001b3a2: entering follower state: follower="Node at d5a7cccb-fd3b-7da8-8d29-6fe2f001b3a2 [Follower]" leader-address= leader-id= 2025-09-14T21:47:56.125Z [INFO] server-c204b0b2-f32f-9b44-7a96-765f4e2b8615: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5a7cccb-fd3b-7da8-8d29-6fe2f001b3a2 Address:d5a7cccb-fd3b-7da8-8d29-6fe2f001b3a2} {Suffrage:Voter ID:server-c204b0b2-f32f-9b44-7a96-765f4e2b8615 Address:c204b0b2-f32f-9b44-7a96-765f4e2b8615} {Suffrage:Voter ID:server-4b2a6508-8035-b4ce-17b2-e8554e09e0b0 Address:4b2a6508-8035-b4ce-17b2-e8554e09e0b0}]" 2025-09-14T21:47:56.125Z [INFO] server-c204b0b2-f32f-9b44-7a96-765f4e2b8615: entering follower state: follower="Node at c204b0b2-f32f-9b44-7a96-765f4e2b8615 [Follower]" leader-address= leader-id= 2025-09-14T21:47:56.125Z [INFO] server-4b2a6508-8035-b4ce-17b2-e8554e09e0b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5a7cccb-fd3b-7da8-8d29-6fe2f001b3a2 Address:d5a7cccb-fd3b-7da8-8d29-6fe2f001b3a2} {Suffrage:Voter ID:server-c204b0b2-f32f-9b44-7a96-765f4e2b8615 Address:c204b0b2-f32f-9b44-7a96-765f4e2b8615} {Suffrage:Voter ID:server-4b2a6508-8035-b4ce-17b2-e8554e09e0b0 Address:4b2a6508-8035-b4ce-17b2-e8554e09e0b0}]" 2025-09-14T21:47:56.125Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:56.125Z [INFO] server-4b2a6508-8035-b4ce-17b2-e8554e09e0b0: entering follower state: follower="Node at 4b2a6508-8035-b4ce-17b2-e8554e09e0b0 [Follower]" leader-address= leader-id= 2025-09-14T21:47:56.632Z [WARN] server-c204b0b2-f32f-9b44-7a96-765f4e2b8615: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:56.632Z [INFO] server-c204b0b2-f32f-9b44-7a96-765f4e2b8615: entering candidate state: node="Node at c204b0b2-f32f-9b44-7a96-765f4e2b8615 [Candidate]" term=2 2025-09-14T21:47:56.632Z [INFO] server-c204b0b2-f32f-9b44-7a96-765f4e2b8615: election won: term=2 tally=2 2025-09-14T21:47:56.632Z [INFO] server-c204b0b2-f32f-9b44-7a96-765f4e2b8615: entering leader state: leader="Node at c204b0b2-f32f-9b44-7a96-765f4e2b8615 [Leader]" 2025-09-14T21:47:56.632Z [INFO] server-c204b0b2-f32f-9b44-7a96-765f4e2b8615: added peer, starting replication: peer=server-d5a7cccb-fd3b-7da8-8d29-6fe2f001b3a2 2025-09-14T21:47:56.632Z [INFO] server-c204b0b2-f32f-9b44-7a96-765f4e2b8615: added peer, starting replication: peer=server-4b2a6508-8035-b4ce-17b2-e8554e09e0b0 2025-09-14T21:47:56.632Z [INFO] server-c204b0b2-f32f-9b44-7a96-765f4e2b8615: pipelining replication: peer="{Voter server-4b2a6508-8035-b4ce-17b2-e8554e09e0b0 4b2a6508-8035-b4ce-17b2-e8554e09e0b0}" 2025-09-14T21:47:56.632Z [INFO] server-c204b0b2-f32f-9b44-7a96-765f4e2b8615: pipelining replication: peer="{Voter server-d5a7cccb-fd3b-7da8-8d29-6fe2f001b3a2 d5a7cccb-fd3b-7da8-8d29-6fe2f001b3a2}" 2025-09-14T21:47:57.637Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:56.63275941 +0000 UTC m=+47.200637682 (1 nodes), 506.994676ms from start of poll, 507.359424ms from cluster start. Timeout at 2025-09-14 21:47:57.637761796 +0000 UTC m=+48.205640071, 1.005002389s after stability 2025-09-14T21:47:57.638Z [INFO] server-c204b0b2-f32f-9b44-7a96-765f4e2b8615: starting snapshot up to: index=12 2025-09-14T21:47:57.638Z [INFO] creating new snapshot: path=/tmp/raft3686180611/snapshots/2-12-1757886477638.tmp 2025-09-14T21:47:57.688Z [INFO] server-c204b0b2-f32f-9b44-7a96-765f4e2b8615: snapshot complete up to: index=12 2025-09-14T21:47:57.689Z [INFO] server-4a406d95-6e56-e981-2e80-6c0b7ec318b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a406d95-6e56-e981-2e80-6c0b7ec318b5 Address:4a406d95-6e56-e981-2e80-6c0b7ec318b5} {Suffrage:Voter ID:server-eddd436f-b70e-2d91-413d-42509e6fafe2 Address:eddd436f-b70e-2d91-413d-42509e6fafe2} {Suffrage:Voter ID:server-d66acbe9-5343-4b20-8ad5-0c2578394362 Address:d66acbe9-5343-4b20-8ad5-0c2578394362}]" 2025-09-14T21:47:57.690Z [INFO] server-4a406d95-6e56-e981-2e80-6c0b7ec318b5: entering follower state: follower="Node at 4a406d95-6e56-e981-2e80-6c0b7ec318b5 [Follower]" leader-address= leader-id= 2025-09-14T21:47:57.690Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a406d95-6e56-e981-2e80-6c0b7ec318b5 Address:4a406d95-6e56-e981-2e80-6c0b7ec318b5} {Suffrage:Voter ID:server-eddd436f-b70e-2d91-413d-42509e6fafe2 Address:eddd436f-b70e-2d91-413d-42509e6fafe2} {Suffrage:Voter ID:server-d66acbe9-5343-4b20-8ad5-0c2578394362 Address:d66acbe9-5343-4b20-8ad5-0c2578394362}]" 2025-09-14T21:47:57.690Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: entering follower state: follower="Node at eddd436f-b70e-2d91-413d-42509e6fafe2 [Follower]" leader-address= leader-id= 2025-09-14T21:47:57.690Z [INFO] server-d66acbe9-5343-4b20-8ad5-0c2578394362: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a406d95-6e56-e981-2e80-6c0b7ec318b5 Address:4a406d95-6e56-e981-2e80-6c0b7ec318b5} {Suffrage:Voter ID:server-eddd436f-b70e-2d91-413d-42509e6fafe2 Address:eddd436f-b70e-2d91-413d-42509e6fafe2} {Suffrage:Voter ID:server-d66acbe9-5343-4b20-8ad5-0c2578394362 Address:d66acbe9-5343-4b20-8ad5-0c2578394362}]" 2025-09-14T21:47:57.690Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:47:57.690Z [INFO] server-d66acbe9-5343-4b20-8ad5-0c2578394362: entering follower state: follower="Node at d66acbe9-5343-4b20-8ad5-0c2578394362 [Follower]" leader-address= leader-id= 2025-09-14T21:47:58.590Z [WARN] server-eddd436f-b70e-2d91-413d-42509e6fafe2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:47:58.590Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: entering candidate state: node="Node at eddd436f-b70e-2d91-413d-42509e6fafe2 [Candidate]" term=2 2025-09-14T21:47:58.592Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: election won: term=2 tally=2 2025-09-14T21:47:58.592Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: entering leader state: leader="Node at eddd436f-b70e-2d91-413d-42509e6fafe2 [Leader]" 2025-09-14T21:47:58.592Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: added peer, starting replication: peer=server-4a406d95-6e56-e981-2e80-6c0b7ec318b5 2025-09-14T21:47:58.592Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: added peer, starting replication: peer=server-d66acbe9-5343-4b20-8ad5-0c2578394362 2025-09-14T21:47:58.594Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: pipelining replication: peer="{Voter server-4a406d95-6e56-e981-2e80-6c0b7ec318b5 4a406d95-6e56-e981-2e80-6c0b7ec318b5}" 2025-09-14T21:47:58.594Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: pipelining replication: peer="{Voter server-d66acbe9-5343-4b20-8ad5-0c2578394362 d66acbe9-5343-4b20-8ad5-0c2578394362}" 2025-09-14T21:47:59.598Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:47:58.592786812 +0000 UTC m=+49.160665088 (1 nodes), 902.574416ms from start of poll, 902.898504ms from cluster start. Timeout at 2025-09-14 21:47:59.597795776 +0000 UTC m=+50.165674050, 1.005008962s after stability 2025-09-14T21:47:59.598Z [INFO] creating new snapshot: path=/tmp/raft3806718845/snapshots/2-15-1757886479598.tmp 2025-09-14T21:47:59.599Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: copied to local snapshot: bytes=71 2025-09-14T21:47:59.599Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: snapshot restore progress: id=2-15-1757886479598 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:59.599Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: restored user snapshot: index=15 2025-09-14T21:47:59.599Z [ERROR] server-eddd436f-b70e-2d91-413d-42509e6fafe2: failed to get log: index=3 error="log not found" 2025-09-14T21:47:59.599Z [ERROR] server-eddd436f-b70e-2d91-413d-42509e6fafe2: failed to get log: index=3 error="log not found" 2025-09-14T21:47:59.599Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: aborting pipeline replication: peer="{Voter server-4a406d95-6e56-e981-2e80-6c0b7ec318b5 4a406d95-6e56-e981-2e80-6c0b7ec318b5}" 2025-09-14T21:47:59.599Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: aborting pipeline replication: peer="{Voter server-d66acbe9-5343-4b20-8ad5-0c2578394362 d66acbe9-5343-4b20-8ad5-0c2578394362}" 2025-09-14T21:47:59.607Z [ERROR] server-eddd436f-b70e-2d91-413d-42509e6fafe2: failed to get log: index=3 error="log not found" 2025-09-14T21:47:59.607Z [ERROR] server-eddd436f-b70e-2d91-413d-42509e6fafe2: failed to get log: index=3 error="log not found" 2025-09-14T21:47:59.607Z [INFO] creating new snapshot: path=/tmp/raft3333033032/snapshots/2-15-1757886479607.tmp 2025-09-14T21:47:59.608Z [INFO] creating new snapshot: path=/tmp/raft1719924851/snapshots/2-15-1757886479608.tmp 2025-09-14T21:47:59.608Z [INFO] server-d66acbe9-5343-4b20-8ad5-0c2578394362: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:59.608Z [INFO] server-4a406d95-6e56-e981-2e80-6c0b7ec318b5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:59.735Z [INFO] server-4a406d95-6e56-e981-2e80-6c0b7ec318b5: copied to local snapshot: bytes=71 2025-09-14T21:47:59.735Z [INFO] server-d66acbe9-5343-4b20-8ad5-0c2578394362: copied to local snapshot: bytes=71 2025-09-14T21:47:59.735Z [INFO] server-4a406d95-6e56-e981-2e80-6c0b7ec318b5: snapshot restore progress: id=2-15-1757886479607 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:59.735Z [INFO] server-d66acbe9-5343-4b20-8ad5-0c2578394362: snapshot restore progress: id=2-15-1757886479608 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:47:59.735Z [INFO] server-d66acbe9-5343-4b20-8ad5-0c2578394362: Installed remote snapshot 2025-09-14T21:47:59.735Z [INFO] server-4a406d95-6e56-e981-2e80-6c0b7ec318b5: Installed remote snapshot 2025-09-14T21:47:59.735Z [WARN] server-4a406d95-6e56-e981-2e80-6c0b7ec318b5: skipping application of old log: index=15 2025-09-14T21:47:59.735Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: pipelining replication: peer="{Voter server-d66acbe9-5343-4b20-8ad5-0c2578394362 d66acbe9-5343-4b20-8ad5-0c2578394362}" 2025-09-14T21:47:59.735Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: pipelining replication: peer="{Voter server-4a406d95-6e56-e981-2e80-6c0b7ec318b5 4a406d95-6e56-e981-2e80-6c0b7ec318b5}" 2025-09-14T21:47:59.735Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:00.740Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:47:59.735900339 +0000 UTC m=+50.303778620 (2 nodes), 1.102µs from start of poll, 2.046012036s from cluster start. Timeout at 2025-09-14 21:48:00.740912453 +0000 UTC m=+51.308790749, 1.005012129s after stability 2025-09-14T21:48:00.752Z [INFO] server-c204b0b2-f32f-9b44-7a96-765f4e2b8615: aborting pipeline replication: peer="{Voter server-4b2a6508-8035-b4ce-17b2-e8554e09e0b0 4b2a6508-8035-b4ce-17b2-e8554e09e0b0}" 2025-09-14T21:48:00.752Z [INFO] server-c204b0b2-f32f-9b44-7a96-765f4e2b8615: aborting pipeline replication: peer="{Voter server-d5a7cccb-fd3b-7da8-8d29-6fe2f001b3a2 d5a7cccb-fd3b-7da8-8d29-6fe2f001b3a2}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true 2025-09-14T21:48:00.753Z [INFO] server-1a7a4ff6-babf-1dc1-10ec-d0779eed532c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a7a4ff6-babf-1dc1-10ec-d0779eed532c Address:1a7a4ff6-babf-1dc1-10ec-d0779eed532c} {Suffrage:Voter ID:server-2ea73cfc-db1c-eb1f-24f1-44b9005fd53b Address:2ea73cfc-db1c-eb1f-24f1-44b9005fd53b} {Suffrage:Voter ID:server-fcc58cde-7ffc-826b-76a6-c5672b3e175f Address:fcc58cde-7ffc-826b-76a6-c5672b3e175f}]" 2025-09-14T21:48:00.753Z [INFO] server-1a7a4ff6-babf-1dc1-10ec-d0779eed532c: entering follower state: follower="Node at 1a7a4ff6-babf-1dc1-10ec-d0779eed532c [Follower]" leader-address= leader-id= 2025-09-14T21:48:00.753Z [INFO] server-2ea73cfc-db1c-eb1f-24f1-44b9005fd53b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a7a4ff6-babf-1dc1-10ec-d0779eed532c Address:1a7a4ff6-babf-1dc1-10ec-d0779eed532c} {Suffrage:Voter ID:server-2ea73cfc-db1c-eb1f-24f1-44b9005fd53b Address:2ea73cfc-db1c-eb1f-24f1-44b9005fd53b} {Suffrage:Voter ID:server-fcc58cde-7ffc-826b-76a6-c5672b3e175f Address:fcc58cde-7ffc-826b-76a6-c5672b3e175f}]" 2025-09-14T21:48:00.753Z [INFO] server-2ea73cfc-db1c-eb1f-24f1-44b9005fd53b: entering follower state: follower="Node at 2ea73cfc-db1c-eb1f-24f1-44b9005fd53b [Follower]" leader-address= leader-id= 2025-09-14T21:48:00.753Z [INFO] server-fcc58cde-7ffc-826b-76a6-c5672b3e175f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a7a4ff6-babf-1dc1-10ec-d0779eed532c Address:1a7a4ff6-babf-1dc1-10ec-d0779eed532c} {Suffrage:Voter ID:server-2ea73cfc-db1c-eb1f-24f1-44b9005fd53b Address:2ea73cfc-db1c-eb1f-24f1-44b9005fd53b} {Suffrage:Voter ID:server-fcc58cde-7ffc-826b-76a6-c5672b3e175f Address:fcc58cde-7ffc-826b-76a6-c5672b3e175f}]" 2025-09-14T21:48:00.753Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:00.753Z [INFO] server-fcc58cde-7ffc-826b-76a6-c5672b3e175f: entering follower state: follower="Node at fcc58cde-7ffc-826b-76a6-c5672b3e175f [Follower]" leader-address= leader-id= 2025-09-14T21:48:01.268Z [WARN] server-1a7a4ff6-babf-1dc1-10ec-d0779eed532c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:01.269Z [INFO] server-1a7a4ff6-babf-1dc1-10ec-d0779eed532c: entering candidate state: node="Node at 1a7a4ff6-babf-1dc1-10ec-d0779eed532c [Candidate]" term=2 2025-09-14T21:48:01.269Z [INFO] server-1a7a4ff6-babf-1dc1-10ec-d0779eed532c: election won: term=2 tally=2 2025-09-14T21:48:01.269Z [INFO] server-1a7a4ff6-babf-1dc1-10ec-d0779eed532c: entering leader state: leader="Node at 1a7a4ff6-babf-1dc1-10ec-d0779eed532c [Leader]" 2025-09-14T21:48:01.269Z [INFO] server-1a7a4ff6-babf-1dc1-10ec-d0779eed532c: added peer, starting replication: peer=server-2ea73cfc-db1c-eb1f-24f1-44b9005fd53b 2025-09-14T21:48:01.269Z [INFO] server-1a7a4ff6-babf-1dc1-10ec-d0779eed532c: added peer, starting replication: peer=server-fcc58cde-7ffc-826b-76a6-c5672b3e175f 2025-09-14T21:48:01.269Z [INFO] server-1a7a4ff6-babf-1dc1-10ec-d0779eed532c: pipelining replication: peer="{Voter server-2ea73cfc-db1c-eb1f-24f1-44b9005fd53b 2ea73cfc-db1c-eb1f-24f1-44b9005fd53b}" 2025-09-14T21:48:01.269Z [INFO] server-1a7a4ff6-babf-1dc1-10ec-d0779eed532c: pipelining replication: peer="{Voter server-fcc58cde-7ffc-826b-76a6-c5672b3e175f fcc58cde-7ffc-826b-76a6-c5672b3e175f}" 2025-09-14T21:48:02.274Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:01.269265528 +0000 UTC m=+51.837143821 (1 nodes), 515.373233ms from start of poll, 515.796703ms from cluster start. Timeout at 2025-09-14 21:48:02.274269349 +0000 UTC m=+52.842147607, 1.005003786s after stability 2025-09-14T21:48:02.275Z [INFO] server-1a7a4ff6-babf-1dc1-10ec-d0779eed532c: starting snapshot up to: index=12 2025-09-14T21:48:02.275Z [INFO] creating new snapshot: path=/tmp/raft2042400414/snapshots/2-12-1757886482275.tmp 2025-09-14T21:48:02.400Z [INFO] server-1a7a4ff6-babf-1dc1-10ec-d0779eed532c: snapshot complete up to: index=12 2025-09-14T21:48:02.401Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3bf73f50-a68b-9cf9-d150-22c060633bc6 Address:3bf73f50-a68b-9cf9-d150-22c060633bc6} {Suffrage:Voter ID:server-e79c8b40-937c-f0ef-930b-1c668670f33e Address:e79c8b40-937c-f0ef-930b-1c668670f33e} {Suffrage:Voter ID:server-0aacb753-bf82-99e4-23f3-da017a22d60d Address:0aacb753-bf82-99e4-23f3-da017a22d60d}]" 2025-09-14T21:48:02.401Z [INFO] server-e79c8b40-937c-f0ef-930b-1c668670f33e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3bf73f50-a68b-9cf9-d150-22c060633bc6 Address:3bf73f50-a68b-9cf9-d150-22c060633bc6} {Suffrage:Voter ID:server-e79c8b40-937c-f0ef-930b-1c668670f33e Address:e79c8b40-937c-f0ef-930b-1c668670f33e} {Suffrage:Voter ID:server-0aacb753-bf82-99e4-23f3-da017a22d60d Address:0aacb753-bf82-99e4-23f3-da017a22d60d}]" 2025-09-14T21:48:02.401Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: entering follower state: follower="Node at 3bf73f50-a68b-9cf9-d150-22c060633bc6 [Follower]" leader-address= leader-id= 2025-09-14T21:48:02.401Z [INFO] server-e79c8b40-937c-f0ef-930b-1c668670f33e: entering follower state: follower="Node at e79c8b40-937c-f0ef-930b-1c668670f33e [Follower]" leader-address= leader-id= 2025-09-14T21:48:02.402Z [INFO] server-0aacb753-bf82-99e4-23f3-da017a22d60d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3bf73f50-a68b-9cf9-d150-22c060633bc6 Address:3bf73f50-a68b-9cf9-d150-22c060633bc6} {Suffrage:Voter ID:server-e79c8b40-937c-f0ef-930b-1c668670f33e Address:e79c8b40-937c-f0ef-930b-1c668670f33e} {Suffrage:Voter ID:server-0aacb753-bf82-99e4-23f3-da017a22d60d Address:0aacb753-bf82-99e4-23f3-da017a22d60d}]" 2025-09-14T21:48:02.402Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:02.402Z [INFO] server-0aacb753-bf82-99e4-23f3-da017a22d60d: entering follower state: follower="Node at 0aacb753-bf82-99e4-23f3-da017a22d60d [Follower]" leader-address= leader-id= 2025-09-14T21:48:02.916Z [WARN] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:02.916Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: entering candidate state: node="Node at 3bf73f50-a68b-9cf9-d150-22c060633bc6 [Candidate]" term=2 2025-09-14T21:48:02.916Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: election won: term=2 tally=2 2025-09-14T21:48:02.916Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: entering leader state: leader="Node at 3bf73f50-a68b-9cf9-d150-22c060633bc6 [Leader]" 2025-09-14T21:48:02.916Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: added peer, starting replication: peer=server-e79c8b40-937c-f0ef-930b-1c668670f33e 2025-09-14T21:48:02.916Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: added peer, starting replication: peer=server-0aacb753-bf82-99e4-23f3-da017a22d60d 2025-09-14T21:48:02.916Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: pipelining replication: peer="{Voter server-e79c8b40-937c-f0ef-930b-1c668670f33e e79c8b40-937c-f0ef-930b-1c668670f33e}" 2025-09-14T21:48:02.916Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: pipelining replication: peer="{Voter server-0aacb753-bf82-99e4-23f3-da017a22d60d 0aacb753-bf82-99e4-23f3-da017a22d60d}" 2025-09-14T21:48:03.921Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:02.916426248 +0000 UTC m=+53.484304523 (1 nodes), 514.350793ms from start of poll, 514.765769ms from cluster start. Timeout at 2025-09-14 21:48:03.921427428 +0000 UTC m=+54.489305692, 1.005001169s after stability 2025-09-14T21:48:03.922Z [INFO] creating new snapshot: path=/tmp/raft4040990141/snapshots/2-15-1757886483922.tmp 2025-09-14T21:48:03.958Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: copied to local snapshot: bytes=71 2025-09-14T21:48:03.958Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: snapshot restore progress: id=2-15-1757886483922 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:03.958Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: removing all old logs from log store 2025-09-14T21:48:03.958Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: compacting logs: from=1 to=2 2025-09-14T21:48:03.958Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: restored user snapshot: index=15 2025-09-14T21:48:03.959Z [ERROR] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: failed to get log: index=2 error="log not found" 2025-09-14T21:48:03.959Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: aborting pipeline replication: peer="{Voter server-0aacb753-bf82-99e4-23f3-da017a22d60d 0aacb753-bf82-99e4-23f3-da017a22d60d}" 2025-09-14T21:48:03.959Z [ERROR] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: failed to get log: index=2 error="log not found" 2025-09-14T21:48:03.959Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: aborting pipeline replication: peer="{Voter server-e79c8b40-937c-f0ef-930b-1c668670f33e e79c8b40-937c-f0ef-930b-1c668670f33e}" 2025-09-14T21:48:03.967Z [ERROR] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: failed to get log: index=2 error="log not found" 2025-09-14T21:48:03.967Z [INFO] creating new snapshot: path=/tmp/raft3236937951/snapshots/2-15-1757886483967.tmp 2025-09-14T21:48:03.967Z [INFO] server-e79c8b40-937c-f0ef-930b-1c668670f33e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:03.968Z [ERROR] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: failed to get log: index=2 error="log not found" 2025-09-14T21:48:03.968Z [INFO] creating new snapshot: path=/tmp/raft1458979249/snapshots/2-15-1757886483968.tmp 2025-09-14T21:48:03.968Z [INFO] server-0aacb753-bf82-99e4-23f3-da017a22d60d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:04.019Z [INFO] server-e79c8b40-937c-f0ef-930b-1c668670f33e: copied to local snapshot: bytes=71 2025-09-14T21:48:04.019Z [INFO] server-0aacb753-bf82-99e4-23f3-da017a22d60d: copied to local snapshot: bytes=71 2025-09-14T21:48:04.020Z [INFO] server-0aacb753-bf82-99e4-23f3-da017a22d60d: snapshot restore progress: id=2-15-1757886483968 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:04.020Z [INFO] server-0aacb753-bf82-99e4-23f3-da017a22d60d: removing all old logs from log store 2025-09-14T21:48:04.020Z [INFO] server-0aacb753-bf82-99e4-23f3-da017a22d60d: compacting logs: from=1 to=2 2025-09-14T21:48:04.020Z [INFO] server-0aacb753-bf82-99e4-23f3-da017a22d60d: Installed remote snapshot 2025-09-14T21:48:04.020Z [INFO] server-e79c8b40-937c-f0ef-930b-1c668670f33e: snapshot restore progress: id=2-15-1757886483967 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:04.020Z [INFO] server-e79c8b40-937c-f0ef-930b-1c668670f33e: removing all old logs from log store 2025-09-14T21:48:04.020Z [INFO] server-e79c8b40-937c-f0ef-930b-1c668670f33e: compacting logs: from=1 to=2 2025-09-14T21:48:04.020Z [INFO] server-e79c8b40-937c-f0ef-930b-1c668670f33e: Installed remote snapshot 2025-09-14T21:48:04.020Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: pipelining replication: peer="{Voter server-0aacb753-bf82-99e4-23f3-da017a22d60d 0aacb753-bf82-99e4-23f3-da017a22d60d}" 2025-09-14T21:48:04.020Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:04.020Z [INFO] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: pipelining replication: peer="{Voter server-e79c8b40-937c-f0ef-930b-1c668670f33e e79c8b40-937c-f0ef-930b-1c668670f33e}" 2025-09-14T21:48:05.025Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:04.020314364 +0000 UTC m=+54.588192642 (2 nodes), 850ns from start of poll, 1.618653888s from cluster start. Timeout at 2025-09-14 21:48:05.025326662 +0000 UTC m=+55.593204951, 1.005012309s after stability 2025-09-14T21:48:05.037Z [INFO] server-1a7a4ff6-babf-1dc1-10ec-d0779eed532c: aborting pipeline replication: peer="{Voter server-fcc58cde-7ffc-826b-76a6-c5672b3e175f fcc58cde-7ffc-826b-76a6-c5672b3e175f}" 2025-09-14T21:48:05.037Z [INFO] server-1a7a4ff6-babf-1dc1-10ec-d0779eed532c: aborting pipeline replication: peer="{Voter server-2ea73cfc-db1c-eb1f-24f1-44b9005fd53b 2ea73cfc-db1c-eb1f-24f1-44b9005fd53b}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false 2025-09-14T21:48:05.038Z [INFO] server-a8724f84-71ef-3e51-ea4d-1189508fc5f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8724f84-71ef-3e51-ea4d-1189508fc5f5 Address:a8724f84-71ef-3e51-ea4d-1189508fc5f5} {Suffrage:Voter ID:server-7d5fff56-0eb1-00d5-5b95-13c19bef1149 Address:7d5fff56-0eb1-00d5-5b95-13c19bef1149} {Suffrage:Voter ID:server-7509dcd1-2696-8b20-8178-4d3905988fe0 Address:7509dcd1-2696-8b20-8178-4d3905988fe0}]" 2025-09-14T21:48:05.038Z [INFO] server-a8724f84-71ef-3e51-ea4d-1189508fc5f5: entering follower state: follower="Node at a8724f84-71ef-3e51-ea4d-1189508fc5f5 [Follower]" leader-address= leader-id= 2025-09-14T21:48:05.038Z [INFO] server-7d5fff56-0eb1-00d5-5b95-13c19bef1149: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8724f84-71ef-3e51-ea4d-1189508fc5f5 Address:a8724f84-71ef-3e51-ea4d-1189508fc5f5} {Suffrage:Voter ID:server-7d5fff56-0eb1-00d5-5b95-13c19bef1149 Address:7d5fff56-0eb1-00d5-5b95-13c19bef1149} {Suffrage:Voter ID:server-7509dcd1-2696-8b20-8178-4d3905988fe0 Address:7509dcd1-2696-8b20-8178-4d3905988fe0}]" 2025-09-14T21:48:05.038Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8724f84-71ef-3e51-ea4d-1189508fc5f5 Address:a8724f84-71ef-3e51-ea4d-1189508fc5f5} {Suffrage:Voter ID:server-7d5fff56-0eb1-00d5-5b95-13c19bef1149 Address:7d5fff56-0eb1-00d5-5b95-13c19bef1149} {Suffrage:Voter ID:server-7509dcd1-2696-8b20-8178-4d3905988fe0 Address:7509dcd1-2696-8b20-8178-4d3905988fe0}]" 2025-09-14T21:48:05.038Z [INFO] server-7d5fff56-0eb1-00d5-5b95-13c19bef1149: entering follower state: follower="Node at 7d5fff56-0eb1-00d5-5b95-13c19bef1149 [Follower]" leader-address= leader-id= 2025-09-14T21:48:05.038Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:05.038Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: entering follower state: follower="Node at 7509dcd1-2696-8b20-8178-4d3905988fe0 [Follower]" leader-address= leader-id= 2025-09-14T21:48:05.699Z [WARN] server-7509dcd1-2696-8b20-8178-4d3905988fe0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:05.699Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: entering candidate state: node="Node at 7509dcd1-2696-8b20-8178-4d3905988fe0 [Candidate]" term=2 2025-09-14T21:48:05.699Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: election won: term=2 tally=2 2025-09-14T21:48:05.699Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: entering leader state: leader="Node at 7509dcd1-2696-8b20-8178-4d3905988fe0 [Leader]" 2025-09-14T21:48:05.699Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: added peer, starting replication: peer=server-a8724f84-71ef-3e51-ea4d-1189508fc5f5 2025-09-14T21:48:05.699Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: added peer, starting replication: peer=server-7d5fff56-0eb1-00d5-5b95-13c19bef1149 2025-09-14T21:48:05.700Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: pipelining replication: peer="{Voter server-a8724f84-71ef-3e51-ea4d-1189508fc5f5 a8724f84-71ef-3e51-ea4d-1189508fc5f5}" 2025-09-14T21:48:05.700Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: pipelining replication: peer="{Voter server-7d5fff56-0eb1-00d5-5b95-13c19bef1149 7d5fff56-0eb1-00d5-5b95-13c19bef1149}" 2025-09-14T21:48:06.705Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:05.70000464 +0000 UTC m=+56.267882901 (1 nodes), 661.106158ms from start of poll, 661.423086ms from cluster start. Timeout at 2025-09-14 21:48:06.70500931 +0000 UTC m=+57.272887569, 1.005004668s after stability 2025-09-14T21:48:06.705Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: starting snapshot up to: index=12 2025-09-14T21:48:06.705Z [INFO] creating new snapshot: path=/tmp/raft1248275639/snapshots/2-12-1757886486705.tmp 2025-09-14T21:48:06.706Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: snapshot complete up to: index=12 2025-09-14T21:48:06.706Z [INFO] creating new snapshot: path=/tmp/raft1248275639/snapshots/2-113-1757886486706.tmp 2025-09-14T21:48:06.707Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: copied to local snapshot: bytes=71 2025-09-14T21:48:06.707Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: snapshot restore progress: id=2-113-1757886486706 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:06.707Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: restored user snapshot: index=113 2025-09-14T21:48:06.707Z [ERROR] server-7509dcd1-2696-8b20-8178-4d3905988fe0: failed to get log: index=23 error="log not found" 2025-09-14T21:48:06.707Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: aborting pipeline replication: peer="{Voter server-7d5fff56-0eb1-00d5-5b95-13c19bef1149 7d5fff56-0eb1-00d5-5b95-13c19bef1149}" 2025-09-14T21:48:06.707Z [ERROR] server-7509dcd1-2696-8b20-8178-4d3905988fe0: failed to get log: index=23 error="log not found" 2025-09-14T21:48:06.707Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: aborting pipeline replication: peer="{Voter server-a8724f84-71ef-3e51-ea4d-1189508fc5f5 a8724f84-71ef-3e51-ea4d-1189508fc5f5}" 2025-09-14T21:48:06.715Z [ERROR] server-7509dcd1-2696-8b20-8178-4d3905988fe0: failed to get log: index=23 error="log not found" 2025-09-14T21:48:06.716Z [INFO] creating new snapshot: path=/tmp/raft1561763167/snapshots/2-113-1757886486716.tmp 2025-09-14T21:48:06.716Z [INFO] server-a8724f84-71ef-3e51-ea4d-1189508fc5f5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:06.716Z [ERROR] server-7509dcd1-2696-8b20-8178-4d3905988fe0: failed to get log: index=23 error="log not found" 2025-09-14T21:48:06.716Z [INFO] server-a8724f84-71ef-3e51-ea4d-1189508fc5f5: copied to local snapshot: bytes=71 2025-09-14T21:48:06.716Z [INFO] creating new snapshot: path=/tmp/raft4254543310/snapshots/2-113-1757886486716.tmp 2025-09-14T21:48:06.716Z [INFO] server-a8724f84-71ef-3e51-ea4d-1189508fc5f5: snapshot restore progress: id=2-113-1757886486716 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:06.717Z [INFO] server-7d5fff56-0eb1-00d5-5b95-13c19bef1149: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:06.717Z [INFO] server-a8724f84-71ef-3e51-ea4d-1189508fc5f5: Installed remote snapshot 2025-09-14T21:48:06.717Z [WARN] server-a8724f84-71ef-3e51-ea4d-1189508fc5f5: skipping application of old log: index=22 2025-09-14T21:48:06.717Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: pipelining replication: peer="{Voter server-a8724f84-71ef-3e51-ea4d-1189508fc5f5 a8724f84-71ef-3e51-ea4d-1189508fc5f5}" 2025-09-14T21:48:06.717Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:06.798Z [INFO] server-7d5fff56-0eb1-00d5-5b95-13c19bef1149: copied to local snapshot: bytes=71 2025-09-14T21:48:06.798Z [INFO] server-7d5fff56-0eb1-00d5-5b95-13c19bef1149: snapshot restore progress: id=2-113-1757886486716 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:06.798Z [INFO] server-7d5fff56-0eb1-00d5-5b95-13c19bef1149: Installed remote snapshot 2025-09-14T21:48:06.798Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: pipelining replication: peer="{Voter server-7d5fff56-0eb1-00d5-5b95-13c19bef1149 7d5fff56-0eb1-00d5-5b95-13c19bef1149}" 2025-09-14T21:48:07.723Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:06.717254964 +0000 UTC m=+57.285133235 (2 nodes), 739ns from start of poll, 1.67867342s from cluster start. Timeout at 2025-09-14 21:48:07.722265025 +0000 UTC m=+58.290143316, 1.005010081s after stability 2025-09-14T21:48:07.734Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: aborting pipeline replication: peer="{Voter server-7d5fff56-0eb1-00d5-5b95-13c19bef1149 7d5fff56-0eb1-00d5-5b95-13c19bef1149}" 2025-09-14T21:48:07.734Z [INFO] server-7509dcd1-2696-8b20-8178-4d3905988fe0: aborting pipeline replication: peer="{Voter server-a8724f84-71ef-3e51-ea4d-1189508fc5f5 a8724f84-71ef-3e51-ea4d-1189508fc5f5}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false 2025-09-14T21:48:07.736Z [INFO] server-95441668-aa39-abd3-a800-b732c66469c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95441668-aa39-abd3-a800-b732c66469c9 Address:95441668-aa39-abd3-a800-b732c66469c9} {Suffrage:Voter ID:server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4 Address:1f4675e6-47c1-d9e0-916e-7d74049f4fa4} {Suffrage:Voter ID:server-fd60f247-7a29-88da-b6b6-2bebf282f940 Address:fd60f247-7a29-88da-b6b6-2bebf282f940}]" 2025-09-14T21:48:07.736Z [INFO] server-95441668-aa39-abd3-a800-b732c66469c9: entering follower state: follower="Node at 95441668-aa39-abd3-a800-b732c66469c9 [Follower]" leader-address= leader-id= 2025-09-14T21:48:07.736Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95441668-aa39-abd3-a800-b732c66469c9 Address:95441668-aa39-abd3-a800-b732c66469c9} {Suffrage:Voter ID:server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4 Address:1f4675e6-47c1-d9e0-916e-7d74049f4fa4} {Suffrage:Voter ID:server-fd60f247-7a29-88da-b6b6-2bebf282f940 Address:fd60f247-7a29-88da-b6b6-2bebf282f940}]" 2025-09-14T21:48:07.736Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: entering follower state: follower="Node at 1f4675e6-47c1-d9e0-916e-7d74049f4fa4 [Follower]" leader-address= leader-id= 2025-09-14T21:48:07.736Z [INFO] server-fd60f247-7a29-88da-b6b6-2bebf282f940: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95441668-aa39-abd3-a800-b732c66469c9 Address:95441668-aa39-abd3-a800-b732c66469c9} {Suffrage:Voter ID:server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4 Address:1f4675e6-47c1-d9e0-916e-7d74049f4fa4} {Suffrage:Voter ID:server-fd60f247-7a29-88da-b6b6-2bebf282f940 Address:fd60f247-7a29-88da-b6b6-2bebf282f940}]" 2025-09-14T21:48:07.736Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:07.736Z [INFO] server-fd60f247-7a29-88da-b6b6-2bebf282f940: entering follower state: follower="Node at fd60f247-7a29-88da-b6b6-2bebf282f940 [Follower]" leader-address= leader-id= 2025-09-14T21:48:08.358Z [WARN] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:08.358Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: entering candidate state: node="Node at 1f4675e6-47c1-d9e0-916e-7d74049f4fa4 [Candidate]" term=2 2025-09-14T21:48:08.359Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: election won: term=2 tally=2 2025-09-14T21:48:08.359Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: entering leader state: leader="Node at 1f4675e6-47c1-d9e0-916e-7d74049f4fa4 [Leader]" 2025-09-14T21:48:08.359Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: added peer, starting replication: peer=server-95441668-aa39-abd3-a800-b732c66469c9 2025-09-14T21:48:08.359Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: added peer, starting replication: peer=server-fd60f247-7a29-88da-b6b6-2bebf282f940 2025-09-14T21:48:08.359Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: pipelining replication: peer="{Voter server-fd60f247-7a29-88da-b6b6-2bebf282f940 fd60f247-7a29-88da-b6b6-2bebf282f940}" 2025-09-14T21:48:08.359Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: pipelining replication: peer="{Voter server-95441668-aa39-abd3-a800-b732c66469c9 95441668-aa39-abd3-a800-b732c66469c9}" 2025-09-14T21:48:09.364Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:08.359086824 +0000 UTC m=+58.926965108 (1 nodes), 622.788679ms from start of poll, 623.116233ms from cluster start. Timeout at 2025-09-14 21:48:09.364088029 +0000 UTC m=+59.931966290, 1.005001182s after stability 2025-09-14T21:48:09.365Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: starting snapshot up to: index=12 2025-09-14T21:48:09.365Z [INFO] creating new snapshot: path=/tmp/raft2017330974/snapshots/2-12-1757886489365.tmp 2025-09-14T21:48:09.366Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: snapshot complete up to: index=12 2025-09-14T21:48:09.366Z [INFO] creating new snapshot: path=/tmp/raft2017330974/snapshots/2-113-1757886489366.tmp 2025-09-14T21:48:09.367Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: copied to local snapshot: bytes=71 2025-09-14T21:48:09.367Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: snapshot restore progress: id=2-113-1757886489366 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:09.367Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: removing all old logs from log store 2025-09-14T21:48:09.367Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: compacting logs: from=1 to=22 2025-09-14T21:48:09.367Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: restored user snapshot: index=113 2025-09-14T21:48:09.367Z [ERROR] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: failed to get log: index=22 error="log not found" 2025-09-14T21:48:09.367Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: aborting pipeline replication: peer="{Voter server-fd60f247-7a29-88da-b6b6-2bebf282f940 fd60f247-7a29-88da-b6b6-2bebf282f940}" 2025-09-14T21:48:09.367Z [ERROR] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: failed to get log: index=22 error="log not found" 2025-09-14T21:48:09.367Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: aborting pipeline replication: peer="{Voter server-95441668-aa39-abd3-a800-b732c66469c9 95441668-aa39-abd3-a800-b732c66469c9}" 2025-09-14T21:48:09.376Z [ERROR] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: failed to get log: index=22 error="log not found" 2025-09-14T21:48:09.376Z [ERROR] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: failed to get log: index=22 error="log not found" 2025-09-14T21:48:09.376Z [INFO] creating new snapshot: path=/tmp/raft2645647459/snapshots/2-113-1757886489376.tmp 2025-09-14T21:48:09.376Z [INFO] creating new snapshot: path=/tmp/raft1188991158/snapshots/2-113-1757886489376.tmp 2025-09-14T21:48:09.376Z [INFO] server-95441668-aa39-abd3-a800-b732c66469c9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:09.376Z [INFO] server-fd60f247-7a29-88da-b6b6-2bebf282f940: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:09.376Z [INFO] server-95441668-aa39-abd3-a800-b732c66469c9: copied to local snapshot: bytes=71 2025-09-14T21:48:09.377Z [INFO] server-fd60f247-7a29-88da-b6b6-2bebf282f940: copied to local snapshot: bytes=71 2025-09-14T21:48:09.377Z [INFO] server-95441668-aa39-abd3-a800-b732c66469c9: snapshot restore progress: id=2-113-1757886489376 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:09.377Z [INFO] server-95441668-aa39-abd3-a800-b732c66469c9: removing all old logs from log store 2025-09-14T21:48:09.377Z [INFO] server-95441668-aa39-abd3-a800-b732c66469c9: compacting logs: from=1 to=22 2025-09-14T21:48:09.377Z [INFO] server-95441668-aa39-abd3-a800-b732c66469c9: Installed remote snapshot 2025-09-14T21:48:09.377Z [INFO] server-fd60f247-7a29-88da-b6b6-2bebf282f940: snapshot restore progress: id=2-113-1757886489376 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:09.377Z [INFO] server-fd60f247-7a29-88da-b6b6-2bebf282f940: removing all old logs from log store 2025-09-14T21:48:09.377Z [INFO] server-fd60f247-7a29-88da-b6b6-2bebf282f940: compacting logs: from=1 to=22 2025-09-14T21:48:09.377Z [INFO] server-fd60f247-7a29-88da-b6b6-2bebf282f940: Installed remote snapshot 2025-09-14T21:48:09.377Z [WARN] server-95441668-aa39-abd3-a800-b732c66469c9: skipping application of old log: index=22 2025-09-14T21:48:09.377Z [WARN] server-fd60f247-7a29-88da-b6b6-2bebf282f940: skipping application of old log: index=113 2025-09-14T21:48:09.377Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: pipelining replication: peer="{Voter server-95441668-aa39-abd3-a800-b732c66469c9 95441668-aa39-abd3-a800-b732c66469c9}" 2025-09-14T21:48:09.377Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: pipelining replication: peer="{Voter server-fd60f247-7a29-88da-b6b6-2bebf282f940 fd60f247-7a29-88da-b6b6-2bebf282f940}" 2025-09-14T21:48:09.377Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:10.383Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:09.377438878 +0000 UTC m=+59.945317146 (2 nodes), 764ns from start of poll, 1.641468271s from cluster start. Timeout at 2025-09-14 21:48:10.382442258 +0000 UTC m=+60.950320517, 1.005003371s after stability 2025-09-14T21:48:10.395Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: aborting pipeline replication: peer="{Voter server-95441668-aa39-abd3-a800-b732c66469c9 95441668-aa39-abd3-a800-b732c66469c9}" 2025-09-14T21:48:10.395Z [INFO] server-1f4675e6-47c1-d9e0-916e-7d74049f4fa4: aborting pipeline replication: peer="{Voter server-fd60f247-7a29-88da-b6b6-2bebf282f940 fd60f247-7a29-88da-b6b6-2bebf282f940}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true 2025-09-14T21:48:10.396Z [INFO] server-7c1433cd-a69f-baa8-9bef-8324a7d0551b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c1433cd-a69f-baa8-9bef-8324a7d0551b Address:7c1433cd-a69f-baa8-9bef-8324a7d0551b} {Suffrage:Voter ID:server-4608d903-70cc-3431-490d-921c333f7dbd Address:4608d903-70cc-3431-490d-921c333f7dbd} {Suffrage:Voter ID:server-8ba781f2-f823-2ee2-f077-558ad644d846 Address:8ba781f2-f823-2ee2-f077-558ad644d846}]" 2025-09-14T21:48:10.396Z [INFO] server-7c1433cd-a69f-baa8-9bef-8324a7d0551b: entering follower state: follower="Node at 7c1433cd-a69f-baa8-9bef-8324a7d0551b [Follower]" leader-address= leader-id= 2025-09-14T21:48:10.396Z [INFO] server-4608d903-70cc-3431-490d-921c333f7dbd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c1433cd-a69f-baa8-9bef-8324a7d0551b Address:7c1433cd-a69f-baa8-9bef-8324a7d0551b} {Suffrage:Voter ID:server-4608d903-70cc-3431-490d-921c333f7dbd Address:4608d903-70cc-3431-490d-921c333f7dbd} {Suffrage:Voter ID:server-8ba781f2-f823-2ee2-f077-558ad644d846 Address:8ba781f2-f823-2ee2-f077-558ad644d846}]" 2025-09-14T21:48:10.396Z [INFO] server-4608d903-70cc-3431-490d-921c333f7dbd: entering follower state: follower="Node at 4608d903-70cc-3431-490d-921c333f7dbd [Follower]" leader-address= leader-id= 2025-09-14T21:48:10.396Z [INFO] server-8ba781f2-f823-2ee2-f077-558ad644d846: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c1433cd-a69f-baa8-9bef-8324a7d0551b Address:7c1433cd-a69f-baa8-9bef-8324a7d0551b} {Suffrage:Voter ID:server-4608d903-70cc-3431-490d-921c333f7dbd Address:4608d903-70cc-3431-490d-921c333f7dbd} {Suffrage:Voter ID:server-8ba781f2-f823-2ee2-f077-558ad644d846 Address:8ba781f2-f823-2ee2-f077-558ad644d846}]" 2025-09-14T21:48:10.396Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:10.396Z [INFO] server-8ba781f2-f823-2ee2-f077-558ad644d846: entering follower state: follower="Node at 8ba781f2-f823-2ee2-f077-558ad644d846 [Follower]" leader-address= leader-id= 2025-09-14T21:48:11.088Z [WARN] server-8ba781f2-f823-2ee2-f077-558ad644d846: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:11.088Z [INFO] server-8ba781f2-f823-2ee2-f077-558ad644d846: entering candidate state: node="Node at 8ba781f2-f823-2ee2-f077-558ad644d846 [Candidate]" term=2 2025-09-14T21:48:11.088Z [INFO] server-8ba781f2-f823-2ee2-f077-558ad644d846: election won: term=2 tally=2 2025-09-14T21:48:11.088Z [INFO] server-8ba781f2-f823-2ee2-f077-558ad644d846: entering leader state: leader="Node at 8ba781f2-f823-2ee2-f077-558ad644d846 [Leader]" 2025-09-14T21:48:11.088Z [INFO] server-8ba781f2-f823-2ee2-f077-558ad644d846: added peer, starting replication: peer=server-7c1433cd-a69f-baa8-9bef-8324a7d0551b 2025-09-14T21:48:11.088Z [INFO] server-8ba781f2-f823-2ee2-f077-558ad644d846: added peer, starting replication: peer=server-4608d903-70cc-3431-490d-921c333f7dbd 2025-09-14T21:48:11.088Z [INFO] server-8ba781f2-f823-2ee2-f077-558ad644d846: pipelining replication: peer="{Voter server-4608d903-70cc-3431-490d-921c333f7dbd 4608d903-70cc-3431-490d-921c333f7dbd}" 2025-09-14T21:48:11.088Z [INFO] server-8ba781f2-f823-2ee2-f077-558ad644d846: pipelining replication: peer="{Voter server-7c1433cd-a69f-baa8-9bef-8324a7d0551b 7c1433cd-a69f-baa8-9bef-8324a7d0551b}" 2025-09-14T21:48:12.094Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:11.088785839 +0000 UTC m=+61.656664106 (1 nodes), 691.804967ms from start of poll, 692.446537ms from cluster start. Timeout at 2025-09-14 21:48:12.093788779 +0000 UTC m=+62.661667038, 1.005002932s after stability 2025-09-14T21:48:12.094Z [INFO] server-8ba781f2-f823-2ee2-f077-558ad644d846: starting snapshot up to: index=12 2025-09-14T21:48:12.094Z [INFO] creating new snapshot: path=/tmp/raft4204788926/snapshots/2-12-1757886492094.tmp 2025-09-14T21:48:12.095Z [INFO] server-8ba781f2-f823-2ee2-f077-558ad644d846: snapshot complete up to: index=12 2025-09-14T21:48:12.096Z [INFO] server-a104f40c-041d-fb46-7da0-43daf7524697: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a104f40c-041d-fb46-7da0-43daf7524697 Address:a104f40c-041d-fb46-7da0-43daf7524697} {Suffrage:Voter ID:server-e923a74e-f617-8ead-eabc-3a1a475f20d2 Address:e923a74e-f617-8ead-eabc-3a1a475f20d2} {Suffrage:Voter ID:server-e0ff53b6-3af5-8b51-d81b-f08fd0a7e727 Address:e0ff53b6-3af5-8b51-d81b-f08fd0a7e727}]" 2025-09-14T21:48:12.096Z [INFO] server-a104f40c-041d-fb46-7da0-43daf7524697: entering follower state: follower="Node at a104f40c-041d-fb46-7da0-43daf7524697 [Follower]" leader-address= leader-id= 2025-09-14T21:48:12.096Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a104f40c-041d-fb46-7da0-43daf7524697 Address:a104f40c-041d-fb46-7da0-43daf7524697} {Suffrage:Voter ID:server-e923a74e-f617-8ead-eabc-3a1a475f20d2 Address:e923a74e-f617-8ead-eabc-3a1a475f20d2} {Suffrage:Voter ID:server-e0ff53b6-3af5-8b51-d81b-f08fd0a7e727 Address:e0ff53b6-3af5-8b51-d81b-f08fd0a7e727}]" 2025-09-14T21:48:12.096Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: entering follower state: follower="Node at e923a74e-f617-8ead-eabc-3a1a475f20d2 [Follower]" leader-address= leader-id= 2025-09-14T21:48:12.096Z [INFO] server-e0ff53b6-3af5-8b51-d81b-f08fd0a7e727: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a104f40c-041d-fb46-7da0-43daf7524697 Address:a104f40c-041d-fb46-7da0-43daf7524697} {Suffrage:Voter ID:server-e923a74e-f617-8ead-eabc-3a1a475f20d2 Address:e923a74e-f617-8ead-eabc-3a1a475f20d2} {Suffrage:Voter ID:server-e0ff53b6-3af5-8b51-d81b-f08fd0a7e727 Address:e0ff53b6-3af5-8b51-d81b-f08fd0a7e727}]" 2025-09-14T21:48:12.096Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:12.096Z [INFO] server-e0ff53b6-3af5-8b51-d81b-f08fd0a7e727: entering follower state: follower="Node at e0ff53b6-3af5-8b51-d81b-f08fd0a7e727 [Follower]" leader-address= leader-id= 2025-09-14T21:48:12.660Z [WARN] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:12.660Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: entering candidate state: node="Node at e923a74e-f617-8ead-eabc-3a1a475f20d2 [Candidate]" term=2 2025-09-14T21:48:12.660Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: election won: term=2 tally=2 2025-09-14T21:48:12.660Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: entering leader state: leader="Node at e923a74e-f617-8ead-eabc-3a1a475f20d2 [Leader]" 2025-09-14T21:48:12.660Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: added peer, starting replication: peer=server-a104f40c-041d-fb46-7da0-43daf7524697 2025-09-14T21:48:12.660Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: added peer, starting replication: peer=server-e0ff53b6-3af5-8b51-d81b-f08fd0a7e727 2025-09-14T21:48:12.660Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: pipelining replication: peer="{Voter server-e0ff53b6-3af5-8b51-d81b-f08fd0a7e727 e0ff53b6-3af5-8b51-d81b-f08fd0a7e727}" 2025-09-14T21:48:12.660Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: pipelining replication: peer="{Voter server-a104f40c-041d-fb46-7da0-43daf7524697 a104f40c-041d-fb46-7da0-43daf7524697}" 2025-09-14T21:48:13.666Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:12.660552053 +0000 UTC m=+63.228430316 (1 nodes), 563.757114ms from start of poll, 564.115186ms from cluster start. Timeout at 2025-09-14 21:48:13.665555024 +0000 UTC m=+64.233433299, 1.005002983s after stability 2025-09-14T21:48:13.666Z [INFO] creating new snapshot: path=/tmp/raft994419550/snapshots/2-113-1757886493666.tmp 2025-09-14T21:48:13.943Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: copied to local snapshot: bytes=71 2025-09-14T21:48:13.943Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: snapshot restore progress: id=2-113-1757886493666 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:13.943Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: restored user snapshot: index=113 2025-09-14T21:48:13.944Z [ERROR] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: failed to get log: index=3 error="log not found" 2025-09-14T21:48:13.944Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: aborting pipeline replication: peer="{Voter server-e0ff53b6-3af5-8b51-d81b-f08fd0a7e727 e0ff53b6-3af5-8b51-d81b-f08fd0a7e727}" 2025-09-14T21:48:13.944Z [ERROR] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: failed to get log: index=3 error="log not found" 2025-09-14T21:48:13.944Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: aborting pipeline replication: peer="{Voter server-a104f40c-041d-fb46-7da0-43daf7524697 a104f40c-041d-fb46-7da0-43daf7524697}" 2025-09-14T21:48:13.951Z [ERROR] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: failed to get log: index=3 error="log not found" 2025-09-14T21:48:13.951Z [INFO] creating new snapshot: path=/tmp/raft1879326517/snapshots/2-113-1757886493951.tmp 2025-09-14T21:48:13.952Z [INFO] server-e0ff53b6-3af5-8b51-d81b-f08fd0a7e727: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:13.953Z [ERROR] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: failed to get log: index=3 error="log not found" 2025-09-14T21:48:13.954Z [INFO] creating new snapshot: path=/tmp/raft788742830/snapshots/2-113-1757886493954.tmp 2025-09-14T21:48:13.954Z [INFO] server-a104f40c-041d-fb46-7da0-43daf7524697: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:14.084Z [INFO] server-a104f40c-041d-fb46-7da0-43daf7524697: copied to local snapshot: bytes=71 2025-09-14T21:48:14.084Z [INFO] server-e0ff53b6-3af5-8b51-d81b-f08fd0a7e727: copied to local snapshot: bytes=71 2025-09-14T21:48:14.085Z [INFO] server-a104f40c-041d-fb46-7da0-43daf7524697: snapshot restore progress: id=2-113-1757886493954 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:14.085Z [INFO] server-a104f40c-041d-fb46-7da0-43daf7524697: Installed remote snapshot 2025-09-14T21:48:14.085Z [INFO] server-e0ff53b6-3af5-8b51-d81b-f08fd0a7e727: snapshot restore progress: id=2-113-1757886493951 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:14.085Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:14.085Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: pipelining replication: peer="{Voter server-a104f40c-041d-fb46-7da0-43daf7524697 a104f40c-041d-fb46-7da0-43daf7524697}" 2025-09-14T21:48:14.085Z [INFO] server-e0ff53b6-3af5-8b51-d81b-f08fd0a7e727: Installed remote snapshot 2025-09-14T21:48:14.085Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: pipelining replication: peer="{Voter server-e0ff53b6-3af5-8b51-d81b-f08fd0a7e727 e0ff53b6-3af5-8b51-d81b-f08fd0a7e727}" 2025-09-14T21:48:15.091Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:14.08537484 +0000 UTC m=+64.653253099 (2 nodes), 794ns from start of poll, 1.988937969s from cluster start. Timeout at 2025-09-14 21:48:15.090387164 +0000 UTC m=+65.658265423, 1.005012324s after stability 2025-09-14T21:48:15.102Z [INFO] server-8ba781f2-f823-2ee2-f077-558ad644d846: aborting pipeline replication: peer="{Voter server-4608d903-70cc-3431-490d-921c333f7dbd 4608d903-70cc-3431-490d-921c333f7dbd}" 2025-09-14T21:48:15.102Z [INFO] server-8ba781f2-f823-2ee2-f077-558ad644d846: aborting pipeline replication: peer="{Voter server-7c1433cd-a69f-baa8-9bef-8324a7d0551b 7c1433cd-a69f-baa8-9bef-8324a7d0551b}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true 2025-09-14T21:48:15.103Z [INFO] server-50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf Address:50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf} {Suffrage:Voter ID:server-251a804e-617b-95f4-9096-5de8c204aaf0 Address:251a804e-617b-95f4-9096-5de8c204aaf0} {Suffrage:Voter ID:server-c2839229-6605-c226-89f8-d5a29a3fec56 Address:c2839229-6605-c226-89f8-d5a29a3fec56}]" 2025-09-14T21:48:15.103Z [INFO] server-50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf: entering follower state: follower="Node at 50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf [Follower]" leader-address= leader-id= 2025-09-14T21:48:15.103Z [INFO] server-251a804e-617b-95f4-9096-5de8c204aaf0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf Address:50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf} {Suffrage:Voter ID:server-251a804e-617b-95f4-9096-5de8c204aaf0 Address:251a804e-617b-95f4-9096-5de8c204aaf0} {Suffrage:Voter ID:server-c2839229-6605-c226-89f8-d5a29a3fec56 Address:c2839229-6605-c226-89f8-d5a29a3fec56}]" 2025-09-14T21:48:15.103Z [INFO] server-251a804e-617b-95f4-9096-5de8c204aaf0: entering follower state: follower="Node at 251a804e-617b-95f4-9096-5de8c204aaf0 [Follower]" leader-address= leader-id= 2025-09-14T21:48:15.103Z [INFO] server-c2839229-6605-c226-89f8-d5a29a3fec56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf Address:50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf} {Suffrage:Voter ID:server-251a804e-617b-95f4-9096-5de8c204aaf0 Address:251a804e-617b-95f4-9096-5de8c204aaf0} {Suffrage:Voter ID:server-c2839229-6605-c226-89f8-d5a29a3fec56 Address:c2839229-6605-c226-89f8-d5a29a3fec56}]" 2025-09-14T21:48:15.104Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:15.104Z [INFO] server-c2839229-6605-c226-89f8-d5a29a3fec56: entering follower state: follower="Node at c2839229-6605-c226-89f8-d5a29a3fec56 [Follower]" leader-address= leader-id= 2025-09-14T21:48:15.679Z [WARN] server-50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:15.679Z [INFO] server-50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf: entering candidate state: node="Node at 50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf [Candidate]" term=2 2025-09-14T21:48:15.679Z [INFO] server-50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf: election won: term=2 tally=2 2025-09-14T21:48:15.679Z [INFO] server-50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf: entering leader state: leader="Node at 50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf [Leader]" 2025-09-14T21:48:15.680Z [INFO] server-50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf: added peer, starting replication: peer=server-251a804e-617b-95f4-9096-5de8c204aaf0 2025-09-14T21:48:15.680Z [INFO] server-50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf: added peer, starting replication: peer=server-c2839229-6605-c226-89f8-d5a29a3fec56 2025-09-14T21:48:15.680Z [INFO] server-50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf: pipelining replication: peer="{Voter server-251a804e-617b-95f4-9096-5de8c204aaf0 251a804e-617b-95f4-9096-5de8c204aaf0}" 2025-09-14T21:48:15.680Z [INFO] server-50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf: pipelining replication: peer="{Voter server-c2839229-6605-c226-89f8-d5a29a3fec56 c2839229-6605-c226-89f8-d5a29a3fec56}" 2025-09-14T21:48:16.685Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:15.680023916 +0000 UTC m=+66.247902178 (1 nodes), 575.994214ms from start of poll, 576.37828ms from cluster start. Timeout at 2025-09-14 21:48:16.685028635 +0000 UTC m=+67.252906909, 1.005004731s after stability 2025-09-14T21:48:16.685Z [INFO] server-50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf: starting snapshot up to: index=12 2025-09-14T21:48:16.686Z [INFO] creating new snapshot: path=/tmp/raft347673438/snapshots/2-12-1757886496686.tmp 2025-09-14T21:48:16.701Z [INFO] server-50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf: snapshot complete up to: index=12 2025-09-14T21:48:16.702Z [INFO] server-eae572b9-0329-8a3e-58c4-58795497d8a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eae572b9-0329-8a3e-58c4-58795497d8a9 Address:eae572b9-0329-8a3e-58c4-58795497d8a9} {Suffrage:Voter ID:server-a0988163-fbd2-519e-177b-c35cf37d2db9 Address:a0988163-fbd2-519e-177b-c35cf37d2db9} {Suffrage:Voter ID:server-c07746fc-a112-b919-c4e0-affa58477424 Address:c07746fc-a112-b919-c4e0-affa58477424}]" 2025-09-14T21:48:16.702Z [INFO] server-eae572b9-0329-8a3e-58c4-58795497d8a9: entering follower state: follower="Node at eae572b9-0329-8a3e-58c4-58795497d8a9 [Follower]" leader-address= leader-id= 2025-09-14T21:48:16.702Z [INFO] server-a0988163-fbd2-519e-177b-c35cf37d2db9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eae572b9-0329-8a3e-58c4-58795497d8a9 Address:eae572b9-0329-8a3e-58c4-58795497d8a9} {Suffrage:Voter ID:server-a0988163-fbd2-519e-177b-c35cf37d2db9 Address:a0988163-fbd2-519e-177b-c35cf37d2db9} {Suffrage:Voter ID:server-c07746fc-a112-b919-c4e0-affa58477424 Address:c07746fc-a112-b919-c4e0-affa58477424}]" 2025-09-14T21:48:16.702Z [INFO] server-a0988163-fbd2-519e-177b-c35cf37d2db9: entering follower state: follower="Node at a0988163-fbd2-519e-177b-c35cf37d2db9 [Follower]" leader-address= leader-id= 2025-09-14T21:48:16.702Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eae572b9-0329-8a3e-58c4-58795497d8a9 Address:eae572b9-0329-8a3e-58c4-58795497d8a9} {Suffrage:Voter ID:server-a0988163-fbd2-519e-177b-c35cf37d2db9 Address:a0988163-fbd2-519e-177b-c35cf37d2db9} {Suffrage:Voter ID:server-c07746fc-a112-b919-c4e0-affa58477424 Address:c07746fc-a112-b919-c4e0-affa58477424}]" 2025-09-14T21:48:16.702Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:16.702Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: entering follower state: follower="Node at c07746fc-a112-b919-c4e0-affa58477424 [Follower]" leader-address= leader-id= 2025-09-14T21:48:17.339Z [WARN] server-c07746fc-a112-b919-c4e0-affa58477424: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:17.339Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: entering candidate state: node="Node at c07746fc-a112-b919-c4e0-affa58477424 [Candidate]" term=2 2025-09-14T21:48:17.340Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: election won: term=2 tally=2 2025-09-14T21:48:17.340Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: entering leader state: leader="Node at c07746fc-a112-b919-c4e0-affa58477424 [Leader]" 2025-09-14T21:48:17.340Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: added peer, starting replication: peer=server-eae572b9-0329-8a3e-58c4-58795497d8a9 2025-09-14T21:48:17.340Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: added peer, starting replication: peer=server-a0988163-fbd2-519e-177b-c35cf37d2db9 2025-09-14T21:48:17.340Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: pipelining replication: peer="{Voter server-eae572b9-0329-8a3e-58c4-58795497d8a9 eae572b9-0329-8a3e-58c4-58795497d8a9}" 2025-09-14T21:48:17.340Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: pipelining replication: peer="{Voter server-a0988163-fbd2-519e-177b-c35cf37d2db9 a0988163-fbd2-519e-177b-c35cf37d2db9}" 2025-09-14T21:48:18.345Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:17.34007639 +0000 UTC m=+67.907954664 (1 nodes), 637.520981ms from start of poll, 637.85473ms from cluster start. Timeout at 2025-09-14 21:48:18.34508231 +0000 UTC m=+68.912960586, 1.005005922s after stability 2025-09-14T21:48:18.355Z [INFO] creating new snapshot: path=/tmp/raft3292138694/snapshots/2-113-1757886498355.tmp 2025-09-14T21:48:18.355Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: copied to local snapshot: bytes=71 2025-09-14T21:48:18.355Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: snapshot restore progress: id=2-113-1757886498355 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:18.355Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: removing all old logs from log store 2025-09-14T21:48:18.355Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: compacting logs: from=1 to=2 2025-09-14T21:48:18.355Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: restored user snapshot: index=113 2025-09-14T21:48:18.355Z [ERROR] server-c07746fc-a112-b919-c4e0-affa58477424: failed to get log: index=2 error="log not found" 2025-09-14T21:48:18.355Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: aborting pipeline replication: peer="{Voter server-a0988163-fbd2-519e-177b-c35cf37d2db9 a0988163-fbd2-519e-177b-c35cf37d2db9}" 2025-09-14T21:48:18.355Z [ERROR] server-c07746fc-a112-b919-c4e0-affa58477424: failed to get log: index=2 error="log not found" 2025-09-14T21:48:18.355Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: aborting pipeline replication: peer="{Voter server-eae572b9-0329-8a3e-58c4-58795497d8a9 eae572b9-0329-8a3e-58c4-58795497d8a9}" 2025-09-14T21:48:18.363Z [ERROR] server-c07746fc-a112-b919-c4e0-affa58477424: failed to get log: index=2 error="log not found" 2025-09-14T21:48:18.363Z [ERROR] server-c07746fc-a112-b919-c4e0-affa58477424: failed to get log: index=2 error="log not found" 2025-09-14T21:48:18.364Z [INFO] creating new snapshot: path=/tmp/raft2911178901/snapshots/2-113-1757886498364.tmp 2025-09-14T21:48:18.364Z [INFO] creating new snapshot: path=/tmp/raft2570025511/snapshots/2-113-1757886498364.tmp 2025-09-14T21:48:18.364Z [INFO] server-eae572b9-0329-8a3e-58c4-58795497d8a9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:18.364Z [INFO] server-a0988163-fbd2-519e-177b-c35cf37d2db9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:18.648Z [INFO] server-eae572b9-0329-8a3e-58c4-58795497d8a9: copied to local snapshot: bytes=71 2025-09-14T21:48:18.648Z [INFO] server-a0988163-fbd2-519e-177b-c35cf37d2db9: copied to local snapshot: bytes=71 2025-09-14T21:48:18.648Z [INFO] server-eae572b9-0329-8a3e-58c4-58795497d8a9: snapshot restore progress: id=2-113-1757886498364 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:18.648Z [INFO] server-a0988163-fbd2-519e-177b-c35cf37d2db9: snapshot restore progress: id=2-113-1757886498364 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:18.648Z [INFO] server-eae572b9-0329-8a3e-58c4-58795497d8a9: removing all old logs from log store 2025-09-14T21:48:18.648Z [INFO] server-eae572b9-0329-8a3e-58c4-58795497d8a9: compacting logs: from=1 to=2 2025-09-14T21:48:18.648Z [INFO] server-eae572b9-0329-8a3e-58c4-58795497d8a9: Installed remote snapshot 2025-09-14T21:48:18.648Z [INFO] server-a0988163-fbd2-519e-177b-c35cf37d2db9: removing all old logs from log store 2025-09-14T21:48:18.648Z [INFO] server-a0988163-fbd2-519e-177b-c35cf37d2db9: compacting logs: from=1 to=2 2025-09-14T21:48:18.648Z [INFO] server-a0988163-fbd2-519e-177b-c35cf37d2db9: Installed remote snapshot 2025-09-14T21:48:18.648Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: pipelining replication: peer="{Voter server-eae572b9-0329-8a3e-58c4-58795497d8a9 eae572b9-0329-8a3e-58c4-58795497d8a9}" 2025-09-14T21:48:18.648Z [WARN] server-a0988163-fbd2-519e-177b-c35cf37d2db9: skipping application of old log: index=113 2025-09-14T21:48:18.648Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:18.648Z [INFO] server-c07746fc-a112-b919-c4e0-affa58477424: pipelining replication: peer="{Voter server-a0988163-fbd2-519e-177b-c35cf37d2db9 a0988163-fbd2-519e-177b-c35cf37d2db9}" 2025-09-14T21:48:19.653Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:18.648903909 +0000 UTC m=+69.216782176 (2 nodes), 665ns from start of poll, 1.946682242s from cluster start. Timeout at 2025-09-14 21:48:19.653917739 +0000 UTC m=+70.221796038, 1.005013862s after stability 2025-09-14T21:48:19.665Z [INFO] server-50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf: aborting pipeline replication: peer="{Voter server-c2839229-6605-c226-89f8-d5a29a3fec56 c2839229-6605-c226-89f8-d5a29a3fec56}" 2025-09-14T21:48:19.665Z [INFO] server-50bef5f5-7d8c-4cb9-8f99-c6e9fa14f5cf: aborting pipeline replication: peer="{Voter server-251a804e-617b-95f4-9096-5de8c204aaf0 251a804e-617b-95f4-9096-5de8c204aaf0}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false 2025-09-14T21:48:19.666Z [INFO] server-f129417c-4acd-e6de-5f81-ddf96c98d4a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f129417c-4acd-e6de-5f81-ddf96c98d4a6 Address:f129417c-4acd-e6de-5f81-ddf96c98d4a6} {Suffrage:Voter ID:server-3e7f9056-cc36-4138-5ee1-4478c432a704 Address:3e7f9056-cc36-4138-5ee1-4478c432a704} {Suffrage:Voter ID:server-6e90427a-2497-9f36-8b7f-4d311fe51646 Address:6e90427a-2497-9f36-8b7f-4d311fe51646}]" 2025-09-14T21:48:19.666Z [INFO] server-f129417c-4acd-e6de-5f81-ddf96c98d4a6: entering follower state: follower="Node at f129417c-4acd-e6de-5f81-ddf96c98d4a6 [Follower]" leader-address= leader-id= 2025-09-14T21:48:19.666Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f129417c-4acd-e6de-5f81-ddf96c98d4a6 Address:f129417c-4acd-e6de-5f81-ddf96c98d4a6} {Suffrage:Voter ID:server-3e7f9056-cc36-4138-5ee1-4478c432a704 Address:3e7f9056-cc36-4138-5ee1-4478c432a704} {Suffrage:Voter ID:server-6e90427a-2497-9f36-8b7f-4d311fe51646 Address:6e90427a-2497-9f36-8b7f-4d311fe51646}]" 2025-09-14T21:48:19.666Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: entering follower state: follower="Node at 3e7f9056-cc36-4138-5ee1-4478c432a704 [Follower]" leader-address= leader-id= 2025-09-14T21:48:19.666Z [INFO] server-6e90427a-2497-9f36-8b7f-4d311fe51646: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f129417c-4acd-e6de-5f81-ddf96c98d4a6 Address:f129417c-4acd-e6de-5f81-ddf96c98d4a6} {Suffrage:Voter ID:server-3e7f9056-cc36-4138-5ee1-4478c432a704 Address:3e7f9056-cc36-4138-5ee1-4478c432a704} {Suffrage:Voter ID:server-6e90427a-2497-9f36-8b7f-4d311fe51646 Address:6e90427a-2497-9f36-8b7f-4d311fe51646}]" 2025-09-14T21:48:19.666Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:19.666Z [INFO] server-6e90427a-2497-9f36-8b7f-4d311fe51646: entering follower state: follower="Node at 6e90427a-2497-9f36-8b7f-4d311fe51646 [Follower]" leader-address= leader-id= 2025-09-14T21:48:20.170Z [WARN] server-3e7f9056-cc36-4138-5ee1-4478c432a704: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:20.170Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: entering candidate state: node="Node at 3e7f9056-cc36-4138-5ee1-4478c432a704 [Candidate]" term=2 2025-09-14T21:48:20.170Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: election won: term=2 tally=2 2025-09-14T21:48:20.171Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: entering leader state: leader="Node at 3e7f9056-cc36-4138-5ee1-4478c432a704 [Leader]" 2025-09-14T21:48:20.171Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: added peer, starting replication: peer=server-f129417c-4acd-e6de-5f81-ddf96c98d4a6 2025-09-14T21:48:20.171Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: added peer, starting replication: peer=server-6e90427a-2497-9f36-8b7f-4d311fe51646 2025-09-14T21:48:20.171Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: pipelining replication: peer="{Voter server-6e90427a-2497-9f36-8b7f-4d311fe51646 6e90427a-2497-9f36-8b7f-4d311fe51646}" 2025-09-14T21:48:20.171Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: pipelining replication: peer="{Voter server-f129417c-4acd-e6de-5f81-ddf96c98d4a6 f129417c-4acd-e6de-5f81-ddf96c98d4a6}" 2025-09-14T21:48:21.176Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:20.171009995 +0000 UTC m=+70.738888270 (1 nodes), 504.148013ms from start of poll, 504.492255ms from cluster start. Timeout at 2025-09-14 21:48:21.176011108 +0000 UTC m=+71.743889372, 1.005001102s after stability 2025-09-14T21:48:21.177Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: starting snapshot up to: index=12 2025-09-14T21:48:21.177Z [INFO] creating new snapshot: path=/tmp/raft2568634495/snapshots/2-12-1757886501177.tmp 2025-09-14T21:48:21.198Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: snapshot complete up to: index=12 2025-09-14T21:48:21.199Z [INFO] creating new snapshot: path=/tmp/raft2568634495/snapshots/2-1013-1757886501199.tmp 2025-09-14T21:48:21.363Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: copied to local snapshot: bytes=71 2025-09-14T21:48:21.363Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: snapshot restore progress: id=2-1013-1757886501199 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:21.363Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: restored user snapshot: index=1013 2025-09-14T21:48:21.363Z [ERROR] server-3e7f9056-cc36-4138-5ee1-4478c432a704: failed to get log: index=23 error="log not found" 2025-09-14T21:48:21.363Z [ERROR] server-3e7f9056-cc36-4138-5ee1-4478c432a704: failed to get log: index=23 error="log not found" 2025-09-14T21:48:21.363Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: aborting pipeline replication: peer="{Voter server-f129417c-4acd-e6de-5f81-ddf96c98d4a6 f129417c-4acd-e6de-5f81-ddf96c98d4a6}" 2025-09-14T21:48:21.363Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: aborting pipeline replication: peer="{Voter server-6e90427a-2497-9f36-8b7f-4d311fe51646 6e90427a-2497-9f36-8b7f-4d311fe51646}" 2025-09-14T21:48:21.372Z [ERROR] server-3e7f9056-cc36-4138-5ee1-4478c432a704: failed to get log: index=23 error="log not found" 2025-09-14T21:48:21.373Z [INFO] creating new snapshot: path=/tmp/raft2607476803/snapshots/2-1013-1757886501373.tmp 2025-09-14T21:48:21.373Z [ERROR] server-3e7f9056-cc36-4138-5ee1-4478c432a704: failed to get log: index=23 error="log not found" 2025-09-14T21:48:21.374Z [INFO] server-f129417c-4acd-e6de-5f81-ddf96c98d4a6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:21.374Z [INFO] creating new snapshot: path=/tmp/raft2015511625/snapshots/2-1013-1757886501374.tmp 2025-09-14T21:48:21.374Z [INFO] server-6e90427a-2497-9f36-8b7f-4d311fe51646: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:21.451Z [INFO] server-f129417c-4acd-e6de-5f81-ddf96c98d4a6: copied to local snapshot: bytes=71 2025-09-14T21:48:21.451Z [INFO] server-6e90427a-2497-9f36-8b7f-4d311fe51646: copied to local snapshot: bytes=71 2025-09-14T21:48:21.451Z [INFO] server-f129417c-4acd-e6de-5f81-ddf96c98d4a6: snapshot restore progress: id=2-1013-1757886501373 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:21.451Z [INFO] server-f129417c-4acd-e6de-5f81-ddf96c98d4a6: Installed remote snapshot 2025-09-14T21:48:21.451Z [INFO] server-6e90427a-2497-9f36-8b7f-4d311fe51646: snapshot restore progress: id=2-1013-1757886501374 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:21.451Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: pipelining replication: peer="{Voter server-f129417c-4acd-e6de-5f81-ddf96c98d4a6 f129417c-4acd-e6de-5f81-ddf96c98d4a6}" 2025-09-14T21:48:21.451Z [INFO] server-6e90427a-2497-9f36-8b7f-4d311fe51646: Installed remote snapshot 2025-09-14T21:48:21.451Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:21.452Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: pipelining replication: peer="{Voter server-6e90427a-2497-9f36-8b7f-4d311fe51646 6e90427a-2497-9f36-8b7f-4d311fe51646}" 2025-09-14T21:48:22.457Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:21.451998295 +0000 UTC m=+72.019876553 (2 nodes), 933ns from start of poll, 1.785480538s from cluster start. Timeout at 2025-09-14 21:48:22.457013175 +0000 UTC m=+73.024891457, 1.005014904s after stability 2025-09-14T21:48:22.469Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: aborting pipeline replication: peer="{Voter server-6e90427a-2497-9f36-8b7f-4d311fe51646 6e90427a-2497-9f36-8b7f-4d311fe51646}" 2025-09-14T21:48:22.469Z [INFO] server-3e7f9056-cc36-4138-5ee1-4478c432a704: aborting pipeline replication: peer="{Voter server-f129417c-4acd-e6de-5f81-ddf96c98d4a6 f129417c-4acd-e6de-5f81-ddf96c98d4a6}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false 2025-09-14T21:48:22.471Z [INFO] server-01b69d89-c8d8-3406-f60a-80d00fc256f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01b69d89-c8d8-3406-f60a-80d00fc256f3 Address:01b69d89-c8d8-3406-f60a-80d00fc256f3} {Suffrage:Voter ID:server-befd431d-e34b-c594-4e9e-d6d640be73be Address:befd431d-e34b-c594-4e9e-d6d640be73be} {Suffrage:Voter ID:server-c159e05f-1075-9ba2-dbef-17b61b031ce3 Address:c159e05f-1075-9ba2-dbef-17b61b031ce3}]" 2025-09-14T21:48:22.471Z [INFO] server-01b69d89-c8d8-3406-f60a-80d00fc256f3: entering follower state: follower="Node at 01b69d89-c8d8-3406-f60a-80d00fc256f3 [Follower]" leader-address= leader-id= 2025-09-14T21:48:22.471Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01b69d89-c8d8-3406-f60a-80d00fc256f3 Address:01b69d89-c8d8-3406-f60a-80d00fc256f3} {Suffrage:Voter ID:server-befd431d-e34b-c594-4e9e-d6d640be73be Address:befd431d-e34b-c594-4e9e-d6d640be73be} {Suffrage:Voter ID:server-c159e05f-1075-9ba2-dbef-17b61b031ce3 Address:c159e05f-1075-9ba2-dbef-17b61b031ce3}]" 2025-09-14T21:48:22.471Z [INFO] server-c159e05f-1075-9ba2-dbef-17b61b031ce3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01b69d89-c8d8-3406-f60a-80d00fc256f3 Address:01b69d89-c8d8-3406-f60a-80d00fc256f3} {Suffrage:Voter ID:server-befd431d-e34b-c594-4e9e-d6d640be73be Address:befd431d-e34b-c594-4e9e-d6d640be73be} {Suffrage:Voter ID:server-c159e05f-1075-9ba2-dbef-17b61b031ce3 Address:c159e05f-1075-9ba2-dbef-17b61b031ce3}]" 2025-09-14T21:48:22.471Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: entering follower state: follower="Node at befd431d-e34b-c594-4e9e-d6d640be73be [Follower]" leader-address= leader-id= 2025-09-14T21:48:22.471Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:22.471Z [INFO] server-c159e05f-1075-9ba2-dbef-17b61b031ce3: entering follower state: follower="Node at c159e05f-1075-9ba2-dbef-17b61b031ce3 [Follower]" leader-address= leader-id= 2025-09-14T21:48:22.980Z [WARN] server-befd431d-e34b-c594-4e9e-d6d640be73be: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:22.980Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: entering candidate state: node="Node at befd431d-e34b-c594-4e9e-d6d640be73be [Candidate]" term=2 2025-09-14T21:48:22.980Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: election won: term=2 tally=2 2025-09-14T21:48:22.980Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: entering leader state: leader="Node at befd431d-e34b-c594-4e9e-d6d640be73be [Leader]" 2025-09-14T21:48:22.980Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: added peer, starting replication: peer=server-01b69d89-c8d8-3406-f60a-80d00fc256f3 2025-09-14T21:48:22.980Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: added peer, starting replication: peer=server-c159e05f-1075-9ba2-dbef-17b61b031ce3 2025-09-14T21:48:22.981Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: pipelining replication: peer="{Voter server-01b69d89-c8d8-3406-f60a-80d00fc256f3 01b69d89-c8d8-3406-f60a-80d00fc256f3}" 2025-09-14T21:48:22.981Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: pipelining replication: peer="{Voter server-c159e05f-1075-9ba2-dbef-17b61b031ce3 c159e05f-1075-9ba2-dbef-17b61b031ce3}" 2025-09-14T21:48:23.986Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:22.980972043 +0000 UTC m=+73.548850302 (1 nodes), 509.757855ms from start of poll, 510.060942ms from cluster start. Timeout at 2025-09-14 21:48:23.985977244 +0000 UTC m=+74.553855504, 1.005005202s after stability 2025-09-14T21:48:23.986Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: starting snapshot up to: index=12 2025-09-14T21:48:23.987Z [INFO] creating new snapshot: path=/tmp/raft2144222587/snapshots/2-12-1757886503987.tmp 2025-09-14T21:48:23.987Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: snapshot complete up to: index=12 2025-09-14T21:48:23.988Z [INFO] creating new snapshot: path=/tmp/raft2144222587/snapshots/2-1013-1757886503988.tmp 2025-09-14T21:48:23.988Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: copied to local snapshot: bytes=71 2025-09-14T21:48:23.988Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: snapshot restore progress: id=2-1013-1757886503988 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:23.988Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: removing all old logs from log store 2025-09-14T21:48:23.988Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: compacting logs: from=1 to=22 2025-09-14T21:48:23.988Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: restored user snapshot: index=1013 2025-09-14T21:48:23.988Z [ERROR] server-befd431d-e34b-c594-4e9e-d6d640be73be: failed to get log: index=22 error="log not found" 2025-09-14T21:48:23.988Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: aborting pipeline replication: peer="{Voter server-c159e05f-1075-9ba2-dbef-17b61b031ce3 c159e05f-1075-9ba2-dbef-17b61b031ce3}" 2025-09-14T21:48:23.988Z [ERROR] server-befd431d-e34b-c594-4e9e-d6d640be73be: failed to get log: index=22 error="log not found" 2025-09-14T21:48:23.988Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: aborting pipeline replication: peer="{Voter server-01b69d89-c8d8-3406-f60a-80d00fc256f3 01b69d89-c8d8-3406-f60a-80d00fc256f3}" 2025-09-14T21:48:23.995Z [ERROR] server-befd431d-e34b-c594-4e9e-d6d640be73be: failed to get log: index=22 error="log not found" 2025-09-14T21:48:23.996Z [INFO] creating new snapshot: path=/tmp/raft785450040/snapshots/2-1013-1757886503996.tmp 2025-09-14T21:48:23.996Z [INFO] server-c159e05f-1075-9ba2-dbef-17b61b031ce3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:23.996Z [INFO] server-c159e05f-1075-9ba2-dbef-17b61b031ce3: copied to local snapshot: bytes=71 2025-09-14T21:48:23.997Z [ERROR] server-befd431d-e34b-c594-4e9e-d6d640be73be: failed to get log: index=22 error="log not found" 2025-09-14T21:48:23.996Z [INFO] server-c159e05f-1075-9ba2-dbef-17b61b031ce3: snapshot restore progress: id=2-1013-1757886503996 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:23.997Z [INFO] server-c159e05f-1075-9ba2-dbef-17b61b031ce3: removing all old logs from log store 2025-09-14T21:48:23.997Z [INFO] server-c159e05f-1075-9ba2-dbef-17b61b031ce3: compacting logs: from=1 to=22 2025-09-14T21:48:23.997Z [INFO] server-c159e05f-1075-9ba2-dbef-17b61b031ce3: Installed remote snapshot 2025-09-14T21:48:23.997Z [WARN] server-c159e05f-1075-9ba2-dbef-17b61b031ce3: skipping application of old log: index=22 2025-09-14T21:48:23.997Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: pipelining replication: peer="{Voter server-c159e05f-1075-9ba2-dbef-17b61b031ce3 c159e05f-1075-9ba2-dbef-17b61b031ce3}" 2025-09-14T21:48:23.997Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:23.997Z [INFO] creating new snapshot: path=/tmp/raft2373624742/snapshots/2-1013-1757886503997.tmp 2025-09-14T21:48:23.997Z [INFO] server-01b69d89-c8d8-3406-f60a-80d00fc256f3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:23.997Z [INFO] server-01b69d89-c8d8-3406-f60a-80d00fc256f3: copied to local snapshot: bytes=71 2025-09-14T21:48:23.997Z [INFO] server-01b69d89-c8d8-3406-f60a-80d00fc256f3: snapshot restore progress: id=2-1013-1757886503997 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:23.997Z [INFO] server-01b69d89-c8d8-3406-f60a-80d00fc256f3: removing all old logs from log store 2025-09-14T21:48:23.997Z [INFO] server-01b69d89-c8d8-3406-f60a-80d00fc256f3: compacting logs: from=1 to=22 2025-09-14T21:48:23.997Z [INFO] server-01b69d89-c8d8-3406-f60a-80d00fc256f3: Installed remote snapshot 2025-09-14T21:48:23.998Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: pipelining replication: peer="{Voter server-01b69d89-c8d8-3406-f60a-80d00fc256f3 01b69d89-c8d8-3406-f60a-80d00fc256f3}" 2025-09-14T21:48:25.003Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:23.997269776 +0000 UTC m=+74.565148047 (2 nodes), 962ns from start of poll, 1.526358687s from cluster start. Timeout at 2025-09-14 21:48:25.002288064 +0000 UTC m=+75.570166326, 1.005018279s after stability 2025-09-14T21:48:25.009Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: aborting pipeline replication: peer="{Voter server-c159e05f-1075-9ba2-dbef-17b61b031ce3 c159e05f-1075-9ba2-dbef-17b61b031ce3}" 2025-09-14T21:48:25.009Z [INFO] server-befd431d-e34b-c594-4e9e-d6d640be73be: aborting pipeline replication: peer="{Voter server-01b69d89-c8d8-3406-f60a-80d00fc256f3 01b69d89-c8d8-3406-f60a-80d00fc256f3}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true 2025-09-14T21:48:25.011Z [INFO] server-6cc25a5c-92fe-bc79-86e5-4811f1de3763: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6cc25a5c-92fe-bc79-86e5-4811f1de3763 Address:6cc25a5c-92fe-bc79-86e5-4811f1de3763} {Suffrage:Voter ID:server-cd5126b9-81df-7b50-9a72-81d90c3bc9d1 Address:cd5126b9-81df-7b50-9a72-81d90c3bc9d1} {Suffrage:Voter ID:server-be462246-91fc-332c-c183-e21636e6c509 Address:be462246-91fc-332c-c183-e21636e6c509}]" 2025-09-14T21:48:25.011Z [INFO] server-6cc25a5c-92fe-bc79-86e5-4811f1de3763: entering follower state: follower="Node at 6cc25a5c-92fe-bc79-86e5-4811f1de3763 [Follower]" leader-address= leader-id= 2025-09-14T21:48:25.011Z [INFO] server-cd5126b9-81df-7b50-9a72-81d90c3bc9d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6cc25a5c-92fe-bc79-86e5-4811f1de3763 Address:6cc25a5c-92fe-bc79-86e5-4811f1de3763} {Suffrage:Voter ID:server-cd5126b9-81df-7b50-9a72-81d90c3bc9d1 Address:cd5126b9-81df-7b50-9a72-81d90c3bc9d1} {Suffrage:Voter ID:server-be462246-91fc-332c-c183-e21636e6c509 Address:be462246-91fc-332c-c183-e21636e6c509}]" 2025-09-14T21:48:25.011Z [INFO] server-cd5126b9-81df-7b50-9a72-81d90c3bc9d1: entering follower state: follower="Node at cd5126b9-81df-7b50-9a72-81d90c3bc9d1 [Follower]" leader-address= leader-id= 2025-09-14T21:48:25.011Z [INFO] server-be462246-91fc-332c-c183-e21636e6c509: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6cc25a5c-92fe-bc79-86e5-4811f1de3763 Address:6cc25a5c-92fe-bc79-86e5-4811f1de3763} {Suffrage:Voter ID:server-cd5126b9-81df-7b50-9a72-81d90c3bc9d1 Address:cd5126b9-81df-7b50-9a72-81d90c3bc9d1} {Suffrage:Voter ID:server-be462246-91fc-332c-c183-e21636e6c509 Address:be462246-91fc-332c-c183-e21636e6c509}]" 2025-09-14T21:48:25.011Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:25.011Z [INFO] server-be462246-91fc-332c-c183-e21636e6c509: entering follower state: follower="Node at be462246-91fc-332c-c183-e21636e6c509 [Follower]" leader-address= leader-id= 2025-09-14T21:48:25.610Z [WARN] server-6cc25a5c-92fe-bc79-86e5-4811f1de3763: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:25.610Z [INFO] server-6cc25a5c-92fe-bc79-86e5-4811f1de3763: entering candidate state: node="Node at 6cc25a5c-92fe-bc79-86e5-4811f1de3763 [Candidate]" term=2 2025-09-14T21:48:25.610Z [INFO] server-6cc25a5c-92fe-bc79-86e5-4811f1de3763: election won: term=2 tally=2 2025-09-14T21:48:25.610Z [INFO] server-6cc25a5c-92fe-bc79-86e5-4811f1de3763: entering leader state: leader="Node at 6cc25a5c-92fe-bc79-86e5-4811f1de3763 [Leader]" 2025-09-14T21:48:25.610Z [INFO] server-6cc25a5c-92fe-bc79-86e5-4811f1de3763: added peer, starting replication: peer=server-cd5126b9-81df-7b50-9a72-81d90c3bc9d1 2025-09-14T21:48:25.610Z [INFO] server-6cc25a5c-92fe-bc79-86e5-4811f1de3763: added peer, starting replication: peer=server-be462246-91fc-332c-c183-e21636e6c509 2025-09-14T21:48:25.610Z [INFO] server-6cc25a5c-92fe-bc79-86e5-4811f1de3763: pipelining replication: peer="{Voter server-cd5126b9-81df-7b50-9a72-81d90c3bc9d1 cd5126b9-81df-7b50-9a72-81d90c3bc9d1}" 2025-09-14T21:48:25.610Z [INFO] server-6cc25a5c-92fe-bc79-86e5-4811f1de3763: pipelining replication: peer="{Voter server-be462246-91fc-332c-c183-e21636e6c509 be462246-91fc-332c-c183-e21636e6c509}" 2025-09-14T21:48:26.615Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:25.610782474 +0000 UTC m=+76.178660754 (1 nodes), 599.168654ms from start of poll, 599.580026ms from cluster start. Timeout at 2025-09-14 21:48:26.615787766 +0000 UTC m=+77.183666029, 1.005005275s after stability 2025-09-14T21:48:26.616Z [INFO] server-6cc25a5c-92fe-bc79-86e5-4811f1de3763: starting snapshot up to: index=12 2025-09-14T21:48:26.616Z [INFO] creating new snapshot: path=/tmp/raft1509186803/snapshots/2-12-1757886506616.tmp 2025-09-14T21:48:26.616Z [INFO] server-6cc25a5c-92fe-bc79-86e5-4811f1de3763: snapshot complete up to: index=12 2025-09-14T21:48:26.617Z [INFO] server-28a659c2-c43c-6853-7088-2e882ff7c590: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-28a659c2-c43c-6853-7088-2e882ff7c590 Address:28a659c2-c43c-6853-7088-2e882ff7c590} {Suffrage:Voter ID:server-98ec5df1-c470-d517-1e27-d2ade02cd320 Address:98ec5df1-c470-d517-1e27-d2ade02cd320} {Suffrage:Voter ID:server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e Address:057ce2ab-8cb9-324e-cd5d-05a52ac7245e}]" 2025-09-14T21:48:26.617Z [INFO] server-28a659c2-c43c-6853-7088-2e882ff7c590: entering follower state: follower="Node at 28a659c2-c43c-6853-7088-2e882ff7c590 [Follower]" leader-address= leader-id= 2025-09-14T21:48:26.618Z [INFO] server-98ec5df1-c470-d517-1e27-d2ade02cd320: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-28a659c2-c43c-6853-7088-2e882ff7c590 Address:28a659c2-c43c-6853-7088-2e882ff7c590} {Suffrage:Voter ID:server-98ec5df1-c470-d517-1e27-d2ade02cd320 Address:98ec5df1-c470-d517-1e27-d2ade02cd320} {Suffrage:Voter ID:server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e Address:057ce2ab-8cb9-324e-cd5d-05a52ac7245e}]" 2025-09-14T21:48:26.618Z [INFO] server-98ec5df1-c470-d517-1e27-d2ade02cd320: entering follower state: follower="Node at 98ec5df1-c470-d517-1e27-d2ade02cd320 [Follower]" leader-address= leader-id= 2025-09-14T21:48:26.618Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-28a659c2-c43c-6853-7088-2e882ff7c590 Address:28a659c2-c43c-6853-7088-2e882ff7c590} {Suffrage:Voter ID:server-98ec5df1-c470-d517-1e27-d2ade02cd320 Address:98ec5df1-c470-d517-1e27-d2ade02cd320} {Suffrage:Voter ID:server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e Address:057ce2ab-8cb9-324e-cd5d-05a52ac7245e}]" 2025-09-14T21:48:26.618Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:26.618Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: entering follower state: follower="Node at 057ce2ab-8cb9-324e-cd5d-05a52ac7245e [Follower]" leader-address= leader-id= 2025-09-14T21:48:27.166Z [WARN] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:27.166Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: entering candidate state: node="Node at 057ce2ab-8cb9-324e-cd5d-05a52ac7245e [Candidate]" term=2 2025-09-14T21:48:27.166Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: election won: term=2 tally=2 2025-09-14T21:48:27.166Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: entering leader state: leader="Node at 057ce2ab-8cb9-324e-cd5d-05a52ac7245e [Leader]" 2025-09-14T21:48:27.166Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: added peer, starting replication: peer=server-28a659c2-c43c-6853-7088-2e882ff7c590 2025-09-14T21:48:27.166Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: added peer, starting replication: peer=server-98ec5df1-c470-d517-1e27-d2ade02cd320 2025-09-14T21:48:27.166Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: pipelining replication: peer="{Voter server-28a659c2-c43c-6853-7088-2e882ff7c590 28a659c2-c43c-6853-7088-2e882ff7c590}" 2025-09-14T21:48:27.166Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: pipelining replication: peer="{Voter server-98ec5df1-c470-d517-1e27-d2ade02cd320 98ec5df1-c470-d517-1e27-d2ade02cd320}" 2025-09-14T21:48:28.172Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:27.166214339 +0000 UTC m=+77.734092605 (1 nodes), 548.088038ms from start of poll, 548.411381ms from cluster start. Timeout at 2025-09-14 21:48:28.171220567 +0000 UTC m=+78.739098829, 1.005006224s after stability 2025-09-14T21:48:28.172Z [INFO] creating new snapshot: path=/tmp/raft1527410179/snapshots/2-1013-1757886508172.tmp 2025-09-14T21:48:28.173Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: copied to local snapshot: bytes=71 2025-09-14T21:48:28.173Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: snapshot restore progress: id=2-1013-1757886508172 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:28.173Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: restored user snapshot: index=1013 2025-09-14T21:48:28.173Z [ERROR] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: failed to get log: index=3 error="log not found" 2025-09-14T21:48:28.173Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: aborting pipeline replication: peer="{Voter server-98ec5df1-c470-d517-1e27-d2ade02cd320 98ec5df1-c470-d517-1e27-d2ade02cd320}" 2025-09-14T21:48:28.173Z [ERROR] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: failed to get log: index=3 error="log not found" 2025-09-14T21:48:28.173Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: aborting pipeline replication: peer="{Voter server-28a659c2-c43c-6853-7088-2e882ff7c590 28a659c2-c43c-6853-7088-2e882ff7c590}" 2025-09-14T21:48:28.180Z [ERROR] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: failed to get log: index=3 error="log not found" 2025-09-14T21:48:28.180Z [INFO] creating new snapshot: path=/tmp/raft2364453357/snapshots/2-1013-1757886508180.tmp 2025-09-14T21:48:28.180Z [INFO] server-98ec5df1-c470-d517-1e27-d2ade02cd320: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:28.180Z [INFO] server-98ec5df1-c470-d517-1e27-d2ade02cd320: copied to local snapshot: bytes=71 2025-09-14T21:48:28.181Z [INFO] server-98ec5df1-c470-d517-1e27-d2ade02cd320: snapshot restore progress: id=2-1013-1757886508180 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:28.181Z [INFO] server-98ec5df1-c470-d517-1e27-d2ade02cd320: Installed remote snapshot 2025-09-14T21:48:28.181Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: pipelining replication: peer="{Voter server-98ec5df1-c470-d517-1e27-d2ade02cd320 98ec5df1-c470-d517-1e27-d2ade02cd320}" 2025-09-14T21:48:28.181Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:28.182Z [ERROR] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: failed to get log: index=3 error="log not found" 2025-09-14T21:48:28.182Z [INFO] creating new snapshot: path=/tmp/raft1116630578/snapshots/2-1013-1757886508182.tmp 2025-09-14T21:48:28.182Z [INFO] server-28a659c2-c43c-6853-7088-2e882ff7c590: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:28.183Z [INFO] server-28a659c2-c43c-6853-7088-2e882ff7c590: copied to local snapshot: bytes=71 2025-09-14T21:48:28.183Z [INFO] server-28a659c2-c43c-6853-7088-2e882ff7c590: snapshot restore progress: id=2-1013-1757886508182 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:28.183Z [INFO] server-28a659c2-c43c-6853-7088-2e882ff7c590: Installed remote snapshot 2025-09-14T21:48:28.184Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: pipelining replication: peer="{Voter server-28a659c2-c43c-6853-7088-2e882ff7c590 28a659c2-c43c-6853-7088-2e882ff7c590}" 2025-09-14T21:48:29.187Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:28.181356099 +0000 UTC m=+78.749234361 (2 nodes), 646ns from start of poll, 1.563553137s from cluster start. Timeout at 2025-09-14 21:48:29.186377369 +0000 UTC m=+79.754255632, 1.005021271s after stability 2025-09-14T21:48:29.199Z [INFO] server-6cc25a5c-92fe-bc79-86e5-4811f1de3763: aborting pipeline replication: peer="{Voter server-cd5126b9-81df-7b50-9a72-81d90c3bc9d1 cd5126b9-81df-7b50-9a72-81d90c3bc9d1}" 2025-09-14T21:48:29.199Z [INFO] server-6cc25a5c-92fe-bc79-86e5-4811f1de3763: aborting pipeline replication: peer="{Voter server-be462246-91fc-332c-c183-e21636e6c509 be462246-91fc-332c-c183-e21636e6c509}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true 2025-09-14T21:48:29.201Z [INFO] server-6d7826f8-74bb-7b60-923d-5df9211334d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d7826f8-74bb-7b60-923d-5df9211334d1 Address:6d7826f8-74bb-7b60-923d-5df9211334d1} {Suffrage:Voter ID:server-5653b578-3e8f-12f6-1fb9-9c88ea148add Address:5653b578-3e8f-12f6-1fb9-9c88ea148add} {Suffrage:Voter ID:server-02d49faf-9ea1-740a-4364-d0551d8fb5d0 Address:02d49faf-9ea1-740a-4364-d0551d8fb5d0}]" 2025-09-14T21:48:29.201Z [INFO] server-6d7826f8-74bb-7b60-923d-5df9211334d1: entering follower state: follower="Node at 6d7826f8-74bb-7b60-923d-5df9211334d1 [Follower]" leader-address= leader-id= 2025-09-14T21:48:29.201Z [INFO] server-5653b578-3e8f-12f6-1fb9-9c88ea148add: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d7826f8-74bb-7b60-923d-5df9211334d1 Address:6d7826f8-74bb-7b60-923d-5df9211334d1} {Suffrage:Voter ID:server-5653b578-3e8f-12f6-1fb9-9c88ea148add Address:5653b578-3e8f-12f6-1fb9-9c88ea148add} {Suffrage:Voter ID:server-02d49faf-9ea1-740a-4364-d0551d8fb5d0 Address:02d49faf-9ea1-740a-4364-d0551d8fb5d0}]" 2025-09-14T21:48:29.201Z [INFO] server-5653b578-3e8f-12f6-1fb9-9c88ea148add: entering follower state: follower="Node at 5653b578-3e8f-12f6-1fb9-9c88ea148add [Follower]" leader-address= leader-id= 2025-09-14T21:48:29.201Z [INFO] server-02d49faf-9ea1-740a-4364-d0551d8fb5d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d7826f8-74bb-7b60-923d-5df9211334d1 Address:6d7826f8-74bb-7b60-923d-5df9211334d1} {Suffrage:Voter ID:server-5653b578-3e8f-12f6-1fb9-9c88ea148add Address:5653b578-3e8f-12f6-1fb9-9c88ea148add} {Suffrage:Voter ID:server-02d49faf-9ea1-740a-4364-d0551d8fb5d0 Address:02d49faf-9ea1-740a-4364-d0551d8fb5d0}]" 2025-09-14T21:48:29.201Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:29.201Z [INFO] server-02d49faf-9ea1-740a-4364-d0551d8fb5d0: entering follower state: follower="Node at 02d49faf-9ea1-740a-4364-d0551d8fb5d0 [Follower]" leader-address= leader-id= 2025-09-14T21:48:29.738Z [WARN] server-5653b578-3e8f-12f6-1fb9-9c88ea148add: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:29.738Z [INFO] server-5653b578-3e8f-12f6-1fb9-9c88ea148add: entering candidate state: node="Node at 5653b578-3e8f-12f6-1fb9-9c88ea148add [Candidate]" term=2 2025-09-14T21:48:29.738Z [INFO] server-5653b578-3e8f-12f6-1fb9-9c88ea148add: election won: term=2 tally=2 2025-09-14T21:48:29.738Z [INFO] server-5653b578-3e8f-12f6-1fb9-9c88ea148add: entering leader state: leader="Node at 5653b578-3e8f-12f6-1fb9-9c88ea148add [Leader]" 2025-09-14T21:48:29.738Z [INFO] server-5653b578-3e8f-12f6-1fb9-9c88ea148add: added peer, starting replication: peer=server-6d7826f8-74bb-7b60-923d-5df9211334d1 2025-09-14T21:48:29.738Z [INFO] server-5653b578-3e8f-12f6-1fb9-9c88ea148add: added peer, starting replication: peer=server-02d49faf-9ea1-740a-4364-d0551d8fb5d0 2025-09-14T21:48:29.739Z [INFO] server-5653b578-3e8f-12f6-1fb9-9c88ea148add: pipelining replication: peer="{Voter server-02d49faf-9ea1-740a-4364-d0551d8fb5d0 02d49faf-9ea1-740a-4364-d0551d8fb5d0}" 2025-09-14T21:48:29.739Z [INFO] server-5653b578-3e8f-12f6-1fb9-9c88ea148add: pipelining replication: peer="{Voter server-6d7826f8-74bb-7b60-923d-5df9211334d1 6d7826f8-74bb-7b60-923d-5df9211334d1}" 2025-09-14T21:48:30.744Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:29.73896162 +0000 UTC m=+80.306839896 (1 nodes), 537.482082ms from start of poll, 537.852212ms from cluster start. Timeout at 2025-09-14 21:48:30.743964933 +0000 UTC m=+81.311843195, 1.005003299s after stability 2025-09-14T21:48:30.746Z [INFO] server-5653b578-3e8f-12f6-1fb9-9c88ea148add: starting snapshot up to: index=12 2025-09-14T21:48:30.746Z [INFO] creating new snapshot: path=/tmp/raft1074807996/snapshots/2-12-1757886510746.tmp 2025-09-14T21:48:30.746Z [INFO] server-5653b578-3e8f-12f6-1fb9-9c88ea148add: snapshot complete up to: index=12 2025-09-14T21:48:30.747Z [INFO] server-02f57d5d-94ac-7a0a-305c-32f8dd22611a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02f57d5d-94ac-7a0a-305c-32f8dd22611a Address:02f57d5d-94ac-7a0a-305c-32f8dd22611a} {Suffrage:Voter ID:server-432c2a69-a834-cccd-2e05-e7e651ad57e5 Address:432c2a69-a834-cccd-2e05-e7e651ad57e5} {Suffrage:Voter ID:server-e5d8d4fd-978d-4462-6b14-1beb75b6d575 Address:e5d8d4fd-978d-4462-6b14-1beb75b6d575}]" 2025-09-14T21:48:30.747Z [INFO] server-02f57d5d-94ac-7a0a-305c-32f8dd22611a: entering follower state: follower="Node at 02f57d5d-94ac-7a0a-305c-32f8dd22611a [Follower]" leader-address= leader-id= 2025-09-14T21:48:30.747Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02f57d5d-94ac-7a0a-305c-32f8dd22611a Address:02f57d5d-94ac-7a0a-305c-32f8dd22611a} {Suffrage:Voter ID:server-432c2a69-a834-cccd-2e05-e7e651ad57e5 Address:432c2a69-a834-cccd-2e05-e7e651ad57e5} {Suffrage:Voter ID:server-e5d8d4fd-978d-4462-6b14-1beb75b6d575 Address:e5d8d4fd-978d-4462-6b14-1beb75b6d575}]" 2025-09-14T21:48:30.747Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: entering follower state: follower="Node at 432c2a69-a834-cccd-2e05-e7e651ad57e5 [Follower]" leader-address= leader-id= 2025-09-14T21:48:30.747Z [INFO] server-e5d8d4fd-978d-4462-6b14-1beb75b6d575: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02f57d5d-94ac-7a0a-305c-32f8dd22611a Address:02f57d5d-94ac-7a0a-305c-32f8dd22611a} {Suffrage:Voter ID:server-432c2a69-a834-cccd-2e05-e7e651ad57e5 Address:432c2a69-a834-cccd-2e05-e7e651ad57e5} {Suffrage:Voter ID:server-e5d8d4fd-978d-4462-6b14-1beb75b6d575 Address:e5d8d4fd-978d-4462-6b14-1beb75b6d575}]" 2025-09-14T21:48:30.747Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:30.748Z [INFO] server-e5d8d4fd-978d-4462-6b14-1beb75b6d575: entering follower state: follower="Node at e5d8d4fd-978d-4462-6b14-1beb75b6d575 [Follower]" leader-address= leader-id= 2025-09-14T21:48:31.305Z [WARN] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:31.305Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: entering candidate state: node="Node at 432c2a69-a834-cccd-2e05-e7e651ad57e5 [Candidate]" term=2 2025-09-14T21:48:31.305Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: election won: term=2 tally=2 2025-09-14T21:48:31.305Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: entering leader state: leader="Node at 432c2a69-a834-cccd-2e05-e7e651ad57e5 [Leader]" 2025-09-14T21:48:31.305Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: added peer, starting replication: peer=server-02f57d5d-94ac-7a0a-305c-32f8dd22611a 2025-09-14T21:48:31.305Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: added peer, starting replication: peer=server-e5d8d4fd-978d-4462-6b14-1beb75b6d575 2025-09-14T21:48:31.305Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: pipelining replication: peer="{Voter server-e5d8d4fd-978d-4462-6b14-1beb75b6d575 e5d8d4fd-978d-4462-6b14-1beb75b6d575}" 2025-09-14T21:48:31.305Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: pipelining replication: peer="{Voter server-02f57d5d-94ac-7a0a-305c-32f8dd22611a 02f57d5d-94ac-7a0a-305c-32f8dd22611a}" 2025-09-14T21:48:32.310Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:31.305204689 +0000 UTC m=+81.873082948 (1 nodes), 557.189401ms from start of poll, 557.503318ms from cluster start. Timeout at 2025-09-14 21:48:32.310207164 +0000 UTC m=+82.878085430, 1.005002482s after stability 2025-09-14T21:48:32.311Z [INFO] creating new snapshot: path=/tmp/raft739321519/snapshots/2-1013-1757886512311.tmp 2025-09-14T21:48:32.312Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: copied to local snapshot: bytes=71 2025-09-14T21:48:32.312Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: snapshot restore progress: id=2-1013-1757886512311 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:32.312Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: removing all old logs from log store 2025-09-14T21:48:32.312Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: compacting logs: from=1 to=2 2025-09-14T21:48:32.312Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: restored user snapshot: index=1013 2025-09-14T21:48:32.312Z [ERROR] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: failed to get log: index=2 error="log not found" 2025-09-14T21:48:32.312Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: aborting pipeline replication: peer="{Voter server-e5d8d4fd-978d-4462-6b14-1beb75b6d575 e5d8d4fd-978d-4462-6b14-1beb75b6d575}" 2025-09-14T21:48:32.312Z [ERROR] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: failed to get log: index=2 error="log not found" 2025-09-14T21:48:32.313Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: aborting pipeline replication: peer="{Voter server-02f57d5d-94ac-7a0a-305c-32f8dd22611a 02f57d5d-94ac-7a0a-305c-32f8dd22611a}" 2025-09-14T21:48:32.322Z [ERROR] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: failed to get log: index=2 error="log not found" 2025-09-14T21:48:32.322Z [INFO] creating new snapshot: path=/tmp/raft1814314280/snapshots/2-1013-1757886512322.tmp 2025-09-14T21:48:32.322Z [INFO] server-e5d8d4fd-978d-4462-6b14-1beb75b6d575: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:32.322Z [INFO] server-e5d8d4fd-978d-4462-6b14-1beb75b6d575: copied to local snapshot: bytes=71 2025-09-14T21:48:32.322Z [ERROR] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: failed to get log: index=2 error="log not found" 2025-09-14T21:48:32.323Z [INFO] server-e5d8d4fd-978d-4462-6b14-1beb75b6d575: snapshot restore progress: id=2-1013-1757886512322 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:32.323Z [INFO] server-e5d8d4fd-978d-4462-6b14-1beb75b6d575: removing all old logs from log store 2025-09-14T21:48:32.323Z [INFO] server-e5d8d4fd-978d-4462-6b14-1beb75b6d575: compacting logs: from=1 to=2 2025-09-14T21:48:32.323Z [INFO] server-e5d8d4fd-978d-4462-6b14-1beb75b6d575: Installed remote snapshot 2025-09-14T21:48:32.323Z [INFO] creating new snapshot: path=/tmp/raft1714456607/snapshots/2-1013-1757886512323.tmp 2025-09-14T21:48:32.323Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: pipelining replication: peer="{Voter server-e5d8d4fd-978d-4462-6b14-1beb75b6d575 e5d8d4fd-978d-4462-6b14-1beb75b6d575}" 2025-09-14T21:48:32.323Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:32.323Z [INFO] server-02f57d5d-94ac-7a0a-305c-32f8dd22611a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:32.323Z [INFO] server-02f57d5d-94ac-7a0a-305c-32f8dd22611a: copied to local snapshot: bytes=71 2025-09-14T21:48:32.323Z [INFO] server-02f57d5d-94ac-7a0a-305c-32f8dd22611a: snapshot restore progress: id=2-1013-1757886512323 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:32.323Z [INFO] server-02f57d5d-94ac-7a0a-305c-32f8dd22611a: removing all old logs from log store 2025-09-14T21:48:32.323Z [INFO] server-02f57d5d-94ac-7a0a-305c-32f8dd22611a: compacting logs: from=1 to=2 2025-09-14T21:48:32.323Z [INFO] server-02f57d5d-94ac-7a0a-305c-32f8dd22611a: Installed remote snapshot 2025-09-14T21:48:32.323Z [INFO] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: pipelining replication: peer="{Voter server-02f57d5d-94ac-7a0a-305c-32f8dd22611a 02f57d5d-94ac-7a0a-305c-32f8dd22611a}" 2025-09-14T21:48:33.329Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:32.323273596 +0000 UTC m=+82.891151873 (2 nodes), 876ns from start of poll, 1.575572243s from cluster start. Timeout at 2025-09-14 21:48:33.328290936 +0000 UTC m=+83.896169217, 1.005017344s after stability 2025-09-14T21:48:33.340Z [INFO] server-5653b578-3e8f-12f6-1fb9-9c88ea148add: aborting pipeline replication: peer="{Voter server-02d49faf-9ea1-740a-4364-d0551d8fb5d0 02d49faf-9ea1-740a-4364-d0551d8fb5d0}" 2025-09-14T21:48:33.340Z [INFO] server-5653b578-3e8f-12f6-1fb9-9c88ea148add: aborting pipeline replication: peer="{Voter server-6d7826f8-74bb-7b60-923d-5df9211334d1 6d7826f8-74bb-7b60-923d-5df9211334d1}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false 2025-09-14T21:48:33.342Z [INFO] server-92bdfc5d-101a-eb78-cf17-f708e8c583aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92bdfc5d-101a-eb78-cf17-f708e8c583aa Address:92bdfc5d-101a-eb78-cf17-f708e8c583aa} {Suffrage:Voter ID:server-63b1da85-619c-89c3-4d72-374118b047a8 Address:63b1da85-619c-89c3-4d72-374118b047a8} {Suffrage:Voter ID:server-c95abc0d-e530-c8dc-3879-408b10353f84 Address:c95abc0d-e530-c8dc-3879-408b10353f84}]" 2025-09-14T21:48:33.342Z [INFO] server-92bdfc5d-101a-eb78-cf17-f708e8c583aa: entering follower state: follower="Node at 92bdfc5d-101a-eb78-cf17-f708e8c583aa [Follower]" leader-address= leader-id= 2025-09-14T21:48:33.342Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92bdfc5d-101a-eb78-cf17-f708e8c583aa Address:92bdfc5d-101a-eb78-cf17-f708e8c583aa} {Suffrage:Voter ID:server-63b1da85-619c-89c3-4d72-374118b047a8 Address:63b1da85-619c-89c3-4d72-374118b047a8} {Suffrage:Voter ID:server-c95abc0d-e530-c8dc-3879-408b10353f84 Address:c95abc0d-e530-c8dc-3879-408b10353f84}]" 2025-09-14T21:48:33.342Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: entering follower state: follower="Node at 63b1da85-619c-89c3-4d72-374118b047a8 [Follower]" leader-address= leader-id= 2025-09-14T21:48:33.343Z [INFO] server-c95abc0d-e530-c8dc-3879-408b10353f84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92bdfc5d-101a-eb78-cf17-f708e8c583aa Address:92bdfc5d-101a-eb78-cf17-f708e8c583aa} {Suffrage:Voter ID:server-63b1da85-619c-89c3-4d72-374118b047a8 Address:63b1da85-619c-89c3-4d72-374118b047a8} {Suffrage:Voter ID:server-c95abc0d-e530-c8dc-3879-408b10353f84 Address:c95abc0d-e530-c8dc-3879-408b10353f84}]" 2025-09-14T21:48:33.343Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:33.343Z [INFO] server-c95abc0d-e530-c8dc-3879-408b10353f84: entering follower state: follower="Node at c95abc0d-e530-c8dc-3879-408b10353f84 [Follower]" leader-address= leader-id= 2025-09-14T21:48:33.906Z [WARN] server-63b1da85-619c-89c3-4d72-374118b047a8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:33.906Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: entering candidate state: node="Node at 63b1da85-619c-89c3-4d72-374118b047a8 [Candidate]" term=2 2025-09-14T21:48:33.906Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: election won: term=2 tally=2 2025-09-14T21:48:33.906Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: entering leader state: leader="Node at 63b1da85-619c-89c3-4d72-374118b047a8 [Leader]" 2025-09-14T21:48:33.906Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: added peer, starting replication: peer=server-92bdfc5d-101a-eb78-cf17-f708e8c583aa 2025-09-14T21:48:33.906Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: added peer, starting replication: peer=server-c95abc0d-e530-c8dc-3879-408b10353f84 2025-09-14T21:48:33.906Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: pipelining replication: peer="{Voter server-c95abc0d-e530-c8dc-3879-408b10353f84 c95abc0d-e530-c8dc-3879-408b10353f84}" 2025-09-14T21:48:33.906Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: pipelining replication: peer="{Voter server-92bdfc5d-101a-eb78-cf17-f708e8c583aa 92bdfc5d-101a-eb78-cf17-f708e8c583aa}" 2025-09-14T21:48:34.912Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:33.906555969 +0000 UTC m=+84.474434227 (1 nodes), 563.471475ms from start of poll, 564.197387ms from cluster start. Timeout at 2025-09-14 21:48:34.911621618 +0000 UTC m=+85.479499877, 1.00506565s after stability 2025-09-14T21:48:34.913Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: starting snapshot up to: index=12 2025-09-14T21:48:34.913Z [INFO] creating new snapshot: path=/tmp/raft2889300730/snapshots/2-12-1757886514913.tmp 2025-09-14T21:48:34.914Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: snapshot complete up to: index=12 2025-09-14T21:48:34.915Z [INFO] creating new snapshot: path=/tmp/raft2889300730/snapshots/2-10013-1757886514915.tmp 2025-09-14T21:48:34.916Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: copied to local snapshot: bytes=71 2025-09-14T21:48:34.916Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: snapshot restore progress: id=2-10013-1757886514915 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:34.916Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: restored user snapshot: index=10013 2025-09-14T21:48:34.916Z [ERROR] server-63b1da85-619c-89c3-4d72-374118b047a8: failed to get log: index=23 error="log not found" 2025-09-14T21:48:34.916Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: aborting pipeline replication: peer="{Voter server-92bdfc5d-101a-eb78-cf17-f708e8c583aa 92bdfc5d-101a-eb78-cf17-f708e8c583aa}" 2025-09-14T21:48:34.916Z [ERROR] server-63b1da85-619c-89c3-4d72-374118b047a8: failed to get log: index=23 error="log not found" 2025-09-14T21:48:34.916Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: aborting pipeline replication: peer="{Voter server-c95abc0d-e530-c8dc-3879-408b10353f84 c95abc0d-e530-c8dc-3879-408b10353f84}" 2025-09-14T21:48:34.923Z [ERROR] server-63b1da85-619c-89c3-4d72-374118b047a8: failed to get log: index=23 error="log not found" 2025-09-14T21:48:34.923Z [INFO] creating new snapshot: path=/tmp/raft2403558468/snapshots/2-10013-1757886514923.tmp 2025-09-14T21:48:34.923Z [INFO] server-92bdfc5d-101a-eb78-cf17-f708e8c583aa: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:34.924Z [INFO] server-92bdfc5d-101a-eb78-cf17-f708e8c583aa: copied to local snapshot: bytes=71 2025-09-14T21:48:34.924Z [INFO] server-92bdfc5d-101a-eb78-cf17-f708e8c583aa: snapshot restore progress: id=2-10013-1757886514923 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:34.924Z [INFO] server-92bdfc5d-101a-eb78-cf17-f708e8c583aa: Installed remote snapshot 2025-09-14T21:48:34.924Z [WARN] server-92bdfc5d-101a-eb78-cf17-f708e8c583aa: skipping application of old log: index=22 2025-09-14T21:48:34.924Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: pipelining replication: peer="{Voter server-92bdfc5d-101a-eb78-cf17-f708e8c583aa 92bdfc5d-101a-eb78-cf17-f708e8c583aa}" 2025-09-14T21:48:34.924Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:34.926Z [ERROR] server-63b1da85-619c-89c3-4d72-374118b047a8: failed to get log: index=23 error="log not found" 2025-09-14T21:48:34.927Z [INFO] creating new snapshot: path=/tmp/raft2543509502/snapshots/2-10013-1757886514927.tmp 2025-09-14T21:48:34.927Z [INFO] server-c95abc0d-e530-c8dc-3879-408b10353f84: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:34.928Z [INFO] server-c95abc0d-e530-c8dc-3879-408b10353f84: copied to local snapshot: bytes=71 2025-09-14T21:48:34.928Z [INFO] server-c95abc0d-e530-c8dc-3879-408b10353f84: snapshot restore progress: id=2-10013-1757886514927 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:34.928Z [INFO] server-c95abc0d-e530-c8dc-3879-408b10353f84: Installed remote snapshot 2025-09-14T21:48:34.928Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: pipelining replication: peer="{Voter server-c95abc0d-e530-c8dc-3879-408b10353f84 c95abc0d-e530-c8dc-3879-408b10353f84}" 2025-09-14T21:48:35.929Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:34.924552236 +0000 UTC m=+85.492430500 (2 nodes), 809ns from start of poll, 1.58219366s from cluster start. Timeout at 2025-09-14 21:48:35.929554686 +0000 UTC m=+86.497432946, 1.005002446s after stability 2025-09-14T21:48:35.941Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: aborting pipeline replication: peer="{Voter server-c95abc0d-e530-c8dc-3879-408b10353f84 c95abc0d-e530-c8dc-3879-408b10353f84}" 2025-09-14T21:48:35.941Z [INFO] server-63b1da85-619c-89c3-4d72-374118b047a8: aborting pipeline replication: peer="{Voter server-92bdfc5d-101a-eb78-cf17-f708e8c583aa 92bdfc5d-101a-eb78-cf17-f708e8c583aa}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false 2025-09-14T21:48:35.942Z [INFO] server-33f671c9-1fd8-c9b7-6221-acd863bf890d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33f671c9-1fd8-c9b7-6221-acd863bf890d Address:33f671c9-1fd8-c9b7-6221-acd863bf890d} {Suffrage:Voter ID:server-075d40c6-b389-5070-3032-314319ad8676 Address:075d40c6-b389-5070-3032-314319ad8676} {Suffrage:Voter ID:server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96 Address:4cc0be23-95d0-4b72-ca5a-75046ddf7d96}]" 2025-09-14T21:48:35.943Z [INFO] server-075d40c6-b389-5070-3032-314319ad8676: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33f671c9-1fd8-c9b7-6221-acd863bf890d Address:33f671c9-1fd8-c9b7-6221-acd863bf890d} {Suffrage:Voter ID:server-075d40c6-b389-5070-3032-314319ad8676 Address:075d40c6-b389-5070-3032-314319ad8676} {Suffrage:Voter ID:server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96 Address:4cc0be23-95d0-4b72-ca5a-75046ddf7d96}]" 2025-09-14T21:48:35.943Z [INFO] server-33f671c9-1fd8-c9b7-6221-acd863bf890d: entering follower state: follower="Node at 33f671c9-1fd8-c9b7-6221-acd863bf890d [Follower]" leader-address= leader-id= 2025-09-14T21:48:35.943Z [INFO] server-075d40c6-b389-5070-3032-314319ad8676: entering follower state: follower="Node at 075d40c6-b389-5070-3032-314319ad8676 [Follower]" leader-address= leader-id= 2025-09-14T21:48:35.943Z [INFO] server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33f671c9-1fd8-c9b7-6221-acd863bf890d Address:33f671c9-1fd8-c9b7-6221-acd863bf890d} {Suffrage:Voter ID:server-075d40c6-b389-5070-3032-314319ad8676 Address:075d40c6-b389-5070-3032-314319ad8676} {Suffrage:Voter ID:server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96 Address:4cc0be23-95d0-4b72-ca5a-75046ddf7d96}]" 2025-09-14T21:48:35.943Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:35.943Z [INFO] server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96: entering follower state: follower="Node at 4cc0be23-95d0-4b72-ca5a-75046ddf7d96 [Follower]" leader-address= leader-id= 2025-09-14T21:48:36.518Z [WARN] server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:36.518Z [INFO] server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96: entering candidate state: node="Node at 4cc0be23-95d0-4b72-ca5a-75046ddf7d96 [Candidate]" term=2 2025-09-14T21:48:36.518Z [INFO] server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96: election won: term=2 tally=2 2025-09-14T21:48:36.518Z [INFO] server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96: entering leader state: leader="Node at 4cc0be23-95d0-4b72-ca5a-75046ddf7d96 [Leader]" 2025-09-14T21:48:36.518Z [INFO] server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96: added peer, starting replication: peer=server-33f671c9-1fd8-c9b7-6221-acd863bf890d 2025-09-14T21:48:36.518Z [INFO] server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96: added peer, starting replication: peer=server-075d40c6-b389-5070-3032-314319ad8676 2025-09-14T21:48:36.519Z [INFO] server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96: pipelining replication: peer="{Voter server-33f671c9-1fd8-c9b7-6221-acd863bf890d 33f671c9-1fd8-c9b7-6221-acd863bf890d}" 2025-09-14T21:48:36.519Z [INFO] server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96: pipelining replication: peer="{Voter server-075d40c6-b389-5070-3032-314319ad8676 075d40c6-b389-5070-3032-314319ad8676}" 2025-09-14T21:48:38.518Z [WARN] server-432c2a69-a834-cccd-2e05-e7e651ad57e5: failed to contact: server-id=server-e5d8d4fd-978d-4462-6b14-1beb75b6d575 time=1.286185438s 2025-09-14T21:48:38.520Z [WARN] server-44db6509-c2cc-56cf-9d4b-a7b59633c136: heartbeat timeout reached, starting election: last-leader-addr=564c2386-ae8b-2d8e-9fe0-1a34ecb724ec last-leader-id=server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec 2025-09-14T21:48:38.520Z [WARN] server-3bf73f50-a68b-9cf9-d150-22c060633bc6: failed to contact: server-id=server-0aacb753-bf82-99e4-23f3-da017a22d60d time=1.299874502s 2025-09-14T21:48:38.520Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:36.519012913 +0000 UTC m=+87.086891192 (1 nodes), 575.859455ms from start of poll, 576.163194ms from cluster start. Timeout at 2025-09-14 21:48:37.524213311 +0000 UTC m=+88.092091582, 1.00520039s after stability 2025-09-14T21:48:38.520Z [WARN] server-d66acbe9-5343-4b20-8ad5-0c2578394362: heartbeat timeout reached, starting election: last-leader-addr=eddd436f-b70e-2d91-413d-42509e6fafe2 last-leader-id=server-eddd436f-b70e-2d91-413d-42509e6fafe2 2025-09-14T21:48:38.520Z [WARN] server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96: failed to contact: server-id=server-33f671c9-1fd8-c9b7-6221-acd863bf890d time=1.29218593s 2025-09-14T21:48:38.520Z [INFO] server-44db6509-c2cc-56cf-9d4b-a7b59633c136: entering candidate state: node="Node at 44db6509-c2cc-56cf-9d4b-a7b59633c136 [Candidate]" term=3 2025-09-14T21:48:38.520Z [WARN] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: failed to contact: server-id=server-a104f40c-041d-fb46-7da0-43daf7524697 time=1.29461876s 2025-09-14T21:48:38.520Z [WARN] server-38971478-809c-95e2-3b17-c84e9880db93: failed to contact: server-id=server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c time=1.296250004s 2025-09-14T21:48:38.520Z [WARN] server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c: heartbeat timeout reached, starting election: last-leader-addr=38971478-809c-95e2-3b17-c84e9880db93 last-leader-id=server-38971478-809c-95e2-3b17-c84e9880db93 2025-09-14T21:48:38.520Z [WARN] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: failed to contact: server-id=server-98ec5df1-c470-d517-1e27-d2ade02cd320 time=1.301148859s 2025-09-14T21:48:38.520Z [WARN] server-33f671c9-1fd8-c9b7-6221-acd863bf890d: heartbeat timeout reached, starting election: last-leader-addr=4cc0be23-95d0-4b72-ca5a-75046ddf7d96 last-leader-id=server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96 2025-09-14T21:48:38.520Z [INFO] server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c: entering candidate state: node="Node at 915e7e29-8f69-5b4f-b2a7-860c5804ae3c [Candidate]" term=3 2025-09-14T21:48:38.520Z [INFO] server-d66acbe9-5343-4b20-8ad5-0c2578394362: entering candidate state: node="Node at d66acbe9-5343-4b20-8ad5-0c2578394362 [Candidate]" term=3 2025-09-14T21:48:38.520Z [ERROR] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: peer has newer term, stopping replication: peer="{Voter server-44db6509-c2cc-56cf-9d4b-a7b59633c136 44db6509-c2cc-56cf-9d4b-a7b59633c136}" 2025-09-14T21:48:38.520Z [WARN] server-4a406d95-6e56-e981-2e80-6c0b7ec318b5: rejecting vote request since we have a leader: from=d66acbe9-5343-4b20-8ad5-0c2578394362 leader=eddd436f-b70e-2d91-413d-42509e6fafe2 leader-id=server-eddd436f-b70e-2d91-413d-42509e6fafe2 2025-09-14T21:48:38.520Z [INFO] server-33f671c9-1fd8-c9b7-6221-acd863bf890d: entering candidate state: node="Node at 33f671c9-1fd8-c9b7-6221-acd863bf890d [Candidate]" term=3 2025-09-14T21:48:38.520Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: aborting pipeline replication: peer="{Voter server-44db6509-c2cc-56cf-9d4b-a7b59633c136 44db6509-c2cc-56cf-9d4b-a7b59633c136}" 2025-09-14T21:48:38.520Z [WARN] server-a45966cf-4970-6e3e-0762-a695f6366ab8: rejecting vote request since we have a leader: from=44db6509-c2cc-56cf-9d4b-a7b59633c136 leader=564c2386-ae8b-2d8e-9fe0-1a34ecb724ec leader-id=server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec 2025-09-14T21:48:38.520Z [ERROR] server-38971478-809c-95e2-3b17-c84e9880db93: peer has newer term, stopping replication: peer="{Voter server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c 915e7e29-8f69-5b4f-b2a7-860c5804ae3c}" 2025-09-14T21:48:38.520Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: aborting pipeline replication: peer="{Voter server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c 915e7e29-8f69-5b4f-b2a7-860c5804ae3c}" 2025-09-14T21:48:38.520Z [WARN] server-38971478-809c-95e2-3b17-c84e9880db93: rejecting vote request since we have a leader: from=915e7e29-8f69-5b4f-b2a7-860c5804ae3c leader=38971478-809c-95e2-3b17-c84e9880db93 leader-id=server-38971478-809c-95e2-3b17-c84e9880db93 2025-09-14T21:48:38.520Z [WARN] server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96: rejecting vote request since we have a leader: from=33f671c9-1fd8-c9b7-6221-acd863bf890d leader=4cc0be23-95d0-4b72-ca5a-75046ddf7d96 leader-id=server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96 2025-09-14T21:48:38.520Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: entering follower state: follower="Node at 564c2386-ae8b-2d8e-9fe0-1a34ecb724ec [Follower]" leader-address= leader-id= 2025-09-14T21:48:38.520Z [WARN] server-075d40c6-b389-5070-3032-314319ad8676: rejecting vote request since we have a leader: from=33f671c9-1fd8-c9b7-6221-acd863bf890d leader=4cc0be23-95d0-4b72-ca5a-75046ddf7d96 leader-id=server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96 2025-09-14T21:48:38.520Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: entering follower state: follower="Node at 38971478-809c-95e2-3b17-c84e9880db93 [Follower]" leader-address= leader-id= 2025-09-14T21:48:38.520Z [WARN] server-d3a15db6-442f-1c38-0601-eab09266c438: rejecting vote request since we have a leader: from=915e7e29-8f69-5b4f-b2a7-860c5804ae3c leader=38971478-809c-95e2-3b17-c84e9880db93 leader-id=server-38971478-809c-95e2-3b17-c84e9880db93 2025-09-14T21:48:38.520Z [INFO] server-44db6509-c2cc-56cf-9d4b-a7b59633c136: election won: term=3 tally=2 2025-09-14T21:48:38.520Z [WARN] server-a104f40c-041d-fb46-7da0-43daf7524697: heartbeat timeout reached, starting election: last-leader-addr=e923a74e-f617-8ead-eabc-3a1a475f20d2 last-leader-id=server-e923a74e-f617-8ead-eabc-3a1a475f20d2 2025-09-14T21:48:38.518Z [WARN] server-eddd436f-b70e-2d91-413d-42509e6fafe2: failed to contact: server-id=server-d66acbe9-5343-4b20-8ad5-0c2578394362 time=1.303296552s 2025-09-14T21:48:38.520Z [INFO] server-44db6509-c2cc-56cf-9d4b-a7b59633c136: entering leader state: leader="Node at 44db6509-c2cc-56cf-9d4b-a7b59633c136 [Leader]" 2025-09-14T21:48:38.519Z [ERROR] server-eddd436f-b70e-2d91-413d-42509e6fafe2: failed to heartbeat to: peer=d66acbe9-5343-4b20-8ad5-0c2578394362 backoff time=10ms error="command timed out" 2025-09-14T21:48:38.520Z [INFO] server-a104f40c-041d-fb46-7da0-43daf7524697: entering candidate state: node="Node at a104f40c-041d-fb46-7da0-43daf7524697 [Candidate]" term=3 2025-09-14T21:48:38.520Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: aborting pipeline replication: peer="{Voter server-d66acbe9-5343-4b20-8ad5-0c2578394362 d66acbe9-5343-4b20-8ad5-0c2578394362}" 2025-09-14T21:48:38.520Z [INFO] server-44db6509-c2cc-56cf-9d4b-a7b59633c136: added peer, starting replication: peer=server-a45966cf-4970-6e3e-0762-a695f6366ab8 2025-09-14T21:48:38.520Z [ERROR] server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96: peer has newer term, stopping replication: peer="{Voter server-33f671c9-1fd8-c9b7-6221-acd863bf890d 33f671c9-1fd8-c9b7-6221-acd863bf890d}" 2025-09-14T21:48:38.520Z [INFO] server-38971478-809c-95e2-3b17-c84e9880db93: aborting pipeline replication: peer="{Voter server-d3a15db6-442f-1c38-0601-eab09266c438 d3a15db6-442f-1c38-0601-eab09266c438}" 2025-09-14T21:48:38.520Z [INFO] server-44db6509-c2cc-56cf-9d4b-a7b59633c136: added peer, starting replication: peer=server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec 2025-09-14T21:48:38.521Z [WARN] server-eddd436f-b70e-2d91-413d-42509e6fafe2: rejecting vote request since we have a leader: from=d66acbe9-5343-4b20-8ad5-0c2578394362 leader=eddd436f-b70e-2d91-413d-42509e6fafe2 leader-id=server-eddd436f-b70e-2d91-413d-42509e6fafe2 2025-09-14T21:48:38.521Z [INFO] server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96: aborting pipeline replication: peer="{Voter server-33f671c9-1fd8-c9b7-6221-acd863bf890d 33f671c9-1fd8-c9b7-6221-acd863bf890d}" 2025-09-14T21:48:38.521Z [INFO] server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96: entering follower state: follower="Node at 4cc0be23-95d0-4b72-ca5a-75046ddf7d96 [Follower]" leader-address= leader-id= 2025-09-14T21:48:38.521Z [INFO] server-44db6509-c2cc-56cf-9d4b-a7b59633c136: pipelining replication: peer="{Voter server-a45966cf-4970-6e3e-0762-a695f6366ab8 a45966cf-4970-6e3e-0762-a695f6366ab8}" 2025-09-14T21:48:38.521Z [INFO] server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96: aborting pipeline replication: peer="{Voter server-075d40c6-b389-5070-3032-314319ad8676 075d40c6-b389-5070-3032-314319ad8676}" 2025-09-14T21:48:38.521Z [ERROR] server-4cc0be23-95d0-4b72-ca5a-75046ddf7d96: peer has newer term, stopping replication: peer="{Voter server-33f671c9-1fd8-c9b7-6221-acd863bf890d 33f671c9-1fd8-c9b7-6221-acd863bf890d}" raft_test.go:1598: Error Apply new log entries: node is not the leader 2025-09-14T21:48:38.521Z [ERROR] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: peer has newer term, stopping replication: peer="{Voter server-a104f40c-041d-fb46-7da0-43daf7524697 a104f40c-041d-fb46-7da0-43daf7524697}" 2025-09-14T21:48:38.521Z [INFO] server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec: aborting pipeline replication: peer="{Voter server-a45966cf-4970-6e3e-0762-a695f6366ab8 a45966cf-4970-6e3e-0762-a695f6366ab8}" 2025-09-14T21:48:38.521Z [WARN] server-e0ff53b6-3af5-8b51-d81b-f08fd0a7e727: rejecting vote request since we have a leader: from=a104f40c-041d-fb46-7da0-43daf7524697 leader=e923a74e-f617-8ead-eabc-3a1a475f20d2 leader-id=server-e923a74e-f617-8ead-eabc-3a1a475f20d2 2025-09-14T21:48:38.521Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: aborting pipeline replication: peer="{Voter server-a104f40c-041d-fb46-7da0-43daf7524697 a104f40c-041d-fb46-7da0-43daf7524697}" 2025-09-14T21:48:38.521Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: entering follower state: follower="Node at e923a74e-f617-8ead-eabc-3a1a475f20d2 [Follower]" leader-address= leader-id= 2025-09-14T21:48:38.521Z [INFO] server-e923a74e-f617-8ead-eabc-3a1a475f20d2: aborting pipeline replication: peer="{Voter server-e0ff53b6-3af5-8b51-d81b-f08fd0a7e727 e0ff53b6-3af5-8b51-d81b-f08fd0a7e727}" 2025-09-14T21:48:38.521Z [INFO] server-44db6509-c2cc-56cf-9d4b-a7b59633c136: pipelining replication: peer="{Voter server-564c2386-ae8b-2d8e-9fe0-1a34ecb724ec 564c2386-ae8b-2d8e-9fe0-1a34ecb724ec}" 2025-09-14T21:48:38.521Z [WARN] server-98ec5df1-c470-d517-1e27-d2ade02cd320: heartbeat timeout reached, starting election: last-leader-addr=057ce2ab-8cb9-324e-cd5d-05a52ac7245e last-leader-id=server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e 2025-09-14T21:48:38.521Z [INFO] server-98ec5df1-c470-d517-1e27-d2ade02cd320: entering candidate state: node="Node at 98ec5df1-c470-d517-1e27-d2ade02cd320 [Candidate]" term=3 2025-09-14T21:48:38.521Z [ERROR] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: peer has newer term, stopping replication: peer="{Voter server-98ec5df1-c470-d517-1e27-d2ade02cd320 98ec5df1-c470-d517-1e27-d2ade02cd320}" 2025-09-14T21:48:38.521Z [WARN] server-28a659c2-c43c-6853-7088-2e882ff7c590: rejecting vote request since we have a leader: from=98ec5df1-c470-d517-1e27-d2ade02cd320 leader=057ce2ab-8cb9-324e-cd5d-05a52ac7245e leader-id=server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e 2025-09-14T21:48:38.521Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: aborting pipeline replication: peer="{Voter server-98ec5df1-c470-d517-1e27-d2ade02cd320 98ec5df1-c470-d517-1e27-d2ade02cd320}" 2025-09-14T21:48:38.521Z [INFO] server-a104f40c-041d-fb46-7da0-43daf7524697: election won: term=3 tally=2 2025-09-14T21:48:38.521Z [INFO] server-a104f40c-041d-fb46-7da0-43daf7524697: entering leader state: leader="Node at a104f40c-041d-fb46-7da0-43daf7524697 [Leader]" 2025-09-14T21:48:38.521Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: entering follower state: follower="Node at 057ce2ab-8cb9-324e-cd5d-05a52ac7245e [Follower]" leader-address= leader-id= 2025-09-14T21:48:38.521Z [INFO] server-a104f40c-041d-fb46-7da0-43daf7524697: added peer, starting replication: peer=server-e923a74e-f617-8ead-eabc-3a1a475f20d2 2025-09-14T21:48:38.521Z [INFO] server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e: aborting pipeline replication: peer="{Voter server-28a659c2-c43c-6853-7088-2e882ff7c590 28a659c2-c43c-6853-7088-2e882ff7c590}" 2025-09-14T21:48:38.521Z [INFO] server-a104f40c-041d-fb46-7da0-43daf7524697: added peer, starting replication: peer=server-e0ff53b6-3af5-8b51-d81b-f08fd0a7e727 2025-09-14T21:48:38.521Z [INFO] server-98ec5df1-c470-d517-1e27-d2ade02cd320: election won: term=3 tally=2 2025-09-14T21:48:38.521Z [INFO] server-98ec5df1-c470-d517-1e27-d2ade02cd320: entering leader state: leader="Node at 98ec5df1-c470-d517-1e27-d2ade02cd320 [Leader]" 2025-09-14T21:48:38.521Z [INFO] server-98ec5df1-c470-d517-1e27-d2ade02cd320: added peer, starting replication: peer=server-28a659c2-c43c-6853-7088-2e882ff7c590 2025-09-14T21:48:38.521Z [INFO] server-a104f40c-041d-fb46-7da0-43daf7524697: pipelining replication: peer="{Voter server-e923a74e-f617-8ead-eabc-3a1a475f20d2 e923a74e-f617-8ead-eabc-3a1a475f20d2}" 2025-09-14T21:48:38.521Z [INFO] server-98ec5df1-c470-d517-1e27-d2ade02cd320: added peer, starting replication: peer=server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e 2025-09-14T21:48:38.521Z [INFO] server-a104f40c-041d-fb46-7da0-43daf7524697: pipelining replication: peer="{Voter server-e0ff53b6-3af5-8b51-d81b-f08fd0a7e727 e0ff53b6-3af5-8b51-d81b-f08fd0a7e727}" 2025-09-14T21:48:38.521Z [INFO] server-98ec5df1-c470-d517-1e27-d2ade02cd320: pipelining replication: peer="{Voter server-28a659c2-c43c-6853-7088-2e882ff7c590 28a659c2-c43c-6853-7088-2e882ff7c590}" 2025-09-14T21:48:38.521Z [INFO] server-98ec5df1-c470-d517-1e27-d2ade02cd320: pipelining replication: peer="{Voter server-057ce2ab-8cb9-324e-cd5d-05a52ac7245e 057ce2ab-8cb9-324e-cd5d-05a52ac7245e}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true 2025-09-14T21:48:38.525Z [INFO] server-2fcf7719-e580-bd0c-8987-dfba587b88db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fcf7719-e580-bd0c-8987-dfba587b88db Address:2fcf7719-e580-bd0c-8987-dfba587b88db} {Suffrage:Voter ID:server-1e186be0-5a3e-bd2c-4588-099976a04aab Address:1e186be0-5a3e-bd2c-4588-099976a04aab} {Suffrage:Voter ID:server-9d317646-b62d-50df-3bb7-01b27b07e7ee Address:9d317646-b62d-50df-3bb7-01b27b07e7ee}]" 2025-09-14T21:48:38.525Z [INFO] server-2fcf7719-e580-bd0c-8987-dfba587b88db: entering follower state: follower="Node at 2fcf7719-e580-bd0c-8987-dfba587b88db [Follower]" leader-address= leader-id= 2025-09-14T21:48:38.525Z [INFO] server-1e186be0-5a3e-bd2c-4588-099976a04aab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fcf7719-e580-bd0c-8987-dfba587b88db Address:2fcf7719-e580-bd0c-8987-dfba587b88db} {Suffrage:Voter ID:server-1e186be0-5a3e-bd2c-4588-099976a04aab Address:1e186be0-5a3e-bd2c-4588-099976a04aab} {Suffrage:Voter ID:server-9d317646-b62d-50df-3bb7-01b27b07e7ee Address:9d317646-b62d-50df-3bb7-01b27b07e7ee}]" 2025-09-14T21:48:38.525Z [INFO] server-1e186be0-5a3e-bd2c-4588-099976a04aab: entering follower state: follower="Node at 1e186be0-5a3e-bd2c-4588-099976a04aab [Follower]" leader-address= leader-id= 2025-09-14T21:48:38.525Z [INFO] server-9d317646-b62d-50df-3bb7-01b27b07e7ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fcf7719-e580-bd0c-8987-dfba587b88db Address:2fcf7719-e580-bd0c-8987-dfba587b88db} {Suffrage:Voter ID:server-1e186be0-5a3e-bd2c-4588-099976a04aab Address:1e186be0-5a3e-bd2c-4588-099976a04aab} {Suffrage:Voter ID:server-9d317646-b62d-50df-3bb7-01b27b07e7ee Address:9d317646-b62d-50df-3bb7-01b27b07e7ee}]" 2025-09-14T21:48:38.525Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:38.525Z [INFO] server-9d317646-b62d-50df-3bb7-01b27b07e7ee: entering follower state: follower="Node at 9d317646-b62d-50df-3bb7-01b27b07e7ee [Follower]" leader-address= leader-id= 2025-09-14T21:48:38.531Z [ERROR] server-eddd436f-b70e-2d91-413d-42509e6fafe2: peer has newer term, stopping replication: peer="{Voter server-d66acbe9-5343-4b20-8ad5-0c2578394362 d66acbe9-5343-4b20-8ad5-0c2578394362}" 2025-09-14T21:48:38.531Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: entering follower state: follower="Node at eddd436f-b70e-2d91-413d-42509e6fafe2 [Follower]" leader-address= leader-id= 2025-09-14T21:48:38.531Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: aborting pipeline replication: peer="{Voter server-4a406d95-6e56-e981-2e80-6c0b7ec318b5 4a406d95-6e56-e981-2e80-6c0b7ec318b5}" 2025-09-14T21:48:39.083Z [WARN] server-9d317646-b62d-50df-3bb7-01b27b07e7ee: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:39.083Z [INFO] server-9d317646-b62d-50df-3bb7-01b27b07e7ee: entering candidate state: node="Node at 9d317646-b62d-50df-3bb7-01b27b07e7ee [Candidate]" term=2 2025-09-14T21:48:39.083Z [INFO] server-9d317646-b62d-50df-3bb7-01b27b07e7ee: election won: term=2 tally=2 2025-09-14T21:48:39.083Z [INFO] server-9d317646-b62d-50df-3bb7-01b27b07e7ee: entering leader state: leader="Node at 9d317646-b62d-50df-3bb7-01b27b07e7ee [Leader]" 2025-09-14T21:48:39.083Z [INFO] server-9d317646-b62d-50df-3bb7-01b27b07e7ee: added peer, starting replication: peer=server-2fcf7719-e580-bd0c-8987-dfba587b88db 2025-09-14T21:48:39.083Z [INFO] server-9d317646-b62d-50df-3bb7-01b27b07e7ee: added peer, starting replication: peer=server-1e186be0-5a3e-bd2c-4588-099976a04aab 2025-09-14T21:48:39.083Z [INFO] server-9d317646-b62d-50df-3bb7-01b27b07e7ee: pipelining replication: peer="{Voter server-2fcf7719-e580-bd0c-8987-dfba587b88db 2fcf7719-e580-bd0c-8987-dfba587b88db}" 2025-09-14T21:48:39.083Z [INFO] server-9d317646-b62d-50df-3bb7-01b27b07e7ee: pipelining replication: peer="{Voter server-1e186be0-5a3e-bd2c-4588-099976a04aab 1e186be0-5a3e-bd2c-4588-099976a04aab}" 2025-09-14T21:48:39.114Z [WARN] server-eddd436f-b70e-2d91-413d-42509e6fafe2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:39.114Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: entering candidate state: node="Node at eddd436f-b70e-2d91-413d-42509e6fafe2 [Candidate]" term=3 2025-09-14T21:48:39.114Z [INFO] server-d66acbe9-5343-4b20-8ad5-0c2578394362: duplicate requestVote for same term: term=3 2025-09-14T21:48:39.114Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: election won: term=3 tally=2 2025-09-14T21:48:39.114Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: entering leader state: leader="Node at eddd436f-b70e-2d91-413d-42509e6fafe2 [Leader]" 2025-09-14T21:48:39.114Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: added peer, starting replication: peer=server-4a406d95-6e56-e981-2e80-6c0b7ec318b5 2025-09-14T21:48:39.114Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: added peer, starting replication: peer=server-d66acbe9-5343-4b20-8ad5-0c2578394362 2025-09-14T21:48:39.114Z [INFO] server-d66acbe9-5343-4b20-8ad5-0c2578394362: entering follower state: follower="Node at d66acbe9-5343-4b20-8ad5-0c2578394362 [Follower]" leader-address=eddd436f-b70e-2d91-413d-42509e6fafe2 leader-id=server-eddd436f-b70e-2d91-413d-42509e6fafe2 2025-09-14T21:48:39.114Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: pipelining replication: peer="{Voter server-4a406d95-6e56-e981-2e80-6c0b7ec318b5 4a406d95-6e56-e981-2e80-6c0b7ec318b5}" 2025-09-14T21:48:39.114Z [INFO] server-eddd436f-b70e-2d91-413d-42509e6fafe2: pipelining replication: peer="{Voter server-d66acbe9-5343-4b20-8ad5-0c2578394362 d66acbe9-5343-4b20-8ad5-0c2578394362}" 2025-09-14T21:48:39.240Z [WARN] server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c: Election timeout reached, restarting election 2025-09-14T21:48:39.240Z [INFO] server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c: entering candidate state: node="Node at 915e7e29-8f69-5b4f-b2a7-860c5804ae3c [Candidate]" term=4 2025-09-14T21:48:39.240Z [WARN] server-d3a15db6-442f-1c38-0601-eab09266c438: rejecting vote request since we have a leader: from=915e7e29-8f69-5b4f-b2a7-860c5804ae3c leader=38971478-809c-95e2-3b17-c84e9880db93 leader-id=server-38971478-809c-95e2-3b17-c84e9880db93 2025-09-14T21:48:39.240Z [INFO] server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c: election won: term=4 tally=2 2025-09-14T21:48:39.240Z [INFO] server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c: entering leader state: leader="Node at 915e7e29-8f69-5b4f-b2a7-860c5804ae3c [Leader]" 2025-09-14T21:48:39.240Z [INFO] server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c: added peer, starting replication: peer=server-38971478-809c-95e2-3b17-c84e9880db93 2025-09-14T21:48:39.240Z [INFO] server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c: added peer, starting replication: peer=server-d3a15db6-442f-1c38-0601-eab09266c438 2025-09-14T21:48:39.240Z [INFO] server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c: pipelining replication: peer="{Voter server-d3a15db6-442f-1c38-0601-eab09266c438 d3a15db6-442f-1c38-0601-eab09266c438}" 2025-09-14T21:48:39.240Z [INFO] server-915e7e29-8f69-5b4f-b2a7-860c5804ae3c: pipelining replication: peer="{Voter server-38971478-809c-95e2-3b17-c84e9880db93 38971478-809c-95e2-3b17-c84e9880db93}" 2025-09-14T21:48:40.089Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:39.083716073 +0000 UTC m=+89.651594336 (1 nodes), 558.224326ms from start of poll, 558.583171ms from cluster start. Timeout at 2025-09-14 21:48:40.088719819 +0000 UTC m=+90.656598095, 1.005003759s after stability 2025-09-14T21:48:40.090Z [INFO] server-9d317646-b62d-50df-3bb7-01b27b07e7ee: starting snapshot up to: index=12 2025-09-14T21:48:40.090Z [INFO] creating new snapshot: path=/tmp/raft451339725/snapshots/2-12-1757886520090.tmp 2025-09-14T21:48:40.094Z [INFO] server-9d317646-b62d-50df-3bb7-01b27b07e7ee: snapshot complete up to: index=12 2025-09-14T21:48:40.095Z [INFO] server-0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f Address:0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f} {Suffrage:Voter ID:server-333bd669-34f6-aabb-e00f-1f26aa872b20 Address:333bd669-34f6-aabb-e00f-1f26aa872b20} {Suffrage:Voter ID:server-c51de05c-77ed-b800-4a06-d1f23986e2c4 Address:c51de05c-77ed-b800-4a06-d1f23986e2c4}]" 2025-09-14T21:48:40.095Z [INFO] server-0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f: entering follower state: follower="Node at 0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f [Follower]" leader-address= leader-id= 2025-09-14T21:48:40.095Z [INFO] server-333bd669-34f6-aabb-e00f-1f26aa872b20: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f Address:0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f} {Suffrage:Voter ID:server-333bd669-34f6-aabb-e00f-1f26aa872b20 Address:333bd669-34f6-aabb-e00f-1f26aa872b20} {Suffrage:Voter ID:server-c51de05c-77ed-b800-4a06-d1f23986e2c4 Address:c51de05c-77ed-b800-4a06-d1f23986e2c4}]" 2025-09-14T21:48:40.095Z [INFO] server-333bd669-34f6-aabb-e00f-1f26aa872b20: entering follower state: follower="Node at 333bd669-34f6-aabb-e00f-1f26aa872b20 [Follower]" leader-address= leader-id= 2025-09-14T21:48:40.095Z [INFO] server-c51de05c-77ed-b800-4a06-d1f23986e2c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f Address:0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f} {Suffrage:Voter ID:server-333bd669-34f6-aabb-e00f-1f26aa872b20 Address:333bd669-34f6-aabb-e00f-1f26aa872b20} {Suffrage:Voter ID:server-c51de05c-77ed-b800-4a06-d1f23986e2c4 Address:c51de05c-77ed-b800-4a06-d1f23986e2c4}]" 2025-09-14T21:48:40.095Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:40.095Z [INFO] server-c51de05c-77ed-b800-4a06-d1f23986e2c4: entering follower state: follower="Node at c51de05c-77ed-b800-4a06-d1f23986e2c4 [Follower]" leader-address= leader-id= 2025-09-14T21:48:40.641Z [WARN] server-333bd669-34f6-aabb-e00f-1f26aa872b20: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:40.641Z [INFO] server-333bd669-34f6-aabb-e00f-1f26aa872b20: entering candidate state: node="Node at 333bd669-34f6-aabb-e00f-1f26aa872b20 [Candidate]" term=2 2025-09-14T21:48:40.641Z [INFO] server-333bd669-34f6-aabb-e00f-1f26aa872b20: election won: term=2 tally=2 2025-09-14T21:48:40.641Z [INFO] server-333bd669-34f6-aabb-e00f-1f26aa872b20: entering leader state: leader="Node at 333bd669-34f6-aabb-e00f-1f26aa872b20 [Leader]" 2025-09-14T21:48:40.641Z [INFO] server-333bd669-34f6-aabb-e00f-1f26aa872b20: added peer, starting replication: peer=server-0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f 2025-09-14T21:48:40.641Z [INFO] server-333bd669-34f6-aabb-e00f-1f26aa872b20: added peer, starting replication: peer=server-c51de05c-77ed-b800-4a06-d1f23986e2c4 2025-09-14T21:48:40.641Z [INFO] server-333bd669-34f6-aabb-e00f-1f26aa872b20: pipelining replication: peer="{Voter server-c51de05c-77ed-b800-4a06-d1f23986e2c4 c51de05c-77ed-b800-4a06-d1f23986e2c4}" 2025-09-14T21:48:40.641Z [INFO] server-333bd669-34f6-aabb-e00f-1f26aa872b20: pipelining replication: peer="{Voter server-0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f 0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f}" 2025-09-14T21:48:41.647Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:40.641474239 +0000 UTC m=+91.209352498 (1 nodes), 546.133083ms from start of poll, 546.465482ms from cluster start. Timeout at 2025-09-14 21:48:41.646478271 +0000 UTC m=+92.214356550, 1.005004052s after stability 2025-09-14T21:48:41.647Z [INFO] creating new snapshot: path=/tmp/raft2789132692/snapshots/2-10013-1757886521647.tmp 2025-09-14T21:48:41.648Z [INFO] server-333bd669-34f6-aabb-e00f-1f26aa872b20: copied to local snapshot: bytes=71 2025-09-14T21:48:41.648Z [INFO] server-333bd669-34f6-aabb-e00f-1f26aa872b20: snapshot restore progress: id=2-10013-1757886521647 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:41.648Z [INFO] server-333bd669-34f6-aabb-e00f-1f26aa872b20: restored user snapshot: index=10013 2025-09-14T21:48:41.648Z [ERROR] server-333bd669-34f6-aabb-e00f-1f26aa872b20: failed to get log: index=3 error="log not found" 2025-09-14T21:48:41.648Z [INFO] server-333bd669-34f6-aabb-e00f-1f26aa872b20: aborting pipeline replication: peer="{Voter server-c51de05c-77ed-b800-4a06-d1f23986e2c4 c51de05c-77ed-b800-4a06-d1f23986e2c4}" 2025-09-14T21:48:41.648Z [ERROR] server-333bd669-34f6-aabb-e00f-1f26aa872b20: failed to get log: index=3 error="log not found" 2025-09-14T21:48:41.648Z [INFO] server-333bd669-34f6-aabb-e00f-1f26aa872b20: aborting pipeline replication: peer="{Voter server-0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f 0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f}" 2025-09-14T21:48:41.655Z [ERROR] server-333bd669-34f6-aabb-e00f-1f26aa872b20: failed to get log: index=3 error="log not found" 2025-09-14T21:48:41.655Z [INFO] creating new snapshot: path=/tmp/raft2346646079/snapshots/2-10013-1757886521655.tmp 2025-09-14T21:48:41.655Z [INFO] server-c51de05c-77ed-b800-4a06-d1f23986e2c4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:41.656Z [INFO] server-c51de05c-77ed-b800-4a06-d1f23986e2c4: copied to local snapshot: bytes=71 2025-09-14T21:48:41.656Z [INFO] server-c51de05c-77ed-b800-4a06-d1f23986e2c4: snapshot restore progress: id=2-10013-1757886521655 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:41.657Z [INFO] server-c51de05c-77ed-b800-4a06-d1f23986e2c4: Installed remote snapshot 2025-09-14T21:48:41.657Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:41.657Z [INFO] server-333bd669-34f6-aabb-e00f-1f26aa872b20: pipelining replication: peer="{Voter server-c51de05c-77ed-b800-4a06-d1f23986e2c4 c51de05c-77ed-b800-4a06-d1f23986e2c4}" 2025-09-14T21:48:41.659Z [ERROR] server-333bd669-34f6-aabb-e00f-1f26aa872b20: failed to get log: index=3 error="log not found" 2025-09-14T21:48:41.659Z [INFO] creating new snapshot: path=/tmp/raft1604152318/snapshots/2-10013-1757886521659.tmp 2025-09-14T21:48:41.659Z [INFO] server-0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:41.660Z [INFO] server-0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f: copied to local snapshot: bytes=71 2025-09-14T21:48:41.660Z [INFO] server-0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f: snapshot restore progress: id=2-10013-1757886521659 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:41.660Z [INFO] server-0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f: Installed remote snapshot 2025-09-14T21:48:41.660Z [INFO] server-333bd669-34f6-aabb-e00f-1f26aa872b20: pipelining replication: peer="{Voter server-0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f 0a3d187f-d0c6-f95a-aa62-3ff1ecc00e8f}" 2025-09-14T21:48:42.663Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:41.657344753 +0000 UTC m=+92.225223028 (2 nodes), 739ns from start of poll, 1.562336012s from cluster start. Timeout at 2025-09-14 21:48:42.662365734 +0000 UTC m=+93.230244028, 1.005021s after stability 2025-09-14T21:48:42.674Z [INFO] server-9d317646-b62d-50df-3bb7-01b27b07e7ee: aborting pipeline replication: peer="{Voter server-1e186be0-5a3e-bd2c-4588-099976a04aab 1e186be0-5a3e-bd2c-4588-099976a04aab}" 2025-09-14T21:48:42.674Z [INFO] server-9d317646-b62d-50df-3bb7-01b27b07e7ee: aborting pipeline replication: peer="{Voter server-2fcf7719-e580-bd0c-8987-dfba587b88db 2fcf7719-e580-bd0c-8987-dfba587b88db}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true 2025-09-14T21:48:42.676Z [INFO] server-8bff343c-4bc1-ebf9-32ce-b92b30946924: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8bff343c-4bc1-ebf9-32ce-b92b30946924 Address:8bff343c-4bc1-ebf9-32ce-b92b30946924} {Suffrage:Voter ID:server-90a53107-df17-22f9-c3f4-b638ab8aaf19 Address:90a53107-df17-22f9-c3f4-b638ab8aaf19} {Suffrage:Voter ID:server-ce9a0073-5ff3-9d01-5354-a519c6ee1273 Address:ce9a0073-5ff3-9d01-5354-a519c6ee1273}]" 2025-09-14T21:48:42.676Z [INFO] server-8bff343c-4bc1-ebf9-32ce-b92b30946924: entering follower state: follower="Node at 8bff343c-4bc1-ebf9-32ce-b92b30946924 [Follower]" leader-address= leader-id= 2025-09-14T21:48:42.676Z [INFO] server-90a53107-df17-22f9-c3f4-b638ab8aaf19: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8bff343c-4bc1-ebf9-32ce-b92b30946924 Address:8bff343c-4bc1-ebf9-32ce-b92b30946924} {Suffrage:Voter ID:server-90a53107-df17-22f9-c3f4-b638ab8aaf19 Address:90a53107-df17-22f9-c3f4-b638ab8aaf19} {Suffrage:Voter ID:server-ce9a0073-5ff3-9d01-5354-a519c6ee1273 Address:ce9a0073-5ff3-9d01-5354-a519c6ee1273}]" 2025-09-14T21:48:42.676Z [INFO] server-90a53107-df17-22f9-c3f4-b638ab8aaf19: entering follower state: follower="Node at 90a53107-df17-22f9-c3f4-b638ab8aaf19 [Follower]" leader-address= leader-id= 2025-09-14T21:48:42.676Z [INFO] server-ce9a0073-5ff3-9d01-5354-a519c6ee1273: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8bff343c-4bc1-ebf9-32ce-b92b30946924 Address:8bff343c-4bc1-ebf9-32ce-b92b30946924} {Suffrage:Voter ID:server-90a53107-df17-22f9-c3f4-b638ab8aaf19 Address:90a53107-df17-22f9-c3f4-b638ab8aaf19} {Suffrage:Voter ID:server-ce9a0073-5ff3-9d01-5354-a519c6ee1273 Address:ce9a0073-5ff3-9d01-5354-a519c6ee1273}]" 2025-09-14T21:48:42.676Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:42.676Z [INFO] server-ce9a0073-5ff3-9d01-5354-a519c6ee1273: entering follower state: follower="Node at ce9a0073-5ff3-9d01-5354-a519c6ee1273 [Follower]" leader-address= leader-id= 2025-09-14T21:48:43.577Z [WARN] server-8bff343c-4bc1-ebf9-32ce-b92b30946924: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:43.577Z [INFO] server-8bff343c-4bc1-ebf9-32ce-b92b30946924: entering candidate state: node="Node at 8bff343c-4bc1-ebf9-32ce-b92b30946924 [Candidate]" term=2 2025-09-14T21:48:43.577Z [INFO] server-8bff343c-4bc1-ebf9-32ce-b92b30946924: election won: term=2 tally=2 2025-09-14T21:48:43.577Z [INFO] server-8bff343c-4bc1-ebf9-32ce-b92b30946924: entering leader state: leader="Node at 8bff343c-4bc1-ebf9-32ce-b92b30946924 [Leader]" 2025-09-14T21:48:43.577Z [INFO] server-8bff343c-4bc1-ebf9-32ce-b92b30946924: added peer, starting replication: peer=server-90a53107-df17-22f9-c3f4-b638ab8aaf19 2025-09-14T21:48:43.577Z [INFO] server-8bff343c-4bc1-ebf9-32ce-b92b30946924: added peer, starting replication: peer=server-ce9a0073-5ff3-9d01-5354-a519c6ee1273 2025-09-14T21:48:43.577Z [INFO] server-8bff343c-4bc1-ebf9-32ce-b92b30946924: pipelining replication: peer="{Voter server-90a53107-df17-22f9-c3f4-b638ab8aaf19 90a53107-df17-22f9-c3f4-b638ab8aaf19}" 2025-09-14T21:48:43.577Z [INFO] server-8bff343c-4bc1-ebf9-32ce-b92b30946924: pipelining replication: peer="{Voter server-ce9a0073-5ff3-9d01-5354-a519c6ee1273 ce9a0073-5ff3-9d01-5354-a519c6ee1273}" 2025-09-14T21:48:44.583Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:43.577758888 +0000 UTC m=+94.145637146 (1 nodes), 901.492257ms from start of poll, 901.805208ms from cluster start. Timeout at 2025-09-14 21:48:44.582831745 +0000 UTC m=+95.150710004, 1.005072858s after stability 2025-09-14T21:48:44.583Z [INFO] server-8bff343c-4bc1-ebf9-32ce-b92b30946924: starting snapshot up to: index=12 2025-09-14T21:48:44.583Z [INFO] creating new snapshot: path=/tmp/raft4277967636/snapshots/2-12-1757886524583.tmp 2025-09-14T21:48:44.583Z [INFO] server-8bff343c-4bc1-ebf9-32ce-b92b30946924: snapshot complete up to: index=12 2025-09-14T21:48:44.584Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60548a44-048f-da2a-3ce1-06a5cd2727e3 Address:60548a44-048f-da2a-3ce1-06a5cd2727e3} {Suffrage:Voter ID:server-0788889b-2a44-053a-c3f2-5192c71d1a2d Address:0788889b-2a44-053a-c3f2-5192c71d1a2d} {Suffrage:Voter ID:server-7ec5727c-f684-c782-0a92-2558db2b2dc6 Address:7ec5727c-f684-c782-0a92-2558db2b2dc6}]" 2025-09-14T21:48:44.585Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: entering follower state: follower="Node at 60548a44-048f-da2a-3ce1-06a5cd2727e3 [Follower]" leader-address= leader-id= 2025-09-14T21:48:44.585Z [INFO] server-0788889b-2a44-053a-c3f2-5192c71d1a2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60548a44-048f-da2a-3ce1-06a5cd2727e3 Address:60548a44-048f-da2a-3ce1-06a5cd2727e3} {Suffrage:Voter ID:server-0788889b-2a44-053a-c3f2-5192c71d1a2d Address:0788889b-2a44-053a-c3f2-5192c71d1a2d} {Suffrage:Voter ID:server-7ec5727c-f684-c782-0a92-2558db2b2dc6 Address:7ec5727c-f684-c782-0a92-2558db2b2dc6}]" 2025-09-14T21:48:44.585Z [INFO] server-0788889b-2a44-053a-c3f2-5192c71d1a2d: entering follower state: follower="Node at 0788889b-2a44-053a-c3f2-5192c71d1a2d [Follower]" leader-address= leader-id= 2025-09-14T21:48:44.585Z [INFO] server-7ec5727c-f684-c782-0a92-2558db2b2dc6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60548a44-048f-da2a-3ce1-06a5cd2727e3 Address:60548a44-048f-da2a-3ce1-06a5cd2727e3} {Suffrage:Voter ID:server-0788889b-2a44-053a-c3f2-5192c71d1a2d Address:0788889b-2a44-053a-c3f2-5192c71d1a2d} {Suffrage:Voter ID:server-7ec5727c-f684-c782-0a92-2558db2b2dc6 Address:7ec5727c-f684-c782-0a92-2558db2b2dc6}]" 2025-09-14T21:48:44.585Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:44.585Z [INFO] server-7ec5727c-f684-c782-0a92-2558db2b2dc6: entering follower state: follower="Node at 7ec5727c-f684-c782-0a92-2558db2b2dc6 [Follower]" leader-address= leader-id= 2025-09-14T21:48:45.147Z [WARN] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:45.147Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: entering candidate state: node="Node at 60548a44-048f-da2a-3ce1-06a5cd2727e3 [Candidate]" term=2 2025-09-14T21:48:45.147Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: election won: term=2 tally=2 2025-09-14T21:48:45.147Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: entering leader state: leader="Node at 60548a44-048f-da2a-3ce1-06a5cd2727e3 [Leader]" 2025-09-14T21:48:45.147Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: added peer, starting replication: peer=server-0788889b-2a44-053a-c3f2-5192c71d1a2d 2025-09-14T21:48:45.147Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: added peer, starting replication: peer=server-7ec5727c-f684-c782-0a92-2558db2b2dc6 2025-09-14T21:48:45.147Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: pipelining replication: peer="{Voter server-0788889b-2a44-053a-c3f2-5192c71d1a2d 0788889b-2a44-053a-c3f2-5192c71d1a2d}" 2025-09-14T21:48:45.147Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: pipelining replication: peer="{Voter server-7ec5727c-f684-c782-0a92-2558db2b2dc6 7ec5727c-f684-c782-0a92-2558db2b2dc6}" 2025-09-14T21:48:46.152Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:45.147220906 +0000 UTC m=+95.715099186 (1 nodes), 561.852191ms from start of poll, 562.365964ms from cluster start. Timeout at 2025-09-14 21:48:46.152224379 +0000 UTC m=+96.720102655, 1.005003469s after stability 2025-09-14T21:48:46.153Z [INFO] creating new snapshot: path=/tmp/raft470665617/snapshots/2-10013-1757886526153.tmp 2025-09-14T21:48:46.153Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: copied to local snapshot: bytes=71 2025-09-14T21:48:46.153Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: snapshot restore progress: id=2-10013-1757886526153 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:46.153Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: removing all old logs from log store 2025-09-14T21:48:46.153Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: compacting logs: from=1 to=2 2025-09-14T21:48:46.153Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: restored user snapshot: index=10013 2025-09-14T21:48:46.153Z [ERROR] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: failed to get log: index=2 error="log not found" 2025-09-14T21:48:46.153Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: aborting pipeline replication: peer="{Voter server-7ec5727c-f684-c782-0a92-2558db2b2dc6 7ec5727c-f684-c782-0a92-2558db2b2dc6}" 2025-09-14T21:48:46.153Z [ERROR] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: failed to get log: index=2 error="log not found" 2025-09-14T21:48:46.154Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: aborting pipeline replication: peer="{Voter server-0788889b-2a44-053a-c3f2-5192c71d1a2d 0788889b-2a44-053a-c3f2-5192c71d1a2d}" 2025-09-14T21:48:46.160Z [ERROR] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: failed to get log: index=2 error="log not found" 2025-09-14T21:48:46.161Z [INFO] creating new snapshot: path=/tmp/raft2337740716/snapshots/2-10013-1757886526161.tmp 2025-09-14T21:48:46.161Z [INFO] server-7ec5727c-f684-c782-0a92-2558db2b2dc6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:46.161Z [INFO] server-7ec5727c-f684-c782-0a92-2558db2b2dc6: copied to local snapshot: bytes=71 2025-09-14T21:48:46.161Z [INFO] server-7ec5727c-f684-c782-0a92-2558db2b2dc6: snapshot restore progress: id=2-10013-1757886526161 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:46.161Z [INFO] server-7ec5727c-f684-c782-0a92-2558db2b2dc6: removing all old logs from log store 2025-09-14T21:48:46.161Z [INFO] server-7ec5727c-f684-c782-0a92-2558db2b2dc6: compacting logs: from=1 to=2 2025-09-14T21:48:46.161Z [INFO] server-7ec5727c-f684-c782-0a92-2558db2b2dc6: Installed remote snapshot 2025-09-14T21:48:46.161Z [ERROR] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: failed to get log: index=2 error="log not found" 2025-09-14T21:48:46.161Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: pipelining replication: peer="{Voter server-7ec5727c-f684-c782-0a92-2558db2b2dc6 7ec5727c-f684-c782-0a92-2558db2b2dc6}" 2025-09-14T21:48:46.162Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:46.162Z [INFO] creating new snapshot: path=/tmp/raft3929264826/snapshots/2-10013-1757886526162.tmp 2025-09-14T21:48:46.162Z [INFO] server-0788889b-2a44-053a-c3f2-5192c71d1a2d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:46.162Z [INFO] server-0788889b-2a44-053a-c3f2-5192c71d1a2d: copied to local snapshot: bytes=71 2025-09-14T21:48:46.163Z [INFO] server-0788889b-2a44-053a-c3f2-5192c71d1a2d: snapshot restore progress: id=2-10013-1757886526162 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-09-14T21:48:46.163Z [INFO] server-0788889b-2a44-053a-c3f2-5192c71d1a2d: removing all old logs from log store 2025-09-14T21:48:46.163Z [INFO] server-0788889b-2a44-053a-c3f2-5192c71d1a2d: compacting logs: from=1 to=2 2025-09-14T21:48:46.163Z [INFO] server-0788889b-2a44-053a-c3f2-5192c71d1a2d: Installed remote snapshot 2025-09-14T21:48:46.163Z [INFO] server-60548a44-048f-da2a-3ce1-06a5cd2727e3: pipelining replication: peer="{Voter server-0788889b-2a44-053a-c3f2-5192c71d1a2d 0788889b-2a44-053a-c3f2-5192c71d1a2d}" 2025-09-14T21:48:47.167Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:46.162018345 +0000 UTC m=+96.729896604 (2 nodes), 636ns from start of poll, 1.577163382s from cluster start. Timeout at 2025-09-14 21:48:47.167050236 +0000 UTC m=+97.734928532, 1.005031928s after stability 2025-09-14T21:48:47.179Z [INFO] server-8bff343c-4bc1-ebf9-32ce-b92b30946924: aborting pipeline replication: peer="{Voter server-ce9a0073-5ff3-9d01-5354-a519c6ee1273 ce9a0073-5ff3-9d01-5354-a519c6ee1273}" 2025-09-14T21:48:47.179Z [INFO] server-8bff343c-4bc1-ebf9-32ce-b92b30946924: aborting pipeline replication: peer="{Voter server-90a53107-df17-22f9-c3f4-b638ab8aaf19 90a53107-df17-22f9-c3f4-b638ab8aaf19}" --- FAIL: TestRaft_UserRestore (84.48s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.70s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.72s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.42s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.34s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.65s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.67s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.25s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.21s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.64s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.82s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.63s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.29s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.70s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.66s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.71s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.56s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.80s) --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.54s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.19s) --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.14s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.60s) --- FAIL: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.58s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.15s) --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.50s) === RUN TestRaft_SendSnapshotFollower 2025-09-14T21:48:47.180Z [INFO] server-72a068ba-8368-bab6-a1b1-96c52c340da0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72a068ba-8368-bab6-a1b1-96c52c340da0 Address:72a068ba-8368-bab6-a1b1-96c52c340da0} {Suffrage:Voter ID:server-dbe852d2-7546-d6b5-0562-f40945198f45 Address:dbe852d2-7546-d6b5-0562-f40945198f45} {Suffrage:Voter ID:server-b01ef8b7-d8a4-871a-4bf3-d39e1e67dddb Address:b01ef8b7-d8a4-871a-4bf3-d39e1e67dddb}]" 2025-09-14T21:48:47.180Z [INFO] server-72a068ba-8368-bab6-a1b1-96c52c340da0: entering follower state: follower="Node at 72a068ba-8368-bab6-a1b1-96c52c340da0 [Follower]" leader-address= leader-id= 2025-09-14T21:48:47.180Z [INFO] server-dbe852d2-7546-d6b5-0562-f40945198f45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72a068ba-8368-bab6-a1b1-96c52c340da0 Address:72a068ba-8368-bab6-a1b1-96c52c340da0} {Suffrage:Voter ID:server-dbe852d2-7546-d6b5-0562-f40945198f45 Address:dbe852d2-7546-d6b5-0562-f40945198f45} {Suffrage:Voter ID:server-b01ef8b7-d8a4-871a-4bf3-d39e1e67dddb Address:b01ef8b7-d8a4-871a-4bf3-d39e1e67dddb}]" 2025-09-14T21:48:47.181Z [INFO] server-b01ef8b7-d8a4-871a-4bf3-d39e1e67dddb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72a068ba-8368-bab6-a1b1-96c52c340da0 Address:72a068ba-8368-bab6-a1b1-96c52c340da0} {Suffrage:Voter ID:server-dbe852d2-7546-d6b5-0562-f40945198f45 Address:dbe852d2-7546-d6b5-0562-f40945198f45} {Suffrage:Voter ID:server-b01ef8b7-d8a4-871a-4bf3-d39e1e67dddb Address:b01ef8b7-d8a4-871a-4bf3-d39e1e67dddb}]" 2025-09-14T21:48:47.181Z [INFO] server-dbe852d2-7546-d6b5-0562-f40945198f45: entering follower state: follower="Node at dbe852d2-7546-d6b5-0562-f40945198f45 [Follower]" leader-address= leader-id= 2025-09-14T21:48:47.181Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:47.181Z [INFO] server-b01ef8b7-d8a4-871a-4bf3-d39e1e67dddb: entering follower state: follower="Node at b01ef8b7-d8a4-871a-4bf3-d39e1e67dddb [Follower]" leader-address= leader-id= 2025-09-14T21:48:47.244Z [WARN] server-72a068ba-8368-bab6-a1b1-96c52c340da0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:47.244Z [INFO] server-72a068ba-8368-bab6-a1b1-96c52c340da0: entering candidate state: node="Node at 72a068ba-8368-bab6-a1b1-96c52c340da0 [Candidate]" term=2 2025-09-14T21:48:47.244Z [INFO] server-72a068ba-8368-bab6-a1b1-96c52c340da0: election won: term=2 tally=2 2025-09-14T21:48:47.244Z [INFO] server-72a068ba-8368-bab6-a1b1-96c52c340da0: entering leader state: leader="Node at 72a068ba-8368-bab6-a1b1-96c52c340da0 [Leader]" 2025-09-14T21:48:47.244Z [INFO] server-72a068ba-8368-bab6-a1b1-96c52c340da0: added peer, starting replication: peer=server-dbe852d2-7546-d6b5-0562-f40945198f45 2025-09-14T21:48:47.244Z [INFO] server-72a068ba-8368-bab6-a1b1-96c52c340da0: added peer, starting replication: peer=server-b01ef8b7-d8a4-871a-4bf3-d39e1e67dddb 2025-09-14T21:48:47.245Z [INFO] server-72a068ba-8368-bab6-a1b1-96c52c340da0: pipelining replication: peer="{Voter server-b01ef8b7-d8a4-871a-4bf3-d39e1e67dddb b01ef8b7-d8a4-871a-4bf3-d39e1e67dddb}" 2025-09-14T21:48:47.245Z [INFO] server-72a068ba-8368-bab6-a1b1-96c52c340da0: pipelining replication: peer="{Voter server-dbe852d2-7546-d6b5-0562-f40945198f45 dbe852d2-7546-d6b5-0562-f40945198f45}" 2025-09-14T21:48:47.350Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:47.244947043 +0000 UTC m=+97.812825307 (2 nodes), 63.897277ms from start of poll, 64.23971ms from cluster start. Timeout at 2025-09-14 21:48:47.349950909 +0000 UTC m=+97.917829168, 105.003861ms after stability 2025-09-14T21:48:47.350Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:47.455Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:47.350203738 +0000 UTC m=+97.918082003 (1 nodes), 1.924µs from start of poll, 169.496406ms from cluster start. Timeout at 2025-09-14 21:48:47.455224049 +0000 UTC m=+98.023102349, 105.020346ms after stability 2025-09-14T21:48:47.455Z [ERROR] server-72a068ba-8368-bab6-a1b1-96c52c340da0: failed to pipeline appendEntries: peer="{Voter server-dbe852d2-7546-d6b5-0562-f40945198f45 dbe852d2-7546-d6b5-0562-f40945198f45}" error="append pipeline closed" 2025-09-14T21:48:47.455Z [INFO] server-72a068ba-8368-bab6-a1b1-96c52c340da0: aborting pipeline replication: peer="{Voter server-dbe852d2-7546-d6b5-0562-f40945198f45 dbe852d2-7546-d6b5-0562-f40945198f45}" 2025-09-14T21:48:47.455Z [ERROR] server-72a068ba-8368-bab6-a1b1-96c52c340da0: failed to appendEntries to: peer="{Voter server-dbe852d2-7546-d6b5-0562-f40945198f45 dbe852d2-7546-d6b5-0562-f40945198f45}" error="failed to connect to peer: dbe852d2-7546-d6b5-0562-f40945198f45" raft_test.go:1627: [INFO] Finished apply without behind follower 2025-09-14T21:48:47.457Z [INFO] server-72a068ba-8368-bab6-a1b1-96c52c340da0: starting snapshot up to: index=102 2025-09-14T21:48:47.457Z [INFO] creating new snapshot: path=/tmp/raft438509124/snapshots/2-102-1757886527457.tmp 2025-09-14T21:48:47.458Z [ERROR] server-72a068ba-8368-bab6-a1b1-96c52c340da0: failed to heartbeat to: peer=dbe852d2-7546-d6b5-0562-f40945198f45 backoff time=10ms error="failed to connect to peer: dbe852d2-7546-d6b5-0562-f40945198f45" 2025-09-14T21:48:47.466Z [ERROR] server-72a068ba-8368-bab6-a1b1-96c52c340da0: failed to appendEntries to: peer="{Voter server-dbe852d2-7546-d6b5-0562-f40945198f45 dbe852d2-7546-d6b5-0562-f40945198f45}" error="failed to connect to peer: dbe852d2-7546-d6b5-0562-f40945198f45" 2025-09-14T21:48:47.468Z [INFO] server-72a068ba-8368-bab6-a1b1-96c52c340da0: compacting logs: from=1 to=92 2025-09-14T21:48:47.468Z [INFO] server-72a068ba-8368-bab6-a1b1-96c52c340da0: snapshot complete up to: index=102 2025-09-14T21:48:47.468Z [ERROR] server-dbe852d2-7546-d6b5-0562-f40945198f45: failed to take snapshot: error="nothing new to snapshot" 2025-09-14T21:48:47.468Z [INFO] server-b01ef8b7-d8a4-871a-4bf3-d39e1e67dddb: starting snapshot up to: index=102 2025-09-14T21:48:47.468Z [INFO] creating new snapshot: path=/tmp/raft1521326145/snapshots/2-102-1757886527468.tmp 2025-09-14T21:48:47.469Z [INFO] server-b01ef8b7-d8a4-871a-4bf3-d39e1e67dddb: compacting logs: from=1 to=92 2025-09-14T21:48:47.469Z [INFO] server-b01ef8b7-d8a4-871a-4bf3-d39e1e67dddb: snapshot complete up to: index=102 2025-09-14T21:48:47.477Z [ERROR] server-72a068ba-8368-bab6-a1b1-96c52c340da0: failed to get log: index=2 error="log not found" 2025-09-14T21:48:47.477Z [INFO] creating new snapshot: path=/tmp/raft2262435474/snapshots/2-102-1757886527477.tmp 2025-09-14T21:48:47.477Z [INFO] server-dbe852d2-7546-d6b5-0562-f40945198f45: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2025-09-14T21:48:47.477Z [INFO] server-dbe852d2-7546-d6b5-0562-f40945198f45: copied to local snapshot: bytes=693 2025-09-14T21:48:47.478Z [INFO] server-dbe852d2-7546-d6b5-0562-f40945198f45: snapshot restore progress: id=2-102-1757886527477 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2025-09-14T21:48:47.478Z [INFO] server-dbe852d2-7546-d6b5-0562-f40945198f45: Installed remote snapshot 2025-09-14T21:48:47.481Z [INFO] server-72a068ba-8368-bab6-a1b1-96c52c340da0: aborting pipeline replication: peer="{Voter server-b01ef8b7-d8a4-871a-4bf3-d39e1e67dddb b01ef8b7-d8a4-871a-4bf3-d39e1e67dddb}" --- PASS: TestRaft_SendSnapshotFollower (0.32s) === RUN TestRaft_SendSnapshotAndLogsFollower 2025-09-14T21:48:47.503Z [INFO] server-c433892e-5dda-8d4c-9ebd-ca43216af4a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c433892e-5dda-8d4c-9ebd-ca43216af4a1 Address:c433892e-5dda-8d4c-9ebd-ca43216af4a1} {Suffrage:Voter ID:server-a3820bc7-ffd7-96a2-da5b-dab01fe76918 Address:a3820bc7-ffd7-96a2-da5b-dab01fe76918} {Suffrage:Voter ID:server-833f8e84-60bd-9362-6ad9-2d84bd538e9f Address:833f8e84-60bd-9362-6ad9-2d84bd538e9f}]" 2025-09-14T21:48:47.503Z [INFO] server-c433892e-5dda-8d4c-9ebd-ca43216af4a1: entering follower state: follower="Node at c433892e-5dda-8d4c-9ebd-ca43216af4a1 [Follower]" leader-address= leader-id= 2025-09-14T21:48:47.503Z [INFO] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c433892e-5dda-8d4c-9ebd-ca43216af4a1 Address:c433892e-5dda-8d4c-9ebd-ca43216af4a1} {Suffrage:Voter ID:server-a3820bc7-ffd7-96a2-da5b-dab01fe76918 Address:a3820bc7-ffd7-96a2-da5b-dab01fe76918} {Suffrage:Voter ID:server-833f8e84-60bd-9362-6ad9-2d84bd538e9f Address:833f8e84-60bd-9362-6ad9-2d84bd538e9f}]" 2025-09-14T21:48:47.503Z [INFO] server-833f8e84-60bd-9362-6ad9-2d84bd538e9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c433892e-5dda-8d4c-9ebd-ca43216af4a1 Address:c433892e-5dda-8d4c-9ebd-ca43216af4a1} {Suffrage:Voter ID:server-a3820bc7-ffd7-96a2-da5b-dab01fe76918 Address:a3820bc7-ffd7-96a2-da5b-dab01fe76918} {Suffrage:Voter ID:server-833f8e84-60bd-9362-6ad9-2d84bd538e9f Address:833f8e84-60bd-9362-6ad9-2d84bd538e9f}]" 2025-09-14T21:48:47.503Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:47.503Z [INFO] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: entering follower state: follower="Node at a3820bc7-ffd7-96a2-da5b-dab01fe76918 [Follower]" leader-address= leader-id= 2025-09-14T21:48:47.503Z [INFO] server-833f8e84-60bd-9362-6ad9-2d84bd538e9f: entering follower state: follower="Node at 833f8e84-60bd-9362-6ad9-2d84bd538e9f [Follower]" leader-address= leader-id= 2025-09-14T21:48:47.557Z [WARN] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:47.557Z [INFO] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: entering candidate state: node="Node at a3820bc7-ffd7-96a2-da5b-dab01fe76918 [Candidate]" term=2 2025-09-14T21:48:47.557Z [INFO] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: election won: term=2 tally=2 2025-09-14T21:48:47.557Z [INFO] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: entering leader state: leader="Node at a3820bc7-ffd7-96a2-da5b-dab01fe76918 [Leader]" 2025-09-14T21:48:47.557Z [INFO] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: added peer, starting replication: peer=server-c433892e-5dda-8d4c-9ebd-ca43216af4a1 2025-09-14T21:48:47.557Z [INFO] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: added peer, starting replication: peer=server-833f8e84-60bd-9362-6ad9-2d84bd538e9f 2025-09-14T21:48:47.557Z [INFO] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: pipelining replication: peer="{Voter server-833f8e84-60bd-9362-6ad9-2d84bd538e9f 833f8e84-60bd-9362-6ad9-2d84bd538e9f}" 2025-09-14T21:48:47.557Z [INFO] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: pipelining replication: peer="{Voter server-c433892e-5dda-8d4c-9ebd-ca43216af4a1 c433892e-5dda-8d4c-9ebd-ca43216af4a1}" 2025-09-14T21:48:47.663Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:47.557684061 +0000 UTC m=+98.125562336 (2 nodes), 54.410923ms from start of poll, 54.716574ms from cluster start. Timeout at 2025-09-14 21:48:47.662701151 +0000 UTC m=+98.230579413, 105.017077ms after stability 2025-09-14T21:48:47.663Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:47.769Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:47.663340387 +0000 UTC m=+98.231218656 (1 nodes), 787ns from start of poll, 160.372894ms from cluster start. Timeout at 2025-09-14 21:48:47.7683547 +0000 UTC m=+98.336232981, 105.014325ms after stability 2025-09-14T21:48:47.769Z [ERROR] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: failed to pipeline appendEntries: peer="{Voter server-c433892e-5dda-8d4c-9ebd-ca43216af4a1 c433892e-5dda-8d4c-9ebd-ca43216af4a1}" error="append pipeline closed" 2025-09-14T21:48:47.769Z [INFO] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: aborting pipeline replication: peer="{Voter server-c433892e-5dda-8d4c-9ebd-ca43216af4a1 c433892e-5dda-8d4c-9ebd-ca43216af4a1}" 2025-09-14T21:48:47.769Z [ERROR] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: failed to appendEntries to: peer="{Voter server-c433892e-5dda-8d4c-9ebd-ca43216af4a1 c433892e-5dda-8d4c-9ebd-ca43216af4a1}" error="failed to connect to peer: c433892e-5dda-8d4c-9ebd-ca43216af4a1" raft_test.go:1669: [INFO] Finished apply without behind follower 2025-09-14T21:48:47.771Z [ERROR] server-c433892e-5dda-8d4c-9ebd-ca43216af4a1: failed to take snapshot: error="nothing new to snapshot" 2025-09-14T21:48:47.771Z [INFO] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: starting snapshot up to: index=102 2025-09-14T21:48:47.771Z [INFO] creating new snapshot: path=/tmp/raft1963662047/snapshots/2-102-1757886527771.tmp 2025-09-14T21:48:47.772Z [INFO] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: compacting logs: from=1 to=92 2025-09-14T21:48:47.772Z [INFO] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: snapshot complete up to: index=102 2025-09-14T21:48:47.772Z [INFO] server-833f8e84-60bd-9362-6ad9-2d84bd538e9f: starting snapshot up to: index=94 2025-09-14T21:48:47.772Z [INFO] creating new snapshot: path=/tmp/raft3879615171/snapshots/2-94-1757886527772.tmp 2025-09-14T21:48:47.772Z [INFO] server-833f8e84-60bd-9362-6ad9-2d84bd538e9f: compacting logs: from=1 to=92 2025-09-14T21:48:47.772Z [INFO] server-833f8e84-60bd-9362-6ad9-2d84bd538e9f: snapshot complete up to: index=94 2025-09-14T21:48:47.773Z [ERROR] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: failed to heartbeat to: peer=c433892e-5dda-8d4c-9ebd-ca43216af4a1 backoff time=10ms error="failed to connect to peer: c433892e-5dda-8d4c-9ebd-ca43216af4a1" raft_test.go:1690: [INFO] Finished apply without behind follower 2025-09-14T21:48:47.779Z [ERROR] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: failed to get log: index=2 error="log not found" 2025-09-14T21:48:47.780Z [INFO] creating new snapshot: path=/tmp/raft1044176650/snapshots/2-102-1757886527780.tmp 2025-09-14T21:48:47.780Z [INFO] server-c433892e-5dda-8d4c-9ebd-ca43216af4a1: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2025-09-14T21:48:47.780Z [INFO] server-c433892e-5dda-8d4c-9ebd-ca43216af4a1: copied to local snapshot: bytes=693 2025-09-14T21:48:47.781Z [INFO] server-c433892e-5dda-8d4c-9ebd-ca43216af4a1: snapshot restore progress: id=2-102-1757886527780 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2025-09-14T21:48:47.781Z [INFO] server-c433892e-5dda-8d4c-9ebd-ca43216af4a1: Installed remote snapshot 2025-09-14T21:48:47.782Z [INFO] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: pipelining replication: peer="{Voter server-c433892e-5dda-8d4c-9ebd-ca43216af4a1 c433892e-5dda-8d4c-9ebd-ca43216af4a1}" 2025-09-14T21:48:47.786Z [INFO] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: aborting pipeline replication: peer="{Voter server-833f8e84-60bd-9362-6ad9-2d84bd538e9f 833f8e84-60bd-9362-6ad9-2d84bd538e9f}" 2025-09-14T21:48:47.786Z [INFO] server-a3820bc7-ffd7-96a2-da5b-dab01fe76918: aborting pipeline replication: peer="{Voter server-c433892e-5dda-8d4c-9ebd-ca43216af4a1 c433892e-5dda-8d4c-9ebd-ca43216af4a1}" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.28s) === RUN TestRaft_ReJoinFollower 2025-09-14T21:48:47.787Z [INFO] server-15f5cc88-1bbe-827a-8882-dddc19ece301: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15f5cc88-1bbe-827a-8882-dddc19ece301 Address:15f5cc88-1bbe-827a-8882-dddc19ece301} {Suffrage:Voter ID:server-f642b670-0ad5-a768-1009-eccc63bf5e54 Address:f642b670-0ad5-a768-1009-eccc63bf5e54} {Suffrage:Voter ID:server-8b4558de-900a-3c2a-fd09-31954a3f0f5d Address:8b4558de-900a-3c2a-fd09-31954a3f0f5d}]" 2025-09-14T21:48:47.787Z [INFO] server-15f5cc88-1bbe-827a-8882-dddc19ece301: entering follower state: follower="Node at 15f5cc88-1bbe-827a-8882-dddc19ece301 [Follower]" leader-address= leader-id= 2025-09-14T21:48:47.787Z [INFO] server-f642b670-0ad5-a768-1009-eccc63bf5e54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15f5cc88-1bbe-827a-8882-dddc19ece301 Address:15f5cc88-1bbe-827a-8882-dddc19ece301} {Suffrage:Voter ID:server-f642b670-0ad5-a768-1009-eccc63bf5e54 Address:f642b670-0ad5-a768-1009-eccc63bf5e54} {Suffrage:Voter ID:server-8b4558de-900a-3c2a-fd09-31954a3f0f5d Address:8b4558de-900a-3c2a-fd09-31954a3f0f5d}]" 2025-09-14T21:48:47.787Z [INFO] server-f642b670-0ad5-a768-1009-eccc63bf5e54: entering follower state: follower="Node at f642b670-0ad5-a768-1009-eccc63bf5e54 [Follower]" leader-address= leader-id= 2025-09-14T21:48:47.787Z [INFO] server-8b4558de-900a-3c2a-fd09-31954a3f0f5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15f5cc88-1bbe-827a-8882-dddc19ece301 Address:15f5cc88-1bbe-827a-8882-dddc19ece301} {Suffrage:Voter ID:server-f642b670-0ad5-a768-1009-eccc63bf5e54 Address:f642b670-0ad5-a768-1009-eccc63bf5e54} {Suffrage:Voter ID:server-8b4558de-900a-3c2a-fd09-31954a3f0f5d Address:8b4558de-900a-3c2a-fd09-31954a3f0f5d}]" 2025-09-14T21:48:47.787Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:47.787Z [INFO] server-8b4558de-900a-3c2a-fd09-31954a3f0f5d: entering follower state: follower="Node at 8b4558de-900a-3c2a-fd09-31954a3f0f5d [Follower]" leader-address= leader-id= 2025-09-14T21:48:47.878Z [WARN] server-15f5cc88-1bbe-827a-8882-dddc19ece301: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:47.878Z [INFO] server-15f5cc88-1bbe-827a-8882-dddc19ece301: entering candidate state: node="Node at 15f5cc88-1bbe-827a-8882-dddc19ece301 [Candidate]" term=2 2025-09-14T21:48:47.878Z [INFO] server-15f5cc88-1bbe-827a-8882-dddc19ece301: election won: term=2 tally=2 2025-09-14T21:48:47.878Z [INFO] server-15f5cc88-1bbe-827a-8882-dddc19ece301: entering leader state: leader="Node at 15f5cc88-1bbe-827a-8882-dddc19ece301 [Leader]" 2025-09-14T21:48:47.878Z [INFO] server-15f5cc88-1bbe-827a-8882-dddc19ece301: added peer, starting replication: peer=server-f642b670-0ad5-a768-1009-eccc63bf5e54 2025-09-14T21:48:47.878Z [INFO] server-15f5cc88-1bbe-827a-8882-dddc19ece301: added peer, starting replication: peer=server-8b4558de-900a-3c2a-fd09-31954a3f0f5d 2025-09-14T21:48:47.878Z [INFO] server-15f5cc88-1bbe-827a-8882-dddc19ece301: pipelining replication: peer="{Voter server-8b4558de-900a-3c2a-fd09-31954a3f0f5d 8b4558de-900a-3c2a-fd09-31954a3f0f5d}" 2025-09-14T21:48:47.878Z [INFO] server-15f5cc88-1bbe-827a-8882-dddc19ece301: pipelining replication: peer="{Voter server-f642b670-0ad5-a768-1009-eccc63bf5e54 f642b670-0ad5-a768-1009-eccc63bf5e54}" 2025-09-14T21:48:47.984Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:47.878230818 +0000 UTC m=+98.446109092 (1 nodes), 90.7865ms from start of poll, 91.067844ms from cluster start. Timeout at 2025-09-14 21:48:47.983231731 +0000 UTC m=+98.551109997, 105.000905ms after stability 2025-09-14T21:48:47.989Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:48.095Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:47.990007998 +0000 UTC m=+98.557886257 (2 nodes), 1.047µs from start of poll, 202.845009ms from cluster start. Timeout at 2025-09-14 21:48:48.09503835 +0000 UTC m=+98.662916612, 105.030355ms after stability 2025-09-14T21:48:48.095Z [INFO] server-15f5cc88-1bbe-827a-8882-dddc19ece301: updating configuration: command=RemoveServer server-id=server-f642b670-0ad5-a768-1009-eccc63bf5e54 server-addr= servers="[{Suffrage:Voter ID:server-15f5cc88-1bbe-827a-8882-dddc19ece301 Address:15f5cc88-1bbe-827a-8882-dddc19ece301} {Suffrage:Voter ID:server-8b4558de-900a-3c2a-fd09-31954a3f0f5d Address:8b4558de-900a-3c2a-fd09-31954a3f0f5d}]" 2025-09-14T21:48:48.095Z [INFO] server-15f5cc88-1bbe-827a-8882-dddc19ece301: removed peer, stopping replication: peer=server-f642b670-0ad5-a768-1009-eccc63bf5e54 last-index=3 2025-09-14T21:48:48.095Z [INFO] server-15f5cc88-1bbe-827a-8882-dddc19ece301: aborting pipeline replication: peer="{Voter server-f642b670-0ad5-a768-1009-eccc63bf5e54 f642b670-0ad5-a768-1009-eccc63bf5e54}" 2025-09-14T21:48:48.194Z [WARN] server-f642b670-0ad5-a768-1009-eccc63bf5e54: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2025-09-14T21:48:48.201Z [INFO] server-15f5cc88-1bbe-827a-8882-dddc19ece301: updating configuration: command=AddVoter server-id=server-f642b670-0ad5-a768-1009-eccc63bf5e54 server-addr=f642b670-0ad5-a768-1009-eccc63bf5e54 servers="[{Suffrage:Voter ID:server-15f5cc88-1bbe-827a-8882-dddc19ece301 Address:15f5cc88-1bbe-827a-8882-dddc19ece301} {Suffrage:Voter ID:server-8b4558de-900a-3c2a-fd09-31954a3f0f5d Address:8b4558de-900a-3c2a-fd09-31954a3f0f5d} {Suffrage:Voter ID:server-f642b670-0ad5-a768-1009-eccc63bf5e54 Address:f642b670-0ad5-a768-1009-eccc63bf5e54}]" 2025-09-14T21:48:48.201Z [INFO] server-15f5cc88-1bbe-827a-8882-dddc19ece301: added peer, starting replication: peer=server-f642b670-0ad5-a768-1009-eccc63bf5e54 2025-09-14T21:48:48.201Z [WARN] server-f642b670-0ad5-a768-1009-eccc63bf5e54: failed to get previous log: previous-index=4 last-index=3 error="log not found" 2025-09-14T21:48:48.201Z [WARN] server-15f5cc88-1bbe-827a-8882-dddc19ece301: appendEntries rejected, sending older logs: peer="{Voter server-f642b670-0ad5-a768-1009-eccc63bf5e54 f642b670-0ad5-a768-1009-eccc63bf5e54}" next=4 2025-09-14T21:48:48.201Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:48.201Z [INFO] server-15f5cc88-1bbe-827a-8882-dddc19ece301: pipelining replication: peer="{Voter server-f642b670-0ad5-a768-1009-eccc63bf5e54 f642b670-0ad5-a768-1009-eccc63bf5e54}" 2025-09-14T21:48:48.307Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:48.201874057 +0000 UTC m=+98.769752340 (1 nodes), 946ns from start of poll, 414.711092ms from cluster start. Timeout at 2025-09-14 21:48:48.306896403 +0000 UTC m=+98.874774686, 105.022346ms after stability 2025-09-14T21:48:48.307Z [INFO] server-15f5cc88-1bbe-827a-8882-dddc19ece301: aborting pipeline replication: peer="{Voter server-8b4558de-900a-3c2a-fd09-31954a3f0f5d 8b4558de-900a-3c2a-fd09-31954a3f0f5d}" 2025-09-14T21:48:48.307Z [INFO] server-15f5cc88-1bbe-827a-8882-dddc19ece301: aborting pipeline replication: peer="{Voter server-f642b670-0ad5-a768-1009-eccc63bf5e54 f642b670-0ad5-a768-1009-eccc63bf5e54}" --- PASS: TestRaft_ReJoinFollower (0.52s) === RUN TestRaft_LeaderLeaseExpire 2025-09-14T21:48:48.308Z [INFO] server-3839dd79-d610-736e-39aa-f079d1ef431d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3839dd79-d610-736e-39aa-f079d1ef431d Address:3839dd79-d610-736e-39aa-f079d1ef431d} {Suffrage:Voter ID:server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159 Address:cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159}]" 2025-09-14T21:48:48.308Z [INFO] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3839dd79-d610-736e-39aa-f079d1ef431d Address:3839dd79-d610-736e-39aa-f079d1ef431d} {Suffrage:Voter ID:server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159 Address:cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159}]" 2025-09-14T21:48:48.308Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:48.308Z [INFO] server-3839dd79-d610-736e-39aa-f079d1ef431d: entering follower state: follower="Node at 3839dd79-d610-736e-39aa-f079d1ef431d [Follower]" leader-address= leader-id= 2025-09-14T21:48:48.309Z [INFO] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: entering follower state: follower="Node at cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159 [Follower]" leader-address= leader-id= 2025-09-14T21:48:48.396Z [WARN] server-3839dd79-d610-736e-39aa-f079d1ef431d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:48.396Z [INFO] server-3839dd79-d610-736e-39aa-f079d1ef431d: entering candidate state: node="Node at 3839dd79-d610-736e-39aa-f079d1ef431d [Candidate]" term=2 2025-09-14T21:48:48.396Z [INFO] server-3839dd79-d610-736e-39aa-f079d1ef431d: election won: term=2 tally=2 2025-09-14T21:48:48.396Z [INFO] server-3839dd79-d610-736e-39aa-f079d1ef431d: entering leader state: leader="Node at 3839dd79-d610-736e-39aa-f079d1ef431d [Leader]" 2025-09-14T21:48:48.396Z [INFO] server-3839dd79-d610-736e-39aa-f079d1ef431d: added peer, starting replication: peer=server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159 2025-09-14T21:48:48.396Z [INFO] server-3839dd79-d610-736e-39aa-f079d1ef431d: pipelining replication: peer="{Voter server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159 cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159}" 2025-09-14T21:48:48.817Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:48.39656303 +0000 UTC m=+98.964441302 (1 nodes), 87.565563ms from start of poll, 87.761459ms from cluster start. Timeout at 2025-09-14 21:48:48.501760512 +0000 UTC m=+99.069638792, 105.19749ms after stability 2025-09-14T21:48:48.818Z [WARN] server-3839dd79-d610-736e-39aa-f079d1ef431d: failed to contact quorum of nodes, stepping down 2025-09-14T21:48:48.818Z [INFO] server-3839dd79-d610-736e-39aa-f079d1ef431d: entering follower state: follower="Node at 3839dd79-d610-736e-39aa-f079d1ef431d [Follower]" leader-address= leader-id= 2025-09-14T21:48:48.818Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:48.818Z [WARN] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: heartbeat timeout reached, starting election: last-leader-addr=3839dd79-d610-736e-39aa-f079d1ef431d last-leader-id=server-3839dd79-d610-736e-39aa-f079d1ef431d 2025-09-14T21:48:48.818Z [INFO] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: entering candidate state: node="Node at cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159 [Candidate]" term=3 2025-09-14T21:48:48.818Z [INFO] server-3839dd79-d610-736e-39aa-f079d1ef431d: aborting pipeline replication: peer="{Voter server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159 cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159}" 2025-09-14T21:48:48.818Z [INFO] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: election won: term=3 tally=2 2025-09-14T21:48:48.818Z [INFO] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: entering leader state: leader="Node at cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159 [Leader]" 2025-09-14T21:48:48.818Z [INFO] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: added peer, starting replication: peer=server-3839dd79-d610-736e-39aa-f079d1ef431d 2025-09-14T21:48:48.818Z [INFO] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: pipelining replication: peer="{Voter server-3839dd79-d610-736e-39aa-f079d1ef431d 3839dd79-d610-736e-39aa-f079d1ef431d}" 2025-09-14T21:48:48.923Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:48.818751095 +0000 UTC m=+99.386629476 (1 nodes), 297.066µs from start of poll, 509.949633ms from cluster start. Timeout at 2025-09-14 21:48:48.923752024 +0000 UTC m=+99.491630285, 105.000809ms after stability raft_test.go:1799: [INFO] Disconnecting Node at 3839dd79-d610-736e-39aa-f079d1ef431d [Follower] 2025-09-14T21:48:48.923Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:48.927Z [ERROR] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: failed to heartbeat to: peer=3839dd79-d610-736e-39aa-f079d1ef431d backoff time=10ms error="failed to connect to peer: 3839dd79-d610-736e-39aa-f079d1ef431d" 2025-09-14T21:48:48.927Z [ERROR] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: failed to pipeline appendEntries: peer="{Voter server-3839dd79-d610-736e-39aa-f079d1ef431d 3839dd79-d610-736e-39aa-f079d1ef431d}" error="append pipeline closed" 2025-09-14T21:48:48.927Z [INFO] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: aborting pipeline replication: peer="{Voter server-3839dd79-d610-736e-39aa-f079d1ef431d 3839dd79-d610-736e-39aa-f079d1ef431d}" 2025-09-14T21:48:48.934Z [ERROR] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: failed to appendEntries to: peer="{Voter server-3839dd79-d610-736e-39aa-f079d1ef431d 3839dd79-d610-736e-39aa-f079d1ef431d}" error="failed to connect to peer: 3839dd79-d610-736e-39aa-f079d1ef431d" 2025-09-14T21:48:48.943Z [ERROR] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: failed to heartbeat to: peer=3839dd79-d610-736e-39aa-f079d1ef431d backoff time=10ms error="failed to connect to peer: 3839dd79-d610-736e-39aa-f079d1ef431d" 2025-09-14T21:48:48.952Z [ERROR] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: failed to appendEntries to: peer="{Voter server-3839dd79-d610-736e-39aa-f079d1ef431d 3839dd79-d610-736e-39aa-f079d1ef431d}" error="failed to connect to peer: 3839dd79-d610-736e-39aa-f079d1ef431d" 2025-09-14T21:48:48.962Z [ERROR] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: failed to heartbeat to: peer=3839dd79-d610-736e-39aa-f079d1ef431d backoff time=10ms error="failed to connect to peer: 3839dd79-d610-736e-39aa-f079d1ef431d" 2025-09-14T21:48:48.970Z [ERROR] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: failed to appendEntries to: peer="{Voter server-3839dd79-d610-736e-39aa-f079d1ef431d 3839dd79-d610-736e-39aa-f079d1ef431d}" error="failed to connect to peer: 3839dd79-d610-736e-39aa-f079d1ef431d" 2025-09-14T21:48:48.976Z [WARN] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: failed to contact: server-id=server-3839dd79-d610-736e-39aa-f079d1ef431d time=54.863077ms 2025-09-14T21:48:48.976Z [WARN] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: failed to contact quorum of nodes, stepping down 2025-09-14T21:48:48.976Z [INFO] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: entering follower state: follower="Node at cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159 [Follower]" leader-address= leader-id= 2025-09-14T21:48:49.050Z [WARN] server-3839dd79-d610-736e-39aa-f079d1ef431d: heartbeat timeout reached, starting election: last-leader-addr=cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159 last-leader-id=server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159 2025-09-14T21:48:49.050Z [INFO] server-3839dd79-d610-736e-39aa-f079d1ef431d: entering candidate state: node="Node at 3839dd79-d610-736e-39aa-f079d1ef431d [Candidate]" term=4 2025-09-14T21:48:49.050Z [ERROR] server-3839dd79-d610-736e-39aa-f079d1ef431d: failed to make requestVote RPC: target="{Voter server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159 cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159}" error="failed to connect to peer: cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159" term=4 2025-09-14T21:48:49.062Z [WARN] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:49.062Z [INFO] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: entering candidate state: node="Node at cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159 [Candidate]" term=4 2025-09-14T21:48:49.062Z [ERROR] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: failed to make requestVote RPC: target="{Voter server-3839dd79-d610-736e-39aa-f079d1ef431d 3839dd79-d610-736e-39aa-f079d1ef431d}" error="failed to connect to peer: 3839dd79-d610-736e-39aa-f079d1ef431d" term=4 2025-09-14T21:48:49.125Z [WARN] server-3839dd79-d610-736e-39aa-f079d1ef431d: Election timeout reached, restarting election 2025-09-14T21:48:49.125Z [INFO] server-3839dd79-d610-736e-39aa-f079d1ef431d: entering candidate state: node="Node at 3839dd79-d610-736e-39aa-f079d1ef431d [Candidate]" term=5 2025-09-14T21:48:49.126Z [ERROR] server-3839dd79-d610-736e-39aa-f079d1ef431d: failed to make requestVote RPC: target="{Voter server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159 cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159}" error="failed to connect to peer: cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159" term=5 2025-09-14T21:48:49.160Z [WARN] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: Election timeout reached, restarting election 2025-09-14T21:48:49.160Z [INFO] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: entering candidate state: node="Node at cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159 [Candidate]" term=5 2025-09-14T21:48:49.160Z [ERROR] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: failed to make requestVote RPC: target="{Voter server-3839dd79-d610-736e-39aa-f079d1ef431d 3839dd79-d610-736e-39aa-f079d1ef431d}" error="failed to connect to peer: 3839dd79-d610-736e-39aa-f079d1ef431d" term=5 2025-09-14T21:48:49.167Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:49.062275222 +0000 UTC m=+99.630153523 (0 nodes), 138.396271ms from start of poll, 753.47368ms from cluster start. Timeout at 2025-09-14 21:48:49.16727949 +0000 UTC m=+99.735157765, 105.004242ms after stability 2025-09-14T21:48:49.193Z [WARN] server-3839dd79-d610-736e-39aa-f079d1ef431d: Election timeout reached, restarting election 2025-09-14T21:48:49.193Z [INFO] server-3839dd79-d610-736e-39aa-f079d1ef431d: entering candidate state: node="Node at 3839dd79-d610-736e-39aa-f079d1ef431d [Candidate]" term=6 2025-09-14T21:48:49.193Z [ERROR] server-3839dd79-d610-736e-39aa-f079d1ef431d: failed to make requestVote RPC: target="{Voter server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159 cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159}" error="failed to connect to peer: cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159" term=6 2025-09-14T21:48:49.241Z [WARN] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: Election timeout reached, restarting election 2025-09-14T21:48:49.241Z [INFO] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: entering candidate state: node="Node at cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159 [Candidate]" term=6 2025-09-14T21:48:49.241Z [ERROR] server-cdb6e77f-e7b4-6e8f-0c54-9831ee3dd159: failed to make requestVote RPC: target="{Voter server-3839dd79-d610-736e-39aa-f079d1ef431d 3839dd79-d610-736e-39aa-f079d1ef431d}" error="failed to connect to peer: 3839dd79-d610-736e-39aa-f079d1ef431d" term=6 --- PASS: TestRaft_LeaderLeaseExpire (0.96s) === RUN TestRaft_Barrier 2025-09-14T21:48:49.274Z [INFO] server-e562165a-3000-ba4b-5776-5ac3b2ec6ac1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e562165a-3000-ba4b-5776-5ac3b2ec6ac1 Address:e562165a-3000-ba4b-5776-5ac3b2ec6ac1} {Suffrage:Voter ID:server-532f3cd7-00b4-8a18-6d45-e1fa0053cc40 Address:532f3cd7-00b4-8a18-6d45-e1fa0053cc40} {Suffrage:Voter ID:server-5fab4665-5ff2-858a-284a-28cb1793cf24 Address:5fab4665-5ff2-858a-284a-28cb1793cf24}]" 2025-09-14T21:48:49.274Z [INFO] server-532f3cd7-00b4-8a18-6d45-e1fa0053cc40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e562165a-3000-ba4b-5776-5ac3b2ec6ac1 Address:e562165a-3000-ba4b-5776-5ac3b2ec6ac1} {Suffrage:Voter ID:server-532f3cd7-00b4-8a18-6d45-e1fa0053cc40 Address:532f3cd7-00b4-8a18-6d45-e1fa0053cc40} {Suffrage:Voter ID:server-5fab4665-5ff2-858a-284a-28cb1793cf24 Address:5fab4665-5ff2-858a-284a-28cb1793cf24}]" 2025-09-14T21:48:49.274Z [INFO] server-e562165a-3000-ba4b-5776-5ac3b2ec6ac1: entering follower state: follower="Node at e562165a-3000-ba4b-5776-5ac3b2ec6ac1 [Follower]" leader-address= leader-id= 2025-09-14T21:48:49.274Z [INFO] server-532f3cd7-00b4-8a18-6d45-e1fa0053cc40: entering follower state: follower="Node at 532f3cd7-00b4-8a18-6d45-e1fa0053cc40 [Follower]" leader-address= leader-id= 2025-09-14T21:48:49.274Z [INFO] server-5fab4665-5ff2-858a-284a-28cb1793cf24: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e562165a-3000-ba4b-5776-5ac3b2ec6ac1 Address:e562165a-3000-ba4b-5776-5ac3b2ec6ac1} {Suffrage:Voter ID:server-532f3cd7-00b4-8a18-6d45-e1fa0053cc40 Address:532f3cd7-00b4-8a18-6d45-e1fa0053cc40} {Suffrage:Voter ID:server-5fab4665-5ff2-858a-284a-28cb1793cf24 Address:5fab4665-5ff2-858a-284a-28cb1793cf24}]" 2025-09-14T21:48:49.274Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:49.274Z [INFO] server-5fab4665-5ff2-858a-284a-28cb1793cf24: entering follower state: follower="Node at 5fab4665-5ff2-858a-284a-28cb1793cf24 [Follower]" leader-address= leader-id= 2025-09-14T21:48:49.357Z [WARN] server-5fab4665-5ff2-858a-284a-28cb1793cf24: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:49.357Z [INFO] server-5fab4665-5ff2-858a-284a-28cb1793cf24: entering candidate state: node="Node at 5fab4665-5ff2-858a-284a-28cb1793cf24 [Candidate]" term=2 2025-09-14T21:48:49.357Z [WARN] server-e562165a-3000-ba4b-5776-5ac3b2ec6ac1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:49.357Z [INFO] server-e562165a-3000-ba4b-5776-5ac3b2ec6ac1: entering candidate state: node="Node at e562165a-3000-ba4b-5776-5ac3b2ec6ac1 [Candidate]" term=2 2025-09-14T21:48:49.357Z [INFO] server-e562165a-3000-ba4b-5776-5ac3b2ec6ac1: duplicate requestVote for same term: term=2 2025-09-14T21:48:49.357Z [INFO] server-5fab4665-5ff2-858a-284a-28cb1793cf24: election won: term=2 tally=2 2025-09-14T21:48:49.357Z [INFO] server-5fab4665-5ff2-858a-284a-28cb1793cf24: entering leader state: leader="Node at 5fab4665-5ff2-858a-284a-28cb1793cf24 [Leader]" 2025-09-14T21:48:49.357Z [INFO] server-5fab4665-5ff2-858a-284a-28cb1793cf24: added peer, starting replication: peer=server-e562165a-3000-ba4b-5776-5ac3b2ec6ac1 2025-09-14T21:48:49.357Z [INFO] server-5fab4665-5ff2-858a-284a-28cb1793cf24: added peer, starting replication: peer=server-532f3cd7-00b4-8a18-6d45-e1fa0053cc40 2025-09-14T21:48:49.357Z [INFO] server-532f3cd7-00b4-8a18-6d45-e1fa0053cc40: duplicate requestVote for same term: term=2 2025-09-14T21:48:49.357Z [WARN] server-5fab4665-5ff2-858a-284a-28cb1793cf24: rejecting vote request since we have a leader: from=e562165a-3000-ba4b-5776-5ac3b2ec6ac1 leader=5fab4665-5ff2-858a-284a-28cb1793cf24 leader-id=server-5fab4665-5ff2-858a-284a-28cb1793cf24 2025-09-14T21:48:49.357Z [INFO] server-e562165a-3000-ba4b-5776-5ac3b2ec6ac1: entering follower state: follower="Node at e562165a-3000-ba4b-5776-5ac3b2ec6ac1 [Follower]" leader-address=5fab4665-5ff2-858a-284a-28cb1793cf24 leader-id=server-5fab4665-5ff2-858a-284a-28cb1793cf24 2025-09-14T21:48:49.357Z [INFO] server-5fab4665-5ff2-858a-284a-28cb1793cf24: pipelining replication: peer="{Voter server-532f3cd7-00b4-8a18-6d45-e1fa0053cc40 532f3cd7-00b4-8a18-6d45-e1fa0053cc40}" 2025-09-14T21:48:49.357Z [INFO] server-5fab4665-5ff2-858a-284a-28cb1793cf24: pipelining replication: peer="{Voter server-e562165a-3000-ba4b-5776-5ac3b2ec6ac1 e562165a-3000-ba4b-5776-5ac3b2ec6ac1}" 2025-09-14T21:48:49.463Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:49.357801644 +0000 UTC m=+99.925679978 (1 nodes), 83.470313ms from start of poll, 83.81988ms from cluster start. Timeout at 2025-09-14 21:48:49.4628066 +0000 UTC m=+100.030684858, 105.00488ms after stability 2025-09-14T21:48:49.477Z [INFO] server-5fab4665-5ff2-858a-284a-28cb1793cf24: aborting pipeline replication: peer="{Voter server-532f3cd7-00b4-8a18-6d45-e1fa0053cc40 532f3cd7-00b4-8a18-6d45-e1fa0053cc40}" 2025-09-14T21:48:49.477Z [INFO] server-5fab4665-5ff2-858a-284a-28cb1793cf24: aborting pipeline replication: peer="{Voter server-e562165a-3000-ba4b-5776-5ac3b2ec6ac1 e562165a-3000-ba4b-5776-5ac3b2ec6ac1}" --- PASS: TestRaft_Barrier (0.20s) === RUN TestRaft_VerifyLeader 2025-09-14T21:48:49.479Z [INFO] server-62de5e29-63a9-89a2-7789-10f6a90587f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62de5e29-63a9-89a2-7789-10f6a90587f2 Address:62de5e29-63a9-89a2-7789-10f6a90587f2} {Suffrage:Voter ID:server-0fc5a27b-e479-7414-b558-f20154ccd453 Address:0fc5a27b-e479-7414-b558-f20154ccd453} {Suffrage:Voter ID:server-e71be3d4-a465-c9ed-7631-10654009c5e1 Address:e71be3d4-a465-c9ed-7631-10654009c5e1}]" 2025-09-14T21:48:49.479Z [INFO] server-62de5e29-63a9-89a2-7789-10f6a90587f2: entering follower state: follower="Node at 62de5e29-63a9-89a2-7789-10f6a90587f2 [Follower]" leader-address= leader-id= 2025-09-14T21:48:49.479Z [INFO] server-0fc5a27b-e479-7414-b558-f20154ccd453: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62de5e29-63a9-89a2-7789-10f6a90587f2 Address:62de5e29-63a9-89a2-7789-10f6a90587f2} {Suffrage:Voter ID:server-0fc5a27b-e479-7414-b558-f20154ccd453 Address:0fc5a27b-e479-7414-b558-f20154ccd453} {Suffrage:Voter ID:server-e71be3d4-a465-c9ed-7631-10654009c5e1 Address:e71be3d4-a465-c9ed-7631-10654009c5e1}]" 2025-09-14T21:48:49.479Z [INFO] server-e71be3d4-a465-c9ed-7631-10654009c5e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62de5e29-63a9-89a2-7789-10f6a90587f2 Address:62de5e29-63a9-89a2-7789-10f6a90587f2} {Suffrage:Voter ID:server-0fc5a27b-e479-7414-b558-f20154ccd453 Address:0fc5a27b-e479-7414-b558-f20154ccd453} {Suffrage:Voter ID:server-e71be3d4-a465-c9ed-7631-10654009c5e1 Address:e71be3d4-a465-c9ed-7631-10654009c5e1}]" 2025-09-14T21:48:49.479Z [INFO] server-0fc5a27b-e479-7414-b558-f20154ccd453: entering follower state: follower="Node at 0fc5a27b-e479-7414-b558-f20154ccd453 [Follower]" leader-address= leader-id= 2025-09-14T21:48:49.479Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:49.479Z [INFO] server-e71be3d4-a465-c9ed-7631-10654009c5e1: entering follower state: follower="Node at e71be3d4-a465-c9ed-7631-10654009c5e1 [Follower]" leader-address= leader-id= 2025-09-14T21:48:49.530Z [WARN] server-e71be3d4-a465-c9ed-7631-10654009c5e1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:49.530Z [INFO] server-e71be3d4-a465-c9ed-7631-10654009c5e1: entering candidate state: node="Node at e71be3d4-a465-c9ed-7631-10654009c5e1 [Candidate]" term=2 2025-09-14T21:48:49.530Z [INFO] server-e71be3d4-a465-c9ed-7631-10654009c5e1: election won: term=2 tally=2 2025-09-14T21:48:49.530Z [INFO] server-e71be3d4-a465-c9ed-7631-10654009c5e1: entering leader state: leader="Node at e71be3d4-a465-c9ed-7631-10654009c5e1 [Leader]" 2025-09-14T21:48:49.530Z [INFO] server-e71be3d4-a465-c9ed-7631-10654009c5e1: added peer, starting replication: peer=server-62de5e29-63a9-89a2-7789-10f6a90587f2 2025-09-14T21:48:49.530Z [INFO] server-e71be3d4-a465-c9ed-7631-10654009c5e1: added peer, starting replication: peer=server-0fc5a27b-e479-7414-b558-f20154ccd453 2025-09-14T21:48:49.530Z [INFO] server-e71be3d4-a465-c9ed-7631-10654009c5e1: pipelining replication: peer="{Voter server-0fc5a27b-e479-7414-b558-f20154ccd453 0fc5a27b-e479-7414-b558-f20154ccd453}" 2025-09-14T21:48:49.530Z [INFO] server-e71be3d4-a465-c9ed-7631-10654009c5e1: pipelining replication: peer="{Voter server-62de5e29-63a9-89a2-7789-10f6a90587f2 62de5e29-63a9-89a2-7789-10f6a90587f2}" 2025-09-14T21:48:49.635Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:49.530542416 +0000 UTC m=+100.098420678 (1 nodes), 51.325983ms from start of poll, 51.60201ms from cluster start. Timeout at 2025-09-14 21:48:49.635547687 +0000 UTC m=+100.203425963, 105.005285ms after stability 2025-09-14T21:48:49.635Z [INFO] server-e71be3d4-a465-c9ed-7631-10654009c5e1: aborting pipeline replication: peer="{Voter server-0fc5a27b-e479-7414-b558-f20154ccd453 0fc5a27b-e479-7414-b558-f20154ccd453}" 2025-09-14T21:48:49.635Z [INFO] server-e71be3d4-a465-c9ed-7631-10654009c5e1: aborting pipeline replication: peer="{Voter server-62de5e29-63a9-89a2-7789-10f6a90587f2 62de5e29-63a9-89a2-7789-10f6a90587f2}" 2025-09-14T21:48:50.135Z [ERROR] server-e71be3d4-a465-c9ed-7631-10654009c5e1: failed to heartbeat to: peer=62de5e29-63a9-89a2-7789-10f6a90587f2 backoff time=10ms error="command timed out" --- PASS: TestRaft_VerifyLeader (0.66s) === RUN TestRaft_VerifyLeader_Single 2025-09-14T21:48:50.136Z [INFO] server-a3ace9d9-1fd3-6ccb-b375-efaf7d8ad6ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3ace9d9-1fd3-6ccb-b375-efaf7d8ad6ac Address:a3ace9d9-1fd3-6ccb-b375-efaf7d8ad6ac}]" 2025-09-14T21:48:50.136Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:50.136Z [INFO] server-a3ace9d9-1fd3-6ccb-b375-efaf7d8ad6ac: entering follower state: follower="Node at a3ace9d9-1fd3-6ccb-b375-efaf7d8ad6ac [Follower]" leader-address= leader-id= 2025-09-14T21:48:50.207Z [WARN] server-a3ace9d9-1fd3-6ccb-b375-efaf7d8ad6ac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:50.207Z [INFO] server-a3ace9d9-1fd3-6ccb-b375-efaf7d8ad6ac: entering candidate state: node="Node at a3ace9d9-1fd3-6ccb-b375-efaf7d8ad6ac [Candidate]" term=2 2025-09-14T21:48:50.207Z [INFO] server-a3ace9d9-1fd3-6ccb-b375-efaf7d8ad6ac: election won: term=2 tally=1 2025-09-14T21:48:50.207Z [INFO] server-a3ace9d9-1fd3-6ccb-b375-efaf7d8ad6ac: entering leader state: leader="Node at a3ace9d9-1fd3-6ccb-b375-efaf7d8ad6ac [Leader]" 2025-09-14T21:48:50.312Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:50.207319149 +0000 UTC m=+100.775197430 (1 nodes), 70.436223ms from start of poll, 70.584693ms from cluster start. Timeout at 2025-09-14 21:48:50.312325919 +0000 UTC m=+100.880204197, 105.006767ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.18s) === RUN TestRaft_VerifyLeader_Fail 2025-09-14T21:48:50.313Z [INFO] server-587c7702-63ed-260e-9dc1-fed0b882a907: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-587c7702-63ed-260e-9dc1-fed0b882a907 Address:587c7702-63ed-260e-9dc1-fed0b882a907} {Suffrage:Voter ID:server-17bdda86-83d0-83b4-a5ee-46856fc5ffe5 Address:17bdda86-83d0-83b4-a5ee-46856fc5ffe5}]" 2025-09-14T21:48:50.313Z [INFO] server-587c7702-63ed-260e-9dc1-fed0b882a907: entering follower state: follower="Node at 587c7702-63ed-260e-9dc1-fed0b882a907 [Follower]" leader-address= leader-id= 2025-09-14T21:48:50.313Z [INFO] server-17bdda86-83d0-83b4-a5ee-46856fc5ffe5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-587c7702-63ed-260e-9dc1-fed0b882a907 Address:587c7702-63ed-260e-9dc1-fed0b882a907} {Suffrage:Voter ID:server-17bdda86-83d0-83b4-a5ee-46856fc5ffe5 Address:17bdda86-83d0-83b4-a5ee-46856fc5ffe5}]" 2025-09-14T21:48:50.313Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:50.313Z [INFO] server-17bdda86-83d0-83b4-a5ee-46856fc5ffe5: entering follower state: follower="Node at 17bdda86-83d0-83b4-a5ee-46856fc5ffe5 [Follower]" leader-address= leader-id= 2025-09-14T21:48:50.367Z [WARN] server-587c7702-63ed-260e-9dc1-fed0b882a907: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:50.368Z [INFO] server-587c7702-63ed-260e-9dc1-fed0b882a907: entering candidate state: node="Node at 587c7702-63ed-260e-9dc1-fed0b882a907 [Candidate]" term=2 2025-09-14T21:48:50.368Z [INFO] server-587c7702-63ed-260e-9dc1-fed0b882a907: election won: term=2 tally=2 2025-09-14T21:48:50.368Z [INFO] server-587c7702-63ed-260e-9dc1-fed0b882a907: entering leader state: leader="Node at 587c7702-63ed-260e-9dc1-fed0b882a907 [Leader]" 2025-09-14T21:48:50.368Z [INFO] server-587c7702-63ed-260e-9dc1-fed0b882a907: added peer, starting replication: peer=server-17bdda86-83d0-83b4-a5ee-46856fc5ffe5 2025-09-14T21:48:50.368Z [INFO] server-587c7702-63ed-260e-9dc1-fed0b882a907: pipelining replication: peer="{Voter server-17bdda86-83d0-83b4-a5ee-46856fc5ffe5 17bdda86-83d0-83b4-a5ee-46856fc5ffe5}" 2025-09-14T21:48:50.474Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:50.368182252 +0000 UTC m=+100.936060520 (1 nodes), 54.37887ms from start of poll, 54.650917ms from cluster start. Timeout at 2025-09-14 21:48:50.473186362 +0000 UTC m=+101.041064638, 105.004118ms after stability 2025-09-14T21:48:50.474Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:50.579Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:50.474143484 +0000 UTC m=+101.042021749 (1 nodes), 560ns from start of poll, 160.612146ms from cluster start. Timeout at 2025-09-14 21:48:50.579169679 +0000 UTC m=+101.147047938, 105.026189ms after stability 2025-09-14T21:48:50.581Z [ERROR] server-587c7702-63ed-260e-9dc1-fed0b882a907: peer has newer term, stopping replication: peer="{Voter server-17bdda86-83d0-83b4-a5ee-46856fc5ffe5 17bdda86-83d0-83b4-a5ee-46856fc5ffe5}" 2025-09-14T21:48:50.581Z [INFO] server-587c7702-63ed-260e-9dc1-fed0b882a907: aborting pipeline replication: peer="{Voter server-17bdda86-83d0-83b4-a5ee-46856fc5ffe5 17bdda86-83d0-83b4-a5ee-46856fc5ffe5}" 2025-09-14T21:48:50.581Z [INFO] server-587c7702-63ed-260e-9dc1-fed0b882a907: entering follower state: follower="Node at 587c7702-63ed-260e-9dc1-fed0b882a907 [Follower]" leader-address= leader-id= --- PASS: TestRaft_VerifyLeader_Fail (0.42s) === RUN TestRaft_VerifyLeader_PartialConnect 2025-09-14T21:48:50.731Z [INFO] server-7409e641-b63e-a7a1-ee6c-131b1607dcf6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7409e641-b63e-a7a1-ee6c-131b1607dcf6 Address:7409e641-b63e-a7a1-ee6c-131b1607dcf6} {Suffrage:Voter ID:server-49235db0-12ff-cd76-d2e9-de1e1273c26c Address:49235db0-12ff-cd76-d2e9-de1e1273c26c} {Suffrage:Voter ID:server-9e10fe04-ccd7-7499-65db-9351072d7b87 Address:9e10fe04-ccd7-7499-65db-9351072d7b87}]" 2025-09-14T21:48:50.731Z [INFO] server-7409e641-b63e-a7a1-ee6c-131b1607dcf6: entering follower state: follower="Node at 7409e641-b63e-a7a1-ee6c-131b1607dcf6 [Follower]" leader-address= leader-id= 2025-09-14T21:48:50.732Z [INFO] server-49235db0-12ff-cd76-d2e9-de1e1273c26c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7409e641-b63e-a7a1-ee6c-131b1607dcf6 Address:7409e641-b63e-a7a1-ee6c-131b1607dcf6} {Suffrage:Voter ID:server-49235db0-12ff-cd76-d2e9-de1e1273c26c Address:49235db0-12ff-cd76-d2e9-de1e1273c26c} {Suffrage:Voter ID:server-9e10fe04-ccd7-7499-65db-9351072d7b87 Address:9e10fe04-ccd7-7499-65db-9351072d7b87}]" 2025-09-14T21:48:50.732Z [INFO] server-49235db0-12ff-cd76-d2e9-de1e1273c26c: entering follower state: follower="Node at 49235db0-12ff-cd76-d2e9-de1e1273c26c [Follower]" leader-address= leader-id= 2025-09-14T21:48:50.732Z [INFO] server-9e10fe04-ccd7-7499-65db-9351072d7b87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7409e641-b63e-a7a1-ee6c-131b1607dcf6 Address:7409e641-b63e-a7a1-ee6c-131b1607dcf6} {Suffrage:Voter ID:server-49235db0-12ff-cd76-d2e9-de1e1273c26c Address:49235db0-12ff-cd76-d2e9-de1e1273c26c} {Suffrage:Voter ID:server-9e10fe04-ccd7-7499-65db-9351072d7b87 Address:9e10fe04-ccd7-7499-65db-9351072d7b87}]" 2025-09-14T21:48:50.732Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:50.732Z [INFO] server-9e10fe04-ccd7-7499-65db-9351072d7b87: entering follower state: follower="Node at 9e10fe04-ccd7-7499-65db-9351072d7b87 [Follower]" leader-address= leader-id= 2025-09-14T21:48:50.815Z [WARN] server-7409e641-b63e-a7a1-ee6c-131b1607dcf6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:50.815Z [INFO] server-7409e641-b63e-a7a1-ee6c-131b1607dcf6: entering candidate state: node="Node at 7409e641-b63e-a7a1-ee6c-131b1607dcf6 [Candidate]" term=2 2025-09-14T21:48:50.815Z [INFO] server-7409e641-b63e-a7a1-ee6c-131b1607dcf6: election won: term=2 tally=2 2025-09-14T21:48:50.815Z [INFO] server-7409e641-b63e-a7a1-ee6c-131b1607dcf6: entering leader state: leader="Node at 7409e641-b63e-a7a1-ee6c-131b1607dcf6 [Leader]" 2025-09-14T21:48:50.815Z [INFO] server-7409e641-b63e-a7a1-ee6c-131b1607dcf6: added peer, starting replication: peer=server-49235db0-12ff-cd76-d2e9-de1e1273c26c 2025-09-14T21:48:50.815Z [INFO] server-7409e641-b63e-a7a1-ee6c-131b1607dcf6: added peer, starting replication: peer=server-9e10fe04-ccd7-7499-65db-9351072d7b87 2025-09-14T21:48:50.815Z [INFO] server-7409e641-b63e-a7a1-ee6c-131b1607dcf6: pipelining replication: peer="{Voter server-9e10fe04-ccd7-7499-65db-9351072d7b87 9e10fe04-ccd7-7499-65db-9351072d7b87}" 2025-09-14T21:48:50.815Z [INFO] server-7409e641-b63e-a7a1-ee6c-131b1607dcf6: pipelining replication: peer="{Voter server-49235db0-12ff-cd76-d2e9-de1e1273c26c 49235db0-12ff-cd76-d2e9-de1e1273c26c}" 2025-09-14T21:48:50.920Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:50.815302089 +0000 UTC m=+101.383180370 (1 nodes), 83.135762ms from start of poll, 83.486697ms from cluster start. Timeout at 2025-09-14 21:48:50.920303229 +0000 UTC m=+101.488181488, 105.001118ms after stability 2025-09-14T21:48:50.926Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:51.031Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:50.926112469 +0000 UTC m=+101.493990730 (2 nodes), 2.31µs from start of poll, 194.297057ms from cluster start. Timeout at 2025-09-14 21:48:51.031131954 +0000 UTC m=+101.599010217, 105.019487ms after stability raft_test.go:1968: [INFO] Disconnecting Node at 49235db0-12ff-cd76-d2e9-de1e1273c26c [Follower] 2025-09-14T21:48:51.031Z [INFO] server-7409e641-b63e-a7a1-ee6c-131b1607dcf6: aborting pipeline replication: peer="{Voter server-9e10fe04-ccd7-7499-65db-9351072d7b87 9e10fe04-ccd7-7499-65db-9351072d7b87}" 2025-09-14T21:48:51.031Z [ERROR] server-7409e641-b63e-a7a1-ee6c-131b1607dcf6: failed to heartbeat to: peer=49235db0-12ff-cd76-d2e9-de1e1273c26c backoff time=10ms error="failed to connect to peer: 49235db0-12ff-cd76-d2e9-de1e1273c26c" 2025-09-14T21:48:51.031Z [INFO] server-7409e641-b63e-a7a1-ee6c-131b1607dcf6: aborting pipeline replication: peer="{Voter server-49235db0-12ff-cd76-d2e9-de1e1273c26c 49235db0-12ff-cd76-d2e9-de1e1273c26c}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.30s) === RUN TestRaft_NotifyCh 2025-09-14T21:48:51.032Z [INFO] server-1376a200-9d2d-b87a-c851-f97a58b9b48d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1376a200-9d2d-b87a-c851-f97a58b9b48d Address:1376a200-9d2d-b87a-c851-f97a58b9b48d}]" 2025-09-14T21:48:51.032Z [INFO] server-1376a200-9d2d-b87a-c851-f97a58b9b48d: entering follower state: follower="Node at 1376a200-9d2d-b87a-c851-f97a58b9b48d [Follower]" leader-address= leader-id= 2025-09-14T21:48:51.116Z [WARN] server-1376a200-9d2d-b87a-c851-f97a58b9b48d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:51.116Z [INFO] server-1376a200-9d2d-b87a-c851-f97a58b9b48d: entering candidate state: node="Node at 1376a200-9d2d-b87a-c851-f97a58b9b48d [Candidate]" term=2 2025-09-14T21:48:51.116Z [INFO] server-1376a200-9d2d-b87a-c851-f97a58b9b48d: election won: term=2 tally=1 2025-09-14T21:48:51.116Z [INFO] server-1376a200-9d2d-b87a-c851-f97a58b9b48d: entering leader state: leader="Node at 1376a200-9d2d-b87a-c851-f97a58b9b48d [Leader]" --- PASS: TestRaft_NotifyCh (0.13s) === RUN TestRaft_AppendEntry 2025-09-14T21:48:51.158Z [INFO] server-f169d1cd-d4bb-ffc1-8284-292a4b81eeea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f169d1cd-d4bb-ffc1-8284-292a4b81eeea Address:f169d1cd-d4bb-ffc1-8284-292a4b81eeea} {Suffrage:Voter ID:server-f0d873d9-81cd-6dc2-14bc-d48fca48b950 Address:f0d873d9-81cd-6dc2-14bc-d48fca48b950} {Suffrage:Voter ID:server-fd969ad0-e9a3-0908-f26b-dd72d7db5cc8 Address:fd969ad0-e9a3-0908-f26b-dd72d7db5cc8}]" 2025-09-14T21:48:51.158Z [INFO] server-f0d873d9-81cd-6dc2-14bc-d48fca48b950: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f169d1cd-d4bb-ffc1-8284-292a4b81eeea Address:f169d1cd-d4bb-ffc1-8284-292a4b81eeea} {Suffrage:Voter ID:server-f0d873d9-81cd-6dc2-14bc-d48fca48b950 Address:f0d873d9-81cd-6dc2-14bc-d48fca48b950} {Suffrage:Voter ID:server-fd969ad0-e9a3-0908-f26b-dd72d7db5cc8 Address:fd969ad0-e9a3-0908-f26b-dd72d7db5cc8}]" 2025-09-14T21:48:51.158Z [INFO] server-f169d1cd-d4bb-ffc1-8284-292a4b81eeea: entering follower state: follower="Node at f169d1cd-d4bb-ffc1-8284-292a4b81eeea [Follower]" leader-address= leader-id= 2025-09-14T21:48:51.158Z [INFO] server-fd969ad0-e9a3-0908-f26b-dd72d7db5cc8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f169d1cd-d4bb-ffc1-8284-292a4b81eeea Address:f169d1cd-d4bb-ffc1-8284-292a4b81eeea} {Suffrage:Voter ID:server-f0d873d9-81cd-6dc2-14bc-d48fca48b950 Address:f0d873d9-81cd-6dc2-14bc-d48fca48b950} {Suffrage:Voter ID:server-fd969ad0-e9a3-0908-f26b-dd72d7db5cc8 Address:fd969ad0-e9a3-0908-f26b-dd72d7db5cc8}]" 2025-09-14T21:48:51.158Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:51.158Z [INFO] server-f0d873d9-81cd-6dc2-14bc-d48fca48b950: entering follower state: follower="Node at f0d873d9-81cd-6dc2-14bc-d48fca48b950 [Follower]" leader-address= leader-id= 2025-09-14T21:48:51.158Z [INFO] server-fd969ad0-e9a3-0908-f26b-dd72d7db5cc8: entering follower state: follower="Node at fd969ad0-e9a3-0908-f26b-dd72d7db5cc8 [Follower]" leader-address= leader-id= 2025-09-14T21:48:51.227Z [WARN] server-f0d873d9-81cd-6dc2-14bc-d48fca48b950: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:51.228Z [INFO] server-f0d873d9-81cd-6dc2-14bc-d48fca48b950: entering candidate state: node="Node at f0d873d9-81cd-6dc2-14bc-d48fca48b950 [Candidate]" term=2 2025-09-14T21:48:51.228Z [INFO] server-f0d873d9-81cd-6dc2-14bc-d48fca48b950: election won: term=2 tally=2 2025-09-14T21:48:51.228Z [INFO] server-f0d873d9-81cd-6dc2-14bc-d48fca48b950: entering leader state: leader="Node at f0d873d9-81cd-6dc2-14bc-d48fca48b950 [Leader]" 2025-09-14T21:48:51.228Z [INFO] server-f0d873d9-81cd-6dc2-14bc-d48fca48b950: added peer, starting replication: peer=server-f169d1cd-d4bb-ffc1-8284-292a4b81eeea 2025-09-14T21:48:51.228Z [INFO] server-f0d873d9-81cd-6dc2-14bc-d48fca48b950: added peer, starting replication: peer=server-fd969ad0-e9a3-0908-f26b-dd72d7db5cc8 2025-09-14T21:48:51.228Z [INFO] server-f0d873d9-81cd-6dc2-14bc-d48fca48b950: pipelining replication: peer="{Voter server-fd969ad0-e9a3-0908-f26b-dd72d7db5cc8 fd969ad0-e9a3-0908-f26b-dd72d7db5cc8}" 2025-09-14T21:48:51.228Z [INFO] server-f0d873d9-81cd-6dc2-14bc-d48fca48b950: pipelining replication: peer="{Voter server-f169d1cd-d4bb-ffc1-8284-292a4b81eeea f169d1cd-d4bb-ffc1-8284-292a4b81eeea}" 2025-09-14T21:48:51.333Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:51.228158143 +0000 UTC m=+101.796036406 (2 nodes), 69.542998ms from start of poll, 69.799177ms from cluster start. Timeout at 2025-09-14 21:48:51.333161122 +0000 UTC m=+101.901039381, 105.002975ms after stability 2025-09-14T21:48:51.333Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:51.439Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:51.333750901 +0000 UTC m=+101.901629164 (1 nodes), 764ns from start of poll, 175.391935ms from cluster start. Timeout at 2025-09-14 21:48:51.438771809 +0000 UTC m=+102.006650091, 105.020927ms after stability 2025-09-14T21:48:51.439Z [WARN] server-f169d1cd-d4bb-ffc1-8284-292a4b81eeea: clearing log suffix: from=1 to=2 2025-09-14T21:48:51.439Z [WARN] server-f169d1cd-d4bb-ffc1-8284-292a4b81eeea: skipping application of old log: index=1 2025-09-14T21:48:51.439Z [WARN] server-f169d1cd-d4bb-ffc1-8284-292a4b81eeea: skipping application of old log: index=1 2025-09-14T21:48:51.439Z [INFO] server-f0d873d9-81cd-6dc2-14bc-d48fca48b950: aborting pipeline replication: peer="{Voter server-f169d1cd-d4bb-ffc1-8284-292a4b81eeea f169d1cd-d4bb-ffc1-8284-292a4b81eeea}" 2025-09-14T21:48:51.439Z [INFO] server-f0d873d9-81cd-6dc2-14bc-d48fca48b950: aborting pipeline replication: peer="{Voter server-fd969ad0-e9a3-0908-f26b-dd72d7db5cc8 fd969ad0-e9a3-0908-f26b-dd72d7db5cc8}" --- PASS: TestRaft_AppendEntry (0.28s) === RUN TestRaft_VotingGrant_WhenLeaderAvailable 2025-09-14T21:48:51.441Z [INFO] server-40fb9454-93f6-b747-8e2b-6188778412bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-40fb9454-93f6-b747-8e2b-6188778412bc Address:40fb9454-93f6-b747-8e2b-6188778412bc} {Suffrage:Voter ID:server-43a0d25d-9968-e64d-8f4a-2132931683f6 Address:43a0d25d-9968-e64d-8f4a-2132931683f6} {Suffrage:Voter ID:server-4d2c791c-2368-885c-d1c2-47ad93ba741c Address:4d2c791c-2368-885c-d1c2-47ad93ba741c}]" 2025-09-14T21:48:51.441Z [INFO] server-40fb9454-93f6-b747-8e2b-6188778412bc: entering follower state: follower="Node at 40fb9454-93f6-b747-8e2b-6188778412bc [Follower]" leader-address= leader-id= 2025-09-14T21:48:51.441Z [INFO] server-43a0d25d-9968-e64d-8f4a-2132931683f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-40fb9454-93f6-b747-8e2b-6188778412bc Address:40fb9454-93f6-b747-8e2b-6188778412bc} {Suffrage:Voter ID:server-43a0d25d-9968-e64d-8f4a-2132931683f6 Address:43a0d25d-9968-e64d-8f4a-2132931683f6} {Suffrage:Voter ID:server-4d2c791c-2368-885c-d1c2-47ad93ba741c Address:4d2c791c-2368-885c-d1c2-47ad93ba741c}]" 2025-09-14T21:48:51.441Z [INFO] server-4d2c791c-2368-885c-d1c2-47ad93ba741c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-40fb9454-93f6-b747-8e2b-6188778412bc Address:40fb9454-93f6-b747-8e2b-6188778412bc} {Suffrage:Voter ID:server-43a0d25d-9968-e64d-8f4a-2132931683f6 Address:43a0d25d-9968-e64d-8f4a-2132931683f6} {Suffrage:Voter ID:server-4d2c791c-2368-885c-d1c2-47ad93ba741c Address:4d2c791c-2368-885c-d1c2-47ad93ba741c}]" 2025-09-14T21:48:51.441Z [INFO] server-43a0d25d-9968-e64d-8f4a-2132931683f6: entering follower state: follower="Node at 43a0d25d-9968-e64d-8f4a-2132931683f6 [Follower]" leader-address= leader-id= 2025-09-14T21:48:51.441Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:51.441Z [INFO] server-4d2c791c-2368-885c-d1c2-47ad93ba741c: entering follower state: follower="Node at 4d2c791c-2368-885c-d1c2-47ad93ba741c [Follower]" leader-address= leader-id= 2025-09-14T21:48:51.495Z [WARN] server-4d2c791c-2368-885c-d1c2-47ad93ba741c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:51.495Z [INFO] server-4d2c791c-2368-885c-d1c2-47ad93ba741c: entering candidate state: node="Node at 4d2c791c-2368-885c-d1c2-47ad93ba741c [Candidate]" term=2 2025-09-14T21:48:51.495Z [INFO] server-4d2c791c-2368-885c-d1c2-47ad93ba741c: election won: term=2 tally=2 2025-09-14T21:48:51.495Z [INFO] server-4d2c791c-2368-885c-d1c2-47ad93ba741c: entering leader state: leader="Node at 4d2c791c-2368-885c-d1c2-47ad93ba741c [Leader]" 2025-09-14T21:48:51.495Z [INFO] server-4d2c791c-2368-885c-d1c2-47ad93ba741c: added peer, starting replication: peer=server-40fb9454-93f6-b747-8e2b-6188778412bc 2025-09-14T21:48:51.495Z [INFO] server-4d2c791c-2368-885c-d1c2-47ad93ba741c: added peer, starting replication: peer=server-43a0d25d-9968-e64d-8f4a-2132931683f6 2025-09-14T21:48:51.495Z [INFO] server-4d2c791c-2368-885c-d1c2-47ad93ba741c: pipelining replication: peer="{Voter server-43a0d25d-9968-e64d-8f4a-2132931683f6 43a0d25d-9968-e64d-8f4a-2132931683f6}" 2025-09-14T21:48:51.495Z [INFO] server-4d2c791c-2368-885c-d1c2-47ad93ba741c: pipelining replication: peer="{Voter server-40fb9454-93f6-b747-8e2b-6188778412bc 40fb9454-93f6-b747-8e2b-6188778412bc}" 2025-09-14T21:48:51.600Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:51.495431517 +0000 UTC m=+102.063309792 (2 nodes), 54.157775ms from start of poll, 54.450211ms from cluster start. Timeout at 2025-09-14 21:48:51.600437278 +0000 UTC m=+102.168315560, 105.005768ms after stability 2025-09-14T21:48:51.600Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:51.705Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:51.600545143 +0000 UTC m=+102.168423422 (1 nodes), 1.614µs from start of poll, 159.563841ms from cluster start. Timeout at 2025-09-14 21:48:51.705585786 +0000 UTC m=+102.273464049, 105.040627ms after stability 2025-09-14T21:48:51.705Z [WARN] server-43a0d25d-9968-e64d-8f4a-2132931683f6: rejecting vote request since we have a leader: from=40fb9454-93f6-b747-8e2b-6188778412bc leader=4d2c791c-2368-885c-d1c2-47ad93ba741c leader-id=server-4d2c791c-2368-885c-d1c2-47ad93ba741c 2025-09-14T21:48:51.705Z [INFO] server-4d2c791c-2368-885c-d1c2-47ad93ba741c: aborting pipeline replication: peer="{Voter server-43a0d25d-9968-e64d-8f4a-2132931683f6 43a0d25d-9968-e64d-8f4a-2132931683f6}" 2025-09-14T21:48:51.705Z [INFO] server-4d2c791c-2368-885c-d1c2-47ad93ba741c: aborting pipeline replication: peer="{Voter server-40fb9454-93f6-b747-8e2b-6188778412bc 40fb9454-93f6-b747-8e2b-6188778412bc}" --- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.30s) === RUN TestRaft_ProtocolVersion_RejectRPC 2025-09-14T21:48:51.741Z [INFO] server-dac12634-35a6-9a07-3f9a-a5db82c3396b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dac12634-35a6-9a07-3f9a-a5db82c3396b Address:dac12634-35a6-9a07-3f9a-a5db82c3396b} {Suffrage:Voter ID:server-d7bd87e0-700c-50b6-0078-e8197029a3f9 Address:d7bd87e0-700c-50b6-0078-e8197029a3f9} {Suffrage:Voter ID:server-12bf272b-8f4b-3aff-c660-eb759e4e46eb Address:12bf272b-8f4b-3aff-c660-eb759e4e46eb}]" 2025-09-14T21:48:51.741Z [INFO] server-d7bd87e0-700c-50b6-0078-e8197029a3f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dac12634-35a6-9a07-3f9a-a5db82c3396b Address:dac12634-35a6-9a07-3f9a-a5db82c3396b} {Suffrage:Voter ID:server-d7bd87e0-700c-50b6-0078-e8197029a3f9 Address:d7bd87e0-700c-50b6-0078-e8197029a3f9} {Suffrage:Voter ID:server-12bf272b-8f4b-3aff-c660-eb759e4e46eb Address:12bf272b-8f4b-3aff-c660-eb759e4e46eb}]" 2025-09-14T21:48:51.741Z [INFO] server-dac12634-35a6-9a07-3f9a-a5db82c3396b: entering follower state: follower="Node at dac12634-35a6-9a07-3f9a-a5db82c3396b [Follower]" leader-address= leader-id= 2025-09-14T21:48:51.741Z [INFO] server-12bf272b-8f4b-3aff-c660-eb759e4e46eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dac12634-35a6-9a07-3f9a-a5db82c3396b Address:dac12634-35a6-9a07-3f9a-a5db82c3396b} {Suffrage:Voter ID:server-d7bd87e0-700c-50b6-0078-e8197029a3f9 Address:d7bd87e0-700c-50b6-0078-e8197029a3f9} {Suffrage:Voter ID:server-12bf272b-8f4b-3aff-c660-eb759e4e46eb Address:12bf272b-8f4b-3aff-c660-eb759e4e46eb}]" 2025-09-14T21:48:51.741Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:51.741Z [INFO] server-d7bd87e0-700c-50b6-0078-e8197029a3f9: entering follower state: follower="Node at d7bd87e0-700c-50b6-0078-e8197029a3f9 [Follower]" leader-address= leader-id= 2025-09-14T21:48:51.741Z [INFO] server-12bf272b-8f4b-3aff-c660-eb759e4e46eb: entering follower state: follower="Node at 12bf272b-8f4b-3aff-c660-eb759e4e46eb [Follower]" leader-address= leader-id= 2025-09-14T21:48:51.809Z [WARN] server-12bf272b-8f4b-3aff-c660-eb759e4e46eb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:51.809Z [INFO] server-12bf272b-8f4b-3aff-c660-eb759e4e46eb: entering candidate state: node="Node at 12bf272b-8f4b-3aff-c660-eb759e4e46eb [Candidate]" term=2 2025-09-14T21:48:51.809Z [INFO] server-12bf272b-8f4b-3aff-c660-eb759e4e46eb: election won: term=2 tally=2 2025-09-14T21:48:51.809Z [INFO] server-12bf272b-8f4b-3aff-c660-eb759e4e46eb: entering leader state: leader="Node at 12bf272b-8f4b-3aff-c660-eb759e4e46eb [Leader]" 2025-09-14T21:48:51.809Z [INFO] server-12bf272b-8f4b-3aff-c660-eb759e4e46eb: added peer, starting replication: peer=server-dac12634-35a6-9a07-3f9a-a5db82c3396b 2025-09-14T21:48:51.809Z [INFO] server-12bf272b-8f4b-3aff-c660-eb759e4e46eb: added peer, starting replication: peer=server-d7bd87e0-700c-50b6-0078-e8197029a3f9 2025-09-14T21:48:51.809Z [INFO] server-12bf272b-8f4b-3aff-c660-eb759e4e46eb: pipelining replication: peer="{Voter server-d7bd87e0-700c-50b6-0078-e8197029a3f9 d7bd87e0-700c-50b6-0078-e8197029a3f9}" 2025-09-14T21:48:51.809Z [INFO] server-12bf272b-8f4b-3aff-c660-eb759e4e46eb: pipelining replication: peer="{Voter server-dac12634-35a6-9a07-3f9a-a5db82c3396b dac12634-35a6-9a07-3f9a-a5db82c3396b}" 2025-09-14T21:48:51.914Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:51.809250392 +0000 UTC m=+102.377128666 (2 nodes), 67.526539ms from start of poll, 67.866701ms from cluster start. Timeout at 2025-09-14 21:48:51.914255613 +0000 UTC m=+102.482133875, 105.005209ms after stability 2025-09-14T21:48:51.914Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:52.020Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:51.914796374 +0000 UTC m=+102.482674649 (1 nodes), 1.972µs from start of poll, 173.412684ms from cluster start. Timeout at 2025-09-14 21:48:52.019822887 +0000 UTC m=+102.587701149, 105.0265ms after stability 2025-09-14T21:48:52.020Z [INFO] server-12bf272b-8f4b-3aff-c660-eb759e4e46eb: aborting pipeline replication: peer="{Voter server-d7bd87e0-700c-50b6-0078-e8197029a3f9 d7bd87e0-700c-50b6-0078-e8197029a3f9}" 2025-09-14T21:48:52.020Z [INFO] server-12bf272b-8f4b-3aff-c660-eb759e4e46eb: aborting pipeline replication: peer="{Voter server-dac12634-35a6-9a07-3f9a-a5db82c3396b dac12634-35a6-9a07-3f9a-a5db82c3396b}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.31s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 raft_test.go:2151: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 raft_test.go:2196: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s) === RUN TestRaft_LeaderID_Propagated 2025-09-14T21:48:52.050Z [INFO] server-0ea574c8-3e5c-c633-c44d-f8517967c0d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ea574c8-3e5c-c633-c44d-f8517967c0d0 Address:0ea574c8-3e5c-c633-c44d-f8517967c0d0} {Suffrage:Voter ID:server-e47445e5-3385-7693-090e-2c47354e4136 Address:e47445e5-3385-7693-090e-2c47354e4136} {Suffrage:Voter ID:server-885727f1-4c0e-ac4d-4871-9ae331ba3614 Address:885727f1-4c0e-ac4d-4871-9ae331ba3614}]" 2025-09-14T21:48:52.050Z [INFO] server-e47445e5-3385-7693-090e-2c47354e4136: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ea574c8-3e5c-c633-c44d-f8517967c0d0 Address:0ea574c8-3e5c-c633-c44d-f8517967c0d0} {Suffrage:Voter ID:server-e47445e5-3385-7693-090e-2c47354e4136 Address:e47445e5-3385-7693-090e-2c47354e4136} {Suffrage:Voter ID:server-885727f1-4c0e-ac4d-4871-9ae331ba3614 Address:885727f1-4c0e-ac4d-4871-9ae331ba3614}]" 2025-09-14T21:48:52.050Z [INFO] server-885727f1-4c0e-ac4d-4871-9ae331ba3614: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ea574c8-3e5c-c633-c44d-f8517967c0d0 Address:0ea574c8-3e5c-c633-c44d-f8517967c0d0} {Suffrage:Voter ID:server-e47445e5-3385-7693-090e-2c47354e4136 Address:e47445e5-3385-7693-090e-2c47354e4136} {Suffrage:Voter ID:server-885727f1-4c0e-ac4d-4871-9ae331ba3614 Address:885727f1-4c0e-ac4d-4871-9ae331ba3614}]" 2025-09-14T21:48:52.050Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:52.050Z [INFO] server-0ea574c8-3e5c-c633-c44d-f8517967c0d0: entering follower state: follower="Node at 0ea574c8-3e5c-c633-c44d-f8517967c0d0 [Follower]" leader-address= leader-id= 2025-09-14T21:48:52.050Z [INFO] server-e47445e5-3385-7693-090e-2c47354e4136: entering follower state: follower="Node at e47445e5-3385-7693-090e-2c47354e4136 [Follower]" leader-address= leader-id= 2025-09-14T21:48:52.050Z [INFO] server-885727f1-4c0e-ac4d-4871-9ae331ba3614: entering follower state: follower="Node at 885727f1-4c0e-ac4d-4871-9ae331ba3614 [Follower]" leader-address= leader-id= 2025-09-14T21:48:52.122Z [WARN] server-e47445e5-3385-7693-090e-2c47354e4136: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:52.122Z [INFO] server-e47445e5-3385-7693-090e-2c47354e4136: entering candidate state: node="Node at e47445e5-3385-7693-090e-2c47354e4136 [Candidate]" term=2 2025-09-14T21:48:52.122Z [INFO] server-e47445e5-3385-7693-090e-2c47354e4136: election won: term=2 tally=2 2025-09-14T21:48:52.122Z [INFO] server-e47445e5-3385-7693-090e-2c47354e4136: entering leader state: leader="Node at e47445e5-3385-7693-090e-2c47354e4136 [Leader]" 2025-09-14T21:48:52.122Z [INFO] server-e47445e5-3385-7693-090e-2c47354e4136: added peer, starting replication: peer=server-0ea574c8-3e5c-c633-c44d-f8517967c0d0 2025-09-14T21:48:52.122Z [INFO] server-e47445e5-3385-7693-090e-2c47354e4136: added peer, starting replication: peer=server-885727f1-4c0e-ac4d-4871-9ae331ba3614 2025-09-14T21:48:52.122Z [INFO] server-e47445e5-3385-7693-090e-2c47354e4136: pipelining replication: peer="{Voter server-0ea574c8-3e5c-c633-c44d-f8517967c0d0 0ea574c8-3e5c-c633-c44d-f8517967c0d0}" 2025-09-14T21:48:52.122Z [INFO] server-e47445e5-3385-7693-090e-2c47354e4136: pipelining replication: peer="{Voter server-885727f1-4c0e-ac4d-4871-9ae331ba3614 885727f1-4c0e-ac4d-4871-9ae331ba3614}" 2025-09-14T21:48:52.228Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:52.122439944 +0000 UTC m=+102.690318206 (1 nodes), 71.793499ms from start of poll, 72.065077ms from cluster start. Timeout at 2025-09-14 21:48:52.22744805 +0000 UTC m=+102.795326313, 105.008107ms after stability 2025-09-14T21:48:52.228Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:52.333Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:52.228472436 +0000 UTC m=+102.796350708 (1 nodes), 482ns from start of poll, 178.097579ms from cluster start. Timeout at 2025-09-14 21:48:52.333487436 +0000 UTC m=+102.901365733, 105.015025ms after stability 2025-09-14T21:48:52.333Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:52.439Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:52.333690415 +0000 UTC m=+102.901568680 (1 nodes), 913ns from start of poll, 283.315551ms from cluster start. Timeout at 2025-09-14 21:48:52.438711983 +0000 UTC m=+103.006590247, 105.021567ms after stability 2025-09-14T21:48:52.439Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:52.544Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:52.439150902 +0000 UTC m=+103.007029178 (1 nodes), 930ns from start of poll, 388.776049ms from cluster start. Timeout at 2025-09-14 21:48:52.544172261 +0000 UTC m=+103.112050543, 105.021365ms after stability 2025-09-14T21:48:52.544Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:52.650Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:52.544414422 +0000 UTC m=+103.112292692 (1 nodes), 1.208µs from start of poll, 494.039563ms from cluster start. Timeout at 2025-09-14 21:48:52.649436588 +0000 UTC m=+103.217314869, 105.022177ms after stability 2025-09-14T21:48:52.650Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:52.756Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:52.650552057 +0000 UTC m=+103.218430330 (1 nodes), 992ns from start of poll, 600.177201ms from cluster start. Timeout at 2025-09-14 21:48:52.755574937 +0000 UTC m=+103.323453218, 105.022888ms after stability 2025-09-14T21:48:52.861Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:52.967Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:52.86196737 +0000 UTC m=+103.429845635 (1 nodes), 1.951µs from start of poll, 811.592506ms from cluster start. Timeout at 2025-09-14 21:48:52.966995538 +0000 UTC m=+103.534873802, 105.028167ms after stability 2025-09-14T21:48:52.967Z [INFO] server-e47445e5-3385-7693-090e-2c47354e4136: aborting pipeline replication: peer="{Voter server-885727f1-4c0e-ac4d-4871-9ae331ba3614 885727f1-4c0e-ac4d-4871-9ae331ba3614}" 2025-09-14T21:48:52.967Z [INFO] server-e47445e5-3385-7693-090e-2c47354e4136: aborting pipeline replication: peer="{Voter server-0ea574c8-3e5c-c633-c44d-f8517967c0d0 0ea574c8-3e5c-c633-c44d-f8517967c0d0}" --- PASS: TestRaft_LeaderID_Propagated (0.92s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer 2025-09-14T21:48:52.969Z [INFO] server-8057495f-d735-3583-8b29-ca8bff1d3983: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8057495f-d735-3583-8b29-ca8bff1d3983 Address:8057495f-d735-3583-8b29-ca8bff1d3983} {Suffrage:Voter ID:server-2ef50e01-74b5-da26-3657-7c234107ce74 Address:2ef50e01-74b5-da26-3657-7c234107ce74} {Suffrage:Voter ID:server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0 Address:13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0}]" 2025-09-14T21:48:52.969Z [INFO] server-8057495f-d735-3583-8b29-ca8bff1d3983: entering follower state: follower="Node at 8057495f-d735-3583-8b29-ca8bff1d3983 [Follower]" leader-address= leader-id= 2025-09-14T21:48:52.969Z [INFO] server-2ef50e01-74b5-da26-3657-7c234107ce74: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8057495f-d735-3583-8b29-ca8bff1d3983 Address:8057495f-d735-3583-8b29-ca8bff1d3983} {Suffrage:Voter ID:server-2ef50e01-74b5-da26-3657-7c234107ce74 Address:2ef50e01-74b5-da26-3657-7c234107ce74} {Suffrage:Voter ID:server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0 Address:13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0}]" 2025-09-14T21:48:52.969Z [INFO] server-2ef50e01-74b5-da26-3657-7c234107ce74: entering follower state: follower="Node at 2ef50e01-74b5-da26-3657-7c234107ce74 [Follower]" leader-address= leader-id= 2025-09-14T21:48:52.969Z [INFO] server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8057495f-d735-3583-8b29-ca8bff1d3983 Address:8057495f-d735-3583-8b29-ca8bff1d3983} {Suffrage:Voter ID:server-2ef50e01-74b5-da26-3657-7c234107ce74 Address:2ef50e01-74b5-da26-3657-7c234107ce74} {Suffrage:Voter ID:server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0 Address:13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0}]" 2025-09-14T21:48:52.969Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:52.969Z [INFO] server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0: entering follower state: follower="Node at 13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0 [Follower]" leader-address= leader-id= 2025-09-14T21:48:53.028Z [WARN] server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:53.028Z [INFO] server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0: entering candidate state: node="Node at 13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0 [Candidate]" term=2 2025-09-14T21:48:53.029Z [INFO] server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0: election won: term=2 tally=2 2025-09-14T21:48:53.029Z [INFO] server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0: entering leader state: leader="Node at 13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0 [Leader]" 2025-09-14T21:48:53.029Z [INFO] server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0: added peer, starting replication: peer=server-8057495f-d735-3583-8b29-ca8bff1d3983 2025-09-14T21:48:53.029Z [INFO] server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0: added peer, starting replication: peer=server-2ef50e01-74b5-da26-3657-7c234107ce74 2025-09-14T21:48:53.029Z [INFO] server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0: pipelining replication: peer="{Voter server-8057495f-d735-3583-8b29-ca8bff1d3983 8057495f-d735-3583-8b29-ca8bff1d3983}" 2025-09-14T21:48:53.029Z [INFO] server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0: pipelining replication: peer="{Voter server-2ef50e01-74b5-da26-3657-7c234107ce74 2ef50e01-74b5-da26-3657-7c234107ce74}" 2025-09-14T21:48:53.134Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:53.029142855 +0000 UTC m=+103.597021113 (1 nodes), 59.384483ms from start of poll, 59.929717ms from cluster start. Timeout at 2025-09-14 21:48:53.134144952 +0000 UTC m=+103.702023216, 105.002103ms after stability 2025-09-14T21:48:53.134Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:53.239Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:53.13428481 +0000 UTC m=+103.702163074 (1 nodes), 1.712µs from start of poll, 165.071678ms from cluster start. Timeout at 2025-09-14 21:48:53.239372173 +0000 UTC m=+103.807250455, 105.087381ms after stability 2025-09-14T21:48:53.239Z [INFO] server-8057495f-d735-3583-8b29-ca8bff1d3983: entering candidate state: node="Node at 8057495f-d735-3583-8b29-ca8bff1d3983 [Candidate]" term=3 2025-09-14T21:48:53.239Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:53.239Z [INFO] server-8057495f-d735-3583-8b29-ca8bff1d3983: election won: term=3 tally=2 2025-09-14T21:48:53.239Z [INFO] server-8057495f-d735-3583-8b29-ca8bff1d3983: entering leader state: leader="Node at 8057495f-d735-3583-8b29-ca8bff1d3983 [Leader]" 2025-09-14T21:48:53.239Z [INFO] server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0: entering follower state: follower="Node at 13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0 [Follower]" leader-address= leader-id= 2025-09-14T21:48:53.239Z [INFO] server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0: aborting pipeline replication: peer="{Voter server-2ef50e01-74b5-da26-3657-7c234107ce74 2ef50e01-74b5-da26-3657-7c234107ce74}" 2025-09-14T21:48:53.239Z [INFO] server-8057495f-d735-3583-8b29-ca8bff1d3983: added peer, starting replication: peer=server-2ef50e01-74b5-da26-3657-7c234107ce74 2025-09-14T21:48:53.239Z [INFO] server-8057495f-d735-3583-8b29-ca8bff1d3983: added peer, starting replication: peer=server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0 2025-09-14T21:48:53.239Z [INFO] server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0: aborting pipeline replication: peer="{Voter server-8057495f-d735-3583-8b29-ca8bff1d3983 8057495f-d735-3583-8b29-ca8bff1d3983}" 2025-09-14T21:48:53.239Z [INFO] server-8057495f-d735-3583-8b29-ca8bff1d3983: pipelining replication: peer="{Voter server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0 13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0}" 2025-09-14T21:48:53.239Z [INFO] server-8057495f-d735-3583-8b29-ca8bff1d3983: pipelining replication: peer="{Voter server-2ef50e01-74b5-da26-3657-7c234107ce74 2ef50e01-74b5-da26-3657-7c234107ce74}" 2025-09-14T21:48:53.344Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:53.239646928 +0000 UTC m=+103.807525209 (1 nodes), 79.619µs from start of poll, 270.433813ms from cluster start. Timeout at 2025-09-14 21:48:53.344667848 +0000 UTC m=+103.912546126, 105.020917ms after stability 2025-09-14T21:48:53.344Z [INFO] server-8057495f-d735-3583-8b29-ca8bff1d3983: aborting pipeline replication: peer="{Voter server-13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0 13fbb9b0-59b7-b645-5cf1-3a1163f5c2a0}" 2025-09-14T21:48:53.344Z [INFO] server-8057495f-d735-3583-8b29-ca8bff1d3983: aborting pipeline replication: peer="{Voter server-2ef50e01-74b5-da26-3657-7c234107ce74 2ef50e01-74b5-da26-3657-7c234107ce74}" --- PASS: TestRaft_LeadershipTransfer (0.38s) === RUN TestRaft_LeadershipTransferWithOneNode 2025-09-14T21:48:53.345Z [INFO] server-e0b7df5c-8ac2-b2c9-717f-a05cd80b5e56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0b7df5c-8ac2-b2c9-717f-a05cd80b5e56 Address:e0b7df5c-8ac2-b2c9-717f-a05cd80b5e56}]" 2025-09-14T21:48:53.345Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:53.345Z [INFO] server-e0b7df5c-8ac2-b2c9-717f-a05cd80b5e56: entering follower state: follower="Node at e0b7df5c-8ac2-b2c9-717f-a05cd80b5e56 [Follower]" leader-address= leader-id= 2025-09-14T21:48:53.445Z [WARN] server-e0b7df5c-8ac2-b2c9-717f-a05cd80b5e56: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:53.445Z [INFO] server-e0b7df5c-8ac2-b2c9-717f-a05cd80b5e56: entering candidate state: node="Node at e0b7df5c-8ac2-b2c9-717f-a05cd80b5e56 [Candidate]" term=2 2025-09-14T21:48:53.445Z [INFO] server-e0b7df5c-8ac2-b2c9-717f-a05cd80b5e56: election won: term=2 tally=1 2025-09-14T21:48:53.445Z [INFO] server-e0b7df5c-8ac2-b2c9-717f-a05cd80b5e56: entering leader state: leader="Node at e0b7df5c-8ac2-b2c9-717f-a05cd80b5e56 [Leader]" 2025-09-14T21:48:53.551Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:53.445930946 +0000 UTC m=+104.013809226 (1 nodes), 100.082635ms from start of poll, 100.217966ms from cluster start. Timeout at 2025-09-14 21:48:53.550934185 +0000 UTC m=+104.118812463, 105.003237ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.21s) === RUN TestRaft_LeadershipTransferWithSevenNodes 2025-09-14T21:48:53.553Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5 Address:fca5bb9b-9360-cdcd-9a25-e4ad2598cae5} {Suffrage:Voter ID:server-3633c8c6-bfd5-bfe4-9e01-8482b0ba2064 Address:3633c8c6-bfd5-bfe4-9e01-8482b0ba2064} {Suffrage:Voter ID:server-6c8e92f6-2698-79b3-f569-8f3793b282c4 Address:6c8e92f6-2698-79b3-f569-8f3793b282c4} {Suffrage:Voter ID:server-fa210c56-f3e2-cef9-a379-00ec4946daa8 Address:fa210c56-f3e2-cef9-a379-00ec4946daa8} {Suffrage:Voter ID:server-4667b539-742b-a9a3-d05e-86f9c726babb Address:4667b539-742b-a9a3-d05e-86f9c726babb} {Suffrage:Voter ID:server-b74d0027-c6a6-1490-063d-9d67aebf3e06 Address:b74d0027-c6a6-1490-063d-9d67aebf3e06} {Suffrage:Voter ID:server-d2d7c728-4147-a781-901e-4b929844d0ed Address:d2d7c728-4147-a781-901e-4b929844d0ed}]" 2025-09-14T21:48:53.553Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: entering follower state: follower="Node at fca5bb9b-9360-cdcd-9a25-e4ad2598cae5 [Follower]" leader-address= leader-id= 2025-09-14T21:48:53.553Z [INFO] server-3633c8c6-bfd5-bfe4-9e01-8482b0ba2064: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5 Address:fca5bb9b-9360-cdcd-9a25-e4ad2598cae5} {Suffrage:Voter ID:server-3633c8c6-bfd5-bfe4-9e01-8482b0ba2064 Address:3633c8c6-bfd5-bfe4-9e01-8482b0ba2064} {Suffrage:Voter ID:server-6c8e92f6-2698-79b3-f569-8f3793b282c4 Address:6c8e92f6-2698-79b3-f569-8f3793b282c4} {Suffrage:Voter ID:server-fa210c56-f3e2-cef9-a379-00ec4946daa8 Address:fa210c56-f3e2-cef9-a379-00ec4946daa8} {Suffrage:Voter ID:server-4667b539-742b-a9a3-d05e-86f9c726babb Address:4667b539-742b-a9a3-d05e-86f9c726babb} {Suffrage:Voter ID:server-b74d0027-c6a6-1490-063d-9d67aebf3e06 Address:b74d0027-c6a6-1490-063d-9d67aebf3e06} {Suffrage:Voter ID:server-d2d7c728-4147-a781-901e-4b929844d0ed Address:d2d7c728-4147-a781-901e-4b929844d0ed}]" 2025-09-14T21:48:53.553Z [INFO] server-3633c8c6-bfd5-bfe4-9e01-8482b0ba2064: entering follower state: follower="Node at 3633c8c6-bfd5-bfe4-9e01-8482b0ba2064 [Follower]" leader-address= leader-id= 2025-09-14T21:48:53.553Z [INFO] server-6c8e92f6-2698-79b3-f569-8f3793b282c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5 Address:fca5bb9b-9360-cdcd-9a25-e4ad2598cae5} {Suffrage:Voter ID:server-3633c8c6-bfd5-bfe4-9e01-8482b0ba2064 Address:3633c8c6-bfd5-bfe4-9e01-8482b0ba2064} {Suffrage:Voter ID:server-6c8e92f6-2698-79b3-f569-8f3793b282c4 Address:6c8e92f6-2698-79b3-f569-8f3793b282c4} {Suffrage:Voter ID:server-fa210c56-f3e2-cef9-a379-00ec4946daa8 Address:fa210c56-f3e2-cef9-a379-00ec4946daa8} {Suffrage:Voter ID:server-4667b539-742b-a9a3-d05e-86f9c726babb Address:4667b539-742b-a9a3-d05e-86f9c726babb} {Suffrage:Voter ID:server-b74d0027-c6a6-1490-063d-9d67aebf3e06 Address:b74d0027-c6a6-1490-063d-9d67aebf3e06} {Suffrage:Voter ID:server-d2d7c728-4147-a781-901e-4b929844d0ed Address:d2d7c728-4147-a781-901e-4b929844d0ed}]" 2025-09-14T21:48:53.553Z [INFO] server-6c8e92f6-2698-79b3-f569-8f3793b282c4: entering follower state: follower="Node at 6c8e92f6-2698-79b3-f569-8f3793b282c4 [Follower]" leader-address= leader-id= 2025-09-14T21:48:53.553Z [INFO] server-fa210c56-f3e2-cef9-a379-00ec4946daa8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5 Address:fca5bb9b-9360-cdcd-9a25-e4ad2598cae5} {Suffrage:Voter ID:server-3633c8c6-bfd5-bfe4-9e01-8482b0ba2064 Address:3633c8c6-bfd5-bfe4-9e01-8482b0ba2064} {Suffrage:Voter ID:server-6c8e92f6-2698-79b3-f569-8f3793b282c4 Address:6c8e92f6-2698-79b3-f569-8f3793b282c4} {Suffrage:Voter ID:server-fa210c56-f3e2-cef9-a379-00ec4946daa8 Address:fa210c56-f3e2-cef9-a379-00ec4946daa8} {Suffrage:Voter ID:server-4667b539-742b-a9a3-d05e-86f9c726babb Address:4667b539-742b-a9a3-d05e-86f9c726babb} {Suffrage:Voter ID:server-b74d0027-c6a6-1490-063d-9d67aebf3e06 Address:b74d0027-c6a6-1490-063d-9d67aebf3e06} {Suffrage:Voter ID:server-d2d7c728-4147-a781-901e-4b929844d0ed Address:d2d7c728-4147-a781-901e-4b929844d0ed}]" 2025-09-14T21:48:53.554Z [INFO] server-fa210c56-f3e2-cef9-a379-00ec4946daa8: entering follower state: follower="Node at fa210c56-f3e2-cef9-a379-00ec4946daa8 [Follower]" leader-address= leader-id= 2025-09-14T21:48:53.554Z [INFO] server-4667b539-742b-a9a3-d05e-86f9c726babb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5 Address:fca5bb9b-9360-cdcd-9a25-e4ad2598cae5} {Suffrage:Voter ID:server-3633c8c6-bfd5-bfe4-9e01-8482b0ba2064 Address:3633c8c6-bfd5-bfe4-9e01-8482b0ba2064} {Suffrage:Voter ID:server-6c8e92f6-2698-79b3-f569-8f3793b282c4 Address:6c8e92f6-2698-79b3-f569-8f3793b282c4} {Suffrage:Voter ID:server-fa210c56-f3e2-cef9-a379-00ec4946daa8 Address:fa210c56-f3e2-cef9-a379-00ec4946daa8} {Suffrage:Voter ID:server-4667b539-742b-a9a3-d05e-86f9c726babb Address:4667b539-742b-a9a3-d05e-86f9c726babb} {Suffrage:Voter ID:server-b74d0027-c6a6-1490-063d-9d67aebf3e06 Address:b74d0027-c6a6-1490-063d-9d67aebf3e06} {Suffrage:Voter ID:server-d2d7c728-4147-a781-901e-4b929844d0ed Address:d2d7c728-4147-a781-901e-4b929844d0ed}]" 2025-09-14T21:48:53.554Z [INFO] server-4667b539-742b-a9a3-d05e-86f9c726babb: entering follower state: follower="Node at 4667b539-742b-a9a3-d05e-86f9c726babb [Follower]" leader-address= leader-id= 2025-09-14T21:48:53.554Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5 Address:fca5bb9b-9360-cdcd-9a25-e4ad2598cae5} {Suffrage:Voter ID:server-3633c8c6-bfd5-bfe4-9e01-8482b0ba2064 Address:3633c8c6-bfd5-bfe4-9e01-8482b0ba2064} {Suffrage:Voter ID:server-6c8e92f6-2698-79b3-f569-8f3793b282c4 Address:6c8e92f6-2698-79b3-f569-8f3793b282c4} {Suffrage:Voter ID:server-fa210c56-f3e2-cef9-a379-00ec4946daa8 Address:fa210c56-f3e2-cef9-a379-00ec4946daa8} {Suffrage:Voter ID:server-4667b539-742b-a9a3-d05e-86f9c726babb Address:4667b539-742b-a9a3-d05e-86f9c726babb} {Suffrage:Voter ID:server-b74d0027-c6a6-1490-063d-9d67aebf3e06 Address:b74d0027-c6a6-1490-063d-9d67aebf3e06} {Suffrage:Voter ID:server-d2d7c728-4147-a781-901e-4b929844d0ed Address:d2d7c728-4147-a781-901e-4b929844d0ed}]" 2025-09-14T21:48:53.554Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: entering follower state: follower="Node at b74d0027-c6a6-1490-063d-9d67aebf3e06 [Follower]" leader-address= leader-id= 2025-09-14T21:48:53.554Z [INFO] server-d2d7c728-4147-a781-901e-4b929844d0ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5 Address:fca5bb9b-9360-cdcd-9a25-e4ad2598cae5} {Suffrage:Voter ID:server-3633c8c6-bfd5-bfe4-9e01-8482b0ba2064 Address:3633c8c6-bfd5-bfe4-9e01-8482b0ba2064} {Suffrage:Voter ID:server-6c8e92f6-2698-79b3-f569-8f3793b282c4 Address:6c8e92f6-2698-79b3-f569-8f3793b282c4} {Suffrage:Voter ID:server-fa210c56-f3e2-cef9-a379-00ec4946daa8 Address:fa210c56-f3e2-cef9-a379-00ec4946daa8} {Suffrage:Voter ID:server-4667b539-742b-a9a3-d05e-86f9c726babb Address:4667b539-742b-a9a3-d05e-86f9c726babb} {Suffrage:Voter ID:server-b74d0027-c6a6-1490-063d-9d67aebf3e06 Address:b74d0027-c6a6-1490-063d-9d67aebf3e06} {Suffrage:Voter ID:server-d2d7c728-4147-a781-901e-4b929844d0ed Address:d2d7c728-4147-a781-901e-4b929844d0ed}]" 2025-09-14T21:48:53.554Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:53.554Z [INFO] server-d2d7c728-4147-a781-901e-4b929844d0ed: entering follower state: follower="Node at d2d7c728-4147-a781-901e-4b929844d0ed [Follower]" leader-address= leader-id= 2025-09-14T21:48:53.614Z [WARN] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:53.614Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: entering candidate state: node="Node at b74d0027-c6a6-1490-063d-9d67aebf3e06 [Candidate]" term=2 2025-09-14T21:48:53.614Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: election won: term=2 tally=4 2025-09-14T21:48:53.614Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: entering leader state: leader="Node at b74d0027-c6a6-1490-063d-9d67aebf3e06 [Leader]" 2025-09-14T21:48:53.614Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: added peer, starting replication: peer=server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5 2025-09-14T21:48:53.614Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: added peer, starting replication: peer=server-3633c8c6-bfd5-bfe4-9e01-8482b0ba2064 2025-09-14T21:48:53.614Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: added peer, starting replication: peer=server-6c8e92f6-2698-79b3-f569-8f3793b282c4 2025-09-14T21:48:53.614Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: added peer, starting replication: peer=server-fa210c56-f3e2-cef9-a379-00ec4946daa8 2025-09-14T21:48:53.614Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: added peer, starting replication: peer=server-4667b539-742b-a9a3-d05e-86f9c726babb 2025-09-14T21:48:53.614Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: added peer, starting replication: peer=server-d2d7c728-4147-a781-901e-4b929844d0ed 2025-09-14T21:48:53.614Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: pipelining replication: peer="{Voter server-6c8e92f6-2698-79b3-f569-8f3793b282c4 6c8e92f6-2698-79b3-f569-8f3793b282c4}" 2025-09-14T21:48:53.614Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: pipelining replication: peer="{Voter server-3633c8c6-bfd5-bfe4-9e01-8482b0ba2064 3633c8c6-bfd5-bfe4-9e01-8482b0ba2064}" 2025-09-14T21:48:53.614Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: pipelining replication: peer="{Voter server-d2d7c728-4147-a781-901e-4b929844d0ed d2d7c728-4147-a781-901e-4b929844d0ed}" 2025-09-14T21:48:53.614Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: pipelining replication: peer="{Voter server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5 fca5bb9b-9360-cdcd-9a25-e4ad2598cae5}" 2025-09-14T21:48:53.614Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: pipelining replication: peer="{Voter server-fa210c56-f3e2-cef9-a379-00ec4946daa8 fa210c56-f3e2-cef9-a379-00ec4946daa8}" 2025-09-14T21:48:53.614Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: pipelining replication: peer="{Voter server-4667b539-742b-a9a3-d05e-86f9c726babb 4667b539-742b-a9a3-d05e-86f9c726babb}" 2025-09-14T21:48:53.720Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:53.614747398 +0000 UTC m=+104.182625661 (1 nodes), 60.337905ms from start of poll, 61.500217ms from cluster start. Timeout at 2025-09-14 21:48:53.719756454 +0000 UTC m=+104.287634725, 105.009064ms after stability 2025-09-14T21:48:53.720Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:53.825Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:53.720718287 +0000 UTC m=+104.288596569 (6 nodes), 2.624µs from start of poll, 167.471125ms from cluster start. Timeout at 2025-09-14 21:48:53.82575096 +0000 UTC m=+104.393629229, 105.03266ms after stability 2025-09-14T21:48:53.825Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:53.931Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:53.825841995 +0000 UTC m=+104.393720259 (1 nodes), 2.223µs from start of poll, 272.594815ms from cluster start. Timeout at 2025-09-14 21:48:53.930874879 +0000 UTC m=+104.498753142, 105.032883ms after stability 2025-09-14T21:48:53.931Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: entering candidate state: node="Node at fca5bb9b-9360-cdcd-9a25-e4ad2598cae5 [Candidate]" term=3 2025-09-14T21:48:53.931Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:53.931Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: election won: term=3 tally=4 2025-09-14T21:48:53.931Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: entering leader state: leader="Node at fca5bb9b-9360-cdcd-9a25-e4ad2598cae5 [Leader]" 2025-09-14T21:48:53.931Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: added peer, starting replication: peer=server-3633c8c6-bfd5-bfe4-9e01-8482b0ba2064 2025-09-14T21:48:53.931Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: entering follower state: follower="Node at b74d0027-c6a6-1490-063d-9d67aebf3e06 [Follower]" leader-address= leader-id= 2025-09-14T21:48:53.931Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: aborting pipeline replication: peer="{Voter server-3633c8c6-bfd5-bfe4-9e01-8482b0ba2064 3633c8c6-bfd5-bfe4-9e01-8482b0ba2064}" 2025-09-14T21:48:53.931Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: added peer, starting replication: peer=server-6c8e92f6-2698-79b3-f569-8f3793b282c4 2025-09-14T21:48:53.931Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: added peer, starting replication: peer=server-fa210c56-f3e2-cef9-a379-00ec4946daa8 2025-09-14T21:48:53.931Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: added peer, starting replication: peer=server-4667b539-742b-a9a3-d05e-86f9c726babb 2025-09-14T21:48:53.931Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: added peer, starting replication: peer=server-b74d0027-c6a6-1490-063d-9d67aebf3e06 2025-09-14T21:48:53.931Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: added peer, starting replication: peer=server-d2d7c728-4147-a781-901e-4b929844d0ed 2025-09-14T21:48:53.931Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: pipelining replication: peer="{Voter server-3633c8c6-bfd5-bfe4-9e01-8482b0ba2064 3633c8c6-bfd5-bfe4-9e01-8482b0ba2064}" 2025-09-14T21:48:53.931Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: pipelining replication: peer="{Voter server-d2d7c728-4147-a781-901e-4b929844d0ed d2d7c728-4147-a781-901e-4b929844d0ed}" 2025-09-14T21:48:53.931Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: aborting pipeline replication: peer="{Voter server-4667b539-742b-a9a3-d05e-86f9c726babb 4667b539-742b-a9a3-d05e-86f9c726babb}" 2025-09-14T21:48:53.932Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: pipelining replication: peer="{Voter server-fa210c56-f3e2-cef9-a379-00ec4946daa8 fa210c56-f3e2-cef9-a379-00ec4946daa8}" 2025-09-14T21:48:53.931Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: aborting pipeline replication: peer="{Voter server-d2d7c728-4147-a781-901e-4b929844d0ed d2d7c728-4147-a781-901e-4b929844d0ed}" 2025-09-14T21:48:53.932Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: pipelining replication: peer="{Voter server-4667b539-742b-a9a3-d05e-86f9c726babb 4667b539-742b-a9a3-d05e-86f9c726babb}" 2025-09-14T21:48:53.931Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: aborting pipeline replication: peer="{Voter server-fa210c56-f3e2-cef9-a379-00ec4946daa8 fa210c56-f3e2-cef9-a379-00ec4946daa8}" 2025-09-14T21:48:53.931Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: aborting pipeline replication: peer="{Voter server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5 fca5bb9b-9360-cdcd-9a25-e4ad2598cae5}" 2025-09-14T21:48:53.932Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: pipelining replication: peer="{Voter server-6c8e92f6-2698-79b3-f569-8f3793b282c4 6c8e92f6-2698-79b3-f569-8f3793b282c4}" 2025-09-14T21:48:53.931Z [INFO] server-b74d0027-c6a6-1490-063d-9d67aebf3e06: aborting pipeline replication: peer="{Voter server-6c8e92f6-2698-79b3-f569-8f3793b282c4 6c8e92f6-2698-79b3-f569-8f3793b282c4}" 2025-09-14T21:48:53.932Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: pipelining replication: peer="{Voter server-b74d0027-c6a6-1490-063d-9d67aebf3e06 b74d0027-c6a6-1490-063d-9d67aebf3e06}" 2025-09-14T21:48:54.037Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:53.931843027 +0000 UTC m=+104.499721303 (1 nodes), 123.304µs from start of poll, 378.595859ms from cluster start. Timeout at 2025-09-14 21:48:54.036875392 +0000 UTC m=+104.604753655, 105.032352ms after stability 2025-09-14T21:48:54.037Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: aborting pipeline replication: peer="{Voter server-4667b539-742b-a9a3-d05e-86f9c726babb 4667b539-742b-a9a3-d05e-86f9c726babb}" 2025-09-14T21:48:54.037Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: aborting pipeline replication: peer="{Voter server-b74d0027-c6a6-1490-063d-9d67aebf3e06 b74d0027-c6a6-1490-063d-9d67aebf3e06}" 2025-09-14T21:48:54.037Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: aborting pipeline replication: peer="{Voter server-3633c8c6-bfd5-bfe4-9e01-8482b0ba2064 3633c8c6-bfd5-bfe4-9e01-8482b0ba2064}" 2025-09-14T21:48:54.037Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: aborting pipeline replication: peer="{Voter server-fa210c56-f3e2-cef9-a379-00ec4946daa8 fa210c56-f3e2-cef9-a379-00ec4946daa8}" 2025-09-14T21:48:54.037Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: aborting pipeline replication: peer="{Voter server-d2d7c728-4147-a781-901e-4b929844d0ed d2d7c728-4147-a781-901e-4b929844d0ed}" 2025-09-14T21:48:54.037Z [INFO] server-fca5bb9b-9360-cdcd-9a25-e4ad2598cae5: aborting pipeline replication: peer="{Voter server-6c8e92f6-2698-79b3-f569-8f3793b282c4 6c8e92f6-2698-79b3-f569-8f3793b282c4}" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.49s) === RUN TestRaft_LeadershipTransferToInvalidID 2025-09-14T21:48:54.039Z [INFO] server-6e2e7569-57bd-5286-4593-7778d8700bf6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e2e7569-57bd-5286-4593-7778d8700bf6 Address:6e2e7569-57bd-5286-4593-7778d8700bf6} {Suffrage:Voter ID:server-948a8a60-d8b9-d514-dcf1-c6279b0fbe2f Address:948a8a60-d8b9-d514-dcf1-c6279b0fbe2f} {Suffrage:Voter ID:server-8043be34-cc58-6bd0-4019-ebd619a7ff92 Address:8043be34-cc58-6bd0-4019-ebd619a7ff92}]" 2025-09-14T21:48:54.039Z [INFO] server-948a8a60-d8b9-d514-dcf1-c6279b0fbe2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e2e7569-57bd-5286-4593-7778d8700bf6 Address:6e2e7569-57bd-5286-4593-7778d8700bf6} {Suffrage:Voter ID:server-948a8a60-d8b9-d514-dcf1-c6279b0fbe2f Address:948a8a60-d8b9-d514-dcf1-c6279b0fbe2f} {Suffrage:Voter ID:server-8043be34-cc58-6bd0-4019-ebd619a7ff92 Address:8043be34-cc58-6bd0-4019-ebd619a7ff92}]" 2025-09-14T21:48:54.039Z [INFO] server-6e2e7569-57bd-5286-4593-7778d8700bf6: entering follower state: follower="Node at 6e2e7569-57bd-5286-4593-7778d8700bf6 [Follower]" leader-address= leader-id= 2025-09-14T21:48:54.039Z [INFO] server-948a8a60-d8b9-d514-dcf1-c6279b0fbe2f: entering follower state: follower="Node at 948a8a60-d8b9-d514-dcf1-c6279b0fbe2f [Follower]" leader-address= leader-id= 2025-09-14T21:48:54.039Z [INFO] server-8043be34-cc58-6bd0-4019-ebd619a7ff92: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e2e7569-57bd-5286-4593-7778d8700bf6 Address:6e2e7569-57bd-5286-4593-7778d8700bf6} {Suffrage:Voter ID:server-948a8a60-d8b9-d514-dcf1-c6279b0fbe2f Address:948a8a60-d8b9-d514-dcf1-c6279b0fbe2f} {Suffrage:Voter ID:server-8043be34-cc58-6bd0-4019-ebd619a7ff92 Address:8043be34-cc58-6bd0-4019-ebd619a7ff92}]" 2025-09-14T21:48:54.039Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:54.039Z [INFO] server-8043be34-cc58-6bd0-4019-ebd619a7ff92: entering follower state: follower="Node at 8043be34-cc58-6bd0-4019-ebd619a7ff92 [Follower]" leader-address= leader-id= 2025-09-14T21:48:54.118Z [WARN] server-948a8a60-d8b9-d514-dcf1-c6279b0fbe2f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:54.118Z [INFO] server-948a8a60-d8b9-d514-dcf1-c6279b0fbe2f: entering candidate state: node="Node at 948a8a60-d8b9-d514-dcf1-c6279b0fbe2f [Candidate]" term=2 2025-09-14T21:48:54.118Z [INFO] server-948a8a60-d8b9-d514-dcf1-c6279b0fbe2f: election won: term=2 tally=2 2025-09-14T21:48:54.118Z [INFO] server-948a8a60-d8b9-d514-dcf1-c6279b0fbe2f: entering leader state: leader="Node at 948a8a60-d8b9-d514-dcf1-c6279b0fbe2f [Leader]" 2025-09-14T21:48:54.118Z [INFO] server-948a8a60-d8b9-d514-dcf1-c6279b0fbe2f: added peer, starting replication: peer=server-6e2e7569-57bd-5286-4593-7778d8700bf6 2025-09-14T21:48:54.118Z [INFO] server-948a8a60-d8b9-d514-dcf1-c6279b0fbe2f: added peer, starting replication: peer=server-8043be34-cc58-6bd0-4019-ebd619a7ff92 2025-09-14T21:48:54.118Z [INFO] server-948a8a60-d8b9-d514-dcf1-c6279b0fbe2f: pipelining replication: peer="{Voter server-6e2e7569-57bd-5286-4593-7778d8700bf6 6e2e7569-57bd-5286-4593-7778d8700bf6}" 2025-09-14T21:48:54.118Z [INFO] server-948a8a60-d8b9-d514-dcf1-c6279b0fbe2f: pipelining replication: peer="{Voter server-8043be34-cc58-6bd0-4019-ebd619a7ff92 8043be34-cc58-6bd0-4019-ebd619a7ff92}" 2025-09-14T21:48:54.223Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:54.118305307 +0000 UTC m=+104.686183583 (1 nodes), 78.754465ms from start of poll, 79.021311ms from cluster start. Timeout at 2025-09-14 21:48:54.223314099 +0000 UTC m=+104.791192376, 105.008793ms after stability 2025-09-14T21:48:54.223Z [INFO] server-948a8a60-d8b9-d514-dcf1-c6279b0fbe2f: aborting pipeline replication: peer="{Voter server-8043be34-cc58-6bd0-4019-ebd619a7ff92 8043be34-cc58-6bd0-4019-ebd619a7ff92}" 2025-09-14T21:48:54.223Z [INFO] server-948a8a60-d8b9-d514-dcf1-c6279b0fbe2f: aborting pipeline replication: peer="{Voter server-6e2e7569-57bd-5286-4593-7778d8700bf6 6e2e7569-57bd-5286-4593-7778d8700bf6}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.19s) === RUN TestRaft_LeadershipTransferToInvalidAddress 2025-09-14T21:48:54.225Z [INFO] server-b43082b9-6c94-11dc-7cd9-7fc24ab4cd90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b43082b9-6c94-11dc-7cd9-7fc24ab4cd90 Address:b43082b9-6c94-11dc-7cd9-7fc24ab4cd90} {Suffrage:Voter ID:server-cd13cadb-ea53-1ee6-6f7e-dba924e2fb19 Address:cd13cadb-ea53-1ee6-6f7e-dba924e2fb19} {Suffrage:Voter ID:server-70b772e7-0164-67f8-9b92-cbc2fbdecc60 Address:70b772e7-0164-67f8-9b92-cbc2fbdecc60}]" 2025-09-14T21:48:54.225Z [INFO] server-cd13cadb-ea53-1ee6-6f7e-dba924e2fb19: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b43082b9-6c94-11dc-7cd9-7fc24ab4cd90 Address:b43082b9-6c94-11dc-7cd9-7fc24ab4cd90} {Suffrage:Voter ID:server-cd13cadb-ea53-1ee6-6f7e-dba924e2fb19 Address:cd13cadb-ea53-1ee6-6f7e-dba924e2fb19} {Suffrage:Voter ID:server-70b772e7-0164-67f8-9b92-cbc2fbdecc60 Address:70b772e7-0164-67f8-9b92-cbc2fbdecc60}]" 2025-09-14T21:48:54.225Z [INFO] server-b43082b9-6c94-11dc-7cd9-7fc24ab4cd90: entering follower state: follower="Node at b43082b9-6c94-11dc-7cd9-7fc24ab4cd90 [Follower]" leader-address= leader-id= 2025-09-14T21:48:54.225Z [INFO] server-70b772e7-0164-67f8-9b92-cbc2fbdecc60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b43082b9-6c94-11dc-7cd9-7fc24ab4cd90 Address:b43082b9-6c94-11dc-7cd9-7fc24ab4cd90} {Suffrage:Voter ID:server-cd13cadb-ea53-1ee6-6f7e-dba924e2fb19 Address:cd13cadb-ea53-1ee6-6f7e-dba924e2fb19} {Suffrage:Voter ID:server-70b772e7-0164-67f8-9b92-cbc2fbdecc60 Address:70b772e7-0164-67f8-9b92-cbc2fbdecc60}]" 2025-09-14T21:48:54.225Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:54.225Z [INFO] server-cd13cadb-ea53-1ee6-6f7e-dba924e2fb19: entering follower state: follower="Node at cd13cadb-ea53-1ee6-6f7e-dba924e2fb19 [Follower]" leader-address= leader-id= 2025-09-14T21:48:54.225Z [INFO] server-70b772e7-0164-67f8-9b92-cbc2fbdecc60: entering follower state: follower="Node at 70b772e7-0164-67f8-9b92-cbc2fbdecc60 [Follower]" leader-address= leader-id= 2025-09-14T21:48:54.281Z [WARN] server-b43082b9-6c94-11dc-7cd9-7fc24ab4cd90: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:54.281Z [INFO] server-b43082b9-6c94-11dc-7cd9-7fc24ab4cd90: entering candidate state: node="Node at b43082b9-6c94-11dc-7cd9-7fc24ab4cd90 [Candidate]" term=2 2025-09-14T21:48:54.281Z [INFO] server-b43082b9-6c94-11dc-7cd9-7fc24ab4cd90: election won: term=2 tally=2 2025-09-14T21:48:54.281Z [INFO] server-b43082b9-6c94-11dc-7cd9-7fc24ab4cd90: entering leader state: leader="Node at b43082b9-6c94-11dc-7cd9-7fc24ab4cd90 [Leader]" 2025-09-14T21:48:54.281Z [INFO] server-b43082b9-6c94-11dc-7cd9-7fc24ab4cd90: added peer, starting replication: peer=server-cd13cadb-ea53-1ee6-6f7e-dba924e2fb19 2025-09-14T21:48:54.281Z [INFO] server-b43082b9-6c94-11dc-7cd9-7fc24ab4cd90: added peer, starting replication: peer=server-70b772e7-0164-67f8-9b92-cbc2fbdecc60 2025-09-14T21:48:54.281Z [INFO] server-b43082b9-6c94-11dc-7cd9-7fc24ab4cd90: pipelining replication: peer="{Voter server-70b772e7-0164-67f8-9b92-cbc2fbdecc60 70b772e7-0164-67f8-9b92-cbc2fbdecc60}" 2025-09-14T21:48:54.281Z [INFO] server-b43082b9-6c94-11dc-7cd9-7fc24ab4cd90: pipelining replication: peer="{Voter server-cd13cadb-ea53-1ee6-6f7e-dba924e2fb19 cd13cadb-ea53-1ee6-6f7e-dba924e2fb19}" 2025-09-14T21:48:54.387Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:54.281944775 +0000 UTC m=+104.849823054 (2 nodes), 56.578825ms from start of poll, 56.911208ms from cluster start. Timeout at 2025-09-14 21:48:54.386947186 +0000 UTC m=+104.954825448, 105.002394ms after stability 2025-09-14T21:48:54.387Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:54.493Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:54.388005922 +0000 UTC m=+104.955884184 (1 nodes), 2.231µs from start of poll, 162.972338ms from cluster start. Timeout at 2025-09-14 21:48:54.493039262 +0000 UTC m=+105.060917577, 105.033393ms after stability 2025-09-14T21:48:54.493Z [INFO] server-b43082b9-6c94-11dc-7cd9-7fc24ab4cd90: aborting pipeline replication: peer="{Voter server-70b772e7-0164-67f8-9b92-cbc2fbdecc60 70b772e7-0164-67f8-9b92-cbc2fbdecc60}" 2025-09-14T21:48:54.493Z [INFO] server-b43082b9-6c94-11dc-7cd9-7fc24ab4cd90: aborting pipeline replication: peer="{Voter server-cd13cadb-ea53-1ee6-6f7e-dba924e2fb19 cd13cadb-ea53-1ee6-6f7e-dba924e2fb19}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.27s) === RUN TestRaft_LeadershipTransferToBehindServer 2025-09-14T21:48:54.494Z [INFO] server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3 Address:0bddb8e7-d0fc-76d0-a385-2172aa3530a3} {Suffrage:Voter ID:server-2bf47b9d-7e24-d617-94be-857272f3bd49 Address:2bf47b9d-7e24-d617-94be-857272f3bd49} {Suffrage:Voter ID:server-b55e4fd5-b036-7834-1ade-2f094da0918d Address:b55e4fd5-b036-7834-1ade-2f094da0918d}]" 2025-09-14T21:48:54.494Z [INFO] server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3: entering follower state: follower="Node at 0bddb8e7-d0fc-76d0-a385-2172aa3530a3 [Follower]" leader-address= leader-id= 2025-09-14T21:48:54.494Z [INFO] server-2bf47b9d-7e24-d617-94be-857272f3bd49: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3 Address:0bddb8e7-d0fc-76d0-a385-2172aa3530a3} {Suffrage:Voter ID:server-2bf47b9d-7e24-d617-94be-857272f3bd49 Address:2bf47b9d-7e24-d617-94be-857272f3bd49} {Suffrage:Voter ID:server-b55e4fd5-b036-7834-1ade-2f094da0918d Address:b55e4fd5-b036-7834-1ade-2f094da0918d}]" 2025-09-14T21:48:54.495Z [INFO] server-b55e4fd5-b036-7834-1ade-2f094da0918d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3 Address:0bddb8e7-d0fc-76d0-a385-2172aa3530a3} {Suffrage:Voter ID:server-2bf47b9d-7e24-d617-94be-857272f3bd49 Address:2bf47b9d-7e24-d617-94be-857272f3bd49} {Suffrage:Voter ID:server-b55e4fd5-b036-7834-1ade-2f094da0918d Address:b55e4fd5-b036-7834-1ade-2f094da0918d}]" 2025-09-14T21:48:54.495Z [INFO] server-2bf47b9d-7e24-d617-94be-857272f3bd49: entering follower state: follower="Node at 2bf47b9d-7e24-d617-94be-857272f3bd49 [Follower]" leader-address= leader-id= 2025-09-14T21:48:54.495Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:54.495Z [INFO] server-b55e4fd5-b036-7834-1ade-2f094da0918d: entering follower state: follower="Node at b55e4fd5-b036-7834-1ade-2f094da0918d [Follower]" leader-address= leader-id= 2025-09-14T21:48:54.550Z [WARN] server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:54.551Z [INFO] server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3: entering candidate state: node="Node at 0bddb8e7-d0fc-76d0-a385-2172aa3530a3 [Candidate]" term=2 2025-09-14T21:48:54.551Z [INFO] server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3: election won: term=2 tally=2 2025-09-14T21:48:54.551Z [INFO] server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3: entering leader state: leader="Node at 0bddb8e7-d0fc-76d0-a385-2172aa3530a3 [Leader]" 2025-09-14T21:48:54.551Z [INFO] server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3: added peer, starting replication: peer=server-2bf47b9d-7e24-d617-94be-857272f3bd49 2025-09-14T21:48:54.551Z [INFO] server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3: added peer, starting replication: peer=server-b55e4fd5-b036-7834-1ade-2f094da0918d 2025-09-14T21:48:54.551Z [INFO] server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3: pipelining replication: peer="{Voter server-b55e4fd5-b036-7834-1ade-2f094da0918d b55e4fd5-b036-7834-1ade-2f094da0918d}" 2025-09-14T21:48:54.551Z [INFO] server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3: pipelining replication: peer="{Voter server-2bf47b9d-7e24-d617-94be-857272f3bd49 2bf47b9d-7e24-d617-94be-857272f3bd49}" 2025-09-14T21:48:54.656Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:54.551148834 +0000 UTC m=+105.119027097 (1 nodes), 56.08574ms from start of poll, 56.414135ms from cluster start. Timeout at 2025-09-14 21:48:54.656152422 +0000 UTC m=+105.224030684, 105.003587ms after stability 2025-09-14T21:48:54.656Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:54.762Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:54.65639799 +0000 UTC m=+105.224276264 (2 nodes), 910ns from start of poll, 161.663302ms from cluster start. Timeout at 2025-09-14 21:48:54.761419915 +0000 UTC m=+105.329298214, 105.02195ms after stability 2025-09-14T21:48:54.792Z [INFO] server-2bf47b9d-7e24-d617-94be-857272f3bd49: entering candidate state: node="Node at 2bf47b9d-7e24-d617-94be-857272f3bd49 [Candidate]" term=3 2025-09-14T21:48:54.792Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:54.792Z [INFO] server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3: entering follower state: follower="Node at 0bddb8e7-d0fc-76d0-a385-2172aa3530a3 [Follower]" leader-address= leader-id= 2025-09-14T21:48:54.792Z [INFO] server-2bf47b9d-7e24-d617-94be-857272f3bd49: election won: term=3 tally=2 2025-09-14T21:48:54.792Z [INFO] server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3: aborting pipeline replication: peer="{Voter server-2bf47b9d-7e24-d617-94be-857272f3bd49 2bf47b9d-7e24-d617-94be-857272f3bd49}" 2025-09-14T21:48:54.792Z [INFO] server-2bf47b9d-7e24-d617-94be-857272f3bd49: entering leader state: leader="Node at 2bf47b9d-7e24-d617-94be-857272f3bd49 [Leader]" 2025-09-14T21:48:54.792Z [INFO] server-2bf47b9d-7e24-d617-94be-857272f3bd49: added peer, starting replication: peer=server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3 2025-09-14T21:48:54.792Z [INFO] server-2bf47b9d-7e24-d617-94be-857272f3bd49: added peer, starting replication: peer=server-b55e4fd5-b036-7834-1ade-2f094da0918d 2025-09-14T21:48:54.792Z [INFO] server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3: aborting pipeline replication: peer="{Voter server-b55e4fd5-b036-7834-1ade-2f094da0918d b55e4fd5-b036-7834-1ade-2f094da0918d}" 2025-09-14T21:48:54.792Z [INFO] server-2bf47b9d-7e24-d617-94be-857272f3bd49: pipelining replication: peer="{Voter server-b55e4fd5-b036-7834-1ade-2f094da0918d b55e4fd5-b036-7834-1ade-2f094da0918d}" 2025-09-14T21:48:54.792Z [INFO] server-2bf47b9d-7e24-d617-94be-857272f3bd49: pipelining replication: peer="{Voter server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3 0bddb8e7-d0fc-76d0-a385-2172aa3530a3}" 2025-09-14T21:48:54.897Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:54.79242868 +0000 UTC m=+105.360306960 (1 nodes), 69.285µs from start of poll, 297.693998ms from cluster start. Timeout at 2025-09-14 21:48:54.897441263 +0000 UTC m=+105.465319546, 105.012586ms after stability 2025-09-14T21:48:54.897Z [INFO] server-2bf47b9d-7e24-d617-94be-857272f3bd49: aborting pipeline replication: peer="{Voter server-b55e4fd5-b036-7834-1ade-2f094da0918d b55e4fd5-b036-7834-1ade-2f094da0918d}" 2025-09-14T21:48:54.897Z [INFO] server-2bf47b9d-7e24-d617-94be-857272f3bd49: aborting pipeline replication: peer="{Voter server-0bddb8e7-d0fc-76d0-a385-2172aa3530a3 0bddb8e7-d0fc-76d0-a385-2172aa3530a3}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.40s) === RUN TestRaft_LeadershipTransferToItself 2025-09-14T21:48:54.898Z [INFO] server-0b521c3c-816a-3b47-96da-a1eec149cb93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b521c3c-816a-3b47-96da-a1eec149cb93 Address:0b521c3c-816a-3b47-96da-a1eec149cb93} {Suffrage:Voter ID:server-7a70bea3-904a-69e2-c6ad-d6da6c1041d6 Address:7a70bea3-904a-69e2-c6ad-d6da6c1041d6} {Suffrage:Voter ID:server-ca7377bd-32d4-4c53-a3d4-a74882e4f35f Address:ca7377bd-32d4-4c53-a3d4-a74882e4f35f}]" 2025-09-14T21:48:54.898Z [INFO] server-7a70bea3-904a-69e2-c6ad-d6da6c1041d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b521c3c-816a-3b47-96da-a1eec149cb93 Address:0b521c3c-816a-3b47-96da-a1eec149cb93} {Suffrage:Voter ID:server-7a70bea3-904a-69e2-c6ad-d6da6c1041d6 Address:7a70bea3-904a-69e2-c6ad-d6da6c1041d6} {Suffrage:Voter ID:server-ca7377bd-32d4-4c53-a3d4-a74882e4f35f Address:ca7377bd-32d4-4c53-a3d4-a74882e4f35f}]" 2025-09-14T21:48:54.898Z [INFO] server-0b521c3c-816a-3b47-96da-a1eec149cb93: entering follower state: follower="Node at 0b521c3c-816a-3b47-96da-a1eec149cb93 [Follower]" leader-address= leader-id= 2025-09-14T21:48:54.898Z [INFO] server-7a70bea3-904a-69e2-c6ad-d6da6c1041d6: entering follower state: follower="Node at 7a70bea3-904a-69e2-c6ad-d6da6c1041d6 [Follower]" leader-address= leader-id= 2025-09-14T21:48:54.898Z [INFO] server-ca7377bd-32d4-4c53-a3d4-a74882e4f35f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b521c3c-816a-3b47-96da-a1eec149cb93 Address:0b521c3c-816a-3b47-96da-a1eec149cb93} {Suffrage:Voter ID:server-7a70bea3-904a-69e2-c6ad-d6da6c1041d6 Address:7a70bea3-904a-69e2-c6ad-d6da6c1041d6} {Suffrage:Voter ID:server-ca7377bd-32d4-4c53-a3d4-a74882e4f35f Address:ca7377bd-32d4-4c53-a3d4-a74882e4f35f}]" 2025-09-14T21:48:54.898Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:54.898Z [INFO] server-ca7377bd-32d4-4c53-a3d4-a74882e4f35f: entering follower state: follower="Node at ca7377bd-32d4-4c53-a3d4-a74882e4f35f [Follower]" leader-address= leader-id= 2025-09-14T21:48:54.952Z [WARN] server-0b521c3c-816a-3b47-96da-a1eec149cb93: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:54.952Z [INFO] server-0b521c3c-816a-3b47-96da-a1eec149cb93: entering candidate state: node="Node at 0b521c3c-816a-3b47-96da-a1eec149cb93 [Candidate]" term=2 2025-09-14T21:48:54.952Z [INFO] server-0b521c3c-816a-3b47-96da-a1eec149cb93: election won: term=2 tally=2 2025-09-14T21:48:54.952Z [INFO] server-0b521c3c-816a-3b47-96da-a1eec149cb93: entering leader state: leader="Node at 0b521c3c-816a-3b47-96da-a1eec149cb93 [Leader]" 2025-09-14T21:48:54.952Z [INFO] server-0b521c3c-816a-3b47-96da-a1eec149cb93: added peer, starting replication: peer=server-7a70bea3-904a-69e2-c6ad-d6da6c1041d6 2025-09-14T21:48:54.952Z [INFO] server-0b521c3c-816a-3b47-96da-a1eec149cb93: added peer, starting replication: peer=server-ca7377bd-32d4-4c53-a3d4-a74882e4f35f 2025-09-14T21:48:54.952Z [INFO] server-0b521c3c-816a-3b47-96da-a1eec149cb93: pipelining replication: peer="{Voter server-ca7377bd-32d4-4c53-a3d4-a74882e4f35f ca7377bd-32d4-4c53-a3d4-a74882e4f35f}" 2025-09-14T21:48:54.952Z [INFO] server-0b521c3c-816a-3b47-96da-a1eec149cb93: pipelining replication: peer="{Voter server-7a70bea3-904a-69e2-c6ad-d6da6c1041d6 7a70bea3-904a-69e2-c6ad-d6da6c1041d6}" 2025-09-14T21:48:55.057Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:54.952474515 +0000 UTC m=+105.520352798 (1 nodes), 53.540659ms from start of poll, 53.811891ms from cluster start. Timeout at 2025-09-14 21:48:55.05747757 +0000 UTC m=+105.625355832, 105.003034ms after stability 2025-09-14T21:48:55.058Z [INFO] server-0b521c3c-816a-3b47-96da-a1eec149cb93: cannot transfer leadership to itself 2025-09-14T21:48:55.058Z [INFO] server-0b521c3c-816a-3b47-96da-a1eec149cb93: aborting pipeline replication: peer="{Voter server-ca7377bd-32d4-4c53-a3d4-a74882e4f35f ca7377bd-32d4-4c53-a3d4-a74882e4f35f}" 2025-09-14T21:48:55.058Z [INFO] server-0b521c3c-816a-3b47-96da-a1eec149cb93: aborting pipeline replication: peer="{Voter server-7a70bea3-904a-69e2-c6ad-d6da6c1041d6 7a70bea3-904a-69e2-c6ad-d6da6c1041d6}" --- PASS: TestRaft_LeadershipTransferToItself (0.16s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests 2025-09-14T21:48:55.059Z [INFO] server-8b99c116-7b99-b34a-d189-abc314c327c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b99c116-7b99-b34a-d189-abc314c327c0 Address:8b99c116-7b99-b34a-d189-abc314c327c0} {Suffrage:Voter ID:server-1e256907-b79b-ceb8-8c73-3c5e8b9ff97b Address:1e256907-b79b-ceb8-8c73-3c5e8b9ff97b} {Suffrage:Voter ID:server-f076d1a5-53de-4036-abbf-8a045eda6ff6 Address:f076d1a5-53de-4036-abbf-8a045eda6ff6}]" 2025-09-14T21:48:55.059Z [INFO] server-8b99c116-7b99-b34a-d189-abc314c327c0: entering follower state: follower="Node at 8b99c116-7b99-b34a-d189-abc314c327c0 [Follower]" leader-address= leader-id= 2025-09-14T21:48:55.059Z [INFO] server-1e256907-b79b-ceb8-8c73-3c5e8b9ff97b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b99c116-7b99-b34a-d189-abc314c327c0 Address:8b99c116-7b99-b34a-d189-abc314c327c0} {Suffrage:Voter ID:server-1e256907-b79b-ceb8-8c73-3c5e8b9ff97b Address:1e256907-b79b-ceb8-8c73-3c5e8b9ff97b} {Suffrage:Voter ID:server-f076d1a5-53de-4036-abbf-8a045eda6ff6 Address:f076d1a5-53de-4036-abbf-8a045eda6ff6}]" 2025-09-14T21:48:55.059Z [INFO] server-1e256907-b79b-ceb8-8c73-3c5e8b9ff97b: entering follower state: follower="Node at 1e256907-b79b-ceb8-8c73-3c5e8b9ff97b [Follower]" leader-address= leader-id= 2025-09-14T21:48:55.059Z [INFO] server-f076d1a5-53de-4036-abbf-8a045eda6ff6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b99c116-7b99-b34a-d189-abc314c327c0 Address:8b99c116-7b99-b34a-d189-abc314c327c0} {Suffrage:Voter ID:server-1e256907-b79b-ceb8-8c73-3c5e8b9ff97b Address:1e256907-b79b-ceb8-8c73-3c5e8b9ff97b} {Suffrage:Voter ID:server-f076d1a5-53de-4036-abbf-8a045eda6ff6 Address:f076d1a5-53de-4036-abbf-8a045eda6ff6}]" 2025-09-14T21:48:55.059Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:55.059Z [INFO] server-f076d1a5-53de-4036-abbf-8a045eda6ff6: entering follower state: follower="Node at f076d1a5-53de-4036-abbf-8a045eda6ff6 [Follower]" leader-address= leader-id= 2025-09-14T21:48:55.116Z [WARN] server-8b99c116-7b99-b34a-d189-abc314c327c0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:55.116Z [INFO] server-8b99c116-7b99-b34a-d189-abc314c327c0: entering candidate state: node="Node at 8b99c116-7b99-b34a-d189-abc314c327c0 [Candidate]" term=2 2025-09-14T21:48:55.116Z [INFO] server-8b99c116-7b99-b34a-d189-abc314c327c0: election won: term=2 tally=2 2025-09-14T21:48:55.116Z [INFO] server-8b99c116-7b99-b34a-d189-abc314c327c0: entering leader state: leader="Node at 8b99c116-7b99-b34a-d189-abc314c327c0 [Leader]" 2025-09-14T21:48:55.116Z [INFO] server-8b99c116-7b99-b34a-d189-abc314c327c0: added peer, starting replication: peer=server-1e256907-b79b-ceb8-8c73-3c5e8b9ff97b 2025-09-14T21:48:55.116Z [INFO] server-8b99c116-7b99-b34a-d189-abc314c327c0: added peer, starting replication: peer=server-f076d1a5-53de-4036-abbf-8a045eda6ff6 2025-09-14T21:48:55.116Z [INFO] server-8b99c116-7b99-b34a-d189-abc314c327c0: pipelining replication: peer="{Voter server-f076d1a5-53de-4036-abbf-8a045eda6ff6 f076d1a5-53de-4036-abbf-8a045eda6ff6}" 2025-09-14T21:48:55.116Z [INFO] server-8b99c116-7b99-b34a-d189-abc314c327c0: pipelining replication: peer="{Voter server-1e256907-b79b-ceb8-8c73-3c5e8b9ff97b 1e256907-b79b-ceb8-8c73-3c5e8b9ff97b}" 2025-09-14T21:48:55.222Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:55.116340193 +0000 UTC m=+105.684218453 (1 nodes), 56.749555ms from start of poll, 57.149282ms from cluster start. Timeout at 2025-09-14 21:48:55.221344545 +0000 UTC m=+105.789222804, 105.004351ms after stability raft_test.go:2457: waiting on future 0 raft_test.go:2457: waiting on future 1 raft_test.go:2457: waiting on future 2 raft_test.go:2457: waiting on future 3 raft_test.go:2457: waiting on future 4 raft_test.go:2457: waiting on future 5 raft_test.go:2457: waiting on future 6 2025-09-14T21:48:55.222Z [INFO] server-8b99c116-7b99-b34a-d189-abc314c327c0: aborting pipeline replication: peer="{Voter server-f076d1a5-53de-4036-abbf-8a045eda6ff6 f076d1a5-53de-4036-abbf-8a045eda6ff6}" 2025-09-14T21:48:55.222Z [INFO] server-8b99c116-7b99-b34a-d189-abc314c327c0: aborting pipeline replication: peer="{Voter server-1e256907-b79b-ceb8-8c73-3c5e8b9ff97b 1e256907-b79b-ceb8-8c73-3c5e8b9ff97b}" --- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.16s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout 2025-09-14T21:48:55.223Z [INFO] server-7b9a5a8f-85e5-0019-590a-d538c4539522: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b9a5a8f-85e5-0019-590a-d538c4539522 Address:7b9a5a8f-85e5-0019-590a-d538c4539522} {Suffrage:Voter ID:server-a6834879-0353-30ec-a694-63e8c096a9d0 Address:a6834879-0353-30ec-a694-63e8c096a9d0} {Suffrage:Voter ID:server-efe994e4-be1e-67f7-48fb-5338f1bddb5f Address:efe994e4-be1e-67f7-48fb-5338f1bddb5f}]" 2025-09-14T21:48:55.223Z [INFO] server-7b9a5a8f-85e5-0019-590a-d538c4539522: entering follower state: follower="Node at 7b9a5a8f-85e5-0019-590a-d538c4539522 [Follower]" leader-address= leader-id= 2025-09-14T21:48:55.223Z [INFO] server-a6834879-0353-30ec-a694-63e8c096a9d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b9a5a8f-85e5-0019-590a-d538c4539522 Address:7b9a5a8f-85e5-0019-590a-d538c4539522} {Suffrage:Voter ID:server-a6834879-0353-30ec-a694-63e8c096a9d0 Address:a6834879-0353-30ec-a694-63e8c096a9d0} {Suffrage:Voter ID:server-efe994e4-be1e-67f7-48fb-5338f1bddb5f Address:efe994e4-be1e-67f7-48fb-5338f1bddb5f}]" 2025-09-14T21:48:55.223Z [INFO] server-a6834879-0353-30ec-a694-63e8c096a9d0: entering follower state: follower="Node at a6834879-0353-30ec-a694-63e8c096a9d0 [Follower]" leader-address= leader-id= 2025-09-14T21:48:55.224Z [INFO] server-efe994e4-be1e-67f7-48fb-5338f1bddb5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b9a5a8f-85e5-0019-590a-d538c4539522 Address:7b9a5a8f-85e5-0019-590a-d538c4539522} {Suffrage:Voter ID:server-a6834879-0353-30ec-a694-63e8c096a9d0 Address:a6834879-0353-30ec-a694-63e8c096a9d0} {Suffrage:Voter ID:server-efe994e4-be1e-67f7-48fb-5338f1bddb5f Address:efe994e4-be1e-67f7-48fb-5338f1bddb5f}]" 2025-09-14T21:48:55.224Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:55.224Z [INFO] server-efe994e4-be1e-67f7-48fb-5338f1bddb5f: entering follower state: follower="Node at efe994e4-be1e-67f7-48fb-5338f1bddb5f [Follower]" leader-address= leader-id= 2025-09-14T21:48:55.276Z [WARN] server-a6834879-0353-30ec-a694-63e8c096a9d0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:55.276Z [INFO] server-a6834879-0353-30ec-a694-63e8c096a9d0: entering candidate state: node="Node at a6834879-0353-30ec-a694-63e8c096a9d0 [Candidate]" term=2 2025-09-14T21:48:55.276Z [INFO] server-a6834879-0353-30ec-a694-63e8c096a9d0: election won: term=2 tally=2 2025-09-14T21:48:55.276Z [INFO] server-a6834879-0353-30ec-a694-63e8c096a9d0: entering leader state: leader="Node at a6834879-0353-30ec-a694-63e8c096a9d0 [Leader]" 2025-09-14T21:48:55.276Z [INFO] server-a6834879-0353-30ec-a694-63e8c096a9d0: added peer, starting replication: peer=server-7b9a5a8f-85e5-0019-590a-d538c4539522 2025-09-14T21:48:55.276Z [INFO] server-a6834879-0353-30ec-a694-63e8c096a9d0: added peer, starting replication: peer=server-efe994e4-be1e-67f7-48fb-5338f1bddb5f 2025-09-14T21:48:55.276Z [INFO] server-a6834879-0353-30ec-a694-63e8c096a9d0: pipelining replication: peer="{Voter server-7b9a5a8f-85e5-0019-590a-d538c4539522 7b9a5a8f-85e5-0019-590a-d538c4539522}" 2025-09-14T21:48:55.276Z [INFO] server-a6834879-0353-30ec-a694-63e8c096a9d0: pipelining replication: peer="{Voter server-efe994e4-be1e-67f7-48fb-5338f1bddb5f efe994e4-be1e-67f7-48fb-5338f1bddb5f}" 2025-09-14T21:48:55.382Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:55.276735511 +0000 UTC m=+105.844613773 (1 nodes), 52.664789ms from start of poll, 53.062846ms from cluster start. Timeout at 2025-09-14 21:48:55.381739065 +0000 UTC m=+105.949617328, 105.003555ms after stability 2025-09-14T21:48:55.382Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:55.487Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:55.382186147 +0000 UTC m=+105.950064410 (2 nodes), 1.904µs from start of poll, 158.513483ms from cluster start. Timeout at 2025-09-14 21:48:55.487219144 +0000 UTC m=+106.055097410, 105.033ms after stability 2025-09-14T21:48:55.760Z [INFO] server-a6834879-0353-30ec-a694-63e8c096a9d0: aborting pipeline replication: peer="{Voter server-efe994e4-be1e-67f7-48fb-5338f1bddb5f efe994e4-be1e-67f7-48fb-5338f1bddb5f}" 2025-09-14T21:48:55.760Z [INFO] server-a6834879-0353-30ec-a694-63e8c096a9d0: aborting pipeline replication: peer="{Voter server-7b9a5a8f-85e5-0019-590a-d538c4539522 7b9a5a8f-85e5-0019-590a-d538c4539522}" --- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.54s) === RUN TestRaft_LeadershipTransferIgnoresNonvoters 2025-09-14T21:48:55.762Z [INFO] server-d8c38a76-d23d-bb70-9df9-ce6320ed3f97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8c38a76-d23d-bb70-9df9-ce6320ed3f97 Address:d8c38a76-d23d-bb70-9df9-ce6320ed3f97} {Suffrage:Voter ID:server-73ffbb16-a15c-13c1-6b9a-700491ce66a9 Address:73ffbb16-a15c-13c1-6b9a-700491ce66a9}]" 2025-09-14T21:48:55.762Z [INFO] server-73ffbb16-a15c-13c1-6b9a-700491ce66a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8c38a76-d23d-bb70-9df9-ce6320ed3f97 Address:d8c38a76-d23d-bb70-9df9-ce6320ed3f97} {Suffrage:Voter ID:server-73ffbb16-a15c-13c1-6b9a-700491ce66a9 Address:73ffbb16-a15c-13c1-6b9a-700491ce66a9}]" 2025-09-14T21:48:55.762Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:55.762Z [INFO] server-d8c38a76-d23d-bb70-9df9-ce6320ed3f97: entering follower state: follower="Node at d8c38a76-d23d-bb70-9df9-ce6320ed3f97 [Follower]" leader-address= leader-id= 2025-09-14T21:48:55.762Z [INFO] server-73ffbb16-a15c-13c1-6b9a-700491ce66a9: entering follower state: follower="Node at 73ffbb16-a15c-13c1-6b9a-700491ce66a9 [Follower]" leader-address= leader-id= 2025-09-14T21:48:55.832Z [WARN] server-73ffbb16-a15c-13c1-6b9a-700491ce66a9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:55.832Z [INFO] server-73ffbb16-a15c-13c1-6b9a-700491ce66a9: entering candidate state: node="Node at 73ffbb16-a15c-13c1-6b9a-700491ce66a9 [Candidate]" term=2 2025-09-14T21:48:55.832Z [INFO] server-73ffbb16-a15c-13c1-6b9a-700491ce66a9: election won: term=2 tally=2 2025-09-14T21:48:55.832Z [INFO] server-73ffbb16-a15c-13c1-6b9a-700491ce66a9: entering leader state: leader="Node at 73ffbb16-a15c-13c1-6b9a-700491ce66a9 [Leader]" 2025-09-14T21:48:55.832Z [INFO] server-73ffbb16-a15c-13c1-6b9a-700491ce66a9: added peer, starting replication: peer=server-d8c38a76-d23d-bb70-9df9-ce6320ed3f97 2025-09-14T21:48:55.832Z [INFO] server-73ffbb16-a15c-13c1-6b9a-700491ce66a9: pipelining replication: peer="{Voter server-d8c38a76-d23d-bb70-9df9-ce6320ed3f97 d8c38a76-d23d-bb70-9df9-ce6320ed3f97}" 2025-09-14T21:48:55.937Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:55.832584647 +0000 UTC m=+106.400462909 (1 nodes), 70.30914ms from start of poll, 70.570459ms from cluster start. Timeout at 2025-09-14 21:48:55.937592583 +0000 UTC m=+106.505470866, 105.007957ms after stability 2025-09-14T21:48:55.938Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:56.043Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:55.93802407 +0000 UTC m=+106.505902348 (1 nodes), 1.879µs from start of poll, 176.009898ms from cluster start. Timeout at 2025-09-14 21:48:56.043143487 +0000 UTC m=+106.611021751, 105.119403ms after stability 2025-09-14T21:48:56.043Z [INFO] server-73ffbb16-a15c-13c1-6b9a-700491ce66a9: updating configuration: command=DemoteVoter server-id=server-d8c38a76-d23d-bb70-9df9-ce6320ed3f97 server-addr= servers="[{Suffrage:Nonvoter ID:server-d8c38a76-d23d-bb70-9df9-ce6320ed3f97 Address:d8c38a76-d23d-bb70-9df9-ce6320ed3f97} {Suffrage:Voter ID:server-73ffbb16-a15c-13c1-6b9a-700491ce66a9 Address:73ffbb16-a15c-13c1-6b9a-700491ce66a9}]" 2025-09-14T21:48:56.043Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:56.149Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:56.043986807 +0000 UTC m=+106.611865066 (1 nodes), 297ns from start of poll, 281.972616ms from cluster start. Timeout at 2025-09-14 21:48:56.149075247 +0000 UTC m=+106.716953507, 105.088441ms after stability 2025-09-14T21:48:56.149Z [INFO] server-73ffbb16-a15c-13c1-6b9a-700491ce66a9: aborting pipeline replication: peer="{Voter server-d8c38a76-d23d-bb70-9df9-ce6320ed3f97 d8c38a76-d23d-bb70-9df9-ce6320ed3f97}" --- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.39s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap 2025-09-14T21:48:56.150Z [INFO] server-e9167798-68d4-3ec5-94bf-9a1e97756ba9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9167798-68d4-3ec5-94bf-9a1e97756ba9 Address:e9167798-68d4-3ec5-94bf-9a1e97756ba9} {Suffrage:Voter ID:server-c01149f0-44dd-15c4-8abb-27de8b50a14a Address:c01149f0-44dd-15c4-8abb-27de8b50a14a}]" 2025-09-14T21:48:56.150Z [INFO] server-e9167798-68d4-3ec5-94bf-9a1e97756ba9: entering follower state: follower="Node at e9167798-68d4-3ec5-94bf-9a1e97756ba9 [Follower]" leader-address= leader-id= 2025-09-14T21:48:56.151Z [INFO] server-c01149f0-44dd-15c4-8abb-27de8b50a14a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9167798-68d4-3ec5-94bf-9a1e97756ba9 Address:e9167798-68d4-3ec5-94bf-9a1e97756ba9} {Suffrage:Voter ID:server-c01149f0-44dd-15c4-8abb-27de8b50a14a Address:c01149f0-44dd-15c4-8abb-27de8b50a14a}]" 2025-09-14T21:48:56.151Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:56.151Z [INFO] server-c01149f0-44dd-15c4-8abb-27de8b50a14a: entering follower state: follower="Node at c01149f0-44dd-15c4-8abb-27de8b50a14a [Follower]" leader-address= leader-id= 2025-09-14T21:48:56.224Z [WARN] server-c01149f0-44dd-15c4-8abb-27de8b50a14a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:56.224Z [INFO] server-c01149f0-44dd-15c4-8abb-27de8b50a14a: entering candidate state: node="Node at c01149f0-44dd-15c4-8abb-27de8b50a14a [Candidate]" term=2 2025-09-14T21:48:56.224Z [INFO] server-c01149f0-44dd-15c4-8abb-27de8b50a14a: election won: term=2 tally=2 2025-09-14T21:48:56.224Z [INFO] server-c01149f0-44dd-15c4-8abb-27de8b50a14a: entering leader state: leader="Node at c01149f0-44dd-15c4-8abb-27de8b50a14a [Leader]" 2025-09-14T21:48:56.224Z [INFO] server-c01149f0-44dd-15c4-8abb-27de8b50a14a: added peer, starting replication: peer=server-e9167798-68d4-3ec5-94bf-9a1e97756ba9 2025-09-14T21:48:56.225Z [INFO] server-c01149f0-44dd-15c4-8abb-27de8b50a14a: pipelining replication: peer="{Voter server-e9167798-68d4-3ec5-94bf-9a1e97756ba9 e9167798-68d4-3ec5-94bf-9a1e97756ba9}" 2025-09-14T21:48:56.330Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:56.22494304 +0000 UTC m=+106.792821391 (1 nodes), 73.891965ms from start of poll, 74.152128ms from cluster start. Timeout at 2025-09-14 21:48:56.329949407 +0000 UTC m=+106.897827667, 105.006276ms after stability 2025-09-14T21:48:56.330Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:56.436Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:56.330722614 +0000 UTC m=+106.898600877 (1 nodes), 476ns from start of poll, 179.931614ms from cluster start. Timeout at 2025-09-14 21:48:56.435759754 +0000 UTC m=+107.003638014, 105.037137ms after stability 2025-09-14T21:48:56.447Z [INFO] server-c01149f0-44dd-15c4-8abb-27de8b50a14a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9167798-68d4-3ec5-94bf-9a1e97756ba9 Address:e9167798-68d4-3ec5-94bf-9a1e97756ba9} {Suffrage:Voter ID:server-c01149f0-44dd-15c4-8abb-27de8b50a14a Address:c01149f0-44dd-15c4-8abb-27de8b50a14a}]" 2025-09-14T21:48:56.447Z [INFO] server-c01149f0-44dd-15c4-8abb-27de8b50a14a: aborting pipeline replication: peer="{Voter server-e9167798-68d4-3ec5-94bf-9a1e97756ba9 e9167798-68d4-3ec5-94bf-9a1e97756ba9}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.30s) === RUN TestRaft_LogStoreIsMonotonic 2025-09-14T21:48:56.448Z [INFO] server-c49c0461-777f-7e7b-f83e-da8488684de9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c49c0461-777f-7e7b-f83e-da8488684de9 Address:c49c0461-777f-7e7b-f83e-da8488684de9}]" 2025-09-14T21:48:56.448Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:56.448Z [INFO] server-c49c0461-777f-7e7b-f83e-da8488684de9: entering follower state: follower="Node at c49c0461-777f-7e7b-f83e-da8488684de9 [Follower]" leader-address= leader-id= 2025-09-14T21:48:56.548Z [WARN] server-c49c0461-777f-7e7b-f83e-da8488684de9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:56.548Z [INFO] server-c49c0461-777f-7e7b-f83e-da8488684de9: entering candidate state: node="Node at c49c0461-777f-7e7b-f83e-da8488684de9 [Candidate]" term=2 2025-09-14T21:48:56.548Z [INFO] server-c49c0461-777f-7e7b-f83e-da8488684de9: election won: term=2 tally=1 2025-09-14T21:48:56.548Z [INFO] server-c49c0461-777f-7e7b-f83e-da8488684de9: entering leader state: leader="Node at c49c0461-777f-7e7b-f83e-da8488684de9 [Leader]" 2025-09-14T21:48:56.654Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:56.548649954 +0000 UTC m=+107.116528212 (1 nodes), 99.998611ms from start of poll, 100.112675ms from cluster start. Timeout at 2025-09-14 21:48:56.653719498 +0000 UTC m=+107.221597756, 105.069544ms after stability --- PASS: TestRaft_LogStoreIsMonotonic (0.21s) === RUN TestRaft_CacheLogWithStoreError 2025-09-14T21:48:56.654Z [INFO] server-c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9 Address:c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9} {Suffrage:Voter ID:server-e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4 Address:e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4}]" 2025-09-14T21:48:56.654Z [INFO] server-c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9: entering follower state: follower="Node at c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9 [Follower]" leader-address= leader-id= 2025-09-14T21:48:56.654Z [INFO] server-e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9 Address:c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9} {Suffrage:Voter ID:server-e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4 Address:e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4}]" 2025-09-14T21:48:56.655Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:56.655Z [INFO] server-e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4: entering follower state: follower="Node at e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4 [Follower]" leader-address= leader-id= 2025-09-14T21:48:56.721Z [WARN] server-c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:56.721Z [INFO] server-c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9: entering candidate state: node="Node at c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9 [Candidate]" term=2 2025-09-14T21:48:56.721Z [INFO] server-c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9: election won: term=2 tally=2 2025-09-14T21:48:56.721Z [INFO] server-c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9: entering leader state: leader="Node at c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9 [Leader]" 2025-09-14T21:48:56.722Z [INFO] server-c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9: added peer, starting replication: peer=server-e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4 2025-09-14T21:48:56.722Z [INFO] server-c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9: pipelining replication: peer="{Voter server-e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4 e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4}" 2025-09-14T21:48:56.828Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:56.722014822 +0000 UTC m=+107.289893095 (1 nodes), 66.981488ms from start of poll, 67.208979ms from cluster start. Timeout at 2025-09-14 21:48:56.827020201 +0000 UTC m=+107.394898478, 105.005383ms after stability 2025-09-14T21:48:56.828Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:56.933Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:56.82814151 +0000 UTC m=+107.396019781 (1 nodes), 1.418µs from start of poll, 173.335665ms from cluster start. Timeout at 2025-09-14 21:48:56.933165529 +0000 UTC m=+107.501043787, 105.024006ms after stability 2025-09-14T21:48:56.934Z [ERROR] server-e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4: failed to append to logs: error="unable to store logs within log store, err: "some error"" 2025-09-14T21:48:56.934Z [WARN] server-e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4: failed to get previous log: previous-index=9 last-index=7 error="log not found" 2025-09-14T21:48:56.934Z [INFO] server-c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9: aborting pipeline replication: peer="{Voter server-e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4 e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4}" 2025-09-14T21:48:56.940Z [INFO] server-c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9: pipelining replication: peer="{Voter server-e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4 e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4}" 2025-09-14T21:48:56.940Z [INFO] server-e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9 Address:c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9} {Suffrage:Voter ID:server-e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4 Address:e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4}]" 2025-09-14T21:48:56.940Z [INFO] server-c75b815a-8c4a-61df-b30d-2ccaf7a2b3a9: aborting pipeline replication: peer="{Voter server-e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4 e0f836a7-4a1b-6f37-41d6-dd0607ad3fd4}" --- PASS: TestRaft_CacheLogWithStoreError (0.29s) === RUN TestRaft_ReloadConfig 2025-09-14T21:48:56.941Z [INFO] server-e2fef0cd-ec60-1d39-8479-9353bd65dc19: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2fef0cd-ec60-1d39-8479-9353bd65dc19 Address:e2fef0cd-ec60-1d39-8479-9353bd65dc19}]" 2025-09-14T21:48:56.941Z [INFO] server-e2fef0cd-ec60-1d39-8479-9353bd65dc19: entering follower state: follower="Node at e2fef0cd-ec60-1d39-8479-9353bd65dc19 [Shutdown]" leader-address= leader-id= --- PASS: TestRaft_ReloadConfig (0.00s) === RUN TestRaft_ReloadConfigValidates 2025-09-14T21:48:56.942Z [INFO] server-bd8f73d3-4bb4-a1fe-d41d-daa31a909e3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bd8f73d3-4bb4-a1fe-d41d-daa31a909e3a Address:bd8f73d3-4bb4-a1fe-d41d-daa31a909e3a}]" 2025-09-14T21:48:56.942Z [INFO] server-bd8f73d3-4bb4-a1fe-d41d-daa31a909e3a: entering follower state: follower="Node at bd8f73d3-4bb4-a1fe-d41d-daa31a909e3a [Shutdown]" leader-address= leader-id= --- PASS: TestRaft_ReloadConfigValidates (0.00s) === RUN TestRaft_InstallSnapshot_InvalidPeers 2025-09-14T21:48:56.942Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)" --- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s) === RUN TestRaft_VoteNotGranted_WhenNodeNotInCluster 2025-09-14T21:48:56.943Z [INFO] server-f931fe05-4494-8775-5a00-7b70ba8ae226: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f931fe05-4494-8775-5a00-7b70ba8ae226 Address:f931fe05-4494-8775-5a00-7b70ba8ae226} {Suffrage:Voter ID:server-2506ff72-13e8-75a2-5207-0d15c7c024a5 Address:2506ff72-13e8-75a2-5207-0d15c7c024a5} {Suffrage:Voter ID:server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e Address:e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e}]" 2025-09-14T21:48:56.943Z [INFO] server-f931fe05-4494-8775-5a00-7b70ba8ae226: entering follower state: follower="Node at f931fe05-4494-8775-5a00-7b70ba8ae226 [Follower]" leader-address= leader-id= 2025-09-14T21:48:56.943Z [INFO] server-2506ff72-13e8-75a2-5207-0d15c7c024a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f931fe05-4494-8775-5a00-7b70ba8ae226 Address:f931fe05-4494-8775-5a00-7b70ba8ae226} {Suffrage:Voter ID:server-2506ff72-13e8-75a2-5207-0d15c7c024a5 Address:2506ff72-13e8-75a2-5207-0d15c7c024a5} {Suffrage:Voter ID:server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e Address:e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e}]" 2025-09-14T21:48:56.943Z [INFO] server-2506ff72-13e8-75a2-5207-0d15c7c024a5: entering follower state: follower="Node at 2506ff72-13e8-75a2-5207-0d15c7c024a5 [Follower]" leader-address= leader-id= 2025-09-14T21:48:56.943Z [INFO] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f931fe05-4494-8775-5a00-7b70ba8ae226 Address:f931fe05-4494-8775-5a00-7b70ba8ae226} {Suffrage:Voter ID:server-2506ff72-13e8-75a2-5207-0d15c7c024a5 Address:2506ff72-13e8-75a2-5207-0d15c7c024a5} {Suffrage:Voter ID:server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e Address:e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e}]" 2025-09-14T21:48:56.943Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:56.943Z [INFO] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: entering follower state: follower="Node at e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e [Follower]" leader-address= leader-id= 2025-09-14T21:48:57.009Z [WARN] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:57.009Z [INFO] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: entering candidate state: node="Node at e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e [Candidate]" term=2 2025-09-14T21:48:57.009Z [INFO] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: election won: term=2 tally=2 2025-09-14T21:48:57.009Z [INFO] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: entering leader state: leader="Node at e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e [Leader]" 2025-09-14T21:48:57.009Z [INFO] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: added peer, starting replication: peer=server-f931fe05-4494-8775-5a00-7b70ba8ae226 2025-09-14T21:48:57.009Z [INFO] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: added peer, starting replication: peer=server-2506ff72-13e8-75a2-5207-0d15c7c024a5 2025-09-14T21:48:57.009Z [INFO] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: pipelining replication: peer="{Voter server-2506ff72-13e8-75a2-5207-0d15c7c024a5 2506ff72-13e8-75a2-5207-0d15c7c024a5}" 2025-09-14T21:48:57.009Z [INFO] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: pipelining replication: peer="{Voter server-f931fe05-4494-8775-5a00-7b70ba8ae226 f931fe05-4494-8775-5a00-7b70ba8ae226}" 2025-09-14T21:48:57.115Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:57.009412681 +0000 UTC m=+107.577290942 (1 nodes), 65.850546ms from start of poll, 66.147383ms from cluster start. Timeout at 2025-09-14 21:48:57.114413922 +0000 UTC m=+107.682292201, 105.001259ms after stability 2025-09-14T21:48:57.121Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:57.227Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:57.121312375 +0000 UTC m=+107.689190646 (2 nodes), 2.528µs from start of poll, 178.047087ms from cluster start. Timeout at 2025-09-14 21:48:57.226346171 +0000 UTC m=+107.794224461, 105.033815ms after stability 2025-09-14T21:48:57.227Z [INFO] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: updating configuration: command=RemoveServer server-id=server-f931fe05-4494-8775-5a00-7b70ba8ae226 server-addr= servers="[{Suffrage:Voter ID:server-2506ff72-13e8-75a2-5207-0d15c7c024a5 Address:2506ff72-13e8-75a2-5207-0d15c7c024a5} {Suffrage:Voter ID:server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e Address:e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e}]" 2025-09-14T21:48:57.227Z [INFO] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: removed peer, stopping replication: peer=server-f931fe05-4494-8775-5a00-7b70ba8ae226 last-index=3 2025-09-14T21:48:57.227Z [INFO] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: aborting pipeline replication: peer="{Voter server-f931fe05-4494-8775-5a00-7b70ba8ae226 f931fe05-4494-8775-5a00-7b70ba8ae226}" 2025-09-14T21:48:57.340Z [ERROR] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: failed to pipeline appendEntries: peer="{Voter server-2506ff72-13e8-75a2-5207-0d15c7c024a5 2506ff72-13e8-75a2-5207-0d15c7c024a5}" error="append pipeline closed" 2025-09-14T21:48:57.340Z [INFO] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: aborting pipeline replication: peer="{Voter server-2506ff72-13e8-75a2-5207-0d15c7c024a5 2506ff72-13e8-75a2-5207-0d15c7c024a5}" 2025-09-14T21:48:57.341Z [ERROR] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: failed to heartbeat to: peer=2506ff72-13e8-75a2-5207-0d15c7c024a5 backoff time=10ms error="failed to connect to peer: 2506ff72-13e8-75a2-5207-0d15c7c024a5" 2025-09-14T21:48:57.348Z [ERROR] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: failed to appendEntries to: peer="{Voter server-2506ff72-13e8-75a2-5207-0d15c7c024a5 2506ff72-13e8-75a2-5207-0d15c7c024a5}" error="failed to connect to peer: 2506ff72-13e8-75a2-5207-0d15c7c024a5" 2025-09-14T21:48:57.358Z [WARN] server-f931fe05-4494-8775-5a00-7b70ba8ae226: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2025-09-14T21:48:57.362Z [ERROR] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: failed to heartbeat to: peer=2506ff72-13e8-75a2-5207-0d15c7c024a5 backoff time=10ms error="failed to connect to peer: 2506ff72-13e8-75a2-5207-0d15c7c024a5" 2025-09-14T21:48:57.367Z [ERROR] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: failed to appendEntries to: peer="{Voter server-2506ff72-13e8-75a2-5207-0d15c7c024a5 2506ff72-13e8-75a2-5207-0d15c7c024a5}" error="failed to connect to peer: 2506ff72-13e8-75a2-5207-0d15c7c024a5" 2025-09-14T21:48:57.382Z [ERROR] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: failed to heartbeat to: peer=2506ff72-13e8-75a2-5207-0d15c7c024a5 backoff time=10ms error="failed to connect to peer: 2506ff72-13e8-75a2-5207-0d15c7c024a5" 2025-09-14T21:48:57.384Z [WARN] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: failed to contact: server-id=server-2506ff72-13e8-75a2-5207-0d15c7c024a5 time=50.630126ms 2025-09-14T21:48:57.384Z [WARN] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: failed to contact quorum of nodes, stepping down 2025-09-14T21:48:57.384Z [INFO] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: entering follower state: follower="Node at e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e [Follower]" leader-address= leader-id= 2025-09-14T21:48:57.388Z [ERROR] server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e: failed to appendEntries to: peer="{Voter server-2506ff72-13e8-75a2-5207-0d15c7c024a5 2506ff72-13e8-75a2-5207-0d15c7c024a5}" error="failed to connect to peer: 2506ff72-13e8-75a2-5207-0d15c7c024a5" 2025-09-14T21:48:57.406Z [WARN] server-2506ff72-13e8-75a2-5207-0d15c7c024a5: heartbeat timeout reached, starting election: last-leader-addr=e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e last-leader-id=server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e 2025-09-14T21:48:57.406Z [INFO] server-2506ff72-13e8-75a2-5207-0d15c7c024a5: entering candidate state: node="Node at 2506ff72-13e8-75a2-5207-0d15c7c024a5 [Candidate]" term=3 2025-09-14T21:48:57.406Z [ERROR] server-2506ff72-13e8-75a2-5207-0d15c7c024a5: failed to make requestVote RPC: target="{Voter server-e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e}" error="failed to connect to peer: e9f3f613-d2ca-eed2-78d6-9e7bb5833e3e" term=3 2025-09-14T21:48:57.439Z [WARN] server-2506ff72-13e8-75a2-5207-0d15c7c024a5: rejecting vote request since node is not in configuration: from=f931fe05-4494-8775-5a00-7b70ba8ae226 --- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.51s) === RUN TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin 2025-09-14T21:48:57.464Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7a2ff82b-903b-6ead-7d69-42500edba3b3 Address:7a2ff82b-903b-6ead-7d69-42500edba3b3} {Suffrage:Voter ID:server-a7d07c55-5063-ce37-ea80-32984534d608 Address:a7d07c55-5063-ce37-ea80-32984534d608} {Suffrage:Voter ID:server-42e37652-ce43-152c-0da0-5e926e8604b8 Address:42e37652-ce43-152c-0da0-5e926e8604b8}]" 2025-09-14T21:48:57.464Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: entering follower state: follower="Node at 7a2ff82b-903b-6ead-7d69-42500edba3b3 [Follower]" leader-address= leader-id= 2025-09-14T21:48:57.464Z [INFO] server-a7d07c55-5063-ce37-ea80-32984534d608: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7a2ff82b-903b-6ead-7d69-42500edba3b3 Address:7a2ff82b-903b-6ead-7d69-42500edba3b3} {Suffrage:Voter ID:server-a7d07c55-5063-ce37-ea80-32984534d608 Address:a7d07c55-5063-ce37-ea80-32984534d608} {Suffrage:Voter ID:server-42e37652-ce43-152c-0da0-5e926e8604b8 Address:42e37652-ce43-152c-0da0-5e926e8604b8}]" 2025-09-14T21:48:57.464Z [INFO] server-a7d07c55-5063-ce37-ea80-32984534d608: entering follower state: follower="Node at a7d07c55-5063-ce37-ea80-32984534d608 [Follower]" leader-address= leader-id= 2025-09-14T21:48:57.464Z [INFO] server-42e37652-ce43-152c-0da0-5e926e8604b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7a2ff82b-903b-6ead-7d69-42500edba3b3 Address:7a2ff82b-903b-6ead-7d69-42500edba3b3} {Suffrage:Voter ID:server-a7d07c55-5063-ce37-ea80-32984534d608 Address:a7d07c55-5063-ce37-ea80-32984534d608} {Suffrage:Voter ID:server-42e37652-ce43-152c-0da0-5e926e8604b8 Address:42e37652-ce43-152c-0da0-5e926e8604b8}]" 2025-09-14T21:48:57.464Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:57.464Z [INFO] server-42e37652-ce43-152c-0da0-5e926e8604b8: entering follower state: follower="Node at 42e37652-ce43-152c-0da0-5e926e8604b8 [Follower]" leader-address= leader-id= 2025-09-14T21:48:57.523Z [WARN] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:57.523Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: entering candidate state: node="Node at 7a2ff82b-903b-6ead-7d69-42500edba3b3 [Candidate]" term=2 2025-09-14T21:48:57.524Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: election won: term=2 tally=2 2025-09-14T21:48:57.524Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: entering leader state: leader="Node at 7a2ff82b-903b-6ead-7d69-42500edba3b3 [Leader]" 2025-09-14T21:48:57.524Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: added peer, starting replication: peer=server-a7d07c55-5063-ce37-ea80-32984534d608 2025-09-14T21:48:57.524Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: added peer, starting replication: peer=server-42e37652-ce43-152c-0da0-5e926e8604b8 2025-09-14T21:48:57.524Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: pipelining replication: peer="{Voter server-a7d07c55-5063-ce37-ea80-32984534d608 a7d07c55-5063-ce37-ea80-32984534d608}" 2025-09-14T21:48:57.524Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: pipelining replication: peer="{Voter server-42e37652-ce43-152c-0da0-5e926e8604b8 42e37652-ce43-152c-0da0-5e926e8604b8}" 2025-09-14T21:48:57.630Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:57.524560502 +0000 UTC m=+108.092438796 (1 nodes), 59.896275ms from start of poll, 60.231698ms from cluster start. Timeout at 2025-09-14 21:48:57.629561722 +0000 UTC m=+108.197439981, 105.001185ms after stability 2025-09-14T21:48:57.635Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:48:57.741Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:57.635914135 +0000 UTC m=+108.203792415 (2 nodes), 3.165µs from start of poll, 171.585317ms from cluster start. Timeout at 2025-09-14 21:48:57.74093518 +0000 UTC m=+108.308813461, 105.021046ms after stability 2025-09-14T21:48:57.743Z [ERROR] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: failed to pipeline appendEntries: peer="{Voter server-a7d07c55-5063-ce37-ea80-32984534d608 a7d07c55-5063-ce37-ea80-32984534d608}" error="append pipeline closed" 2025-09-14T21:48:57.743Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: aborting pipeline replication: peer="{Voter server-a7d07c55-5063-ce37-ea80-32984534d608 a7d07c55-5063-ce37-ea80-32984534d608}" 2025-09-14T21:48:57.745Z [ERROR] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: failed to heartbeat to: peer=a7d07c55-5063-ce37-ea80-32984534d608 backoff time=10ms error="failed to connect to peer: a7d07c55-5063-ce37-ea80-32984534d608" 2025-09-14T21:48:57.754Z [ERROR] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: failed to appendEntries to: peer="{Voter server-a7d07c55-5063-ce37-ea80-32984534d608 a7d07c55-5063-ce37-ea80-32984534d608}" error="failed to connect to peer: a7d07c55-5063-ce37-ea80-32984534d608" 2025-09-14T21:48:57.761Z [ERROR] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: failed to heartbeat to: peer=a7d07c55-5063-ce37-ea80-32984534d608 backoff time=10ms error="failed to connect to peer: a7d07c55-5063-ce37-ea80-32984534d608" 2025-09-14T21:48:57.774Z [ERROR] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: failed to appendEntries to: peer="{Voter server-a7d07c55-5063-ce37-ea80-32984534d608 a7d07c55-5063-ce37-ea80-32984534d608}" error="failed to connect to peer: a7d07c55-5063-ce37-ea80-32984534d608" 2025-09-14T21:48:57.781Z [ERROR] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: failed to heartbeat to: peer=a7d07c55-5063-ce37-ea80-32984534d608 backoff time=10ms error="failed to connect to peer: a7d07c55-5063-ce37-ea80-32984534d608" 2025-09-14T21:48:57.785Z [WARN] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: failed to contact: server-id=server-a7d07c55-5063-ce37-ea80-32984534d608 time=50.684011ms 2025-09-14T21:48:57.794Z [ERROR] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: failed to appendEntries to: peer="{Voter server-a7d07c55-5063-ce37-ea80-32984534d608 a7d07c55-5063-ce37-ea80-32984534d608}" error="failed to connect to peer: a7d07c55-5063-ce37-ea80-32984534d608" 2025-09-14T21:48:57.801Z [ERROR] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: failed to heartbeat to: peer=a7d07c55-5063-ce37-ea80-32984534d608 backoff time=20ms error="failed to connect to peer: a7d07c55-5063-ce37-ea80-32984534d608" 2025-09-14T21:48:57.803Z [WARN] server-a7d07c55-5063-ce37-ea80-32984534d608: heartbeat timeout reached, starting election: last-leader-addr=7a2ff82b-903b-6ead-7d69-42500edba3b3 last-leader-id=server-7a2ff82b-903b-6ead-7d69-42500edba3b3 2025-09-14T21:48:57.803Z [INFO] server-a7d07c55-5063-ce37-ea80-32984534d608: entering candidate state: node="Node at a7d07c55-5063-ce37-ea80-32984534d608 [Candidate]" term=3 2025-09-14T21:48:57.803Z [ERROR] server-a7d07c55-5063-ce37-ea80-32984534d608: failed to make requestVote RPC: target="{Voter server-42e37652-ce43-152c-0da0-5e926e8604b8 42e37652-ce43-152c-0da0-5e926e8604b8}" error="failed to connect to peer: 42e37652-ce43-152c-0da0-5e926e8604b8" term=3 2025-09-14T21:48:57.803Z [ERROR] server-a7d07c55-5063-ce37-ea80-32984534d608: failed to make requestVote RPC: target="{Voter server-7a2ff82b-903b-6ead-7d69-42500edba3b3 7a2ff82b-903b-6ead-7d69-42500edba3b3}" error="failed to connect to peer: 7a2ff82b-903b-6ead-7d69-42500edba3b3" term=3 2025-09-14T21:48:57.822Z [ERROR] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: failed to appendEntries to: peer="{Voter server-a7d07c55-5063-ce37-ea80-32984534d608 a7d07c55-5063-ce37-ea80-32984534d608}" error="failed to connect to peer: a7d07c55-5063-ce37-ea80-32984534d608" 2025-09-14T21:48:57.830Z [ERROR] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: failed to heartbeat to: peer=a7d07c55-5063-ce37-ea80-32984534d608 backoff time=25ms error="failed to connect to peer: a7d07c55-5063-ce37-ea80-32984534d608" 2025-09-14T21:48:57.831Z [WARN] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: failed to contact: server-id=server-a7d07c55-5063-ce37-ea80-32984534d608 time=96.686804ms 2025-09-14T21:48:57.847Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: updating configuration: command=RemoveServer server-id=server-a7d07c55-5063-ce37-ea80-32984534d608 server-addr= servers="[{Suffrage:Voter ID:server-7a2ff82b-903b-6ead-7d69-42500edba3b3 Address:7a2ff82b-903b-6ead-7d69-42500edba3b3} {Suffrage:Voter ID:server-42e37652-ce43-152c-0da0-5e926e8604b8 Address:42e37652-ce43-152c-0da0-5e926e8604b8}]" 2025-09-14T21:48:57.847Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: removed peer, stopping replication: peer=server-a7d07c55-5063-ce37-ea80-32984534d608 last-index=3 2025-09-14T21:48:57.847Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: updating configuration: command=AddNonvoter server-id=server-a7d07c55-5063-ce37-ea80-32984534d608 server-addr=a7d07c55-5063-ce37-ea80-32984534d608 servers="[{Suffrage:Voter ID:server-7a2ff82b-903b-6ead-7d69-42500edba3b3 Address:7a2ff82b-903b-6ead-7d69-42500edba3b3} {Suffrage:Voter ID:server-42e37652-ce43-152c-0da0-5e926e8604b8 Address:42e37652-ce43-152c-0da0-5e926e8604b8} {Suffrage:Nonvoter ID:server-a7d07c55-5063-ce37-ea80-32984534d608 Address:a7d07c55-5063-ce37-ea80-32984534d608}]" 2025-09-14T21:48:57.847Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: added peer, starting replication: peer=server-a7d07c55-5063-ce37-ea80-32984534d608 2025-09-14T21:48:57.847Z [ERROR] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: failed to appendEntries to: peer="{Nonvoter server-a7d07c55-5063-ce37-ea80-32984534d608 a7d07c55-5063-ce37-ea80-32984534d608}" error="failed to connect to peer: a7d07c55-5063-ce37-ea80-32984534d608" 2025-09-14T21:48:57.861Z [WARN] server-a7d07c55-5063-ce37-ea80-32984534d608: Election timeout reached, restarting election 2025-09-14T21:48:57.861Z [INFO] server-a7d07c55-5063-ce37-ea80-32984534d608: entering candidate state: node="Node at a7d07c55-5063-ce37-ea80-32984534d608 [Candidate]" term=23 2025-09-14T21:48:57.861Z [WARN] server-42e37652-ce43-152c-0da0-5e926e8604b8: rejecting vote request since we have a leader: from=a7d07c55-5063-ce37-ea80-32984534d608 leader=7a2ff82b-903b-6ead-7d69-42500edba3b3 leader-id=server-7a2ff82b-903b-6ead-7d69-42500edba3b3 2025-09-14T21:48:57.861Z [WARN] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: rejecting vote request since we have a leader: from=a7d07c55-5063-ce37-ea80-32984534d608 leader=7a2ff82b-903b-6ead-7d69-42500edba3b3 leader-id=server-7a2ff82b-903b-6ead-7d69-42500edba3b3 2025-09-14T21:48:57.867Z [ERROR] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: peer has newer term, stopping replication: peer="{Nonvoter server-a7d07c55-5063-ce37-ea80-32984534d608 a7d07c55-5063-ce37-ea80-32984534d608}" 2025-09-14T21:48:57.867Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: entering follower state: follower="Node at 7a2ff82b-903b-6ead-7d69-42500edba3b3 [Follower]" leader-address= leader-id= 2025-09-14T21:48:57.867Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: aborting pipeline replication: peer="{Voter server-42e37652-ce43-152c-0da0-5e926e8604b8 42e37652-ce43-152c-0da0-5e926e8604b8}" 2025-09-14T21:48:57.870Z [ERROR] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: peer has newer term, stopping replication: peer="{Voter server-a7d07c55-5063-ce37-ea80-32984534d608 a7d07c55-5063-ce37-ea80-32984534d608}" 2025-09-14T21:48:57.934Z [WARN] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:57.934Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: entering candidate state: node="Node at 7a2ff82b-903b-6ead-7d69-42500edba3b3 [Candidate]" term=3 2025-09-14T21:48:57.934Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: election won: term=3 tally=2 2025-09-14T21:48:57.934Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: entering leader state: leader="Node at 7a2ff82b-903b-6ead-7d69-42500edba3b3 [Leader]" 2025-09-14T21:48:57.934Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: added peer, starting replication: peer=server-42e37652-ce43-152c-0da0-5e926e8604b8 2025-09-14T21:48:57.935Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: added peer, starting replication: peer=server-a7d07c55-5063-ce37-ea80-32984534d608 2025-09-14T21:48:57.935Z [ERROR] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: peer has newer term, stopping replication: peer="{Nonvoter server-a7d07c55-5063-ce37-ea80-32984534d608 a7d07c55-5063-ce37-ea80-32984534d608}" 2025-09-14T21:48:57.935Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: pipelining replication: peer="{Voter server-42e37652-ce43-152c-0da0-5e926e8604b8 42e37652-ce43-152c-0da0-5e926e8604b8}" 2025-09-14T21:48:57.935Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: aborting pipeline replication: peer="{Voter server-42e37652-ce43-152c-0da0-5e926e8604b8 42e37652-ce43-152c-0da0-5e926e8604b8}" 2025-09-14T21:48:57.935Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: entering follower state: follower="Node at 7a2ff82b-903b-6ead-7d69-42500edba3b3 [Follower]" leader-address= leader-id= 2025-09-14T21:48:57.953Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:57.953Z [WARN] server-a7d07c55-5063-ce37-ea80-32984534d608: Election timeout reached, restarting election 2025-09-14T21:48:57.953Z [INFO] server-a7d07c55-5063-ce37-ea80-32984534d608: entering candidate state: node="Node at a7d07c55-5063-ce37-ea80-32984534d608 [Candidate]" term=24 2025-09-14T21:48:57.953Z [WARN] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: rejecting vote request since node is not a voter: from=a7d07c55-5063-ce37-ea80-32984534d608 2025-09-14T21:48:57.953Z [WARN] server-42e37652-ce43-152c-0da0-5e926e8604b8: rejecting vote request since we have a leader: from=a7d07c55-5063-ce37-ea80-32984534d608 leader=7a2ff82b-903b-6ead-7d69-42500edba3b3 leader-id=server-7a2ff82b-903b-6ead-7d69-42500edba3b3 2025-09-14T21:48:57.994Z [WARN] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:57.994Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: entering candidate state: node="Node at 7a2ff82b-903b-6ead-7d69-42500edba3b3 [Candidate]" term=25 2025-09-14T21:48:57.994Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: election won: term=25 tally=2 2025-09-14T21:48:57.994Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: entering leader state: leader="Node at 7a2ff82b-903b-6ead-7d69-42500edba3b3 [Leader]" 2025-09-14T21:48:57.994Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: added peer, starting replication: peer=server-42e37652-ce43-152c-0da0-5e926e8604b8 2025-09-14T21:48:57.994Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: added peer, starting replication: peer=server-a7d07c55-5063-ce37-ea80-32984534d608 2025-09-14T21:48:57.995Z [WARN] server-a7d07c55-5063-ce37-ea80-32984534d608: failed to get previous log: previous-index=5 last-index=2 error="log not found" 2025-09-14T21:48:57.995Z [INFO] server-a7d07c55-5063-ce37-ea80-32984534d608: entering follower state: follower="Node at a7d07c55-5063-ce37-ea80-32984534d608 [Follower]" leader-address=7a2ff82b-903b-6ead-7d69-42500edba3b3 leader-id=server-7a2ff82b-903b-6ead-7d69-42500edba3b3 2025-09-14T21:48:57.995Z [WARN] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: appendEntries rejected, sending older logs: peer="{Nonvoter server-a7d07c55-5063-ce37-ea80-32984534d608 a7d07c55-5063-ce37-ea80-32984534d608}" next=3 2025-09-14T21:48:57.995Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: pipelining replication: peer="{Voter server-42e37652-ce43-152c-0da0-5e926e8604b8 42e37652-ce43-152c-0da0-5e926e8604b8}" 2025-09-14T21:48:57.995Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: pipelining replication: peer="{Nonvoter server-a7d07c55-5063-ce37-ea80-32984534d608 a7d07c55-5063-ce37-ea80-32984534d608}" 2025-09-14T21:48:58.100Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:57.995065396 +0000 UTC m=+108.562943671 (1 nodes), 41.657156ms from start of poll, 530.736573ms from cluster start. Timeout at 2025-09-14 21:48:58.100071382 +0000 UTC m=+108.667949645, 105.005974ms after stability 2025-09-14T21:48:58.120Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: updating configuration: command=RemoveServer server-id=server-a7d07c55-5063-ce37-ea80-32984534d608 server-addr= servers="[{Suffrage:Voter ID:server-7a2ff82b-903b-6ead-7d69-42500edba3b3 Address:7a2ff82b-903b-6ead-7d69-42500edba3b3} {Suffrage:Voter ID:server-42e37652-ce43-152c-0da0-5e926e8604b8 Address:42e37652-ce43-152c-0da0-5e926e8604b8}]" 2025-09-14T21:48:58.120Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: removed peer, stopping replication: peer=server-a7d07c55-5063-ce37-ea80-32984534d608 last-index=107 2025-09-14T21:48:58.120Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: aborting pipeline replication: peer="{Nonvoter server-a7d07c55-5063-ce37-ea80-32984534d608 a7d07c55-5063-ce37-ea80-32984534d608}" 2025-09-14T21:48:58.121Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: updating configuration: command=AddVoter server-id=server-a7d07c55-5063-ce37-ea80-32984534d608 server-addr=a7d07c55-5063-ce37-ea80-32984534d608 servers="[{Suffrage:Voter ID:server-7a2ff82b-903b-6ead-7d69-42500edba3b3 Address:7a2ff82b-903b-6ead-7d69-42500edba3b3} {Suffrage:Voter ID:server-42e37652-ce43-152c-0da0-5e926e8604b8 Address:42e37652-ce43-152c-0da0-5e926e8604b8} {Suffrage:Voter ID:server-a7d07c55-5063-ce37-ea80-32984534d608 Address:a7d07c55-5063-ce37-ea80-32984534d608}]" 2025-09-14T21:48:58.121Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: added peer, starting replication: peer=server-a7d07c55-5063-ce37-ea80-32984534d608 2025-09-14T21:48:58.121Z [WARN] server-a7d07c55-5063-ce37-ea80-32984534d608: failed to get previous log: previous-index=108 last-index=107 error="log not found" 2025-09-14T21:48:58.121Z [WARN] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: appendEntries rejected, sending older logs: peer="{Voter server-a7d07c55-5063-ce37-ea80-32984534d608 a7d07c55-5063-ce37-ea80-32984534d608}" next=108 2025-09-14T21:48:58.121Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: pipelining replication: peer="{Voter server-a7d07c55-5063-ce37-ea80-32984534d608 a7d07c55-5063-ce37-ea80-32984534d608}" 2025-09-14T21:48:59.189Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:59.295Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:59.189235454 +0000 UTC m=+109.757113715 (1 nodes), 759ns from start of poll, 1.724906617s from cluster start. Timeout at 2025-09-14 21:48:59.294259612 +0000 UTC m=+109.862137913, 105.024198ms after stability 2025-09-14T21:48:59.295Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: aborting pipeline replication: peer="{Voter server-a7d07c55-5063-ce37-ea80-32984534d608 a7d07c55-5063-ce37-ea80-32984534d608}" 2025-09-14T21:48:59.295Z [INFO] server-7a2ff82b-903b-6ead-7d69-42500edba3b3: aborting pipeline replication: peer="{Voter server-42e37652-ce43-152c-0da0-5e926e8604b8 42e37652-ce43-152c-0da0-5e926e8604b8}" --- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.85s) === RUN TestRaft_FollowerRemovalNoElection 2025-09-14T21:48:59.296Z [INFO] server-672c1920-9d5d-7290-fa0f-7ca59d79fdab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-672c1920-9d5d-7290-fa0f-7ca59d79fdab Address:672c1920-9d5d-7290-fa0f-7ca59d79fdab} {Suffrage:Voter ID:server-7b447be6-5883-7b77-1cfc-367803da99a3 Address:7b447be6-5883-7b77-1cfc-367803da99a3} {Suffrage:Voter ID:server-36321267-59f0-e755-ef5d-9294bb9c40a3 Address:36321267-59f0-e755-ef5d-9294bb9c40a3}]" 2025-09-14T21:48:59.296Z [INFO] server-672c1920-9d5d-7290-fa0f-7ca59d79fdab: entering follower state: follower="Node at 672c1920-9d5d-7290-fa0f-7ca59d79fdab [Follower]" leader-address= leader-id= 2025-09-14T21:48:59.296Z [INFO] server-7b447be6-5883-7b77-1cfc-367803da99a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-672c1920-9d5d-7290-fa0f-7ca59d79fdab Address:672c1920-9d5d-7290-fa0f-7ca59d79fdab} {Suffrage:Voter ID:server-7b447be6-5883-7b77-1cfc-367803da99a3 Address:7b447be6-5883-7b77-1cfc-367803da99a3} {Suffrage:Voter ID:server-36321267-59f0-e755-ef5d-9294bb9c40a3 Address:36321267-59f0-e755-ef5d-9294bb9c40a3}]" 2025-09-14T21:48:59.296Z [INFO] server-7b447be6-5883-7b77-1cfc-367803da99a3: entering follower state: follower="Node at 7b447be6-5883-7b77-1cfc-367803da99a3 [Follower]" leader-address= leader-id= 2025-09-14T21:48:59.296Z [INFO] server-36321267-59f0-e755-ef5d-9294bb9c40a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-672c1920-9d5d-7290-fa0f-7ca59d79fdab Address:672c1920-9d5d-7290-fa0f-7ca59d79fdab} {Suffrage:Voter ID:server-7b447be6-5883-7b77-1cfc-367803da99a3 Address:7b447be6-5883-7b77-1cfc-367803da99a3} {Suffrage:Voter ID:server-36321267-59f0-e755-ef5d-9294bb9c40a3 Address:36321267-59f0-e755-ef5d-9294bb9c40a3}]" 2025-09-14T21:48:59.296Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:59.296Z [INFO] server-36321267-59f0-e755-ef5d-9294bb9c40a3: entering follower state: follower="Node at 36321267-59f0-e755-ef5d-9294bb9c40a3 [Follower]" leader-address= leader-id= 2025-09-14T21:48:59.398Z [WARN] server-36321267-59f0-e755-ef5d-9294bb9c40a3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:48:59.398Z [INFO] server-36321267-59f0-e755-ef5d-9294bb9c40a3: entering candidate state: node="Node at 36321267-59f0-e755-ef5d-9294bb9c40a3 [Candidate]" term=2 2025-09-14T21:48:59.398Z [INFO] server-36321267-59f0-e755-ef5d-9294bb9c40a3: election won: term=2 tally=2 2025-09-14T21:48:59.398Z [INFO] server-36321267-59f0-e755-ef5d-9294bb9c40a3: entering leader state: leader="Node at 36321267-59f0-e755-ef5d-9294bb9c40a3 [Leader]" 2025-09-14T21:48:59.398Z [INFO] server-36321267-59f0-e755-ef5d-9294bb9c40a3: added peer, starting replication: peer=server-672c1920-9d5d-7290-fa0f-7ca59d79fdab 2025-09-14T21:48:59.398Z [INFO] server-36321267-59f0-e755-ef5d-9294bb9c40a3: added peer, starting replication: peer=server-7b447be6-5883-7b77-1cfc-367803da99a3 2025-09-14T21:48:59.398Z [INFO] server-36321267-59f0-e755-ef5d-9294bb9c40a3: pipelining replication: peer="{Voter server-672c1920-9d5d-7290-fa0f-7ca59d79fdab 672c1920-9d5d-7290-fa0f-7ca59d79fdab}" 2025-09-14T21:48:59.398Z [INFO] server-36321267-59f0-e755-ef5d-9294bb9c40a3: pipelining replication: peer="{Voter server-7b447be6-5883-7b77-1cfc-367803da99a3 7b447be6-5883-7b77-1cfc-367803da99a3}" 2025-09-14T21:48:59.604Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:59.398381631 +0000 UTC m=+109.966259906 (1 nodes), 101.833617ms from start of poll, 102.323603ms from cluster start. Timeout at 2025-09-14 21:48:59.603391413 +0000 UTC m=+110.171269688, 205.009782ms after stability 2025-09-14T21:48:59.604Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:48:59.809Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:48:59.604634494 +0000 UTC m=+110.172512776 (1 nodes), 3.196µs from start of poll, 308.576473ms from cluster start. Timeout at 2025-09-14 21:48:59.809674406 +0000 UTC m=+110.377552668, 205.039892ms after stability 2025-09-14T21:48:59.815Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:49:00.021Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:48:59.815689894 +0000 UTC m=+110.383568160 (2 nodes), 1.982µs from start of poll, 519.631857ms from cluster start. Timeout at 2025-09-14 21:49:00.020719614 +0000 UTC m=+110.588597923, 205.029763ms after stability 2025-09-14T21:49:00.021Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:49:00.227Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:49:00.021614763 +0000 UTC m=+110.589493025 (1 nodes), 1.655µs from start of poll, 725.556722ms from cluster start. Timeout at 2025-09-14 21:49:00.226630474 +0000 UTC m=+110.794508755, 205.01573ms after stability raft_test.go:2961: [INFO] restarting Node at 672c1920-9d5d-7290-fa0f-7ca59d79fdab [Follower] 2025-09-14T21:49:00.227Z [INFO] server-672c1920-9d5d-7290-fa0f-7ca59d79fdab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-672c1920-9d5d-7290-fa0f-7ca59d79fdab Address:672c1920-9d5d-7290-fa0f-7ca59d79fdab} {Suffrage:Voter ID:server-7b447be6-5883-7b77-1cfc-367803da99a3 Address:7b447be6-5883-7b77-1cfc-367803da99a3} {Suffrage:Voter ID:server-36321267-59f0-e755-ef5d-9294bb9c40a3 Address:36321267-59f0-e755-ef5d-9294bb9c40a3}]" 2025-09-14T21:49:00.227Z [INFO] server-672c1920-9d5d-7290-fa0f-7ca59d79fdab: entering follower state: follower="Node at 672c1920-9d5d-7290-fa0f-7ca59d79fdab [Follower]" leader-address= leader-id= 2025-09-14T21:49:00.477Z [INFO] server-36321267-59f0-e755-ef5d-9294bb9c40a3: aborting pipeline replication: peer="{Voter server-7b447be6-5883-7b77-1cfc-367803da99a3 7b447be6-5883-7b77-1cfc-367803da99a3}" 2025-09-14T21:49:00.859Z [ERROR] server-36321267-59f0-e755-ef5d-9294bb9c40a3: failed to pipeline appendEntries: peer="{Voter server-672c1920-9d5d-7290-fa0f-7ca59d79fdab 672c1920-9d5d-7290-fa0f-7ca59d79fdab}" error="command enqueue timeout" 2025-09-14T21:49:00.859Z [INFO] server-36321267-59f0-e755-ef5d-9294bb9c40a3: aborting pipeline replication: peer="{Voter server-672c1920-9d5d-7290-fa0f-7ca59d79fdab 672c1920-9d5d-7290-fa0f-7ca59d79fdab}" 2025-09-14T21:49:00.991Z [ERROR] server-36321267-59f0-e755-ef5d-9294bb9c40a3: failed to heartbeat to: peer=672c1920-9d5d-7290-fa0f-7ca59d79fdab backoff time=10ms error="command timed out" --- PASS: TestRaft_FollowerRemovalNoElection (1.70s) === RUN TestRaft_VoteWithNoIDNoAddr 2025-09-14T21:49:00.993Z [INFO] server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 Address:d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2} {Suffrage:Voter ID:server-ee65bab0-a64b-c4d9-0be7-42e198beaf09 Address:ee65bab0-a64b-c4d9-0be7-42e198beaf09} {Suffrage:Voter ID:server-70a153e8-5cdb-9d89-d720-d2fa666640c9 Address:70a153e8-5cdb-9d89-d720-d2fa666640c9}]" 2025-09-14T21:49:00.993Z [INFO] server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2: entering follower state: follower="Node at d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 [Follower]" leader-address= leader-id= 2025-09-14T21:49:00.993Z [INFO] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 Address:d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2} {Suffrage:Voter ID:server-ee65bab0-a64b-c4d9-0be7-42e198beaf09 Address:ee65bab0-a64b-c4d9-0be7-42e198beaf09} {Suffrage:Voter ID:server-70a153e8-5cdb-9d89-d720-d2fa666640c9 Address:70a153e8-5cdb-9d89-d720-d2fa666640c9}]" 2025-09-14T21:49:00.994Z [INFO] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: entering follower state: follower="Node at ee65bab0-a64b-c4d9-0be7-42e198beaf09 [Follower]" leader-address= leader-id= 2025-09-14T21:49:00.994Z [INFO] server-70a153e8-5cdb-9d89-d720-d2fa666640c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 Address:d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2} {Suffrage:Voter ID:server-ee65bab0-a64b-c4d9-0be7-42e198beaf09 Address:ee65bab0-a64b-c4d9-0be7-42e198beaf09} {Suffrage:Voter ID:server-70a153e8-5cdb-9d89-d720-d2fa666640c9 Address:70a153e8-5cdb-9d89-d720-d2fa666640c9}]" 2025-09-14T21:49:00.994Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:49:00.994Z [INFO] server-70a153e8-5cdb-9d89-d720-d2fa666640c9: entering follower state: follower="Node at 70a153e8-5cdb-9d89-d720-d2fa666640c9 [Follower]" leader-address= leader-id= 2025-09-14T21:49:01.047Z [WARN] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-09-14T21:49:01.047Z [INFO] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: entering candidate state: node="Node at ee65bab0-a64b-c4d9-0be7-42e198beaf09 [Candidate]" term=2 2025-09-14T21:49:01.047Z [INFO] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: election won: term=2 tally=2 2025-09-14T21:49:01.047Z [INFO] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: entering leader state: leader="Node at ee65bab0-a64b-c4d9-0be7-42e198beaf09 [Leader]" 2025-09-14T21:49:01.047Z [INFO] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: added peer, starting replication: peer=server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 2025-09-14T21:49:01.047Z [INFO] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: added peer, starting replication: peer=server-70a153e8-5cdb-9d89-d720-d2fa666640c9 2025-09-14T21:49:01.047Z [INFO] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: pipelining replication: peer="{Voter server-70a153e8-5cdb-9d89-d720-d2fa666640c9 70a153e8-5cdb-9d89-d720-d2fa666640c9}" 2025-09-14T21:49:01.047Z [INFO] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: pipelining replication: peer="{Voter server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2}" 2025-09-14T21:49:01.153Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:49:01.047398584 +0000 UTC m=+111.615276859 (1 nodes), 53.185314ms from start of poll, 54.567989ms from cluster start. Timeout at 2025-09-14 21:49:01.152402139 +0000 UTC m=+111.720280415, 105.003556ms after stability 2025-09-14T21:49:01.153Z [INFO] cluster: starting stability test: raft-state=Leader 2025-09-14T21:49:01.258Z [INFO] cluster: stable state for Leader reached at 2025-09-14 21:49:01.153311766 +0000 UTC m=+111.721190037 (1 nodes), 917ns from start of poll, 160.481167ms from cluster start. Timeout at 2025-09-14 21:49:01.258318548 +0000 UTC m=+111.826196812, 105.006775ms after stability 2025-09-14T21:49:01.264Z [INFO] cluster: starting stability test: raft-state=Follower 2025-09-14T21:49:01.369Z [INFO] cluster: stable state for Follower reached at 2025-09-14 21:49:01.264285787 +0000 UTC m=+111.832164048 (2 nodes), 2.519µs from start of poll, 271.455178ms from cluster start. Timeout at 2025-09-14 21:49:01.36932356 +0000 UTC m=+111.937201820, 105.037772ms after stability 2025-09-14T21:49:01.370Z [ERROR] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to heartbeat to: peer=70a153e8-5cdb-9d89-d720-d2fa666640c9 backoff time=10ms error="failed to connect to peer: 70a153e8-5cdb-9d89-d720-d2fa666640c9" 2025-09-14T21:49:01.374Z [ERROR] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to pipeline appendEntries: peer="{Voter server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2}" error="append pipeline closed" 2025-09-14T21:49:01.374Z [INFO] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: aborting pipeline replication: peer="{Voter server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2}" 2025-09-14T21:49:01.376Z [ERROR] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to pipeline appendEntries: peer="{Voter server-70a153e8-5cdb-9d89-d720-d2fa666640c9 70a153e8-5cdb-9d89-d720-d2fa666640c9}" error="append pipeline closed" 2025-09-14T21:49:01.376Z [INFO] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: aborting pipeline replication: peer="{Voter server-70a153e8-5cdb-9d89-d720-d2fa666640c9 70a153e8-5cdb-9d89-d720-d2fa666640c9}" 2025-09-14T21:49:01.376Z [ERROR] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to heartbeat to: peer=d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 backoff time=10ms error="failed to connect to peer: d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2" 2025-09-14T21:49:01.383Z [ERROR] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to appendEntries to: peer="{Voter server-70a153e8-5cdb-9d89-d720-d2fa666640c9 70a153e8-5cdb-9d89-d720-d2fa666640c9}" error="failed to connect to peer: 70a153e8-5cdb-9d89-d720-d2fa666640c9" 2025-09-14T21:49:01.384Z [ERROR] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to appendEntries to: peer="{Voter server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2}" error="failed to connect to peer: d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2" 2025-09-14T21:49:01.390Z [ERROR] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to heartbeat to: peer=70a153e8-5cdb-9d89-d720-d2fa666640c9 backoff time=10ms error="failed to connect to peer: 70a153e8-5cdb-9d89-d720-d2fa666640c9" 2025-09-14T21:49:01.395Z [ERROR] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to heartbeat to: peer=d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 backoff time=10ms error="failed to connect to peer: d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2" 2025-09-14T21:49:01.403Z [ERROR] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to appendEntries to: peer="{Voter server-70a153e8-5cdb-9d89-d720-d2fa666640c9 70a153e8-5cdb-9d89-d720-d2fa666640c9}" error="failed to connect to peer: 70a153e8-5cdb-9d89-d720-d2fa666640c9" 2025-09-14T21:49:01.404Z [ERROR] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to appendEntries to: peer="{Voter server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2}" error="failed to connect to peer: d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2" 2025-09-14T21:49:01.406Z [ERROR] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to heartbeat to: peer=70a153e8-5cdb-9d89-d720-d2fa666640c9 backoff time=10ms error="failed to connect to peer: 70a153e8-5cdb-9d89-d720-d2fa666640c9" 2025-09-14T21:49:01.413Z [ERROR] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to heartbeat to: peer=d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 backoff time=10ms error="failed to connect to peer: d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2" 2025-09-14T21:49:01.418Z [WARN] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to contact: server-id=server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 time=50.020288ms 2025-09-14T21:49:01.421Z [ERROR] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to appendEntries to: peer="{Voter server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2}" error="failed to connect to peer: d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2" 2025-09-14T21:49:01.423Z [ERROR] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to appendEntries to: peer="{Voter server-70a153e8-5cdb-9d89-d720-d2fa666640c9 70a153e8-5cdb-9d89-d720-d2fa666640c9}" error="failed to connect to peer: 70a153e8-5cdb-9d89-d720-d2fa666640c9" 2025-09-14T21:49:01.424Z [ERROR] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to heartbeat to: peer=70a153e8-5cdb-9d89-d720-d2fa666640c9 backoff time=20ms error="failed to connect to peer: 70a153e8-5cdb-9d89-d720-d2fa666640c9" 2025-09-14T21:49:01.428Z [WARN] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to contact: server-id=server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 time=60.490522ms 2025-09-14T21:49:01.428Z [WARN] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to contact: server-id=server-70a153e8-5cdb-9d89-d720-d2fa666640c9 time=59.191599ms 2025-09-14T21:49:01.428Z [WARN] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to contact quorum of nodes, stepping down 2025-09-14T21:49:01.428Z [INFO] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: entering follower state: follower="Node at ee65bab0-a64b-c4d9-0be7-42e198beaf09 [Follower]" leader-address= leader-id= 2025-09-14T21:49:01.431Z [ERROR] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to heartbeat to: peer=d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 backoff time=20ms error="failed to connect to peer: d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2" 2025-09-14T21:49:01.447Z [WARN] server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2: heartbeat timeout reached, starting election: last-leader-addr=ee65bab0-a64b-c4d9-0be7-42e198beaf09 last-leader-id=server-ee65bab0-a64b-c4d9-0be7-42e198beaf09 2025-09-14T21:49:01.447Z [INFO] server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2: entering candidate state: node="Node at d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 [Candidate]" term=3 2025-09-14T21:49:01.447Z [WARN] server-70a153e8-5cdb-9d89-d720-d2fa666640c9: rejecting vote request since we have a leader: from=d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 leader=ee65bab0-a64b-c4d9-0be7-42e198beaf09 leader-id=server-ee65bab0-a64b-c4d9-0be7-42e198beaf09 2025-09-14T21:49:01.447Z [ERROR] server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2: failed to make requestVote RPC: target="{Voter server-ee65bab0-a64b-c4d9-0be7-42e198beaf09 ee65bab0-a64b-c4d9-0be7-42e198beaf09}" error="failed to connect to peer: ee65bab0-a64b-c4d9-0be7-42e198beaf09" term=3 2025-09-14T21:49:01.448Z [ERROR] server-ee65bab0-a64b-c4d9-0be7-42e198beaf09: failed to appendEntries to: peer="{Voter server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2}" error="failed to connect to peer: d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2" 2025-09-14T21:49:01.475Z [INFO] server-d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2: entering follower state: follower="Node at d2cdaf5f-b2c0-21b3-f617-b45a9e8dd1a2 [Follower]" leader-address= leader-id= --- PASS: TestRaft_VoteWithNoIDNoAddr (0.48s) === RUN TestRaft_runFollower_State_Transition === RUN TestRaft_runFollower_State_Transition/NonVoter 2025-09-14T21:49:01.476Z [INFO] 127.0.0.1:36227 :: starting node: addr=127.0.0.1:36227 2025-09-14T21:49:01.476Z [INFO] 127.0.0.1:36227 :: initial configuration: index=0 servers=[] 2025-09-14T21:49:01.476Z [INFO] 127.0.0.1:36227 :: entering follower state: follower="Node at 127.0.0.1:36227 [Follower]" leader-address= leader-id= 2025-09-14T21:49:01.574Z [WARN] 127.0.0.1:36227 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election === RUN TestRaft_runFollower_State_Transition/Voter 2025-09-14T21:49:01.627Z [INFO] 127.0.0.1:44125 :: starting node: addr=127.0.0.1:44125 2025-09-14T21:49:01.627Z [INFO] 127.0.0.1:44125 :: initial configuration: index=0 servers=[] 2025-09-14T21:49:01.627Z [INFO] 127.0.0.1:44125 :: entering follower state: follower="Node at 127.0.0.1:44125 [Follower]" leader-address= leader-id= 2025-09-14T21:49:01.714Z [WARN] 127.0.0.1:44125 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= === RUN TestRaft_runFollower_State_Transition/Not_in_Config 2025-09-14T21:49:01.779Z [INFO] 127.0.0.1:45183 :: starting node: addr=127.0.0.1:45183 2025-09-14T21:49:01.779Z [INFO] 127.0.0.1:45183 :: initial configuration: index=0 servers=[] 2025-09-14T21:49:01.779Z [INFO] 127.0.0.1:45183 :: entering follower state: follower="Node at 127.0.0.1:45183 [Follower]" leader-address= leader-id= 2025-09-14T21:49:01.872Z [WARN] 127.0.0.1:45183 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election --- PASS: TestRaft_runFollower_State_Transition (0.45s) --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s) === RUN TestRaft_runFollower_ReloadTimeoutConfigs 2025-09-14T21:49:01.931Z [INFO] 127.0.0.1:40819 :: starting node: addr=127.0.0.1:40819 2025-09-14T21:49:01.931Z [INFO] 127.0.0.1:40819 :: initial configuration: index=0 servers=[] 2025-09-14T21:49:01.931Z [INFO] 127.0.0.1:40819 :: entering follower state: follower="Node at 127.0.0.1:40819 [Follower]" leader-address= leader-id= 2025-09-14T21:49:01.931Z [WARN] 127.0.0.1:40819 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= --- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s) === RUN TestSaturationMetric === RUN TestSaturationMetric/without_smoothing --- PASS: TestSaturationMetric (0.00s) --- PASS: TestSaturationMetric/without_smoothing (0.00s) === RUN TestSaturationMetric_IncorrectUsage === RUN TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_first --- PASS: TestSaturationMetric_IncorrectUsage (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_EmptyAddr --- PASS: TestTCPTransport_EmptyAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) === RUN TestOverrideNotifyBool --- PASS: TestOverrideNotifyBool (0.00s) FAIL FAIL github.com/hashicorp/raft 112.660s ? github.com/hashicorp/raft/bench [no test files] FAIL dh_auto_test: error: cd obj-x86_64-linux-gnu && go test -vet=off -v -p 40 github.com/hashicorp/raft github.com/hashicorp/raft/bench returned exit code 1 make[1]: *** [debian/rules:17: override_dh_auto_test] Error 25 make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' make: *** [debian/rules:14: binary] Error 2 dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2 I: copying local configuration E: Failed autobuilding of package I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/3199738 and its subdirectories