I: pbuilder: network access will be disabled during build I: Current time: Tue Jan 28 02:38:43 -12 2025 I: pbuilder-time-stamp: 1738075123 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/unstable-reproducible-base.tgz] I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: Copying source file I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc] I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz] I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz] I: Extracting source dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: unsupported subcommand dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying 0001-Skip-unreliable-tests.patch I: Not using root during the build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/5020/tmp/hooks/D02_print_environment starting I: set BUILDDIR='/build/reproducible-path' BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' BUILDUSERNAME='pbuilder1' BUILD_ARCH='armhf' DEBIAN_FRONTEND='noninteractive' DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=3 ' DISTRIBUTION='unstable' HOME='/root' HOST_ARCH='armhf' IFS=' ' INVOCATION_ID='28571caa5af849b4bffd30c66d7e80c7' LANG='C' LANGUAGE='en_US:en' LC_ALL='C' MAIL='/var/mail/root' OPTIND='1' PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' PBCURRENTCOMMANDLINEOPERATION='build' PBUILDER_OPERATION='build' PBUILDER_PKGDATADIR='/usr/share/pbuilder' PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' PBUILDER_SYSCONFDIR='/etc' PPID='5020' PS1='# ' PS2='> ' PS4='+ ' PWD='/' SHELL='/bin/bash' SHLVL='2' SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.mPRYFda3/pbuilderrc_mml7 --distribution unstable --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/unstable-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.mPRYFda3/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc' SUDO_GID='113' SUDO_UID='107' SUDO_USER='jenkins' TERM='unknown' TZ='/usr/share/zoneinfo/Etc/GMT+12' USER='root' _='/usr/bin/systemd-run' http_proxy='http://10.0.0.15:3142/' I: uname -a Linux wbq0 6.1.0-30-armmp #1 SMP Debian 6.1.124-1 (2025-01-12) armv7l GNU/Linux I: ls -l /bin lrwxrwxrwx 1 root root 7 Nov 22 14:40 /bin -> usr/bin I: user script /srv/workspace/pbuilder/5020/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: armhf Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 19568 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however: Package debhelper-compat is not installed. pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however: Package dh-sequence-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however: Package golang-github-stretchr-testify-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.23-go{a} golang-1.23-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libunistring5{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 0 packages upgraded, 106 newly installed, 0 to remove and 0 not upgraded. Need to get 123 MB of archives. After unpacking 681 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian unstable/main armhf sensible-utils all 0.0.24 [24.8 kB] Get: 2 http://deb.debian.org/debian unstable/main armhf libmagic-mgc armhf 1:5.45-3+b1 [314 kB] Get: 3 http://deb.debian.org/debian unstable/main armhf libmagic1t64 armhf 1:5.45-3+b1 [98.5 kB] Get: 4 http://deb.debian.org/debian unstable/main armhf file armhf 1:5.45-3+b1 [42.3 kB] Get: 5 http://deb.debian.org/debian unstable/main armhf gettext-base armhf 0.23.1-1 [238 kB] Get: 6 http://deb.debian.org/debian unstable/main armhf libuchardet0 armhf 0.0.8-1+b2 [65.6 kB] Get: 7 http://deb.debian.org/debian unstable/main armhf groff-base armhf 1.23.0-7 [1095 kB] Get: 8 http://deb.debian.org/debian unstable/main armhf bsdextrautils armhf 2.40.4-2 [84.6 kB] Get: 9 http://deb.debian.org/debian unstable/main armhf libpipeline1 armhf 1.5.8-1 [35.0 kB] Get: 10 http://deb.debian.org/debian unstable/main armhf man-db armhf 2.13.0-1 [1382 kB] Get: 11 http://deb.debian.org/debian unstable/main armhf m4 armhf 1.4.19-5 [272 kB] Get: 12 http://deb.debian.org/debian unstable/main armhf autoconf all 2.72-3 [493 kB] Get: 13 http://deb.debian.org/debian unstable/main armhf autotools-dev all 20220109.1 [51.6 kB] Get: 14 http://deb.debian.org/debian unstable/main armhf automake all 1:1.17-2 [861 kB] Get: 15 http://deb.debian.org/debian unstable/main armhf autopoint all 0.23.1-1 [770 kB] Get: 16 http://deb.debian.org/debian unstable/main armhf libdebhelper-perl all 13.24.1 [90.9 kB] Get: 17 http://deb.debian.org/debian unstable/main armhf libtool all 2.5.4-2 [539 kB] Get: 18 http://deb.debian.org/debian unstable/main armhf dh-autoreconf all 20 [17.1 kB] Get: 19 http://deb.debian.org/debian unstable/main armhf libarchive-zip-perl all 1.68-1 [104 kB] Get: 20 http://deb.debian.org/debian unstable/main armhf libfile-stripnondeterminism-perl all 1.14.1-1 [19.6 kB] Get: 21 http://deb.debian.org/debian unstable/main armhf dh-strip-nondeterminism all 1.14.1-1 [8584 B] Get: 22 http://deb.debian.org/debian unstable/main armhf libelf1t64 armhf 0.192-4 [184 kB] Get: 23 http://deb.debian.org/debian unstable/main armhf dwz armhf 0.15-1+b2 [106 kB] Get: 24 http://deb.debian.org/debian unstable/main armhf libunistring5 armhf 1.3-1 [444 kB] Get: 25 http://deb.debian.org/debian unstable/main armhf libicu72 armhf 72.1-6 [9086 kB] Get: 26 http://deb.debian.org/debian unstable/main armhf libxml2 armhf 2.12.7+dfsg+really2.9.14-0.2+b1 [605 kB] Get: 27 http://deb.debian.org/debian unstable/main armhf gettext armhf 0.23.1-1 [1560 kB] Get: 28 http://deb.debian.org/debian unstable/main armhf intltool-debian all 0.35.0+20060710.6 [22.9 kB] Get: 29 http://deb.debian.org/debian unstable/main armhf po-debconf all 1.0.21+nmu1 [248 kB] Get: 30 http://deb.debian.org/debian unstable/main armhf debhelper all 13.24.1 [920 kB] Get: 31 http://deb.debian.org/debian unstable/main armhf dh-golang all 1.62 [24.7 kB] Get: 32 http://deb.debian.org/debian unstable/main armhf golang-1.23-src all 1.23.5-1 [19.1 MB] Get: 33 http://deb.debian.org/debian unstable/main armhf golang-1.23-go armhf 1.23.5-1 [23.6 MB] Get: 34 http://deb.debian.org/debian unstable/main armhf golang-src all 2:1.23~2 [5088 B] Get: 35 http://deb.debian.org/debian unstable/main armhf golang-go armhf 2:1.23~2 [44.3 kB] Get: 36 http://deb.debian.org/debian unstable/main armhf golang-any armhf 2:1.23~2 [5172 B] Get: 37 http://deb.debian.org/debian unstable/main armhf golang-github-alecthomas-units-dev all 0.0~git20240626.61d1e34-1 [9032 B] Get: 38 http://deb.debian.org/debian unstable/main armhf golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 39 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 40 http://deb.debian.org/debian unstable/main armhf golang-golang-x-sys-dev all 0.22.0-1 [431 kB] Get: 41 http://deb.debian.org/debian unstable/main armhf golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B] Get: 42 http://deb.debian.org/debian unstable/main armhf golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB] Get: 43 http://deb.debian.org/debian unstable/main armhf golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB] Get: 44 http://deb.debian.org/debian unstable/main armhf golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB] Get: 45 http://deb.debian.org/debian unstable/main armhf golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 46 http://deb.debian.org/debian unstable/main armhf golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB] Get: 47 http://deb.debian.org/debian unstable/main armhf golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB] Get: 48 http://deb.debian.org/debian unstable/main armhf golang-github-stretchr-testify-dev all 1.9.0-1 [74.7 kB] Get: 49 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 50 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB] Get: 51 http://deb.debian.org/debian unstable/main armhf golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB] Get: 52 http://deb.debian.org/debian unstable/main armhf golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B] Get: 53 http://deb.debian.org/debian unstable/main armhf golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB] Get: 54 http://deb.debian.org/debian unstable/main armhf golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 55 http://deb.debian.org/debian unstable/main armhf golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB] Get: 56 http://deb.debian.org/debian unstable/main armhf golang-google-protobuf-dev all 1.33.0-1 [625 kB] Get: 57 http://deb.debian.org/debian unstable/main armhf golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB] Get: 58 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B] Get: 59 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 60 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB] Get: 61 http://deb.debian.org/debian unstable/main armhf golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB] Get: 62 http://deb.debian.org/debian unstable/main armhf golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB] Get: 63 http://deb.debian.org/debian unstable/main armhf golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB] Get: 64 http://deb.debian.org/debian unstable/main armhf golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 65 http://deb.debian.org/debian unstable/main armhf golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB] Get: 66 http://deb.debian.org/debian unstable/main armhf golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB] Get: 67 http://deb.debian.org/debian unstable/main armhf golang-github-klauspost-compress-dev all 1.17.11+ds1-1 [33.3 MB] Get: 68 http://deb.debian.org/debian unstable/main armhf golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB] Get: 69 http://deb.debian.org/debian unstable/main armhf golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB] Get: 70 http://deb.debian.org/debian unstable/main armhf golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB] Get: 71 http://deb.debian.org/debian unstable/main armhf golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB] Get: 72 http://deb.debian.org/debian unstable/main armhf golang-golang-x-term-dev all 0.22.0-1 [14.8 kB] Get: 73 http://deb.debian.org/debian unstable/main armhf golang-golang-x-text-dev all 0.16.0-1 [4003 kB] Get: 74 http://deb.debian.org/debian unstable/main armhf golang-golang-x-crypto-dev all 1:0.25.0-1 [1682 kB] Get: 75 http://deb.debian.org/debian unstable/main armhf golang-golang-x-net-dev all 1:0.27.0-1 [898 kB] Get: 76 http://deb.debian.org/debian unstable/main armhf golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB] Get: 77 http://deb.debian.org/debian unstable/main armhf golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 78 http://deb.debian.org/debian unstable/main armhf golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB] Get: 79 http://deb.debian.org/debian unstable/main armhf golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB] Get: 80 http://deb.debian.org/debian unstable/main armhf golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B] Get: 81 http://deb.debian.org/debian unstable/main armhf golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 82 http://deb.debian.org/debian unstable/main armhf golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB] Get: 83 http://deb.debian.org/debian unstable/main armhf golang-golang-x-oauth2-dev all 0.15.0-1 [38.0 kB] Get: 84 http://deb.debian.org/debian unstable/main armhf golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB] Get: 85 http://deb.debian.org/debian unstable/main armhf golang-github-prometheus-common-dev all 0.61.0-1 [120 kB] Get: 86 http://deb.debian.org/debian unstable/main armhf golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB] Get: 87 http://deb.debian.org/debian unstable/main armhf golang-github-prometheus-procfs-dev all 0.14.0-1 [183 kB] Get: 88 http://deb.debian.org/debian unstable/main armhf golang-github-prometheus-client-golang-dev all 1.20.5-1 [188 kB] Get: 89 http://deb.debian.org/debian unstable/main armhf golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB] Get: 90 http://deb.debian.org/debian unstable/main armhf golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB] Get: 91 http://deb.debian.org/debian unstable/main armhf golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B] Get: 92 http://deb.debian.org/debian unstable/main armhf golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 93 http://deb.debian.org/debian unstable/main armhf golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 94 http://deb.debian.org/debian unstable/main armhf libsasl2-modules-db armhf 2.1.28+dfsg1-8+b1 [18.6 kB] Get: 95 http://deb.debian.org/debian unstable/main armhf libsasl2-2 armhf 2.1.28+dfsg1-8+b1 [50.6 kB] Get: 96 http://deb.debian.org/debian unstable/main armhf libssl-dev armhf 3.4.0-2 [2431 kB] Get: 97 http://deb.debian.org/debian unstable/main armhf libsasl2-dev armhf 2.1.28+dfsg1-8+b1 [219 kB] Get: 98 http://deb.debian.org/debian unstable/main armhf golang-github-kr-text-dev all 0.2.0-2 [10.2 kB] Get: 99 http://deb.debian.org/debian unstable/main armhf golang-golang-x-mod-dev all 0.19.0-1 [104 kB] Get: 100 http://deb.debian.org/debian unstable/main armhf golang-github-yuin-goldmark-dev all 1.7.4-1 [113 kB] Get: 101 http://deb.debian.org/debian unstable/main armhf golang-golang-x-tools-dev all 1:0.23.0+ds-1 [1593 kB] Get: 102 http://deb.debian.org/debian unstable/main armhf golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB] Get: 103 http://deb.debian.org/debian unstable/main armhf golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB] Get: 104 http://deb.debian.org/debian unstable/main armhf golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 105 http://deb.debian.org/debian unstable/main armhf golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 106 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 123 MB in 13s (9639 kB/s) Preconfiguring packages ... Selecting previously unselected package sensible-utils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19568 files and directories currently installed.) Preparing to unpack .../000-sensible-utils_0.0.24_all.deb ... Unpacking sensible-utils (0.0.24) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3+b1_armhf.deb ... Unpacking libmagic-mgc (1:5.45-3+b1) ... Selecting previously unselected package libmagic1t64:armhf. Preparing to unpack .../002-libmagic1t64_1%3a5.45-3+b1_armhf.deb ... Unpacking libmagic1t64:armhf (1:5.45-3+b1) ... Selecting previously unselected package file. Preparing to unpack .../003-file_1%3a5.45-3+b1_armhf.deb ... Unpacking file (1:5.45-3+b1) ... Selecting previously unselected package gettext-base. Preparing to unpack .../004-gettext-base_0.23.1-1_armhf.deb ... Unpacking gettext-base (0.23.1-1) ... Selecting previously unselected package libuchardet0:armhf. Preparing to unpack .../005-libuchardet0_0.0.8-1+b2_armhf.deb ... Unpacking libuchardet0:armhf (0.0.8-1+b2) ... Selecting previously unselected package groff-base. Preparing to unpack .../006-groff-base_1.23.0-7_armhf.deb ... Unpacking groff-base (1.23.0-7) ... Selecting previously unselected package bsdextrautils. Preparing to unpack .../007-bsdextrautils_2.40.4-2_armhf.deb ... Unpacking bsdextrautils (2.40.4-2) ... Selecting previously unselected package libpipeline1:armhf. Preparing to unpack .../008-libpipeline1_1.5.8-1_armhf.deb ... Unpacking libpipeline1:armhf (1.5.8-1) ... Selecting previously unselected package man-db. Preparing to unpack .../009-man-db_2.13.0-1_armhf.deb ... Unpacking man-db (2.13.0-1) ... Selecting previously unselected package m4. Preparing to unpack .../010-m4_1.4.19-5_armhf.deb ... Unpacking m4 (1.4.19-5) ... Selecting previously unselected package autoconf. Preparing to unpack .../011-autoconf_2.72-3_all.deb ... Unpacking autoconf (2.72-3) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ... Unpacking autotools-dev (20220109.1) ... Selecting previously unselected package automake. Preparing to unpack .../013-automake_1%3a1.17-2_all.deb ... Unpacking automake (1:1.17-2) ... Selecting previously unselected package autopoint. Preparing to unpack .../014-autopoint_0.23.1-1_all.deb ... Unpacking autopoint (0.23.1-1) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../015-libdebhelper-perl_13.24.1_all.deb ... Unpacking libdebhelper-perl (13.24.1) ... Selecting previously unselected package libtool. Preparing to unpack .../016-libtool_2.5.4-2_all.deb ... Unpacking libtool (2.5.4-2) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../017-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.1-1_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.14.1-1) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../020-dh-strip-nondeterminism_1.14.1-1_all.deb ... Unpacking dh-strip-nondeterminism (1.14.1-1) ... Selecting previously unselected package libelf1t64:armhf. Preparing to unpack .../021-libelf1t64_0.192-4_armhf.deb ... Unpacking libelf1t64:armhf (0.192-4) ... Selecting previously unselected package dwz. Preparing to unpack .../022-dwz_0.15-1+b2_armhf.deb ... Unpacking dwz (0.15-1+b2) ... Selecting previously unselected package libunistring5:armhf. Preparing to unpack .../023-libunistring5_1.3-1_armhf.deb ... Unpacking libunistring5:armhf (1.3-1) ... Selecting previously unselected package libicu72:armhf. Preparing to unpack .../024-libicu72_72.1-6_armhf.deb ... Unpacking libicu72:armhf (72.1-6) ... Selecting previously unselected package libxml2:armhf. Preparing to unpack .../025-libxml2_2.12.7+dfsg+really2.9.14-0.2+b1_armhf.deb ... Unpacking libxml2:armhf (2.12.7+dfsg+really2.9.14-0.2+b1) ... Selecting previously unselected package gettext. Preparing to unpack .../026-gettext_0.23.1-1_armhf.deb ... Unpacking gettext (0.23.1-1) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../027-intltool-debian_0.35.0+20060710.6_all.deb ... Unpacking intltool-debian (0.35.0+20060710.6) ... Selecting previously unselected package po-debconf. Preparing to unpack .../028-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../029-debhelper_13.24.1_all.deb ... Unpacking debhelper (13.24.1) ... Selecting previously unselected package dh-golang. Preparing to unpack .../030-dh-golang_1.62_all.deb ... Unpacking dh-golang (1.62) ... Selecting previously unselected package golang-1.23-src. Preparing to unpack .../031-golang-1.23-src_1.23.5-1_all.deb ... Unpacking golang-1.23-src (1.23.5-1) ... Selecting previously unselected package golang-1.23-go. Preparing to unpack .../032-golang-1.23-go_1.23.5-1_armhf.deb ... Unpacking golang-1.23-go (1.23.5-1) ... Selecting previously unselected package golang-src. Preparing to unpack .../033-golang-src_2%3a1.23~2_all.deb ... Unpacking golang-src (2:1.23~2) ... Selecting previously unselected package golang-go:armhf. Preparing to unpack .../034-golang-go_2%3a1.23~2_armhf.deb ... Unpacking golang-go:armhf (2:1.23~2) ... Selecting previously unselected package golang-any:armhf. Preparing to unpack .../035-golang-any_2%3a1.23~2_armhf.deb ... Unpacking golang-any:armhf (2:1.23~2) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../036-golang-github-alecthomas-units-dev_0.0~git20240626.61d1e34-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../037-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../038-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../039-golang-golang-x-sys-dev_0.22.0-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.22.0-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../040-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../041-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../042-golang-github-fatih-color-dev_1.17.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.17.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../043-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../044-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../045-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../046-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../047-golang-github-stretchr-testify-dev_1.9.0-1_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.9.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../048-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../049-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../050-golang-github-pkg-errors-dev_0.9.1-3_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-3) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../051-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../052-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../053-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-google-go-cmp-dev. Preparing to unpack .../054-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ... Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ... Selecting previously unselected package golang-google-protobuf-dev. Preparing to unpack .../055-golang-google-protobuf-dev_1.33.0-1_all.deb ... Unpacking golang-google-protobuf-dev (1.33.0-1) ... Selecting previously unselected package golang-github-golang-protobuf-1-5-dev. Preparing to unpack .../056-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ... Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../057-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../058-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../059-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../060-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../061-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../062-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../063-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../064-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../065-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ... Selecting previously unselected package golang-github-klauspost-compress-dev. Preparing to unpack .../066-golang-github-klauspost-compress-dev_1.17.11+ds1-1_all.deb ... Unpacking golang-github-klauspost-compress-dev (1.17.11+ds1-1) ... Selecting previously unselected package golang-github-kylelemons-godebug-dev. Preparing to unpack .../067-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ... Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../068-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../069-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ... Selecting previously unselected package golang-github-jmespath-go-jmespath-dev. Preparing to unpack .../070-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ... Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Selecting previously unselected package golang-golang-x-term-dev. Preparing to unpack .../071-golang-golang-x-term-dev_0.22.0-1_all.deb ... Unpacking golang-golang-x-term-dev (0.22.0-1) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../072-golang-golang-x-text-dev_0.16.0-1_all.deb ... Unpacking golang-golang-x-text-dev (0.16.0-1) ... Selecting previously unselected package golang-golang-x-crypto-dev. Preparing to unpack .../073-golang-golang-x-crypto-dev_1%3a0.25.0-1_all.deb ... Unpacking golang-golang-x-crypto-dev (1:0.25.0-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../074-golang-golang-x-net-dev_1%3a0.27.0-1_all.deb ... Unpacking golang-golang-x-net-dev (1:0.27.0-1) ... Selecting previously unselected package golang-github-aws-aws-sdk-go-dev. Preparing to unpack .../075-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ... Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../076-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-kit-log-dev. Preparing to unpack .../077-golang-github-go-kit-log-dev_0.2.1-1_all.deb ... Unpacking golang-github-go-kit-log-dev (0.2.1-1) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../078-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Selecting previously unselected package golang-github-munnerz-goautoneg-dev. Preparing to unpack .../079-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ... Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../080-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../081-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../082-golang-golang-x-oauth2-dev_0.15.0-1_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.15.0-1) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../083-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../084-golang-github-prometheus-common-dev_0.61.0-1_all.deb ... Unpacking golang-github-prometheus-common-dev (0.61.0-1) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../085-golang-golang-x-sync-dev_0.7.0-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.7.0-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../086-golang-github-prometheus-procfs-dev_0.14.0-1_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.14.0-1) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../087-golang-github-prometheus-client-golang-dev_1.20.5-1_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.20.5-1) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../088-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../089-golang-github-creack-pty-dev_1.1.21-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.21-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../090-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../091-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../092-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:armhf. Preparing to unpack .../093-libsasl2-modules-db_2.1.28+dfsg1-8+b1_armhf.deb ... Unpacking libsasl2-modules-db:armhf (2.1.28+dfsg1-8+b1) ... Selecting previously unselected package libsasl2-2:armhf. Preparing to unpack .../094-libsasl2-2_2.1.28+dfsg1-8+b1_armhf.deb ... Unpacking libsasl2-2:armhf (2.1.28+dfsg1-8+b1) ... Selecting previously unselected package libssl-dev:armhf. Preparing to unpack .../095-libssl-dev_3.4.0-2_armhf.deb ... Unpacking libssl-dev:armhf (3.4.0-2) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../096-libsasl2-dev_2.1.28+dfsg1-8+b1_armhf.deb ... Unpacking libsasl2-dev (2.1.28+dfsg1-8+b1) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../097-golang-github-kr-text-dev_0.2.0-2_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-2) ... Selecting previously unselected package golang-golang-x-mod-dev. Preparing to unpack .../098-golang-golang-x-mod-dev_0.19.0-1_all.deb ... Unpacking golang-golang-x-mod-dev (0.19.0-1) ... Selecting previously unselected package golang-github-yuin-goldmark-dev. Preparing to unpack .../099-golang-github-yuin-goldmark-dev_1.7.4-1_all.deb ... Unpacking golang-github-yuin-goldmark-dev (1.7.4-1) ... Selecting previously unselected package golang-golang-x-tools-dev. Preparing to unpack .../100-golang-golang-x-tools-dev_1%3a0.23.0+ds-1_all.deb ... Unpacking golang-golang-x-tools-dev (1:0.23.0+ds-1) ... Selecting previously unselected package golang-github-rogpeppe-go-internal-dev. Preparing to unpack .../101-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ... Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../102-golang-github-kr-pretty-dev_0.3.1-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.3.1-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../103-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../104-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../105-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up libpipeline1:armhf (1.5.8-1) ... Setting up golang-github-google-go-cmp-dev (0.6.0-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Setting up libicu72:armhf (72.1-6) ... Setting up bsdextrautils (2.40.4-2) ... Setting up golang-github-creack-pty-dev (1.1.21-1) ... Setting up golang-github-stretchr-objx-dev (0.5.2-1) ... Setting up libmagic-mgc (1:5.45-3+b1) ... Setting up golang-github-pkg-errors-dev (0.9.1-3) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ... Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ... Setting up libdebhelper-perl (13.24.1) ... Setting up golang-golang-x-sys-dev (0.22.0-1) ... Setting up libmagic1t64:armhf (1:5.45-3+b1) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.23.1-1) ... Setting up m4 (1.4.19-5) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up golang-golang-x-term-dev (0.22.0-1) ... Setting up file (1:5.45-3+b1) ... Setting up golang-github-beorn7-perks-dev (1.0.1-1) ... Setting up libelf1t64:armhf (0.192-4) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:armhf (2.1.28+dfsg1-8+b1) ... Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ... Setting up autotools-dev (20220109.1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-google-protobuf-dev (1.33.0-1) ... Setting up golang-github-kr-text-dev (0.2.0-2) ... Setting up golang-github-stretchr-testify-dev (1.9.0-1) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Setting up golang-github-yuin-goldmark-dev (1.7.4-1) ... Setting up libunistring5:armhf (1.3-1) ... Setting up libssl-dev:armhf (3.4.0-2) ... Setting up golang-golang-x-sync-dev (0.7.0-1) ... Setting up golang-golang-x-mod-dev (0.19.0-1) ... Setting up golang-1.23-src (1.23.5-1) ... Setting up autopoint (0.23.1-1) ... Setting up golang-github-klauspost-compress-dev (1.17.11+ds1-1) ... Setting up libsasl2-2:armhf (2.1.28+dfsg1-8+b1) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Setting up autoconf (2.72-3) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Setting up dwz (0.15-1+b2) ... Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ... Setting up sensible-utils (0.0.24) ... Setting up golang-golang-x-text-dev (0.16.0-1) ... Setting up libuchardet0:armhf (0.0.8-1+b2) ... Setting up golang-github-prometheus-procfs-dev (0.14.0-1) ... Setting up golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ... Setting up golang-golang-x-crypto-dev (1:0.25.0-1) ... Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-golang-x-oauth2-dev (0.15.0-1) ... Setting up golang-src (2:1.23~2) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Setting up libxml2:armhf (2.12.7+dfsg+really2.9.14-0.2+b1) ... Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Setting up automake (1:1.17-2) ... update-alternatives: using /usr/bin/automake-1.17 to provide /usr/bin/automake (automake) in auto mode Setting up libfile-stripnondeterminism-perl (1.14.1-1) ... Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ... Setting up golang-github-go-kit-log-dev (0.2.1-1) ... Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ... Setting up gettext (0.23.1-1) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Setting up libtool (2.5.4-2) ... Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ... Setting up golang-github-json-iterator-go-dev (1.1.12-2) ... Setting up intltool-debian (0.35.0+20060710.6) ... Setting up dh-autoreconf (20) ... Setting up golang-1.23-go (1.23.5-1) ... Setting up libsasl2-dev (2.1.28+dfsg1-8+b1) ... Setting up golang-golang-x-net-dev (1:0.27.0-1) ... Setting up dh-strip-nondeterminism (1.14.1-1) ... Setting up groff-base (1.23.0-7) ... Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ... Setting up golang-github-fatih-color-dev (1.17.0-1) ... Setting up golang-go:armhf (2:1.23~2) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up golang-golang-x-tools-dev (1:0.23.0+ds-1) ... Setting up man-db (2.13.0-1) ... Not building database; man-db/auto-update is not 'true'. Setting up golang-any:armhf (2:1.23~2) ... Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up golang-github-kr-pretty-dev (0.3.1-1) ... Setting up debhelper (13.24.1) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Setting up dh-golang (1.62) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Setting up golang-github-prometheus-common-dev (0.61.0-1) ... Setting up golang-github-prometheus-client-golang-dev (1.20.5-1) ... Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ... Processing triggers for libc-bin (2.40-6) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps I: Building the package I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.5.0-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.5.0-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Daniel Swarbrick dpkg-source --before-build . dpkg-buildpackage: info: host architecture armhf debian/rules clean dh clean --buildsystem=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules binary dh binary --buildsystem=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-arm-linux-gnueabihf && go install -trimpath -v -p 3 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/unsafeheader internal/goarch internal/cpu internal/byteorder internal/abi internal/chacha8rand internal/bytealg internal/coverage/rtcov internal/godebugs internal/goexperiment internal/goos internal/profilerecord internal/runtime/atomic internal/runtime/syscall internal/stringslite runtime/internal/math runtime/internal/sys internal/runtime/exithook internal/race sync/atomic unicode unicode/utf8 container/list runtime internal/asan internal/itoa internal/msan math/bits math crypto/internal/alias crypto/internal/boring/sig cmp encoding unicode/utf16 log/internal internal/nettrace vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/alias crypto/subtle internal/reflectlite sync iter slices internal/weak maps internal/testlog internal/bisect internal/singleflight errors sort io internal/godebug internal/oserror syscall bytes strings hash strconv bufio crypto/cipher crypto/internal/randutil crypto reflect crypto/internal/boring path math/rand time internal/syscall/unix internal/syscall/execenv github.com/hashicorp/golang-lru/simplelru github.com/hashicorp/go-immutable-radix vendor/golang.org/x/net/dns/dnsmessage context io/fs internal/poll internal/filepathlite math/rand/v2 os internal/concurrent unique internal/fmtsort encoding/binary net/netip encoding/base64 runtime/cgo hash/crc32 fmt crypto/aes crypto/des crypto/internal/edwards25519/field crypto/internal/nistec/fiat math/big encoding/json crypto/rand log net compress/flate crypto/internal/nistec compress/gzip crypto/internal/bigmod crypto/internal/boring/bbig crypto/sha512 crypto/ecdh crypto/elliptic encoding/asn1 crypto/internal/edwards25519 vendor/golang.org/x/crypto/cryptobyte crypto/ed25519 crypto/hmac vendor/golang.org/x/crypto/chacha20 crypto/ecdsa vendor/golang.org/x/crypto/internal/poly1305 vendor/golang.org/x/crypto/chacha20poly1305 vendor/golang.org/x/crypto/hkdf vendor/golang.org/x/sys/cpu crypto/internal/hpke vendor/golang.org/x/crypto/sha3 crypto/md5 crypto/rc4 crypto/rsa crypto/internal/mlkem768 crypto/sha1 crypto/sha256 crypto/dsa encoding/hex encoding/pem crypto/x509/pkix net/url path/filepath vendor/golang.org/x/text/transform vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/text/unicode/norm vendor/golang.org/x/text/secure/bidirule vendor/golang.org/x/net/http2/hpack mime vendor/golang.org/x/net/idna mime/quotedprintable net/http/internal net/http/internal/ascii os/signal golang.org/x/sys/unix crypto/x509 net/textproto vendor/golang.org/x/net/http/httpguts vendor/golang.org/x/net/http/httpproxy mime/multipart io/ioutil regexp/syntax crypto/tls regexp github.com/mattn/go-isatty github.com/mattn/go-colorable github.com/fatih/color internal/saferio github.com/hashicorp/go-hclog encoding/gob go/token html text/template/parse hash/crc64 flag text/template internal/sysinfo runtime/debug net/http/httptrace runtime/trace net/http testing html/template net/rpc github.com/armon/go-metrics github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench debian/rules override_dh_auto_test make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' dh_auto_test cd obj-arm-linux-gnueabihf && go test -vet=off -v -p 3 github.com/hashicorp/raft github.com/hashicorp/raft/bench ? github.com/hashicorp/raft/bench [no test files] === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir 2025-01-28T14:46:56.937Z [INFO] creating new snapshot: path=/tmp/raft283189632/raft1113880266/snapshots/3-10-1738075616937.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.01s) === RUN TestFileSS_CreateSnapshot 2025-01-28T14:46:56.948Z [INFO] creating new snapshot: path=/tmp/raft1913927195/snapshots/3-10-1738075616948.tmp 2025-01-28T14:46:56.952Z [WARN] found temporary snapshot: name=3-10-1738075616948.tmp --- PASS: TestFileSS_CreateSnapshot (0.01s) === RUN TestFileSS_CancelSnapshot 2025-01-28T14:46:56.962Z [INFO] creating new snapshot: path=/tmp/raft220914051/snapshots/3-10-1738075616962.tmp --- PASS: TestFileSS_CancelSnapshot (0.01s) === RUN TestFileSS_Retention 2025-01-28T14:46:56.970Z [INFO] creating new snapshot: path=/tmp/raft1508563252/snapshots/3-10-1738075616970.tmp 2025-01-28T14:46:56.979Z [INFO] creating new snapshot: path=/tmp/raft1508563252/snapshots/3-11-1738075616979.tmp 2025-01-28T14:46:56.990Z [INFO] creating new snapshot: path=/tmp/raft1508563252/snapshots/3-12-1738075616989.tmp 2025-01-28T14:46:57.000Z [INFO] reaping snapshot: path=/tmp/raft1508563252/snapshots/3-10-1738075616970 2025-01-28T14:46:57.001Z [INFO] creating new snapshot: path=/tmp/raft1508563252/snapshots/3-13-1738075617001.tmp 2025-01-28T14:46:57.011Z [INFO] reaping snapshot: path=/tmp/raft1508563252/snapshots/3-11-1738075616979 2025-01-28T14:46:57.013Z [INFO] creating new snapshot: path=/tmp/raft1508563252/snapshots/3-14-1738075617013.tmp 2025-01-28T14:46:57.023Z [INFO] reaping snapshot: path=/tmp/raft1508563252/snapshots/3-12-1738075616989 --- PASS: TestFileSS_Retention (0.06s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering 2025-01-28T14:46:57.036Z [INFO] creating new snapshot: path=/tmp/raft967178168/snapshots/5-130350-1738075617036.tmp 2025-01-28T14:46:57.046Z [INFO] creating new snapshot: path=/tmp/raft967178168/snapshots/36-204917-1738075617046.tmp --- PASS: TestFileSS_Ordering (0.02s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestRaft_RestartFollower_LongInitialHeartbeat --- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestLogCacheWithBackendStoreError --- PASS: TestLogCacheWithBackendStoreError (0.00s) === RUN TestOldestLog === RUN TestOldestLog/empty_logs === RUN TestOldestLog/simple_case --- PASS: TestOldestLog (0.00s) --- PASS: TestOldestLog/empty_logs (0.00s) --- PASS: TestOldestLog/simple_case (0.00s) === RUN TestEmitsLogStoreMetrics --- PASS: TestEmitsLogStoreMetrics (0.01s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.02s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.01s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.02s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams --- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.06s) === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 2025-01-28T14:46:57.258Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 2025-01-28T14:46:57.318Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 2025-01-28T14:46:57.381Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.18s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.06s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.06s) === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.01s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.02s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.03s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:839: countingWriter: 2025-01-28T14:46:57.443Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-01-28T14:46:57.449Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-01-28T14:46:57.460Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-01-28T14:46:57.481Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-01-28T14:46:57.522Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-01-28T14:46:57.603Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-01-28T14:46:57.764Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-01-28T14:46:58.085Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-01-28T14:46:58.726Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-01-28T14:46:59.730Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2025-01-28T14:47:00.733Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop 2025-01-28T14:47:01.463Z [INFO] server-3358659b-b3cf-31b1-e880-6378dbcd77c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3358659b-b3cf-31b1-e880-6378dbcd77c4 Address:3358659b-b3cf-31b1-e880-6378dbcd77c4}]" --- PASS: TestRaft_StartStop (0.01s) === RUN TestRaft_AfterShutdown 2025-01-28T14:47:01.471Z [INFO] server-a67335e7-2352-3a79-b424-4492a7a4a57e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a67335e7-2352-3a79-b424-4492a7a4a57e Address:a67335e7-2352-3a79-b424-4492a7a4a57e}]" --- PASS: TestRaft_AfterShutdown (0.01s) === RUN TestRaft_LiveBootstrap 2025-01-28T14:47:01.485Z [INFO] server-79546aa9-537c-b229-10aa-db3e354499e8: initial configuration: index=0 servers=[] 2025-01-28T14:47:01.486Z [INFO] server-7724a847-9ed4-aa13-63c6-6094d6b5e8d6: initial configuration: index=0 servers=[] 2025-01-28T14:47:01.486Z [INFO] server-79546aa9-537c-b229-10aa-db3e354499e8: entering follower state: follower="Node at 79546aa9-537c-b229-10aa-db3e354499e8 [Follower]" leader-address= leader-id= 2025-01-28T14:47:01.487Z [INFO] server-7724a847-9ed4-aa13-63c6-6094d6b5e8d6: entering follower state: follower="Node at 7724a847-9ed4-aa13-63c6-6094d6b5e8d6 [Follower]" leader-address= leader-id= 2025-01-28T14:47:01.487Z [INFO] server-45feca13-291e-9a5e-0004-3a482b181ca4: initial configuration: index=0 servers=[] 2025-01-28T14:47:01.487Z [INFO] server-45feca13-291e-9a5e-0004-3a482b181ca4: entering follower state: follower="Node at 45feca13-291e-9a5e-0004-3a482b181ca4 [Follower]" leader-address= leader-id= 2025-01-28T14:47:01.488Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:01.541Z [WARN] server-45feca13-291e-9a5e-0004-3a482b181ca4: no known peers, aborting election 2025-01-28T14:47:01.550Z [WARN] server-79546aa9-537c-b229-10aa-db3e354499e8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:01.550Z [INFO] server-79546aa9-537c-b229-10aa-db3e354499e8: entering candidate state: node="Node at 79546aa9-537c-b229-10aa-db3e354499e8 [Candidate]" term=2 2025-01-28T14:47:01.551Z [INFO] server-79546aa9-537c-b229-10aa-db3e354499e8: election won: term=2 tally=2 2025-01-28T14:47:01.551Z [INFO] server-79546aa9-537c-b229-10aa-db3e354499e8: entering leader state: leader="Node at 79546aa9-537c-b229-10aa-db3e354499e8 [Leader]" 2025-01-28T14:47:01.552Z [INFO] server-79546aa9-537c-b229-10aa-db3e354499e8: added peer, starting replication: peer=server-7724a847-9ed4-aa13-63c6-6094d6b5e8d6 2025-01-28T14:47:01.552Z [INFO] server-79546aa9-537c-b229-10aa-db3e354499e8: added peer, starting replication: peer=server-45feca13-291e-9a5e-0004-3a482b181ca4 2025-01-28T14:47:01.552Z [WARN] server-7724a847-9ed4-aa13-63c6-6094d6b5e8d6: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2025-01-28T14:47:01.553Z [WARN] server-45feca13-291e-9a5e-0004-3a482b181ca4: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2025-01-28T14:47:01.553Z [WARN] server-79546aa9-537c-b229-10aa-db3e354499e8: appendEntries rejected, sending older logs: peer="{Voter server-7724a847-9ed4-aa13-63c6-6094d6b5e8d6 7724a847-9ed4-aa13-63c6-6094d6b5e8d6}" next=1 2025-01-28T14:47:01.554Z [WARN] server-79546aa9-537c-b229-10aa-db3e354499e8: appendEntries rejected, sending older logs: peer="{Voter server-45feca13-291e-9a5e-0004-3a482b181ca4 45feca13-291e-9a5e-0004-3a482b181ca4}" next=1 2025-01-28T14:47:01.555Z [INFO] server-79546aa9-537c-b229-10aa-db3e354499e8: pipelining replication: peer="{Voter server-7724a847-9ed4-aa13-63c6-6094d6b5e8d6 7724a847-9ed4-aa13-63c6-6094d6b5e8d6}" 2025-01-28T14:47:01.555Z [INFO] server-79546aa9-537c-b229-10aa-db3e354499e8: pipelining replication: peer="{Voter server-45feca13-291e-9a5e-0004-3a482b181ca4 45feca13-291e-9a5e-0004-3a482b181ca4}" 2025-01-28T14:47:01.657Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:01.551970922 +0000 UTC m=+4.635377280 (2 nodes), 63.063205ms from start of poll, 66.794983ms from cluster start. Timeout at 2025-01-28 14:47:01.65699259 +0000 UTC m=+4.740398615, 105.021335ms after stability 2025-01-28T14:47:01.658Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:01.763Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:01.658408749 +0000 UTC m=+4.741814440 (1 nodes), 12.334µs from start of poll, 173.232143ms from cluster start. Timeout at 2025-01-28 14:47:01.763607755 +0000 UTC m=+4.847014779, 105.200339ms after stability 2025-01-28T14:47:01.778Z [INFO] server-79546aa9-537c-b229-10aa-db3e354499e8: aborting pipeline replication: peer="{Voter server-45feca13-291e-9a5e-0004-3a482b181ca4 45feca13-291e-9a5e-0004-3a482b181ca4}" 2025-01-28T14:47:01.778Z [INFO] server-79546aa9-537c-b229-10aa-db3e354499e8: aborting pipeline replication: peer="{Voter server-7724a847-9ed4-aa13-63c6-6094d6b5e8d6 7724a847-9ed4-aa13-63c6-6094d6b5e8d6}" --- PASS: TestRaft_LiveBootstrap (0.31s) === RUN TestRaft_LiveBootstrap_From_NonVoter 2025-01-28T14:47:01.793Z [INFO] server-48bb254e-ac56-6b6c-0499-21b5f200f092: initial configuration: index=0 servers=[] 2025-01-28T14:47:01.794Z [INFO] server-25fdb754-f485-ac62-c7bb-c30efa6ef312: initial configuration: index=0 servers=[] 2025-01-28T14:47:01.793Z [INFO] server-48bb254e-ac56-6b6c-0499-21b5f200f092: entering follower state: follower="Node at 48bb254e-ac56-6b6c-0499-21b5f200f092 [Follower]" leader-address= leader-id= 2025-01-28T14:47:01.794Z [INFO] server-25fdb754-f485-ac62-c7bb-c30efa6ef312: entering follower state: follower="Node at 25fdb754-f485-ac62-c7bb-c30efa6ef312 [Follower]" leader-address= leader-id= --- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.01s) === RUN TestRaft_RecoverCluster_NoState 2025-01-28T14:47:01.804Z [INFO] server-23e37d06-0887-185c-d158-99d3aed2a00c: initial configuration: index=0 servers=[] 2025-01-28T14:47:01.804Z [INFO] server-23e37d06-0887-185c-d158-99d3aed2a00c: entering follower state: follower="Node at 23e37d06-0887-185c-d158-99d3aed2a00c [Shutdown]" leader-address= leader-id= --- PASS: TestRaft_RecoverCluster_NoState (0.01s) === RUN TestRaft_RecoverCluster === RUN TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs 2025-01-28T14:47:01.818Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df6943aa-1ab1-b097-3f72-549d08243b9e Address:df6943aa-1ab1-b097-3f72-549d08243b9e} {Suffrage:Voter ID:server-1852e1aa-4cc1-a195-8079-5984b0db113a Address:1852e1aa-4cc1-a195-8079-5984b0db113a} {Suffrage:Voter ID:server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1 Address:fe8c31c3-3ad4-ff83-d85a-35a0960b6de1}]" 2025-01-28T14:47:01.819Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: entering follower state: follower="Node at df6943aa-1ab1-b097-3f72-549d08243b9e [Follower]" leader-address= leader-id= 2025-01-28T14:47:01.820Z [INFO] server-1852e1aa-4cc1-a195-8079-5984b0db113a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df6943aa-1ab1-b097-3f72-549d08243b9e Address:df6943aa-1ab1-b097-3f72-549d08243b9e} {Suffrage:Voter ID:server-1852e1aa-4cc1-a195-8079-5984b0db113a Address:1852e1aa-4cc1-a195-8079-5984b0db113a} {Suffrage:Voter ID:server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1 Address:fe8c31c3-3ad4-ff83-d85a-35a0960b6de1}]" 2025-01-28T14:47:01.820Z [INFO] server-1852e1aa-4cc1-a195-8079-5984b0db113a: entering follower state: follower="Node at 1852e1aa-4cc1-a195-8079-5984b0db113a [Follower]" leader-address= leader-id= 2025-01-28T14:47:01.821Z [INFO] server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df6943aa-1ab1-b097-3f72-549d08243b9e Address:df6943aa-1ab1-b097-3f72-549d08243b9e} {Suffrage:Voter ID:server-1852e1aa-4cc1-a195-8079-5984b0db113a Address:1852e1aa-4cc1-a195-8079-5984b0db113a} {Suffrage:Voter ID:server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1 Address:fe8c31c3-3ad4-ff83-d85a-35a0960b6de1}]" 2025-01-28T14:47:01.822Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:01.822Z [INFO] server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1: entering follower state: follower="Node at fe8c31c3-3ad4-ff83-d85a-35a0960b6de1 [Follower]" leader-address= leader-id= 2025-01-28T14:47:01.874Z [WARN] server-df6943aa-1ab1-b097-3f72-549d08243b9e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:01.874Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: entering candidate state: node="Node at df6943aa-1ab1-b097-3f72-549d08243b9e [Candidate]" term=2 2025-01-28T14:47:01.875Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: election won: term=2 tally=2 2025-01-28T14:47:01.875Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: entering leader state: leader="Node at df6943aa-1ab1-b097-3f72-549d08243b9e [Leader]" 2025-01-28T14:47:01.875Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: added peer, starting replication: peer=server-1852e1aa-4cc1-a195-8079-5984b0db113a 2025-01-28T14:47:01.876Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: added peer, starting replication: peer=server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1 2025-01-28T14:47:01.877Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: pipelining replication: peer="{Voter server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1 fe8c31c3-3ad4-ff83-d85a-35a0960b6de1}" 2025-01-28T14:47:01.879Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: pipelining replication: peer="{Voter server-1852e1aa-4cc1-a195-8079-5984b0db113a 1852e1aa-4cc1-a195-8079-5984b0db113a}" 2025-01-28T14:47:01.983Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:01.877683259 +0000 UTC m=+4.961089616 (1 nodes), 55.357976ms from start of poll, 60.373792ms from cluster start. Timeout at 2025-01-28 14:47:01.982718593 +0000 UTC m=+5.066124951, 105.035335ms after stability 2025-01-28T14:47:01.983Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: aborting pipeline replication: peer="{Voter server-1852e1aa-4cc1-a195-8079-5984b0db113a 1852e1aa-4cc1-a195-8079-5984b0db113a}" 2025-01-28T14:47:01.983Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: aborting pipeline replication: peer="{Voter server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1 fe8c31c3-3ad4-ff83-d85a-35a0960b6de1}" 2025-01-28T14:47:01.985Z [INFO] creating new snapshot: path=/tmp/raft3361722972/snapshots/2-2-1738075621984.tmp 2025-01-28T14:47:01.993Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: starting restore from snapshot: id=2-2-1738075621984 last-index=2 last-term=2 size-in-bytes=1 2025-01-28T14:47:01.995Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: snapshot restore progress: id=2-2-1738075621984 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2025-01-28T14:47:01.995Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: restored from snapshot: id=2-2-1738075621984 last-index=2 last-term=2 size-in-bytes=1 2025-01-28T14:47:01.996Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df6943aa-1ab1-b097-3f72-549d08243b9e Address:df6943aa-1ab1-b097-3f72-549d08243b9e} {Suffrage:Voter ID:server-1852e1aa-4cc1-a195-8079-5984b0db113a Address:1852e1aa-4cc1-a195-8079-5984b0db113a} {Suffrage:Voter ID:server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1 Address:fe8c31c3-3ad4-ff83-d85a-35a0960b6de1}]" 2025-01-28T14:47:01.996Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: entering follower state: follower="Node at df6943aa-1ab1-b097-3f72-549d08243b9e [Follower]" leader-address= leader-id= 2025-01-28T14:47:01.997Z [INFO] creating new snapshot: path=/tmp/raft2582477310/snapshots/2-2-1738075621996.tmp 2025-01-28T14:47:02.004Z [INFO] server-1852e1aa-4cc1-a195-8079-5984b0db113a: starting restore from snapshot: id=2-2-1738075621996 last-index=2 last-term=2 size-in-bytes=1 2025-01-28T14:47:02.006Z [INFO] server-1852e1aa-4cc1-a195-8079-5984b0db113a: snapshot restore progress: id=2-2-1738075621996 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2025-01-28T14:47:02.007Z [INFO] server-1852e1aa-4cc1-a195-8079-5984b0db113a: restored from snapshot: id=2-2-1738075621996 last-index=2 last-term=2 size-in-bytes=1 2025-01-28T14:47:02.007Z [INFO] server-1852e1aa-4cc1-a195-8079-5984b0db113a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df6943aa-1ab1-b097-3f72-549d08243b9e Address:df6943aa-1ab1-b097-3f72-549d08243b9e} {Suffrage:Voter ID:server-1852e1aa-4cc1-a195-8079-5984b0db113a Address:1852e1aa-4cc1-a195-8079-5984b0db113a} {Suffrage:Voter ID:server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1 Address:fe8c31c3-3ad4-ff83-d85a-35a0960b6de1}]" 2025-01-28T14:47:02.007Z [INFO] server-1852e1aa-4cc1-a195-8079-5984b0db113a: entering follower state: follower="Node at 1852e1aa-4cc1-a195-8079-5984b0db113a [Follower]" leader-address= leader-id= 2025-01-28T14:47:02.008Z [INFO] creating new snapshot: path=/tmp/raft1264297467/snapshots/2-2-1738075622008.tmp 2025-01-28T14:47:02.016Z [INFO] server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1: starting restore from snapshot: id=2-2-1738075622008 last-index=2 last-term=2 size-in-bytes=1 2025-01-28T14:47:02.018Z [INFO] server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1: snapshot restore progress: id=2-2-1738075622008 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2025-01-28T14:47:02.018Z [INFO] server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1: restored from snapshot: id=2-2-1738075622008 last-index=2 last-term=2 size-in-bytes=1 2025-01-28T14:47:02.018Z [INFO] server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df6943aa-1ab1-b097-3f72-549d08243b9e Address:df6943aa-1ab1-b097-3f72-549d08243b9e} {Suffrage:Voter ID:server-1852e1aa-4cc1-a195-8079-5984b0db113a Address:1852e1aa-4cc1-a195-8079-5984b0db113a} {Suffrage:Voter ID:server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1 Address:fe8c31c3-3ad4-ff83-d85a-35a0960b6de1}]" 2025-01-28T14:47:02.019Z [INFO] server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1: entering follower state: follower="Node at fe8c31c3-3ad4-ff83-d85a-35a0960b6de1 [Follower]" leader-address= leader-id= 2025-01-28T14:47:02.074Z [WARN] server-df6943aa-1ab1-b097-3f72-549d08243b9e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:02.074Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: entering candidate state: node="Node at df6943aa-1ab1-b097-3f72-549d08243b9e [Candidate]" term=3 2025-01-28T14:47:02.075Z [WARN] server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:02.075Z [INFO] server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1: entering candidate state: node="Node at fe8c31c3-3ad4-ff83-d85a-35a0960b6de1 [Candidate]" term=3 2025-01-28T14:47:02.076Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: election won: term=3 tally=2 2025-01-28T14:47:02.076Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: entering leader state: leader="Node at df6943aa-1ab1-b097-3f72-549d08243b9e [Leader]" 2025-01-28T14:47:02.076Z [INFO] server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1: duplicate requestVote for same term: term=3 2025-01-28T14:47:02.076Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: added peer, starting replication: peer=server-1852e1aa-4cc1-a195-8079-5984b0db113a 2025-01-28T14:47:02.076Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: added peer, starting replication: peer=server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1 2025-01-28T14:47:02.076Z [INFO] server-1852e1aa-4cc1-a195-8079-5984b0db113a: duplicate requestVote for same term: term=3 2025-01-28T14:47:02.077Z [WARN] server-df6943aa-1ab1-b097-3f72-549d08243b9e: rejecting vote request since we have a leader: from=fe8c31c3-3ad4-ff83-d85a-35a0960b6de1 leader=df6943aa-1ab1-b097-3f72-549d08243b9e leader-id=server-df6943aa-1ab1-b097-3f72-549d08243b9e 2025-01-28T14:47:02.078Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: pipelining replication: peer="{Voter server-1852e1aa-4cc1-a195-8079-5984b0db113a 1852e1aa-4cc1-a195-8079-5984b0db113a}" 2025-01-28T14:47:02.078Z [INFO] server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1: entering follower state: follower="Node at fe8c31c3-3ad4-ff83-d85a-35a0960b6de1 [Follower]" leader-address=df6943aa-1ab1-b097-3f72-549d08243b9e leader-id=server-df6943aa-1ab1-b097-3f72-549d08243b9e 2025-01-28T14:47:02.078Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: pipelining replication: peer="{Voter server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1 fe8c31c3-3ad4-ff83-d85a-35a0960b6de1}" 2025-01-28T14:47:02.334Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:02.441Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:02.335187841 +0000 UTC m=+5.418594199 (1 nodes), 435.013µs from start of poll, 517.878375ms from cluster start. Timeout at 2025-01-28 14:47:02.440277509 +0000 UTC m=+5.523683867, 105.089668ms after stability 2025-01-28T14:47:02.442Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: aborting pipeline replication: peer="{Voter server-1852e1aa-4cc1-a195-8079-5984b0db113a 1852e1aa-4cc1-a195-8079-5984b0db113a}" 2025-01-28T14:47:02.442Z [INFO] server-df6943aa-1ab1-b097-3f72-549d08243b9e: aborting pipeline replication: peer="{Voter server-fe8c31c3-3ad4-ff83-d85a-35a0960b6de1 fe8c31c3-3ad4-ff83-d85a-35a0960b6de1}" 2025-01-28T14:47:02.942Z [ERROR] server-df6943aa-1ab1-b097-3f72-549d08243b9e: failed to heartbeat to: peer=1852e1aa-4cc1-a195-8079-5984b0db113a backoff time=10ms error="command timed out" === RUN TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs 2025-01-28T14:47:02.965Z [INFO] server-94d9935c-4fa0-7c31-b12c-73ad442e6428: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94d9935c-4fa0-7c31-b12c-73ad442e6428 Address:94d9935c-4fa0-7c31-b12c-73ad442e6428} {Suffrage:Voter ID:server-97c5019d-853d-2af6-38ab-20c5409bbc12 Address:97c5019d-853d-2af6-38ab-20c5409bbc12} {Suffrage:Voter ID:server-1a0e1a4d-9257-48ca-5915-cf97777ef904 Address:1a0e1a4d-9257-48ca-5915-cf97777ef904}]" 2025-01-28T14:47:02.966Z [INFO] server-94d9935c-4fa0-7c31-b12c-73ad442e6428: entering follower state: follower="Node at 94d9935c-4fa0-7c31-b12c-73ad442e6428 [Follower]" leader-address= leader-id= 2025-01-28T14:47:02.967Z [INFO] server-97c5019d-853d-2af6-38ab-20c5409bbc12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94d9935c-4fa0-7c31-b12c-73ad442e6428 Address:94d9935c-4fa0-7c31-b12c-73ad442e6428} {Suffrage:Voter ID:server-97c5019d-853d-2af6-38ab-20c5409bbc12 Address:97c5019d-853d-2af6-38ab-20c5409bbc12} {Suffrage:Voter ID:server-1a0e1a4d-9257-48ca-5915-cf97777ef904 Address:1a0e1a4d-9257-48ca-5915-cf97777ef904}]" 2025-01-28T14:47:02.968Z [INFO] server-97c5019d-853d-2af6-38ab-20c5409bbc12: entering follower state: follower="Node at 97c5019d-853d-2af6-38ab-20c5409bbc12 [Follower]" leader-address= leader-id= 2025-01-28T14:47:02.969Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94d9935c-4fa0-7c31-b12c-73ad442e6428 Address:94d9935c-4fa0-7c31-b12c-73ad442e6428} {Suffrage:Voter ID:server-97c5019d-853d-2af6-38ab-20c5409bbc12 Address:97c5019d-853d-2af6-38ab-20c5409bbc12} {Suffrage:Voter ID:server-1a0e1a4d-9257-48ca-5915-cf97777ef904 Address:1a0e1a4d-9257-48ca-5915-cf97777ef904}]" 2025-01-28T14:47:02.969Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:02.969Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: entering follower state: follower="Node at 1a0e1a4d-9257-48ca-5915-cf97777ef904 [Follower]" leader-address= leader-id= 2025-01-28T14:47:03.022Z [WARN] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:03.022Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: entering candidate state: node="Node at 1a0e1a4d-9257-48ca-5915-cf97777ef904 [Candidate]" term=2 2025-01-28T14:47:03.023Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: election won: term=2 tally=2 2025-01-28T14:47:03.023Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: entering leader state: leader="Node at 1a0e1a4d-9257-48ca-5915-cf97777ef904 [Leader]" 2025-01-28T14:47:03.024Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: added peer, starting replication: peer=server-94d9935c-4fa0-7c31-b12c-73ad442e6428 2025-01-28T14:47:03.024Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: added peer, starting replication: peer=server-97c5019d-853d-2af6-38ab-20c5409bbc12 2025-01-28T14:47:03.025Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: pipelining replication: peer="{Voter server-94d9935c-4fa0-7c31-b12c-73ad442e6428 94d9935c-4fa0-7c31-b12c-73ad442e6428}" 2025-01-28T14:47:03.026Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: pipelining replication: peer="{Voter server-97c5019d-853d-2af6-38ab-20c5409bbc12 97c5019d-853d-2af6-38ab-20c5409bbc12}" 2025-01-28T14:47:03.130Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:03.024219297 +0000 UTC m=+6.107626654 (1 nodes), 54.620288ms from start of poll, 59.478765ms from cluster start. Timeout at 2025-01-28 14:47:03.129263298 +0000 UTC m=+6.212669322, 105.042668ms after stability 2025-01-28T14:47:03.138Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: aborting pipeline replication: peer="{Voter server-97c5019d-853d-2af6-38ab-20c5409bbc12 97c5019d-853d-2af6-38ab-20c5409bbc12}" 2025-01-28T14:47:03.139Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: aborting pipeline replication: peer="{Voter server-94d9935c-4fa0-7c31-b12c-73ad442e6428 94d9935c-4fa0-7c31-b12c-73ad442e6428}" 2025-01-28T14:47:03.140Z [INFO] creating new snapshot: path=/tmp/raft2035946799/snapshots/2-6-1738075623140.tmp 2025-01-28T14:47:03.148Z [INFO] server-94d9935c-4fa0-7c31-b12c-73ad442e6428: starting restore from snapshot: id=2-6-1738075623140 last-index=6 last-term=2 size-in-bytes=25 2025-01-28T14:47:03.150Z [INFO] server-94d9935c-4fa0-7c31-b12c-73ad442e6428: snapshot restore progress: id=2-6-1738075623140 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2025-01-28T14:47:03.151Z [INFO] server-94d9935c-4fa0-7c31-b12c-73ad442e6428: restored from snapshot: id=2-6-1738075623140 last-index=6 last-term=2 size-in-bytes=25 2025-01-28T14:47:03.151Z [INFO] server-94d9935c-4fa0-7c31-b12c-73ad442e6428: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94d9935c-4fa0-7c31-b12c-73ad442e6428 Address:94d9935c-4fa0-7c31-b12c-73ad442e6428} {Suffrage:Voter ID:server-97c5019d-853d-2af6-38ab-20c5409bbc12 Address:97c5019d-853d-2af6-38ab-20c5409bbc12} {Suffrage:Voter ID:server-1a0e1a4d-9257-48ca-5915-cf97777ef904 Address:1a0e1a4d-9257-48ca-5915-cf97777ef904}]" 2025-01-28T14:47:03.151Z [INFO] server-94d9935c-4fa0-7c31-b12c-73ad442e6428: entering follower state: follower="Node at 94d9935c-4fa0-7c31-b12c-73ad442e6428 [Follower]" leader-address= leader-id= 2025-01-28T14:47:03.152Z [INFO] creating new snapshot: path=/tmp/raft3191791572/snapshots/2-6-1738075623152.tmp 2025-01-28T14:47:03.160Z [INFO] server-97c5019d-853d-2af6-38ab-20c5409bbc12: starting restore from snapshot: id=2-6-1738075623152 last-index=6 last-term=2 size-in-bytes=25 2025-01-28T14:47:03.161Z [INFO] server-97c5019d-853d-2af6-38ab-20c5409bbc12: snapshot restore progress: id=2-6-1738075623152 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2025-01-28T14:47:03.162Z [INFO] server-97c5019d-853d-2af6-38ab-20c5409bbc12: restored from snapshot: id=2-6-1738075623152 last-index=6 last-term=2 size-in-bytes=25 2025-01-28T14:47:03.162Z [INFO] server-97c5019d-853d-2af6-38ab-20c5409bbc12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94d9935c-4fa0-7c31-b12c-73ad442e6428 Address:94d9935c-4fa0-7c31-b12c-73ad442e6428} {Suffrage:Voter ID:server-97c5019d-853d-2af6-38ab-20c5409bbc12 Address:97c5019d-853d-2af6-38ab-20c5409bbc12} {Suffrage:Voter ID:server-1a0e1a4d-9257-48ca-5915-cf97777ef904 Address:1a0e1a4d-9257-48ca-5915-cf97777ef904}]" 2025-01-28T14:47:03.163Z [INFO] server-97c5019d-853d-2af6-38ab-20c5409bbc12: entering follower state: follower="Node at 97c5019d-853d-2af6-38ab-20c5409bbc12 [Follower]" leader-address= leader-id= 2025-01-28T14:47:03.163Z [INFO] creating new snapshot: path=/tmp/raft2635543855/snapshots/2-6-1738075623163.tmp 2025-01-28T14:47:03.171Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: starting restore from snapshot: id=2-6-1738075623163 last-index=6 last-term=2 size-in-bytes=25 2025-01-28T14:47:03.173Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: snapshot restore progress: id=2-6-1738075623163 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2025-01-28T14:47:03.173Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: restored from snapshot: id=2-6-1738075623163 last-index=6 last-term=2 size-in-bytes=25 2025-01-28T14:47:03.173Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94d9935c-4fa0-7c31-b12c-73ad442e6428 Address:94d9935c-4fa0-7c31-b12c-73ad442e6428} {Suffrage:Voter ID:server-97c5019d-853d-2af6-38ab-20c5409bbc12 Address:97c5019d-853d-2af6-38ab-20c5409bbc12} {Suffrage:Voter ID:server-1a0e1a4d-9257-48ca-5915-cf97777ef904 Address:1a0e1a4d-9257-48ca-5915-cf97777ef904}]" 2025-01-28T14:47:03.174Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: entering follower state: follower="Node at 1a0e1a4d-9257-48ca-5915-cf97777ef904 [Follower]" leader-address= leader-id= 2025-01-28T14:47:03.226Z [WARN] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:03.226Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: entering candidate state: node="Node at 1a0e1a4d-9257-48ca-5915-cf97777ef904 [Candidate]" term=3 2025-01-28T14:47:03.227Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: election won: term=3 tally=2 2025-01-28T14:47:03.227Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: entering leader state: leader="Node at 1a0e1a4d-9257-48ca-5915-cf97777ef904 [Leader]" 2025-01-28T14:47:03.228Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: added peer, starting replication: peer=server-94d9935c-4fa0-7c31-b12c-73ad442e6428 2025-01-28T14:47:03.228Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: added peer, starting replication: peer=server-97c5019d-853d-2af6-38ab-20c5409bbc12 2025-01-28T14:47:03.229Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: pipelining replication: peer="{Voter server-94d9935c-4fa0-7c31-b12c-73ad442e6428 94d9935c-4fa0-7c31-b12c-73ad442e6428}" 2025-01-28T14:47:03.229Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: pipelining replication: peer="{Voter server-97c5019d-853d-2af6-38ab-20c5409bbc12 97c5019d-853d-2af6-38ab-20c5409bbc12}" 2025-01-28T14:47:03.489Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:03.597Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:03.490645811 +0000 UTC m=+6.574052168 (1 nodes), 421.679µs from start of poll, 525.904279ms from cluster start. Timeout at 2025-01-28 14:47:03.595681811 +0000 UTC m=+6.679088169, 105.036001ms after stability 2025-01-28T14:47:03.598Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: aborting pipeline replication: peer="{Voter server-97c5019d-853d-2af6-38ab-20c5409bbc12 97c5019d-853d-2af6-38ab-20c5409bbc12}" 2025-01-28T14:47:03.598Z [INFO] server-1a0e1a4d-9257-48ca-5915-cf97777ef904: aborting pipeline replication: peer="{Voter server-94d9935c-4fa0-7c31-b12c-73ad442e6428 94d9935c-4fa0-7c31-b12c-73ad442e6428}" === RUN TestRaft_RecoverCluster/snapshot,_with_trailing_logs 2025-01-28T14:47:03.621Z [INFO] server-f7b027f1-68b8-2216-8157-cedc103a184e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f7b027f1-68b8-2216-8157-cedc103a184e Address:f7b027f1-68b8-2216-8157-cedc103a184e} {Suffrage:Voter ID:server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9 Address:bbc63f83-1cb3-4bcf-2919-31b9c55552e9} {Suffrage:Voter ID:server-cc998fce-243f-d7c9-9c87-63072b683c42 Address:cc998fce-243f-d7c9-9c87-63072b683c42}]" 2025-01-28T14:47:03.622Z [INFO] server-f7b027f1-68b8-2216-8157-cedc103a184e: entering follower state: follower="Node at f7b027f1-68b8-2216-8157-cedc103a184e [Follower]" leader-address= leader-id= 2025-01-28T14:47:03.623Z [INFO] server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f7b027f1-68b8-2216-8157-cedc103a184e Address:f7b027f1-68b8-2216-8157-cedc103a184e} {Suffrage:Voter ID:server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9 Address:bbc63f83-1cb3-4bcf-2919-31b9c55552e9} {Suffrage:Voter ID:server-cc998fce-243f-d7c9-9c87-63072b683c42 Address:cc998fce-243f-d7c9-9c87-63072b683c42}]" 2025-01-28T14:47:03.623Z [INFO] server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9: entering follower state: follower="Node at bbc63f83-1cb3-4bcf-2919-31b9c55552e9 [Follower]" leader-address= leader-id= 2025-01-28T14:47:03.624Z [INFO] server-cc998fce-243f-d7c9-9c87-63072b683c42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f7b027f1-68b8-2216-8157-cedc103a184e Address:f7b027f1-68b8-2216-8157-cedc103a184e} {Suffrage:Voter ID:server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9 Address:bbc63f83-1cb3-4bcf-2919-31b9c55552e9} {Suffrage:Voter ID:server-cc998fce-243f-d7c9-9c87-63072b683c42 Address:cc998fce-243f-d7c9-9c87-63072b683c42}]" 2025-01-28T14:47:03.625Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:03.625Z [INFO] server-cc998fce-243f-d7c9-9c87-63072b683c42: entering follower state: follower="Node at cc998fce-243f-d7c9-9c87-63072b683c42 [Follower]" leader-address= leader-id= 2025-01-28T14:47:03.685Z [WARN] server-cc998fce-243f-d7c9-9c87-63072b683c42: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:03.685Z [INFO] server-cc998fce-243f-d7c9-9c87-63072b683c42: entering candidate state: node="Node at cc998fce-243f-d7c9-9c87-63072b683c42 [Candidate]" term=2 2025-01-28T14:47:03.686Z [INFO] server-cc998fce-243f-d7c9-9c87-63072b683c42: election won: term=2 tally=2 2025-01-28T14:47:03.686Z [INFO] server-cc998fce-243f-d7c9-9c87-63072b683c42: entering leader state: leader="Node at cc998fce-243f-d7c9-9c87-63072b683c42 [Leader]" 2025-01-28T14:47:03.686Z [INFO] server-cc998fce-243f-d7c9-9c87-63072b683c42: added peer, starting replication: peer=server-f7b027f1-68b8-2216-8157-cedc103a184e 2025-01-28T14:47:03.687Z [INFO] server-cc998fce-243f-d7c9-9c87-63072b683c42: added peer, starting replication: peer=server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9 2025-01-28T14:47:03.688Z [INFO] server-cc998fce-243f-d7c9-9c87-63072b683c42: pipelining replication: peer="{Voter server-f7b027f1-68b8-2216-8157-cedc103a184e f7b027f1-68b8-2216-8157-cedc103a184e}" 2025-01-28T14:47:03.688Z [INFO] server-cc998fce-243f-d7c9-9c87-63072b683c42: pipelining replication: peer="{Voter server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9 bbc63f83-1cb3-4bcf-2919-31b9c55552e9}" 2025-01-28T14:47:03.792Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:03.687126977 +0000 UTC m=+6.770533001 (1 nodes), 61.932505ms from start of poll, 66.840984ms from cluster start. Timeout at 2025-01-28 14:47:03.792146644 +0000 UTC m=+6.875553668, 105.020667ms after stability 2025-01-28T14:47:03.845Z [INFO] server-cc998fce-243f-d7c9-9c87-63072b683c42: aborting pipeline replication: peer="{Voter server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9 bbc63f83-1cb3-4bcf-2919-31b9c55552e9}" 2025-01-28T14:47:03.845Z [INFO] server-cc998fce-243f-d7c9-9c87-63072b683c42: aborting pipeline replication: peer="{Voter server-f7b027f1-68b8-2216-8157-cedc103a184e f7b027f1-68b8-2216-8157-cedc103a184e}" 2025-01-28T14:47:03.846Z [INFO] creating new snapshot: path=/tmp/raft3439626179/snapshots/2-27-1738075623846.tmp 2025-01-28T14:47:03.855Z [INFO] server-f7b027f1-68b8-2216-8157-cedc103a184e: starting restore from snapshot: id=2-27-1738075623846 last-index=27 last-term=2 size-in-bytes=168 2025-01-28T14:47:03.858Z [INFO] server-f7b027f1-68b8-2216-8157-cedc103a184e: snapshot restore progress: id=2-27-1738075623846 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2025-01-28T14:47:03.858Z [INFO] server-f7b027f1-68b8-2216-8157-cedc103a184e: restored from snapshot: id=2-27-1738075623846 last-index=27 last-term=2 size-in-bytes=168 2025-01-28T14:47:03.858Z [INFO] server-f7b027f1-68b8-2216-8157-cedc103a184e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f7b027f1-68b8-2216-8157-cedc103a184e Address:f7b027f1-68b8-2216-8157-cedc103a184e} {Suffrage:Voter ID:server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9 Address:bbc63f83-1cb3-4bcf-2919-31b9c55552e9} {Suffrage:Voter ID:server-cc998fce-243f-d7c9-9c87-63072b683c42 Address:cc998fce-243f-d7c9-9c87-63072b683c42}]" 2025-01-28T14:47:03.859Z [INFO] server-f7b027f1-68b8-2216-8157-cedc103a184e: entering follower state: follower="Node at f7b027f1-68b8-2216-8157-cedc103a184e [Follower]" leader-address= leader-id= 2025-01-28T14:47:03.860Z [INFO] creating new snapshot: path=/tmp/raft1996937746/snapshots/2-27-1738075623860.tmp 2025-01-28T14:47:03.869Z [INFO] server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9: starting restore from snapshot: id=2-27-1738075623860 last-index=27 last-term=2 size-in-bytes=168 2025-01-28T14:47:03.871Z [INFO] server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9: snapshot restore progress: id=2-27-1738075623860 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2025-01-28T14:47:03.872Z [INFO] server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9: restored from snapshot: id=2-27-1738075623860 last-index=27 last-term=2 size-in-bytes=168 2025-01-28T14:47:03.872Z [INFO] server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f7b027f1-68b8-2216-8157-cedc103a184e Address:f7b027f1-68b8-2216-8157-cedc103a184e} {Suffrage:Voter ID:server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9 Address:bbc63f83-1cb3-4bcf-2919-31b9c55552e9} {Suffrage:Voter ID:server-cc998fce-243f-d7c9-9c87-63072b683c42 Address:cc998fce-243f-d7c9-9c87-63072b683c42}]" 2025-01-28T14:47:03.873Z [INFO] server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9: entering follower state: follower="Node at bbc63f83-1cb3-4bcf-2919-31b9c55552e9 [Follower]" leader-address= leader-id= 2025-01-28T14:47:03.874Z [INFO] creating new snapshot: path=/tmp/raft2051273926/snapshots/2-27-1738075623874.tmp 2025-01-28T14:47:03.881Z [INFO] server-cc998fce-243f-d7c9-9c87-63072b683c42: starting restore from snapshot: id=2-27-1738075623874 last-index=27 last-term=2 size-in-bytes=168 2025-01-28T14:47:03.883Z [INFO] server-cc998fce-243f-d7c9-9c87-63072b683c42: snapshot restore progress: id=2-27-1738075623874 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2025-01-28T14:47:03.883Z [INFO] server-cc998fce-243f-d7c9-9c87-63072b683c42: restored from snapshot: id=2-27-1738075623874 last-index=27 last-term=2 size-in-bytes=168 2025-01-28T14:47:03.883Z [INFO] server-cc998fce-243f-d7c9-9c87-63072b683c42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f7b027f1-68b8-2216-8157-cedc103a184e Address:f7b027f1-68b8-2216-8157-cedc103a184e} {Suffrage:Voter ID:server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9 Address:bbc63f83-1cb3-4bcf-2919-31b9c55552e9} {Suffrage:Voter ID:server-cc998fce-243f-d7c9-9c87-63072b683c42 Address:cc998fce-243f-d7c9-9c87-63072b683c42}]" 2025-01-28T14:47:03.883Z [INFO] server-cc998fce-243f-d7c9-9c87-63072b683c42: entering follower state: follower="Node at cc998fce-243f-d7c9-9c87-63072b683c42 [Follower]" leader-address= leader-id= 2025-01-28T14:47:03.923Z [WARN] server-f7b027f1-68b8-2216-8157-cedc103a184e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:03.923Z [INFO] server-f7b027f1-68b8-2216-8157-cedc103a184e: entering candidate state: node="Node at f7b027f1-68b8-2216-8157-cedc103a184e [Candidate]" term=3 2025-01-28T14:47:03.923Z [INFO] server-f7b027f1-68b8-2216-8157-cedc103a184e: election won: term=3 tally=2 2025-01-28T14:47:03.924Z [INFO] server-f7b027f1-68b8-2216-8157-cedc103a184e: entering leader state: leader="Node at f7b027f1-68b8-2216-8157-cedc103a184e [Leader]" 2025-01-28T14:47:03.924Z [INFO] server-f7b027f1-68b8-2216-8157-cedc103a184e: added peer, starting replication: peer=server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9 2025-01-28T14:47:03.924Z [INFO] server-f7b027f1-68b8-2216-8157-cedc103a184e: added peer, starting replication: peer=server-cc998fce-243f-d7c9-9c87-63072b683c42 2025-01-28T14:47:03.925Z [INFO] server-f7b027f1-68b8-2216-8157-cedc103a184e: pipelining replication: peer="{Voter server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9 bbc63f83-1cb3-4bcf-2919-31b9c55552e9}" 2025-01-28T14:47:03.925Z [INFO] server-f7b027f1-68b8-2216-8157-cedc103a184e: pipelining replication: peer="{Voter server-cc998fce-243f-d7c9-9c87-63072b683c42 cc998fce-243f-d7c9-9c87-63072b683c42}" 2025-01-28T14:47:04.198Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:04.305Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:04.199727528 +0000 UTC m=+7.283134219 (1 nodes), 443.679µs from start of poll, 579.442202ms from cluster start. Timeout at 2025-01-28 14:47:04.304887199 +0000 UTC m=+7.388293224, 105.159005ms after stability 2025-01-28T14:47:04.306Z [INFO] server-f7b027f1-68b8-2216-8157-cedc103a184e: aborting pipeline replication: peer="{Voter server-cc998fce-243f-d7c9-9c87-63072b683c42 cc998fce-243f-d7c9-9c87-63072b683c42}" 2025-01-28T14:47:04.306Z [INFO] server-f7b027f1-68b8-2216-8157-cedc103a184e: aborting pipeline replication: peer="{Voter server-bbc63f83-1cb3-4bcf-2919-31b9c55552e9 bbc63f83-1cb3-4bcf-2919-31b9c55552e9}" 2025-01-28T14:47:04.807Z [ERROR] server-f7b027f1-68b8-2216-8157-cedc103a184e: failed to heartbeat to: peer=bbc63f83-1cb3-4bcf-2919-31b9c55552e9 backoff time=10ms error="command timed out" --- PASS: TestRaft_RecoverCluster (3.01s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (1.15s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.66s) --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (1.21s) === RUN TestRaft_HasExistingState raft_test.go:258: Skipping unreliable test --- SKIP: TestRaft_HasExistingState (0.00s) === RUN TestRaft_SingleNode 2025-01-28T14:47:04.825Z [INFO] server-1e288a7f-7f12-3fc8-e3fd-cdd537b4d129: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e288a7f-7f12-3fc8-e3fd-cdd537b4d129 Address:1e288a7f-7f12-3fc8-e3fd-cdd537b4d129}]" 2025-01-28T14:47:04.826Z [INFO] server-1e288a7f-7f12-3fc8-e3fd-cdd537b4d129: entering follower state: follower="Node at 1e288a7f-7f12-3fc8-e3fd-cdd537b4d129 [Follower]" leader-address= leader-id= 2025-01-28T14:47:04.906Z [WARN] server-1e288a7f-7f12-3fc8-e3fd-cdd537b4d129: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:04.906Z [INFO] server-1e288a7f-7f12-3fc8-e3fd-cdd537b4d129: entering candidate state: node="Node at 1e288a7f-7f12-3fc8-e3fd-cdd537b4d129 [Candidate]" term=2 2025-01-28T14:47:04.907Z [INFO] server-1e288a7f-7f12-3fc8-e3fd-cdd537b4d129: election won: term=2 tally=1 2025-01-28T14:47:04.907Z [INFO] server-1e288a7f-7f12-3fc8-e3fd-cdd537b4d129: entering leader state: leader="Node at 1e288a7f-7f12-3fc8-e3fd-cdd537b4d129 [Leader]" --- PASS: TestRaft_SingleNode (0.09s) === RUN TestRaft_TripleNode 2025-01-28T14:47:04.922Z [INFO] server-54ffa07a-76e2-8807-cd00-76ea5a156cd5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54ffa07a-76e2-8807-cd00-76ea5a156cd5 Address:54ffa07a-76e2-8807-cd00-76ea5a156cd5} {Suffrage:Voter ID:server-b8987b37-93de-fc75-3558-7f04ffd4418d Address:b8987b37-93de-fc75-3558-7f04ffd4418d} {Suffrage:Voter ID:server-d1070395-f174-c0ce-f8cb-cf3543699be1 Address:d1070395-f174-c0ce-f8cb-cf3543699be1}]" 2025-01-28T14:47:04.923Z [INFO] server-54ffa07a-76e2-8807-cd00-76ea5a156cd5: entering follower state: follower="Node at 54ffa07a-76e2-8807-cd00-76ea5a156cd5 [Follower]" leader-address= leader-id= 2025-01-28T14:47:04.924Z [INFO] server-b8987b37-93de-fc75-3558-7f04ffd4418d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54ffa07a-76e2-8807-cd00-76ea5a156cd5 Address:54ffa07a-76e2-8807-cd00-76ea5a156cd5} {Suffrage:Voter ID:server-b8987b37-93de-fc75-3558-7f04ffd4418d Address:b8987b37-93de-fc75-3558-7f04ffd4418d} {Suffrage:Voter ID:server-d1070395-f174-c0ce-f8cb-cf3543699be1 Address:d1070395-f174-c0ce-f8cb-cf3543699be1}]" 2025-01-28T14:47:04.924Z [INFO] server-b8987b37-93de-fc75-3558-7f04ffd4418d: entering follower state: follower="Node at b8987b37-93de-fc75-3558-7f04ffd4418d [Follower]" leader-address= leader-id= 2025-01-28T14:47:04.925Z [INFO] server-d1070395-f174-c0ce-f8cb-cf3543699be1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54ffa07a-76e2-8807-cd00-76ea5a156cd5 Address:54ffa07a-76e2-8807-cd00-76ea5a156cd5} {Suffrage:Voter ID:server-b8987b37-93de-fc75-3558-7f04ffd4418d Address:b8987b37-93de-fc75-3558-7f04ffd4418d} {Suffrage:Voter ID:server-d1070395-f174-c0ce-f8cb-cf3543699be1 Address:d1070395-f174-c0ce-f8cb-cf3543699be1}]" 2025-01-28T14:47:04.925Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:04.925Z [INFO] server-d1070395-f174-c0ce-f8cb-cf3543699be1: entering follower state: follower="Node at d1070395-f174-c0ce-f8cb-cf3543699be1 [Follower]" leader-address= leader-id= 2025-01-28T14:47:04.983Z [WARN] server-d1070395-f174-c0ce-f8cb-cf3543699be1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:04.983Z [INFO] server-d1070395-f174-c0ce-f8cb-cf3543699be1: entering candidate state: node="Node at d1070395-f174-c0ce-f8cb-cf3543699be1 [Candidate]" term=2 2025-01-28T14:47:04.984Z [INFO] server-d1070395-f174-c0ce-f8cb-cf3543699be1: election won: term=2 tally=2 2025-01-28T14:47:04.984Z [INFO] server-d1070395-f174-c0ce-f8cb-cf3543699be1: entering leader state: leader="Node at d1070395-f174-c0ce-f8cb-cf3543699be1 [Leader]" 2025-01-28T14:47:04.985Z [INFO] server-d1070395-f174-c0ce-f8cb-cf3543699be1: added peer, starting replication: peer=server-54ffa07a-76e2-8807-cd00-76ea5a156cd5 2025-01-28T14:47:04.985Z [INFO] server-d1070395-f174-c0ce-f8cb-cf3543699be1: added peer, starting replication: peer=server-b8987b37-93de-fc75-3558-7f04ffd4418d 2025-01-28T14:47:04.986Z [INFO] server-d1070395-f174-c0ce-f8cb-cf3543699be1: pipelining replication: peer="{Voter server-54ffa07a-76e2-8807-cd00-76ea5a156cd5 54ffa07a-76e2-8807-cd00-76ea5a156cd5}" 2025-01-28T14:47:04.987Z [INFO] server-d1070395-f174-c0ce-f8cb-cf3543699be1: pipelining replication: peer="{Voter server-b8987b37-93de-fc75-3558-7f04ffd4418d b8987b37-93de-fc75-3558-7f04ffd4418d}" 2025-01-28T14:47:05.090Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:04.98517618 +0000 UTC m=+8.068582871 (2 nodes), 59.114422ms from start of poll, 63.945898ms from cluster start. Timeout at 2025-01-28 14:47:05.090215514 +0000 UTC m=+8.173621539, 105.038668ms after stability 2025-01-28T14:47:05.091Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:05.197Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:05.091284663 +0000 UTC m=+8.174690354 (1 nodes), 12.667µs from start of poll, 170.053381ms from cluster start. Timeout at 2025-01-28 14:47:05.196366333 +0000 UTC m=+8.279772691, 105.082337ms after stability 2025-01-28T14:47:05.212Z [INFO] server-d1070395-f174-c0ce-f8cb-cf3543699be1: aborting pipeline replication: peer="{Voter server-b8987b37-93de-fc75-3558-7f04ffd4418d b8987b37-93de-fc75-3558-7f04ffd4418d}" 2025-01-28T14:47:05.212Z [INFO] server-d1070395-f174-c0ce-f8cb-cf3543699be1: aborting pipeline replication: peer="{Voter server-54ffa07a-76e2-8807-cd00-76ea5a156cd5 54ffa07a-76e2-8807-cd00-76ea5a156cd5}" 2025-01-28T14:47:05.714Z [ERROR] server-d1070395-f174-c0ce-f8cb-cf3543699be1: failed to heartbeat to: peer=b8987b37-93de-fc75-3558-7f04ffd4418d backoff time=10ms error="command timed out" --- PASS: TestRaft_TripleNode (0.81s) === RUN TestRaft_LeaderFail 2025-01-28T14:47:05.732Z [INFO] server-e86384bf-8c6b-4166-35b1-83d62a5346db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e86384bf-8c6b-4166-35b1-83d62a5346db Address:e86384bf-8c6b-4166-35b1-83d62a5346db} {Suffrage:Voter ID:server-a4321908-d7fe-45f3-5b12-771cfcb77ecf Address:a4321908-d7fe-45f3-5b12-771cfcb77ecf} {Suffrage:Voter ID:server-fcf8ed41-29dd-91bd-1de3-270377a51fba Address:fcf8ed41-29dd-91bd-1de3-270377a51fba}]" 2025-01-28T14:47:05.733Z [INFO] server-e86384bf-8c6b-4166-35b1-83d62a5346db: entering follower state: follower="Node at e86384bf-8c6b-4166-35b1-83d62a5346db [Follower]" leader-address= leader-id= 2025-01-28T14:47:05.734Z [INFO] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e86384bf-8c6b-4166-35b1-83d62a5346db Address:e86384bf-8c6b-4166-35b1-83d62a5346db} {Suffrage:Voter ID:server-a4321908-d7fe-45f3-5b12-771cfcb77ecf Address:a4321908-d7fe-45f3-5b12-771cfcb77ecf} {Suffrage:Voter ID:server-fcf8ed41-29dd-91bd-1de3-270377a51fba Address:fcf8ed41-29dd-91bd-1de3-270377a51fba}]" 2025-01-28T14:47:05.734Z [INFO] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: entering follower state: follower="Node at a4321908-d7fe-45f3-5b12-771cfcb77ecf [Follower]" leader-address= leader-id= 2025-01-28T14:47:05.735Z [INFO] server-fcf8ed41-29dd-91bd-1de3-270377a51fba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e86384bf-8c6b-4166-35b1-83d62a5346db Address:e86384bf-8c6b-4166-35b1-83d62a5346db} {Suffrage:Voter ID:server-a4321908-d7fe-45f3-5b12-771cfcb77ecf Address:a4321908-d7fe-45f3-5b12-771cfcb77ecf} {Suffrage:Voter ID:server-fcf8ed41-29dd-91bd-1de3-270377a51fba Address:fcf8ed41-29dd-91bd-1de3-270377a51fba}]" 2025-01-28T14:47:05.735Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:05.736Z [INFO] server-fcf8ed41-29dd-91bd-1de3-270377a51fba: entering follower state: follower="Node at fcf8ed41-29dd-91bd-1de3-270377a51fba [Follower]" leader-address= leader-id= 2025-01-28T14:47:05.797Z [WARN] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:05.797Z [INFO] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: entering candidate state: node="Node at a4321908-d7fe-45f3-5b12-771cfcb77ecf [Candidate]" term=2 2025-01-28T14:47:05.798Z [INFO] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: election won: term=2 tally=2 2025-01-28T14:47:05.798Z [INFO] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: entering leader state: leader="Node at a4321908-d7fe-45f3-5b12-771cfcb77ecf [Leader]" 2025-01-28T14:47:05.798Z [INFO] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: added peer, starting replication: peer=server-e86384bf-8c6b-4166-35b1-83d62a5346db 2025-01-28T14:47:05.799Z [INFO] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: added peer, starting replication: peer=server-fcf8ed41-29dd-91bd-1de3-270377a51fba 2025-01-28T14:47:05.800Z [INFO] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: pipelining replication: peer="{Voter server-e86384bf-8c6b-4166-35b1-83d62a5346db e86384bf-8c6b-4166-35b1-83d62a5346db}" 2025-01-28T14:47:05.801Z [INFO] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: pipelining replication: peer="{Voter server-fcf8ed41-29dd-91bd-1de3-270377a51fba fcf8ed41-29dd-91bd-1de3-270377a51fba}" 2025-01-28T14:47:05.904Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:05.798934339 +0000 UTC m=+8.882340697 (2 nodes), 62.799532ms from start of poll, 67.630675ms from cluster start. Timeout at 2025-01-28 14:47:05.903956339 +0000 UTC m=+8.987362030, 105.021333ms after stability 2025-01-28T14:47:05.904Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:06.010Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:05.90485315 +0000 UTC m=+8.988258841 (1 nodes), 11µs from start of poll, 173.548819ms from cluster start. Timeout at 2025-01-28 14:47:06.009907151 +0000 UTC m=+9.093313509, 105.054668ms after stability raft_test.go:378: [INFO] Disconnecting Node at a4321908-d7fe-45f3-5b12-771cfcb77ecf [Leader] 2025-01-28T14:47:06.029Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to pipeline appendEntries: peer="{Voter server-fcf8ed41-29dd-91bd-1de3-270377a51fba fcf8ed41-29dd-91bd-1de3-270377a51fba}" error="append pipeline closed" 2025-01-28T14:47:06.029Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to pipeline appendEntries: peer="{Voter server-e86384bf-8c6b-4166-35b1-83d62a5346db e86384bf-8c6b-4166-35b1-83d62a5346db}" error="append pipeline closed" 2025-01-28T14:47:06.029Z [INFO] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: aborting pipeline replication: peer="{Voter server-e86384bf-8c6b-4166-35b1-83d62a5346db e86384bf-8c6b-4166-35b1-83d62a5346db}" 2025-01-28T14:47:06.030Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:06.029Z [INFO] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: aborting pipeline replication: peer="{Voter server-fcf8ed41-29dd-91bd-1de3-270377a51fba fcf8ed41-29dd-91bd-1de3-270377a51fba}" 2025-01-28T14:47:06.030Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to heartbeat to: peer=fcf8ed41-29dd-91bd-1de3-270377a51fba backoff time=10ms error="failed to connect to peer: fcf8ed41-29dd-91bd-1de3-270377a51fba" 2025-01-28T14:47:06.033Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to heartbeat to: peer=e86384bf-8c6b-4166-35b1-83d62a5346db backoff time=10ms error="failed to connect to peer: e86384bf-8c6b-4166-35b1-83d62a5346db" 2025-01-28T14:47:06.037Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to appendEntries to: peer="{Voter server-e86384bf-8c6b-4166-35b1-83d62a5346db e86384bf-8c6b-4166-35b1-83d62a5346db}" error="failed to connect to peer: e86384bf-8c6b-4166-35b1-83d62a5346db" 2025-01-28T14:47:06.041Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to appendEntries to: peer="{Voter server-fcf8ed41-29dd-91bd-1de3-270377a51fba fcf8ed41-29dd-91bd-1de3-270377a51fba}" error="failed to connect to peer: fcf8ed41-29dd-91bd-1de3-270377a51fba" 2025-01-28T14:47:06.050Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to heartbeat to: peer=fcf8ed41-29dd-91bd-1de3-270377a51fba backoff time=10ms error="failed to connect to peer: fcf8ed41-29dd-91bd-1de3-270377a51fba" 2025-01-28T14:47:06.053Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to heartbeat to: peer=e86384bf-8c6b-4166-35b1-83d62a5346db backoff time=10ms error="failed to connect to peer: e86384bf-8c6b-4166-35b1-83d62a5346db" 2025-01-28T14:47:06.055Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to appendEntries to: peer="{Voter server-e86384bf-8c6b-4166-35b1-83d62a5346db e86384bf-8c6b-4166-35b1-83d62a5346db}" error="failed to connect to peer: e86384bf-8c6b-4166-35b1-83d62a5346db" 2025-01-28T14:47:06.061Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to appendEntries to: peer="{Voter server-fcf8ed41-29dd-91bd-1de3-270377a51fba fcf8ed41-29dd-91bd-1de3-270377a51fba}" error="failed to connect to peer: fcf8ed41-29dd-91bd-1de3-270377a51fba" 2025-01-28T14:47:06.069Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to heartbeat to: peer=fcf8ed41-29dd-91bd-1de3-270377a51fba backoff time=10ms error="failed to connect to peer: fcf8ed41-29dd-91bd-1de3-270377a51fba" 2025-01-28T14:47:06.073Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to heartbeat to: peer=e86384bf-8c6b-4166-35b1-83d62a5346db backoff time=10ms error="failed to connect to peer: e86384bf-8c6b-4166-35b1-83d62a5346db" 2025-01-28T14:47:06.073Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to appendEntries to: peer="{Voter server-e86384bf-8c6b-4166-35b1-83d62a5346db e86384bf-8c6b-4166-35b1-83d62a5346db}" error="failed to connect to peer: e86384bf-8c6b-4166-35b1-83d62a5346db" 2025-01-28T14:47:06.074Z [WARN] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to contact: server-id=server-e86384bf-8c6b-4166-35b1-83d62a5346db time=50.701172ms 2025-01-28T14:47:06.074Z [WARN] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to contact: server-id=server-fcf8ed41-29dd-91bd-1de3-270377a51fba time=51.33019ms 2025-01-28T14:47:06.075Z [WARN] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to contact quorum of nodes, stepping down 2025-01-28T14:47:06.075Z [INFO] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: entering follower state: follower="Node at a4321908-d7fe-45f3-5b12-771cfcb77ecf [Follower]" leader-address= leader-id= 2025-01-28T14:47:06.080Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to appendEntries to: peer="{Voter server-fcf8ed41-29dd-91bd-1de3-270377a51fba fcf8ed41-29dd-91bd-1de3-270377a51fba}" error="failed to connect to peer: fcf8ed41-29dd-91bd-1de3-270377a51fba" 2025-01-28T14:47:06.109Z [WARN] server-e86384bf-8c6b-4166-35b1-83d62a5346db: heartbeat timeout reached, starting election: last-leader-addr=a4321908-d7fe-45f3-5b12-771cfcb77ecf last-leader-id=server-a4321908-d7fe-45f3-5b12-771cfcb77ecf 2025-01-28T14:47:06.109Z [INFO] server-e86384bf-8c6b-4166-35b1-83d62a5346db: entering candidate state: node="Node at e86384bf-8c6b-4166-35b1-83d62a5346db [Candidate]" term=3 2025-01-28T14:47:06.110Z [WARN] server-fcf8ed41-29dd-91bd-1de3-270377a51fba: rejecting vote request since we have a leader: from=e86384bf-8c6b-4166-35b1-83d62a5346db leader=a4321908-d7fe-45f3-5b12-771cfcb77ecf leader-id=server-a4321908-d7fe-45f3-5b12-771cfcb77ecf 2025-01-28T14:47:06.110Z [ERROR] server-e86384bf-8c6b-4166-35b1-83d62a5346db: failed to make requestVote RPC: target="{Voter server-a4321908-d7fe-45f3-5b12-771cfcb77ecf a4321908-d7fe-45f3-5b12-771cfcb77ecf}" error="failed to connect to peer: a4321908-d7fe-45f3-5b12-771cfcb77ecf" term=3 2025-01-28T14:47:06.142Z [WARN] server-fcf8ed41-29dd-91bd-1de3-270377a51fba: heartbeat timeout reached, starting election: last-leader-addr=a4321908-d7fe-45f3-5b12-771cfcb77ecf last-leader-id=server-a4321908-d7fe-45f3-5b12-771cfcb77ecf 2025-01-28T14:47:06.142Z [INFO] server-fcf8ed41-29dd-91bd-1de3-270377a51fba: entering candidate state: node="Node at fcf8ed41-29dd-91bd-1de3-270377a51fba [Candidate]" term=3 2025-01-28T14:47:06.142Z [INFO] server-e86384bf-8c6b-4166-35b1-83d62a5346db: duplicate requestVote for same term: term=3 2025-01-28T14:47:06.142Z [ERROR] server-fcf8ed41-29dd-91bd-1de3-270377a51fba: failed to make requestVote RPC: target="{Voter server-a4321908-d7fe-45f3-5b12-771cfcb77ecf a4321908-d7fe-45f3-5b12-771cfcb77ecf}" error="failed to connect to peer: a4321908-d7fe-45f3-5b12-771cfcb77ecf" term=3 2025-01-28T14:47:06.166Z [WARN] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:06.166Z [INFO] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: entering candidate state: node="Node at a4321908-d7fe-45f3-5b12-771cfcb77ecf [Candidate]" term=3 2025-01-28T14:47:06.167Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to make requestVote RPC: target="{Voter server-fcf8ed41-29dd-91bd-1de3-270377a51fba fcf8ed41-29dd-91bd-1de3-270377a51fba}" error="failed to connect to peer: fcf8ed41-29dd-91bd-1de3-270377a51fba" term=3 2025-01-28T14:47:06.167Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to make requestVote RPC: target="{Voter server-e86384bf-8c6b-4166-35b1-83d62a5346db e86384bf-8c6b-4166-35b1-83d62a5346db}" error="failed to connect to peer: e86384bf-8c6b-4166-35b1-83d62a5346db" term=3 2025-01-28T14:47:06.183Z [WARN] server-e86384bf-8c6b-4166-35b1-83d62a5346db: Election timeout reached, restarting election 2025-01-28T14:47:06.183Z [INFO] server-e86384bf-8c6b-4166-35b1-83d62a5346db: entering candidate state: node="Node at e86384bf-8c6b-4166-35b1-83d62a5346db [Candidate]" term=4 2025-01-28T14:47:06.183Z [ERROR] server-e86384bf-8c6b-4166-35b1-83d62a5346db: failed to make requestVote RPC: target="{Voter server-a4321908-d7fe-45f3-5b12-771cfcb77ecf a4321908-d7fe-45f3-5b12-771cfcb77ecf}" error="failed to connect to peer: a4321908-d7fe-45f3-5b12-771cfcb77ecf" term=4 2025-01-28T14:47:06.183Z [INFO] server-fcf8ed41-29dd-91bd-1de3-270377a51fba: entering follower state: follower="Node at fcf8ed41-29dd-91bd-1de3-270377a51fba [Follower]" leader-address= leader-id= 2025-01-28T14:47:06.183Z [INFO] server-e86384bf-8c6b-4166-35b1-83d62a5346db: election won: term=4 tally=2 2025-01-28T14:47:06.183Z [INFO] server-e86384bf-8c6b-4166-35b1-83d62a5346db: entering leader state: leader="Node at e86384bf-8c6b-4166-35b1-83d62a5346db [Leader]" 2025-01-28T14:47:06.183Z [INFO] server-e86384bf-8c6b-4166-35b1-83d62a5346db: added peer, starting replication: peer=server-a4321908-d7fe-45f3-5b12-771cfcb77ecf 2025-01-28T14:47:06.184Z [INFO] server-e86384bf-8c6b-4166-35b1-83d62a5346db: added peer, starting replication: peer=server-fcf8ed41-29dd-91bd-1de3-270377a51fba 2025-01-28T14:47:06.184Z [ERROR] server-e86384bf-8c6b-4166-35b1-83d62a5346db: failed to appendEntries to: peer="{Voter server-a4321908-d7fe-45f3-5b12-771cfcb77ecf a4321908-d7fe-45f3-5b12-771cfcb77ecf}" error="failed to connect to peer: a4321908-d7fe-45f3-5b12-771cfcb77ecf" 2025-01-28T14:47:06.184Z [INFO] server-e86384bf-8c6b-4166-35b1-83d62a5346db: pipelining replication: peer="{Voter server-fcf8ed41-29dd-91bd-1de3-270377a51fba fcf8ed41-29dd-91bd-1de3-270377a51fba}" 2025-01-28T14:47:06.192Z [ERROR] server-e86384bf-8c6b-4166-35b1-83d62a5346db: failed to heartbeat to: peer=a4321908-d7fe-45f3-5b12-771cfcb77ecf backoff time=10ms error="failed to connect to peer: a4321908-d7fe-45f3-5b12-771cfcb77ecf" 2025-01-28T14:47:06.194Z [ERROR] server-e86384bf-8c6b-4166-35b1-83d62a5346db: failed to appendEntries to: peer="{Voter server-a4321908-d7fe-45f3-5b12-771cfcb77ecf a4321908-d7fe-45f3-5b12-771cfcb77ecf}" error="failed to connect to peer: a4321908-d7fe-45f3-5b12-771cfcb77ecf" 2025-01-28T14:47:06.211Z [ERROR] server-e86384bf-8c6b-4166-35b1-83d62a5346db: failed to heartbeat to: peer=a4321908-d7fe-45f3-5b12-771cfcb77ecf backoff time=10ms error="failed to connect to peer: a4321908-d7fe-45f3-5b12-771cfcb77ecf" 2025-01-28T14:47:06.213Z [ERROR] server-e86384bf-8c6b-4166-35b1-83d62a5346db: failed to appendEntries to: peer="{Voter server-a4321908-d7fe-45f3-5b12-771cfcb77ecf a4321908-d7fe-45f3-5b12-771cfcb77ecf}" error="failed to connect to peer: a4321908-d7fe-45f3-5b12-771cfcb77ecf" 2025-01-28T14:47:06.232Z [ERROR] server-e86384bf-8c6b-4166-35b1-83d62a5346db: failed to heartbeat to: peer=a4321908-d7fe-45f3-5b12-771cfcb77ecf backoff time=10ms error="failed to connect to peer: a4321908-d7fe-45f3-5b12-771cfcb77ecf" 2025-01-28T14:47:06.234Z [WARN] server-e86384bf-8c6b-4166-35b1-83d62a5346db: failed to contact: server-id=server-a4321908-d7fe-45f3-5b12-771cfcb77ecf time=50.907845ms 2025-01-28T14:47:06.241Z [ERROR] server-e86384bf-8c6b-4166-35b1-83d62a5346db: failed to appendEntries to: peer="{Voter server-a4321908-d7fe-45f3-5b12-771cfcb77ecf a4321908-d7fe-45f3-5b12-771cfcb77ecf}" error="failed to connect to peer: a4321908-d7fe-45f3-5b12-771cfcb77ecf" 2025-01-28T14:47:06.245Z [WARN] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: Election timeout reached, restarting election 2025-01-28T14:47:06.245Z [INFO] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: entering candidate state: node="Node at a4321908-d7fe-45f3-5b12-771cfcb77ecf [Candidate]" term=4 2025-01-28T14:47:06.246Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to make requestVote RPC: target="{Voter server-e86384bf-8c6b-4166-35b1-83d62a5346db e86384bf-8c6b-4166-35b1-83d62a5346db}" error="failed to connect to peer: e86384bf-8c6b-4166-35b1-83d62a5346db" term=4 2025-01-28T14:47:06.246Z [ERROR] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: failed to make requestVote RPC: target="{Voter server-fcf8ed41-29dd-91bd-1de3-270377a51fba fcf8ed41-29dd-91bd-1de3-270377a51fba}" error="failed to connect to peer: fcf8ed41-29dd-91bd-1de3-270377a51fba" term=4 2025-01-28T14:47:06.250Z [ERROR] server-e86384bf-8c6b-4166-35b1-83d62a5346db: failed to heartbeat to: peer=a4321908-d7fe-45f3-5b12-771cfcb77ecf backoff time=20ms error="failed to connect to peer: a4321908-d7fe-45f3-5b12-771cfcb77ecf" 2025-01-28T14:47:06.279Z [ERROR] server-e86384bf-8c6b-4166-35b1-83d62a5346db: failed to heartbeat to: peer=a4321908-d7fe-45f3-5b12-771cfcb77ecf backoff time=25ms error="failed to connect to peer: a4321908-d7fe-45f3-5b12-771cfcb77ecf" 2025-01-28T14:47:06.281Z [WARN] server-e86384bf-8c6b-4166-35b1-83d62a5346db: failed to contact: server-id=server-a4321908-d7fe-45f3-5b12-771cfcb77ecf time=97.76957ms 2025-01-28T14:47:06.289Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:06.183963436 +0000 UTC m=+9.267368461 (1 nodes), 153.731564ms from start of poll, 452.658439ms from cluster start. Timeout at 2025-01-28 14:47:06.28898277 +0000 UTC m=+9.372388461, 105.02ms after stability 2025-01-28T14:47:06.291Z [ERROR] server-e86384bf-8c6b-4166-35b1-83d62a5346db: failed to appendEntries to: peer="{Voter server-a4321908-d7fe-45f3-5b12-771cfcb77ecf a4321908-d7fe-45f3-5b12-771cfcb77ecf}" error="failed to connect to peer: a4321908-d7fe-45f3-5b12-771cfcb77ecf" raft_test.go:413: [INFO] Reconnecting Node at a4321908-d7fe-45f3-5b12-771cfcb77ecf [Candidate] 2025-01-28T14:47:06.312Z [INFO] server-a4321908-d7fe-45f3-5b12-771cfcb77ecf: entering follower state: follower="Node at a4321908-d7fe-45f3-5b12-771cfcb77ecf [Follower]" leader-address=e86384bf-8c6b-4166-35b1-83d62a5346db leader-id=server-e86384bf-8c6b-4166-35b1-83d62a5346db 2025-01-28T14:47:06.372Z [INFO] server-e86384bf-8c6b-4166-35b1-83d62a5346db: pipelining replication: peer="{Voter server-a4321908-d7fe-45f3-5b12-771cfcb77ecf a4321908-d7fe-45f3-5b12-771cfcb77ecf}" 2025-01-28T14:47:06.378Z [INFO] server-e86384bf-8c6b-4166-35b1-83d62a5346db: aborting pipeline replication: peer="{Voter server-fcf8ed41-29dd-91bd-1de3-270377a51fba fcf8ed41-29dd-91bd-1de3-270377a51fba}" 2025-01-28T14:47:06.378Z [INFO] server-e86384bf-8c6b-4166-35b1-83d62a5346db: aborting pipeline replication: peer="{Voter server-a4321908-d7fe-45f3-5b12-771cfcb77ecf a4321908-d7fe-45f3-5b12-771cfcb77ecf}" --- PASS: TestRaft_LeaderFail (0.66s) === RUN TestRaft_BehindFollower 2025-01-28T14:47:06.397Z [INFO] server-27452489-e8d3-7a9f-381e-dd328430c8dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27452489-e8d3-7a9f-381e-dd328430c8dc Address:27452489-e8d3-7a9f-381e-dd328430c8dc} {Suffrage:Voter ID:server-84aa2645-417b-90ca-0165-0e53534a0f9b Address:84aa2645-417b-90ca-0165-0e53534a0f9b} {Suffrage:Voter ID:server-8230fc43-42c1-42b0-0f4e-84c48f032580 Address:8230fc43-42c1-42b0-0f4e-84c48f032580}]" 2025-01-28T14:47:06.397Z [INFO] server-27452489-e8d3-7a9f-381e-dd328430c8dc: entering follower state: follower="Node at 27452489-e8d3-7a9f-381e-dd328430c8dc [Follower]" leader-address= leader-id= 2025-01-28T14:47:06.398Z [INFO] server-84aa2645-417b-90ca-0165-0e53534a0f9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27452489-e8d3-7a9f-381e-dd328430c8dc Address:27452489-e8d3-7a9f-381e-dd328430c8dc} {Suffrage:Voter ID:server-84aa2645-417b-90ca-0165-0e53534a0f9b Address:84aa2645-417b-90ca-0165-0e53534a0f9b} {Suffrage:Voter ID:server-8230fc43-42c1-42b0-0f4e-84c48f032580 Address:8230fc43-42c1-42b0-0f4e-84c48f032580}]" 2025-01-28T14:47:06.399Z [INFO] server-84aa2645-417b-90ca-0165-0e53534a0f9b: entering follower state: follower="Node at 84aa2645-417b-90ca-0165-0e53534a0f9b [Follower]" leader-address= leader-id= 2025-01-28T14:47:06.400Z [INFO] server-8230fc43-42c1-42b0-0f4e-84c48f032580: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27452489-e8d3-7a9f-381e-dd328430c8dc Address:27452489-e8d3-7a9f-381e-dd328430c8dc} {Suffrage:Voter ID:server-84aa2645-417b-90ca-0165-0e53534a0f9b Address:84aa2645-417b-90ca-0165-0e53534a0f9b} {Suffrage:Voter ID:server-8230fc43-42c1-42b0-0f4e-84c48f032580 Address:8230fc43-42c1-42b0-0f4e-84c48f032580}]" 2025-01-28T14:47:06.400Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:06.401Z [INFO] server-8230fc43-42c1-42b0-0f4e-84c48f032580: entering follower state: follower="Node at 8230fc43-42c1-42b0-0f4e-84c48f032580 [Follower]" leader-address= leader-id= 2025-01-28T14:47:06.454Z [WARN] server-27452489-e8d3-7a9f-381e-dd328430c8dc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:06.454Z [INFO] server-27452489-e8d3-7a9f-381e-dd328430c8dc: entering candidate state: node="Node at 27452489-e8d3-7a9f-381e-dd328430c8dc [Candidate]" term=2 2025-01-28T14:47:06.455Z [INFO] server-27452489-e8d3-7a9f-381e-dd328430c8dc: election won: term=2 tally=2 2025-01-28T14:47:06.455Z [INFO] server-27452489-e8d3-7a9f-381e-dd328430c8dc: entering leader state: leader="Node at 27452489-e8d3-7a9f-381e-dd328430c8dc [Leader]" 2025-01-28T14:47:06.456Z [INFO] server-27452489-e8d3-7a9f-381e-dd328430c8dc: added peer, starting replication: peer=server-84aa2645-417b-90ca-0165-0e53534a0f9b 2025-01-28T14:47:06.456Z [INFO] server-27452489-e8d3-7a9f-381e-dd328430c8dc: added peer, starting replication: peer=server-8230fc43-42c1-42b0-0f4e-84c48f032580 2025-01-28T14:47:06.457Z [INFO] server-27452489-e8d3-7a9f-381e-dd328430c8dc: pipelining replication: peer="{Voter server-84aa2645-417b-90ca-0165-0e53534a0f9b 84aa2645-417b-90ca-0165-0e53534a0f9b}" 2025-01-28T14:47:06.457Z [INFO] server-27452489-e8d3-7a9f-381e-dd328430c8dc: pipelining replication: peer="{Voter server-8230fc43-42c1-42b0-0f4e-84c48f032580 8230fc43-42c1-42b0-0f4e-84c48f032580}" 2025-01-28T14:47:06.562Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:06.456151517 +0000 UTC m=+9.539558208 (1 nodes), 55.098969ms from start of poll, 59.97678ms from cluster start. Timeout at 2025-01-28 14:47:06.561190852 +0000 UTC m=+9.644596543, 105.038335ms after stability 2025-01-28T14:47:06.562Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:06.668Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:06.562911353 +0000 UTC m=+9.646317044 (2 nodes), 13.667µs from start of poll, 166.735616ms from cluster start. Timeout at 2025-01-28 14:47:06.667997355 +0000 UTC m=+9.751403380, 105.086336ms after stability 2025-01-28T14:47:06.669Z [ERROR] server-27452489-e8d3-7a9f-381e-dd328430c8dc: failed to pipeline appendEntries: peer="{Voter server-84aa2645-417b-90ca-0165-0e53534a0f9b 84aa2645-417b-90ca-0165-0e53534a0f9b}" error="append pipeline closed" 2025-01-28T14:47:06.669Z [INFO] server-27452489-e8d3-7a9f-381e-dd328430c8dc: aborting pipeline replication: peer="{Voter server-84aa2645-417b-90ca-0165-0e53534a0f9b 84aa2645-417b-90ca-0165-0e53534a0f9b}" 2025-01-28T14:47:06.670Z [ERROR] server-27452489-e8d3-7a9f-381e-dd328430c8dc: failed to appendEntries to: peer="{Voter server-84aa2645-417b-90ca-0165-0e53534a0f9b 84aa2645-417b-90ca-0165-0e53534a0f9b}" error="failed to connect to peer: 84aa2645-417b-90ca-0165-0e53534a0f9b" 2025-01-28T14:47:06.673Z [ERROR] server-27452489-e8d3-7a9f-381e-dd328430c8dc: failed to heartbeat to: peer=84aa2645-417b-90ca-0165-0e53534a0f9b backoff time=10ms error="failed to connect to peer: 84aa2645-417b-90ca-0165-0e53534a0f9b" 2025-01-28T14:47:06.688Z [ERROR] server-27452489-e8d3-7a9f-381e-dd328430c8dc: failed to appendEntries to: peer="{Voter server-84aa2645-417b-90ca-0165-0e53534a0f9b 84aa2645-417b-90ca-0165-0e53534a0f9b}" error="failed to connect to peer: 84aa2645-417b-90ca-0165-0e53534a0f9b" 2025-01-28T14:47:06.698Z [ERROR] server-27452489-e8d3-7a9f-381e-dd328430c8dc: failed to heartbeat to: peer=84aa2645-417b-90ca-0165-0e53534a0f9b backoff time=10ms error="failed to connect to peer: 84aa2645-417b-90ca-0165-0e53534a0f9b" 2025-01-28T14:47:06.699Z [ERROR] server-27452489-e8d3-7a9f-381e-dd328430c8dc: failed to appendEntries to: peer="{Voter server-84aa2645-417b-90ca-0165-0e53534a0f9b 84aa2645-417b-90ca-0165-0e53534a0f9b}" error="failed to connect to peer: 84aa2645-417b-90ca-0165-0e53534a0f9b" 2025-01-28T14:47:06.716Z [ERROR] server-27452489-e8d3-7a9f-381e-dd328430c8dc: failed to heartbeat to: peer=84aa2645-417b-90ca-0165-0e53534a0f9b backoff time=10ms error="failed to connect to peer: 84aa2645-417b-90ca-0165-0e53534a0f9b" 2025-01-28T14:47:06.718Z [WARN] server-27452489-e8d3-7a9f-381e-dd328430c8dc: failed to contact: server-id=server-84aa2645-417b-90ca-0165-0e53534a0f9b time=50.244492ms 2025-01-28T14:47:06.720Z [ERROR] server-27452489-e8d3-7a9f-381e-dd328430c8dc: failed to appendEntries to: peer="{Voter server-84aa2645-417b-90ca-0165-0e53534a0f9b 84aa2645-417b-90ca-0165-0e53534a0f9b}" error="failed to connect to peer: 84aa2645-417b-90ca-0165-0e53534a0f9b" raft_test.go:459: [INFO] Finished apply without behind follower 2025-01-28T14:47:06.764Z [INFO] server-27452489-e8d3-7a9f-381e-dd328430c8dc: pipelining replication: peer="{Voter server-84aa2645-417b-90ca-0165-0e53534a0f9b 84aa2645-417b-90ca-0165-0e53534a0f9b}" 2025-01-28T14:47:06.773Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:06.880Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:06.774070955 +0000 UTC m=+9.857476647 (1 nodes), 12.334µs from start of poll, 377.895219ms from cluster start. Timeout at 2025-01-28 14:47:06.879167291 +0000 UTC m=+9.962572982, 105.096335ms after stability 2025-01-28T14:47:06.880Z [INFO] server-27452489-e8d3-7a9f-381e-dd328430c8dc: aborting pipeline replication: peer="{Voter server-8230fc43-42c1-42b0-0f4e-84c48f032580 8230fc43-42c1-42b0-0f4e-84c48f032580}" 2025-01-28T14:47:06.881Z [INFO] server-27452489-e8d3-7a9f-381e-dd328430c8dc: aborting pipeline replication: peer="{Voter server-84aa2645-417b-90ca-0165-0e53534a0f9b 84aa2645-417b-90ca-0165-0e53534a0f9b}" 2025-01-28T14:47:07.381Z [ERROR] server-27452489-e8d3-7a9f-381e-dd328430c8dc: failed to heartbeat to: peer=84aa2645-417b-90ca-0165-0e53534a0f9b backoff time=10ms error="command timed out" --- PASS: TestRaft_BehindFollower (1.00s) === RUN TestRaft_ApplyNonLeader 2025-01-28T14:47:07.400Z [INFO] server-70acce05-c77b-6d47-99c8-d16b2b1b214b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70acce05-c77b-6d47-99c8-d16b2b1b214b Address:70acce05-c77b-6d47-99c8-d16b2b1b214b} {Suffrage:Voter ID:server-dfa1fdab-b4a7-68ad-6a37-aaac9a70bb79 Address:dfa1fdab-b4a7-68ad-6a37-aaac9a70bb79} {Suffrage:Voter ID:server-c1998568-e227-8e8f-dd1d-64639753dbd5 Address:c1998568-e227-8e8f-dd1d-64639753dbd5}]" 2025-01-28T14:47:07.401Z [INFO] server-70acce05-c77b-6d47-99c8-d16b2b1b214b: entering follower state: follower="Node at 70acce05-c77b-6d47-99c8-d16b2b1b214b [Follower]" leader-address= leader-id= 2025-01-28T14:47:07.402Z [INFO] server-dfa1fdab-b4a7-68ad-6a37-aaac9a70bb79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70acce05-c77b-6d47-99c8-d16b2b1b214b Address:70acce05-c77b-6d47-99c8-d16b2b1b214b} {Suffrage:Voter ID:server-dfa1fdab-b4a7-68ad-6a37-aaac9a70bb79 Address:dfa1fdab-b4a7-68ad-6a37-aaac9a70bb79} {Suffrage:Voter ID:server-c1998568-e227-8e8f-dd1d-64639753dbd5 Address:c1998568-e227-8e8f-dd1d-64639753dbd5}]" 2025-01-28T14:47:07.402Z [INFO] server-dfa1fdab-b4a7-68ad-6a37-aaac9a70bb79: entering follower state: follower="Node at dfa1fdab-b4a7-68ad-6a37-aaac9a70bb79 [Follower]" leader-address= leader-id= 2025-01-28T14:47:07.403Z [INFO] server-c1998568-e227-8e8f-dd1d-64639753dbd5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70acce05-c77b-6d47-99c8-d16b2b1b214b Address:70acce05-c77b-6d47-99c8-d16b2b1b214b} {Suffrage:Voter ID:server-dfa1fdab-b4a7-68ad-6a37-aaac9a70bb79 Address:dfa1fdab-b4a7-68ad-6a37-aaac9a70bb79} {Suffrage:Voter ID:server-c1998568-e227-8e8f-dd1d-64639753dbd5 Address:c1998568-e227-8e8f-dd1d-64639753dbd5}]" 2025-01-28T14:47:07.404Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:07.404Z [INFO] server-c1998568-e227-8e8f-dd1d-64639753dbd5: entering follower state: follower="Node at c1998568-e227-8e8f-dd1d-64639753dbd5 [Follower]" leader-address= leader-id= 2025-01-28T14:47:07.478Z [WARN] server-70acce05-c77b-6d47-99c8-d16b2b1b214b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:07.478Z [INFO] server-70acce05-c77b-6d47-99c8-d16b2b1b214b: entering candidate state: node="Node at 70acce05-c77b-6d47-99c8-d16b2b1b214b [Candidate]" term=2 2025-01-28T14:47:07.479Z [INFO] server-70acce05-c77b-6d47-99c8-d16b2b1b214b: election won: term=2 tally=2 2025-01-28T14:47:07.479Z [INFO] server-70acce05-c77b-6d47-99c8-d16b2b1b214b: entering leader state: leader="Node at 70acce05-c77b-6d47-99c8-d16b2b1b214b [Leader]" 2025-01-28T14:47:07.480Z [INFO] server-70acce05-c77b-6d47-99c8-d16b2b1b214b: added peer, starting replication: peer=server-dfa1fdab-b4a7-68ad-6a37-aaac9a70bb79 2025-01-28T14:47:07.480Z [INFO] server-70acce05-c77b-6d47-99c8-d16b2b1b214b: added peer, starting replication: peer=server-c1998568-e227-8e8f-dd1d-64639753dbd5 2025-01-28T14:47:07.481Z [INFO] server-70acce05-c77b-6d47-99c8-d16b2b1b214b: pipelining replication: peer="{Voter server-dfa1fdab-b4a7-68ad-6a37-aaac9a70bb79 dfa1fdab-b4a7-68ad-6a37-aaac9a70bb79}" 2025-01-28T14:47:07.482Z [INFO] server-70acce05-c77b-6d47-99c8-d16b2b1b214b: pipelining replication: peer="{Voter server-c1998568-e227-8e8f-dd1d-64639753dbd5 c1998568-e227-8e8f-dd1d-64639753dbd5}" 2025-01-28T14:47:07.586Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:07.479986246 +0000 UTC m=+10.563392604 (1 nodes), 75.636246ms from start of poll, 80.539058ms from cluster start. Timeout at 2025-01-28 14:47:07.585026581 +0000 UTC m=+10.668432605, 105.040001ms after stability 2025-01-28T14:47:07.586Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:07.692Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:07.586637412 +0000 UTC m=+10.670043103 (2 nodes), 14.667µs from start of poll, 187.189557ms from cluster start. Timeout at 2025-01-28 14:47:07.691714748 +0000 UTC m=+10.775120772, 105.077669ms after stability 2025-01-28T14:47:07.692Z [INFO] server-70acce05-c77b-6d47-99c8-d16b2b1b214b: aborting pipeline replication: peer="{Voter server-c1998568-e227-8e8f-dd1d-64639753dbd5 c1998568-e227-8e8f-dd1d-64639753dbd5}" 2025-01-28T14:47:07.692Z [INFO] server-70acce05-c77b-6d47-99c8-d16b2b1b214b: aborting pipeline replication: peer="{Voter server-dfa1fdab-b4a7-68ad-6a37-aaac9a70bb79 dfa1fdab-b4a7-68ad-6a37-aaac9a70bb79}" --- PASS: TestRaft_ApplyNonLeader (0.31s) === RUN TestRaft_ApplyConcurrent 2025-01-28T14:47:07.711Z [INFO] server-e776d1bb-0d70-30d8-28fb-dd78713fadcd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e776d1bb-0d70-30d8-28fb-dd78713fadcd Address:e776d1bb-0d70-30d8-28fb-dd78713fadcd} {Suffrage:Voter ID:server-e035fc49-bd0e-2ae8-910b-ad3a5fc9b747 Address:e035fc49-bd0e-2ae8-910b-ad3a5fc9b747} {Suffrage:Voter ID:server-b18dffe5-c0fa-955e-2a3a-d5368a14682e Address:b18dffe5-c0fa-955e-2a3a-d5368a14682e}]" 2025-01-28T14:47:07.712Z [INFO] server-e776d1bb-0d70-30d8-28fb-dd78713fadcd: entering follower state: follower="Node at e776d1bb-0d70-30d8-28fb-dd78713fadcd [Follower]" leader-address= leader-id= 2025-01-28T14:47:07.713Z [INFO] server-e035fc49-bd0e-2ae8-910b-ad3a5fc9b747: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e776d1bb-0d70-30d8-28fb-dd78713fadcd Address:e776d1bb-0d70-30d8-28fb-dd78713fadcd} {Suffrage:Voter ID:server-e035fc49-bd0e-2ae8-910b-ad3a5fc9b747 Address:e035fc49-bd0e-2ae8-910b-ad3a5fc9b747} {Suffrage:Voter ID:server-b18dffe5-c0fa-955e-2a3a-d5368a14682e Address:b18dffe5-c0fa-955e-2a3a-d5368a14682e}]" 2025-01-28T14:47:07.713Z [INFO] server-e035fc49-bd0e-2ae8-910b-ad3a5fc9b747: entering follower state: follower="Node at e035fc49-bd0e-2ae8-910b-ad3a5fc9b747 [Follower]" leader-address= leader-id= 2025-01-28T14:47:07.714Z [INFO] server-b18dffe5-c0fa-955e-2a3a-d5368a14682e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e776d1bb-0d70-30d8-28fb-dd78713fadcd Address:e776d1bb-0d70-30d8-28fb-dd78713fadcd} {Suffrage:Voter ID:server-e035fc49-bd0e-2ae8-910b-ad3a5fc9b747 Address:e035fc49-bd0e-2ae8-910b-ad3a5fc9b747} {Suffrage:Voter ID:server-b18dffe5-c0fa-955e-2a3a-d5368a14682e Address:b18dffe5-c0fa-955e-2a3a-d5368a14682e}]" 2025-01-28T14:47:07.715Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:07.715Z [INFO] server-b18dffe5-c0fa-955e-2a3a-d5368a14682e: entering follower state: follower="Node at b18dffe5-c0fa-955e-2a3a-d5368a14682e [Follower]" leader-address= leader-id= 2025-01-28T14:47:07.826Z [WARN] server-b18dffe5-c0fa-955e-2a3a-d5368a14682e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:07.827Z [INFO] server-b18dffe5-c0fa-955e-2a3a-d5368a14682e: entering candidate state: node="Node at b18dffe5-c0fa-955e-2a3a-d5368a14682e [Candidate]" term=2 2025-01-28T14:47:07.828Z [INFO] server-b18dffe5-c0fa-955e-2a3a-d5368a14682e: election won: term=2 tally=2 2025-01-28T14:47:07.828Z [INFO] server-b18dffe5-c0fa-955e-2a3a-d5368a14682e: entering leader state: leader="Node at b18dffe5-c0fa-955e-2a3a-d5368a14682e [Leader]" 2025-01-28T14:47:07.828Z [INFO] server-b18dffe5-c0fa-955e-2a3a-d5368a14682e: added peer, starting replication: peer=server-e776d1bb-0d70-30d8-28fb-dd78713fadcd 2025-01-28T14:47:07.829Z [INFO] server-b18dffe5-c0fa-955e-2a3a-d5368a14682e: added peer, starting replication: peer=server-e035fc49-bd0e-2ae8-910b-ad3a5fc9b747 2025-01-28T14:47:07.830Z [INFO] server-b18dffe5-c0fa-955e-2a3a-d5368a14682e: pipelining replication: peer="{Voter server-e776d1bb-0d70-30d8-28fb-dd78713fadcd e776d1bb-0d70-30d8-28fb-dd78713fadcd}" 2025-01-28T14:47:07.830Z [INFO] server-b18dffe5-c0fa-955e-2a3a-d5368a14682e: pipelining replication: peer="{Voter server-e035fc49-bd0e-2ae8-910b-ad3a5fc9b747 e035fc49-bd0e-2ae8-910b-ad3a5fc9b747}" 2025-01-28T14:47:08.034Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:07.828632263 +0000 UTC m=+10.912038621 (1 nodes), 113.279363ms from start of poll, 117.931501ms from cluster start. Timeout at 2025-01-28 14:47:08.033671931 +0000 UTC m=+11.117077955, 205.039334ms after stability 2025-01-28T14:47:08.068Z [INFO] server-b18dffe5-c0fa-955e-2a3a-d5368a14682e: aborting pipeline replication: peer="{Voter server-e035fc49-bd0e-2ae8-910b-ad3a5fc9b747 e035fc49-bd0e-2ae8-910b-ad3a5fc9b747}" 2025-01-28T14:47:08.068Z [INFO] server-b18dffe5-c0fa-955e-2a3a-d5368a14682e: aborting pipeline replication: peer="{Voter server-e776d1bb-0d70-30d8-28fb-dd78713fadcd e776d1bb-0d70-30d8-28fb-dd78713fadcd}" --- PASS: TestRaft_ApplyConcurrent (0.38s) === RUN TestRaft_ApplyConcurrent_Timeout 2025-01-28T14:47:08.089Z [INFO] server-3a03d5bc-14c7-bd88-eb45-6c53a2c0baa2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a03d5bc-14c7-bd88-eb45-6c53a2c0baa2 Address:3a03d5bc-14c7-bd88-eb45-6c53a2c0baa2}]" 2025-01-28T14:47:08.090Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:08.090Z [INFO] server-3a03d5bc-14c7-bd88-eb45-6c53a2c0baa2: entering follower state: follower="Node at 3a03d5bc-14c7-bd88-eb45-6c53a2c0baa2 [Follower]" leader-address= leader-id= 2025-01-28T14:47:08.277Z [WARN] server-3a03d5bc-14c7-bd88-eb45-6c53a2c0baa2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:08.278Z [INFO] server-3a03d5bc-14c7-bd88-eb45-6c53a2c0baa2: entering candidate state: node="Node at 3a03d5bc-14c7-bd88-eb45-6c53a2c0baa2 [Candidate]" term=2 2025-01-28T14:47:08.278Z [INFO] server-3a03d5bc-14c7-bd88-eb45-6c53a2c0baa2: election won: term=2 tally=1 2025-01-28T14:47:08.278Z [INFO] server-3a03d5bc-14c7-bd88-eb45-6c53a2c0baa2: entering leader state: leader="Node at 3a03d5bc-14c7-bd88-eb45-6c53a2c0baa2 [Leader]" 2025-01-28T14:47:08.481Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:08.279074636 +0000 UTC m=+11.362480661 (1 nodes), 188.431594ms from start of poll, 190.327651ms from cluster start. Timeout at 2025-01-28 14:47:08.480140972 +0000 UTC m=+11.563546996, 201.066335ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.40s) === RUN TestRaft_JoinNode 2025-01-28T14:47:08.493Z [INFO] server-b5e086a3-a5d6-b7e9-110c-cb4a8866a1bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5e086a3-a5d6-b7e9-110c-cb4a8866a1bb Address:b5e086a3-a5d6-b7e9-110c-cb4a8866a1bb} {Suffrage:Voter ID:server-1de2e149-53ee-70ae-41bc-5cb5868bca3b Address:1de2e149-53ee-70ae-41bc-5cb5868bca3b}]" 2025-01-28T14:47:08.493Z [INFO] server-b5e086a3-a5d6-b7e9-110c-cb4a8866a1bb: entering follower state: follower="Node at b5e086a3-a5d6-b7e9-110c-cb4a8866a1bb [Follower]" leader-address= leader-id= 2025-01-28T14:47:08.494Z [INFO] server-1de2e149-53ee-70ae-41bc-5cb5868bca3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5e086a3-a5d6-b7e9-110c-cb4a8866a1bb Address:b5e086a3-a5d6-b7e9-110c-cb4a8866a1bb} {Suffrage:Voter ID:server-1de2e149-53ee-70ae-41bc-5cb5868bca3b Address:1de2e149-53ee-70ae-41bc-5cb5868bca3b}]" 2025-01-28T14:47:08.495Z [INFO] server-1de2e149-53ee-70ae-41bc-5cb5868bca3b: entering follower state: follower="Node at 1de2e149-53ee-70ae-41bc-5cb5868bca3b [Follower]" leader-address= leader-id= 2025-01-28T14:47:08.498Z [INFO] server-6bef4d51-4f51-ac84-ad3d-f283191476da: initial configuration: index=0 servers=[] 2025-01-28T14:47:08.498Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:08.499Z [INFO] server-6bef4d51-4f51-ac84-ad3d-f283191476da: entering follower state: follower="Node at 6bef4d51-4f51-ac84-ad3d-f283191476da [Follower]" leader-address= leader-id= 2025-01-28T14:47:08.569Z [WARN] server-1de2e149-53ee-70ae-41bc-5cb5868bca3b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:08.569Z [INFO] server-1de2e149-53ee-70ae-41bc-5cb5868bca3b: entering candidate state: node="Node at 1de2e149-53ee-70ae-41bc-5cb5868bca3b [Candidate]" term=2 2025-01-28T14:47:08.570Z [INFO] server-1de2e149-53ee-70ae-41bc-5cb5868bca3b: election won: term=2 tally=2 2025-01-28T14:47:08.570Z [INFO] server-1de2e149-53ee-70ae-41bc-5cb5868bca3b: entering leader state: leader="Node at 1de2e149-53ee-70ae-41bc-5cb5868bca3b [Leader]" 2025-01-28T14:47:08.571Z [INFO] server-1de2e149-53ee-70ae-41bc-5cb5868bca3b: added peer, starting replication: peer=server-b5e086a3-a5d6-b7e9-110c-cb4a8866a1bb 2025-01-28T14:47:08.572Z [INFO] server-1de2e149-53ee-70ae-41bc-5cb5868bca3b: pipelining replication: peer="{Voter server-b5e086a3-a5d6-b7e9-110c-cb4a8866a1bb b5e086a3-a5d6-b7e9-110c-cb4a8866a1bb}" 2025-01-28T14:47:08.592Z [WARN] server-6bef4d51-4f51-ac84-ad3d-f283191476da: no known peers, aborting election 2025-01-28T14:47:08.676Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:08.571304979 +0000 UTC m=+11.654711337 (1 nodes), 72.354815ms from start of poll, 79.278021ms from cluster start. Timeout at 2025-01-28 14:47:08.676341979 +0000 UTC m=+11.759748004, 105.036667ms after stability 2025-01-28T14:47:08.677Z [INFO] server-1de2e149-53ee-70ae-41bc-5cb5868bca3b: updating configuration: command=AddVoter server-id=server-6bef4d51-4f51-ac84-ad3d-f283191476da server-addr=6bef4d51-4f51-ac84-ad3d-f283191476da servers="[{Suffrage:Voter ID:server-b5e086a3-a5d6-b7e9-110c-cb4a8866a1bb Address:b5e086a3-a5d6-b7e9-110c-cb4a8866a1bb} {Suffrage:Voter ID:server-1de2e149-53ee-70ae-41bc-5cb5868bca3b Address:1de2e149-53ee-70ae-41bc-5cb5868bca3b} {Suffrage:Voter ID:server-6bef4d51-4f51-ac84-ad3d-f283191476da Address:6bef4d51-4f51-ac84-ad3d-f283191476da}]" 2025-01-28T14:47:08.678Z [INFO] server-1de2e149-53ee-70ae-41bc-5cb5868bca3b: added peer, starting replication: peer=server-6bef4d51-4f51-ac84-ad3d-f283191476da 2025-01-28T14:47:08.678Z [WARN] server-6bef4d51-4f51-ac84-ad3d-f283191476da: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2025-01-28T14:47:08.679Z [WARN] server-1de2e149-53ee-70ae-41bc-5cb5868bca3b: appendEntries rejected, sending older logs: peer="{Voter server-6bef4d51-4f51-ac84-ad3d-f283191476da 6bef4d51-4f51-ac84-ad3d-f283191476da}" next=1 2025-01-28T14:47:08.680Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:08.681Z [INFO] server-1de2e149-53ee-70ae-41bc-5cb5868bca3b: pipelining replication: peer="{Voter server-6bef4d51-4f51-ac84-ad3d-f283191476da 6bef4d51-4f51-ac84-ad3d-f283191476da}" 2025-01-28T14:47:08.785Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:08.680373884 +0000 UTC m=+11.763779575 (1 nodes), 11.667µs from start of poll, 188.346259ms from cluster start. Timeout at 2025-01-28 14:47:08.785443885 +0000 UTC m=+11.868849910, 105.070335ms after stability 2025-01-28T14:47:08.786Z [INFO] server-1de2e149-53ee-70ae-41bc-5cb5868bca3b: aborting pipeline replication: peer="{Voter server-6bef4d51-4f51-ac84-ad3d-f283191476da 6bef4d51-4f51-ac84-ad3d-f283191476da}" 2025-01-28T14:47:08.786Z [INFO] server-1de2e149-53ee-70ae-41bc-5cb5868bca3b: aborting pipeline replication: peer="{Voter server-b5e086a3-a5d6-b7e9-110c-cb4a8866a1bb b5e086a3-a5d6-b7e9-110c-cb4a8866a1bb}" --- PASS: TestRaft_JoinNode (0.31s) === RUN TestRaft_JoinNode_ConfigStore 2025-01-28T14:47:08.798Z [INFO] server-c9ae9745-3747-ad0d-8d0e-981f4739124a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9ae9745-3747-ad0d-8d0e-981f4739124a Address:c9ae9745-3747-ad0d-8d0e-981f4739124a}]" 2025-01-28T14:47:08.799Z [INFO] server-c9ae9745-3747-ad0d-8d0e-981f4739124a: entering follower state: follower="Node at c9ae9745-3747-ad0d-8d0e-981f4739124a [Follower]" leader-address= leader-id= 2025-01-28T14:47:08.802Z [INFO] server-27322f22-a8f6-ebbc-2383-913b2318d7b3: initial configuration: index=0 servers=[] 2025-01-28T14:47:08.802Z [INFO] server-27322f22-a8f6-ebbc-2383-913b2318d7b3: entering follower state: follower="Node at 27322f22-a8f6-ebbc-2383-913b2318d7b3 [Follower]" leader-address= leader-id= 2025-01-28T14:47:08.805Z [INFO] server-68bb3633-f9f3-bd67-ee3d-a9f6ca0940fe: initial configuration: index=0 servers=[] 2025-01-28T14:47:08.806Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:08.806Z [INFO] server-68bb3633-f9f3-bd67-ee3d-a9f6ca0940fe: entering follower state: follower="Node at 68bb3633-f9f3-bd67-ee3d-a9f6ca0940fe [Follower]" leader-address= leader-id= 2025-01-28T14:47:08.870Z [WARN] server-68bb3633-f9f3-bd67-ee3d-a9f6ca0940fe: no known peers, aborting election 2025-01-28T14:47:08.870Z [WARN] server-c9ae9745-3747-ad0d-8d0e-981f4739124a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:08.871Z [INFO] server-c9ae9745-3747-ad0d-8d0e-981f4739124a: entering candidate state: node="Node at c9ae9745-3747-ad0d-8d0e-981f4739124a [Candidate]" term=2 2025-01-28T14:47:08.871Z [INFO] server-c9ae9745-3747-ad0d-8d0e-981f4739124a: election won: term=2 tally=1 2025-01-28T14:47:08.871Z [INFO] server-c9ae9745-3747-ad0d-8d0e-981f4739124a: entering leader state: leader="Node at c9ae9745-3747-ad0d-8d0e-981f4739124a [Leader]" 2025-01-28T14:47:08.873Z [WARN] server-27322f22-a8f6-ebbc-2383-913b2318d7b3: no known peers, aborting election 2025-01-28T14:47:08.977Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:08.872019573 +0000 UTC m=+11.955425931 (1 nodes), 65.532946ms from start of poll, 74.42321ms from cluster start. Timeout at 2025-01-28 14:47:08.977056574 +0000 UTC m=+12.060462932, 105.037001ms after stability 2025-01-28T14:47:08.978Z [INFO] server-c9ae9745-3747-ad0d-8d0e-981f4739124a: updating configuration: command=AddVoter server-id=server-27322f22-a8f6-ebbc-2383-913b2318d7b3 server-addr=27322f22-a8f6-ebbc-2383-913b2318d7b3 servers="[{Suffrage:Voter ID:server-c9ae9745-3747-ad0d-8d0e-981f4739124a Address:c9ae9745-3747-ad0d-8d0e-981f4739124a} {Suffrage:Voter ID:server-27322f22-a8f6-ebbc-2383-913b2318d7b3 Address:27322f22-a8f6-ebbc-2383-913b2318d7b3}]" 2025-01-28T14:47:08.979Z [INFO] server-c9ae9745-3747-ad0d-8d0e-981f4739124a: added peer, starting replication: peer=server-27322f22-a8f6-ebbc-2383-913b2318d7b3 2025-01-28T14:47:08.980Z [WARN] server-27322f22-a8f6-ebbc-2383-913b2318d7b3: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2025-01-28T14:47:08.980Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:08.981Z [WARN] server-c9ae9745-3747-ad0d-8d0e-981f4739124a: appendEntries rejected, sending older logs: peer="{Voter server-27322f22-a8f6-ebbc-2383-913b2318d7b3 27322f22-a8f6-ebbc-2383-913b2318d7b3}" next=1 2025-01-28T14:47:08.983Z [INFO] server-c9ae9745-3747-ad0d-8d0e-981f4739124a: pipelining replication: peer="{Voter server-27322f22-a8f6-ebbc-2383-913b2318d7b3 27322f22-a8f6-ebbc-2383-913b2318d7b3}" 2025-01-28T14:47:09.086Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:08.980580796 +0000 UTC m=+12.063986821 (1 nodes), 12.001µs from start of poll, 182.9841ms from cluster start. Timeout at 2025-01-28 14:47:09.085657131 +0000 UTC m=+12.169063156, 105.076335ms after stability 2025-01-28T14:47:09.086Z [INFO] server-c9ae9745-3747-ad0d-8d0e-981f4739124a: updating configuration: command=AddVoter server-id=server-68bb3633-f9f3-bd67-ee3d-a9f6ca0940fe server-addr=68bb3633-f9f3-bd67-ee3d-a9f6ca0940fe servers="[{Suffrage:Voter ID:server-c9ae9745-3747-ad0d-8d0e-981f4739124a Address:c9ae9745-3747-ad0d-8d0e-981f4739124a} {Suffrage:Voter ID:server-27322f22-a8f6-ebbc-2383-913b2318d7b3 Address:27322f22-a8f6-ebbc-2383-913b2318d7b3} {Suffrage:Voter ID:server-68bb3633-f9f3-bd67-ee3d-a9f6ca0940fe Address:68bb3633-f9f3-bd67-ee3d-a9f6ca0940fe}]" 2025-01-28T14:47:09.087Z [INFO] server-c9ae9745-3747-ad0d-8d0e-981f4739124a: added peer, starting replication: peer=server-68bb3633-f9f3-bd67-ee3d-a9f6ca0940fe 2025-01-28T14:47:09.088Z [WARN] server-68bb3633-f9f3-bd67-ee3d-a9f6ca0940fe: failed to get previous log: previous-index=4 last-index=0 error="log not found" 2025-01-28T14:47:09.089Z [WARN] server-c9ae9745-3747-ad0d-8d0e-981f4739124a: appendEntries rejected, sending older logs: peer="{Voter server-68bb3633-f9f3-bd67-ee3d-a9f6ca0940fe 68bb3633-f9f3-bd67-ee3d-a9f6ca0940fe}" next=1 2025-01-28T14:47:09.090Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:09.090Z [INFO] server-c9ae9745-3747-ad0d-8d0e-981f4739124a: pipelining replication: peer="{Voter server-68bb3633-f9f3-bd67-ee3d-a9f6ca0940fe 68bb3633-f9f3-bd67-ee3d-a9f6ca0940fe}" 2025-01-28T14:47:09.196Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:09.090504393 +0000 UTC m=+12.173910084 (1 nodes), 12.334µs from start of poll, 292.907363ms from cluster start. Timeout at 2025-01-28 14:47:09.195558728 +0000 UTC m=+12.278964752, 105.054668ms after stability 2025-01-28T14:47:09.196Z [INFO] server-c9ae9745-3747-ad0d-8d0e-981f4739124a: aborting pipeline replication: peer="{Voter server-27322f22-a8f6-ebbc-2383-913b2318d7b3 27322f22-a8f6-ebbc-2383-913b2318d7b3}" 2025-01-28T14:47:09.196Z [INFO] server-c9ae9745-3747-ad0d-8d0e-981f4739124a: aborting pipeline replication: peer="{Voter server-68bb3633-f9f3-bd67-ee3d-a9f6ca0940fe 68bb3633-f9f3-bd67-ee3d-a9f6ca0940fe}" --- PASS: TestRaft_JoinNode_ConfigStore (0.41s) === RUN TestRaft_RemoveFollower 2025-01-28T14:47:09.207Z [INFO] server-23fab774-ab30-2a89-1897-b1298a7e24d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-23fab774-ab30-2a89-1897-b1298a7e24d7 Address:23fab774-ab30-2a89-1897-b1298a7e24d7} {Suffrage:Voter ID:server-2c71e324-1188-437b-3808-ae43c76baf5c Address:2c71e324-1188-437b-3808-ae43c76baf5c} {Suffrage:Voter ID:server-9a9b8701-2e2e-8656-7935-597f1257e969 Address:9a9b8701-2e2e-8656-7935-597f1257e969}]" 2025-01-28T14:47:09.207Z [INFO] server-23fab774-ab30-2a89-1897-b1298a7e24d7: entering follower state: follower="Node at 23fab774-ab30-2a89-1897-b1298a7e24d7 [Follower]" leader-address= leader-id= 2025-01-28T14:47:09.207Z [INFO] server-2c71e324-1188-437b-3808-ae43c76baf5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-23fab774-ab30-2a89-1897-b1298a7e24d7 Address:23fab774-ab30-2a89-1897-b1298a7e24d7} {Suffrage:Voter ID:server-2c71e324-1188-437b-3808-ae43c76baf5c Address:2c71e324-1188-437b-3808-ae43c76baf5c} {Suffrage:Voter ID:server-9a9b8701-2e2e-8656-7935-597f1257e969 Address:9a9b8701-2e2e-8656-7935-597f1257e969}]" 2025-01-28T14:47:09.208Z [INFO] server-2c71e324-1188-437b-3808-ae43c76baf5c: entering follower state: follower="Node at 2c71e324-1188-437b-3808-ae43c76baf5c [Follower]" leader-address= leader-id= 2025-01-28T14:47:09.209Z [INFO] server-9a9b8701-2e2e-8656-7935-597f1257e969: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-23fab774-ab30-2a89-1897-b1298a7e24d7 Address:23fab774-ab30-2a89-1897-b1298a7e24d7} {Suffrage:Voter ID:server-2c71e324-1188-437b-3808-ae43c76baf5c Address:2c71e324-1188-437b-3808-ae43c76baf5c} {Suffrage:Voter ID:server-9a9b8701-2e2e-8656-7935-597f1257e969 Address:9a9b8701-2e2e-8656-7935-597f1257e969}]" 2025-01-28T14:47:09.209Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:09.209Z [INFO] server-9a9b8701-2e2e-8656-7935-597f1257e969: entering follower state: follower="Node at 9a9b8701-2e2e-8656-7935-597f1257e969 [Follower]" leader-address= leader-id= 2025-01-28T14:47:09.276Z [WARN] server-23fab774-ab30-2a89-1897-b1298a7e24d7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:09.276Z [INFO] server-23fab774-ab30-2a89-1897-b1298a7e24d7: entering candidate state: node="Node at 23fab774-ab30-2a89-1897-b1298a7e24d7 [Candidate]" term=2 2025-01-28T14:47:09.277Z [INFO] server-23fab774-ab30-2a89-1897-b1298a7e24d7: election won: term=2 tally=2 2025-01-28T14:47:09.277Z [INFO] server-23fab774-ab30-2a89-1897-b1298a7e24d7: entering leader state: leader="Node at 23fab774-ab30-2a89-1897-b1298a7e24d7 [Leader]" 2025-01-28T14:47:09.278Z [INFO] server-23fab774-ab30-2a89-1897-b1298a7e24d7: added peer, starting replication: peer=server-2c71e324-1188-437b-3808-ae43c76baf5c 2025-01-28T14:47:09.278Z [INFO] server-23fab774-ab30-2a89-1897-b1298a7e24d7: added peer, starting replication: peer=server-9a9b8701-2e2e-8656-7935-597f1257e969 2025-01-28T14:47:09.279Z [INFO] server-23fab774-ab30-2a89-1897-b1298a7e24d7: pipelining replication: peer="{Voter server-2c71e324-1188-437b-3808-ae43c76baf5c 2c71e324-1188-437b-3808-ae43c76baf5c}" 2025-01-28T14:47:09.280Z [INFO] server-23fab774-ab30-2a89-1897-b1298a7e24d7: pipelining replication: peer="{Voter server-9a9b8701-2e2e-8656-7935-597f1257e969 9a9b8701-2e2e-8656-7935-597f1257e969}" 2025-01-28T14:47:09.384Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:09.278105963 +0000 UTC m=+12.361512654 (1 nodes), 68.087022ms from start of poll, 71.599126ms from cluster start. Timeout at 2025-01-28 14:47:09.383126297 +0000 UTC m=+12.466531988, 105.019334ms after stability 2025-01-28T14:47:09.390Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:09.495Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:09.390386629 +0000 UTC m=+12.473792320 (2 nodes), 16.333µs from start of poll, 183.878792ms from cluster start. Timeout at 2025-01-28 14:47:09.495482299 +0000 UTC m=+12.578888323, 105.096003ms after stability 2025-01-28T14:47:09.496Z [INFO] server-23fab774-ab30-2a89-1897-b1298a7e24d7: updating configuration: command=RemoveServer server-id=server-2c71e324-1188-437b-3808-ae43c76baf5c server-addr= servers="[{Suffrage:Voter ID:server-23fab774-ab30-2a89-1897-b1298a7e24d7 Address:23fab774-ab30-2a89-1897-b1298a7e24d7} {Suffrage:Voter ID:server-9a9b8701-2e2e-8656-7935-597f1257e969 Address:9a9b8701-2e2e-8656-7935-597f1257e969}]" 2025-01-28T14:47:09.497Z [INFO] server-23fab774-ab30-2a89-1897-b1298a7e24d7: removed peer, stopping replication: peer=server-2c71e324-1188-437b-3808-ae43c76baf5c last-index=3 2025-01-28T14:47:09.497Z [INFO] server-23fab774-ab30-2a89-1897-b1298a7e24d7: aborting pipeline replication: peer="{Voter server-2c71e324-1188-437b-3808-ae43c76baf5c 2c71e324-1188-437b-3808-ae43c76baf5c}" 2025-01-28T14:47:09.587Z [WARN] server-2c71e324-1188-437b-3808-ae43c76baf5c: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2025-01-28T14:47:09.604Z [INFO] server-23fab774-ab30-2a89-1897-b1298a7e24d7: aborting pipeline replication: peer="{Voter server-9a9b8701-2e2e-8656-7935-597f1257e969 9a9b8701-2e2e-8656-7935-597f1257e969}" --- PASS: TestRaft_RemoveFollower (0.41s) === RUN TestRaft_RemoveLeader 2025-01-28T14:47:09.622Z [INFO] server-0bb198d2-3144-9050-708d-ba597a89098c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bb198d2-3144-9050-708d-ba597a89098c Address:0bb198d2-3144-9050-708d-ba597a89098c} {Suffrage:Voter ID:server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe Address:3800ac0c-3deb-159b-bef8-f7ea1c026bbe} {Suffrage:Voter ID:server-20938b53-c978-b131-c99d-33b8cb97c052 Address:20938b53-c978-b131-c99d-33b8cb97c052}]" 2025-01-28T14:47:09.622Z [INFO] server-0bb198d2-3144-9050-708d-ba597a89098c: entering follower state: follower="Node at 0bb198d2-3144-9050-708d-ba597a89098c [Follower]" leader-address= leader-id= 2025-01-28T14:47:09.624Z [INFO] server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bb198d2-3144-9050-708d-ba597a89098c Address:0bb198d2-3144-9050-708d-ba597a89098c} {Suffrage:Voter ID:server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe Address:3800ac0c-3deb-159b-bef8-f7ea1c026bbe} {Suffrage:Voter ID:server-20938b53-c978-b131-c99d-33b8cb97c052 Address:20938b53-c978-b131-c99d-33b8cb97c052}]" 2025-01-28T14:47:09.624Z [INFO] server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe: entering follower state: follower="Node at 3800ac0c-3deb-159b-bef8-f7ea1c026bbe [Follower]" leader-address= leader-id= 2025-01-28T14:47:09.625Z [INFO] server-20938b53-c978-b131-c99d-33b8cb97c052: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bb198d2-3144-9050-708d-ba597a89098c Address:0bb198d2-3144-9050-708d-ba597a89098c} {Suffrage:Voter ID:server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe Address:3800ac0c-3deb-159b-bef8-f7ea1c026bbe} {Suffrage:Voter ID:server-20938b53-c978-b131-c99d-33b8cb97c052 Address:20938b53-c978-b131-c99d-33b8cb97c052}]" 2025-01-28T14:47:09.625Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:09.626Z [INFO] server-20938b53-c978-b131-c99d-33b8cb97c052: entering follower state: follower="Node at 20938b53-c978-b131-c99d-33b8cb97c052 [Follower]" leader-address= leader-id= 2025-01-28T14:47:09.689Z [WARN] server-20938b53-c978-b131-c99d-33b8cb97c052: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:09.690Z [WARN] server-0bb198d2-3144-9050-708d-ba597a89098c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:09.690Z [INFO] server-20938b53-c978-b131-c99d-33b8cb97c052: entering candidate state: node="Node at 20938b53-c978-b131-c99d-33b8cb97c052 [Candidate]" term=2 2025-01-28T14:47:09.690Z [INFO] server-0bb198d2-3144-9050-708d-ba597a89098c: entering candidate state: node="Node at 0bb198d2-3144-9050-708d-ba597a89098c [Candidate]" term=2 2025-01-28T14:47:09.691Z [INFO] server-20938b53-c978-b131-c99d-33b8cb97c052: duplicate requestVote for same term: term=2 2025-01-28T14:47:09.691Z [INFO] server-0bb198d2-3144-9050-708d-ba597a89098c: duplicate requestVote for same term: term=2 2025-01-28T14:47:09.691Z [INFO] server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe: duplicate requestVote for same term: term=2 2025-01-28T14:47:09.691Z [INFO] server-20938b53-c978-b131-c99d-33b8cb97c052: election won: term=2 tally=2 2025-01-28T14:47:09.691Z [INFO] server-20938b53-c978-b131-c99d-33b8cb97c052: entering leader state: leader="Node at 20938b53-c978-b131-c99d-33b8cb97c052 [Leader]" 2025-01-28T14:47:09.691Z [INFO] server-20938b53-c978-b131-c99d-33b8cb97c052: added peer, starting replication: peer=server-0bb198d2-3144-9050-708d-ba597a89098c 2025-01-28T14:47:09.691Z [INFO] server-20938b53-c978-b131-c99d-33b8cb97c052: added peer, starting replication: peer=server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe 2025-01-28T14:47:09.692Z [INFO] server-0bb198d2-3144-9050-708d-ba597a89098c: entering follower state: follower="Node at 0bb198d2-3144-9050-708d-ba597a89098c [Follower]" leader-address=20938b53-c978-b131-c99d-33b8cb97c052 leader-id=server-20938b53-c978-b131-c99d-33b8cb97c052 2025-01-28T14:47:09.693Z [INFO] server-20938b53-c978-b131-c99d-33b8cb97c052: pipelining replication: peer="{Voter server-0bb198d2-3144-9050-708d-ba597a89098c 0bb198d2-3144-9050-708d-ba597a89098c}" 2025-01-28T14:47:09.693Z [INFO] server-20938b53-c978-b131-c99d-33b8cb97c052: pipelining replication: peer="{Voter server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe 3800ac0c-3deb-159b-bef8-f7ea1c026bbe}" 2025-01-28T14:47:09.798Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:09.692929611 +0000 UTC m=+12.776335969 (1 nodes), 66.818984ms from start of poll, 71.74013ms from cluster start. Timeout at 2025-01-28 14:47:09.797950279 +0000 UTC m=+12.881356303, 105.020334ms after stability 2025-01-28T14:47:09.803Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:09.909Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:09.804017909 +0000 UTC m=+12.887423600 (2 nodes), 15µs from start of poll, 182.827761ms from cluster start. Timeout at 2025-01-28 14:47:09.909115912 +0000 UTC m=+12.992521936, 105.098336ms after stability 2025-01-28T14:47:09.909Z [INFO] server-20938b53-c978-b131-c99d-33b8cb97c052: updating configuration: command=RemoveServer server-id=server-20938b53-c978-b131-c99d-33b8cb97c052 server-addr= servers="[{Suffrage:Voter ID:server-0bb198d2-3144-9050-708d-ba597a89098c Address:0bb198d2-3144-9050-708d-ba597a89098c} {Suffrage:Voter ID:server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe Address:3800ac0c-3deb-159b-bef8-f7ea1c026bbe}]" 2025-01-28T14:47:09.912Z [INFO] server-20938b53-c978-b131-c99d-33b8cb97c052: removed ourself, shutting down 2025-01-28T14:47:09.912Z [INFO] server-20938b53-c978-b131-c99d-33b8cb97c052: aborting pipeline replication: peer="{Voter server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe 3800ac0c-3deb-159b-bef8-f7ea1c026bbe}" 2025-01-28T14:47:09.912Z [INFO] server-20938b53-c978-b131-c99d-33b8cb97c052: aborting pipeline replication: peer="{Voter server-0bb198d2-3144-9050-708d-ba597a89098c 0bb198d2-3144-9050-708d-ba597a89098c}" 2025-01-28T14:47:09.976Z [WARN] server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe: heartbeat timeout reached, starting election: last-leader-addr=20938b53-c978-b131-c99d-33b8cb97c052 last-leader-id=server-20938b53-c978-b131-c99d-33b8cb97c052 2025-01-28T14:47:09.976Z [INFO] server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe: entering candidate state: node="Node at 3800ac0c-3deb-159b-bef8-f7ea1c026bbe [Candidate]" term=3 2025-01-28T14:47:09.977Z [WARN] server-0bb198d2-3144-9050-708d-ba597a89098c: rejecting vote request since we have a leader: from=3800ac0c-3deb-159b-bef8-f7ea1c026bbe leader=20938b53-c978-b131-c99d-33b8cb97c052 leader-id=server-20938b53-c978-b131-c99d-33b8cb97c052 2025-01-28T14:47:10.003Z [WARN] server-0bb198d2-3144-9050-708d-ba597a89098c: heartbeat timeout reached, starting election: last-leader-addr=20938b53-c978-b131-c99d-33b8cb97c052 last-leader-id=server-20938b53-c978-b131-c99d-33b8cb97c052 2025-01-28T14:47:10.004Z [INFO] server-0bb198d2-3144-9050-708d-ba597a89098c: entering candidate state: node="Node at 0bb198d2-3144-9050-708d-ba597a89098c [Candidate]" term=3 2025-01-28T14:47:10.004Z [INFO] server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe: duplicate requestVote for same term: term=3 2025-01-28T14:47:10.074Z [WARN] server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe: Election timeout reached, restarting election 2025-01-28T14:47:10.075Z [INFO] server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe: entering candidate state: node="Node at 3800ac0c-3deb-159b-bef8-f7ea1c026bbe [Candidate]" term=4 2025-01-28T14:47:10.075Z [INFO] server-0bb198d2-3144-9050-708d-ba597a89098c: entering follower state: follower="Node at 0bb198d2-3144-9050-708d-ba597a89098c [Follower]" leader-address= leader-id= 2025-01-28T14:47:10.076Z [INFO] server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe: election won: term=4 tally=2 2025-01-28T14:47:10.076Z [INFO] server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe: entering leader state: leader="Node at 3800ac0c-3deb-159b-bef8-f7ea1c026bbe [Leader]" 2025-01-28T14:47:10.076Z [INFO] server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe: added peer, starting replication: peer=server-0bb198d2-3144-9050-708d-ba597a89098c 2025-01-28T14:47:10.078Z [INFO] server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe: pipelining replication: peer="{Voter server-0bb198d2-3144-9050-708d-ba597a89098c 0bb198d2-3144-9050-708d-ba597a89098c}" 2025-01-28T14:47:10.124Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:10.231Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:10.125987468 +0000 UTC m=+13.209393826 (1 nodes), 960.695µs from start of poll, 504.797987ms from cluster start. Timeout at 2025-01-28 14:47:10.231064804 +0000 UTC m=+13.314471162, 105.077336ms after stability 2025-01-28T14:47:10.232Z [INFO] server-3800ac0c-3deb-159b-bef8-f7ea1c026bbe: aborting pipeline replication: peer="{Voter server-0bb198d2-3144-9050-708d-ba597a89098c 0bb198d2-3144-9050-708d-ba597a89098c}" --- PASS: TestRaft_RemoveLeader (0.63s) === RUN TestRaft_RemoveLeader_NoShutdown 2025-01-28T14:47:10.249Z [INFO] server-03944325-2272-53cd-dabd-1e894a3c7765: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03944325-2272-53cd-dabd-1e894a3c7765 Address:03944325-2272-53cd-dabd-1e894a3c7765} {Suffrage:Voter ID:server-e6d775a2-4723-0609-3a27-5fe05c4fd77d Address:e6d775a2-4723-0609-3a27-5fe05c4fd77d} {Suffrage:Voter ID:server-519e4f44-a153-9c68-28fd-d67ac519194f Address:519e4f44-a153-9c68-28fd-d67ac519194f}]" 2025-01-28T14:47:10.250Z [INFO] server-03944325-2272-53cd-dabd-1e894a3c7765: entering follower state: follower="Node at 03944325-2272-53cd-dabd-1e894a3c7765 [Follower]" leader-address= leader-id= 2025-01-28T14:47:10.251Z [INFO] server-e6d775a2-4723-0609-3a27-5fe05c4fd77d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03944325-2272-53cd-dabd-1e894a3c7765 Address:03944325-2272-53cd-dabd-1e894a3c7765} {Suffrage:Voter ID:server-e6d775a2-4723-0609-3a27-5fe05c4fd77d Address:e6d775a2-4723-0609-3a27-5fe05c4fd77d} {Suffrage:Voter ID:server-519e4f44-a153-9c68-28fd-d67ac519194f Address:519e4f44-a153-9c68-28fd-d67ac519194f}]" 2025-01-28T14:47:10.252Z [INFO] server-e6d775a2-4723-0609-3a27-5fe05c4fd77d: entering follower state: follower="Node at e6d775a2-4723-0609-3a27-5fe05c4fd77d [Follower]" leader-address= leader-id= 2025-01-28T14:47:10.253Z [INFO] server-519e4f44-a153-9c68-28fd-d67ac519194f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03944325-2272-53cd-dabd-1e894a3c7765 Address:03944325-2272-53cd-dabd-1e894a3c7765} {Suffrage:Voter ID:server-e6d775a2-4723-0609-3a27-5fe05c4fd77d Address:e6d775a2-4723-0609-3a27-5fe05c4fd77d} {Suffrage:Voter ID:server-519e4f44-a153-9c68-28fd-d67ac519194f Address:519e4f44-a153-9c68-28fd-d67ac519194f}]" 2025-01-28T14:47:10.253Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:10.253Z [INFO] server-519e4f44-a153-9c68-28fd-d67ac519194f: entering follower state: follower="Node at 519e4f44-a153-9c68-28fd-d67ac519194f [Follower]" leader-address= leader-id= 2025-01-28T14:47:10.309Z [WARN] server-e6d775a2-4723-0609-3a27-5fe05c4fd77d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:10.310Z [INFO] server-e6d775a2-4723-0609-3a27-5fe05c4fd77d: entering candidate state: node="Node at e6d775a2-4723-0609-3a27-5fe05c4fd77d [Candidate]" term=2 2025-01-28T14:47:10.311Z [INFO] server-e6d775a2-4723-0609-3a27-5fe05c4fd77d: election won: term=2 tally=2 2025-01-28T14:47:10.311Z [INFO] server-e6d775a2-4723-0609-3a27-5fe05c4fd77d: entering leader state: leader="Node at e6d775a2-4723-0609-3a27-5fe05c4fd77d [Leader]" 2025-01-28T14:47:10.311Z [INFO] server-e6d775a2-4723-0609-3a27-5fe05c4fd77d: added peer, starting replication: peer=server-03944325-2272-53cd-dabd-1e894a3c7765 2025-01-28T14:47:10.311Z [INFO] server-e6d775a2-4723-0609-3a27-5fe05c4fd77d: added peer, starting replication: peer=server-519e4f44-a153-9c68-28fd-d67ac519194f 2025-01-28T14:47:10.312Z [INFO] server-e6d775a2-4723-0609-3a27-5fe05c4fd77d: pipelining replication: peer="{Voter server-519e4f44-a153-9c68-28fd-d67ac519194f 519e4f44-a153-9c68-28fd-d67ac519194f}" 2025-01-28T14:47:10.313Z [INFO] server-e6d775a2-4723-0609-3a27-5fe05c4fd77d: pipelining replication: peer="{Voter server-03944325-2272-53cd-dabd-1e894a3c7765 03944325-2272-53cd-dabd-1e894a3c7765}" 2025-01-28T14:47:10.416Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:10.311385639 +0000 UTC m=+13.394792330 (2 nodes), 57.298701ms from start of poll, 62.620859ms from cluster start. Timeout at 2025-01-28 14:47:10.416431641 +0000 UTC m=+13.499837665, 105.045335ms after stability 2025-01-28T14:47:10.417Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:10.523Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:10.417200447 +0000 UTC m=+13.500606472 (1 nodes), 12.334µs from start of poll, 168.435001ms from cluster start. Timeout at 2025-01-28 14:47:10.522280116 +0000 UTC m=+13.605686140, 105.079668ms after stability 2025-01-28T14:47:10.564Z [INFO] server-e6d775a2-4723-0609-3a27-5fe05c4fd77d: updating configuration: command=RemoveServer server-id=server-e6d775a2-4723-0609-3a27-5fe05c4fd77d server-addr= servers="[{Suffrage:Voter ID:server-03944325-2272-53cd-dabd-1e894a3c7765 Address:03944325-2272-53cd-dabd-1e894a3c7765} {Suffrage:Voter ID:server-519e4f44-a153-9c68-28fd-d67ac519194f Address:519e4f44-a153-9c68-28fd-d67ac519194f}]" 2025-01-28T14:47:10.566Z [INFO] server-e6d775a2-4723-0609-3a27-5fe05c4fd77d: removed ourself, transitioning to follower 2025-01-28T14:47:10.566Z [INFO] server-e6d775a2-4723-0609-3a27-5fe05c4fd77d: entering follower state: follower="Node at e6d775a2-4723-0609-3a27-5fe05c4fd77d [Follower]" leader-address= leader-id= 2025-01-28T14:47:10.567Z [INFO] server-e6d775a2-4723-0609-3a27-5fe05c4fd77d: aborting pipeline replication: peer="{Voter server-03944325-2272-53cd-dabd-1e894a3c7765 03944325-2272-53cd-dabd-1e894a3c7765}" 2025-01-28T14:47:10.567Z [INFO] server-e6d775a2-4723-0609-3a27-5fe05c4fd77d: aborting pipeline replication: peer="{Voter server-519e4f44-a153-9c68-28fd-d67ac519194f 519e4f44-a153-9c68-28fd-d67ac519194f}" 2025-01-28T14:47:10.635Z [WARN] server-519e4f44-a153-9c68-28fd-d67ac519194f: heartbeat timeout reached, starting election: last-leader-addr=e6d775a2-4723-0609-3a27-5fe05c4fd77d last-leader-id=server-e6d775a2-4723-0609-3a27-5fe05c4fd77d 2025-01-28T14:47:10.636Z [INFO] server-519e4f44-a153-9c68-28fd-d67ac519194f: entering candidate state: node="Node at 519e4f44-a153-9c68-28fd-d67ac519194f [Candidate]" term=3 2025-01-28T14:47:10.636Z [WARN] server-03944325-2272-53cd-dabd-1e894a3c7765: rejecting vote request since we have a leader: from=519e4f44-a153-9c68-28fd-d67ac519194f leader=e6d775a2-4723-0609-3a27-5fe05c4fd77d leader-id=server-e6d775a2-4723-0609-3a27-5fe05c4fd77d 2025-01-28T14:47:10.647Z [WARN] server-03944325-2272-53cd-dabd-1e894a3c7765: heartbeat timeout reached, starting election: last-leader-addr=e6d775a2-4723-0609-3a27-5fe05c4fd77d last-leader-id=server-e6d775a2-4723-0609-3a27-5fe05c4fd77d 2025-01-28T14:47:10.647Z [INFO] server-03944325-2272-53cd-dabd-1e894a3c7765: entering candidate state: node="Node at 03944325-2272-53cd-dabd-1e894a3c7765 [Candidate]" term=3 2025-01-28T14:47:10.648Z [INFO] server-519e4f44-a153-9c68-28fd-d67ac519194f: duplicate requestVote for same term: term=3 2025-01-28T14:47:10.661Z [WARN] server-e6d775a2-4723-0609-3a27-5fe05c4fd77d: not part of stable configuration, aborting election 2025-01-28T14:47:10.676Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:10.706Z [WARN] server-519e4f44-a153-9c68-28fd-d67ac519194f: Election timeout reached, restarting election 2025-01-28T14:47:10.707Z [INFO] server-519e4f44-a153-9c68-28fd-d67ac519194f: entering candidate state: node="Node at 519e4f44-a153-9c68-28fd-d67ac519194f [Candidate]" term=4 2025-01-28T14:47:10.707Z [INFO] server-03944325-2272-53cd-dabd-1e894a3c7765: entering follower state: follower="Node at 03944325-2272-53cd-dabd-1e894a3c7765 [Follower]" leader-address= leader-id= 2025-01-28T14:47:10.708Z [INFO] server-519e4f44-a153-9c68-28fd-d67ac519194f: election won: term=4 tally=2 2025-01-28T14:47:10.708Z [INFO] server-519e4f44-a153-9c68-28fd-d67ac519194f: entering leader state: leader="Node at 519e4f44-a153-9c68-28fd-d67ac519194f [Leader]" 2025-01-28T14:47:10.708Z [INFO] server-519e4f44-a153-9c68-28fd-d67ac519194f: added peer, starting replication: peer=server-03944325-2272-53cd-dabd-1e894a3c7765 2025-01-28T14:47:10.710Z [INFO] server-519e4f44-a153-9c68-28fd-d67ac519194f: pipelining replication: peer="{Voter server-03944325-2272-53cd-dabd-1e894a3c7765 03944325-2272-53cd-dabd-1e894a3c7765}" 2025-01-28T14:47:10.814Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:10.709024111 +0000 UTC m=+13.792430469 (1 nodes), 31.853946ms from start of poll, 460.258998ms from cluster start. Timeout at 2025-01-28 14:47:10.814045446 +0000 UTC m=+13.897451137, 105.020668ms after stability 2025-01-28T14:47:10.920Z [INFO] server-519e4f44-a153-9c68-28fd-d67ac519194f: aborting pipeline replication: peer="{Voter server-03944325-2272-53cd-dabd-1e894a3c7765 03944325-2272-53cd-dabd-1e894a3c7765}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.69s) === RUN TestRaft_RemoveFollower_SplitCluster 2025-01-28T14:47:10.940Z [INFO] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91 Address:936c0f23-73e1-513b-2c9e-6ce7e97dbc91} {Suffrage:Voter ID:server-c7a92434-7a92-4fd8-b335-7567fef318aa Address:c7a92434-7a92-4fd8-b335-7567fef318aa} {Suffrage:Voter ID:server-fc3157a1-e38e-1410-4693-3b06129397c4 Address:fc3157a1-e38e-1410-4693-3b06129397c4} {Suffrage:Voter ID:server-6b6349ac-b54a-04f0-d1ad-b695097b35c5 Address:6b6349ac-b54a-04f0-d1ad-b695097b35c5}]" 2025-01-28T14:47:10.941Z [INFO] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: entering follower state: follower="Node at 936c0f23-73e1-513b-2c9e-6ce7e97dbc91 [Follower]" leader-address= leader-id= 2025-01-28T14:47:10.942Z [INFO] server-c7a92434-7a92-4fd8-b335-7567fef318aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91 Address:936c0f23-73e1-513b-2c9e-6ce7e97dbc91} {Suffrage:Voter ID:server-c7a92434-7a92-4fd8-b335-7567fef318aa Address:c7a92434-7a92-4fd8-b335-7567fef318aa} {Suffrage:Voter ID:server-fc3157a1-e38e-1410-4693-3b06129397c4 Address:fc3157a1-e38e-1410-4693-3b06129397c4} {Suffrage:Voter ID:server-6b6349ac-b54a-04f0-d1ad-b695097b35c5 Address:6b6349ac-b54a-04f0-d1ad-b695097b35c5}]" 2025-01-28T14:47:10.943Z [INFO] server-c7a92434-7a92-4fd8-b335-7567fef318aa: entering follower state: follower="Node at c7a92434-7a92-4fd8-b335-7567fef318aa [Follower]" leader-address= leader-id= 2025-01-28T14:47:10.943Z [INFO] server-fc3157a1-e38e-1410-4693-3b06129397c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91 Address:936c0f23-73e1-513b-2c9e-6ce7e97dbc91} {Suffrage:Voter ID:server-c7a92434-7a92-4fd8-b335-7567fef318aa Address:c7a92434-7a92-4fd8-b335-7567fef318aa} {Suffrage:Voter ID:server-fc3157a1-e38e-1410-4693-3b06129397c4 Address:fc3157a1-e38e-1410-4693-3b06129397c4} {Suffrage:Voter ID:server-6b6349ac-b54a-04f0-d1ad-b695097b35c5 Address:6b6349ac-b54a-04f0-d1ad-b695097b35c5}]" 2025-01-28T14:47:10.944Z [INFO] server-fc3157a1-e38e-1410-4693-3b06129397c4: entering follower state: follower="Node at fc3157a1-e38e-1410-4693-3b06129397c4 [Follower]" leader-address= leader-id= 2025-01-28T14:47:10.945Z [INFO] server-6b6349ac-b54a-04f0-d1ad-b695097b35c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91 Address:936c0f23-73e1-513b-2c9e-6ce7e97dbc91} {Suffrage:Voter ID:server-c7a92434-7a92-4fd8-b335-7567fef318aa Address:c7a92434-7a92-4fd8-b335-7567fef318aa} {Suffrage:Voter ID:server-fc3157a1-e38e-1410-4693-3b06129397c4 Address:fc3157a1-e38e-1410-4693-3b06129397c4} {Suffrage:Voter ID:server-6b6349ac-b54a-04f0-d1ad-b695097b35c5 Address:6b6349ac-b54a-04f0-d1ad-b695097b35c5}]" 2025-01-28T14:47:10.946Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:10.946Z [INFO] server-6b6349ac-b54a-04f0-d1ad-b695097b35c5: entering follower state: follower="Node at 6b6349ac-b54a-04f0-d1ad-b695097b35c5 [Follower]" leader-address= leader-id= 2025-01-28T14:47:11.003Z [WARN] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:11.004Z [INFO] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: entering candidate state: node="Node at 936c0f23-73e1-513b-2c9e-6ce7e97dbc91 [Candidate]" term=2 2025-01-28T14:47:11.005Z [INFO] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: election won: term=2 tally=3 2025-01-28T14:47:11.005Z [INFO] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: entering leader state: leader="Node at 936c0f23-73e1-513b-2c9e-6ce7e97dbc91 [Leader]" 2025-01-28T14:47:11.005Z [INFO] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: added peer, starting replication: peer=server-c7a92434-7a92-4fd8-b335-7567fef318aa 2025-01-28T14:47:11.005Z [INFO] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: added peer, starting replication: peer=server-fc3157a1-e38e-1410-4693-3b06129397c4 2025-01-28T14:47:11.006Z [INFO] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: added peer, starting replication: peer=server-6b6349ac-b54a-04f0-d1ad-b695097b35c5 2025-01-28T14:47:11.007Z [INFO] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: pipelining replication: peer="{Voter server-c7a92434-7a92-4fd8-b335-7567fef318aa c7a92434-7a92-4fd8-b335-7567fef318aa}" 2025-01-28T14:47:11.007Z [INFO] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: pipelining replication: peer="{Voter server-fc3157a1-e38e-1410-4693-3b06129397c4 fc3157a1-e38e-1410-4693-3b06129397c4}" 2025-01-28T14:47:11.008Z [INFO] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: pipelining replication: peer="{Voter server-6b6349ac-b54a-04f0-d1ad-b695097b35c5 6b6349ac-b54a-04f0-d1ad-b695097b35c5}" 2025-01-28T14:47:11.112Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:11.005818922 +0000 UTC m=+14.089225280 (1 nodes), 59.534434ms from start of poll, 66.233966ms from cluster start. Timeout at 2025-01-28 14:47:11.110839923 +0000 UTC m=+14.194245947, 105.020667ms after stability 2025-01-28T14:47:11.217Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:11.323Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:11.218204228 +0000 UTC m=+14.301610252 (3 nodes), 18µs from start of poll, 278.618938ms from cluster start. Timeout at 2025-01-28 14:47:11.323309564 +0000 UTC m=+14.406715589, 105.105337ms after stability 2025-01-28T14:47:11.324Z [INFO] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: updating configuration: command=RemoveServer server-id=server-6b6349ac-b54a-04f0-d1ad-b695097b35c5 server-addr= servers="[{Suffrage:Voter ID:server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91 Address:936c0f23-73e1-513b-2c9e-6ce7e97dbc91} {Suffrage:Voter ID:server-c7a92434-7a92-4fd8-b335-7567fef318aa Address:c7a92434-7a92-4fd8-b335-7567fef318aa} {Suffrage:Voter ID:server-fc3157a1-e38e-1410-4693-3b06129397c4 Address:fc3157a1-e38e-1410-4693-3b06129397c4}]" 2025-01-28T14:47:11.325Z [ERROR] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to pipeline appendEntries: peer="{Voter server-fc3157a1-e38e-1410-4693-3b06129397c4 fc3157a1-e38e-1410-4693-3b06129397c4}" error="append pipeline closed" 2025-01-28T14:47:11.325Z [INFO] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: removed peer, stopping replication: peer=server-6b6349ac-b54a-04f0-d1ad-b695097b35c5 last-index=3 2025-01-28T14:47:11.325Z [ERROR] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to pipeline appendEntries: peer="{Voter server-c7a92434-7a92-4fd8-b335-7567fef318aa c7a92434-7a92-4fd8-b335-7567fef318aa}" error="append pipeline closed" 2025-01-28T14:47:11.325Z [INFO] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: aborting pipeline replication: peer="{Voter server-c7a92434-7a92-4fd8-b335-7567fef318aa c7a92434-7a92-4fd8-b335-7567fef318aa}" 2025-01-28T14:47:11.325Z [INFO] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: aborting pipeline replication: peer="{Voter server-6b6349ac-b54a-04f0-d1ad-b695097b35c5 6b6349ac-b54a-04f0-d1ad-b695097b35c5}" 2025-01-28T14:47:11.325Z [INFO] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: aborting pipeline replication: peer="{Voter server-fc3157a1-e38e-1410-4693-3b06129397c4 fc3157a1-e38e-1410-4693-3b06129397c4}" 2025-01-28T14:47:11.326Z [ERROR] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to appendEntries to: peer="{Voter server-fc3157a1-e38e-1410-4693-3b06129397c4 fc3157a1-e38e-1410-4693-3b06129397c4}" error="failed to connect to peer: fc3157a1-e38e-1410-4693-3b06129397c4" 2025-01-28T14:47:11.328Z [ERROR] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to heartbeat to: peer=fc3157a1-e38e-1410-4693-3b06129397c4 backoff time=10ms error="failed to connect to peer: fc3157a1-e38e-1410-4693-3b06129397c4" 2025-01-28T14:47:11.333Z [ERROR] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to heartbeat to: peer=c7a92434-7a92-4fd8-b335-7567fef318aa backoff time=10ms error="failed to connect to peer: c7a92434-7a92-4fd8-b335-7567fef318aa" 2025-01-28T14:47:11.335Z [ERROR] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to appendEntries to: peer="{Voter server-c7a92434-7a92-4fd8-b335-7567fef318aa c7a92434-7a92-4fd8-b335-7567fef318aa}" error="failed to connect to peer: c7a92434-7a92-4fd8-b335-7567fef318aa" 2025-01-28T14:47:11.343Z [ERROR] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to appendEntries to: peer="{Voter server-fc3157a1-e38e-1410-4693-3b06129397c4 fc3157a1-e38e-1410-4693-3b06129397c4}" error="failed to connect to peer: fc3157a1-e38e-1410-4693-3b06129397c4" 2025-01-28T14:47:11.345Z [ERROR] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to heartbeat to: peer=fc3157a1-e38e-1410-4693-3b06129397c4 backoff time=10ms error="failed to connect to peer: fc3157a1-e38e-1410-4693-3b06129397c4" 2025-01-28T14:47:11.351Z [ERROR] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to heartbeat to: peer=c7a92434-7a92-4fd8-b335-7567fef318aa backoff time=10ms error="failed to connect to peer: c7a92434-7a92-4fd8-b335-7567fef318aa" 2025-01-28T14:47:11.354Z [ERROR] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to appendEntries to: peer="{Voter server-c7a92434-7a92-4fd8-b335-7567fef318aa c7a92434-7a92-4fd8-b335-7567fef318aa}" error="failed to connect to peer: c7a92434-7a92-4fd8-b335-7567fef318aa" 2025-01-28T14:47:11.360Z [ERROR] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to appendEntries to: peer="{Voter server-fc3157a1-e38e-1410-4693-3b06129397c4 fc3157a1-e38e-1410-4693-3b06129397c4}" error="failed to connect to peer: fc3157a1-e38e-1410-4693-3b06129397c4" 2025-01-28T14:47:11.362Z [ERROR] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to heartbeat to: peer=fc3157a1-e38e-1410-4693-3b06129397c4 backoff time=10ms error="failed to connect to peer: fc3157a1-e38e-1410-4693-3b06129397c4" 2025-01-28T14:47:11.370Z [WARN] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to contact: server-id=server-fc3157a1-e38e-1410-4693-3b06129397c4 time=51.244855ms 2025-01-28T14:47:11.371Z [ERROR] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to appendEntries to: peer="{Voter server-c7a92434-7a92-4fd8-b335-7567fef318aa c7a92434-7a92-4fd8-b335-7567fef318aa}" error="failed to connect to peer: c7a92434-7a92-4fd8-b335-7567fef318aa" 2025-01-28T14:47:11.373Z [ERROR] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to heartbeat to: peer=c7a92434-7a92-4fd8-b335-7567fef318aa backoff time=10ms error="failed to connect to peer: c7a92434-7a92-4fd8-b335-7567fef318aa" 2025-01-28T14:47:11.378Z [WARN] server-c7a92434-7a92-4fd8-b335-7567fef318aa: heartbeat timeout reached, starting election: last-leader-addr=936c0f23-73e1-513b-2c9e-6ce7e97dbc91 last-leader-id=server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91 2025-01-28T14:47:11.378Z [INFO] server-c7a92434-7a92-4fd8-b335-7567fef318aa: entering candidate state: node="Node at c7a92434-7a92-4fd8-b335-7567fef318aa [Candidate]" term=3 2025-01-28T14:47:11.378Z [ERROR] server-c7a92434-7a92-4fd8-b335-7567fef318aa: failed to make requestVote RPC: target="{Voter server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91 936c0f23-73e1-513b-2c9e-6ce7e97dbc91}" error="failed to connect to peer: 936c0f23-73e1-513b-2c9e-6ce7e97dbc91" term=3 2025-01-28T14:47:11.379Z [WARN] server-fc3157a1-e38e-1410-4693-3b06129397c4: rejecting vote request since we have a leader: from=c7a92434-7a92-4fd8-b335-7567fef318aa leader=936c0f23-73e1-513b-2c9e-6ce7e97dbc91 leader-id=server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91 2025-01-28T14:47:11.378Z [ERROR] server-c7a92434-7a92-4fd8-b335-7567fef318aa: failed to make requestVote RPC: target="{Voter server-6b6349ac-b54a-04f0-d1ad-b695097b35c5 6b6349ac-b54a-04f0-d1ad-b695097b35c5}" error="failed to connect to peer: 6b6349ac-b54a-04f0-d1ad-b695097b35c5" term=3 2025-01-28T14:47:11.381Z [WARN] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to contact: server-id=server-c7a92434-7a92-4fd8-b335-7567fef318aa time=57.708047ms 2025-01-28T14:47:11.381Z [WARN] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to contact: server-id=server-fc3157a1-e38e-1410-4693-3b06129397c4 time=61.863837ms 2025-01-28T14:47:11.381Z [WARN] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to contact quorum of nodes, stepping down 2025-01-28T14:47:11.382Z [INFO] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: entering follower state: follower="Node at 936c0f23-73e1-513b-2c9e-6ce7e97dbc91 [Follower]" leader-address= leader-id= 2025-01-28T14:47:11.382Z [ERROR] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to heartbeat to: peer=fc3157a1-e38e-1410-4693-3b06129397c4 backoff time=20ms error="failed to connect to peer: fc3157a1-e38e-1410-4693-3b06129397c4" 2025-01-28T14:47:11.382Z [ERROR] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to appendEntries to: peer="{Voter server-c7a92434-7a92-4fd8-b335-7567fef318aa c7a92434-7a92-4fd8-b335-7567fef318aa}" error="failed to connect to peer: c7a92434-7a92-4fd8-b335-7567fef318aa" 2025-01-28T14:47:11.382Z [ERROR] server-936c0f23-73e1-513b-2c9e-6ce7e97dbc91: failed to appendEntries to: peer="{Voter server-fc3157a1-e38e-1410-4693-3b06129397c4 fc3157a1-e38e-1410-4693-3b06129397c4}" error="failed to connect to peer: fc3157a1-e38e-1410-4693-3b06129397c4" --- PASS: TestRaft_RemoveFollower_SplitCluster (0.46s) === RUN TestRaft_AddKnownPeer 2025-01-28T14:47:11.403Z [INFO] server-590c519e-4232-2b34-c578-3ea3353245d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-590c519e-4232-2b34-c578-3ea3353245d7 Address:590c519e-4232-2b34-c578-3ea3353245d7} {Suffrage:Voter ID:server-7a3c82da-b7dc-927b-f82b-1658e9cdb1d1 Address:7a3c82da-b7dc-927b-f82b-1658e9cdb1d1} {Suffrage:Voter ID:server-43254534-c596-2eea-1096-1a56d7141b20 Address:43254534-c596-2eea-1096-1a56d7141b20}]" 2025-01-28T14:47:11.403Z [INFO] server-590c519e-4232-2b34-c578-3ea3353245d7: entering follower state: follower="Node at 590c519e-4232-2b34-c578-3ea3353245d7 [Follower]" leader-address= leader-id= 2025-01-28T14:47:11.404Z [INFO] server-7a3c82da-b7dc-927b-f82b-1658e9cdb1d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-590c519e-4232-2b34-c578-3ea3353245d7 Address:590c519e-4232-2b34-c578-3ea3353245d7} {Suffrage:Voter ID:server-7a3c82da-b7dc-927b-f82b-1658e9cdb1d1 Address:7a3c82da-b7dc-927b-f82b-1658e9cdb1d1} {Suffrage:Voter ID:server-43254534-c596-2eea-1096-1a56d7141b20 Address:43254534-c596-2eea-1096-1a56d7141b20}]" 2025-01-28T14:47:11.405Z [INFO] server-7a3c82da-b7dc-927b-f82b-1658e9cdb1d1: entering follower state: follower="Node at 7a3c82da-b7dc-927b-f82b-1658e9cdb1d1 [Follower]" leader-address= leader-id= 2025-01-28T14:47:11.406Z [INFO] server-43254534-c596-2eea-1096-1a56d7141b20: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-590c519e-4232-2b34-c578-3ea3353245d7 Address:590c519e-4232-2b34-c578-3ea3353245d7} {Suffrage:Voter ID:server-7a3c82da-b7dc-927b-f82b-1658e9cdb1d1 Address:7a3c82da-b7dc-927b-f82b-1658e9cdb1d1} {Suffrage:Voter ID:server-43254534-c596-2eea-1096-1a56d7141b20 Address:43254534-c596-2eea-1096-1a56d7141b20}]" 2025-01-28T14:47:11.406Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:11.407Z [INFO] server-43254534-c596-2eea-1096-1a56d7141b20: entering follower state: follower="Node at 43254534-c596-2eea-1096-1a56d7141b20 [Follower]" leader-address= leader-id= 2025-01-28T14:47:11.461Z [WARN] server-590c519e-4232-2b34-c578-3ea3353245d7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:11.461Z [INFO] server-590c519e-4232-2b34-c578-3ea3353245d7: entering candidate state: node="Node at 590c519e-4232-2b34-c578-3ea3353245d7 [Candidate]" term=2 2025-01-28T14:47:11.462Z [INFO] server-590c519e-4232-2b34-c578-3ea3353245d7: election won: term=2 tally=2 2025-01-28T14:47:11.462Z [INFO] server-590c519e-4232-2b34-c578-3ea3353245d7: entering leader state: leader="Node at 590c519e-4232-2b34-c578-3ea3353245d7 [Leader]" 2025-01-28T14:47:11.462Z [INFO] server-590c519e-4232-2b34-c578-3ea3353245d7: added peer, starting replication: peer=server-7a3c82da-b7dc-927b-f82b-1658e9cdb1d1 2025-01-28T14:47:11.463Z [INFO] server-590c519e-4232-2b34-c578-3ea3353245d7: added peer, starting replication: peer=server-43254534-c596-2eea-1096-1a56d7141b20 2025-01-28T14:47:11.464Z [INFO] server-590c519e-4232-2b34-c578-3ea3353245d7: pipelining replication: peer="{Voter server-7a3c82da-b7dc-927b-f82b-1658e9cdb1d1 7a3c82da-b7dc-927b-f82b-1658e9cdb1d1}" 2025-01-28T14:47:11.464Z [INFO] server-590c519e-4232-2b34-c578-3ea3353245d7: pipelining replication: peer="{Voter server-43254534-c596-2eea-1096-1a56d7141b20 43254534-c596-2eea-1096-1a56d7141b20}" 2025-01-28T14:47:11.569Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:11.463016829 +0000 UTC m=+14.546423187 (1 nodes), 55.979995ms from start of poll, 61.002478ms from cluster start. Timeout at 2025-01-28 14:47:11.568036497 +0000 UTC m=+14.651442522, 105.019335ms after stability 2025-01-28T14:47:11.569Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:11.675Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:11.569973005 +0000 UTC m=+14.653379363 (2 nodes), 16.668µs from start of poll, 167.958654ms from cluster start. Timeout at 2025-01-28 14:47:11.675066674 +0000 UTC m=+14.758472698, 105.093335ms after stability 2025-01-28T14:47:11.675Z [INFO] server-590c519e-4232-2b34-c578-3ea3353245d7: updating configuration: command=AddVoter server-id=server-7a3c82da-b7dc-927b-f82b-1658e9cdb1d1 server-addr=7a3c82da-b7dc-927b-f82b-1658e9cdb1d1 servers="[{Suffrage:Voter ID:server-590c519e-4232-2b34-c578-3ea3353245d7 Address:590c519e-4232-2b34-c578-3ea3353245d7} {Suffrage:Voter ID:server-7a3c82da-b7dc-927b-f82b-1658e9cdb1d1 Address:7a3c82da-b7dc-927b-f82b-1658e9cdb1d1} {Suffrage:Voter ID:server-43254534-c596-2eea-1096-1a56d7141b20 Address:43254534-c596-2eea-1096-1a56d7141b20}]" 2025-01-28T14:47:11.680Z [INFO] server-590c519e-4232-2b34-c578-3ea3353245d7: aborting pipeline replication: peer="{Voter server-43254534-c596-2eea-1096-1a56d7141b20 43254534-c596-2eea-1096-1a56d7141b20}" 2025-01-28T14:47:11.680Z [INFO] server-590c519e-4232-2b34-c578-3ea3353245d7: aborting pipeline replication: peer="{Voter server-7a3c82da-b7dc-927b-f82b-1658e9cdb1d1 7a3c82da-b7dc-927b-f82b-1658e9cdb1d1}" --- PASS: TestRaft_AddKnownPeer (0.29s) === RUN TestRaft_RemoveUnknownPeer 2025-01-28T14:47:11.697Z [INFO] server-62c0d1ef-75d6-2746-3250-de3d8ace4f16: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62c0d1ef-75d6-2746-3250-de3d8ace4f16 Address:62c0d1ef-75d6-2746-3250-de3d8ace4f16} {Suffrage:Voter ID:server-971de209-6a92-8bfc-97e5-fa010cd91795 Address:971de209-6a92-8bfc-97e5-fa010cd91795} {Suffrage:Voter ID:server-0e079425-9518-c376-3564-dd199eec0866 Address:0e079425-9518-c376-3564-dd199eec0866}]" 2025-01-28T14:47:11.698Z [INFO] server-62c0d1ef-75d6-2746-3250-de3d8ace4f16: entering follower state: follower="Node at 62c0d1ef-75d6-2746-3250-de3d8ace4f16 [Follower]" leader-address= leader-id= 2025-01-28T14:47:11.699Z [INFO] server-971de209-6a92-8bfc-97e5-fa010cd91795: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62c0d1ef-75d6-2746-3250-de3d8ace4f16 Address:62c0d1ef-75d6-2746-3250-de3d8ace4f16} {Suffrage:Voter ID:server-971de209-6a92-8bfc-97e5-fa010cd91795 Address:971de209-6a92-8bfc-97e5-fa010cd91795} {Suffrage:Voter ID:server-0e079425-9518-c376-3564-dd199eec0866 Address:0e079425-9518-c376-3564-dd199eec0866}]" 2025-01-28T14:47:11.700Z [INFO] server-971de209-6a92-8bfc-97e5-fa010cd91795: entering follower state: follower="Node at 971de209-6a92-8bfc-97e5-fa010cd91795 [Follower]" leader-address= leader-id= 2025-01-28T14:47:11.701Z [INFO] server-0e079425-9518-c376-3564-dd199eec0866: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62c0d1ef-75d6-2746-3250-de3d8ace4f16 Address:62c0d1ef-75d6-2746-3250-de3d8ace4f16} {Suffrage:Voter ID:server-971de209-6a92-8bfc-97e5-fa010cd91795 Address:971de209-6a92-8bfc-97e5-fa010cd91795} {Suffrage:Voter ID:server-0e079425-9518-c376-3564-dd199eec0866 Address:0e079425-9518-c376-3564-dd199eec0866}]" 2025-01-28T14:47:11.702Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:11.702Z [INFO] server-0e079425-9518-c376-3564-dd199eec0866: entering follower state: follower="Node at 0e079425-9518-c376-3564-dd199eec0866 [Follower]" leader-address= leader-id= 2025-01-28T14:47:11.758Z [WARN] server-971de209-6a92-8bfc-97e5-fa010cd91795: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:11.758Z [INFO] server-971de209-6a92-8bfc-97e5-fa010cd91795: entering candidate state: node="Node at 971de209-6a92-8bfc-97e5-fa010cd91795 [Candidate]" term=2 2025-01-28T14:47:11.759Z [INFO] server-971de209-6a92-8bfc-97e5-fa010cd91795: election won: term=2 tally=2 2025-01-28T14:47:11.759Z [INFO] server-971de209-6a92-8bfc-97e5-fa010cd91795: entering leader state: leader="Node at 971de209-6a92-8bfc-97e5-fa010cd91795 [Leader]" 2025-01-28T14:47:11.760Z [INFO] server-971de209-6a92-8bfc-97e5-fa010cd91795: added peer, starting replication: peer=server-62c0d1ef-75d6-2746-3250-de3d8ace4f16 2025-01-28T14:47:11.760Z [INFO] server-971de209-6a92-8bfc-97e5-fa010cd91795: added peer, starting replication: peer=server-0e079425-9518-c376-3564-dd199eec0866 2025-01-28T14:47:11.761Z [INFO] server-971de209-6a92-8bfc-97e5-fa010cd91795: pipelining replication: peer="{Voter server-62c0d1ef-75d6-2746-3250-de3d8ace4f16 62c0d1ef-75d6-2746-3250-de3d8ace4f16}" 2025-01-28T14:47:11.761Z [INFO] server-971de209-6a92-8bfc-97e5-fa010cd91795: pipelining replication: peer="{Voter server-0e079425-9518-c376-3564-dd199eec0866 0e079425-9518-c376-3564-dd199eec0866}" 2025-01-28T14:47:11.865Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:11.759980312 +0000 UTC m=+14.843386670 (1 nodes), 57.667045ms from start of poll, 63.222877ms from cluster start. Timeout at 2025-01-28 14:47:11.865017314 +0000 UTC m=+14.948423672, 105.037002ms after stability 2025-01-28T14:47:11.865Z [INFO] server-971de209-6a92-8bfc-97e5-fa010cd91795: updating configuration: command=RemoveServer server-id=eb2a8234-0b01-7cfb-eea6-d505480a7285 server-addr= servers="[{Suffrage:Voter ID:server-62c0d1ef-75d6-2746-3250-de3d8ace4f16 Address:62c0d1ef-75d6-2746-3250-de3d8ace4f16} {Suffrage:Voter ID:server-971de209-6a92-8bfc-97e5-fa010cd91795 Address:971de209-6a92-8bfc-97e5-fa010cd91795} {Suffrage:Voter ID:server-0e079425-9518-c376-3564-dd199eec0866 Address:0e079425-9518-c376-3564-dd199eec0866}]" 2025-01-28T14:47:11.868Z [INFO] server-971de209-6a92-8bfc-97e5-fa010cd91795: aborting pipeline replication: peer="{Voter server-0e079425-9518-c376-3564-dd199eec0866 0e079425-9518-c376-3564-dd199eec0866}" 2025-01-28T14:47:11.868Z [INFO] server-971de209-6a92-8bfc-97e5-fa010cd91795: aborting pipeline replication: peer="{Voter server-62c0d1ef-75d6-2746-3250-de3d8ace4f16 62c0d1ef-75d6-2746-3250-de3d8ace4f16}" --- PASS: TestRaft_RemoveUnknownPeer (0.19s) === RUN TestRaft_SnapshotRestore 2025-01-28T14:47:11.881Z [INFO] server-884a97c4-a05f-e5ae-65f2-e50401e8f4a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-884a97c4-a05f-e5ae-65f2-e50401e8f4a1 Address:884a97c4-a05f-e5ae-65f2-e50401e8f4a1}]" 2025-01-28T14:47:11.881Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:11.882Z [INFO] server-884a97c4-a05f-e5ae-65f2-e50401e8f4a1: entering follower state: follower="Node at 884a97c4-a05f-e5ae-65f2-e50401e8f4a1 [Follower]" leader-address= leader-id= 2025-01-28T14:47:11.963Z [WARN] server-884a97c4-a05f-e5ae-65f2-e50401e8f4a1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:11.963Z [INFO] server-884a97c4-a05f-e5ae-65f2-e50401e8f4a1: entering candidate state: node="Node at 884a97c4-a05f-e5ae-65f2-e50401e8f4a1 [Candidate]" term=2 2025-01-28T14:47:11.964Z [INFO] server-884a97c4-a05f-e5ae-65f2-e50401e8f4a1: election won: term=2 tally=1 2025-01-28T14:47:11.964Z [INFO] server-884a97c4-a05f-e5ae-65f2-e50401e8f4a1: entering leader state: leader="Node at 884a97c4-a05f-e5ae-65f2-e50401e8f4a1 [Leader]" 2025-01-28T14:47:12.071Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:11.964751392 +0000 UTC m=+15.048157750 (1 nodes), 82.689122ms from start of poll, 84.348838ms from cluster start. Timeout at 2025-01-28 14:47:12.069825728 +0000 UTC m=+15.153231752, 105.074002ms after stability 2025-01-28T14:47:12.122Z [INFO] server-884a97c4-a05f-e5ae-65f2-e50401e8f4a1: starting snapshot up to: index=102 2025-01-28T14:47:12.122Z [INFO] creating new snapshot: path=/tmp/raft2293657919/snapshots/2-102-1738075632122.tmp 2025-01-28T14:47:12.129Z [INFO] server-884a97c4-a05f-e5ae-65f2-e50401e8f4a1: compacting logs: from=1 to=92 2025-01-28T14:47:12.129Z [INFO] server-884a97c4-a05f-e5ae-65f2-e50401e8f4a1: snapshot complete up to: index=102 2025-01-28T14:47:12.132Z [INFO] server-884a97c4-a05f-e5ae-65f2-e50401e8f4a1: starting restore from snapshot: id=2-102-1738075632122 last-index=102 last-term=2 size-in-bytes=693 2025-01-28T14:47:12.135Z [INFO] server-884a97c4-a05f-e5ae-65f2-e50401e8f4a1: snapshot restore progress: id=2-102-1738075632122 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2025-01-28T14:47:12.135Z [INFO] server-884a97c4-a05f-e5ae-65f2-e50401e8f4a1: restored from snapshot: id=2-102-1738075632122 last-index=102 last-term=2 size-in-bytes=693 2025-01-28T14:47:12.135Z [INFO] server-884a97c4-a05f-e5ae-65f2-e50401e8f4a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-884a97c4-a05f-e5ae-65f2-e50401e8f4a1 Address:884a97c4-a05f-e5ae-65f2-e50401e8f4a1}]" --- PASS: TestRaft_SnapshotRestore (0.26s) === RUN TestRaft_RestoreSnapshotOnStartup_Monotonic 2025-01-28T14:47:12.144Z [INFO] server-62d76cd7-062f-60f2-a993-deb148e8cb66: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62d76cd7-062f-60f2-a993-deb148e8cb66 Address:62d76cd7-062f-60f2-a993-deb148e8cb66}]" 2025-01-28T14:47:12.145Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:12.145Z [INFO] server-62d76cd7-062f-60f2-a993-deb148e8cb66: entering follower state: follower="Node at 62d76cd7-062f-60f2-a993-deb148e8cb66 [Follower]" leader-address= leader-id= 2025-01-28T14:47:12.214Z [WARN] server-62d76cd7-062f-60f2-a993-deb148e8cb66: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:12.214Z [INFO] server-62d76cd7-062f-60f2-a993-deb148e8cb66: entering candidate state: node="Node at 62d76cd7-062f-60f2-a993-deb148e8cb66 [Candidate]" term=2 2025-01-28T14:47:12.215Z [INFO] server-62d76cd7-062f-60f2-a993-deb148e8cb66: election won: term=2 tally=1 2025-01-28T14:47:12.215Z [INFO] server-62d76cd7-062f-60f2-a993-deb148e8cb66: entering leader state: leader="Node at 62d76cd7-062f-60f2-a993-deb148e8cb66 [Leader]" 2025-01-28T14:47:12.321Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:12.215670508 +0000 UTC m=+15.299076866 (1 nodes), 70.329422ms from start of poll, 71.827466ms from cluster start. Timeout at 2025-01-28 14:47:12.320729177 +0000 UTC m=+15.404135201, 105.058335ms after stability 2025-01-28T14:47:12.368Z [INFO] server-62d76cd7-062f-60f2-a993-deb148e8cb66: starting snapshot up to: index=102 2025-01-28T14:47:12.368Z [INFO] creating new snapshot: path=/tmp/raft1978134631/snapshots/2-102-1738075632368.tmp 2025-01-28T14:47:12.374Z [INFO] server-62d76cd7-062f-60f2-a993-deb148e8cb66: compacting logs: from=1 to=92 2025-01-28T14:47:12.375Z [INFO] server-62d76cd7-062f-60f2-a993-deb148e8cb66: snapshot complete up to: index=102 2025-01-28T14:47:12.377Z [INFO] server-62d76cd7-062f-60f2-a993-deb148e8cb66: starting restore from snapshot: id=2-102-1738075632368 last-index=102 last-term=2 size-in-bytes=693 2025-01-28T14:47:12.380Z [INFO] server-62d76cd7-062f-60f2-a993-deb148e8cb66: snapshot restore progress: id=2-102-1738075632368 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2025-01-28T14:47:12.380Z [INFO] server-62d76cd7-062f-60f2-a993-deb148e8cb66: restored from snapshot: id=2-102-1738075632368 last-index=102 last-term=2 size-in-bytes=693 2025-01-28T14:47:12.380Z [INFO] server-62d76cd7-062f-60f2-a993-deb148e8cb66: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62d76cd7-062f-60f2-a993-deb148e8cb66 Address:62d76cd7-062f-60f2-a993-deb148e8cb66}]" 2025-01-28T14:47:12.380Z [INFO] server-62d76cd7-062f-60f2-a993-deb148e8cb66: entering follower state: follower="Node at 62d76cd7-062f-60f2-a993-deb148e8cb66 [Shutdown]" leader-address= leader-id= --- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.24s) === RUN TestRaft_SnapshotRestore_Progress 2025-01-28T14:47:12.390Z [INFO] server-26c267a5-1274-601c-7df4-affc5dde90d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26c267a5-1274-601c-7df4-affc5dde90d7 Address:26c267a5-1274-601c-7df4-affc5dde90d7}]" 2025-01-28T14:47:12.390Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:12.390Z [INFO] server-26c267a5-1274-601c-7df4-affc5dde90d7: entering follower state: follower="Node at 26c267a5-1274-601c-7df4-affc5dde90d7 [Follower]" leader-address= leader-id= 2025-01-28T14:47:12.492Z [WARN] server-26c267a5-1274-601c-7df4-affc5dde90d7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:12.492Z [INFO] server-26c267a5-1274-601c-7df4-affc5dde90d7: entering candidate state: node="Node at 26c267a5-1274-601c-7df4-affc5dde90d7 [Candidate]" term=2 2025-01-28T14:47:12.492Z [INFO] server-26c267a5-1274-601c-7df4-affc5dde90d7: election won: term=2 tally=1 2025-01-28T14:47:12.493Z [INFO] server-26c267a5-1274-601c-7df4-affc5dde90d7: entering leader state: leader="Node at 26c267a5-1274-601c-7df4-affc5dde90d7 [Leader]" 2025-01-28T14:47:12.599Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:12.493309751 +0000 UTC m=+15.576716108 (1 nodes), 102.635713ms from start of poll, 104.156425ms from cluster start. Timeout at 2025-01-28 14:47:12.598344753 +0000 UTC m=+15.681750777, 105.034669ms after stability 2025-01-28T14:47:12.647Z [INFO] server-26c267a5-1274-601c-7df4-affc5dde90d7: starting snapshot up to: index=102 2025-01-28T14:47:12.648Z [INFO] creating new snapshot: path=/tmp/raft3979978216/snapshots/2-102-1738075632647.tmp 2025-01-28T14:47:12.655Z [INFO] server-26c267a5-1274-601c-7df4-affc5dde90d7: compacting logs: from=1 to=92 2025-01-28T14:47:12.655Z [INFO] server-26c267a5-1274-601c-7df4-affc5dde90d7: snapshot complete up to: index=102 --- PASS: TestRaft_SnapshotRestore_Progress (0.28s) === RUN TestRaft_NoRestoreOnStart 2025-01-28T14:47:12.672Z [INFO] server-ca8f3287-0908-181a-20c5-650ee85ec296: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca8f3287-0908-181a-20c5-650ee85ec296 Address:ca8f3287-0908-181a-20c5-650ee85ec296}]" 2025-01-28T14:47:12.673Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:12.673Z [INFO] server-ca8f3287-0908-181a-20c5-650ee85ec296: entering follower state: follower="Node at ca8f3287-0908-181a-20c5-650ee85ec296 [Follower]" leader-address= leader-id= 2025-01-28T14:47:12.735Z [WARN] server-ca8f3287-0908-181a-20c5-650ee85ec296: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:12.735Z [INFO] server-ca8f3287-0908-181a-20c5-650ee85ec296: entering candidate state: node="Node at ca8f3287-0908-181a-20c5-650ee85ec296 [Candidate]" term=2 2025-01-28T14:47:12.735Z [INFO] server-ca8f3287-0908-181a-20c5-650ee85ec296: election won: term=2 tally=1 2025-01-28T14:47:12.736Z [INFO] server-ca8f3287-0908-181a-20c5-650ee85ec296: entering leader state: leader="Node at ca8f3287-0908-181a-20c5-650ee85ec296 [Leader]" 2025-01-28T14:47:12.843Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:12.737152657 +0000 UTC m=+15.820559015 (1 nodes), 63.772227ms from start of poll, 65.260605ms from cluster start. Timeout at 2025-01-28 14:47:12.842193992 +0000 UTC m=+15.925600016, 105.041001ms after stability 2025-01-28T14:47:12.899Z [INFO] server-ca8f3287-0908-181a-20c5-650ee85ec296: starting snapshot up to: index=102 2025-01-28T14:47:12.899Z [INFO] creating new snapshot: path=/tmp/raft1739837881/snapshots/2-102-1738075632899.tmp 2025-01-28T14:47:12.906Z [INFO] server-ca8f3287-0908-181a-20c5-650ee85ec296: compacting logs: from=1 to=92 2025-01-28T14:47:12.906Z [INFO] server-ca8f3287-0908-181a-20c5-650ee85ec296: snapshot complete up to: index=102 2025-01-28T14:47:12.909Z [INFO] server-ca8f3287-0908-181a-20c5-650ee85ec296: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca8f3287-0908-181a-20c5-650ee85ec296 Address:ca8f3287-0908-181a-20c5-650ee85ec296}]" --- PASS: TestRaft_NoRestoreOnStart (0.24s) 2025-01-28T14:47:12.909Z [INFO] server-ca8f3287-0908-181a-20c5-650ee85ec296: entering follower state: follower="Node at ca8f3287-0908-181a-20c5-650ee85ec296 [Follower]" leader-address= leader-id= === RUN TestRaft_SnapshotRestore_PeerChange raft_test.go:1266: Skipping unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot 2025-01-28T14:47:12.915Z [INFO] server-57196a83-cc55-e28e-15f3-812732bdaa86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57196a83-cc55-e28e-15f3-812732bdaa86 Address:57196a83-cc55-e28e-15f3-812732bdaa86}]" 2025-01-28T14:47:12.916Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:12.916Z [INFO] server-57196a83-cc55-e28e-15f3-812732bdaa86: entering follower state: follower="Node at 57196a83-cc55-e28e-15f3-812732bdaa86 [Follower]" leader-address= leader-id= 2025-01-28T14:47:12.964Z [WARN] server-ca8f3287-0908-181a-20c5-650ee85ec296: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:12.964Z [INFO] server-ca8f3287-0908-181a-20c5-650ee85ec296: entering candidate state: node="Node at ca8f3287-0908-181a-20c5-650ee85ec296 [Candidate]" term=3 2025-01-28T14:47:12.965Z [INFO] server-ca8f3287-0908-181a-20c5-650ee85ec296: election won: term=3 tally=1 2025-01-28T14:47:12.965Z [INFO] server-ca8f3287-0908-181a-20c5-650ee85ec296: entering leader state: leader="Node at ca8f3287-0908-181a-20c5-650ee85ec296 [Leader]" 2025-01-28T14:47:12.987Z [WARN] server-57196a83-cc55-e28e-15f3-812732bdaa86: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:12.987Z [INFO] server-57196a83-cc55-e28e-15f3-812732bdaa86: entering candidate state: node="Node at 57196a83-cc55-e28e-15f3-812732bdaa86 [Candidate]" term=2 2025-01-28T14:47:12.988Z [INFO] server-57196a83-cc55-e28e-15f3-812732bdaa86: election won: term=2 tally=1 2025-01-28T14:47:12.988Z [INFO] server-57196a83-cc55-e28e-15f3-812732bdaa86: entering leader state: leader="Node at 57196a83-cc55-e28e-15f3-812732bdaa86 [Leader]" 2025-01-28T14:47:13.095Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:12.988784794 +0000 UTC m=+16.072190819 (1 nodes), 72.542154ms from start of poll, 74.008198ms from cluster start. Timeout at 2025-01-28 14:47:13.093822795 +0000 UTC m=+16.177228820, 105.038001ms after stability 2025-01-28T14:47:13.125Z [INFO] server-57196a83-cc55-e28e-15f3-812732bdaa86: starting snapshot up to: index=63 2025-01-28T14:47:13.125Z [INFO] creating new snapshot: path=/tmp/raft713996494/snapshots/2-63-1738075633125.tmp 2025-01-28T14:47:13.134Z [INFO] server-57196a83-cc55-e28e-15f3-812732bdaa86: compacting logs: from=1 to=63 2025-01-28T14:47:13.134Z [INFO] server-57196a83-cc55-e28e-15f3-812732bdaa86: snapshot complete up to: index=63 --- PASS: TestRaft_AutoSnapshot (0.34s) === RUN TestRaft_UserSnapshot 2025-01-28T14:47:13.252Z [INFO] server-51b56a40-9b2b-2990-8cfb-737ad7096eac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-51b56a40-9b2b-2990-8cfb-737ad7096eac Address:51b56a40-9b2b-2990-8cfb-737ad7096eac}]" 2025-01-28T14:47:13.252Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:13.253Z [INFO] server-51b56a40-9b2b-2990-8cfb-737ad7096eac: entering follower state: follower="Node at 51b56a40-9b2b-2990-8cfb-737ad7096eac [Follower]" leader-address= leader-id= 2025-01-28T14:47:13.305Z [WARN] server-51b56a40-9b2b-2990-8cfb-737ad7096eac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:13.305Z [INFO] server-51b56a40-9b2b-2990-8cfb-737ad7096eac: entering candidate state: node="Node at 51b56a40-9b2b-2990-8cfb-737ad7096eac [Candidate]" term=2 2025-01-28T14:47:13.306Z [INFO] server-51b56a40-9b2b-2990-8cfb-737ad7096eac: election won: term=2 tally=1 2025-01-28T14:47:13.306Z [INFO] server-51b56a40-9b2b-2990-8cfb-737ad7096eac: entering leader state: leader="Node at 51b56a40-9b2b-2990-8cfb-737ad7096eac [Leader]" 2025-01-28T14:47:13.413Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:13.307035909 +0000 UTC m=+16.390442267 (1 nodes), 53.948935ms from start of poll, 55.491647ms from cluster start. Timeout at 2025-01-28 14:47:13.412072243 +0000 UTC m=+16.495478601, 105.036334ms after stability 2025-01-28T14:47:13.414Z [ERROR] server-51b56a40-9b2b-2990-8cfb-737ad7096eac: failed to take snapshot: error="nothing new to snapshot" 2025-01-28T14:47:13.419Z [INFO] server-51b56a40-9b2b-2990-8cfb-737ad7096eac: starting snapshot up to: index=12 2025-01-28T14:47:13.420Z [INFO] creating new snapshot: path=/tmp/raft2817216302/snapshots/2-12-1738075633420.tmp 2025-01-28T14:47:13.426Z [INFO] server-51b56a40-9b2b-2990-8cfb-737ad7096eac: compacting logs: from=1 to=2 2025-01-28T14:47:13.426Z [INFO] server-51b56a40-9b2b-2990-8cfb-737ad7096eac: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.18s) === RUN TestRaft_UserRestore === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false 2025-01-28T14:47:13.443Z [INFO] server-1226ab04-35b0-2be9-5be4-78b115ecd48a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1226ab04-35b0-2be9-5be4-78b115ecd48a Address:1226ab04-35b0-2be9-5be4-78b115ecd48a} {Suffrage:Voter ID:server-84b8fbe0-13b4-b759-c8dc-2380bb74f023 Address:84b8fbe0-13b4-b759-c8dc-2380bb74f023} {Suffrage:Voter ID:server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c Address:1dfb61d6-ddb4-bc22-0bf3-78241ca9782c}]" 2025-01-28T14:47:13.443Z [INFO] server-1226ab04-35b0-2be9-5be4-78b115ecd48a: entering follower state: follower="Node at 1226ab04-35b0-2be9-5be4-78b115ecd48a [Follower]" leader-address= leader-id= 2025-01-28T14:47:13.444Z [INFO] server-84b8fbe0-13b4-b759-c8dc-2380bb74f023: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1226ab04-35b0-2be9-5be4-78b115ecd48a Address:1226ab04-35b0-2be9-5be4-78b115ecd48a} {Suffrage:Voter ID:server-84b8fbe0-13b4-b759-c8dc-2380bb74f023 Address:84b8fbe0-13b4-b759-c8dc-2380bb74f023} {Suffrage:Voter ID:server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c Address:1dfb61d6-ddb4-bc22-0bf3-78241ca9782c}]" 2025-01-28T14:47:13.445Z [INFO] server-84b8fbe0-13b4-b759-c8dc-2380bb74f023: entering follower state: follower="Node at 84b8fbe0-13b4-b759-c8dc-2380bb74f023 [Follower]" leader-address= leader-id= 2025-01-28T14:47:13.446Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1226ab04-35b0-2be9-5be4-78b115ecd48a Address:1226ab04-35b0-2be9-5be4-78b115ecd48a} {Suffrage:Voter ID:server-84b8fbe0-13b4-b759-c8dc-2380bb74f023 Address:84b8fbe0-13b4-b759-c8dc-2380bb74f023} {Suffrage:Voter ID:server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c Address:1dfb61d6-ddb4-bc22-0bf3-78241ca9782c}]" 2025-01-28T14:47:13.446Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:13.446Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: entering follower state: follower="Node at 1dfb61d6-ddb4-bc22-0bf3-78241ca9782c [Follower]" leader-address= leader-id= 2025-01-28T14:47:13.970Z [WARN] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:13.970Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: entering candidate state: node="Node at 1dfb61d6-ddb4-bc22-0bf3-78241ca9782c [Candidate]" term=2 2025-01-28T14:47:13.971Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: election won: term=2 tally=2 2025-01-28T14:47:13.971Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: entering leader state: leader="Node at 1dfb61d6-ddb4-bc22-0bf3-78241ca9782c [Leader]" 2025-01-28T14:47:13.971Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: added peer, starting replication: peer=server-1226ab04-35b0-2be9-5be4-78b115ecd48a 2025-01-28T14:47:13.972Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: added peer, starting replication: peer=server-84b8fbe0-13b4-b759-c8dc-2380bb74f023 2025-01-28T14:47:13.973Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: pipelining replication: peer="{Voter server-84b8fbe0-13b4-b759-c8dc-2380bb74f023 84b8fbe0-13b4-b759-c8dc-2380bb74f023}" 2025-01-28T14:47:13.976Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: pipelining replication: peer="{Voter server-1226ab04-35b0-2be9-5be4-78b115ecd48a 1226ab04-35b0-2be9-5be4-78b115ecd48a}" 2025-01-28T14:47:14.980Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:13.975182746 +0000 UTC m=+17.058589437 (1 nodes), 528.206015ms from start of poll, 532.988157ms from cluster start. Timeout at 2025-01-28 14:47:14.980205079 +0000 UTC m=+18.063611104, 1.005021667s after stability 2025-01-28T14:47:14.987Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: starting snapshot up to: index=12 2025-01-28T14:47:14.987Z [INFO] creating new snapshot: path=/tmp/raft172538491/snapshots/2-12-1738075634987.tmp 2025-01-28T14:47:14.994Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: snapshot complete up to: index=12 2025-01-28T14:47:15.004Z [INFO] creating new snapshot: path=/tmp/raft172538491/snapshots/2-23-1738075635004.tmp 2025-01-28T14:47:15.011Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: copied to local snapshot: bytes=71 2025-01-28T14:47:15.013Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: snapshot restore progress: id=2-23-1738075635004 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:15.013Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: restored user snapshot: index=23 2025-01-28T14:47:15.014Z [ERROR] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: failed to get log: index=23 error="log not found" 2025-01-28T14:47:15.014Z [ERROR] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: failed to get log: index=23 error="log not found" 2025-01-28T14:47:15.014Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: aborting pipeline replication: peer="{Voter server-84b8fbe0-13b4-b759-c8dc-2380bb74f023 84b8fbe0-13b4-b759-c8dc-2380bb74f023}" 2025-01-28T14:47:15.014Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: aborting pipeline replication: peer="{Voter server-1226ab04-35b0-2be9-5be4-78b115ecd48a 1226ab04-35b0-2be9-5be4-78b115ecd48a}" 2025-01-28T14:47:15.022Z [ERROR] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: failed to get log: index=23 error="log not found" 2025-01-28T14:47:15.025Z [ERROR] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: failed to get log: index=23 error="log not found" 2025-01-28T14:47:15.027Z [INFO] creating new snapshot: path=/tmp/raft1680502720/snapshots/2-23-1738075635026.tmp 2025-01-28T14:47:15.034Z [INFO] creating new snapshot: path=/tmp/raft4173689595/snapshots/2-23-1738075635034.tmp 2025-01-28T14:47:15.035Z [INFO] server-1226ab04-35b0-2be9-5be4-78b115ecd48a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:15.038Z [INFO] server-84b8fbe0-13b4-b759-c8dc-2380bb74f023: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:15.043Z [INFO] server-1226ab04-35b0-2be9-5be4-78b115ecd48a: copied to local snapshot: bytes=71 2025-01-28T14:47:15.046Z [INFO] server-84b8fbe0-13b4-b759-c8dc-2380bb74f023: copied to local snapshot: bytes=71 2025-01-28T14:47:15.046Z [INFO] server-1226ab04-35b0-2be9-5be4-78b115ecd48a: snapshot restore progress: id=2-23-1738075635026 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:15.047Z [INFO] server-1226ab04-35b0-2be9-5be4-78b115ecd48a: Installed remote snapshot 2025-01-28T14:47:15.048Z [INFO] server-84b8fbe0-13b4-b759-c8dc-2380bb74f023: snapshot restore progress: id=2-23-1738075635034 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:15.049Z [INFO] server-84b8fbe0-13b4-b759-c8dc-2380bb74f023: Installed remote snapshot 2025-01-28T14:47:15.049Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: pipelining replication: peer="{Voter server-1226ab04-35b0-2be9-5be4-78b115ecd48a 1226ab04-35b0-2be9-5be4-78b115ecd48a}" 2025-01-28T14:47:15.049Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:15.051Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: pipelining replication: peer="{Voter server-84b8fbe0-13b4-b759-c8dc-2380bb74f023 84b8fbe0-13b4-b759-c8dc-2380bb74f023}" 2025-01-28T14:47:16.056Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:15.050115993 +0000 UTC m=+18.133522017 (2 nodes), 15.667µs from start of poll, 1.607920737s from cluster start. Timeout at 2025-01-28 14:47:16.055211329 +0000 UTC m=+19.138617686, 1.005095669s after stability 2025-01-28T14:47:16.075Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: aborting pipeline replication: peer="{Voter server-84b8fbe0-13b4-b759-c8dc-2380bb74f023 84b8fbe0-13b4-b759-c8dc-2380bb74f023}" 2025-01-28T14:47:16.075Z [INFO] server-1dfb61d6-ddb4-bc22-0bf3-78241ca9782c: aborting pipeline replication: peer="{Voter server-1226ab04-35b0-2be9-5be4-78b115ecd48a 1226ab04-35b0-2be9-5be4-78b115ecd48a}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false 2025-01-28T14:47:16.100Z [INFO] server-b48fead8-5972-2638-6f3c-ccb7a1f78a17: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b48fead8-5972-2638-6f3c-ccb7a1f78a17 Address:b48fead8-5972-2638-6f3c-ccb7a1f78a17} {Suffrage:Voter ID:server-ccf56c4c-764e-ee95-f90c-3c19dffcac3a Address:ccf56c4c-764e-ee95-f90c-3c19dffcac3a} {Suffrage:Voter ID:server-17035c84-a8b1-8e78-0330-ee95b2f35c9d Address:17035c84-a8b1-8e78-0330-ee95b2f35c9d}]" 2025-01-28T14:47:16.101Z [INFO] server-b48fead8-5972-2638-6f3c-ccb7a1f78a17: entering follower state: follower="Node at b48fead8-5972-2638-6f3c-ccb7a1f78a17 [Follower]" leader-address= leader-id= 2025-01-28T14:47:16.102Z [INFO] server-ccf56c4c-764e-ee95-f90c-3c19dffcac3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b48fead8-5972-2638-6f3c-ccb7a1f78a17 Address:b48fead8-5972-2638-6f3c-ccb7a1f78a17} {Suffrage:Voter ID:server-ccf56c4c-764e-ee95-f90c-3c19dffcac3a Address:ccf56c4c-764e-ee95-f90c-3c19dffcac3a} {Suffrage:Voter ID:server-17035c84-a8b1-8e78-0330-ee95b2f35c9d Address:17035c84-a8b1-8e78-0330-ee95b2f35c9d}]" 2025-01-28T14:47:16.102Z [INFO] server-ccf56c4c-764e-ee95-f90c-3c19dffcac3a: entering follower state: follower="Node at ccf56c4c-764e-ee95-f90c-3c19dffcac3a [Follower]" leader-address= leader-id= 2025-01-28T14:47:16.103Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b48fead8-5972-2638-6f3c-ccb7a1f78a17 Address:b48fead8-5972-2638-6f3c-ccb7a1f78a17} {Suffrage:Voter ID:server-ccf56c4c-764e-ee95-f90c-3c19dffcac3a Address:ccf56c4c-764e-ee95-f90c-3c19dffcac3a} {Suffrage:Voter ID:server-17035c84-a8b1-8e78-0330-ee95b2f35c9d Address:17035c84-a8b1-8e78-0330-ee95b2f35c9d}]" 2025-01-28T14:47:16.103Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:16.104Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: entering follower state: follower="Node at 17035c84-a8b1-8e78-0330-ee95b2f35c9d [Follower]" leader-address= leader-id= 2025-01-28T14:47:16.703Z [WARN] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:16.703Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: entering candidate state: node="Node at 17035c84-a8b1-8e78-0330-ee95b2f35c9d [Candidate]" term=2 2025-01-28T14:47:16.704Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: election won: term=2 tally=2 2025-01-28T14:47:16.704Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: entering leader state: leader="Node at 17035c84-a8b1-8e78-0330-ee95b2f35c9d [Leader]" 2025-01-28T14:47:16.705Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: added peer, starting replication: peer=server-b48fead8-5972-2638-6f3c-ccb7a1f78a17 2025-01-28T14:47:16.705Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: added peer, starting replication: peer=server-ccf56c4c-764e-ee95-f90c-3c19dffcac3a 2025-01-28T14:47:16.706Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: pipelining replication: peer="{Voter server-b48fead8-5972-2638-6f3c-ccb7a1f78a17 b48fead8-5972-2638-6f3c-ccb7a1f78a17}" 2025-01-28T14:47:16.707Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: pipelining replication: peer="{Voter server-ccf56c4c-764e-ee95-f90c-3c19dffcac3a ccf56c4c-764e-ee95-f90c-3c19dffcac3a}" 2025-01-28T14:47:17.711Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:16.705240132 +0000 UTC m=+19.788646156 (1 nodes), 601.087512ms from start of poll, 605.761984ms from cluster start. Timeout at 2025-01-28 14:47:17.710279466 +0000 UTC m=+20.793685490, 1.005039334s after stability 2025-01-28T14:47:17.721Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: starting snapshot up to: index=12 2025-01-28T14:47:17.721Z [INFO] creating new snapshot: path=/tmp/raft2278412720/snapshots/2-12-1738075637721.tmp 2025-01-28T14:47:17.728Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: snapshot complete up to: index=12 2025-01-28T14:47:17.737Z [INFO] creating new snapshot: path=/tmp/raft2278412720/snapshots/2-23-1738075637737.tmp 2025-01-28T14:47:17.744Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: copied to local snapshot: bytes=71 2025-01-28T14:47:17.746Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: snapshot restore progress: id=2-23-1738075637737 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:17.747Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: removing all old logs from log store 2025-01-28T14:47:17.747Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: compacting logs: from=1 to=22 2025-01-28T14:47:17.747Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: restored user snapshot: index=23 2025-01-28T14:47:17.747Z [ERROR] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: failed to get log: index=22 error="log not found" 2025-01-28T14:47:17.748Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: aborting pipeline replication: peer="{Voter server-ccf56c4c-764e-ee95-f90c-3c19dffcac3a ccf56c4c-764e-ee95-f90c-3c19dffcac3a}" 2025-01-28T14:47:17.748Z [ERROR] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: failed to get log: index=22 error="log not found" 2025-01-28T14:47:17.748Z [ERROR] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: failed to get log: index=22 error="log not found" 2025-01-28T14:47:17.748Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: aborting pipeline replication: peer="{Voter server-b48fead8-5972-2638-6f3c-ccb7a1f78a17 b48fead8-5972-2638-6f3c-ccb7a1f78a17}" 2025-01-28T14:47:17.752Z [INFO] creating new snapshot: path=/tmp/raft2291532567/snapshots/2-23-1738075637752.tmp 2025-01-28T14:47:17.755Z [INFO] server-ccf56c4c-764e-ee95-f90c-3c19dffcac3a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:17.756Z [ERROR] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: failed to get log: index=22 error="log not found" 2025-01-28T14:47:17.759Z [INFO] server-ccf56c4c-764e-ee95-f90c-3c19dffcac3a: copied to local snapshot: bytes=71 2025-01-28T14:47:17.761Z [INFO] creating new snapshot: path=/tmp/raft1291913436/snapshots/2-23-1738075637761.tmp 2025-01-28T14:47:17.762Z [INFO] server-ccf56c4c-764e-ee95-f90c-3c19dffcac3a: snapshot restore progress: id=2-23-1738075637752 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:17.762Z [INFO] server-ccf56c4c-764e-ee95-f90c-3c19dffcac3a: removing all old logs from log store 2025-01-28T14:47:17.762Z [INFO] server-ccf56c4c-764e-ee95-f90c-3c19dffcac3a: compacting logs: from=1 to=22 2025-01-28T14:47:17.763Z [INFO] server-ccf56c4c-764e-ee95-f90c-3c19dffcac3a: Installed remote snapshot 2025-01-28T14:47:17.764Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: pipelining replication: peer="{Voter server-ccf56c4c-764e-ee95-f90c-3c19dffcac3a ccf56c4c-764e-ee95-f90c-3c19dffcac3a}" 2025-01-28T14:47:17.764Z [INFO] server-b48fead8-5972-2638-6f3c-ccb7a1f78a17: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:17.765Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:17.768Z [INFO] server-b48fead8-5972-2638-6f3c-ccb7a1f78a17: copied to local snapshot: bytes=71 2025-01-28T14:47:17.771Z [INFO] server-b48fead8-5972-2638-6f3c-ccb7a1f78a17: snapshot restore progress: id=2-23-1738075637761 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:17.771Z [INFO] server-b48fead8-5972-2638-6f3c-ccb7a1f78a17: removing all old logs from log store 2025-01-28T14:47:17.771Z [INFO] server-b48fead8-5972-2638-6f3c-ccb7a1f78a17: compacting logs: from=1 to=22 2025-01-28T14:47:17.772Z [INFO] server-b48fead8-5972-2638-6f3c-ccb7a1f78a17: Installed remote snapshot 2025-01-28T14:47:17.773Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: pipelining replication: peer="{Voter server-b48fead8-5972-2638-6f3c-ccb7a1f78a17 b48fead8-5972-2638-6f3c-ccb7a1f78a17}" 2025-01-28T14:47:18.771Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:17.765522278 +0000 UTC m=+20.848927969 (2 nodes), 16.334µs from start of poll, 1.666043797s from cluster start. Timeout at 2025-01-28 14:47:18.77061428 +0000 UTC m=+21.854020304, 1.005092335s after stability 2025-01-28T14:47:18.790Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: aborting pipeline replication: peer="{Voter server-b48fead8-5972-2638-6f3c-ccb7a1f78a17 b48fead8-5972-2638-6f3c-ccb7a1f78a17}" 2025-01-28T14:47:18.790Z [INFO] server-17035c84-a8b1-8e78-0330-ee95b2f35c9d: aborting pipeline replication: peer="{Voter server-ccf56c4c-764e-ee95-f90c-3c19dffcac3a ccf56c4c-764e-ee95-f90c-3c19dffcac3a}" === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true 2025-01-28T14:47:18.815Z [INFO] server-4ff75592-9024-530c-833a-dc93f2cf222f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ff75592-9024-530c-833a-dc93f2cf222f Address:4ff75592-9024-530c-833a-dc93f2cf222f} {Suffrage:Voter ID:server-d9d3529d-9515-358b-f54a-9e5fd80525d8 Address:d9d3529d-9515-358b-f54a-9e5fd80525d8} {Suffrage:Voter ID:server-677a9cf7-1e63-5934-ed6a-b93205f9e737 Address:677a9cf7-1e63-5934-ed6a-b93205f9e737}]" 2025-01-28T14:47:18.816Z [INFO] server-4ff75592-9024-530c-833a-dc93f2cf222f: entering follower state: follower="Node at 4ff75592-9024-530c-833a-dc93f2cf222f [Follower]" leader-address= leader-id= 2025-01-28T14:47:18.816Z [INFO] server-d9d3529d-9515-358b-f54a-9e5fd80525d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ff75592-9024-530c-833a-dc93f2cf222f Address:4ff75592-9024-530c-833a-dc93f2cf222f} {Suffrage:Voter ID:server-d9d3529d-9515-358b-f54a-9e5fd80525d8 Address:d9d3529d-9515-358b-f54a-9e5fd80525d8} {Suffrage:Voter ID:server-677a9cf7-1e63-5934-ed6a-b93205f9e737 Address:677a9cf7-1e63-5934-ed6a-b93205f9e737}]" 2025-01-28T14:47:18.817Z [INFO] server-d9d3529d-9515-358b-f54a-9e5fd80525d8: entering follower state: follower="Node at d9d3529d-9515-358b-f54a-9e5fd80525d8 [Follower]" leader-address= leader-id= 2025-01-28T14:47:18.818Z [INFO] server-677a9cf7-1e63-5934-ed6a-b93205f9e737: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ff75592-9024-530c-833a-dc93f2cf222f Address:4ff75592-9024-530c-833a-dc93f2cf222f} {Suffrage:Voter ID:server-d9d3529d-9515-358b-f54a-9e5fd80525d8 Address:d9d3529d-9515-358b-f54a-9e5fd80525d8} {Suffrage:Voter ID:server-677a9cf7-1e63-5934-ed6a-b93205f9e737 Address:677a9cf7-1e63-5934-ed6a-b93205f9e737}]" 2025-01-28T14:47:18.818Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:18.819Z [INFO] server-677a9cf7-1e63-5934-ed6a-b93205f9e737: entering follower state: follower="Node at 677a9cf7-1e63-5934-ed6a-b93205f9e737 [Follower]" leader-address= leader-id= 2025-01-28T14:47:19.444Z [WARN] server-677a9cf7-1e63-5934-ed6a-b93205f9e737: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:19.444Z [INFO] server-677a9cf7-1e63-5934-ed6a-b93205f9e737: entering candidate state: node="Node at 677a9cf7-1e63-5934-ed6a-b93205f9e737 [Candidate]" term=2 2025-01-28T14:47:19.446Z [INFO] server-677a9cf7-1e63-5934-ed6a-b93205f9e737: election won: term=2 tally=2 2025-01-28T14:47:19.446Z [INFO] server-677a9cf7-1e63-5934-ed6a-b93205f9e737: entering leader state: leader="Node at 677a9cf7-1e63-5934-ed6a-b93205f9e737 [Leader]" 2025-01-28T14:47:19.446Z [INFO] server-677a9cf7-1e63-5934-ed6a-b93205f9e737: added peer, starting replication: peer=server-4ff75592-9024-530c-833a-dc93f2cf222f 2025-01-28T14:47:19.446Z [INFO] server-677a9cf7-1e63-5934-ed6a-b93205f9e737: added peer, starting replication: peer=server-d9d3529d-9515-358b-f54a-9e5fd80525d8 2025-01-28T14:47:19.447Z [INFO] server-677a9cf7-1e63-5934-ed6a-b93205f9e737: pipelining replication: peer="{Voter server-4ff75592-9024-530c-833a-dc93f2cf222f 4ff75592-9024-530c-833a-dc93f2cf222f}" 2025-01-28T14:47:19.447Z [INFO] server-677a9cf7-1e63-5934-ed6a-b93205f9e737: pipelining replication: peer="{Voter server-d9d3529d-9515-358b-f54a-9e5fd80525d8 d9d3529d-9515-358b-f54a-9e5fd80525d8}" 2025-01-28T14:47:20.452Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:19.44656252 +0000 UTC m=+22.529967878 (1 nodes), 627.413628ms from start of poll, 632.306107ms from cluster start. Timeout at 2025-01-28 14:47:20.451580854 +0000 UTC m=+23.534986879, 1.005019001s after stability 2025-01-28T14:47:20.460Z [INFO] server-677a9cf7-1e63-5934-ed6a-b93205f9e737: starting snapshot up to: index=12 2025-01-28T14:47:20.460Z [INFO] creating new snapshot: path=/tmp/raft2604533988/snapshots/2-12-1738075640460.tmp 2025-01-28T14:47:20.467Z [INFO] server-677a9cf7-1e63-5934-ed6a-b93205f9e737: snapshot complete up to: index=12 2025-01-28T14:47:20.485Z [INFO] server-a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa Address:a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa} {Suffrage:Voter ID:server-f507ec73-74b1-74b3-f0b0-8e1e844cb235 Address:f507ec73-74b1-74b3-f0b0-8e1e844cb235} {Suffrage:Voter ID:server-7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78 Address:7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78}]" 2025-01-28T14:47:20.486Z [INFO] server-a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa: entering follower state: follower="Node at a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa [Follower]" leader-address= leader-id= 2025-01-28T14:47:20.487Z [INFO] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa Address:a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa} {Suffrage:Voter ID:server-f507ec73-74b1-74b3-f0b0-8e1e844cb235 Address:f507ec73-74b1-74b3-f0b0-8e1e844cb235} {Suffrage:Voter ID:server-7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78 Address:7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78}]" 2025-01-28T14:47:20.488Z [INFO] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: entering follower state: follower="Node at f507ec73-74b1-74b3-f0b0-8e1e844cb235 [Follower]" leader-address= leader-id= 2025-01-28T14:47:20.488Z [INFO] server-7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa Address:a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa} {Suffrage:Voter ID:server-f507ec73-74b1-74b3-f0b0-8e1e844cb235 Address:f507ec73-74b1-74b3-f0b0-8e1e844cb235} {Suffrage:Voter ID:server-7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78 Address:7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78}]" 2025-01-28T14:47:20.489Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:20.489Z [INFO] server-7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78: entering follower state: follower="Node at 7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78 [Follower]" leader-address= leader-id= 2025-01-28T14:47:21.104Z [WARN] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:21.104Z [INFO] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: entering candidate state: node="Node at f507ec73-74b1-74b3-f0b0-8e1e844cb235 [Candidate]" term=2 2025-01-28T14:47:21.105Z [INFO] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: election won: term=2 tally=2 2025-01-28T14:47:21.106Z [INFO] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: entering leader state: leader="Node at f507ec73-74b1-74b3-f0b0-8e1e844cb235 [Leader]" 2025-01-28T14:47:21.106Z [INFO] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: added peer, starting replication: peer=server-a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa 2025-01-28T14:47:21.106Z [INFO] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: added peer, starting replication: peer=server-7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78 2025-01-28T14:47:21.108Z [INFO] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: pipelining replication: peer="{Voter server-a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa}" 2025-01-28T14:47:21.108Z [INFO] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: pipelining replication: peer="{Voter server-7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78 7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78}" 2025-01-28T14:47:22.111Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:21.106248795 +0000 UTC m=+24.189655486 (1 nodes), 616.747977ms from start of poll, 621.601121ms from cluster start. Timeout at 2025-01-28 14:47:22.111291463 +0000 UTC m=+25.194697821, 1.005042335s after stability 2025-01-28T14:47:22.114Z [INFO] creating new snapshot: path=/tmp/raft656545943/snapshots/2-13-1738075642114.tmp 2025-01-28T14:47:22.121Z [INFO] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: copied to local snapshot: bytes=71 2025-01-28T14:47:22.123Z [INFO] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: snapshot restore progress: id=2-13-1738075642114 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:22.124Z [INFO] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: restored user snapshot: index=13 2025-01-28T14:47:22.125Z [ERROR] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: failed to get log: index=3 error="log not found" 2025-01-28T14:47:22.125Z [INFO] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: aborting pipeline replication: peer="{Voter server-a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa}" 2025-01-28T14:47:22.125Z [ERROR] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: failed to get log: index=3 error="log not found" 2025-01-28T14:47:22.126Z [INFO] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: aborting pipeline replication: peer="{Voter server-7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78 7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78}" 2025-01-28T14:47:22.133Z [ERROR] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: failed to get log: index=3 error="log not found" 2025-01-28T14:47:22.135Z [ERROR] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: failed to get log: index=3 error="log not found" 2025-01-28T14:47:22.137Z [INFO] creating new snapshot: path=/tmp/raft1878919715/snapshots/2-13-1738075642137.tmp 2025-01-28T14:47:22.140Z [INFO] creating new snapshot: path=/tmp/raft2425637237/snapshots/2-13-1738075642140.tmp 2025-01-28T14:47:22.141Z [INFO] server-a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:22.144Z [INFO] server-7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:22.145Z [INFO] server-a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa: copied to local snapshot: bytes=71 2025-01-28T14:47:22.153Z [INFO] server-7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78: copied to local snapshot: bytes=71 2025-01-28T14:47:22.156Z [INFO] server-a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa: snapshot restore progress: id=2-13-1738075642137 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:22.157Z [INFO] server-a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa: Installed remote snapshot 2025-01-28T14:47:22.163Z [INFO] server-7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78: snapshot restore progress: id=2-13-1738075642140 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:22.163Z [INFO] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: pipelining replication: peer="{Voter server-a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa a0f0a4a8-93b0-0591-dbfc-a0ece30e1aaa}" 2025-01-28T14:47:22.163Z [INFO] server-7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78: Installed remote snapshot 2025-01-28T14:47:22.164Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:22.164Z [INFO] server-f507ec73-74b1-74b3-f0b0-8e1e844cb235: pipelining replication: peer="{Voter server-7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78 7508b3ae-4f6c-1bd2-1d2e-8000a5d2cb78}" 2025-01-28T14:47:23.169Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:22.164148204 +0000 UTC m=+25.247553228 (2 nodes), 9.667µs from start of poll, 1.679498863s from cluster start. Timeout at 2025-01-28 14:47:23.169332876 +0000 UTC m=+26.252738901, 1.005185673s after stability 2025-01-28T14:47:23.186Z [INFO] server-677a9cf7-1e63-5934-ed6a-b93205f9e737: aborting pipeline replication: peer="{Voter server-4ff75592-9024-530c-833a-dc93f2cf222f 4ff75592-9024-530c-833a-dc93f2cf222f}" 2025-01-28T14:47:23.186Z [INFO] server-677a9cf7-1e63-5934-ed6a-b93205f9e737: aborting pipeline replication: peer="{Voter server-d9d3529d-9515-358b-f54a-9e5fd80525d8 d9d3529d-9515-358b-f54a-9e5fd80525d8}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true 2025-01-28T14:47:23.207Z [INFO] server-0fe07390-0968-ea5e-60d1-4c2f880ef77b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0fe07390-0968-ea5e-60d1-4c2f880ef77b Address:0fe07390-0968-ea5e-60d1-4c2f880ef77b} {Suffrage:Voter ID:server-f17939a7-fb26-6f31-6b75-88ca50ed48d1 Address:f17939a7-fb26-6f31-6b75-88ca50ed48d1} {Suffrage:Voter ID:server-dc182dbf-004c-763d-20be-9957a307222e Address:dc182dbf-004c-763d-20be-9957a307222e}]" 2025-01-28T14:47:23.207Z [INFO] server-0fe07390-0968-ea5e-60d1-4c2f880ef77b: entering follower state: follower="Node at 0fe07390-0968-ea5e-60d1-4c2f880ef77b [Follower]" leader-address= leader-id= 2025-01-28T14:47:23.208Z [INFO] server-f17939a7-fb26-6f31-6b75-88ca50ed48d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0fe07390-0968-ea5e-60d1-4c2f880ef77b Address:0fe07390-0968-ea5e-60d1-4c2f880ef77b} {Suffrage:Voter ID:server-f17939a7-fb26-6f31-6b75-88ca50ed48d1 Address:f17939a7-fb26-6f31-6b75-88ca50ed48d1} {Suffrage:Voter ID:server-dc182dbf-004c-763d-20be-9957a307222e Address:dc182dbf-004c-763d-20be-9957a307222e}]" 2025-01-28T14:47:23.209Z [INFO] server-f17939a7-fb26-6f31-6b75-88ca50ed48d1: entering follower state: follower="Node at f17939a7-fb26-6f31-6b75-88ca50ed48d1 [Follower]" leader-address= leader-id= 2025-01-28T14:47:23.210Z [INFO] server-dc182dbf-004c-763d-20be-9957a307222e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0fe07390-0968-ea5e-60d1-4c2f880ef77b Address:0fe07390-0968-ea5e-60d1-4c2f880ef77b} {Suffrage:Voter ID:server-f17939a7-fb26-6f31-6b75-88ca50ed48d1 Address:f17939a7-fb26-6f31-6b75-88ca50ed48d1} {Suffrage:Voter ID:server-dc182dbf-004c-763d-20be-9957a307222e Address:dc182dbf-004c-763d-20be-9957a307222e}]" 2025-01-28T14:47:23.210Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:23.210Z [INFO] server-dc182dbf-004c-763d-20be-9957a307222e: entering follower state: follower="Node at dc182dbf-004c-763d-20be-9957a307222e [Follower]" leader-address= leader-id= 2025-01-28T14:47:23.714Z [WARN] server-f17939a7-fb26-6f31-6b75-88ca50ed48d1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:23.714Z [INFO] server-f17939a7-fb26-6f31-6b75-88ca50ed48d1: entering candidate state: node="Node at f17939a7-fb26-6f31-6b75-88ca50ed48d1 [Candidate]" term=2 2025-01-28T14:47:23.715Z [INFO] server-f17939a7-fb26-6f31-6b75-88ca50ed48d1: election won: term=2 tally=2 2025-01-28T14:47:23.716Z [INFO] server-f17939a7-fb26-6f31-6b75-88ca50ed48d1: entering leader state: leader="Node at f17939a7-fb26-6f31-6b75-88ca50ed48d1 [Leader]" 2025-01-28T14:47:23.716Z [INFO] server-f17939a7-fb26-6f31-6b75-88ca50ed48d1: added peer, starting replication: peer=server-0fe07390-0968-ea5e-60d1-4c2f880ef77b 2025-01-28T14:47:23.716Z [INFO] server-f17939a7-fb26-6f31-6b75-88ca50ed48d1: added peer, starting replication: peer=server-dc182dbf-004c-763d-20be-9957a307222e 2025-01-28T14:47:23.717Z [INFO] server-f17939a7-fb26-6f31-6b75-88ca50ed48d1: pipelining replication: peer="{Voter server-0fe07390-0968-ea5e-60d1-4c2f880ef77b 0fe07390-0968-ea5e-60d1-4c2f880ef77b}" 2025-01-28T14:47:23.718Z [INFO] server-f17939a7-fb26-6f31-6b75-88ca50ed48d1: pipelining replication: peer="{Voter server-dc182dbf-004c-763d-20be-9957a307222e dc182dbf-004c-763d-20be-9957a307222e}" 2025-01-28T14:47:24.722Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:23.716378956 +0000 UTC m=+26.799785314 (1 nodes), 505.56801ms from start of poll, 510.853834ms from cluster start. Timeout at 2025-01-28 14:47:24.721419957 +0000 UTC m=+27.804825982, 1.005040668s after stability 2025-01-28T14:47:24.733Z [INFO] server-f17939a7-fb26-6f31-6b75-88ca50ed48d1: starting snapshot up to: index=12 2025-01-28T14:47:24.733Z [INFO] creating new snapshot: path=/tmp/raft2364932551/snapshots/2-12-1738075644733.tmp 2025-01-28T14:47:24.740Z [INFO] server-f17939a7-fb26-6f31-6b75-88ca50ed48d1: snapshot complete up to: index=12 2025-01-28T14:47:24.771Z [INFO] server-6bae03fd-423d-baff-c73a-4c44b86b68e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6bae03fd-423d-baff-c73a-4c44b86b68e5 Address:6bae03fd-423d-baff-c73a-4c44b86b68e5} {Suffrage:Voter ID:server-018cc126-6d81-d83e-7e22-6c832322804d Address:018cc126-6d81-d83e-7e22-6c832322804d} {Suffrage:Voter ID:server-f86d04aa-4db5-551f-bd4e-29adf720afd7 Address:f86d04aa-4db5-551f-bd4e-29adf720afd7}]" 2025-01-28T14:47:24.771Z [INFO] server-6bae03fd-423d-baff-c73a-4c44b86b68e5: entering follower state: follower="Node at 6bae03fd-423d-baff-c73a-4c44b86b68e5 [Follower]" leader-address= leader-id= 2025-01-28T14:47:24.773Z [INFO] server-018cc126-6d81-d83e-7e22-6c832322804d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6bae03fd-423d-baff-c73a-4c44b86b68e5 Address:6bae03fd-423d-baff-c73a-4c44b86b68e5} {Suffrage:Voter ID:server-018cc126-6d81-d83e-7e22-6c832322804d Address:018cc126-6d81-d83e-7e22-6c832322804d} {Suffrage:Voter ID:server-f86d04aa-4db5-551f-bd4e-29adf720afd7 Address:f86d04aa-4db5-551f-bd4e-29adf720afd7}]" 2025-01-28T14:47:24.773Z [INFO] server-018cc126-6d81-d83e-7e22-6c832322804d: entering follower state: follower="Node at 018cc126-6d81-d83e-7e22-6c832322804d [Follower]" leader-address= leader-id= 2025-01-28T14:47:24.774Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6bae03fd-423d-baff-c73a-4c44b86b68e5 Address:6bae03fd-423d-baff-c73a-4c44b86b68e5} {Suffrage:Voter ID:server-018cc126-6d81-d83e-7e22-6c832322804d Address:018cc126-6d81-d83e-7e22-6c832322804d} {Suffrage:Voter ID:server-f86d04aa-4db5-551f-bd4e-29adf720afd7 Address:f86d04aa-4db5-551f-bd4e-29adf720afd7}]" 2025-01-28T14:47:24.774Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:24.774Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: entering follower state: follower="Node at f86d04aa-4db5-551f-bd4e-29adf720afd7 [Follower]" leader-address= leader-id= 2025-01-28T14:47:25.354Z [WARN] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:25.355Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: entering candidate state: node="Node at f86d04aa-4db5-551f-bd4e-29adf720afd7 [Candidate]" term=2 2025-01-28T14:47:25.356Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: election won: term=2 tally=2 2025-01-28T14:47:25.356Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: entering leader state: leader="Node at f86d04aa-4db5-551f-bd4e-29adf720afd7 [Leader]" 2025-01-28T14:47:25.356Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: added peer, starting replication: peer=server-6bae03fd-423d-baff-c73a-4c44b86b68e5 2025-01-28T14:47:25.356Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: added peer, starting replication: peer=server-018cc126-6d81-d83e-7e22-6c832322804d 2025-01-28T14:47:25.358Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: pipelining replication: peer="{Voter server-018cc126-6d81-d83e-7e22-6c832322804d 018cc126-6d81-d83e-7e22-6c832322804d}" 2025-01-28T14:47:25.359Z [WARN] server-6bae03fd-423d-baff-c73a-4c44b86b68e5: rejecting vote request since our last term is greater: candidate=f86d04aa-4db5-551f-bd4e-29adf720afd7 last-term=2 last-candidate-term=1 2025-01-28T14:47:25.360Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: pipelining replication: peer="{Voter server-6bae03fd-423d-baff-c73a-4c44b86b68e5 6bae03fd-423d-baff-c73a-4c44b86b68e5}" 2025-01-28T14:47:26.365Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:25.360013422 +0000 UTC m=+28.443419780 (1 nodes), 585.074371ms from start of poll, 590.229191ms from cluster start. Timeout at 2025-01-28 14:47:26.365111759 +0000 UTC m=+29.448518450, 1.00509867s after stability 2025-01-28T14:47:26.368Z [INFO] creating new snapshot: path=/tmp/raft1136064849/snapshots/2-13-1738075646368.tmp 2025-01-28T14:47:26.375Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: copied to local snapshot: bytes=71 2025-01-28T14:47:26.376Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: snapshot restore progress: id=2-13-1738075646368 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:26.377Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: removing all old logs from log store 2025-01-28T14:47:26.377Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: compacting logs: from=1 to=2 2025-01-28T14:47:26.377Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: restored user snapshot: index=13 2025-01-28T14:47:26.377Z [ERROR] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: failed to get log: index=2 error="log not found" 2025-01-28T14:47:26.377Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: aborting pipeline replication: peer="{Voter server-6bae03fd-423d-baff-c73a-4c44b86b68e5 6bae03fd-423d-baff-c73a-4c44b86b68e5}" 2025-01-28T14:47:26.377Z [ERROR] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: failed to get log: index=2 error="log not found" 2025-01-28T14:47:26.378Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: aborting pipeline replication: peer="{Voter server-018cc126-6d81-d83e-7e22-6c832322804d 018cc126-6d81-d83e-7e22-6c832322804d}" 2025-01-28T14:47:26.383Z [ERROR] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: failed to get log: index=2 error="log not found" 2025-01-28T14:47:26.385Z [ERROR] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: failed to get log: index=2 error="log not found" 2025-01-28T14:47:26.388Z [INFO] creating new snapshot: path=/tmp/raft3671420036/snapshots/2-13-1738075646388.tmp 2025-01-28T14:47:26.390Z [INFO] creating new snapshot: path=/tmp/raft2647387347/snapshots/2-13-1738075646390.tmp 2025-01-28T14:47:26.392Z [INFO] server-018cc126-6d81-d83e-7e22-6c832322804d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:26.394Z [INFO] server-6bae03fd-423d-baff-c73a-4c44b86b68e5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:26.396Z [INFO] server-018cc126-6d81-d83e-7e22-6c832322804d: copied to local snapshot: bytes=71 2025-01-28T14:47:26.399Z [INFO] server-6bae03fd-423d-baff-c73a-4c44b86b68e5: copied to local snapshot: bytes=71 2025-01-28T14:47:26.400Z [INFO] server-018cc126-6d81-d83e-7e22-6c832322804d: snapshot restore progress: id=2-13-1738075646388 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:26.400Z [INFO] server-018cc126-6d81-d83e-7e22-6c832322804d: removing all old logs from log store 2025-01-28T14:47:26.400Z [INFO] server-018cc126-6d81-d83e-7e22-6c832322804d: compacting logs: from=1 to=2 2025-01-28T14:47:26.401Z [INFO] server-018cc126-6d81-d83e-7e22-6c832322804d: Installed remote snapshot 2025-01-28T14:47:26.408Z [WARN] server-018cc126-6d81-d83e-7e22-6c832322804d: skipping application of old log: index=13 2025-01-28T14:47:26.408Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: pipelining replication: peer="{Voter server-018cc126-6d81-d83e-7e22-6c832322804d 018cc126-6d81-d83e-7e22-6c832322804d}" 2025-01-28T14:47:26.408Z [INFO] server-6bae03fd-423d-baff-c73a-4c44b86b68e5: snapshot restore progress: id=2-13-1738075646390 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:26.409Z [INFO] server-6bae03fd-423d-baff-c73a-4c44b86b68e5: removing all old logs from log store 2025-01-28T14:47:26.409Z [INFO] server-6bae03fd-423d-baff-c73a-4c44b86b68e5: compacting logs: from=1 to=2 2025-01-28T14:47:26.409Z [INFO] server-6bae03fd-423d-baff-c73a-4c44b86b68e5: Installed remote snapshot 2025-01-28T14:47:26.410Z [WARN] server-6bae03fd-423d-baff-c73a-4c44b86b68e5: skipping application of old log: index=13 2025-01-28T14:47:26.412Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:26.412Z [INFO] server-f86d04aa-4db5-551f-bd4e-29adf720afd7: pipelining replication: peer="{Voter server-6bae03fd-423d-baff-c73a-4c44b86b68e5 6bae03fd-423d-baff-c73a-4c44b86b68e5}" 2025-01-28T14:47:27.419Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:26.412929017 +0000 UTC m=+29.496334041 (2 nodes), 8.667µs from start of poll, 1.643143452s from cluster start. Timeout at 2025-01-28 14:47:27.418209359 +0000 UTC m=+30.501615716, 1.005281675s after stability 2025-01-28T14:47:27.440Z [INFO] server-f17939a7-fb26-6f31-6b75-88ca50ed48d1: aborting pipeline replication: peer="{Voter server-dc182dbf-004c-763d-20be-9957a307222e dc182dbf-004c-763d-20be-9957a307222e}" 2025-01-28T14:47:27.440Z [INFO] server-f17939a7-fb26-6f31-6b75-88ca50ed48d1: aborting pipeline replication: peer="{Voter server-0fe07390-0968-ea5e-60d1-4c2f880ef77b 0fe07390-0968-ea5e-60d1-4c2f880ef77b}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false 2025-01-28T14:47:27.466Z [INFO] server-c423691e-e148-ddfa-255b-fd8beaeafc7b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c423691e-e148-ddfa-255b-fd8beaeafc7b Address:c423691e-e148-ddfa-255b-fd8beaeafc7b} {Suffrage:Voter ID:server-9bf51f08-6efa-d61b-a29b-84fbc7861362 Address:9bf51f08-6efa-d61b-a29b-84fbc7861362} {Suffrage:Voter ID:server-32d4a8ac-aa2e-4064-2d52-965243d61aac Address:32d4a8ac-aa2e-4064-2d52-965243d61aac}]" 2025-01-28T14:47:27.466Z [INFO] server-c423691e-e148-ddfa-255b-fd8beaeafc7b: entering follower state: follower="Node at c423691e-e148-ddfa-255b-fd8beaeafc7b [Follower]" leader-address= leader-id= 2025-01-28T14:47:27.467Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c423691e-e148-ddfa-255b-fd8beaeafc7b Address:c423691e-e148-ddfa-255b-fd8beaeafc7b} {Suffrage:Voter ID:server-9bf51f08-6efa-d61b-a29b-84fbc7861362 Address:9bf51f08-6efa-d61b-a29b-84fbc7861362} {Suffrage:Voter ID:server-32d4a8ac-aa2e-4064-2d52-965243d61aac Address:32d4a8ac-aa2e-4064-2d52-965243d61aac}]" 2025-01-28T14:47:27.467Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: entering follower state: follower="Node at 9bf51f08-6efa-d61b-a29b-84fbc7861362 [Follower]" leader-address= leader-id= 2025-01-28T14:47:27.467Z [INFO] server-32d4a8ac-aa2e-4064-2d52-965243d61aac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c423691e-e148-ddfa-255b-fd8beaeafc7b Address:c423691e-e148-ddfa-255b-fd8beaeafc7b} {Suffrage:Voter ID:server-9bf51f08-6efa-d61b-a29b-84fbc7861362 Address:9bf51f08-6efa-d61b-a29b-84fbc7861362} {Suffrage:Voter ID:server-32d4a8ac-aa2e-4064-2d52-965243d61aac Address:32d4a8ac-aa2e-4064-2d52-965243d61aac}]" 2025-01-28T14:47:27.468Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:27.468Z [INFO] server-32d4a8ac-aa2e-4064-2d52-965243d61aac: entering follower state: follower="Node at 32d4a8ac-aa2e-4064-2d52-965243d61aac [Follower]" leader-address= leader-id= 2025-01-28T14:47:28.188Z [WARN] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:28.188Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: entering candidate state: node="Node at 9bf51f08-6efa-d61b-a29b-84fbc7861362 [Candidate]" term=2 2025-01-28T14:47:28.189Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: election won: term=2 tally=2 2025-01-28T14:47:28.189Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: entering leader state: leader="Node at 9bf51f08-6efa-d61b-a29b-84fbc7861362 [Leader]" 2025-01-28T14:47:28.190Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: added peer, starting replication: peer=server-c423691e-e148-ddfa-255b-fd8beaeafc7b 2025-01-28T14:47:28.190Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: added peer, starting replication: peer=server-32d4a8ac-aa2e-4064-2d52-965243d61aac 2025-01-28T14:47:28.191Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: pipelining replication: peer="{Voter server-c423691e-e148-ddfa-255b-fd8beaeafc7b c423691e-e148-ddfa-255b-fd8beaeafc7b}" 2025-01-28T14:47:28.192Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: pipelining replication: peer="{Voter server-32d4a8ac-aa2e-4064-2d52-965243d61aac 32d4a8ac-aa2e-4064-2d52-965243d61aac}" 2025-01-28T14:47:29.195Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:28.190134116 +0000 UTC m=+31.273540807 (1 nodes), 721.842765ms from start of poll, 726.028223ms from cluster start. Timeout at 2025-01-28 14:47:29.19518945 +0000 UTC m=+32.278595475, 1.005054668s after stability 2025-01-28T14:47:29.206Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: starting snapshot up to: index=12 2025-01-28T14:47:29.206Z [INFO] creating new snapshot: path=/tmp/raft1870196071/snapshots/2-12-1738075649206.tmp 2025-01-28T14:47:29.213Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: snapshot complete up to: index=12 2025-01-28T14:47:29.220Z [INFO] creating new snapshot: path=/tmp/raft1870196071/snapshots/2-23-1738075649220.tmp 2025-01-28T14:47:29.225Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: copied to local snapshot: bytes=71 2025-01-28T14:47:29.227Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: snapshot restore progress: id=2-23-1738075649220 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:29.228Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: restored user snapshot: index=23 2025-01-28T14:47:29.228Z [ERROR] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: failed to get log: index=23 error="log not found" 2025-01-28T14:47:29.228Z [ERROR] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: failed to get log: index=23 error="log not found" 2025-01-28T14:47:29.228Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: aborting pipeline replication: peer="{Voter server-c423691e-e148-ddfa-255b-fd8beaeafc7b c423691e-e148-ddfa-255b-fd8beaeafc7b}" 2025-01-28T14:47:29.229Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: aborting pipeline replication: peer="{Voter server-32d4a8ac-aa2e-4064-2d52-965243d61aac 32d4a8ac-aa2e-4064-2d52-965243d61aac}" 2025-01-28T14:47:29.237Z [ERROR] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: failed to get log: index=23 error="log not found" 2025-01-28T14:47:29.245Z [ERROR] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: failed to get log: index=23 error="log not found" 2025-01-28T14:47:29.245Z [INFO] creating new snapshot: path=/tmp/raft3072491704/snapshots/2-23-1738075649245.tmp 2025-01-28T14:47:29.250Z [INFO] server-c423691e-e148-ddfa-255b-fd8beaeafc7b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:29.253Z [INFO] creating new snapshot: path=/tmp/raft1651862045/snapshots/2-23-1738075649253.tmp 2025-01-28T14:47:29.255Z [INFO] server-c423691e-e148-ddfa-255b-fd8beaeafc7b: copied to local snapshot: bytes=71 2025-01-28T14:47:29.256Z [INFO] server-32d4a8ac-aa2e-4064-2d52-965243d61aac: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:29.257Z [INFO] server-c423691e-e148-ddfa-255b-fd8beaeafc7b: snapshot restore progress: id=2-23-1738075649245 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:29.257Z [INFO] server-c423691e-e148-ddfa-255b-fd8beaeafc7b: Installed remote snapshot 2025-01-28T14:47:29.258Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: pipelining replication: peer="{Voter server-c423691e-e148-ddfa-255b-fd8beaeafc7b c423691e-e148-ddfa-255b-fd8beaeafc7b}" 2025-01-28T14:47:29.259Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:29.260Z [INFO] server-32d4a8ac-aa2e-4064-2d52-965243d61aac: copied to local snapshot: bytes=71 2025-01-28T14:47:29.261Z [INFO] server-32d4a8ac-aa2e-4064-2d52-965243d61aac: snapshot restore progress: id=2-23-1738075649253 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:29.262Z [INFO] server-32d4a8ac-aa2e-4064-2d52-965243d61aac: Installed remote snapshot 2025-01-28T14:47:29.263Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: pipelining replication: peer="{Voter server-32d4a8ac-aa2e-4064-2d52-965243d61aac 32d4a8ac-aa2e-4064-2d52-965243d61aac}" 2025-01-28T14:47:30.265Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:29.259314193 +0000 UTC m=+32.342719218 (2 nodes), 9.667µs from start of poll, 1.795206634s from cluster start. Timeout at 2025-01-28 14:47:30.264519533 +0000 UTC m=+33.347925558, 1.00520634s after stability 2025-01-28T14:47:30.285Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: aborting pipeline replication: peer="{Voter server-32d4a8ac-aa2e-4064-2d52-965243d61aac 32d4a8ac-aa2e-4064-2d52-965243d61aac}" 2025-01-28T14:47:30.285Z [INFO] server-9bf51f08-6efa-d61b-a29b-84fbc7861362: aborting pipeline replication: peer="{Voter server-c423691e-e148-ddfa-255b-fd8beaeafc7b c423691e-e148-ddfa-255b-fd8beaeafc7b}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false 2025-01-28T14:47:30.310Z [INFO] server-38c5302d-dc2c-f38a-f9a2-1c455ce74c38: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38c5302d-dc2c-f38a-f9a2-1c455ce74c38 Address:38c5302d-dc2c-f38a-f9a2-1c455ce74c38} {Suffrage:Voter ID:server-c1e37476-0327-fcca-2379-37c54a428795 Address:c1e37476-0327-fcca-2379-37c54a428795} {Suffrage:Voter ID:server-d584fb87-a943-9514-dc9f-51329f22d5fc Address:d584fb87-a943-9514-dc9f-51329f22d5fc}]" 2025-01-28T14:47:30.311Z [INFO] server-38c5302d-dc2c-f38a-f9a2-1c455ce74c38: entering follower state: follower="Node at 38c5302d-dc2c-f38a-f9a2-1c455ce74c38 [Follower]" leader-address= leader-id= 2025-01-28T14:47:30.312Z [INFO] server-c1e37476-0327-fcca-2379-37c54a428795: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38c5302d-dc2c-f38a-f9a2-1c455ce74c38 Address:38c5302d-dc2c-f38a-f9a2-1c455ce74c38} {Suffrage:Voter ID:server-c1e37476-0327-fcca-2379-37c54a428795 Address:c1e37476-0327-fcca-2379-37c54a428795} {Suffrage:Voter ID:server-d584fb87-a943-9514-dc9f-51329f22d5fc Address:d584fb87-a943-9514-dc9f-51329f22d5fc}]" 2025-01-28T14:47:30.312Z [INFO] server-c1e37476-0327-fcca-2379-37c54a428795: entering follower state: follower="Node at c1e37476-0327-fcca-2379-37c54a428795 [Follower]" leader-address= leader-id= 2025-01-28T14:47:30.313Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38c5302d-dc2c-f38a-f9a2-1c455ce74c38 Address:38c5302d-dc2c-f38a-f9a2-1c455ce74c38} {Suffrage:Voter ID:server-c1e37476-0327-fcca-2379-37c54a428795 Address:c1e37476-0327-fcca-2379-37c54a428795} {Suffrage:Voter ID:server-d584fb87-a943-9514-dc9f-51329f22d5fc Address:d584fb87-a943-9514-dc9f-51329f22d5fc}]" 2025-01-28T14:47:30.314Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:30.314Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: entering follower state: follower="Node at d584fb87-a943-9514-dc9f-51329f22d5fc [Follower]" leader-address= leader-id= 2025-01-28T14:47:30.898Z [WARN] server-d584fb87-a943-9514-dc9f-51329f22d5fc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:30.898Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: entering candidate state: node="Node at d584fb87-a943-9514-dc9f-51329f22d5fc [Candidate]" term=2 2025-01-28T14:47:30.899Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: election won: term=2 tally=2 2025-01-28T14:47:30.900Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: entering leader state: leader="Node at d584fb87-a943-9514-dc9f-51329f22d5fc [Leader]" 2025-01-28T14:47:30.900Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: added peer, starting replication: peer=server-38c5302d-dc2c-f38a-f9a2-1c455ce74c38 2025-01-28T14:47:30.900Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: added peer, starting replication: peer=server-c1e37476-0327-fcca-2379-37c54a428795 2025-01-28T14:47:30.902Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: pipelining replication: peer="{Voter server-c1e37476-0327-fcca-2379-37c54a428795 c1e37476-0327-fcca-2379-37c54a428795}" 2025-01-28T14:47:30.902Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: pipelining replication: peer="{Voter server-38c5302d-dc2c-f38a-f9a2-1c455ce74c38 38c5302d-dc2c-f38a-f9a2-1c455ce74c38}" 2025-01-28T14:47:31.906Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:30.900581924 +0000 UTC m=+33.983988281 (1 nodes), 586.138402ms from start of poll, 590.877877ms from cluster start. Timeout at 2025-01-28 14:47:31.905618591 +0000 UTC m=+34.989024949, 1.005036668s after stability 2025-01-28T14:47:31.914Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: starting snapshot up to: index=12 2025-01-28T14:47:31.914Z [INFO] creating new snapshot: path=/tmp/raft1285573842/snapshots/2-12-1738075651914.tmp 2025-01-28T14:47:31.922Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: snapshot complete up to: index=12 2025-01-28T14:47:31.937Z [INFO] creating new snapshot: path=/tmp/raft1285573842/snapshots/2-23-1738075651936.tmp 2025-01-28T14:47:31.946Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: copied to local snapshot: bytes=71 2025-01-28T14:47:31.948Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: snapshot restore progress: id=2-23-1738075651936 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:31.948Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: removing all old logs from log store 2025-01-28T14:47:31.948Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: compacting logs: from=1 to=22 2025-01-28T14:47:31.948Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: restored user snapshot: index=23 2025-01-28T14:47:31.949Z [ERROR] server-d584fb87-a943-9514-dc9f-51329f22d5fc: failed to get log: index=22 error="log not found" 2025-01-28T14:47:31.949Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: aborting pipeline replication: peer="{Voter server-38c5302d-dc2c-f38a-f9a2-1c455ce74c38 38c5302d-dc2c-f38a-f9a2-1c455ce74c38}" 2025-01-28T14:47:31.949Z [ERROR] server-d584fb87-a943-9514-dc9f-51329f22d5fc: failed to get log: index=22 error="log not found" 2025-01-28T14:47:31.949Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: aborting pipeline replication: peer="{Voter server-c1e37476-0327-fcca-2379-37c54a428795 c1e37476-0327-fcca-2379-37c54a428795}" 2025-01-28T14:47:31.957Z [ERROR] server-d584fb87-a943-9514-dc9f-51329f22d5fc: failed to get log: index=22 error="log not found" 2025-01-28T14:47:31.958Z [ERROR] server-d584fb87-a943-9514-dc9f-51329f22d5fc: failed to get log: index=22 error="log not found" 2025-01-28T14:47:31.963Z [INFO] creating new snapshot: path=/tmp/raft946878823/snapshots/2-23-1738075651963.tmp 2025-01-28T14:47:31.963Z [INFO] creating new snapshot: path=/tmp/raft3328387268/snapshots/2-23-1738075651963.tmp 2025-01-28T14:47:31.965Z [INFO] server-c1e37476-0327-fcca-2379-37c54a428795: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:31.965Z [INFO] server-38c5302d-dc2c-f38a-f9a2-1c455ce74c38: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:31.968Z [INFO] server-c1e37476-0327-fcca-2379-37c54a428795: copied to local snapshot: bytes=71 2025-01-28T14:47:31.968Z [INFO] server-38c5302d-dc2c-f38a-f9a2-1c455ce74c38: copied to local snapshot: bytes=71 2025-01-28T14:47:31.969Z [INFO] server-38c5302d-dc2c-f38a-f9a2-1c455ce74c38: snapshot restore progress: id=2-23-1738075651963 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:31.969Z [INFO] server-c1e37476-0327-fcca-2379-37c54a428795: snapshot restore progress: id=2-23-1738075651963 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:31.969Z [INFO] server-38c5302d-dc2c-f38a-f9a2-1c455ce74c38: removing all old logs from log store 2025-01-28T14:47:31.969Z [INFO] server-38c5302d-dc2c-f38a-f9a2-1c455ce74c38: compacting logs: from=1 to=22 2025-01-28T14:47:31.969Z [INFO] server-c1e37476-0327-fcca-2379-37c54a428795: removing all old logs from log store 2025-01-28T14:47:31.969Z [INFO] server-c1e37476-0327-fcca-2379-37c54a428795: compacting logs: from=1 to=22 2025-01-28T14:47:31.970Z [INFO] server-38c5302d-dc2c-f38a-f9a2-1c455ce74c38: Installed remote snapshot 2025-01-28T14:47:31.970Z [INFO] server-c1e37476-0327-fcca-2379-37c54a428795: Installed remote snapshot 2025-01-28T14:47:31.970Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: pipelining replication: peer="{Voter server-c1e37476-0327-fcca-2379-37c54a428795 c1e37476-0327-fcca-2379-37c54a428795}" 2025-01-28T14:47:31.971Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: pipelining replication: peer="{Voter server-38c5302d-dc2c-f38a-f9a2-1c455ce74c38 38c5302d-dc2c-f38a-f9a2-1c455ce74c38}" 2025-01-28T14:47:31.971Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:32.977Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:31.971306381 +0000 UTC m=+35.054711405 (2 nodes), 8µs from start of poll, 1.661601001s from cluster start. Timeout at 2025-01-28 14:47:32.976529388 +0000 UTC m=+36.059935412, 1.005224007s after stability 2025-01-28T14:47:32.990Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: aborting pipeline replication: peer="{Voter server-c1e37476-0327-fcca-2379-37c54a428795 c1e37476-0327-fcca-2379-37c54a428795}" 2025-01-28T14:47:32.990Z [INFO] server-d584fb87-a943-9514-dc9f-51329f22d5fc: aborting pipeline replication: peer="{Voter server-38c5302d-dc2c-f38a-f9a2-1c455ce74c38 38c5302d-dc2c-f38a-f9a2-1c455ce74c38}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true 2025-01-28T14:47:33.016Z [INFO] server-cad7fdee-dd1b-b5b3-8983-5972aea7e140: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cad7fdee-dd1b-b5b3-8983-5972aea7e140 Address:cad7fdee-dd1b-b5b3-8983-5972aea7e140} {Suffrage:Voter ID:server-bb582f92-2308-3fa9-7ed0-4487ae2e1e2d Address:bb582f92-2308-3fa9-7ed0-4487ae2e1e2d} {Suffrage:Voter ID:server-e76b4dbc-59c9-5243-69d8-ec7d906c1c31 Address:e76b4dbc-59c9-5243-69d8-ec7d906c1c31}]" 2025-01-28T14:47:33.018Z [INFO] server-bb582f92-2308-3fa9-7ed0-4487ae2e1e2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cad7fdee-dd1b-b5b3-8983-5972aea7e140 Address:cad7fdee-dd1b-b5b3-8983-5972aea7e140} {Suffrage:Voter ID:server-bb582f92-2308-3fa9-7ed0-4487ae2e1e2d Address:bb582f92-2308-3fa9-7ed0-4487ae2e1e2d} {Suffrage:Voter ID:server-e76b4dbc-59c9-5243-69d8-ec7d906c1c31 Address:e76b4dbc-59c9-5243-69d8-ec7d906c1c31}]" 2025-01-28T14:47:33.018Z [INFO] server-cad7fdee-dd1b-b5b3-8983-5972aea7e140: entering follower state: follower="Node at cad7fdee-dd1b-b5b3-8983-5972aea7e140 [Follower]" leader-address= leader-id= 2025-01-28T14:47:33.018Z [INFO] server-bb582f92-2308-3fa9-7ed0-4487ae2e1e2d: entering follower state: follower="Node at bb582f92-2308-3fa9-7ed0-4487ae2e1e2d [Follower]" leader-address= leader-id= 2025-01-28T14:47:33.019Z [INFO] server-e76b4dbc-59c9-5243-69d8-ec7d906c1c31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cad7fdee-dd1b-b5b3-8983-5972aea7e140 Address:cad7fdee-dd1b-b5b3-8983-5972aea7e140} {Suffrage:Voter ID:server-bb582f92-2308-3fa9-7ed0-4487ae2e1e2d Address:bb582f92-2308-3fa9-7ed0-4487ae2e1e2d} {Suffrage:Voter ID:server-e76b4dbc-59c9-5243-69d8-ec7d906c1c31 Address:e76b4dbc-59c9-5243-69d8-ec7d906c1c31}]" 2025-01-28T14:47:33.020Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:33.020Z [INFO] server-e76b4dbc-59c9-5243-69d8-ec7d906c1c31: entering follower state: follower="Node at e76b4dbc-59c9-5243-69d8-ec7d906c1c31 [Follower]" leader-address= leader-id= 2025-01-28T14:47:33.560Z [WARN] server-e76b4dbc-59c9-5243-69d8-ec7d906c1c31: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:33.560Z [INFO] server-e76b4dbc-59c9-5243-69d8-ec7d906c1c31: entering candidate state: node="Node at e76b4dbc-59c9-5243-69d8-ec7d906c1c31 [Candidate]" term=2 2025-01-28T14:47:33.562Z [INFO] server-e76b4dbc-59c9-5243-69d8-ec7d906c1c31: election won: term=2 tally=2 2025-01-28T14:47:33.562Z [INFO] server-e76b4dbc-59c9-5243-69d8-ec7d906c1c31: entering leader state: leader="Node at e76b4dbc-59c9-5243-69d8-ec7d906c1c31 [Leader]" 2025-01-28T14:47:33.562Z [INFO] server-e76b4dbc-59c9-5243-69d8-ec7d906c1c31: added peer, starting replication: peer=server-cad7fdee-dd1b-b5b3-8983-5972aea7e140 2025-01-28T14:47:33.562Z [INFO] server-e76b4dbc-59c9-5243-69d8-ec7d906c1c31: added peer, starting replication: peer=server-bb582f92-2308-3fa9-7ed0-4487ae2e1e2d 2025-01-28T14:47:33.563Z [INFO] server-e76b4dbc-59c9-5243-69d8-ec7d906c1c31: pipelining replication: peer="{Voter server-cad7fdee-dd1b-b5b3-8983-5972aea7e140 cad7fdee-dd1b-b5b3-8983-5972aea7e140}" 2025-01-28T14:47:33.564Z [INFO] server-e76b4dbc-59c9-5243-69d8-ec7d906c1c31: pipelining replication: peer="{Voter server-bb582f92-2308-3fa9-7ed0-4487ae2e1e2d bb582f92-2308-3fa9-7ed0-4487ae2e1e2d}" 2025-01-28T14:47:34.567Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:33.562538292 +0000 UTC m=+36.645944984 (1 nodes), 542.121097ms from start of poll, 546.985908ms from cluster start. Timeout at 2025-01-28 14:47:34.567560959 +0000 UTC m=+37.650966984, 1.005022s after stability 2025-01-28T14:47:34.576Z [INFO] server-e76b4dbc-59c9-5243-69d8-ec7d906c1c31: starting snapshot up to: index=12 2025-01-28T14:47:34.576Z [INFO] creating new snapshot: path=/tmp/raft4240931850/snapshots/2-12-1738075654576.tmp 2025-01-28T14:47:34.583Z [INFO] server-e76b4dbc-59c9-5243-69d8-ec7d906c1c31: snapshot complete up to: index=12 2025-01-28T14:47:34.594Z [INFO] server-0c3d5030-e31e-a65f-ab98-18c6e1f535bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c3d5030-e31e-a65f-ab98-18c6e1f535bb Address:0c3d5030-e31e-a65f-ab98-18c6e1f535bb} {Suffrage:Voter ID:server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867 Address:d9bdee70-33be-8ce5-cc95-b4ae2f9be867} {Suffrage:Voter ID:server-dbda4c06-c499-520d-52bc-2f751a7c30b4 Address:dbda4c06-c499-520d-52bc-2f751a7c30b4}]" 2025-01-28T14:47:34.595Z [INFO] server-0c3d5030-e31e-a65f-ab98-18c6e1f535bb: entering follower state: follower="Node at 0c3d5030-e31e-a65f-ab98-18c6e1f535bb [Follower]" leader-address= leader-id= 2025-01-28T14:47:34.596Z [INFO] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c3d5030-e31e-a65f-ab98-18c6e1f535bb Address:0c3d5030-e31e-a65f-ab98-18c6e1f535bb} {Suffrage:Voter ID:server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867 Address:d9bdee70-33be-8ce5-cc95-b4ae2f9be867} {Suffrage:Voter ID:server-dbda4c06-c499-520d-52bc-2f751a7c30b4 Address:dbda4c06-c499-520d-52bc-2f751a7c30b4}]" 2025-01-28T14:47:34.596Z [INFO] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: entering follower state: follower="Node at d9bdee70-33be-8ce5-cc95-b4ae2f9be867 [Follower]" leader-address= leader-id= 2025-01-28T14:47:34.597Z [INFO] server-dbda4c06-c499-520d-52bc-2f751a7c30b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c3d5030-e31e-a65f-ab98-18c6e1f535bb Address:0c3d5030-e31e-a65f-ab98-18c6e1f535bb} {Suffrage:Voter ID:server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867 Address:d9bdee70-33be-8ce5-cc95-b4ae2f9be867} {Suffrage:Voter ID:server-dbda4c06-c499-520d-52bc-2f751a7c30b4 Address:dbda4c06-c499-520d-52bc-2f751a7c30b4}]" 2025-01-28T14:47:34.597Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:34.598Z [INFO] server-dbda4c06-c499-520d-52bc-2f751a7c30b4: entering follower state: follower="Node at dbda4c06-c499-520d-52bc-2f751a7c30b4 [Follower]" leader-address= leader-id= 2025-01-28T14:47:35.165Z [WARN] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:35.165Z [INFO] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: entering candidate state: node="Node at d9bdee70-33be-8ce5-cc95-b4ae2f9be867 [Candidate]" term=2 2025-01-28T14:47:35.166Z [INFO] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: election won: term=2 tally=2 2025-01-28T14:47:35.167Z [INFO] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: entering leader state: leader="Node at d9bdee70-33be-8ce5-cc95-b4ae2f9be867 [Leader]" 2025-01-28T14:47:35.167Z [INFO] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: added peer, starting replication: peer=server-0c3d5030-e31e-a65f-ab98-18c6e1f535bb 2025-01-28T14:47:35.167Z [INFO] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: added peer, starting replication: peer=server-dbda4c06-c499-520d-52bc-2f751a7c30b4 2025-01-28T14:47:35.169Z [INFO] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: pipelining replication: peer="{Voter server-0c3d5030-e31e-a65f-ab98-18c6e1f535bb 0c3d5030-e31e-a65f-ab98-18c6e1f535bb}" 2025-01-28T14:47:35.169Z [INFO] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: pipelining replication: peer="{Voter server-dbda4c06-c499-520d-52bc-2f751a7c30b4 dbda4c06-c499-520d-52bc-2f751a7c30b4}" 2025-01-28T14:47:36.173Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:35.167439277 +0000 UTC m=+38.250845635 (1 nodes), 569.245235ms from start of poll, 574.337053ms from cluster start. Timeout at 2025-01-28 14:47:36.17245361 +0000 UTC m=+39.255858635, 1.005013s after stability 2025-01-28T14:47:36.174Z [INFO] creating new snapshot: path=/tmp/raft3868837951/snapshots/2-14-1738075656174.tmp 2025-01-28T14:47:36.179Z [INFO] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: copied to local snapshot: bytes=71 2025-01-28T14:47:36.182Z [INFO] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: snapshot restore progress: id=2-14-1738075656174 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:36.182Z [INFO] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: restored user snapshot: index=14 2025-01-28T14:47:36.183Z [ERROR] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: failed to get log: index=3 error="log not found" 2025-01-28T14:47:36.183Z [INFO] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: aborting pipeline replication: peer="{Voter server-dbda4c06-c499-520d-52bc-2f751a7c30b4 dbda4c06-c499-520d-52bc-2f751a7c30b4}" 2025-01-28T14:47:36.184Z [ERROR] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: failed to get log: index=3 error="log not found" 2025-01-28T14:47:36.184Z [INFO] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: aborting pipeline replication: peer="{Voter server-0c3d5030-e31e-a65f-ab98-18c6e1f535bb 0c3d5030-e31e-a65f-ab98-18c6e1f535bb}" 2025-01-28T14:47:36.191Z [ERROR] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: failed to get log: index=3 error="log not found" 2025-01-28T14:47:36.192Z [ERROR] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: failed to get log: index=3 error="log not found" 2025-01-28T14:47:36.194Z [INFO] creating new snapshot: path=/tmp/raft457258100/snapshots/2-14-1738075656194.tmp 2025-01-28T14:47:36.195Z [INFO] creating new snapshot: path=/tmp/raft1367413152/snapshots/2-14-1738075656195.tmp 2025-01-28T14:47:36.196Z [INFO] server-0c3d5030-e31e-a65f-ab98-18c6e1f535bb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:36.198Z [INFO] server-dbda4c06-c499-520d-52bc-2f751a7c30b4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:36.199Z [INFO] server-0c3d5030-e31e-a65f-ab98-18c6e1f535bb: copied to local snapshot: bytes=71 2025-01-28T14:47:36.200Z [INFO] server-dbda4c06-c499-520d-52bc-2f751a7c30b4: copied to local snapshot: bytes=71 2025-01-28T14:47:36.201Z [INFO] server-0c3d5030-e31e-a65f-ab98-18c6e1f535bb: snapshot restore progress: id=2-14-1738075656194 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:36.201Z [INFO] server-0c3d5030-e31e-a65f-ab98-18c6e1f535bb: Installed remote snapshot 2025-01-28T14:47:36.202Z [INFO] server-dbda4c06-c499-520d-52bc-2f751a7c30b4: snapshot restore progress: id=2-14-1738075656195 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:36.202Z [INFO] server-dbda4c06-c499-520d-52bc-2f751a7c30b4: Installed remote snapshot 2025-01-28T14:47:36.203Z [INFO] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: pipelining replication: peer="{Voter server-0c3d5030-e31e-a65f-ab98-18c6e1f535bb 0c3d5030-e31e-a65f-ab98-18c6e1f535bb}" 2025-01-28T14:47:36.203Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:36.204Z [INFO] server-d9bdee70-33be-8ce5-cc95-b4ae2f9be867: pipelining replication: peer="{Voter server-dbda4c06-c499-520d-52bc-2f751a7c30b4 dbda4c06-c499-520d-52bc-2f751a7c30b4}" 2025-01-28T14:47:37.209Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:36.203855383 +0000 UTC m=+39.287261740 (2 nodes), 55.001µs from start of poll, 1.610753158s from cluster start. Timeout at 2025-01-28 14:47:37.209075055 +0000 UTC m=+40.292481413, 1.005219673s after stability 2025-01-28T14:47:37.229Z [INFO] server-e76b4dbc-59c9-5243-69d8-ec7d906c1c31: aborting pipeline replication: peer="{Voter server-cad7fdee-dd1b-b5b3-8983-5972aea7e140 cad7fdee-dd1b-b5b3-8983-5972aea7e140}" 2025-01-28T14:47:37.229Z [INFO] server-e76b4dbc-59c9-5243-69d8-ec7d906c1c31: aborting pipeline replication: peer="{Voter server-bb582f92-2308-3fa9-7ed0-4487ae2e1e2d bb582f92-2308-3fa9-7ed0-4487ae2e1e2d}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true 2025-01-28T14:47:37.251Z [INFO] server-b62ac313-8e79-67ab-1df5-b2c266cd8ce8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b62ac313-8e79-67ab-1df5-b2c266cd8ce8 Address:b62ac313-8e79-67ab-1df5-b2c266cd8ce8} {Suffrage:Voter ID:server-806d8b01-e676-6255-2534-d40684ac061c Address:806d8b01-e676-6255-2534-d40684ac061c} {Suffrage:Voter ID:server-199bf27d-a513-3515-241b-2c3295ed158d Address:199bf27d-a513-3515-241b-2c3295ed158d}]" 2025-01-28T14:47:37.251Z [INFO] server-b62ac313-8e79-67ab-1df5-b2c266cd8ce8: entering follower state: follower="Node at b62ac313-8e79-67ab-1df5-b2c266cd8ce8 [Follower]" leader-address= leader-id= 2025-01-28T14:47:37.252Z [INFO] server-806d8b01-e676-6255-2534-d40684ac061c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b62ac313-8e79-67ab-1df5-b2c266cd8ce8 Address:b62ac313-8e79-67ab-1df5-b2c266cd8ce8} {Suffrage:Voter ID:server-806d8b01-e676-6255-2534-d40684ac061c Address:806d8b01-e676-6255-2534-d40684ac061c} {Suffrage:Voter ID:server-199bf27d-a513-3515-241b-2c3295ed158d Address:199bf27d-a513-3515-241b-2c3295ed158d}]" 2025-01-28T14:47:37.253Z [INFO] server-806d8b01-e676-6255-2534-d40684ac061c: entering follower state: follower="Node at 806d8b01-e676-6255-2534-d40684ac061c [Follower]" leader-address= leader-id= 2025-01-28T14:47:37.254Z [INFO] server-199bf27d-a513-3515-241b-2c3295ed158d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b62ac313-8e79-67ab-1df5-b2c266cd8ce8 Address:b62ac313-8e79-67ab-1df5-b2c266cd8ce8} {Suffrage:Voter ID:server-806d8b01-e676-6255-2534-d40684ac061c Address:806d8b01-e676-6255-2534-d40684ac061c} {Suffrage:Voter ID:server-199bf27d-a513-3515-241b-2c3295ed158d Address:199bf27d-a513-3515-241b-2c3295ed158d}]" 2025-01-28T14:47:37.254Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:37.254Z [INFO] server-199bf27d-a513-3515-241b-2c3295ed158d: entering follower state: follower="Node at 199bf27d-a513-3515-241b-2c3295ed158d [Follower]" leader-address= leader-id= 2025-01-28T14:47:37.757Z [WARN] server-199bf27d-a513-3515-241b-2c3295ed158d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:37.757Z [INFO] server-199bf27d-a513-3515-241b-2c3295ed158d: entering candidate state: node="Node at 199bf27d-a513-3515-241b-2c3295ed158d [Candidate]" term=2 2025-01-28T14:47:37.758Z [INFO] server-199bf27d-a513-3515-241b-2c3295ed158d: election won: term=2 tally=2 2025-01-28T14:47:37.758Z [INFO] server-199bf27d-a513-3515-241b-2c3295ed158d: entering leader state: leader="Node at 199bf27d-a513-3515-241b-2c3295ed158d [Leader]" 2025-01-28T14:47:37.759Z [INFO] server-199bf27d-a513-3515-241b-2c3295ed158d: added peer, starting replication: peer=server-b62ac313-8e79-67ab-1df5-b2c266cd8ce8 2025-01-28T14:47:37.759Z [INFO] server-199bf27d-a513-3515-241b-2c3295ed158d: added peer, starting replication: peer=server-806d8b01-e676-6255-2534-d40684ac061c 2025-01-28T14:47:37.760Z [INFO] server-199bf27d-a513-3515-241b-2c3295ed158d: pipelining replication: peer="{Voter server-b62ac313-8e79-67ab-1df5-b2c266cd8ce8 b62ac313-8e79-67ab-1df5-b2c266cd8ce8}" 2025-01-28T14:47:37.761Z [INFO] server-199bf27d-a513-3515-241b-2c3295ed158d: pipelining replication: peer="{Voter server-806d8b01-e676-6255-2534-d40684ac061c 806d8b01-e676-6255-2534-d40684ac061c}" 2025-01-28T14:47:38.765Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:37.759387902 +0000 UTC m=+40.842794259 (1 nodes), 504.529313ms from start of poll, 509.795803ms from cluster start. Timeout at 2025-01-28 14:47:38.764411569 +0000 UTC m=+41.847817593, 1.005023334s after stability 2025-01-28T14:47:38.772Z [INFO] server-199bf27d-a513-3515-241b-2c3295ed158d: starting snapshot up to: index=12 2025-01-28T14:47:38.772Z [INFO] creating new snapshot: path=/tmp/raft1921814711/snapshots/2-12-1738075658772.tmp 2025-01-28T14:47:38.776Z [INFO] server-199bf27d-a513-3515-241b-2c3295ed158d: snapshot complete up to: index=12 2025-01-28T14:47:38.791Z [INFO] server-ebc7083a-6739-76b2-4583-bc6fc34432e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ebc7083a-6739-76b2-4583-bc6fc34432e8 Address:ebc7083a-6739-76b2-4583-bc6fc34432e8} {Suffrage:Voter ID:server-7497df4b-5a7a-1c46-9b74-89cb4986583b Address:7497df4b-5a7a-1c46-9b74-89cb4986583b} {Suffrage:Voter ID:server-99561da2-acdf-619f-ae13-365931711180 Address:99561da2-acdf-619f-ae13-365931711180}]" 2025-01-28T14:47:38.791Z [INFO] server-ebc7083a-6739-76b2-4583-bc6fc34432e8: entering follower state: follower="Node at ebc7083a-6739-76b2-4583-bc6fc34432e8 [Follower]" leader-address= leader-id= 2025-01-28T14:47:38.792Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ebc7083a-6739-76b2-4583-bc6fc34432e8 Address:ebc7083a-6739-76b2-4583-bc6fc34432e8} {Suffrage:Voter ID:server-7497df4b-5a7a-1c46-9b74-89cb4986583b Address:7497df4b-5a7a-1c46-9b74-89cb4986583b} {Suffrage:Voter ID:server-99561da2-acdf-619f-ae13-365931711180 Address:99561da2-acdf-619f-ae13-365931711180}]" 2025-01-28T14:47:38.792Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: entering follower state: follower="Node at 7497df4b-5a7a-1c46-9b74-89cb4986583b [Follower]" leader-address= leader-id= 2025-01-28T14:47:38.792Z [INFO] server-99561da2-acdf-619f-ae13-365931711180: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ebc7083a-6739-76b2-4583-bc6fc34432e8 Address:ebc7083a-6739-76b2-4583-bc6fc34432e8} {Suffrage:Voter ID:server-7497df4b-5a7a-1c46-9b74-89cb4986583b Address:7497df4b-5a7a-1c46-9b74-89cb4986583b} {Suffrage:Voter ID:server-99561da2-acdf-619f-ae13-365931711180 Address:99561da2-acdf-619f-ae13-365931711180}]" 2025-01-28T14:47:38.792Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:38.792Z [INFO] server-99561da2-acdf-619f-ae13-365931711180: entering follower state: follower="Node at 99561da2-acdf-619f-ae13-365931711180 [Follower]" leader-address= leader-id= 2025-01-28T14:47:39.310Z [WARN] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:39.310Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: entering candidate state: node="Node at 7497df4b-5a7a-1c46-9b74-89cb4986583b [Candidate]" term=2 2025-01-28T14:47:39.311Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: election won: term=2 tally=2 2025-01-28T14:47:39.311Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: entering leader state: leader="Node at 7497df4b-5a7a-1c46-9b74-89cb4986583b [Leader]" 2025-01-28T14:47:39.312Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: added peer, starting replication: peer=server-ebc7083a-6739-76b2-4583-bc6fc34432e8 2025-01-28T14:47:39.312Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: added peer, starting replication: peer=server-99561da2-acdf-619f-ae13-365931711180 2025-01-28T14:47:39.313Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: pipelining replication: peer="{Voter server-ebc7083a-6739-76b2-4583-bc6fc34432e8 ebc7083a-6739-76b2-4583-bc6fc34432e8}" 2025-01-28T14:47:39.314Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: pipelining replication: peer="{Voter server-99561da2-acdf-619f-ae13-365931711180 99561da2-acdf-619f-ae13-365931711180}" 2025-01-28T14:47:40.318Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:39.312480015 +0000 UTC m=+42.395886373 (1 nodes), 519.397755ms from start of poll, 522.939194ms from cluster start. Timeout at 2025-01-28 14:47:40.317503349 +0000 UTC m=+43.400909373, 1.005023s after stability 2025-01-28T14:47:40.320Z [INFO] creating new snapshot: path=/tmp/raft3960274483/snapshots/2-14-1738075660320.tmp 2025-01-28T14:47:40.328Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: copied to local snapshot: bytes=71 2025-01-28T14:47:40.330Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: snapshot restore progress: id=2-14-1738075660320 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:40.330Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: removing all old logs from log store 2025-01-28T14:47:40.330Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: compacting logs: from=1 to=2 2025-01-28T14:47:40.331Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: restored user snapshot: index=14 2025-01-28T14:47:40.331Z [ERROR] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: failed to get log: index=2 error="log not found" 2025-01-28T14:47:40.331Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: aborting pipeline replication: peer="{Voter server-99561da2-acdf-619f-ae13-365931711180 99561da2-acdf-619f-ae13-365931711180}" 2025-01-28T14:47:40.331Z [ERROR] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: failed to get log: index=2 error="log not found" 2025-01-28T14:47:40.331Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: aborting pipeline replication: peer="{Voter server-ebc7083a-6739-76b2-4583-bc6fc34432e8 ebc7083a-6739-76b2-4583-bc6fc34432e8}" 2025-01-28T14:47:40.339Z [ERROR] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: failed to get log: index=2 error="log not found" 2025-01-28T14:47:40.339Z [ERROR] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: failed to get log: index=2 error="log not found" 2025-01-28T14:47:40.348Z [INFO] creating new snapshot: path=/tmp/raft3906985356/snapshots/2-14-1738075660348.tmp 2025-01-28T14:47:40.355Z [INFO] creating new snapshot: path=/tmp/raft3568417245/snapshots/2-14-1738075660354.tmp 2025-01-28T14:47:40.356Z [INFO] server-99561da2-acdf-619f-ae13-365931711180: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:40.359Z [INFO] server-99561da2-acdf-619f-ae13-365931711180: copied to local snapshot: bytes=71 2025-01-28T14:47:40.360Z [INFO] server-ebc7083a-6739-76b2-4583-bc6fc34432e8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:40.361Z [INFO] server-99561da2-acdf-619f-ae13-365931711180: snapshot restore progress: id=2-14-1738075660348 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:40.361Z [INFO] server-99561da2-acdf-619f-ae13-365931711180: removing all old logs from log store 2025-01-28T14:47:40.361Z [INFO] server-99561da2-acdf-619f-ae13-365931711180: compacting logs: from=1 to=2 2025-01-28T14:47:40.361Z [INFO] server-99561da2-acdf-619f-ae13-365931711180: Installed remote snapshot 2025-01-28T14:47:40.363Z [INFO] server-ebc7083a-6739-76b2-4583-bc6fc34432e8: copied to local snapshot: bytes=71 2025-01-28T14:47:40.363Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: pipelining replication: peer="{Voter server-99561da2-acdf-619f-ae13-365931711180 99561da2-acdf-619f-ae13-365931711180}" 2025-01-28T14:47:40.363Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:40.364Z [INFO] server-ebc7083a-6739-76b2-4583-bc6fc34432e8: snapshot restore progress: id=2-14-1738075660354 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:40.364Z [INFO] server-ebc7083a-6739-76b2-4583-bc6fc34432e8: removing all old logs from log store 2025-01-28T14:47:40.364Z [INFO] server-ebc7083a-6739-76b2-4583-bc6fc34432e8: compacting logs: from=1 to=2 2025-01-28T14:47:40.364Z [INFO] server-ebc7083a-6739-76b2-4583-bc6fc34432e8: Installed remote snapshot 2025-01-28T14:47:40.365Z [INFO] server-7497df4b-5a7a-1c46-9b74-89cb4986583b: pipelining replication: peer="{Voter server-ebc7083a-6739-76b2-4583-bc6fc34432e8 ebc7083a-6739-76b2-4583-bc6fc34432e8}" 2025-01-28T14:47:41.370Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:40.363964569 +0000 UTC m=+43.447369593 (2 nodes), 9.667µs from start of poll, 1.574422414s from cluster start. Timeout at 2025-01-28 14:47:41.369271577 +0000 UTC m=+44.452678268, 1.005308675s after stability 2025-01-28T14:47:41.391Z [INFO] server-199bf27d-a513-3515-241b-2c3295ed158d: aborting pipeline replication: peer="{Voter server-806d8b01-e676-6255-2534-d40684ac061c 806d8b01-e676-6255-2534-d40684ac061c}" 2025-01-28T14:47:41.391Z [INFO] server-199bf27d-a513-3515-241b-2c3295ed158d: aborting pipeline replication: peer="{Voter server-b62ac313-8e79-67ab-1df5-b2c266cd8ce8 b62ac313-8e79-67ab-1df5-b2c266cd8ce8}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false 2025-01-28T14:47:41.407Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-541f64d1-861e-e225-e2bb-bc12467aad06 Address:541f64d1-861e-e225-e2bb-bc12467aad06} {Suffrage:Voter ID:server-c1d90d27-5eb2-d124-263e-38a4c1230359 Address:c1d90d27-5eb2-d124-263e-38a4c1230359} {Suffrage:Voter ID:server-c7fc58ce-93ea-adc0-5c6a-39e639d65742 Address:c7fc58ce-93ea-adc0-5c6a-39e639d65742}]" 2025-01-28T14:47:41.407Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: entering follower state: follower="Node at 541f64d1-861e-e225-e2bb-bc12467aad06 [Follower]" leader-address= leader-id= 2025-01-28T14:47:41.408Z [INFO] server-c1d90d27-5eb2-d124-263e-38a4c1230359: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-541f64d1-861e-e225-e2bb-bc12467aad06 Address:541f64d1-861e-e225-e2bb-bc12467aad06} {Suffrage:Voter ID:server-c1d90d27-5eb2-d124-263e-38a4c1230359 Address:c1d90d27-5eb2-d124-263e-38a4c1230359} {Suffrage:Voter ID:server-c7fc58ce-93ea-adc0-5c6a-39e639d65742 Address:c7fc58ce-93ea-adc0-5c6a-39e639d65742}]" 2025-01-28T14:47:41.409Z [INFO] server-c1d90d27-5eb2-d124-263e-38a4c1230359: entering follower state: follower="Node at c1d90d27-5eb2-d124-263e-38a4c1230359 [Follower]" leader-address= leader-id= 2025-01-28T14:47:41.409Z [INFO] server-c7fc58ce-93ea-adc0-5c6a-39e639d65742: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-541f64d1-861e-e225-e2bb-bc12467aad06 Address:541f64d1-861e-e225-e2bb-bc12467aad06} {Suffrage:Voter ID:server-c1d90d27-5eb2-d124-263e-38a4c1230359 Address:c1d90d27-5eb2-d124-263e-38a4c1230359} {Suffrage:Voter ID:server-c7fc58ce-93ea-adc0-5c6a-39e639d65742 Address:c7fc58ce-93ea-adc0-5c6a-39e639d65742}]" 2025-01-28T14:47:41.409Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:41.409Z [INFO] server-c7fc58ce-93ea-adc0-5c6a-39e639d65742: entering follower state: follower="Node at c7fc58ce-93ea-adc0-5c6a-39e639d65742 [Follower]" leader-address= leader-id= 2025-01-28T14:47:42.164Z [WARN] server-541f64d1-861e-e225-e2bb-bc12467aad06: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:42.164Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: entering candidate state: node="Node at 541f64d1-861e-e225-e2bb-bc12467aad06 [Candidate]" term=2 2025-01-28T14:47:42.166Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: election won: term=2 tally=2 2025-01-28T14:47:42.166Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: entering leader state: leader="Node at 541f64d1-861e-e225-e2bb-bc12467aad06 [Leader]" 2025-01-28T14:47:42.166Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: added peer, starting replication: peer=server-c1d90d27-5eb2-d124-263e-38a4c1230359 2025-01-28T14:47:42.167Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: added peer, starting replication: peer=server-c7fc58ce-93ea-adc0-5c6a-39e639d65742 2025-01-28T14:47:42.169Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: pipelining replication: peer="{Voter server-c1d90d27-5eb2-d124-263e-38a4c1230359 c1d90d27-5eb2-d124-263e-38a4c1230359}" 2025-01-28T14:47:42.170Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: pipelining replication: peer="{Voter server-c7fc58ce-93ea-adc0-5c6a-39e639d65742 c7fc58ce-93ea-adc0-5c6a-39e639d65742}" 2025-01-28T14:47:43.172Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:42.166750763 +0000 UTC m=+45.250157121 (1 nodes), 757.118147ms from start of poll, 759.776559ms from cluster start. Timeout at 2025-01-28 14:47:43.171764764 +0000 UTC m=+46.255169455, 1.005012334s after stability 2025-01-28T14:47:43.191Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: starting snapshot up to: index=12 2025-01-28T14:47:43.191Z [INFO] creating new snapshot: path=/tmp/raft1314639082/snapshots/2-12-1738075663191.tmp 2025-01-28T14:47:43.195Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: snapshot complete up to: index=12 2025-01-28T14:47:43.201Z [INFO] creating new snapshot: path=/tmp/raft1314639082/snapshots/2-23-1738075663201.tmp 2025-01-28T14:47:43.205Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: copied to local snapshot: bytes=71 2025-01-28T14:47:43.207Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: snapshot restore progress: id=2-23-1738075663201 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:43.207Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: restored user snapshot: index=23 2025-01-28T14:47:43.207Z [ERROR] server-541f64d1-861e-e225-e2bb-bc12467aad06: failed to get log: index=23 error="log not found" 2025-01-28T14:47:43.208Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: aborting pipeline replication: peer="{Voter server-c1d90d27-5eb2-d124-263e-38a4c1230359 c1d90d27-5eb2-d124-263e-38a4c1230359}" 2025-01-28T14:47:43.207Z [ERROR] server-541f64d1-861e-e225-e2bb-bc12467aad06: failed to get log: index=23 error="log not found" 2025-01-28T14:47:43.208Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: aborting pipeline replication: peer="{Voter server-c7fc58ce-93ea-adc0-5c6a-39e639d65742 c7fc58ce-93ea-adc0-5c6a-39e639d65742}" 2025-01-28T14:47:43.214Z [ERROR] server-541f64d1-861e-e225-e2bb-bc12467aad06: failed to get log: index=23 error="log not found" 2025-01-28T14:47:43.215Z [ERROR] server-541f64d1-861e-e225-e2bb-bc12467aad06: failed to get log: index=23 error="log not found" 2025-01-28T14:47:43.217Z [INFO] creating new snapshot: path=/tmp/raft1599551405/snapshots/2-23-1738075663217.tmp 2025-01-28T14:47:43.218Z [INFO] creating new snapshot: path=/tmp/raft2577505623/snapshots/2-23-1738075663218.tmp 2025-01-28T14:47:43.219Z [INFO] server-c1d90d27-5eb2-d124-263e-38a4c1230359: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:43.220Z [INFO] server-c7fc58ce-93ea-adc0-5c6a-39e639d65742: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:43.221Z [INFO] server-c1d90d27-5eb2-d124-263e-38a4c1230359: copied to local snapshot: bytes=71 2025-01-28T14:47:43.223Z [INFO] server-c1d90d27-5eb2-d124-263e-38a4c1230359: snapshot restore progress: id=2-23-1738075663217 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:43.223Z [INFO] server-c1d90d27-5eb2-d124-263e-38a4c1230359: Installed remote snapshot 2025-01-28T14:47:43.224Z [INFO] server-c7fc58ce-93ea-adc0-5c6a-39e639d65742: copied to local snapshot: bytes=71 2025-01-28T14:47:43.224Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: pipelining replication: peer="{Voter server-c1d90d27-5eb2-d124-263e-38a4c1230359 c1d90d27-5eb2-d124-263e-38a4c1230359}" 2025-01-28T14:47:43.225Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:43.225Z [INFO] server-c7fc58ce-93ea-adc0-5c6a-39e639d65742: snapshot restore progress: id=2-23-1738075663218 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:43.226Z [INFO] server-c7fc58ce-93ea-adc0-5c6a-39e639d65742: Installed remote snapshot 2025-01-28T14:47:43.227Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: pipelining replication: peer="{Voter server-c7fc58ce-93ea-adc0-5c6a-39e639d65742 c7fc58ce-93ea-adc0-5c6a-39e639d65742}" 2025-01-28T14:47:44.231Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:43.225627203 +0000 UTC m=+46.309031894 (2 nodes), 9.667µs from start of poll, 1.818651332s from cluster start. Timeout at 2025-01-28 14:47:44.23097188 +0000 UTC m=+47.314377571, 1.005345677s after stability 2025-01-28T14:47:44.250Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: aborting pipeline replication: peer="{Voter server-c7fc58ce-93ea-adc0-5c6a-39e639d65742 c7fc58ce-93ea-adc0-5c6a-39e639d65742}" 2025-01-28T14:47:44.251Z [INFO] server-541f64d1-861e-e225-e2bb-bc12467aad06: aborting pipeline replication: peer="{Voter server-c1d90d27-5eb2-d124-263e-38a4c1230359 c1d90d27-5eb2-d124-263e-38a4c1230359}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false 2025-01-28T14:47:44.274Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5 Address:e7c6f8f8-cdef-8d82-e864-83ca31daaff5} {Suffrage:Voter ID:server-20f127e6-58df-01b2-44af-b7412c8a286f Address:20f127e6-58df-01b2-44af-b7412c8a286f} {Suffrage:Voter ID:server-c9525c34-7e72-5157-463b-041c518d97fa Address:c9525c34-7e72-5157-463b-041c518d97fa}]" 2025-01-28T14:47:44.274Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: entering follower state: follower="Node at e7c6f8f8-cdef-8d82-e864-83ca31daaff5 [Follower]" leader-address= leader-id= 2025-01-28T14:47:44.275Z [INFO] server-20f127e6-58df-01b2-44af-b7412c8a286f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5 Address:e7c6f8f8-cdef-8d82-e864-83ca31daaff5} {Suffrage:Voter ID:server-20f127e6-58df-01b2-44af-b7412c8a286f Address:20f127e6-58df-01b2-44af-b7412c8a286f} {Suffrage:Voter ID:server-c9525c34-7e72-5157-463b-041c518d97fa Address:c9525c34-7e72-5157-463b-041c518d97fa}]" 2025-01-28T14:47:44.276Z [INFO] server-20f127e6-58df-01b2-44af-b7412c8a286f: entering follower state: follower="Node at 20f127e6-58df-01b2-44af-b7412c8a286f [Follower]" leader-address= leader-id= 2025-01-28T14:47:44.277Z [INFO] server-c9525c34-7e72-5157-463b-041c518d97fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5 Address:e7c6f8f8-cdef-8d82-e864-83ca31daaff5} {Suffrage:Voter ID:server-20f127e6-58df-01b2-44af-b7412c8a286f Address:20f127e6-58df-01b2-44af-b7412c8a286f} {Suffrage:Voter ID:server-c9525c34-7e72-5157-463b-041c518d97fa Address:c9525c34-7e72-5157-463b-041c518d97fa}]" 2025-01-28T14:47:44.277Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:44.277Z [INFO] server-c9525c34-7e72-5157-463b-041c518d97fa: entering follower state: follower="Node at c9525c34-7e72-5157-463b-041c518d97fa [Follower]" leader-address= leader-id= 2025-01-28T14:47:44.853Z [WARN] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:44.854Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: entering candidate state: node="Node at e7c6f8f8-cdef-8d82-e864-83ca31daaff5 [Candidate]" term=2 2025-01-28T14:47:44.855Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: election won: term=2 tally=2 2025-01-28T14:47:44.855Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: entering leader state: leader="Node at e7c6f8f8-cdef-8d82-e864-83ca31daaff5 [Leader]" 2025-01-28T14:47:44.855Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: added peer, starting replication: peer=server-20f127e6-58df-01b2-44af-b7412c8a286f 2025-01-28T14:47:44.855Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: added peer, starting replication: peer=server-c9525c34-7e72-5157-463b-041c518d97fa 2025-01-28T14:47:44.857Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: pipelining replication: peer="{Voter server-20f127e6-58df-01b2-44af-b7412c8a286f 20f127e6-58df-01b2-44af-b7412c8a286f}" 2025-01-28T14:47:44.857Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: pipelining replication: peer="{Voter server-c9525c34-7e72-5157-463b-041c518d97fa c9525c34-7e72-5157-463b-041c518d97fa}" 2025-01-28T14:47:45.861Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:44.85594361 +0000 UTC m=+47.939350301 (1 nodes), 578.139832ms from start of poll, 583.187649ms from cluster start. Timeout at 2025-01-28 14:47:45.860989612 +0000 UTC m=+48.944395636, 1.005045335s after stability 2025-01-28T14:47:45.870Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: starting snapshot up to: index=12 2025-01-28T14:47:45.870Z [INFO] creating new snapshot: path=/tmp/raft3587898646/snapshots/2-12-1738075665870.tmp 2025-01-28T14:47:45.874Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: snapshot complete up to: index=12 2025-01-28T14:47:45.880Z [INFO] creating new snapshot: path=/tmp/raft3587898646/snapshots/2-23-1738075665880.tmp 2025-01-28T14:47:45.888Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: copied to local snapshot: bytes=71 2025-01-28T14:47:45.891Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: snapshot restore progress: id=2-23-1738075665880 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:45.891Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: removing all old logs from log store 2025-01-28T14:47:45.891Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: compacting logs: from=1 to=22 2025-01-28T14:47:45.891Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: restored user snapshot: index=23 2025-01-28T14:47:45.892Z [ERROR] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: failed to get log: index=22 error="log not found" 2025-01-28T14:47:45.892Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: aborting pipeline replication: peer="{Voter server-c9525c34-7e72-5157-463b-041c518d97fa c9525c34-7e72-5157-463b-041c518d97fa}" 2025-01-28T14:47:45.892Z [ERROR] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: failed to get log: index=22 error="log not found" 2025-01-28T14:47:45.892Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: aborting pipeline replication: peer="{Voter server-20f127e6-58df-01b2-44af-b7412c8a286f 20f127e6-58df-01b2-44af-b7412c8a286f}" 2025-01-28T14:47:45.898Z [ERROR] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: failed to get log: index=22 error="log not found" 2025-01-28T14:47:45.900Z [ERROR] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: failed to get log: index=22 error="log not found" 2025-01-28T14:47:45.901Z [INFO] creating new snapshot: path=/tmp/raft2000382351/snapshots/2-23-1738075665901.tmp 2025-01-28T14:47:45.905Z [INFO] server-c9525c34-7e72-5157-463b-041c518d97fa: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:45.908Z [INFO] server-c9525c34-7e72-5157-463b-041c518d97fa: copied to local snapshot: bytes=71 2025-01-28T14:47:45.910Z [INFO] server-c9525c34-7e72-5157-463b-041c518d97fa: snapshot restore progress: id=2-23-1738075665901 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:45.910Z [INFO] server-c9525c34-7e72-5157-463b-041c518d97fa: removing all old logs from log store 2025-01-28T14:47:45.910Z [INFO] server-c9525c34-7e72-5157-463b-041c518d97fa: compacting logs: from=1 to=22 2025-01-28T14:47:45.910Z [INFO] server-c9525c34-7e72-5157-463b-041c518d97fa: Installed remote snapshot 2025-01-28T14:47:45.913Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: pipelining replication: peer="{Voter server-c9525c34-7e72-5157-463b-041c518d97fa c9525c34-7e72-5157-463b-041c518d97fa}" 2025-01-28T14:47:45.914Z [INFO] creating new snapshot: path=/tmp/raft3574893251/snapshots/2-23-1738075665914.tmp 2025-01-28T14:47:45.914Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:45.916Z [INFO] server-20f127e6-58df-01b2-44af-b7412c8a286f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:45.919Z [INFO] server-20f127e6-58df-01b2-44af-b7412c8a286f: copied to local snapshot: bytes=71 2025-01-28T14:47:45.920Z [INFO] server-20f127e6-58df-01b2-44af-b7412c8a286f: snapshot restore progress: id=2-23-1738075665914 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:45.920Z [INFO] server-20f127e6-58df-01b2-44af-b7412c8a286f: removing all old logs from log store 2025-01-28T14:47:45.920Z [INFO] server-20f127e6-58df-01b2-44af-b7412c8a286f: compacting logs: from=1 to=22 2025-01-28T14:47:45.920Z [INFO] server-20f127e6-58df-01b2-44af-b7412c8a286f: Installed remote snapshot 2025-01-28T14:47:45.921Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: pipelining replication: peer="{Voter server-20f127e6-58df-01b2-44af-b7412c8a286f 20f127e6-58df-01b2-44af-b7412c8a286f}" 2025-01-28T14:47:46.921Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:45.914374704 +0000 UTC m=+48.997779728 (2 nodes), 8.333µs from start of poll, 1.641617076s from cluster start. Timeout at 2025-01-28 14:47:46.919519375 +0000 UTC m=+50.002924399, 1.005144671s after stability 2025-01-28T14:47:46.941Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: aborting pipeline replication: peer="{Voter server-c9525c34-7e72-5157-463b-041c518d97fa c9525c34-7e72-5157-463b-041c518d97fa}" 2025-01-28T14:47:46.941Z [INFO] server-e7c6f8f8-cdef-8d82-e864-83ca31daaff5: aborting pipeline replication: peer="{Voter server-20f127e6-58df-01b2-44af-b7412c8a286f 20f127e6-58df-01b2-44af-b7412c8a286f}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true 2025-01-28T14:47:46.961Z [INFO] server-91ed2ad3-61f0-e889-7647-ffe91f847860: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91ed2ad3-61f0-e889-7647-ffe91f847860 Address:91ed2ad3-61f0-e889-7647-ffe91f847860} {Suffrage:Voter ID:server-b38473ca-6276-4d1b-e145-f87a31535893 Address:b38473ca-6276-4d1b-e145-f87a31535893} {Suffrage:Voter ID:server-5f3ea835-045e-5e32-edb3-d81c30891701 Address:5f3ea835-045e-5e32-edb3-d81c30891701}]" 2025-01-28T14:47:46.961Z [INFO] server-91ed2ad3-61f0-e889-7647-ffe91f847860: entering follower state: follower="Node at 91ed2ad3-61f0-e889-7647-ffe91f847860 [Follower]" leader-address= leader-id= 2025-01-28T14:47:46.962Z [INFO] server-b38473ca-6276-4d1b-e145-f87a31535893: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91ed2ad3-61f0-e889-7647-ffe91f847860 Address:91ed2ad3-61f0-e889-7647-ffe91f847860} {Suffrage:Voter ID:server-b38473ca-6276-4d1b-e145-f87a31535893 Address:b38473ca-6276-4d1b-e145-f87a31535893} {Suffrage:Voter ID:server-5f3ea835-045e-5e32-edb3-d81c30891701 Address:5f3ea835-045e-5e32-edb3-d81c30891701}]" 2025-01-28T14:47:46.962Z [INFO] server-b38473ca-6276-4d1b-e145-f87a31535893: entering follower state: follower="Node at b38473ca-6276-4d1b-e145-f87a31535893 [Follower]" leader-address= leader-id= 2025-01-28T14:47:46.963Z [INFO] server-5f3ea835-045e-5e32-edb3-d81c30891701: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91ed2ad3-61f0-e889-7647-ffe91f847860 Address:91ed2ad3-61f0-e889-7647-ffe91f847860} {Suffrage:Voter ID:server-b38473ca-6276-4d1b-e145-f87a31535893 Address:b38473ca-6276-4d1b-e145-f87a31535893} {Suffrage:Voter ID:server-5f3ea835-045e-5e32-edb3-d81c30891701 Address:5f3ea835-045e-5e32-edb3-d81c30891701}]" 2025-01-28T14:47:46.963Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:46.963Z [INFO] server-5f3ea835-045e-5e32-edb3-d81c30891701: entering follower state: follower="Node at 5f3ea835-045e-5e32-edb3-d81c30891701 [Follower]" leader-address= leader-id= 2025-01-28T14:47:47.692Z [WARN] server-91ed2ad3-61f0-e889-7647-ffe91f847860: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:47.693Z [INFO] server-91ed2ad3-61f0-e889-7647-ffe91f847860: entering candidate state: node="Node at 91ed2ad3-61f0-e889-7647-ffe91f847860 [Candidate]" term=2 2025-01-28T14:47:47.694Z [INFO] server-91ed2ad3-61f0-e889-7647-ffe91f847860: election won: term=2 tally=2 2025-01-28T14:47:47.694Z [INFO] server-91ed2ad3-61f0-e889-7647-ffe91f847860: entering leader state: leader="Node at 91ed2ad3-61f0-e889-7647-ffe91f847860 [Leader]" 2025-01-28T14:47:47.694Z [INFO] server-91ed2ad3-61f0-e889-7647-ffe91f847860: added peer, starting replication: peer=server-b38473ca-6276-4d1b-e145-f87a31535893 2025-01-28T14:47:47.695Z [INFO] server-91ed2ad3-61f0-e889-7647-ffe91f847860: added peer, starting replication: peer=server-5f3ea835-045e-5e32-edb3-d81c30891701 2025-01-28T14:47:47.696Z [INFO] server-91ed2ad3-61f0-e889-7647-ffe91f847860: pipelining replication: peer="{Voter server-b38473ca-6276-4d1b-e145-f87a31535893 b38473ca-6276-4d1b-e145-f87a31535893}" 2025-01-28T14:47:47.697Z [INFO] server-91ed2ad3-61f0-e889-7647-ffe91f847860: pipelining replication: peer="{Voter server-5f3ea835-045e-5e32-edb3-d81c30891701 5f3ea835-045e-5e32-edb3-d81c30891701}" 2025-01-28T14:47:48.700Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:47.694743901 +0000 UTC m=+50.778150258 (1 nodes), 731.213711ms from start of poll, 734.131464ms from cluster start. Timeout at 2025-01-28 14:47:48.699763901 +0000 UTC m=+51.783169925, 1.005019667s after stability 2025-01-28T14:47:48.708Z [INFO] server-91ed2ad3-61f0-e889-7647-ffe91f847860: starting snapshot up to: index=12 2025-01-28T14:47:48.708Z [INFO] creating new snapshot: path=/tmp/raft3470672819/snapshots/2-12-1738075668708.tmp 2025-01-28T14:47:48.712Z [INFO] server-91ed2ad3-61f0-e889-7647-ffe91f847860: snapshot complete up to: index=12 2025-01-28T14:47:48.724Z [INFO] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd Address:bb52452f-f1be-9b4d-6a01-ebc8ba97cabd} {Suffrage:Voter ID:server-b8fb1426-7437-8a41-6fb0-6533fd60b357 Address:b8fb1426-7437-8a41-6fb0-6533fd60b357} {Suffrage:Voter ID:server-5bd05750-583d-16dc-cbab-1bc2a5fc8d03 Address:5bd05750-583d-16dc-cbab-1bc2a5fc8d03}]" 2025-01-28T14:47:48.725Z [INFO] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: entering follower state: follower="Node at bb52452f-f1be-9b4d-6a01-ebc8ba97cabd [Follower]" leader-address= leader-id= 2025-01-28T14:47:48.726Z [INFO] server-b8fb1426-7437-8a41-6fb0-6533fd60b357: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd Address:bb52452f-f1be-9b4d-6a01-ebc8ba97cabd} {Suffrage:Voter ID:server-b8fb1426-7437-8a41-6fb0-6533fd60b357 Address:b8fb1426-7437-8a41-6fb0-6533fd60b357} {Suffrage:Voter ID:server-5bd05750-583d-16dc-cbab-1bc2a5fc8d03 Address:5bd05750-583d-16dc-cbab-1bc2a5fc8d03}]" 2025-01-28T14:47:48.727Z [INFO] server-b8fb1426-7437-8a41-6fb0-6533fd60b357: entering follower state: follower="Node at b8fb1426-7437-8a41-6fb0-6533fd60b357 [Follower]" leader-address= leader-id= 2025-01-28T14:47:48.728Z [INFO] server-5bd05750-583d-16dc-cbab-1bc2a5fc8d03: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd Address:bb52452f-f1be-9b4d-6a01-ebc8ba97cabd} {Suffrage:Voter ID:server-b8fb1426-7437-8a41-6fb0-6533fd60b357 Address:b8fb1426-7437-8a41-6fb0-6533fd60b357} {Suffrage:Voter ID:server-5bd05750-583d-16dc-cbab-1bc2a5fc8d03 Address:5bd05750-583d-16dc-cbab-1bc2a5fc8d03}]" 2025-01-28T14:47:48.728Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:48.728Z [INFO] server-5bd05750-583d-16dc-cbab-1bc2a5fc8d03: entering follower state: follower="Node at 5bd05750-583d-16dc-cbab-1bc2a5fc8d03 [Follower]" leader-address= leader-id= 2025-01-28T14:47:49.251Z [WARN] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:49.251Z [INFO] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: entering candidate state: node="Node at bb52452f-f1be-9b4d-6a01-ebc8ba97cabd [Candidate]" term=2 2025-01-28T14:47:49.252Z [INFO] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: election won: term=2 tally=2 2025-01-28T14:47:49.252Z [INFO] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: entering leader state: leader="Node at bb52452f-f1be-9b4d-6a01-ebc8ba97cabd [Leader]" 2025-01-28T14:47:49.252Z [INFO] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: added peer, starting replication: peer=server-b8fb1426-7437-8a41-6fb0-6533fd60b357 2025-01-28T14:47:49.253Z [INFO] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: added peer, starting replication: peer=server-5bd05750-583d-16dc-cbab-1bc2a5fc8d03 2025-01-28T14:47:49.254Z [INFO] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: pipelining replication: peer="{Voter server-b8fb1426-7437-8a41-6fb0-6533fd60b357 b8fb1426-7437-8a41-6fb0-6533fd60b357}" 2025-01-28T14:47:49.254Z [INFO] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: pipelining replication: peer="{Voter server-5bd05750-583d-16dc-cbab-1bc2a5fc8d03 5bd05750-583d-16dc-cbab-1bc2a5fc8d03}" 2025-01-28T14:47:50.258Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:49.252814164 +0000 UTC m=+52.336220521 (1 nodes), 523.995892ms from start of poll, 529.401719ms from cluster start. Timeout at 2025-01-28 14:47:50.257838831 +0000 UTC m=+53.341244188, 1.005023667s after stability 2025-01-28T14:47:50.259Z [INFO] creating new snapshot: path=/tmp/raft1063363986/snapshots/2-15-1738075670259.tmp 2025-01-28T14:47:50.263Z [INFO] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: copied to local snapshot: bytes=71 2025-01-28T14:47:50.265Z [INFO] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: snapshot restore progress: id=2-15-1738075670259 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:50.265Z [INFO] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: restored user snapshot: index=15 2025-01-28T14:47:50.266Z [ERROR] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: failed to get log: index=3 error="log not found" 2025-01-28T14:47:50.266Z [INFO] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: aborting pipeline replication: peer="{Voter server-b8fb1426-7437-8a41-6fb0-6533fd60b357 b8fb1426-7437-8a41-6fb0-6533fd60b357}" 2025-01-28T14:47:50.266Z [ERROR] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: failed to get log: index=3 error="log not found" 2025-01-28T14:47:50.266Z [INFO] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: aborting pipeline replication: peer="{Voter server-5bd05750-583d-16dc-cbab-1bc2a5fc8d03 5bd05750-583d-16dc-cbab-1bc2a5fc8d03}" 2025-01-28T14:47:50.273Z [ERROR] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: failed to get log: index=3 error="log not found" 2025-01-28T14:47:50.275Z [INFO] creating new snapshot: path=/tmp/raft1006998353/snapshots/2-15-1738075670275.tmp 2025-01-28T14:47:50.276Z [ERROR] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: failed to get log: index=3 error="log not found" 2025-01-28T14:47:50.277Z [INFO] server-5bd05750-583d-16dc-cbab-1bc2a5fc8d03: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:50.280Z [INFO] creating new snapshot: path=/tmp/raft3734439899/snapshots/2-15-1738075670280.tmp 2025-01-28T14:47:50.283Z [INFO] server-5bd05750-583d-16dc-cbab-1bc2a5fc8d03: copied to local snapshot: bytes=71 2025-01-28T14:47:50.283Z [INFO] server-b8fb1426-7437-8a41-6fb0-6533fd60b357: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:50.287Z [INFO] server-5bd05750-583d-16dc-cbab-1bc2a5fc8d03: snapshot restore progress: id=2-15-1738075670275 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:50.287Z [INFO] server-5bd05750-583d-16dc-cbab-1bc2a5fc8d03: Installed remote snapshot 2025-01-28T14:47:50.288Z [INFO] server-b8fb1426-7437-8a41-6fb0-6533fd60b357: copied to local snapshot: bytes=71 2025-01-28T14:47:50.289Z [INFO] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: pipelining replication: peer="{Voter server-5bd05750-583d-16dc-cbab-1bc2a5fc8d03 5bd05750-583d-16dc-cbab-1bc2a5fc8d03}" 2025-01-28T14:47:50.297Z [INFO] server-b8fb1426-7437-8a41-6fb0-6533fd60b357: snapshot restore progress: id=2-15-1738075670280 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:50.299Z [INFO] server-b8fb1426-7437-8a41-6fb0-6533fd60b357: Installed remote snapshot 2025-01-28T14:47:50.299Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:50.302Z [INFO] server-bb52452f-f1be-9b4d-6a01-ebc8ba97cabd: pipelining replication: peer="{Voter server-b8fb1426-7437-8a41-6fb0-6533fd60b357 b8fb1426-7437-8a41-6fb0-6533fd60b357}" 2025-01-28T14:47:51.306Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:50.299769917 +0000 UTC m=+53.383174941 (2 nodes), 10.667µs from start of poll, 1.576356139s from cluster start. Timeout at 2025-01-28 14:47:51.305112261 +0000 UTC m=+54.388518618, 1.005343677s after stability 2025-01-28T14:47:51.326Z [INFO] server-91ed2ad3-61f0-e889-7647-ffe91f847860: aborting pipeline replication: peer="{Voter server-5f3ea835-045e-5e32-edb3-d81c30891701 5f3ea835-045e-5e32-edb3-d81c30891701}" 2025-01-28T14:47:51.326Z [INFO] server-91ed2ad3-61f0-e889-7647-ffe91f847860: aborting pipeline replication: peer="{Voter server-b38473ca-6276-4d1b-e145-f87a31535893 b38473ca-6276-4d1b-e145-f87a31535893}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true 2025-01-28T14:47:51.345Z [INFO] server-36cc9eee-2f39-55d1-243e-166dafc379e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36cc9eee-2f39-55d1-243e-166dafc379e0 Address:36cc9eee-2f39-55d1-243e-166dafc379e0} {Suffrage:Voter ID:server-c0fbd62f-fc1d-1366-1ad4-3edece012848 Address:c0fbd62f-fc1d-1366-1ad4-3edece012848} {Suffrage:Voter ID:server-1e0b3e56-ce15-a30a-33d5-5d1af16656e0 Address:1e0b3e56-ce15-a30a-33d5-5d1af16656e0}]" 2025-01-28T14:47:51.348Z [INFO] server-c0fbd62f-fc1d-1366-1ad4-3edece012848: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36cc9eee-2f39-55d1-243e-166dafc379e0 Address:36cc9eee-2f39-55d1-243e-166dafc379e0} {Suffrage:Voter ID:server-c0fbd62f-fc1d-1366-1ad4-3edece012848 Address:c0fbd62f-fc1d-1366-1ad4-3edece012848} {Suffrage:Voter ID:server-1e0b3e56-ce15-a30a-33d5-5d1af16656e0 Address:1e0b3e56-ce15-a30a-33d5-5d1af16656e0}]" 2025-01-28T14:47:51.353Z [INFO] server-36cc9eee-2f39-55d1-243e-166dafc379e0: entering follower state: follower="Node at 36cc9eee-2f39-55d1-243e-166dafc379e0 [Follower]" leader-address= leader-id= 2025-01-28T14:47:51.355Z [INFO] server-c0fbd62f-fc1d-1366-1ad4-3edece012848: entering follower state: follower="Node at c0fbd62f-fc1d-1366-1ad4-3edece012848 [Follower]" leader-address= leader-id= 2025-01-28T14:47:51.349Z [INFO] server-1e0b3e56-ce15-a30a-33d5-5d1af16656e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36cc9eee-2f39-55d1-243e-166dafc379e0 Address:36cc9eee-2f39-55d1-243e-166dafc379e0} {Suffrage:Voter ID:server-c0fbd62f-fc1d-1366-1ad4-3edece012848 Address:c0fbd62f-fc1d-1366-1ad4-3edece012848} {Suffrage:Voter ID:server-1e0b3e56-ce15-a30a-33d5-5d1af16656e0 Address:1e0b3e56-ce15-a30a-33d5-5d1af16656e0}]" 2025-01-28T14:47:51.356Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:51.356Z [INFO] server-1e0b3e56-ce15-a30a-33d5-5d1af16656e0: entering follower state: follower="Node at 1e0b3e56-ce15-a30a-33d5-5d1af16656e0 [Follower]" leader-address= leader-id= 2025-01-28T14:47:51.884Z [WARN] server-36cc9eee-2f39-55d1-243e-166dafc379e0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:51.885Z [INFO] server-36cc9eee-2f39-55d1-243e-166dafc379e0: entering candidate state: node="Node at 36cc9eee-2f39-55d1-243e-166dafc379e0 [Candidate]" term=2 2025-01-28T14:47:51.890Z [INFO] server-36cc9eee-2f39-55d1-243e-166dafc379e0: election won: term=2 tally=2 2025-01-28T14:47:51.891Z [INFO] server-36cc9eee-2f39-55d1-243e-166dafc379e0: entering leader state: leader="Node at 36cc9eee-2f39-55d1-243e-166dafc379e0 [Leader]" 2025-01-28T14:47:51.891Z [INFO] server-36cc9eee-2f39-55d1-243e-166dafc379e0: added peer, starting replication: peer=server-c0fbd62f-fc1d-1366-1ad4-3edece012848 2025-01-28T14:47:51.891Z [INFO] server-36cc9eee-2f39-55d1-243e-166dafc379e0: added peer, starting replication: peer=server-1e0b3e56-ce15-a30a-33d5-5d1af16656e0 2025-01-28T14:47:51.893Z [INFO] server-36cc9eee-2f39-55d1-243e-166dafc379e0: pipelining replication: peer="{Voter server-1e0b3e56-ce15-a30a-33d5-5d1af16656e0 1e0b3e56-ce15-a30a-33d5-5d1af16656e0}" 2025-01-28T14:47:51.895Z [INFO] server-36cc9eee-2f39-55d1-243e-166dafc379e0: pipelining replication: peer="{Voter server-c0fbd62f-fc1d-1366-1ad4-3edece012848 c0fbd62f-fc1d-1366-1ad4-3edece012848}" 2025-01-28T14:47:52.898Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:51.893155896 +0000 UTC m=+54.976562253 (1 nodes), 536.505263ms from start of poll, 548.237945ms from cluster start. Timeout at 2025-01-28 14:47:52.898178563 +0000 UTC m=+55.981583588, 1.005021335s after stability 2025-01-28T14:47:52.902Z [INFO] server-36cc9eee-2f39-55d1-243e-166dafc379e0: starting snapshot up to: index=12 2025-01-28T14:47:52.902Z [INFO] creating new snapshot: path=/tmp/raft3564985301/snapshots/2-12-1738075672902.tmp 2025-01-28T14:47:52.907Z [INFO] server-36cc9eee-2f39-55d1-243e-166dafc379e0: snapshot complete up to: index=12 2025-01-28T14:47:52.918Z [INFO] server-60ac5e63-3a0b-93e9-9fef-7d5db297c0ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60ac5e63-3a0b-93e9-9fef-7d5db297c0ff Address:60ac5e63-3a0b-93e9-9fef-7d5db297c0ff} {Suffrage:Voter ID:server-b635cf67-4438-7efa-f573-372b1284a9ac Address:b635cf67-4438-7efa-f573-372b1284a9ac} {Suffrage:Voter ID:server-ac05d0aa-5e14-5de7-1638-0049e764b418 Address:ac05d0aa-5e14-5de7-1638-0049e764b418}]" 2025-01-28T14:47:52.918Z [INFO] server-60ac5e63-3a0b-93e9-9fef-7d5db297c0ff: entering follower state: follower="Node at 60ac5e63-3a0b-93e9-9fef-7d5db297c0ff [Follower]" leader-address= leader-id= 2025-01-28T14:47:52.919Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60ac5e63-3a0b-93e9-9fef-7d5db297c0ff Address:60ac5e63-3a0b-93e9-9fef-7d5db297c0ff} {Suffrage:Voter ID:server-b635cf67-4438-7efa-f573-372b1284a9ac Address:b635cf67-4438-7efa-f573-372b1284a9ac} {Suffrage:Voter ID:server-ac05d0aa-5e14-5de7-1638-0049e764b418 Address:ac05d0aa-5e14-5de7-1638-0049e764b418}]" 2025-01-28T14:47:52.919Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: entering follower state: follower="Node at b635cf67-4438-7efa-f573-372b1284a9ac [Follower]" leader-address= leader-id= 2025-01-28T14:47:52.920Z [INFO] server-ac05d0aa-5e14-5de7-1638-0049e764b418: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60ac5e63-3a0b-93e9-9fef-7d5db297c0ff Address:60ac5e63-3a0b-93e9-9fef-7d5db297c0ff} {Suffrage:Voter ID:server-b635cf67-4438-7efa-f573-372b1284a9ac Address:b635cf67-4438-7efa-f573-372b1284a9ac} {Suffrage:Voter ID:server-ac05d0aa-5e14-5de7-1638-0049e764b418 Address:ac05d0aa-5e14-5de7-1638-0049e764b418}]" 2025-01-28T14:47:52.920Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:52.920Z [INFO] server-ac05d0aa-5e14-5de7-1638-0049e764b418: entering follower state: follower="Node at ac05d0aa-5e14-5de7-1638-0049e764b418 [Follower]" leader-address= leader-id= 2025-01-28T14:47:53.438Z [WARN] server-b635cf67-4438-7efa-f573-372b1284a9ac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:53.438Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: entering candidate state: node="Node at b635cf67-4438-7efa-f573-372b1284a9ac [Candidate]" term=2 2025-01-28T14:47:53.439Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: election won: term=2 tally=2 2025-01-28T14:47:53.439Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: entering leader state: leader="Node at b635cf67-4438-7efa-f573-372b1284a9ac [Leader]" 2025-01-28T14:47:53.439Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: added peer, starting replication: peer=server-60ac5e63-3a0b-93e9-9fef-7d5db297c0ff 2025-01-28T14:47:53.442Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: added peer, starting replication: peer=server-ac05d0aa-5e14-5de7-1638-0049e764b418 2025-01-28T14:47:53.456Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: pipelining replication: peer="{Voter server-60ac5e63-3a0b-93e9-9fef-7d5db297c0ff 60ac5e63-3a0b-93e9-9fef-7d5db297c0ff}" 2025-01-28T14:47:53.458Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: pipelining replication: peer="{Voter server-ac05d0aa-5e14-5de7-1638-0049e764b418 ac05d0aa-5e14-5de7-1638-0049e764b418}" 2025-01-28T14:47:54.445Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:53.439864822 +0000 UTC m=+56.523269846 (1 nodes), 519.498759ms from start of poll, 522.230506ms from cluster start. Timeout at 2025-01-28 14:47:54.444877489 +0000 UTC m=+57.528282513, 1.005012667s after stability 2025-01-28T14:47:54.446Z [INFO] creating new snapshot: path=/tmp/raft1104535091/snapshots/2-15-1738075674446.tmp 2025-01-28T14:47:54.451Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: copied to local snapshot: bytes=71 2025-01-28T14:47:54.452Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: snapshot restore progress: id=2-15-1738075674446 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:54.453Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: removing all old logs from log store 2025-01-28T14:47:54.453Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: compacting logs: from=1 to=2 2025-01-28T14:47:54.453Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: restored user snapshot: index=15 2025-01-28T14:47:54.453Z [ERROR] server-b635cf67-4438-7efa-f573-372b1284a9ac: failed to get log: index=2 error="log not found" 2025-01-28T14:47:54.453Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: aborting pipeline replication: peer="{Voter server-ac05d0aa-5e14-5de7-1638-0049e764b418 ac05d0aa-5e14-5de7-1638-0049e764b418}" 2025-01-28T14:47:54.453Z [ERROR] server-b635cf67-4438-7efa-f573-372b1284a9ac: failed to get log: index=2 error="log not found" 2025-01-28T14:47:54.454Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: aborting pipeline replication: peer="{Voter server-60ac5e63-3a0b-93e9-9fef-7d5db297c0ff 60ac5e63-3a0b-93e9-9fef-7d5db297c0ff}" 2025-01-28T14:47:54.460Z [ERROR] server-b635cf67-4438-7efa-f573-372b1284a9ac: failed to get log: index=2 error="log not found" 2025-01-28T14:47:54.462Z [ERROR] server-b635cf67-4438-7efa-f573-372b1284a9ac: failed to get log: index=2 error="log not found" 2025-01-28T14:47:54.462Z [INFO] creating new snapshot: path=/tmp/raft2247567764/snapshots/2-15-1738075674462.tmp 2025-01-28T14:47:54.468Z [INFO] server-60ac5e63-3a0b-93e9-9fef-7d5db297c0ff: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:54.474Z [INFO] creating new snapshot: path=/tmp/raft1343579901/snapshots/2-15-1738075674474.tmp 2025-01-28T14:47:54.477Z [INFO] server-60ac5e63-3a0b-93e9-9fef-7d5db297c0ff: copied to local snapshot: bytes=71 2025-01-28T14:47:54.477Z [INFO] server-ac05d0aa-5e14-5de7-1638-0049e764b418: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:54.479Z [INFO] server-60ac5e63-3a0b-93e9-9fef-7d5db297c0ff: snapshot restore progress: id=2-15-1738075674462 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:54.479Z [INFO] server-60ac5e63-3a0b-93e9-9fef-7d5db297c0ff: removing all old logs from log store 2025-01-28T14:47:54.479Z [INFO] server-60ac5e63-3a0b-93e9-9fef-7d5db297c0ff: compacting logs: from=1 to=2 2025-01-28T14:47:54.480Z [INFO] server-60ac5e63-3a0b-93e9-9fef-7d5db297c0ff: Installed remote snapshot 2025-01-28T14:47:54.480Z [INFO] server-ac05d0aa-5e14-5de7-1638-0049e764b418: copied to local snapshot: bytes=71 2025-01-28T14:47:54.481Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:54.480Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: pipelining replication: peer="{Voter server-60ac5e63-3a0b-93e9-9fef-7d5db297c0ff 60ac5e63-3a0b-93e9-9fef-7d5db297c0ff}" 2025-01-28T14:47:54.482Z [INFO] server-ac05d0aa-5e14-5de7-1638-0049e764b418: snapshot restore progress: id=2-15-1738075674474 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:54.482Z [INFO] server-ac05d0aa-5e14-5de7-1638-0049e764b418: removing all old logs from log store 2025-01-28T14:47:54.482Z [INFO] server-ac05d0aa-5e14-5de7-1638-0049e764b418: compacting logs: from=1 to=2 2025-01-28T14:47:54.482Z [INFO] server-ac05d0aa-5e14-5de7-1638-0049e764b418: Installed remote snapshot 2025-01-28T14:47:54.483Z [INFO] server-b635cf67-4438-7efa-f573-372b1284a9ac: pipelining replication: peer="{Voter server-ac05d0aa-5e14-5de7-1638-0049e764b418 ac05d0aa-5e14-5de7-1638-0049e764b418}" 2025-01-28T14:47:55.486Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:54.481243077 +0000 UTC m=+57.564647768 (2 nodes), 7.334µs from start of poll, 1.563608428s from cluster start. Timeout at 2025-01-28 14:47:55.486488417 +0000 UTC m=+58.569893442, 1.005245674s after stability 2025-01-28T14:47:55.503Z [INFO] server-36cc9eee-2f39-55d1-243e-166dafc379e0: aborting pipeline replication: peer="{Voter server-c0fbd62f-fc1d-1366-1ad4-3edece012848 c0fbd62f-fc1d-1366-1ad4-3edece012848}" 2025-01-28T14:47:55.503Z [INFO] server-36cc9eee-2f39-55d1-243e-166dafc379e0: aborting pipeline replication: peer="{Voter server-1e0b3e56-ce15-a30a-33d5-5d1af16656e0 1e0b3e56-ce15-a30a-33d5-5d1af16656e0}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false 2025-01-28T14:47:55.523Z [INFO] server-99072298-1ad9-33df-1284-b10c1061da78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-99072298-1ad9-33df-1284-b10c1061da78 Address:99072298-1ad9-33df-1284-b10c1061da78} {Suffrage:Voter ID:server-527ea787-4f72-17a2-fc7e-7b8b74974f56 Address:527ea787-4f72-17a2-fc7e-7b8b74974f56} {Suffrage:Voter ID:server-ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3 Address:ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3}]" 2025-01-28T14:47:55.523Z [INFO] server-99072298-1ad9-33df-1284-b10c1061da78: entering follower state: follower="Node at 99072298-1ad9-33df-1284-b10c1061da78 [Follower]" leader-address= leader-id= 2025-01-28T14:47:55.523Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-99072298-1ad9-33df-1284-b10c1061da78 Address:99072298-1ad9-33df-1284-b10c1061da78} {Suffrage:Voter ID:server-527ea787-4f72-17a2-fc7e-7b8b74974f56 Address:527ea787-4f72-17a2-fc7e-7b8b74974f56} {Suffrage:Voter ID:server-ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3 Address:ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3}]" 2025-01-28T14:47:55.524Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: entering follower state: follower="Node at 527ea787-4f72-17a2-fc7e-7b8b74974f56 [Follower]" leader-address= leader-id= 2025-01-28T14:47:55.524Z [INFO] server-ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-99072298-1ad9-33df-1284-b10c1061da78 Address:99072298-1ad9-33df-1284-b10c1061da78} {Suffrage:Voter ID:server-527ea787-4f72-17a2-fc7e-7b8b74974f56 Address:527ea787-4f72-17a2-fc7e-7b8b74974f56} {Suffrage:Voter ID:server-ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3 Address:ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3}]" 2025-01-28T14:47:55.524Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:55.524Z [INFO] server-ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3: entering follower state: follower="Node at ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3 [Follower]" leader-address= leader-id= 2025-01-28T14:47:56.075Z [WARN] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:56.075Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: entering candidate state: node="Node at 527ea787-4f72-17a2-fc7e-7b8b74974f56 [Candidate]" term=2 2025-01-28T14:47:56.076Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: election won: term=2 tally=2 2025-01-28T14:47:56.076Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: entering leader state: leader="Node at 527ea787-4f72-17a2-fc7e-7b8b74974f56 [Leader]" 2025-01-28T14:47:56.076Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: added peer, starting replication: peer=server-99072298-1ad9-33df-1284-b10c1061da78 2025-01-28T14:47:56.076Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: added peer, starting replication: peer=server-ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3 2025-01-28T14:47:56.077Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: pipelining replication: peer="{Voter server-99072298-1ad9-33df-1284-b10c1061da78 99072298-1ad9-33df-1284-b10c1061da78}" 2025-01-28T14:47:56.077Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: pipelining replication: peer="{Voter server-ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3 ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3}" 2025-01-28T14:47:57.082Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:56.076457444 +0000 UTC m=+59.159862468 (1 nodes), 551.515376ms from start of poll, 554.153121ms from cluster start. Timeout at 2025-01-28 14:47:57.081470111 +0000 UTC m=+60.164874802, 1.005012334s after stability 2025-01-28T14:47:57.086Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: starting snapshot up to: index=12 2025-01-28T14:47:57.087Z [INFO] creating new snapshot: path=/tmp/raft3769061558/snapshots/2-12-1738075677087.tmp 2025-01-28T14:47:57.090Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: snapshot complete up to: index=12 2025-01-28T14:47:57.101Z [INFO] creating new snapshot: path=/tmp/raft3769061558/snapshots/2-113-1738075677101.tmp 2025-01-28T14:47:57.106Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: copied to local snapshot: bytes=71 2025-01-28T14:47:57.108Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: snapshot restore progress: id=2-113-1738075677101 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:57.108Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: restored user snapshot: index=113 2025-01-28T14:47:57.108Z [ERROR] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: failed to get log: index=23 error="log not found" 2025-01-28T14:47:57.108Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: aborting pipeline replication: peer="{Voter server-99072298-1ad9-33df-1284-b10c1061da78 99072298-1ad9-33df-1284-b10c1061da78}" 2025-01-28T14:47:57.108Z [ERROR] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: failed to get log: index=23 error="log not found" 2025-01-28T14:47:57.109Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: aborting pipeline replication: peer="{Voter server-ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3 ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3}" 2025-01-28T14:47:57.118Z [ERROR] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: failed to get log: index=23 error="log not found" 2025-01-28T14:47:57.119Z [ERROR] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: failed to get log: index=23 error="log not found" 2025-01-28T14:47:57.136Z [INFO] creating new snapshot: path=/tmp/raft1917326730/snapshots/2-113-1738075677136.tmp 2025-01-28T14:47:57.139Z [INFO] server-99072298-1ad9-33df-1284-b10c1061da78: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:57.140Z [INFO] creating new snapshot: path=/tmp/raft4064220794/snapshots/2-113-1738075677140.tmp 2025-01-28T14:47:57.142Z [INFO] server-ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:57.142Z [INFO] server-99072298-1ad9-33df-1284-b10c1061da78: copied to local snapshot: bytes=71 2025-01-28T14:47:57.144Z [INFO] server-99072298-1ad9-33df-1284-b10c1061da78: snapshot restore progress: id=2-113-1738075677136 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:57.144Z [INFO] server-99072298-1ad9-33df-1284-b10c1061da78: Installed remote snapshot 2025-01-28T14:47:57.144Z [INFO] server-ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3: copied to local snapshot: bytes=71 2025-01-28T14:47:57.145Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: pipelining replication: peer="{Voter server-99072298-1ad9-33df-1284-b10c1061da78 99072298-1ad9-33df-1284-b10c1061da78}" 2025-01-28T14:47:57.146Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:57.146Z [INFO] server-ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3: snapshot restore progress: id=2-113-1738075677140 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:57.146Z [INFO] server-ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3: Installed remote snapshot 2025-01-28T14:47:57.147Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: pipelining replication: peer="{Voter server-ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3 ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3}" 2025-01-28T14:47:58.152Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:57.146333545 +0000 UTC m=+60.229738236 (2 nodes), 8µs from start of poll, 1.624028889s from cluster start. Timeout at 2025-01-28 14:47:58.151634221 +0000 UTC m=+61.235040579, 1.005302343s after stability 2025-01-28T14:47:58.169Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: aborting pipeline replication: peer="{Voter server-99072298-1ad9-33df-1284-b10c1061da78 99072298-1ad9-33df-1284-b10c1061da78}" 2025-01-28T14:47:58.169Z [INFO] server-527ea787-4f72-17a2-fc7e-7b8b74974f56: aborting pipeline replication: peer="{Voter server-ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3 ac7419ce-610e-ddda-e2b7-9e2f5abb5bd3}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false 2025-01-28T14:47:58.199Z [INFO] server-ee8041f9-8ff0-6c9a-7e70-1e3a6d037379: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee8041f9-8ff0-6c9a-7e70-1e3a6d037379 Address:ee8041f9-8ff0-6c9a-7e70-1e3a6d037379} {Suffrage:Voter ID:server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8 Address:f1f21a48-a3f9-eca1-2244-ac9fa37313f8} {Suffrage:Voter ID:server-35185b9a-5a01-5523-f0ef-6182e329818a Address:35185b9a-5a01-5523-f0ef-6182e329818a}]" 2025-01-28T14:47:58.199Z [INFO] server-ee8041f9-8ff0-6c9a-7e70-1e3a6d037379: entering follower state: follower="Node at ee8041f9-8ff0-6c9a-7e70-1e3a6d037379 [Follower]" leader-address= leader-id= 2025-01-28T14:47:58.200Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee8041f9-8ff0-6c9a-7e70-1e3a6d037379 Address:ee8041f9-8ff0-6c9a-7e70-1e3a6d037379} {Suffrage:Voter ID:server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8 Address:f1f21a48-a3f9-eca1-2244-ac9fa37313f8} {Suffrage:Voter ID:server-35185b9a-5a01-5523-f0ef-6182e329818a Address:35185b9a-5a01-5523-f0ef-6182e329818a}]" 2025-01-28T14:47:58.201Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: entering follower state: follower="Node at f1f21a48-a3f9-eca1-2244-ac9fa37313f8 [Follower]" leader-address= leader-id= 2025-01-28T14:47:58.201Z [INFO] server-35185b9a-5a01-5523-f0ef-6182e329818a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee8041f9-8ff0-6c9a-7e70-1e3a6d037379 Address:ee8041f9-8ff0-6c9a-7e70-1e3a6d037379} {Suffrage:Voter ID:server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8 Address:f1f21a48-a3f9-eca1-2244-ac9fa37313f8} {Suffrage:Voter ID:server-35185b9a-5a01-5523-f0ef-6182e329818a Address:35185b9a-5a01-5523-f0ef-6182e329818a}]" 2025-01-28T14:47:58.201Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:47:58.201Z [INFO] server-35185b9a-5a01-5523-f0ef-6182e329818a: entering follower state: follower="Node at 35185b9a-5a01-5523-f0ef-6182e329818a [Follower]" leader-address= leader-id= 2025-01-28T14:47:58.779Z [WARN] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:47:58.779Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: entering candidate state: node="Node at f1f21a48-a3f9-eca1-2244-ac9fa37313f8 [Candidate]" term=2 2025-01-28T14:47:58.780Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: election won: term=2 tally=2 2025-01-28T14:47:58.780Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: entering leader state: leader="Node at f1f21a48-a3f9-eca1-2244-ac9fa37313f8 [Leader]" 2025-01-28T14:47:58.780Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: added peer, starting replication: peer=server-ee8041f9-8ff0-6c9a-7e70-1e3a6d037379 2025-01-28T14:47:58.781Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: added peer, starting replication: peer=server-35185b9a-5a01-5523-f0ef-6182e329818a 2025-01-28T14:47:58.785Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: pipelining replication: peer="{Voter server-ee8041f9-8ff0-6c9a-7e70-1e3a6d037379 ee8041f9-8ff0-6c9a-7e70-1e3a6d037379}" 2025-01-28T14:47:58.787Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: pipelining replication: peer="{Voter server-35185b9a-5a01-5523-f0ef-6182e329818a 35185b9a-5a01-5523-f0ef-6182e329818a}" 2025-01-28T14:47:59.787Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:47:58.78120209 +0000 UTC m=+61.864607114 (1 nodes), 579.392871ms from start of poll, 582.360625ms from cluster start. Timeout at 2025-01-28 14:47:59.786222091 +0000 UTC m=+62.869626782, 1.005019668s after stability 2025-01-28T14:47:59.793Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: starting snapshot up to: index=12 2025-01-28T14:47:59.793Z [INFO] creating new snapshot: path=/tmp/raft3157924745/snapshots/2-12-1738075679793.tmp 2025-01-28T14:47:59.797Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: snapshot complete up to: index=12 2025-01-28T14:47:59.804Z [INFO] creating new snapshot: path=/tmp/raft3157924745/snapshots/2-113-1738075679804.tmp 2025-01-28T14:47:59.808Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: copied to local snapshot: bytes=71 2025-01-28T14:47:59.810Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: snapshot restore progress: id=2-113-1738075679804 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:59.810Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: removing all old logs from log store 2025-01-28T14:47:59.810Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: compacting logs: from=1 to=22 2025-01-28T14:47:59.810Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: restored user snapshot: index=113 2025-01-28T14:47:59.810Z [ERROR] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: failed to get log: index=22 error="log not found" 2025-01-28T14:47:59.810Z [ERROR] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: failed to get log: index=22 error="log not found" 2025-01-28T14:47:59.810Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: aborting pipeline replication: peer="{Voter server-ee8041f9-8ff0-6c9a-7e70-1e3a6d037379 ee8041f9-8ff0-6c9a-7e70-1e3a6d037379}" 2025-01-28T14:47:59.810Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: aborting pipeline replication: peer="{Voter server-35185b9a-5a01-5523-f0ef-6182e329818a 35185b9a-5a01-5523-f0ef-6182e329818a}" 2025-01-28T14:47:59.818Z [ERROR] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: failed to get log: index=22 error="log not found" 2025-01-28T14:47:59.819Z [ERROR] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: failed to get log: index=22 error="log not found" 2025-01-28T14:47:59.823Z [INFO] creating new snapshot: path=/tmp/raft3670346112/snapshots/2-113-1738075679823.tmp 2025-01-28T14:47:59.829Z [INFO] creating new snapshot: path=/tmp/raft1653024333/snapshots/2-113-1738075679829.tmp 2025-01-28T14:47:59.836Z [INFO] server-35185b9a-5a01-5523-f0ef-6182e329818a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:59.839Z [INFO] server-35185b9a-5a01-5523-f0ef-6182e329818a: copied to local snapshot: bytes=71 2025-01-28T14:47:59.841Z [INFO] server-35185b9a-5a01-5523-f0ef-6182e329818a: snapshot restore progress: id=2-113-1738075679823 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:59.841Z [INFO] server-35185b9a-5a01-5523-f0ef-6182e329818a: removing all old logs from log store 2025-01-28T14:47:59.841Z [INFO] server-35185b9a-5a01-5523-f0ef-6182e329818a: compacting logs: from=1 to=22 2025-01-28T14:47:59.841Z [INFO] server-35185b9a-5a01-5523-f0ef-6182e329818a: Installed remote snapshot 2025-01-28T14:47:59.842Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: pipelining replication: peer="{Voter server-35185b9a-5a01-5523-f0ef-6182e329818a 35185b9a-5a01-5523-f0ef-6182e329818a}" 2025-01-28T14:47:59.843Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:47:59.847Z [INFO] server-ee8041f9-8ff0-6c9a-7e70-1e3a6d037379: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:59.849Z [INFO] server-ee8041f9-8ff0-6c9a-7e70-1e3a6d037379: copied to local snapshot: bytes=71 2025-01-28T14:47:59.852Z [INFO] server-ee8041f9-8ff0-6c9a-7e70-1e3a6d037379: snapshot restore progress: id=2-113-1738075679829 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:47:59.852Z [INFO] server-ee8041f9-8ff0-6c9a-7e70-1e3a6d037379: removing all old logs from log store 2025-01-28T14:47:59.852Z [INFO] server-ee8041f9-8ff0-6c9a-7e70-1e3a6d037379: compacting logs: from=1 to=22 2025-01-28T14:47:59.852Z [INFO] server-ee8041f9-8ff0-6c9a-7e70-1e3a6d037379: Installed remote snapshot 2025-01-28T14:47:59.853Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: pipelining replication: peer="{Voter server-ee8041f9-8ff0-6c9a-7e70-1e3a6d037379 ee8041f9-8ff0-6c9a-7e70-1e3a6d037379}" 2025-01-28T14:48:00.848Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:47:59.843199624 +0000 UTC m=+62.926604315 (2 nodes), 6.666µs from start of poll, 1.644357826s from cluster start. Timeout at 2025-01-28 14:48:00.848474632 +0000 UTC m=+63.931879657, 1.005275342s after stability 2025-01-28T14:48:00.867Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: aborting pipeline replication: peer="{Voter server-35185b9a-5a01-5523-f0ef-6182e329818a 35185b9a-5a01-5523-f0ef-6182e329818a}" 2025-01-28T14:48:00.867Z [INFO] server-f1f21a48-a3f9-eca1-2244-ac9fa37313f8: aborting pipeline replication: peer="{Voter server-ee8041f9-8ff0-6c9a-7e70-1e3a6d037379 ee8041f9-8ff0-6c9a-7e70-1e3a6d037379}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true 2025-01-28T14:48:00.879Z [INFO] server-1cf17c00-771a-8084-cb31-ef3934ccad69: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cf17c00-771a-8084-cb31-ef3934ccad69 Address:1cf17c00-771a-8084-cb31-ef3934ccad69} {Suffrage:Voter ID:server-3068c999-712b-9dd4-25d1-ad785e71f54a Address:3068c999-712b-9dd4-25d1-ad785e71f54a} {Suffrage:Voter ID:server-91815518-0e58-36e5-ac77-c6d606e0eec1 Address:91815518-0e58-36e5-ac77-c6d606e0eec1}]" 2025-01-28T14:48:00.880Z [INFO] server-1cf17c00-771a-8084-cb31-ef3934ccad69: entering follower state: follower="Node at 1cf17c00-771a-8084-cb31-ef3934ccad69 [Follower]" leader-address= leader-id= 2025-01-28T14:48:00.880Z [INFO] server-3068c999-712b-9dd4-25d1-ad785e71f54a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cf17c00-771a-8084-cb31-ef3934ccad69 Address:1cf17c00-771a-8084-cb31-ef3934ccad69} {Suffrage:Voter ID:server-3068c999-712b-9dd4-25d1-ad785e71f54a Address:3068c999-712b-9dd4-25d1-ad785e71f54a} {Suffrage:Voter ID:server-91815518-0e58-36e5-ac77-c6d606e0eec1 Address:91815518-0e58-36e5-ac77-c6d606e0eec1}]" 2025-01-28T14:48:00.881Z [INFO] server-91815518-0e58-36e5-ac77-c6d606e0eec1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cf17c00-771a-8084-cb31-ef3934ccad69 Address:1cf17c00-771a-8084-cb31-ef3934ccad69} {Suffrage:Voter ID:server-3068c999-712b-9dd4-25d1-ad785e71f54a Address:3068c999-712b-9dd4-25d1-ad785e71f54a} {Suffrage:Voter ID:server-91815518-0e58-36e5-ac77-c6d606e0eec1 Address:91815518-0e58-36e5-ac77-c6d606e0eec1}]" 2025-01-28T14:48:00.881Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:00.881Z [INFO] server-3068c999-712b-9dd4-25d1-ad785e71f54a: entering follower state: follower="Node at 3068c999-712b-9dd4-25d1-ad785e71f54a [Follower]" leader-address= leader-id= 2025-01-28T14:48:00.881Z [INFO] server-91815518-0e58-36e5-ac77-c6d606e0eec1: entering follower state: follower="Node at 91815518-0e58-36e5-ac77-c6d606e0eec1 [Follower]" leader-address= leader-id= 2025-01-28T14:48:01.414Z [WARN] server-91815518-0e58-36e5-ac77-c6d606e0eec1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:01.414Z [INFO] server-91815518-0e58-36e5-ac77-c6d606e0eec1: entering candidate state: node="Node at 91815518-0e58-36e5-ac77-c6d606e0eec1 [Candidate]" term=2 2025-01-28T14:48:01.415Z [INFO] server-91815518-0e58-36e5-ac77-c6d606e0eec1: election won: term=2 tally=2 2025-01-28T14:48:01.415Z [INFO] server-91815518-0e58-36e5-ac77-c6d606e0eec1: entering leader state: leader="Node at 91815518-0e58-36e5-ac77-c6d606e0eec1 [Leader]" 2025-01-28T14:48:01.415Z [INFO] server-91815518-0e58-36e5-ac77-c6d606e0eec1: added peer, starting replication: peer=server-1cf17c00-771a-8084-cb31-ef3934ccad69 2025-01-28T14:48:01.415Z [INFO] server-91815518-0e58-36e5-ac77-c6d606e0eec1: added peer, starting replication: peer=server-3068c999-712b-9dd4-25d1-ad785e71f54a 2025-01-28T14:48:01.416Z [INFO] server-91815518-0e58-36e5-ac77-c6d606e0eec1: pipelining replication: peer="{Voter server-3068c999-712b-9dd4-25d1-ad785e71f54a 3068c999-712b-9dd4-25d1-ad785e71f54a}" 2025-01-28T14:48:01.417Z [INFO] server-91815518-0e58-36e5-ac77-c6d606e0eec1: pipelining replication: peer="{Voter server-1cf17c00-771a-8084-cb31-ef3934ccad69 1cf17c00-771a-8084-cb31-ef3934ccad69}" 2025-01-28T14:48:02.427Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:01.416402005 +0000 UTC m=+64.499806696 (1 nodes), 534.985885ms from start of poll, 537.103948ms from cluster start. Timeout at 2025-01-28 14:48:02.421487341 +0000 UTC m=+65.504893032, 1.005086336s after stability 2025-01-28T14:48:02.441Z [INFO] server-91815518-0e58-36e5-ac77-c6d606e0eec1: starting snapshot up to: index=12 2025-01-28T14:48:02.441Z [INFO] creating new snapshot: path=/tmp/raft2173949338/snapshots/2-12-1738075682441.tmp 2025-01-28T14:48:02.448Z [INFO] server-91815518-0e58-36e5-ac77-c6d606e0eec1: snapshot complete up to: index=12 2025-01-28T14:48:02.481Z [INFO] server-0ea1d87d-35be-1ad0-59f8-9bc6b48a4081: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ea1d87d-35be-1ad0-59f8-9bc6b48a4081 Address:0ea1d87d-35be-1ad0-59f8-9bc6b48a4081} {Suffrage:Voter ID:server-db5d333b-7afc-fc9b-acb1-82caafa66ae9 Address:db5d333b-7afc-fc9b-acb1-82caafa66ae9} {Suffrage:Voter ID:server-16e15cc4-70b3-6d04-e109-efa7fedf2cfc Address:16e15cc4-70b3-6d04-e109-efa7fedf2cfc}]" 2025-01-28T14:48:02.482Z [INFO] server-0ea1d87d-35be-1ad0-59f8-9bc6b48a4081: entering follower state: follower="Node at 0ea1d87d-35be-1ad0-59f8-9bc6b48a4081 [Follower]" leader-address= leader-id= 2025-01-28T14:48:02.482Z [INFO] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ea1d87d-35be-1ad0-59f8-9bc6b48a4081 Address:0ea1d87d-35be-1ad0-59f8-9bc6b48a4081} {Suffrage:Voter ID:server-db5d333b-7afc-fc9b-acb1-82caafa66ae9 Address:db5d333b-7afc-fc9b-acb1-82caafa66ae9} {Suffrage:Voter ID:server-16e15cc4-70b3-6d04-e109-efa7fedf2cfc Address:16e15cc4-70b3-6d04-e109-efa7fedf2cfc}]" 2025-01-28T14:48:02.482Z [INFO] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: entering follower state: follower="Node at db5d333b-7afc-fc9b-acb1-82caafa66ae9 [Follower]" leader-address= leader-id= 2025-01-28T14:48:02.483Z [INFO] server-16e15cc4-70b3-6d04-e109-efa7fedf2cfc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ea1d87d-35be-1ad0-59f8-9bc6b48a4081 Address:0ea1d87d-35be-1ad0-59f8-9bc6b48a4081} {Suffrage:Voter ID:server-db5d333b-7afc-fc9b-acb1-82caafa66ae9 Address:db5d333b-7afc-fc9b-acb1-82caafa66ae9} {Suffrage:Voter ID:server-16e15cc4-70b3-6d04-e109-efa7fedf2cfc Address:16e15cc4-70b3-6d04-e109-efa7fedf2cfc}]" 2025-01-28T14:48:02.483Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:02.483Z [INFO] server-16e15cc4-70b3-6d04-e109-efa7fedf2cfc: entering follower state: follower="Node at 16e15cc4-70b3-6d04-e109-efa7fedf2cfc [Follower]" leader-address= leader-id= 2025-01-28T14:48:03.047Z [WARN] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:03.048Z [INFO] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: entering candidate state: node="Node at db5d333b-7afc-fc9b-acb1-82caafa66ae9 [Candidate]" term=2 2025-01-28T14:48:03.048Z [INFO] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: election won: term=2 tally=2 2025-01-28T14:48:03.048Z [INFO] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: entering leader state: leader="Node at db5d333b-7afc-fc9b-acb1-82caafa66ae9 [Leader]" 2025-01-28T14:48:03.049Z [INFO] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: added peer, starting replication: peer=server-0ea1d87d-35be-1ad0-59f8-9bc6b48a4081 2025-01-28T14:48:03.049Z [INFO] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: added peer, starting replication: peer=server-16e15cc4-70b3-6d04-e109-efa7fedf2cfc 2025-01-28T14:48:03.050Z [INFO] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: pipelining replication: peer="{Voter server-0ea1d87d-35be-1ad0-59f8-9bc6b48a4081 0ea1d87d-35be-1ad0-59f8-9bc6b48a4081}" 2025-01-28T14:48:03.050Z [INFO] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: pipelining replication: peer="{Voter server-16e15cc4-70b3-6d04-e109-efa7fedf2cfc 16e15cc4-70b3-6d04-e109-efa7fedf2cfc}" 2025-01-28T14:48:04.054Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:03.049086819 +0000 UTC m=+66.132491844 (1 nodes), 565.30312ms from start of poll, 568.768556ms from cluster start. Timeout at 2025-01-28 14:48:04.054106486 +0000 UTC m=+67.137511510, 1.005019666s after stability 2025-01-28T14:48:04.055Z [INFO] creating new snapshot: path=/tmp/raft2155486308/snapshots/2-113-1738075684055.tmp 2025-01-28T14:48:04.060Z [INFO] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: copied to local snapshot: bytes=71 2025-01-28T14:48:04.062Z [INFO] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: snapshot restore progress: id=2-113-1738075684055 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:04.063Z [INFO] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: restored user snapshot: index=113 2025-01-28T14:48:04.063Z [ERROR] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: failed to get log: index=3 error="log not found" 2025-01-28T14:48:04.063Z [ERROR] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: failed to get log: index=3 error="log not found" 2025-01-28T14:48:04.063Z [INFO] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: aborting pipeline replication: peer="{Voter server-0ea1d87d-35be-1ad0-59f8-9bc6b48a4081 0ea1d87d-35be-1ad0-59f8-9bc6b48a4081}" 2025-01-28T14:48:04.063Z [INFO] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: aborting pipeline replication: peer="{Voter server-16e15cc4-70b3-6d04-e109-efa7fedf2cfc 16e15cc4-70b3-6d04-e109-efa7fedf2cfc}" 2025-01-28T14:48:04.063Z [ERROR] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: failed to get log: index=3 error="log not found" 2025-01-28T14:48:04.066Z [INFO] creating new snapshot: path=/tmp/raft2810845969/snapshots/2-113-1738075684066.tmp 2025-01-28T14:48:04.068Z [INFO] server-16e15cc4-70b3-6d04-e109-efa7fedf2cfc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:04.070Z [INFO] server-16e15cc4-70b3-6d04-e109-efa7fedf2cfc: copied to local snapshot: bytes=71 2025-01-28T14:48:04.071Z [ERROR] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: failed to get log: index=3 error="log not found" 2025-01-28T14:48:04.072Z [INFO] server-16e15cc4-70b3-6d04-e109-efa7fedf2cfc: snapshot restore progress: id=2-113-1738075684066 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:04.072Z [INFO] server-16e15cc4-70b3-6d04-e109-efa7fedf2cfc: Installed remote snapshot 2025-01-28T14:48:04.072Z [INFO] creating new snapshot: path=/tmp/raft1581725978/snapshots/2-113-1738075684072.tmp 2025-01-28T14:48:04.074Z [INFO] server-0ea1d87d-35be-1ad0-59f8-9bc6b48a4081: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:04.075Z [INFO] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: pipelining replication: peer="{Voter server-16e15cc4-70b3-6d04-e109-efa7fedf2cfc 16e15cc4-70b3-6d04-e109-efa7fedf2cfc}" 2025-01-28T14:48:04.075Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:04.079Z [INFO] server-0ea1d87d-35be-1ad0-59f8-9bc6b48a4081: copied to local snapshot: bytes=71 2025-01-28T14:48:04.080Z [INFO] server-0ea1d87d-35be-1ad0-59f8-9bc6b48a4081: snapshot restore progress: id=2-113-1738075684072 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:04.081Z [INFO] server-0ea1d87d-35be-1ad0-59f8-9bc6b48a4081: Installed remote snapshot 2025-01-28T14:48:04.081Z [INFO] server-db5d333b-7afc-fc9b-acb1-82caafa66ae9: pipelining replication: peer="{Voter server-0ea1d87d-35be-1ad0-59f8-9bc6b48a4081 0ea1d87d-35be-1ad0-59f8-9bc6b48a4081}" 2025-01-28T14:48:05.081Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:04.075644968 +0000 UTC m=+67.159049660 (2 nodes), 9.001µs from start of poll, 1.595326372s from cluster start. Timeout at 2025-01-28 14:48:05.080822973 +0000 UTC m=+68.164227998, 1.005178338s after stability 2025-01-28T14:48:05.094Z [INFO] server-91815518-0e58-36e5-ac77-c6d606e0eec1: aborting pipeline replication: peer="{Voter server-3068c999-712b-9dd4-25d1-ad785e71f54a 3068c999-712b-9dd4-25d1-ad785e71f54a}" 2025-01-28T14:48:05.094Z [INFO] server-91815518-0e58-36e5-ac77-c6d606e0eec1: aborting pipeline replication: peer="{Voter server-1cf17c00-771a-8084-cb31-ef3934ccad69 1cf17c00-771a-8084-cb31-ef3934ccad69}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true 2025-01-28T14:48:05.104Z [INFO] server-fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c Address:fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c} {Suffrage:Voter ID:server-36d02b3b-2e5c-7d34-c23f-f15f45d4b6aa Address:36d02b3b-2e5c-7d34-c23f-f15f45d4b6aa} {Suffrage:Voter ID:server-4071d816-93ca-339a-f9c1-5f652d35b900 Address:4071d816-93ca-339a-f9c1-5f652d35b900}]" 2025-01-28T14:48:05.105Z [INFO] server-36d02b3b-2e5c-7d34-c23f-f15f45d4b6aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c Address:fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c} {Suffrage:Voter ID:server-36d02b3b-2e5c-7d34-c23f-f15f45d4b6aa Address:36d02b3b-2e5c-7d34-c23f-f15f45d4b6aa} {Suffrage:Voter ID:server-4071d816-93ca-339a-f9c1-5f652d35b900 Address:4071d816-93ca-339a-f9c1-5f652d35b900}]" 2025-01-28T14:48:05.106Z [INFO] server-fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c: entering follower state: follower="Node at fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c [Follower]" leader-address= leader-id= 2025-01-28T14:48:05.106Z [INFO] server-36d02b3b-2e5c-7d34-c23f-f15f45d4b6aa: entering follower state: follower="Node at 36d02b3b-2e5c-7d34-c23f-f15f45d4b6aa [Follower]" leader-address= leader-id= 2025-01-28T14:48:05.107Z [INFO] server-4071d816-93ca-339a-f9c1-5f652d35b900: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c Address:fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c} {Suffrage:Voter ID:server-36d02b3b-2e5c-7d34-c23f-f15f45d4b6aa Address:36d02b3b-2e5c-7d34-c23f-f15f45d4b6aa} {Suffrage:Voter ID:server-4071d816-93ca-339a-f9c1-5f652d35b900 Address:4071d816-93ca-339a-f9c1-5f652d35b900}]" 2025-01-28T14:48:05.107Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:05.107Z [INFO] server-4071d816-93ca-339a-f9c1-5f652d35b900: entering follower state: follower="Node at 4071d816-93ca-339a-f9c1-5f652d35b900 [Follower]" leader-address= leader-id= 2025-01-28T14:48:05.707Z [WARN] server-fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:05.707Z [INFO] server-fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c: entering candidate state: node="Node at fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c [Candidate]" term=2 2025-01-28T14:48:05.708Z [INFO] server-fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c: election won: term=2 tally=2 2025-01-28T14:48:05.708Z [INFO] server-fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c: entering leader state: leader="Node at fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c [Leader]" 2025-01-28T14:48:05.708Z [INFO] server-fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c: added peer, starting replication: peer=server-36d02b3b-2e5c-7d34-c23f-f15f45d4b6aa 2025-01-28T14:48:05.708Z [INFO] server-fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c: added peer, starting replication: peer=server-4071d816-93ca-339a-f9c1-5f652d35b900 2025-01-28T14:48:05.709Z [INFO] server-fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c: pipelining replication: peer="{Voter server-36d02b3b-2e5c-7d34-c23f-f15f45d4b6aa 36d02b3b-2e5c-7d34-c23f-f15f45d4b6aa}" 2025-01-28T14:48:05.711Z [INFO] server-fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c: pipelining replication: peer="{Voter server-4071d816-93ca-339a-f9c1-5f652d35b900 4071d816-93ca-339a-f9c1-5f652d35b900}" 2025-01-28T14:48:06.714Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:05.709077472 +0000 UTC m=+68.792482163 (1 nodes), 601.22852ms from start of poll, 604.642955ms from cluster start. Timeout at 2025-01-28 14:48:06.714102139 +0000 UTC m=+69.797507164, 1.005025001s after stability 2025-01-28T14:48:06.721Z [INFO] server-fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c: starting snapshot up to: index=12 2025-01-28T14:48:06.721Z [INFO] creating new snapshot: path=/tmp/raft715067995/snapshots/2-12-1738075686721.tmp 2025-01-28T14:48:06.725Z [INFO] server-fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c: snapshot complete up to: index=12 2025-01-28T14:48:06.739Z [INFO] server-3c37f80e-90f7-498e-cc2d-503f26d3f261: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c37f80e-90f7-498e-cc2d-503f26d3f261 Address:3c37f80e-90f7-498e-cc2d-503f26d3f261} {Suffrage:Voter ID:server-9515b21a-4ea1-2479-266f-4ed1a16e4eec Address:9515b21a-4ea1-2479-266f-4ed1a16e4eec} {Suffrage:Voter ID:server-80451566-3d16-6fd0-2dca-11a25a538455 Address:80451566-3d16-6fd0-2dca-11a25a538455}]" 2025-01-28T14:48:06.739Z [INFO] server-3c37f80e-90f7-498e-cc2d-503f26d3f261: entering follower state: follower="Node at 3c37f80e-90f7-498e-cc2d-503f26d3f261 [Follower]" leader-address= leader-id= 2025-01-28T14:48:06.739Z [INFO] server-9515b21a-4ea1-2479-266f-4ed1a16e4eec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c37f80e-90f7-498e-cc2d-503f26d3f261 Address:3c37f80e-90f7-498e-cc2d-503f26d3f261} {Suffrage:Voter ID:server-9515b21a-4ea1-2479-266f-4ed1a16e4eec Address:9515b21a-4ea1-2479-266f-4ed1a16e4eec} {Suffrage:Voter ID:server-80451566-3d16-6fd0-2dca-11a25a538455 Address:80451566-3d16-6fd0-2dca-11a25a538455}]" 2025-01-28T14:48:06.740Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c37f80e-90f7-498e-cc2d-503f26d3f261 Address:3c37f80e-90f7-498e-cc2d-503f26d3f261} {Suffrage:Voter ID:server-9515b21a-4ea1-2479-266f-4ed1a16e4eec Address:9515b21a-4ea1-2479-266f-4ed1a16e4eec} {Suffrage:Voter ID:server-80451566-3d16-6fd0-2dca-11a25a538455 Address:80451566-3d16-6fd0-2dca-11a25a538455}]" 2025-01-28T14:48:06.740Z [INFO] server-9515b21a-4ea1-2479-266f-4ed1a16e4eec: entering follower state: follower="Node at 9515b21a-4ea1-2479-266f-4ed1a16e4eec [Follower]" leader-address= leader-id= 2025-01-28T14:48:06.740Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:06.740Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: entering follower state: follower="Node at 80451566-3d16-6fd0-2dca-11a25a538455 [Follower]" leader-address= leader-id= 2025-01-28T14:48:07.351Z [WARN] server-80451566-3d16-6fd0-2dca-11a25a538455: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:07.351Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: entering candidate state: node="Node at 80451566-3d16-6fd0-2dca-11a25a538455 [Candidate]" term=2 2025-01-28T14:48:07.352Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: election won: term=2 tally=2 2025-01-28T14:48:07.352Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: entering leader state: leader="Node at 80451566-3d16-6fd0-2dca-11a25a538455 [Leader]" 2025-01-28T14:48:07.352Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: added peer, starting replication: peer=server-3c37f80e-90f7-498e-cc2d-503f26d3f261 2025-01-28T14:48:07.353Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: added peer, starting replication: peer=server-9515b21a-4ea1-2479-266f-4ed1a16e4eec 2025-01-28T14:48:07.354Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: pipelining replication: peer="{Voter server-3c37f80e-90f7-498e-cc2d-503f26d3f261 3c37f80e-90f7-498e-cc2d-503f26d3f261}" 2025-01-28T14:48:07.355Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: pipelining replication: peer="{Voter server-9515b21a-4ea1-2479-266f-4ed1a16e4eec 9515b21a-4ea1-2479-266f-4ed1a16e4eec}" 2025-01-28T14:48:08.361Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:07.353155625 +0000 UTC m=+70.436560649 (1 nodes), 612.469187ms from start of poll, 614.70292ms from cluster start. Timeout at 2025-01-28 14:48:08.358273295 +0000 UTC m=+71.441677986, 1.005117337s after stability 2025-01-28T14:48:08.362Z [INFO] creating new snapshot: path=/tmp/raft2368570834/snapshots/2-113-1738075688362.tmp 2025-01-28T14:48:08.366Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: copied to local snapshot: bytes=71 2025-01-28T14:48:08.368Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: snapshot restore progress: id=2-113-1738075688362 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:08.368Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: removing all old logs from log store 2025-01-28T14:48:08.368Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: compacting logs: from=1 to=2 2025-01-28T14:48:08.368Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: restored user snapshot: index=113 2025-01-28T14:48:08.368Z [ERROR] server-80451566-3d16-6fd0-2dca-11a25a538455: failed to get log: index=2 error="log not found" 2025-01-28T14:48:08.369Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: aborting pipeline replication: peer="{Voter server-9515b21a-4ea1-2479-266f-4ed1a16e4eec 9515b21a-4ea1-2479-266f-4ed1a16e4eec}" 2025-01-28T14:48:08.369Z [ERROR] server-80451566-3d16-6fd0-2dca-11a25a538455: failed to get log: index=2 error="log not found" 2025-01-28T14:48:08.369Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: aborting pipeline replication: peer="{Voter server-3c37f80e-90f7-498e-cc2d-503f26d3f261 3c37f80e-90f7-498e-cc2d-503f26d3f261}" 2025-01-28T14:48:08.376Z [ERROR] server-80451566-3d16-6fd0-2dca-11a25a538455: failed to get log: index=2 error="log not found" 2025-01-28T14:48:08.378Z [INFO] creating new snapshot: path=/tmp/raft1238104714/snapshots/2-113-1738075688378.tmp 2025-01-28T14:48:08.380Z [ERROR] server-80451566-3d16-6fd0-2dca-11a25a538455: failed to get log: index=2 error="log not found" 2025-01-28T14:48:08.380Z [INFO] server-9515b21a-4ea1-2479-266f-4ed1a16e4eec: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:08.390Z [INFO] server-9515b21a-4ea1-2479-266f-4ed1a16e4eec: copied to local snapshot: bytes=71 2025-01-28T14:48:08.391Z [INFO] creating new snapshot: path=/tmp/raft1867329354/snapshots/2-113-1738075688390.tmp 2025-01-28T14:48:08.392Z [INFO] server-9515b21a-4ea1-2479-266f-4ed1a16e4eec: snapshot restore progress: id=2-113-1738075688378 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:08.392Z [INFO] server-9515b21a-4ea1-2479-266f-4ed1a16e4eec: removing all old logs from log store 2025-01-28T14:48:08.392Z [INFO] server-9515b21a-4ea1-2479-266f-4ed1a16e4eec: compacting logs: from=1 to=2 2025-01-28T14:48:08.392Z [INFO] server-9515b21a-4ea1-2479-266f-4ed1a16e4eec: Installed remote snapshot 2025-01-28T14:48:08.392Z [INFO] server-3c37f80e-90f7-498e-cc2d-503f26d3f261: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:08.393Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: pipelining replication: peer="{Voter server-9515b21a-4ea1-2479-266f-4ed1a16e4eec 9515b21a-4ea1-2479-266f-4ed1a16e4eec}" 2025-01-28T14:48:08.395Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:08.395Z [INFO] server-3c37f80e-90f7-498e-cc2d-503f26d3f261: copied to local snapshot: bytes=71 2025-01-28T14:48:08.396Z [INFO] server-3c37f80e-90f7-498e-cc2d-503f26d3f261: snapshot restore progress: id=2-113-1738075688390 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:08.396Z [INFO] server-3c37f80e-90f7-498e-cc2d-503f26d3f261: removing all old logs from log store 2025-01-28T14:48:08.397Z [INFO] server-3c37f80e-90f7-498e-cc2d-503f26d3f261: compacting logs: from=1 to=2 2025-01-28T14:48:08.397Z [INFO] server-3c37f80e-90f7-498e-cc2d-503f26d3f261: Installed remote snapshot 2025-01-28T14:48:08.397Z [INFO] server-80451566-3d16-6fd0-2dca-11a25a538455: pipelining replication: peer="{Voter server-3c37f80e-90f7-498e-cc2d-503f26d3f261 3c37f80e-90f7-498e-cc2d-503f26d3f261}" 2025-01-28T14:48:09.400Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:08.395144899 +0000 UTC m=+71.478549590 (2 nodes), 7.666µs from start of poll, 1.656691861s from cluster start. Timeout at 2025-01-28 14:48:09.400388907 +0000 UTC m=+72.483793931, 1.005244341s after stability 2025-01-28T14:48:09.415Z [INFO] server-fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c: aborting pipeline replication: peer="{Voter server-4071d816-93ca-339a-f9c1-5f652d35b900 4071d816-93ca-339a-f9c1-5f652d35b900}" 2025-01-28T14:48:09.416Z [INFO] server-fe0351f2-2ea1-fcdf-4e10-8f422e7ef51c: aborting pipeline replication: peer="{Voter server-36d02b3b-2e5c-7d34-c23f-f15f45d4b6aa 36d02b3b-2e5c-7d34-c23f-f15f45d4b6aa}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false 2025-01-28T14:48:09.429Z [INFO] server-bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9 Address:bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9} {Suffrage:Voter ID:server-cd8ee909-9a12-9e35-2c85-7164cba4429d Address:cd8ee909-9a12-9e35-2c85-7164cba4429d} {Suffrage:Voter ID:server-7c555bf9-8b41-2ade-331c-b3045e0b5bce Address:7c555bf9-8b41-2ade-331c-b3045e0b5bce}]" 2025-01-28T14:48:09.429Z [INFO] server-bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9: entering follower state: follower="Node at bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9 [Follower]" leader-address= leader-id= 2025-01-28T14:48:09.430Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9 Address:bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9} {Suffrage:Voter ID:server-cd8ee909-9a12-9e35-2c85-7164cba4429d Address:cd8ee909-9a12-9e35-2c85-7164cba4429d} {Suffrage:Voter ID:server-7c555bf9-8b41-2ade-331c-b3045e0b5bce Address:7c555bf9-8b41-2ade-331c-b3045e0b5bce}]" 2025-01-28T14:48:09.430Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: entering follower state: follower="Node at cd8ee909-9a12-9e35-2c85-7164cba4429d [Follower]" leader-address= leader-id= 2025-01-28T14:48:09.430Z [INFO] server-7c555bf9-8b41-2ade-331c-b3045e0b5bce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9 Address:bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9} {Suffrage:Voter ID:server-cd8ee909-9a12-9e35-2c85-7164cba4429d Address:cd8ee909-9a12-9e35-2c85-7164cba4429d} {Suffrage:Voter ID:server-7c555bf9-8b41-2ade-331c-b3045e0b5bce Address:7c555bf9-8b41-2ade-331c-b3045e0b5bce}]" 2025-01-28T14:48:09.431Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:09.431Z [INFO] server-7c555bf9-8b41-2ade-331c-b3045e0b5bce: entering follower state: follower="Node at 7c555bf9-8b41-2ade-331c-b3045e0b5bce [Follower]" leader-address= leader-id= 2025-01-28T14:48:10.211Z [WARN] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:10.211Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: entering candidate state: node="Node at cd8ee909-9a12-9e35-2c85-7164cba4429d [Candidate]" term=2 2025-01-28T14:48:10.212Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: election won: term=2 tally=2 2025-01-28T14:48:10.212Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: entering leader state: leader="Node at cd8ee909-9a12-9e35-2c85-7164cba4429d [Leader]" 2025-01-28T14:48:10.212Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: added peer, starting replication: peer=server-bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9 2025-01-28T14:48:10.212Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: added peer, starting replication: peer=server-7c555bf9-8b41-2ade-331c-b3045e0b5bce 2025-01-28T14:48:10.213Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: pipelining replication: peer="{Voter server-7c555bf9-8b41-2ade-331c-b3045e0b5bce 7c555bf9-8b41-2ade-331c-b3045e0b5bce}" 2025-01-28T14:48:10.215Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: pipelining replication: peer="{Voter server-bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9 bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9}" 2025-01-28T14:48:11.218Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:10.213207553 +0000 UTC m=+73.296612577 (1 nodes), 782.010555ms from start of poll, 784.621966ms from cluster start. Timeout at 2025-01-28 14:48:11.218217219 +0000 UTC m=+74.301621910, 1.005009333s after stability 2025-01-28T14:48:11.222Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: starting snapshot up to: index=12 2025-01-28T14:48:11.222Z [INFO] creating new snapshot: path=/tmp/raft1853178945/snapshots/2-12-1738075691222.tmp 2025-01-28T14:48:11.228Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: snapshot complete up to: index=12 2025-01-28T14:48:11.249Z [INFO] creating new snapshot: path=/tmp/raft1853178945/snapshots/2-1013-1738075691249.tmp 2025-01-28T14:48:11.253Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: copied to local snapshot: bytes=71 2025-01-28T14:48:11.254Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: snapshot restore progress: id=2-1013-1738075691249 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:11.255Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: restored user snapshot: index=1013 2025-01-28T14:48:11.255Z [ERROR] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: failed to get log: index=23 error="log not found" 2025-01-28T14:48:11.255Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: aborting pipeline replication: peer="{Voter server-bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9 bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9}" 2025-01-28T14:48:11.255Z [ERROR] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: failed to get log: index=23 error="log not found" 2025-01-28T14:48:11.255Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: aborting pipeline replication: peer="{Voter server-7c555bf9-8b41-2ade-331c-b3045e0b5bce 7c555bf9-8b41-2ade-331c-b3045e0b5bce}" 2025-01-28T14:48:11.263Z [ERROR] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: failed to get log: index=23 error="log not found" 2025-01-28T14:48:11.263Z [ERROR] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: failed to get log: index=23 error="log not found" 2025-01-28T14:48:11.265Z [INFO] creating new snapshot: path=/tmp/raft4057434078/snapshots/2-1013-1738075691265.tmp 2025-01-28T14:48:11.266Z [INFO] creating new snapshot: path=/tmp/raft2952460223/snapshots/2-1013-1738075691266.tmp 2025-01-28T14:48:11.268Z [INFO] server-7c555bf9-8b41-2ade-331c-b3045e0b5bce: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:11.268Z [INFO] server-bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:11.273Z [INFO] server-bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9: copied to local snapshot: bytes=71 2025-01-28T14:48:11.274Z [INFO] server-bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9: snapshot restore progress: id=2-1013-1738075691265 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:11.274Z [INFO] server-bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9: Installed remote snapshot 2025-01-28T14:48:11.275Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: pipelining replication: peer="{Voter server-bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9 bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9}" 2025-01-28T14:48:11.276Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:11.278Z [INFO] server-7c555bf9-8b41-2ade-331c-b3045e0b5bce: copied to local snapshot: bytes=71 2025-01-28T14:48:11.280Z [INFO] server-7c555bf9-8b41-2ade-331c-b3045e0b5bce: snapshot restore progress: id=2-1013-1738075691266 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:11.280Z [INFO] server-7c555bf9-8b41-2ade-331c-b3045e0b5bce: Installed remote snapshot 2025-01-28T14:48:11.281Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: pipelining replication: peer="{Voter server-7c555bf9-8b41-2ade-331c-b3045e0b5bce 7c555bf9-8b41-2ade-331c-b3045e0b5bce}" 2025-01-28T14:48:12.282Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:11.276536795 +0000 UTC m=+74.359941819 (2 nodes), 6.667µs from start of poll, 1.847951208s from cluster start. Timeout at 2025-01-28 14:48:12.281784136 +0000 UTC m=+75.365189493, 1.005247674s after stability 2025-01-28T14:48:12.296Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: aborting pipeline replication: peer="{Voter server-7c555bf9-8b41-2ade-331c-b3045e0b5bce 7c555bf9-8b41-2ade-331c-b3045e0b5bce}" 2025-01-28T14:48:12.296Z [INFO] server-cd8ee909-9a12-9e35-2c85-7164cba4429d: aborting pipeline replication: peer="{Voter server-bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9 bfb4c83d-d7eb-2b5b-06d5-a0612343b7d9}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false 2025-01-28T14:48:12.309Z [INFO] server-342a93e5-d0a2-3cae-43f2-29f68eb32825: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-342a93e5-d0a2-3cae-43f2-29f68eb32825 Address:342a93e5-d0a2-3cae-43f2-29f68eb32825} {Suffrage:Voter ID:server-ec198e85-9faf-b877-8970-04c3b24214cb Address:ec198e85-9faf-b877-8970-04c3b24214cb} {Suffrage:Voter ID:server-fe702b79-a2fb-8bd2-23c1-1567725bd0e8 Address:fe702b79-a2fb-8bd2-23c1-1567725bd0e8}]" 2025-01-28T14:48:12.310Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-342a93e5-d0a2-3cae-43f2-29f68eb32825 Address:342a93e5-d0a2-3cae-43f2-29f68eb32825} {Suffrage:Voter ID:server-ec198e85-9faf-b877-8970-04c3b24214cb Address:ec198e85-9faf-b877-8970-04c3b24214cb} {Suffrage:Voter ID:server-fe702b79-a2fb-8bd2-23c1-1567725bd0e8 Address:fe702b79-a2fb-8bd2-23c1-1567725bd0e8}]" 2025-01-28T14:48:12.310Z [INFO] server-fe702b79-a2fb-8bd2-23c1-1567725bd0e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-342a93e5-d0a2-3cae-43f2-29f68eb32825 Address:342a93e5-d0a2-3cae-43f2-29f68eb32825} {Suffrage:Voter ID:server-ec198e85-9faf-b877-8970-04c3b24214cb Address:ec198e85-9faf-b877-8970-04c3b24214cb} {Suffrage:Voter ID:server-fe702b79-a2fb-8bd2-23c1-1567725bd0e8 Address:fe702b79-a2fb-8bd2-23c1-1567725bd0e8}]" 2025-01-28T14:48:12.311Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:12.311Z [INFO] server-342a93e5-d0a2-3cae-43f2-29f68eb32825: entering follower state: follower="Node at 342a93e5-d0a2-3cae-43f2-29f68eb32825 [Follower]" leader-address= leader-id= 2025-01-28T14:48:12.311Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: entering follower state: follower="Node at ec198e85-9faf-b877-8970-04c3b24214cb [Follower]" leader-address= leader-id= 2025-01-28T14:48:12.311Z [INFO] server-fe702b79-a2fb-8bd2-23c1-1567725bd0e8: entering follower state: follower="Node at fe702b79-a2fb-8bd2-23c1-1567725bd0e8 [Follower]" leader-address= leader-id= 2025-01-28T14:48:12.845Z [WARN] server-ec198e85-9faf-b877-8970-04c3b24214cb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:12.846Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: entering candidate state: node="Node at ec198e85-9faf-b877-8970-04c3b24214cb [Candidate]" term=2 2025-01-28T14:48:12.846Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: election won: term=2 tally=2 2025-01-28T14:48:12.846Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: entering leader state: leader="Node at ec198e85-9faf-b877-8970-04c3b24214cb [Leader]" 2025-01-28T14:48:12.846Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: added peer, starting replication: peer=server-342a93e5-d0a2-3cae-43f2-29f68eb32825 2025-01-28T14:48:12.846Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: added peer, starting replication: peer=server-fe702b79-a2fb-8bd2-23c1-1567725bd0e8 2025-01-28T14:48:12.847Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: pipelining replication: peer="{Voter server-fe702b79-a2fb-8bd2-23c1-1567725bd0e8 fe702b79-a2fb-8bd2-23c1-1567725bd0e8}" 2025-01-28T14:48:12.847Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: pipelining replication: peer="{Voter server-342a93e5-d0a2-3cae-43f2-29f68eb32825 342a93e5-d0a2-3cae-43f2-29f68eb32825}" 2025-01-28T14:48:13.857Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:12.847337106 +0000 UTC m=+75.930741797 (1 nodes), 536.097919ms from start of poll, 538.46599ms from cluster start. Timeout at 2025-01-28 14:48:13.852426109 +0000 UTC m=+76.935830800, 1.005089003s after stability 2025-01-28T14:48:13.872Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: starting snapshot up to: index=12 2025-01-28T14:48:13.872Z [INFO] creating new snapshot: path=/tmp/raft2419961856/snapshots/2-12-1738075693872.tmp 2025-01-28T14:48:13.878Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: snapshot complete up to: index=12 2025-01-28T14:48:13.885Z [INFO] creating new snapshot: path=/tmp/raft2419961856/snapshots/2-1013-1738075693885.tmp 2025-01-28T14:48:13.890Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: copied to local snapshot: bytes=71 2025-01-28T14:48:13.891Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: snapshot restore progress: id=2-1013-1738075693885 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:13.892Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: removing all old logs from log store 2025-01-28T14:48:13.892Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: compacting logs: from=1 to=22 2025-01-28T14:48:13.892Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: restored user snapshot: index=1013 2025-01-28T14:48:13.892Z [ERROR] server-ec198e85-9faf-b877-8970-04c3b24214cb: failed to get log: index=22 error="log not found" 2025-01-28T14:48:13.892Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: aborting pipeline replication: peer="{Voter server-fe702b79-a2fb-8bd2-23c1-1567725bd0e8 fe702b79-a2fb-8bd2-23c1-1567725bd0e8}" 2025-01-28T14:48:13.892Z [ERROR] server-ec198e85-9faf-b877-8970-04c3b24214cb: failed to get log: index=22 error="log not found" 2025-01-28T14:48:13.893Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: aborting pipeline replication: peer="{Voter server-342a93e5-d0a2-3cae-43f2-29f68eb32825 342a93e5-d0a2-3cae-43f2-29f68eb32825}" 2025-01-28T14:48:13.900Z [ERROR] server-ec198e85-9faf-b877-8970-04c3b24214cb: failed to get log: index=22 error="log not found" 2025-01-28T14:48:13.903Z [ERROR] server-ec198e85-9faf-b877-8970-04c3b24214cb: failed to get log: index=22 error="log not found" 2025-01-28T14:48:13.903Z [INFO] creating new snapshot: path=/tmp/raft2104220590/snapshots/2-1013-1738075693903.tmp 2025-01-28T14:48:13.905Z [INFO] server-fe702b79-a2fb-8bd2-23c1-1567725bd0e8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:13.910Z [INFO] server-fe702b79-a2fb-8bd2-23c1-1567725bd0e8: copied to local snapshot: bytes=71 2025-01-28T14:48:13.911Z [INFO] server-fe702b79-a2fb-8bd2-23c1-1567725bd0e8: snapshot restore progress: id=2-1013-1738075693903 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:13.911Z [INFO] server-fe702b79-a2fb-8bd2-23c1-1567725bd0e8: removing all old logs from log store 2025-01-28T14:48:13.911Z [INFO] server-fe702b79-a2fb-8bd2-23c1-1567725bd0e8: compacting logs: from=1 to=22 2025-01-28T14:48:13.911Z [INFO] server-fe702b79-a2fb-8bd2-23c1-1567725bd0e8: Installed remote snapshot 2025-01-28T14:48:13.912Z [INFO] creating new snapshot: path=/tmp/raft1238313485/snapshots/2-1013-1738075693912.tmp 2025-01-28T14:48:13.913Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: pipelining replication: peer="{Voter server-fe702b79-a2fb-8bd2-23c1-1567725bd0e8 fe702b79-a2fb-8bd2-23c1-1567725bd0e8}" 2025-01-28T14:48:13.913Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:13.914Z [INFO] server-342a93e5-d0a2-3cae-43f2-29f68eb32825: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:13.917Z [INFO] server-342a93e5-d0a2-3cae-43f2-29f68eb32825: copied to local snapshot: bytes=71 2025-01-28T14:48:13.918Z [INFO] server-342a93e5-d0a2-3cae-43f2-29f68eb32825: snapshot restore progress: id=2-1013-1738075693912 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:13.919Z [INFO] server-342a93e5-d0a2-3cae-43f2-29f68eb32825: removing all old logs from log store 2025-01-28T14:48:13.919Z [INFO] server-342a93e5-d0a2-3cae-43f2-29f68eb32825: compacting logs: from=1 to=22 2025-01-28T14:48:13.919Z [INFO] server-342a93e5-d0a2-3cae-43f2-29f68eb32825: Installed remote snapshot 2025-01-28T14:48:13.920Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: pipelining replication: peer="{Voter server-342a93e5-d0a2-3cae-43f2-29f68eb32825 342a93e5-d0a2-3cae-43f2-29f68eb32825}" 2025-01-28T14:48:14.920Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:13.913615102 +0000 UTC m=+76.997019793 (2 nodes), 6.667µs from start of poll, 1.604743986s from cluster start. Timeout at 2025-01-28 14:48:14.918839443 +0000 UTC m=+78.002244134, 1.005224341s after stability 2025-01-28T14:48:14.935Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: aborting pipeline replication: peer="{Voter server-fe702b79-a2fb-8bd2-23c1-1567725bd0e8 fe702b79-a2fb-8bd2-23c1-1567725bd0e8}" 2025-01-28T14:48:14.935Z [INFO] server-ec198e85-9faf-b877-8970-04c3b24214cb: aborting pipeline replication: peer="{Voter server-342a93e5-d0a2-3cae-43f2-29f68eb32825 342a93e5-d0a2-3cae-43f2-29f68eb32825}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true 2025-01-28T14:48:14.970Z [INFO] server-7f6f48a9-7f76-8a0c-321b-fc74a0b1063e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f6f48a9-7f76-8a0c-321b-fc74a0b1063e Address:7f6f48a9-7f76-8a0c-321b-fc74a0b1063e} {Suffrage:Voter ID:server-0a406d7e-1a30-97a6-97d9-a198341370d4 Address:0a406d7e-1a30-97a6-97d9-a198341370d4} {Suffrage:Voter ID:server-9c47da70-2268-570c-c05c-4a549f3e503a Address:9c47da70-2268-570c-c05c-4a549f3e503a}]" 2025-01-28T14:48:14.970Z [INFO] server-7f6f48a9-7f76-8a0c-321b-fc74a0b1063e: entering follower state: follower="Node at 7f6f48a9-7f76-8a0c-321b-fc74a0b1063e [Follower]" leader-address= leader-id= 2025-01-28T14:48:14.971Z [INFO] server-0a406d7e-1a30-97a6-97d9-a198341370d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f6f48a9-7f76-8a0c-321b-fc74a0b1063e Address:7f6f48a9-7f76-8a0c-321b-fc74a0b1063e} {Suffrage:Voter ID:server-0a406d7e-1a30-97a6-97d9-a198341370d4 Address:0a406d7e-1a30-97a6-97d9-a198341370d4} {Suffrage:Voter ID:server-9c47da70-2268-570c-c05c-4a549f3e503a Address:9c47da70-2268-570c-c05c-4a549f3e503a}]" 2025-01-28T14:48:14.971Z [INFO] server-0a406d7e-1a30-97a6-97d9-a198341370d4: entering follower state: follower="Node at 0a406d7e-1a30-97a6-97d9-a198341370d4 [Follower]" leader-address= leader-id= 2025-01-28T14:48:14.971Z [INFO] server-9c47da70-2268-570c-c05c-4a549f3e503a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f6f48a9-7f76-8a0c-321b-fc74a0b1063e Address:7f6f48a9-7f76-8a0c-321b-fc74a0b1063e} {Suffrage:Voter ID:server-0a406d7e-1a30-97a6-97d9-a198341370d4 Address:0a406d7e-1a30-97a6-97d9-a198341370d4} {Suffrage:Voter ID:server-9c47da70-2268-570c-c05c-4a549f3e503a Address:9c47da70-2268-570c-c05c-4a549f3e503a}]" 2025-01-28T14:48:14.972Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:14.972Z [INFO] server-9c47da70-2268-570c-c05c-4a549f3e503a: entering follower state: follower="Node at 9c47da70-2268-570c-c05c-4a549f3e503a [Follower]" leader-address= leader-id= 2025-01-28T14:48:15.626Z [WARN] server-0a406d7e-1a30-97a6-97d9-a198341370d4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:15.626Z [INFO] server-0a406d7e-1a30-97a6-97d9-a198341370d4: entering candidate state: node="Node at 0a406d7e-1a30-97a6-97d9-a198341370d4 [Candidate]" term=2 2025-01-28T14:48:15.627Z [INFO] server-0a406d7e-1a30-97a6-97d9-a198341370d4: election won: term=2 tally=2 2025-01-28T14:48:15.627Z [INFO] server-0a406d7e-1a30-97a6-97d9-a198341370d4: entering leader state: leader="Node at 0a406d7e-1a30-97a6-97d9-a198341370d4 [Leader]" 2025-01-28T14:48:15.627Z [INFO] server-0a406d7e-1a30-97a6-97d9-a198341370d4: added peer, starting replication: peer=server-7f6f48a9-7f76-8a0c-321b-fc74a0b1063e 2025-01-28T14:48:15.627Z [INFO] server-0a406d7e-1a30-97a6-97d9-a198341370d4: added peer, starting replication: peer=server-9c47da70-2268-570c-c05c-4a549f3e503a 2025-01-28T14:48:15.628Z [INFO] server-0a406d7e-1a30-97a6-97d9-a198341370d4: pipelining replication: peer="{Voter server-7f6f48a9-7f76-8a0c-321b-fc74a0b1063e 7f6f48a9-7f76-8a0c-321b-fc74a0b1063e}" 2025-01-28T14:48:15.629Z [INFO] server-0a406d7e-1a30-97a6-97d9-a198341370d4: pipelining replication: peer="{Voter server-9c47da70-2268-570c-c05c-4a549f3e503a 9c47da70-2268-570c-c05c-4a549f3e503a}" 2025-01-28T14:48:16.633Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:15.627445661 +0000 UTC m=+78.710850352 (1 nodes), 655.249457ms from start of poll, 677.173775ms from cluster start. Timeout at 2025-01-28 14:48:16.632457329 +0000 UTC m=+79.715862020, 1.005011668s after stability 2025-01-28T14:48:16.638Z [INFO] server-0a406d7e-1a30-97a6-97d9-a198341370d4: starting snapshot up to: index=12 2025-01-28T14:48:16.638Z [INFO] creating new snapshot: path=/tmp/raft1002841784/snapshots/2-12-1738075696638.tmp 2025-01-28T14:48:16.642Z [INFO] server-0a406d7e-1a30-97a6-97d9-a198341370d4: snapshot complete up to: index=12 2025-01-28T14:48:16.666Z [INFO] server-6065237c-1a54-1c4c-0f7a-22e5cdb6d113: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6065237c-1a54-1c4c-0f7a-22e5cdb6d113 Address:6065237c-1a54-1c4c-0f7a-22e5cdb6d113} {Suffrage:Voter ID:server-59534215-baee-5112-685b-5f864bddd655 Address:59534215-baee-5112-685b-5f864bddd655} {Suffrage:Voter ID:server-59bb37bb-db13-50dd-ea44-3385e8588740 Address:59bb37bb-db13-50dd-ea44-3385e8588740}]" 2025-01-28T14:48:16.666Z [INFO] server-6065237c-1a54-1c4c-0f7a-22e5cdb6d113: entering follower state: follower="Node at 6065237c-1a54-1c4c-0f7a-22e5cdb6d113 [Follower]" leader-address= leader-id= 2025-01-28T14:48:16.667Z [INFO] server-59534215-baee-5112-685b-5f864bddd655: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6065237c-1a54-1c4c-0f7a-22e5cdb6d113 Address:6065237c-1a54-1c4c-0f7a-22e5cdb6d113} {Suffrage:Voter ID:server-59534215-baee-5112-685b-5f864bddd655 Address:59534215-baee-5112-685b-5f864bddd655} {Suffrage:Voter ID:server-59bb37bb-db13-50dd-ea44-3385e8588740 Address:59bb37bb-db13-50dd-ea44-3385e8588740}]" 2025-01-28T14:48:16.667Z [INFO] server-59534215-baee-5112-685b-5f864bddd655: entering follower state: follower="Node at 59534215-baee-5112-685b-5f864bddd655 [Follower]" leader-address= leader-id= 2025-01-28T14:48:16.668Z [INFO] server-59bb37bb-db13-50dd-ea44-3385e8588740: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6065237c-1a54-1c4c-0f7a-22e5cdb6d113 Address:6065237c-1a54-1c4c-0f7a-22e5cdb6d113} {Suffrage:Voter ID:server-59534215-baee-5112-685b-5f864bddd655 Address:59534215-baee-5112-685b-5f864bddd655} {Suffrage:Voter ID:server-59bb37bb-db13-50dd-ea44-3385e8588740 Address:59bb37bb-db13-50dd-ea44-3385e8588740}]" 2025-01-28T14:48:16.668Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:16.668Z [INFO] server-59bb37bb-db13-50dd-ea44-3385e8588740: entering follower state: follower="Node at 59bb37bb-db13-50dd-ea44-3385e8588740 [Follower]" leader-address= leader-id= 2025-01-28T14:48:17.244Z [WARN] server-59bb37bb-db13-50dd-ea44-3385e8588740: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:17.244Z [INFO] server-59bb37bb-db13-50dd-ea44-3385e8588740: entering candidate state: node="Node at 59bb37bb-db13-50dd-ea44-3385e8588740 [Candidate]" term=2 2025-01-28T14:48:17.245Z [INFO] server-59bb37bb-db13-50dd-ea44-3385e8588740: election won: term=2 tally=2 2025-01-28T14:48:17.245Z [INFO] server-59bb37bb-db13-50dd-ea44-3385e8588740: entering leader state: leader="Node at 59bb37bb-db13-50dd-ea44-3385e8588740 [Leader]" 2025-01-28T14:48:17.245Z [INFO] server-59bb37bb-db13-50dd-ea44-3385e8588740: added peer, starting replication: peer=server-6065237c-1a54-1c4c-0f7a-22e5cdb6d113 2025-01-28T14:48:17.245Z [INFO] server-59bb37bb-db13-50dd-ea44-3385e8588740: added peer, starting replication: peer=server-59534215-baee-5112-685b-5f864bddd655 2025-01-28T14:48:17.246Z [INFO] server-59bb37bb-db13-50dd-ea44-3385e8588740: pipelining replication: peer="{Voter server-59534215-baee-5112-685b-5f864bddd655 59534215-baee-5112-685b-5f864bddd655}" 2025-01-28T14:48:17.246Z [INFO] server-59bb37bb-db13-50dd-ea44-3385e8588740: pipelining replication: peer="{Voter server-6065237c-1a54-1c4c-0f7a-22e5cdb6d113 6065237c-1a54-1c4c-0f7a-22e5cdb6d113}" 2025-01-28T14:48:18.251Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:17.245737717 +0000 UTC m=+80.329142741 (1 nodes), 577.375812ms from start of poll, 579.717215ms from cluster start. Timeout at 2025-01-28 14:48:18.250758051 +0000 UTC m=+81.334163075, 1.005020334s after stability 2025-01-28T14:48:18.252Z [INFO] creating new snapshot: path=/tmp/raft2755412486/snapshots/2-1013-1738075698252.tmp 2025-01-28T14:48:18.258Z [INFO] server-59bb37bb-db13-50dd-ea44-3385e8588740: copied to local snapshot: bytes=71 2025-01-28T14:48:18.260Z [INFO] server-59bb37bb-db13-50dd-ea44-3385e8588740: snapshot restore progress: id=2-1013-1738075698252 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:18.260Z [INFO] server-59bb37bb-db13-50dd-ea44-3385e8588740: restored user snapshot: index=1013 2025-01-28T14:48:18.260Z [ERROR] server-59bb37bb-db13-50dd-ea44-3385e8588740: failed to get log: index=3 error="log not found" 2025-01-28T14:48:18.260Z [INFO] server-59bb37bb-db13-50dd-ea44-3385e8588740: aborting pipeline replication: peer="{Voter server-6065237c-1a54-1c4c-0f7a-22e5cdb6d113 6065237c-1a54-1c4c-0f7a-22e5cdb6d113}" 2025-01-28T14:48:18.260Z [ERROR] server-59bb37bb-db13-50dd-ea44-3385e8588740: failed to get log: index=3 error="log not found" 2025-01-28T14:48:18.261Z [INFO] server-59bb37bb-db13-50dd-ea44-3385e8588740: aborting pipeline replication: peer="{Voter server-59534215-baee-5112-685b-5f864bddd655 59534215-baee-5112-685b-5f864bddd655}" 2025-01-28T14:48:18.269Z [ERROR] server-59bb37bb-db13-50dd-ea44-3385e8588740: failed to get log: index=3 error="log not found" 2025-01-28T14:48:18.269Z [ERROR] server-59bb37bb-db13-50dd-ea44-3385e8588740: failed to get log: index=3 error="log not found" 2025-01-28T14:48:18.271Z [INFO] creating new snapshot: path=/tmp/raft1819036198/snapshots/2-1013-1738075698271.tmp 2025-01-28T14:48:18.271Z [INFO] creating new snapshot: path=/tmp/raft1622522723/snapshots/2-1013-1738075698271.tmp 2025-01-28T14:48:18.274Z [INFO] server-59534215-baee-5112-685b-5f864bddd655: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:18.274Z [INFO] server-6065237c-1a54-1c4c-0f7a-22e5cdb6d113: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:18.276Z [INFO] server-59534215-baee-5112-685b-5f864bddd655: copied to local snapshot: bytes=71 2025-01-28T14:48:18.276Z [INFO] server-6065237c-1a54-1c4c-0f7a-22e5cdb6d113: copied to local snapshot: bytes=71 2025-01-28T14:48:18.278Z [INFO] server-59534215-baee-5112-685b-5f864bddd655: snapshot restore progress: id=2-1013-1738075698271 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:18.278Z [INFO] server-6065237c-1a54-1c4c-0f7a-22e5cdb6d113: snapshot restore progress: id=2-1013-1738075698271 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:18.278Z [INFO] server-59534215-baee-5112-685b-5f864bddd655: Installed remote snapshot 2025-01-28T14:48:18.279Z [INFO] server-6065237c-1a54-1c4c-0f7a-22e5cdb6d113: Installed remote snapshot 2025-01-28T14:48:18.279Z [INFO] server-59bb37bb-db13-50dd-ea44-3385e8588740: pipelining replication: peer="{Voter server-6065237c-1a54-1c4c-0f7a-22e5cdb6d113 6065237c-1a54-1c4c-0f7a-22e5cdb6d113}" 2025-01-28T14:48:18.280Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:18.280Z [WARN] server-59534215-baee-5112-685b-5f864bddd655: skipping application of old log: index=1013 2025-01-28T14:48:18.280Z [INFO] server-59bb37bb-db13-50dd-ea44-3385e8588740: pipelining replication: peer="{Voter server-59534215-baee-5112-685b-5f864bddd655 59534215-baee-5112-685b-5f864bddd655}" 2025-01-28T14:48:19.287Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:18.280420442 +0000 UTC m=+81.363825133 (2 nodes), 7.333µs from start of poll, 1.614399607s from cluster start. Timeout at 2025-01-28 14:48:19.285889124 +0000 UTC m=+82.369294481, 1.005469348s after stability 2025-01-28T14:48:19.304Z [INFO] server-0a406d7e-1a30-97a6-97d9-a198341370d4: aborting pipeline replication: peer="{Voter server-7f6f48a9-7f76-8a0c-321b-fc74a0b1063e 7f6f48a9-7f76-8a0c-321b-fc74a0b1063e}" 2025-01-28T14:48:19.304Z [INFO] server-0a406d7e-1a30-97a6-97d9-a198341370d4: aborting pipeline replication: peer="{Voter server-9c47da70-2268-570c-c05c-4a549f3e503a 9c47da70-2268-570c-c05c-4a549f3e503a}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true 2025-01-28T14:48:19.318Z [INFO] server-b3bd3d9e-323e-c040-6bce-3b3117c8385c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3bd3d9e-323e-c040-6bce-3b3117c8385c Address:b3bd3d9e-323e-c040-6bce-3b3117c8385c} {Suffrage:Voter ID:server-9847a3c5-7878-ee88-fbe5-3e6f389019b1 Address:9847a3c5-7878-ee88-fbe5-3e6f389019b1} {Suffrage:Voter ID:server-2ccdaf21-38c6-83c8-04ac-671ed5f54eac Address:2ccdaf21-38c6-83c8-04ac-671ed5f54eac}]" 2025-01-28T14:48:19.318Z [INFO] server-b3bd3d9e-323e-c040-6bce-3b3117c8385c: entering follower state: follower="Node at b3bd3d9e-323e-c040-6bce-3b3117c8385c [Follower]" leader-address= leader-id= 2025-01-28T14:48:19.318Z [INFO] server-9847a3c5-7878-ee88-fbe5-3e6f389019b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3bd3d9e-323e-c040-6bce-3b3117c8385c Address:b3bd3d9e-323e-c040-6bce-3b3117c8385c} {Suffrage:Voter ID:server-9847a3c5-7878-ee88-fbe5-3e6f389019b1 Address:9847a3c5-7878-ee88-fbe5-3e6f389019b1} {Suffrage:Voter ID:server-2ccdaf21-38c6-83c8-04ac-671ed5f54eac Address:2ccdaf21-38c6-83c8-04ac-671ed5f54eac}]" 2025-01-28T14:48:19.319Z [INFO] server-9847a3c5-7878-ee88-fbe5-3e6f389019b1: entering follower state: follower="Node at 9847a3c5-7878-ee88-fbe5-3e6f389019b1 [Follower]" leader-address= leader-id= 2025-01-28T14:48:19.325Z [INFO] server-2ccdaf21-38c6-83c8-04ac-671ed5f54eac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3bd3d9e-323e-c040-6bce-3b3117c8385c Address:b3bd3d9e-323e-c040-6bce-3b3117c8385c} {Suffrage:Voter ID:server-9847a3c5-7878-ee88-fbe5-3e6f389019b1 Address:9847a3c5-7878-ee88-fbe5-3e6f389019b1} {Suffrage:Voter ID:server-2ccdaf21-38c6-83c8-04ac-671ed5f54eac Address:2ccdaf21-38c6-83c8-04ac-671ed5f54eac}]" 2025-01-28T14:48:19.326Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:19.328Z [INFO] server-2ccdaf21-38c6-83c8-04ac-671ed5f54eac: entering follower state: follower="Node at 2ccdaf21-38c6-83c8-04ac-671ed5f54eac [Follower]" leader-address= leader-id= 2025-01-28T14:48:19.839Z [WARN] server-2ccdaf21-38c6-83c8-04ac-671ed5f54eac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:19.840Z [INFO] server-2ccdaf21-38c6-83c8-04ac-671ed5f54eac: entering candidate state: node="Node at 2ccdaf21-38c6-83c8-04ac-671ed5f54eac [Candidate]" term=2 2025-01-28T14:48:19.841Z [INFO] server-2ccdaf21-38c6-83c8-04ac-671ed5f54eac: election won: term=2 tally=2 2025-01-28T14:48:19.842Z [INFO] server-2ccdaf21-38c6-83c8-04ac-671ed5f54eac: entering leader state: leader="Node at 2ccdaf21-38c6-83c8-04ac-671ed5f54eac [Leader]" 2025-01-28T14:48:19.842Z [INFO] server-2ccdaf21-38c6-83c8-04ac-671ed5f54eac: added peer, starting replication: peer=server-b3bd3d9e-323e-c040-6bce-3b3117c8385c 2025-01-28T14:48:19.842Z [INFO] server-2ccdaf21-38c6-83c8-04ac-671ed5f54eac: added peer, starting replication: peer=server-9847a3c5-7878-ee88-fbe5-3e6f389019b1 2025-01-28T14:48:19.844Z [INFO] server-2ccdaf21-38c6-83c8-04ac-671ed5f54eac: pipelining replication: peer="{Voter server-9847a3c5-7878-ee88-fbe5-3e6f389019b1 9847a3c5-7878-ee88-fbe5-3e6f389019b1}" 2025-01-28T14:48:19.845Z [INFO] server-2ccdaf21-38c6-83c8-04ac-671ed5f54eac: pipelining replication: peer="{Voter server-b3bd3d9e-323e-c040-6bce-3b3117c8385c b3bd3d9e-323e-c040-6bce-3b3117c8385c}" 2025-01-28T14:48:20.848Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:19.842850173 +0000 UTC m=+82.926256531 (1 nodes), 516.483338ms from start of poll, 525.402269ms from cluster start. Timeout at 2025-01-28 14:48:20.847894174 +0000 UTC m=+83.931300198, 1.005043667s after stability 2025-01-28T14:48:20.858Z [INFO] server-2ccdaf21-38c6-83c8-04ac-671ed5f54eac: starting snapshot up to: index=12 2025-01-28T14:48:20.858Z [INFO] creating new snapshot: path=/tmp/raft299851659/snapshots/2-12-1738075700858.tmp 2025-01-28T14:48:20.876Z [INFO] server-2ccdaf21-38c6-83c8-04ac-671ed5f54eac: snapshot complete up to: index=12 2025-01-28T14:48:20.886Z [INFO] server-c3b633cc-60ce-aeac-a5c9-2437f2bb711b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3b633cc-60ce-aeac-a5c9-2437f2bb711b Address:c3b633cc-60ce-aeac-a5c9-2437f2bb711b} {Suffrage:Voter ID:server-622bf10a-b5d0-f174-c1c8-1b771b0914e9 Address:622bf10a-b5d0-f174-c1c8-1b771b0914e9} {Suffrage:Voter ID:server-577cee66-e0a3-b801-e8af-c79721ad7e75 Address:577cee66-e0a3-b801-e8af-c79721ad7e75}]" 2025-01-28T14:48:20.886Z [INFO] server-c3b633cc-60ce-aeac-a5c9-2437f2bb711b: entering follower state: follower="Node at c3b633cc-60ce-aeac-a5c9-2437f2bb711b [Follower]" leader-address= leader-id= 2025-01-28T14:48:20.886Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3b633cc-60ce-aeac-a5c9-2437f2bb711b Address:c3b633cc-60ce-aeac-a5c9-2437f2bb711b} {Suffrage:Voter ID:server-622bf10a-b5d0-f174-c1c8-1b771b0914e9 Address:622bf10a-b5d0-f174-c1c8-1b771b0914e9} {Suffrage:Voter ID:server-577cee66-e0a3-b801-e8af-c79721ad7e75 Address:577cee66-e0a3-b801-e8af-c79721ad7e75}]" 2025-01-28T14:48:20.887Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: entering follower state: follower="Node at 622bf10a-b5d0-f174-c1c8-1b771b0914e9 [Follower]" leader-address= leader-id= 2025-01-28T14:48:20.887Z [INFO] server-577cee66-e0a3-b801-e8af-c79721ad7e75: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3b633cc-60ce-aeac-a5c9-2437f2bb711b Address:c3b633cc-60ce-aeac-a5c9-2437f2bb711b} {Suffrage:Voter ID:server-622bf10a-b5d0-f174-c1c8-1b771b0914e9 Address:622bf10a-b5d0-f174-c1c8-1b771b0914e9} {Suffrage:Voter ID:server-577cee66-e0a3-b801-e8af-c79721ad7e75 Address:577cee66-e0a3-b801-e8af-c79721ad7e75}]" 2025-01-28T14:48:20.887Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:20.887Z [INFO] server-577cee66-e0a3-b801-e8af-c79721ad7e75: entering follower state: follower="Node at 577cee66-e0a3-b801-e8af-c79721ad7e75 [Follower]" leader-address= leader-id= 2025-01-28T14:48:21.581Z [WARN] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:21.581Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: entering candidate state: node="Node at 622bf10a-b5d0-f174-c1c8-1b771b0914e9 [Candidate]" term=2 2025-01-28T14:48:21.582Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: election won: term=2 tally=2 2025-01-28T14:48:21.582Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: entering leader state: leader="Node at 622bf10a-b5d0-f174-c1c8-1b771b0914e9 [Leader]" 2025-01-28T14:48:21.582Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: added peer, starting replication: peer=server-c3b633cc-60ce-aeac-a5c9-2437f2bb711b 2025-01-28T14:48:21.582Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: added peer, starting replication: peer=server-577cee66-e0a3-b801-e8af-c79721ad7e75 2025-01-28T14:48:21.583Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: pipelining replication: peer="{Voter server-577cee66-e0a3-b801-e8af-c79721ad7e75 577cee66-e0a3-b801-e8af-c79721ad7e75}" 2025-01-28T14:48:21.583Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: pipelining replication: peer="{Voter server-c3b633cc-60ce-aeac-a5c9-2437f2bb711b c3b633cc-60ce-aeac-a5c9-2437f2bb711b}" 2025-01-28T14:48:22.588Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:21.582727506 +0000 UTC m=+84.666132197 (1 nodes), 694.847967ms from start of poll, 697.172703ms from cluster start. Timeout at 2025-01-28 14:48:22.587747174 +0000 UTC m=+85.671152198, 1.005020001s after stability 2025-01-28T14:48:22.589Z [INFO] creating new snapshot: path=/tmp/raft1310620579/snapshots/2-1013-1738075702589.tmp 2025-01-28T14:48:22.594Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: copied to local snapshot: bytes=71 2025-01-28T14:48:22.596Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: snapshot restore progress: id=2-1013-1738075702589 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:22.596Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: removing all old logs from log store 2025-01-28T14:48:22.596Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: compacting logs: from=1 to=2 2025-01-28T14:48:22.596Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: restored user snapshot: index=1013 2025-01-28T14:48:22.597Z [ERROR] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: failed to get log: index=2 error="log not found" 2025-01-28T14:48:22.597Z [ERROR] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: failed to get log: index=2 error="log not found" 2025-01-28T14:48:22.597Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: aborting pipeline replication: peer="{Voter server-577cee66-e0a3-b801-e8af-c79721ad7e75 577cee66-e0a3-b801-e8af-c79721ad7e75}" 2025-01-28T14:48:22.597Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: aborting pipeline replication: peer="{Voter server-c3b633cc-60ce-aeac-a5c9-2437f2bb711b c3b633cc-60ce-aeac-a5c9-2437f2bb711b}" 2025-01-28T14:48:22.606Z [ERROR] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: failed to get log: index=2 error="log not found" 2025-01-28T14:48:22.617Z [ERROR] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: failed to get log: index=2 error="log not found" 2025-01-28T14:48:22.621Z [INFO] creating new snapshot: path=/tmp/raft1371859071/snapshots/2-1013-1738075702621.tmp 2025-01-28T14:48:22.624Z [INFO] server-577cee66-e0a3-b801-e8af-c79721ad7e75: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:22.626Z [INFO] creating new snapshot: path=/tmp/raft2633116013/snapshots/2-1013-1738075702626.tmp 2025-01-28T14:48:22.628Z [INFO] server-577cee66-e0a3-b801-e8af-c79721ad7e75: copied to local snapshot: bytes=71 2025-01-28T14:48:22.628Z [INFO] server-c3b633cc-60ce-aeac-a5c9-2437f2bb711b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:22.630Z [INFO] server-577cee66-e0a3-b801-e8af-c79721ad7e75: snapshot restore progress: id=2-1013-1738075702621 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:22.630Z [INFO] server-577cee66-e0a3-b801-e8af-c79721ad7e75: removing all old logs from log store 2025-01-28T14:48:22.630Z [INFO] server-577cee66-e0a3-b801-e8af-c79721ad7e75: compacting logs: from=1 to=2 2025-01-28T14:48:22.630Z [INFO] server-577cee66-e0a3-b801-e8af-c79721ad7e75: Installed remote snapshot 2025-01-28T14:48:22.631Z [INFO] server-c3b633cc-60ce-aeac-a5c9-2437f2bb711b: copied to local snapshot: bytes=71 2025-01-28T14:48:22.632Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: pipelining replication: peer="{Voter server-577cee66-e0a3-b801-e8af-c79721ad7e75 577cee66-e0a3-b801-e8af-c79721ad7e75}" 2025-01-28T14:48:22.632Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:22.632Z [INFO] server-c3b633cc-60ce-aeac-a5c9-2437f2bb711b: snapshot restore progress: id=2-1013-1738075702626 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:22.632Z [INFO] server-c3b633cc-60ce-aeac-a5c9-2437f2bb711b: removing all old logs from log store 2025-01-28T14:48:22.632Z [INFO] server-c3b633cc-60ce-aeac-a5c9-2437f2bb711b: compacting logs: from=1 to=2 2025-01-28T14:48:22.633Z [INFO] server-c3b633cc-60ce-aeac-a5c9-2437f2bb711b: Installed remote snapshot 2025-01-28T14:48:22.633Z [INFO] server-622bf10a-b5d0-f174-c1c8-1b771b0914e9: pipelining replication: peer="{Voter server-c3b633cc-60ce-aeac-a5c9-2437f2bb711b c3b633cc-60ce-aeac-a5c9-2437f2bb711b}" 2025-01-28T14:48:23.638Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:22.632672352 +0000 UTC m=+85.716077043 (2 nodes), 7.667µs from start of poll, 1.747117549s from cluster start. Timeout at 2025-01-28 14:48:23.638124365 +0000 UTC m=+86.721529389, 1.005452346s after stability 2025-01-28T14:48:23.653Z [INFO] server-2ccdaf21-38c6-83c8-04ac-671ed5f54eac: aborting pipeline replication: peer="{Voter server-9847a3c5-7878-ee88-fbe5-3e6f389019b1 9847a3c5-7878-ee88-fbe5-3e6f389019b1}" 2025-01-28T14:48:23.653Z [INFO] server-2ccdaf21-38c6-83c8-04ac-671ed5f54eac: aborting pipeline replication: peer="{Voter server-b3bd3d9e-323e-c040-6bce-3b3117c8385c b3bd3d9e-323e-c040-6bce-3b3117c8385c}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false 2025-01-28T14:48:23.666Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1 Address:e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1} {Suffrage:Voter ID:server-d6b9257c-51c8-af4e-866d-554a5007d821 Address:d6b9257c-51c8-af4e-866d-554a5007d821} {Suffrage:Voter ID:server-8818d339-8684-0522-92a9-44a956aa148f Address:8818d339-8684-0522-92a9-44a956aa148f}]" 2025-01-28T14:48:23.666Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: entering follower state: follower="Node at e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1 [Follower]" leader-address= leader-id= 2025-01-28T14:48:23.667Z [INFO] server-d6b9257c-51c8-af4e-866d-554a5007d821: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1 Address:e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1} {Suffrage:Voter ID:server-d6b9257c-51c8-af4e-866d-554a5007d821 Address:d6b9257c-51c8-af4e-866d-554a5007d821} {Suffrage:Voter ID:server-8818d339-8684-0522-92a9-44a956aa148f Address:8818d339-8684-0522-92a9-44a956aa148f}]" 2025-01-28T14:48:23.667Z [INFO] server-d6b9257c-51c8-af4e-866d-554a5007d821: entering follower state: follower="Node at d6b9257c-51c8-af4e-866d-554a5007d821 [Follower]" leader-address= leader-id= 2025-01-28T14:48:23.667Z [INFO] server-8818d339-8684-0522-92a9-44a956aa148f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1 Address:e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1} {Suffrage:Voter ID:server-d6b9257c-51c8-af4e-866d-554a5007d821 Address:d6b9257c-51c8-af4e-866d-554a5007d821} {Suffrage:Voter ID:server-8818d339-8684-0522-92a9-44a956aa148f Address:8818d339-8684-0522-92a9-44a956aa148f}]" 2025-01-28T14:48:23.667Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:23.667Z [INFO] server-8818d339-8684-0522-92a9-44a956aa148f: entering follower state: follower="Node at 8818d339-8684-0522-92a9-44a956aa148f [Follower]" leader-address= leader-id= 2025-01-28T14:48:24.223Z [WARN] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:24.223Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: entering candidate state: node="Node at e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1 [Candidate]" term=2 2025-01-28T14:48:24.224Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: election won: term=2 tally=2 2025-01-28T14:48:24.224Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: entering leader state: leader="Node at e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1 [Leader]" 2025-01-28T14:48:24.224Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: added peer, starting replication: peer=server-d6b9257c-51c8-af4e-866d-554a5007d821 2025-01-28T14:48:24.225Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: added peer, starting replication: peer=server-8818d339-8684-0522-92a9-44a956aa148f 2025-01-28T14:48:24.225Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: pipelining replication: peer="{Voter server-8818d339-8684-0522-92a9-44a956aa148f 8818d339-8684-0522-92a9-44a956aa148f}" 2025-01-28T14:48:24.227Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: pipelining replication: peer="{Voter server-d6b9257c-51c8-af4e-866d-554a5007d821 d6b9257c-51c8-af4e-866d-554a5007d821}" 2025-01-28T14:48:25.231Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:24.226462681 +0000 UTC m=+87.309868039 (1 nodes), 558.515252ms from start of poll, 560.854989ms from cluster start. Timeout at 2025-01-28 14:48:25.231473015 +0000 UTC m=+88.314878039, 1.00501s after stability 2025-01-28T14:48:25.236Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: starting snapshot up to: index=12 2025-01-28T14:48:25.237Z [INFO] creating new snapshot: path=/tmp/raft2833461981/snapshots/2-12-1738075705237.tmp 2025-01-28T14:48:25.240Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: snapshot complete up to: index=12 2025-01-28T14:48:25.246Z [INFO] creating new snapshot: path=/tmp/raft2833461981/snapshots/2-10013-1738075705246.tmp 2025-01-28T14:48:25.251Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: copied to local snapshot: bytes=71 2025-01-28T14:48:25.252Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: snapshot restore progress: id=2-10013-1738075705246 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:25.253Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: restored user snapshot: index=10013 2025-01-28T14:48:25.253Z [ERROR] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: failed to get log: index=23 error="log not found" 2025-01-28T14:48:25.253Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: aborting pipeline replication: peer="{Voter server-8818d339-8684-0522-92a9-44a956aa148f 8818d339-8684-0522-92a9-44a956aa148f}" 2025-01-28T14:48:25.254Z [ERROR] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: failed to get log: index=23 error="log not found" 2025-01-28T14:48:25.254Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: aborting pipeline replication: peer="{Voter server-d6b9257c-51c8-af4e-866d-554a5007d821 d6b9257c-51c8-af4e-866d-554a5007d821}" 2025-01-28T14:48:25.261Z [ERROR] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: failed to get log: index=23 error="log not found" 2025-01-28T14:48:25.267Z [INFO] creating new snapshot: path=/tmp/raft4065308969/snapshots/2-10013-1738075705267.tmp 2025-01-28T14:48:25.268Z [ERROR] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: failed to get log: index=23 error="log not found" 2025-01-28T14:48:25.270Z [INFO] server-8818d339-8684-0522-92a9-44a956aa148f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:25.272Z [INFO] creating new snapshot: path=/tmp/raft316726399/snapshots/2-10013-1738075705272.tmp 2025-01-28T14:48:25.272Z [INFO] server-8818d339-8684-0522-92a9-44a956aa148f: copied to local snapshot: bytes=71 2025-01-28T14:48:25.273Z [INFO] server-8818d339-8684-0522-92a9-44a956aa148f: snapshot restore progress: id=2-10013-1738075705267 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:25.274Z [INFO] server-8818d339-8684-0522-92a9-44a956aa148f: Installed remote snapshot 2025-01-28T14:48:25.274Z [INFO] server-d6b9257c-51c8-af4e-866d-554a5007d821: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:25.275Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: pipelining replication: peer="{Voter server-8818d339-8684-0522-92a9-44a956aa148f 8818d339-8684-0522-92a9-44a956aa148f}" 2025-01-28T14:48:25.275Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:25.277Z [INFO] server-d6b9257c-51c8-af4e-866d-554a5007d821: copied to local snapshot: bytes=71 2025-01-28T14:48:25.279Z [INFO] server-d6b9257c-51c8-af4e-866d-554a5007d821: snapshot restore progress: id=2-10013-1738075705272 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:25.279Z [INFO] server-d6b9257c-51c8-af4e-866d-554a5007d821: Installed remote snapshot 2025-01-28T14:48:25.280Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: pipelining replication: peer="{Voter server-d6b9257c-51c8-af4e-866d-554a5007d821 d6b9257c-51c8-af4e-866d-554a5007d821}" 2025-01-28T14:48:26.281Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:25.275328162 +0000 UTC m=+88.358732853 (2 nodes), 8.334µs from start of poll, 1.609719803s from cluster start. Timeout at 2025-01-28 14:48:26.280763842 +0000 UTC m=+89.364168866, 1.005436013s after stability 2025-01-28T14:48:26.308Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: aborting pipeline replication: peer="{Voter server-8818d339-8684-0522-92a9-44a956aa148f 8818d339-8684-0522-92a9-44a956aa148f}" 2025-01-28T14:48:26.308Z [INFO] server-e7cf8dca-da8d-b7f3-98d1-7bb15b3994b1: aborting pipeline replication: peer="{Voter server-d6b9257c-51c8-af4e-866d-554a5007d821 d6b9257c-51c8-af4e-866d-554a5007d821}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false 2025-01-28T14:48:26.322Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ff64675-a041-218a-bfb7-7309389e7ef0 Address:2ff64675-a041-218a-bfb7-7309389e7ef0} {Suffrage:Voter ID:server-385b77c0-7d33-6911-3190-c3ba095b5eda Address:385b77c0-7d33-6911-3190-c3ba095b5eda} {Suffrage:Voter ID:server-eb806551-8db7-5a37-e142-e2f33f3531cd Address:eb806551-8db7-5a37-e142-e2f33f3531cd}]" 2025-01-28T14:48:26.322Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: entering follower state: follower="Node at 2ff64675-a041-218a-bfb7-7309389e7ef0 [Follower]" leader-address= leader-id= 2025-01-28T14:48:26.322Z [INFO] server-385b77c0-7d33-6911-3190-c3ba095b5eda: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ff64675-a041-218a-bfb7-7309389e7ef0 Address:2ff64675-a041-218a-bfb7-7309389e7ef0} {Suffrage:Voter ID:server-385b77c0-7d33-6911-3190-c3ba095b5eda Address:385b77c0-7d33-6911-3190-c3ba095b5eda} {Suffrage:Voter ID:server-eb806551-8db7-5a37-e142-e2f33f3531cd Address:eb806551-8db7-5a37-e142-e2f33f3531cd}]" 2025-01-28T14:48:26.323Z [INFO] server-385b77c0-7d33-6911-3190-c3ba095b5eda: entering follower state: follower="Node at 385b77c0-7d33-6911-3190-c3ba095b5eda [Follower]" leader-address= leader-id= 2025-01-28T14:48:26.323Z [INFO] server-eb806551-8db7-5a37-e142-e2f33f3531cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ff64675-a041-218a-bfb7-7309389e7ef0 Address:2ff64675-a041-218a-bfb7-7309389e7ef0} {Suffrage:Voter ID:server-385b77c0-7d33-6911-3190-c3ba095b5eda Address:385b77c0-7d33-6911-3190-c3ba095b5eda} {Suffrage:Voter ID:server-eb806551-8db7-5a37-e142-e2f33f3531cd Address:eb806551-8db7-5a37-e142-e2f33f3531cd}]" 2025-01-28T14:48:26.323Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:26.323Z [INFO] server-eb806551-8db7-5a37-e142-e2f33f3531cd: entering follower state: follower="Node at eb806551-8db7-5a37-e142-e2f33f3531cd [Follower]" leader-address= leader-id= 2025-01-28T14:48:26.874Z [WARN] server-2ff64675-a041-218a-bfb7-7309389e7ef0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:26.874Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: entering candidate state: node="Node at 2ff64675-a041-218a-bfb7-7309389e7ef0 [Candidate]" term=2 2025-01-28T14:48:26.875Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: election won: term=2 tally=2 2025-01-28T14:48:26.875Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: entering leader state: leader="Node at 2ff64675-a041-218a-bfb7-7309389e7ef0 [Leader]" 2025-01-28T14:48:26.875Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: added peer, starting replication: peer=server-385b77c0-7d33-6911-3190-c3ba095b5eda 2025-01-28T14:48:26.875Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: added peer, starting replication: peer=server-eb806551-8db7-5a37-e142-e2f33f3531cd 2025-01-28T14:48:26.876Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: pipelining replication: peer="{Voter server-385b77c0-7d33-6911-3190-c3ba095b5eda 385b77c0-7d33-6911-3190-c3ba095b5eda}" 2025-01-28T14:48:26.876Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: pipelining replication: peer="{Voter server-eb806551-8db7-5a37-e142-e2f33f3531cd eb806551-8db7-5a37-e142-e2f33f3531cd}" 2025-01-28T14:48:27.881Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:26.876171701 +0000 UTC m=+89.959576725 (1 nodes), 552.561075ms from start of poll, 554.760141ms from cluster start. Timeout at 2025-01-28 14:48:27.881195035 +0000 UTC m=+90.964600059, 1.005023334s after stability 2025-01-28T14:48:27.886Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: starting snapshot up to: index=12 2025-01-28T14:48:27.886Z [INFO] creating new snapshot: path=/tmp/raft2936564990/snapshots/2-12-1738075707886.tmp 2025-01-28T14:48:27.890Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: snapshot complete up to: index=12 2025-01-28T14:48:27.896Z [INFO] creating new snapshot: path=/tmp/raft2936564990/snapshots/2-10013-1738075707896.tmp 2025-01-28T14:48:27.899Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: copied to local snapshot: bytes=71 2025-01-28T14:48:27.901Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: snapshot restore progress: id=2-10013-1738075707896 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:27.901Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: removing all old logs from log store 2025-01-28T14:48:27.901Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: compacting logs: from=1 to=22 2025-01-28T14:48:27.901Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: restored user snapshot: index=10013 2025-01-28T14:48:27.901Z [ERROR] server-2ff64675-a041-218a-bfb7-7309389e7ef0: failed to get log: index=22 error="log not found" 2025-01-28T14:48:27.901Z [ERROR] server-2ff64675-a041-218a-bfb7-7309389e7ef0: failed to get log: index=22 error="log not found" 2025-01-28T14:48:27.902Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: aborting pipeline replication: peer="{Voter server-385b77c0-7d33-6911-3190-c3ba095b5eda 385b77c0-7d33-6911-3190-c3ba095b5eda}" 2025-01-28T14:48:27.902Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: aborting pipeline replication: peer="{Voter server-eb806551-8db7-5a37-e142-e2f33f3531cd eb806551-8db7-5a37-e142-e2f33f3531cd}" 2025-01-28T14:48:27.909Z [ERROR] server-2ff64675-a041-218a-bfb7-7309389e7ef0: failed to get log: index=22 error="log not found" 2025-01-28T14:48:27.910Z [ERROR] server-2ff64675-a041-218a-bfb7-7309389e7ef0: failed to get log: index=22 error="log not found" 2025-01-28T14:48:27.911Z [INFO] creating new snapshot: path=/tmp/raft1699396691/snapshots/2-10013-1738075707911.tmp 2025-01-28T14:48:27.913Z [INFO] creating new snapshot: path=/tmp/raft3793172068/snapshots/2-10013-1738075707913.tmp 2025-01-28T14:48:27.914Z [INFO] server-385b77c0-7d33-6911-3190-c3ba095b5eda: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:27.915Z [INFO] server-eb806551-8db7-5a37-e142-e2f33f3531cd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:27.917Z [INFO] server-385b77c0-7d33-6911-3190-c3ba095b5eda: copied to local snapshot: bytes=71 2025-01-28T14:48:27.918Z [INFO] server-eb806551-8db7-5a37-e142-e2f33f3531cd: copied to local snapshot: bytes=71 2025-01-28T14:48:27.918Z [INFO] server-385b77c0-7d33-6911-3190-c3ba095b5eda: snapshot restore progress: id=2-10013-1738075707911 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:27.918Z [INFO] server-385b77c0-7d33-6911-3190-c3ba095b5eda: removing all old logs from log store 2025-01-28T14:48:27.918Z [INFO] server-385b77c0-7d33-6911-3190-c3ba095b5eda: compacting logs: from=1 to=22 2025-01-28T14:48:27.918Z [INFO] server-385b77c0-7d33-6911-3190-c3ba095b5eda: Installed remote snapshot 2025-01-28T14:48:27.919Z [WARN] server-385b77c0-7d33-6911-3190-c3ba095b5eda: skipping application of old log: index=22 2025-01-28T14:48:27.919Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: pipelining replication: peer="{Voter server-385b77c0-7d33-6911-3190-c3ba095b5eda 385b77c0-7d33-6911-3190-c3ba095b5eda}" 2025-01-28T14:48:27.920Z [INFO] server-eb806551-8db7-5a37-e142-e2f33f3531cd: snapshot restore progress: id=2-10013-1738075707913 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:27.920Z [INFO] server-eb806551-8db7-5a37-e142-e2f33f3531cd: removing all old logs from log store 2025-01-28T14:48:27.920Z [INFO] server-eb806551-8db7-5a37-e142-e2f33f3531cd: compacting logs: from=1 to=22 2025-01-28T14:48:27.920Z [INFO] server-eb806551-8db7-5a37-e142-e2f33f3531cd: Installed remote snapshot 2025-01-28T14:48:27.920Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:27.921Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: pipelining replication: peer="{Voter server-eb806551-8db7-5a37-e142-e2f33f3531cd eb806551-8db7-5a37-e142-e2f33f3531cd}" 2025-01-28T14:48:28.926Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:27.920472046 +0000 UTC m=+91.003877404 (2 nodes), 8.667µs from start of poll, 1.59906082s from cluster start. Timeout at 2025-01-28 14:48:28.925782056 +0000 UTC m=+92.009187080, 1.005309676s after stability 2025-01-28T14:48:28.951Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: aborting pipeline replication: peer="{Voter server-eb806551-8db7-5a37-e142-e2f33f3531cd eb806551-8db7-5a37-e142-e2f33f3531cd}" 2025-01-28T14:48:28.951Z [INFO] server-2ff64675-a041-218a-bfb7-7309389e7ef0: aborting pipeline replication: peer="{Voter server-385b77c0-7d33-6911-3190-c3ba095b5eda 385b77c0-7d33-6911-3190-c3ba095b5eda}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true 2025-01-28T14:48:28.963Z [INFO] server-d367fa51-6540-caf5-9e58-22323dcb89ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d367fa51-6540-caf5-9e58-22323dcb89ff Address:d367fa51-6540-caf5-9e58-22323dcb89ff} {Suffrage:Voter ID:server-0682e837-5360-7fa3-5476-d534c6de53c2 Address:0682e837-5360-7fa3-5476-d534c6de53c2} {Suffrage:Voter ID:server-db5e5232-35e9-1d78-0360-2dc309460a53 Address:db5e5232-35e9-1d78-0360-2dc309460a53}]" 2025-01-28T14:48:28.963Z [INFO] server-0682e837-5360-7fa3-5476-d534c6de53c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d367fa51-6540-caf5-9e58-22323dcb89ff Address:d367fa51-6540-caf5-9e58-22323dcb89ff} {Suffrage:Voter ID:server-0682e837-5360-7fa3-5476-d534c6de53c2 Address:0682e837-5360-7fa3-5476-d534c6de53c2} {Suffrage:Voter ID:server-db5e5232-35e9-1d78-0360-2dc309460a53 Address:db5e5232-35e9-1d78-0360-2dc309460a53}]" 2025-01-28T14:48:28.963Z [INFO] server-d367fa51-6540-caf5-9e58-22323dcb89ff: entering follower state: follower="Node at d367fa51-6540-caf5-9e58-22323dcb89ff [Follower]" leader-address= leader-id= 2025-01-28T14:48:28.964Z [INFO] server-0682e837-5360-7fa3-5476-d534c6de53c2: entering follower state: follower="Node at 0682e837-5360-7fa3-5476-d534c6de53c2 [Follower]" leader-address= leader-id= 2025-01-28T14:48:28.964Z [INFO] server-db5e5232-35e9-1d78-0360-2dc309460a53: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d367fa51-6540-caf5-9e58-22323dcb89ff Address:d367fa51-6540-caf5-9e58-22323dcb89ff} {Suffrage:Voter ID:server-0682e837-5360-7fa3-5476-d534c6de53c2 Address:0682e837-5360-7fa3-5476-d534c6de53c2} {Suffrage:Voter ID:server-db5e5232-35e9-1d78-0360-2dc309460a53 Address:db5e5232-35e9-1d78-0360-2dc309460a53}]" 2025-01-28T14:48:28.964Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:28.964Z [INFO] server-db5e5232-35e9-1d78-0360-2dc309460a53: entering follower state: follower="Node at db5e5232-35e9-1d78-0360-2dc309460a53 [Follower]" leader-address= leader-id= 2025-01-28T14:48:29.490Z [WARN] server-d367fa51-6540-caf5-9e58-22323dcb89ff: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:29.491Z [INFO] server-d367fa51-6540-caf5-9e58-22323dcb89ff: entering candidate state: node="Node at d367fa51-6540-caf5-9e58-22323dcb89ff [Candidate]" term=2 2025-01-28T14:48:29.491Z [INFO] server-d367fa51-6540-caf5-9e58-22323dcb89ff: election won: term=2 tally=2 2025-01-28T14:48:29.491Z [INFO] server-d367fa51-6540-caf5-9e58-22323dcb89ff: entering leader state: leader="Node at d367fa51-6540-caf5-9e58-22323dcb89ff [Leader]" 2025-01-28T14:48:29.491Z [INFO] server-d367fa51-6540-caf5-9e58-22323dcb89ff: added peer, starting replication: peer=server-0682e837-5360-7fa3-5476-d534c6de53c2 2025-01-28T14:48:29.491Z [INFO] server-d367fa51-6540-caf5-9e58-22323dcb89ff: added peer, starting replication: peer=server-db5e5232-35e9-1d78-0360-2dc309460a53 2025-01-28T14:48:29.492Z [INFO] server-d367fa51-6540-caf5-9e58-22323dcb89ff: pipelining replication: peer="{Voter server-0682e837-5360-7fa3-5476-d534c6de53c2 0682e837-5360-7fa3-5476-d534c6de53c2}" 2025-01-28T14:48:29.492Z [INFO] server-d367fa51-6540-caf5-9e58-22323dcb89ff: pipelining replication: peer="{Voter server-db5e5232-35e9-1d78-0360-2dc309460a53 db5e5232-35e9-1d78-0360-2dc309460a53}" 2025-01-28T14:48:30.497Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:29.492048717 +0000 UTC m=+92.575453741 (1 nodes), 527.387662ms from start of poll, 529.401055ms from cluster start. Timeout at 2025-01-28 14:48:30.497092718 +0000 UTC m=+93.580498742, 1.005045001s after stability 2025-01-28T14:48:30.501Z [INFO] server-d367fa51-6540-caf5-9e58-22323dcb89ff: starting snapshot up to: index=12 2025-01-28T14:48:30.501Z [INFO] creating new snapshot: path=/tmp/raft3576803583/snapshots/2-12-1738075710501.tmp 2025-01-28T14:48:30.507Z [INFO] server-d367fa51-6540-caf5-9e58-22323dcb89ff: snapshot complete up to: index=12 2025-01-28T14:48:30.532Z [INFO] server-a007de91-f4df-05bf-e6be-220d9180296c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a007de91-f4df-05bf-e6be-220d9180296c Address:a007de91-f4df-05bf-e6be-220d9180296c} {Suffrage:Voter ID:server-6d74e363-32a7-7140-f5d0-2908e65ca9ed Address:6d74e363-32a7-7140-f5d0-2908e65ca9ed} {Suffrage:Voter ID:server-be78217d-49d1-f754-b626-ffbbdcdb9017 Address:be78217d-49d1-f754-b626-ffbbdcdb9017}]" 2025-01-28T14:48:30.532Z [INFO] server-a007de91-f4df-05bf-e6be-220d9180296c: entering follower state: follower="Node at a007de91-f4df-05bf-e6be-220d9180296c [Follower]" leader-address= leader-id= 2025-01-28T14:48:30.532Z [INFO] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a007de91-f4df-05bf-e6be-220d9180296c Address:a007de91-f4df-05bf-e6be-220d9180296c} {Suffrage:Voter ID:server-6d74e363-32a7-7140-f5d0-2908e65ca9ed Address:6d74e363-32a7-7140-f5d0-2908e65ca9ed} {Suffrage:Voter ID:server-be78217d-49d1-f754-b626-ffbbdcdb9017 Address:be78217d-49d1-f754-b626-ffbbdcdb9017}]" 2025-01-28T14:48:30.533Z [INFO] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: entering follower state: follower="Node at 6d74e363-32a7-7140-f5d0-2908e65ca9ed [Follower]" leader-address= leader-id= 2025-01-28T14:48:30.533Z [INFO] server-be78217d-49d1-f754-b626-ffbbdcdb9017: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a007de91-f4df-05bf-e6be-220d9180296c Address:a007de91-f4df-05bf-e6be-220d9180296c} {Suffrage:Voter ID:server-6d74e363-32a7-7140-f5d0-2908e65ca9ed Address:6d74e363-32a7-7140-f5d0-2908e65ca9ed} {Suffrage:Voter ID:server-be78217d-49d1-f754-b626-ffbbdcdb9017 Address:be78217d-49d1-f754-b626-ffbbdcdb9017}]" 2025-01-28T14:48:30.533Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:30.533Z [INFO] server-be78217d-49d1-f754-b626-ffbbdcdb9017: entering follower state: follower="Node at be78217d-49d1-f754-b626-ffbbdcdb9017 [Follower]" leader-address= leader-id= 2025-01-28T14:48:31.082Z [WARN] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:31.082Z [INFO] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: entering candidate state: node="Node at 6d74e363-32a7-7140-f5d0-2908e65ca9ed [Candidate]" term=2 2025-01-28T14:48:31.083Z [INFO] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: election won: term=2 tally=2 2025-01-28T14:48:31.083Z [INFO] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: entering leader state: leader="Node at 6d74e363-32a7-7140-f5d0-2908e65ca9ed [Leader]" 2025-01-28T14:48:31.083Z [INFO] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: added peer, starting replication: peer=server-a007de91-f4df-05bf-e6be-220d9180296c 2025-01-28T14:48:31.083Z [INFO] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: added peer, starting replication: peer=server-be78217d-49d1-f754-b626-ffbbdcdb9017 2025-01-28T14:48:31.084Z [INFO] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: pipelining replication: peer="{Voter server-a007de91-f4df-05bf-e6be-220d9180296c a007de91-f4df-05bf-e6be-220d9180296c}" 2025-01-28T14:48:31.084Z [INFO] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: pipelining replication: peer="{Voter server-be78217d-49d1-f754-b626-ffbbdcdb9017 be78217d-49d1-f754-b626-ffbbdcdb9017}" 2025-01-28T14:48:32.090Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:31.083532978 +0000 UTC m=+94.166938003 (1 nodes), 549.793661ms from start of poll, 551.984059ms from cluster start. Timeout at 2025-01-28 14:48:32.088948324 +0000 UTC m=+95.172355015, 1.005417012s after stability 2025-01-28T14:48:32.091Z [INFO] creating new snapshot: path=/tmp/raft3162444630/snapshots/2-10013-1738075712091.tmp 2025-01-28T14:48:32.095Z [INFO] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: copied to local snapshot: bytes=71 2025-01-28T14:48:32.097Z [INFO] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: snapshot restore progress: id=2-10013-1738075712091 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:32.097Z [INFO] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: restored user snapshot: index=10013 2025-01-28T14:48:32.097Z [ERROR] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: failed to get log: index=3 error="log not found" 2025-01-28T14:48:32.097Z [INFO] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: aborting pipeline replication: peer="{Voter server-a007de91-f4df-05bf-e6be-220d9180296c a007de91-f4df-05bf-e6be-220d9180296c}" 2025-01-28T14:48:32.097Z [ERROR] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: failed to get log: index=3 error="log not found" 2025-01-28T14:48:32.098Z [INFO] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: aborting pipeline replication: peer="{Voter server-be78217d-49d1-f754-b626-ffbbdcdb9017 be78217d-49d1-f754-b626-ffbbdcdb9017}" 2025-01-28T14:48:32.107Z [ERROR] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: failed to get log: index=3 error="log not found" 2025-01-28T14:48:32.109Z [INFO] creating new snapshot: path=/tmp/raft75711000/snapshots/2-10013-1738075712109.tmp 2025-01-28T14:48:32.110Z [ERROR] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: failed to get log: index=3 error="log not found" 2025-01-28T14:48:32.112Z [INFO] server-be78217d-49d1-f754-b626-ffbbdcdb9017: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:32.113Z [INFO] creating new snapshot: path=/tmp/raft74456927/snapshots/2-10013-1738075712113.tmp 2025-01-28T14:48:32.114Z [INFO] server-be78217d-49d1-f754-b626-ffbbdcdb9017: copied to local snapshot: bytes=71 2025-01-28T14:48:32.115Z [INFO] server-a007de91-f4df-05bf-e6be-220d9180296c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:32.116Z [INFO] server-be78217d-49d1-f754-b626-ffbbdcdb9017: snapshot restore progress: id=2-10013-1738075712109 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:32.116Z [INFO] server-be78217d-49d1-f754-b626-ffbbdcdb9017: Installed remote snapshot 2025-01-28T14:48:32.117Z [INFO] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: pipelining replication: peer="{Voter server-be78217d-49d1-f754-b626-ffbbdcdb9017 be78217d-49d1-f754-b626-ffbbdcdb9017}" 2025-01-28T14:48:32.117Z [INFO] server-a007de91-f4df-05bf-e6be-220d9180296c: copied to local snapshot: bytes=71 2025-01-28T14:48:32.118Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:32.118Z [INFO] server-a007de91-f4df-05bf-e6be-220d9180296c: snapshot restore progress: id=2-10013-1738075712113 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:32.119Z [INFO] server-a007de91-f4df-05bf-e6be-220d9180296c: Installed remote snapshot 2025-01-28T14:48:32.119Z [INFO] server-6d74e363-32a7-7140-f5d0-2908e65ca9ed: pipelining replication: peer="{Voter server-a007de91-f4df-05bf-e6be-220d9180296c a007de91-f4df-05bf-e6be-220d9180296c}" 2025-01-28T14:48:33.123Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:32.118172037 +0000 UTC m=+95.201577062 (2 nodes), 7.334µs from start of poll, 1.586623118s from cluster start. Timeout at 2025-01-28 14:48:33.123543382 +0000 UTC m=+96.206948406, 1.005371344s after stability 2025-01-28T14:48:33.139Z [INFO] server-d367fa51-6540-caf5-9e58-22323dcb89ff: aborting pipeline replication: peer="{Voter server-db5e5232-35e9-1d78-0360-2dc309460a53 db5e5232-35e9-1d78-0360-2dc309460a53}" 2025-01-28T14:48:33.140Z [INFO] server-d367fa51-6540-caf5-9e58-22323dcb89ff: aborting pipeline replication: peer="{Voter server-0682e837-5360-7fa3-5476-d534c6de53c2 0682e837-5360-7fa3-5476-d534c6de53c2}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true 2025-01-28T14:48:33.151Z [INFO] server-bc82fe28-6aeb-0a59-7442-be916a5753a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc82fe28-6aeb-0a59-7442-be916a5753a8 Address:bc82fe28-6aeb-0a59-7442-be916a5753a8} {Suffrage:Voter ID:server-77fc1848-4a8d-31ff-3deb-7368d978c495 Address:77fc1848-4a8d-31ff-3deb-7368d978c495} {Suffrage:Voter ID:server-6687842b-c9af-9218-8782-c176f6d73242 Address:6687842b-c9af-9218-8782-c176f6d73242}]" 2025-01-28T14:48:33.152Z [INFO] server-bc82fe28-6aeb-0a59-7442-be916a5753a8: entering follower state: follower="Node at bc82fe28-6aeb-0a59-7442-be916a5753a8 [Follower]" leader-address= leader-id= 2025-01-28T14:48:33.152Z [INFO] server-77fc1848-4a8d-31ff-3deb-7368d978c495: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc82fe28-6aeb-0a59-7442-be916a5753a8 Address:bc82fe28-6aeb-0a59-7442-be916a5753a8} {Suffrage:Voter ID:server-77fc1848-4a8d-31ff-3deb-7368d978c495 Address:77fc1848-4a8d-31ff-3deb-7368d978c495} {Suffrage:Voter ID:server-6687842b-c9af-9218-8782-c176f6d73242 Address:6687842b-c9af-9218-8782-c176f6d73242}]" 2025-01-28T14:48:33.153Z [INFO] server-77fc1848-4a8d-31ff-3deb-7368d978c495: entering follower state: follower="Node at 77fc1848-4a8d-31ff-3deb-7368d978c495 [Follower]" leader-address= leader-id= 2025-01-28T14:48:33.153Z [INFO] server-6687842b-c9af-9218-8782-c176f6d73242: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc82fe28-6aeb-0a59-7442-be916a5753a8 Address:bc82fe28-6aeb-0a59-7442-be916a5753a8} {Suffrage:Voter ID:server-77fc1848-4a8d-31ff-3deb-7368d978c495 Address:77fc1848-4a8d-31ff-3deb-7368d978c495} {Suffrage:Voter ID:server-6687842b-c9af-9218-8782-c176f6d73242 Address:6687842b-c9af-9218-8782-c176f6d73242}]" 2025-01-28T14:48:33.153Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:33.153Z [INFO] server-6687842b-c9af-9218-8782-c176f6d73242: entering follower state: follower="Node at 6687842b-c9af-9218-8782-c176f6d73242 [Follower]" leader-address= leader-id= 2025-01-28T14:48:33.856Z [WARN] server-bc82fe28-6aeb-0a59-7442-be916a5753a8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:33.856Z [INFO] server-bc82fe28-6aeb-0a59-7442-be916a5753a8: entering candidate state: node="Node at bc82fe28-6aeb-0a59-7442-be916a5753a8 [Candidate]" term=2 2025-01-28T14:48:33.857Z [INFO] server-bc82fe28-6aeb-0a59-7442-be916a5753a8: election won: term=2 tally=2 2025-01-28T14:48:33.857Z [INFO] server-bc82fe28-6aeb-0a59-7442-be916a5753a8: entering leader state: leader="Node at bc82fe28-6aeb-0a59-7442-be916a5753a8 [Leader]" 2025-01-28T14:48:33.857Z [INFO] server-bc82fe28-6aeb-0a59-7442-be916a5753a8: added peer, starting replication: peer=server-77fc1848-4a8d-31ff-3deb-7368d978c495 2025-01-28T14:48:33.857Z [INFO] server-bc82fe28-6aeb-0a59-7442-be916a5753a8: added peer, starting replication: peer=server-6687842b-c9af-9218-8782-c176f6d73242 2025-01-28T14:48:33.858Z [INFO] server-bc82fe28-6aeb-0a59-7442-be916a5753a8: pipelining replication: peer="{Voter server-6687842b-c9af-9218-8782-c176f6d73242 6687842b-c9af-9218-8782-c176f6d73242}" 2025-01-28T14:48:33.858Z [INFO] server-bc82fe28-6aeb-0a59-7442-be916a5753a8: pipelining replication: peer="{Voter server-77fc1848-4a8d-31ff-3deb-7368d978c495 77fc1848-4a8d-31ff-3deb-7368d978c495}" 2025-01-28T14:48:34.863Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:33.857351685 +0000 UTC m=+96.940756710 (1 nodes), 703.526893ms from start of poll, 706.008967ms from cluster start. Timeout at 2025-01-28 14:48:34.862363352 +0000 UTC m=+97.945768043, 1.005011333s after stability 2025-01-28T14:48:34.886Z [INFO] server-bc82fe28-6aeb-0a59-7442-be916a5753a8: starting snapshot up to: index=12 2025-01-28T14:48:34.886Z [INFO] creating new snapshot: path=/tmp/raft3590269609/snapshots/2-12-1738075714886.tmp 2025-01-28T14:48:34.890Z [INFO] server-bc82fe28-6aeb-0a59-7442-be916a5753a8: snapshot complete up to: index=12 2025-01-28T14:48:34.899Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7cb18d25-4f78-51d1-3966-08f87c6cf016 Address:7cb18d25-4f78-51d1-3966-08f87c6cf016} {Suffrage:Voter ID:server-fe60e4bd-553c-0f14-8d47-afd82330b409 Address:fe60e4bd-553c-0f14-8d47-afd82330b409} {Suffrage:Voter ID:server-9e6b65ef-a63b-e514-909a-c8207b4711d6 Address:9e6b65ef-a63b-e514-909a-c8207b4711d6}]" 2025-01-28T14:48:34.900Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: entering follower state: follower="Node at 7cb18d25-4f78-51d1-3966-08f87c6cf016 [Follower]" leader-address= leader-id= 2025-01-28T14:48:34.900Z [INFO] server-fe60e4bd-553c-0f14-8d47-afd82330b409: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7cb18d25-4f78-51d1-3966-08f87c6cf016 Address:7cb18d25-4f78-51d1-3966-08f87c6cf016} {Suffrage:Voter ID:server-fe60e4bd-553c-0f14-8d47-afd82330b409 Address:fe60e4bd-553c-0f14-8d47-afd82330b409} {Suffrage:Voter ID:server-9e6b65ef-a63b-e514-909a-c8207b4711d6 Address:9e6b65ef-a63b-e514-909a-c8207b4711d6}]" 2025-01-28T14:48:34.901Z [INFO] server-fe60e4bd-553c-0f14-8d47-afd82330b409: entering follower state: follower="Node at fe60e4bd-553c-0f14-8d47-afd82330b409 [Follower]" leader-address= leader-id= 2025-01-28T14:48:34.901Z [INFO] server-9e6b65ef-a63b-e514-909a-c8207b4711d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7cb18d25-4f78-51d1-3966-08f87c6cf016 Address:7cb18d25-4f78-51d1-3966-08f87c6cf016} {Suffrage:Voter ID:server-fe60e4bd-553c-0f14-8d47-afd82330b409 Address:fe60e4bd-553c-0f14-8d47-afd82330b409} {Suffrage:Voter ID:server-9e6b65ef-a63b-e514-909a-c8207b4711d6 Address:9e6b65ef-a63b-e514-909a-c8207b4711d6}]" 2025-01-28T14:48:34.901Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:34.901Z [INFO] server-9e6b65ef-a63b-e514-909a-c8207b4711d6: entering follower state: follower="Node at 9e6b65ef-a63b-e514-909a-c8207b4711d6 [Follower]" leader-address= leader-id= 2025-01-28T14:48:35.461Z [WARN] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:35.462Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: entering candidate state: node="Node at 7cb18d25-4f78-51d1-3966-08f87c6cf016 [Candidate]" term=2 2025-01-28T14:48:35.462Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: election won: term=2 tally=2 2025-01-28T14:48:35.462Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: entering leader state: leader="Node at 7cb18d25-4f78-51d1-3966-08f87c6cf016 [Leader]" 2025-01-28T14:48:35.463Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: added peer, starting replication: peer=server-fe60e4bd-553c-0f14-8d47-afd82330b409 2025-01-28T14:48:35.463Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: added peer, starting replication: peer=server-9e6b65ef-a63b-e514-909a-c8207b4711d6 2025-01-28T14:48:35.464Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: pipelining replication: peer="{Voter server-9e6b65ef-a63b-e514-909a-c8207b4711d6 9e6b65ef-a63b-e514-909a-c8207b4711d6}" 2025-01-28T14:48:35.464Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: pipelining replication: peer="{Voter server-fe60e4bd-553c-0f14-8d47-afd82330b409 fe60e4bd-553c-0f14-8d47-afd82330b409}" 2025-01-28T14:48:36.470Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:35.464354742 +0000 UTC m=+98.547759767 (1 nodes), 562.698044ms from start of poll, 565.01978ms from cluster start. Timeout at 2025-01-28 14:48:36.469400078 +0000 UTC m=+99.552804769, 1.005045002s after stability 2025-01-28T14:48:36.485Z [INFO] creating new snapshot: path=/tmp/raft2220644668/snapshots/2-10013-1738075716485.tmp 2025-01-28T14:48:36.489Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: copied to local snapshot: bytes=71 2025-01-28T14:48:36.491Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: snapshot restore progress: id=2-10013-1738075716485 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:36.491Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: removing all old logs from log store 2025-01-28T14:48:36.491Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: compacting logs: from=1 to=2 2025-01-28T14:48:36.491Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: restored user snapshot: index=10013 2025-01-28T14:48:36.491Z [ERROR] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: failed to get log: index=3 error="log not found" 2025-01-28T14:48:36.491Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: aborting pipeline replication: peer="{Voter server-fe60e4bd-553c-0f14-8d47-afd82330b409 fe60e4bd-553c-0f14-8d47-afd82330b409}" 2025-01-28T14:48:36.492Z [ERROR] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: failed to get log: index=2 error="log not found" 2025-01-28T14:48:36.492Z [ERROR] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: failed to get log: index=2 error="log not found" 2025-01-28T14:48:36.492Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: aborting pipeline replication: peer="{Voter server-9e6b65ef-a63b-e514-909a-c8207b4711d6 9e6b65ef-a63b-e514-909a-c8207b4711d6}" 2025-01-28T14:48:36.494Z [INFO] creating new snapshot: path=/tmp/raft1505187362/snapshots/2-10013-1738075716494.tmp 2025-01-28T14:48:36.495Z [INFO] server-fe60e4bd-553c-0f14-8d47-afd82330b409: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:36.498Z [INFO] server-fe60e4bd-553c-0f14-8d47-afd82330b409: copied to local snapshot: bytes=71 2025-01-28T14:48:36.500Z [INFO] server-fe60e4bd-553c-0f14-8d47-afd82330b409: snapshot restore progress: id=2-10013-1738075716494 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:36.500Z [INFO] server-fe60e4bd-553c-0f14-8d47-afd82330b409: removing all old logs from log store 2025-01-28T14:48:36.500Z [INFO] server-fe60e4bd-553c-0f14-8d47-afd82330b409: compacting logs: from=1 to=2 2025-01-28T14:48:36.500Z [INFO] server-fe60e4bd-553c-0f14-8d47-afd82330b409: Installed remote snapshot 2025-01-28T14:48:36.500Z [ERROR] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: failed to get log: index=2 error="log not found" 2025-01-28T14:48:36.501Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: pipelining replication: peer="{Voter server-fe60e4bd-553c-0f14-8d47-afd82330b409 fe60e4bd-553c-0f14-8d47-afd82330b409}" 2025-01-28T14:48:36.502Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:36.503Z [INFO] creating new snapshot: path=/tmp/raft1641485788/snapshots/2-10013-1738075716503.tmp 2025-01-28T14:48:36.505Z [INFO] server-9e6b65ef-a63b-e514-909a-c8207b4711d6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:36.508Z [INFO] server-9e6b65ef-a63b-e514-909a-c8207b4711d6: copied to local snapshot: bytes=71 2025-01-28T14:48:36.509Z [INFO] server-9e6b65ef-a63b-e514-909a-c8207b4711d6: snapshot restore progress: id=2-10013-1738075716503 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2025-01-28T14:48:36.509Z [INFO] server-9e6b65ef-a63b-e514-909a-c8207b4711d6: removing all old logs from log store 2025-01-28T14:48:36.509Z [INFO] server-9e6b65ef-a63b-e514-909a-c8207b4711d6: compacting logs: from=1 to=2 2025-01-28T14:48:36.509Z [INFO] server-9e6b65ef-a63b-e514-909a-c8207b4711d6: Installed remote snapshot 2025-01-28T14:48:36.510Z [INFO] server-7cb18d25-4f78-51d1-3966-08f87c6cf016: pipelining replication: peer="{Voter server-9e6b65ef-a63b-e514-909a-c8207b4711d6 9e6b65ef-a63b-e514-909a-c8207b4711d6}" 2025-01-28T14:48:37.508Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:36.502789581 +0000 UTC m=+99.586194272 (2 nodes), 9µs from start of poll, 1.603454285s from cluster start. Timeout at 2025-01-28 14:48:37.508242261 +0000 UTC m=+100.591647286, 1.005453014s after stability 2025-01-28T14:48:37.523Z [INFO] server-bc82fe28-6aeb-0a59-7442-be916a5753a8: aborting pipeline replication: peer="{Voter server-77fc1848-4a8d-31ff-3deb-7368d978c495 77fc1848-4a8d-31ff-3deb-7368d978c495}" 2025-01-28T14:48:37.523Z [INFO] server-bc82fe28-6aeb-0a59-7442-be916a5753a8: aborting pipeline replication: peer="{Voter server-6687842b-c9af-9218-8782-c176f6d73242 6687842b-c9af-9218-8782-c176f6d73242}" --- PASS: TestRaft_UserRestore (84.10s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.66s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.71s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.39s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.25s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.85s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.70s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.23s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.16s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.86s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.69s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.38s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.17s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.68s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.68s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.22s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.32s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.88s) --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.64s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.37s) --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.35s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.66s) --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.64s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.19s) --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.38s) === RUN TestRaft_SendSnapshotFollower 2025-01-28T14:48:37.535Z [INFO] server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb Address:36dfa77f-42ea-aecc-e97a-0fe8582e3fdb} {Suffrage:Voter ID:server-7bacb526-80a8-a9cf-06f9-21febbffef8e Address:7bacb526-80a8-a9cf-06f9-21febbffef8e} {Suffrage:Voter ID:server-316ada7c-e3c1-50a4-7107-315421fc3f4f Address:316ada7c-e3c1-50a4-7107-315421fc3f4f}]" 2025-01-28T14:48:37.535Z [INFO] server-7bacb526-80a8-a9cf-06f9-21febbffef8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb Address:36dfa77f-42ea-aecc-e97a-0fe8582e3fdb} {Suffrage:Voter ID:server-7bacb526-80a8-a9cf-06f9-21febbffef8e Address:7bacb526-80a8-a9cf-06f9-21febbffef8e} {Suffrage:Voter ID:server-316ada7c-e3c1-50a4-7107-315421fc3f4f Address:316ada7c-e3c1-50a4-7107-315421fc3f4f}]" 2025-01-28T14:48:37.536Z [INFO] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb Address:36dfa77f-42ea-aecc-e97a-0fe8582e3fdb} {Suffrage:Voter ID:server-7bacb526-80a8-a9cf-06f9-21febbffef8e Address:7bacb526-80a8-a9cf-06f9-21febbffef8e} {Suffrage:Voter ID:server-316ada7c-e3c1-50a4-7107-315421fc3f4f Address:316ada7c-e3c1-50a4-7107-315421fc3f4f}]" 2025-01-28T14:48:37.536Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:37.536Z [INFO] server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb: entering follower state: follower="Node at 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb [Follower]" leader-address= leader-id= 2025-01-28T14:48:37.537Z [INFO] server-7bacb526-80a8-a9cf-06f9-21febbffef8e: entering follower state: follower="Node at 7bacb526-80a8-a9cf-06f9-21febbffef8e [Follower]" leader-address= leader-id= 2025-01-28T14:48:37.537Z [INFO] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: entering follower state: follower="Node at 316ada7c-e3c1-50a4-7107-315421fc3f4f [Follower]" leader-address= leader-id= 2025-01-28T14:48:37.591Z [WARN] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:37.591Z [INFO] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: entering candidate state: node="Node at 316ada7c-e3c1-50a4-7107-315421fc3f4f [Candidate]" term=2 2025-01-28T14:48:37.591Z [INFO] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: election won: term=2 tally=2 2025-01-28T14:48:37.591Z [INFO] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: entering leader state: leader="Node at 316ada7c-e3c1-50a4-7107-315421fc3f4f [Leader]" 2025-01-28T14:48:37.592Z [INFO] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: added peer, starting replication: peer=server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb 2025-01-28T14:48:37.592Z [INFO] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: added peer, starting replication: peer=server-7bacb526-80a8-a9cf-06f9-21febbffef8e 2025-01-28T14:48:37.593Z [INFO] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: pipelining replication: peer="{Voter server-7bacb526-80a8-a9cf-06f9-21febbffef8e 7bacb526-80a8-a9cf-06f9-21febbffef8e}" 2025-01-28T14:48:37.593Z [INFO] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: pipelining replication: peer="{Voter server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb}" 2025-01-28T14:48:37.698Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:37.592478942 +0000 UTC m=+100.675883966 (2 nodes), 55.722321ms from start of poll, 57.803383ms from cluster start. Timeout at 2025-01-28 14:48:37.697488609 +0000 UTC m=+100.780893634, 105.009668ms after stability 2025-01-28T14:48:37.698Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:37.805Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:37.698465423 +0000 UTC m=+100.781870447 (1 nodes), 6.667µs from start of poll, 163.789864ms from cluster start. Timeout at 2025-01-28 14:48:37.803892769 +0000 UTC m=+100.887298126, 105.427679ms after stability 2025-01-28T14:48:37.806Z [ERROR] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: failed to pipeline appendEntries: peer="{Voter server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb}" error="append pipeline closed" 2025-01-28T14:48:37.806Z [INFO] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: aborting pipeline replication: peer="{Voter server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb}" 2025-01-28T14:48:37.806Z [ERROR] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: failed to appendEntries to: peer="{Voter server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb}" error="failed to connect to peer: 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb" 2025-01-28T14:48:37.807Z [ERROR] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: failed to heartbeat to: peer=36dfa77f-42ea-aecc-e97a-0fe8582e3fdb backoff time=10ms error="failed to connect to peer: 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb" 2025-01-28T14:48:37.817Z [ERROR] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: failed to appendEntries to: peer="{Voter server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb}" error="failed to connect to peer: 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb" 2025-01-28T14:48:37.823Z [ERROR] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: failed to heartbeat to: peer=36dfa77f-42ea-aecc-e97a-0fe8582e3fdb backoff time=10ms error="failed to connect to peer: 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb" 2025-01-28T14:48:37.827Z [ERROR] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: failed to appendEntries to: peer="{Voter server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb}" error="failed to connect to peer: 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb" 2025-01-28T14:48:37.842Z [ERROR] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: failed to heartbeat to: peer=36dfa77f-42ea-aecc-e97a-0fe8582e3fdb backoff time=10ms error="failed to connect to peer: 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb" 2025-01-28T14:48:37.847Z [ERROR] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: failed to appendEntries to: peer="{Voter server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb}" error="failed to connect to peer: 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb" raft_test.go:1627: [INFO] Finished apply without behind follower 2025-01-28T14:48:37.855Z [ERROR] server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb: failed to take snapshot: error="nothing new to snapshot" 2025-01-28T14:48:37.855Z [INFO] server-7bacb526-80a8-a9cf-06f9-21febbffef8e: starting snapshot up to: index=101 2025-01-28T14:48:37.855Z [INFO] creating new snapshot: path=/tmp/raft854547184/snapshots/2-101-1738075717855.tmp 2025-01-28T14:48:37.859Z [INFO] server-7bacb526-80a8-a9cf-06f9-21febbffef8e: compacting logs: from=1 to=92 2025-01-28T14:48:37.859Z [INFO] server-7bacb526-80a8-a9cf-06f9-21febbffef8e: snapshot complete up to: index=101 2025-01-28T14:48:37.859Z [WARN] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: failed to contact: server-id=server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb time=54.792627ms 2025-01-28T14:48:37.860Z [ERROR] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: failed to heartbeat to: peer=36dfa77f-42ea-aecc-e97a-0fe8582e3fdb backoff time=20ms error="failed to connect to peer: 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb" 2025-01-28T14:48:37.860Z [INFO] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: starting snapshot up to: index=102 2025-01-28T14:48:37.860Z [INFO] creating new snapshot: path=/tmp/raft1106525746/snapshots/2-102-1738075717860.tmp 2025-01-28T14:48:37.864Z [INFO] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: compacting logs: from=1 to=92 2025-01-28T14:48:37.864Z [INFO] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: snapshot complete up to: index=102 2025-01-28T14:48:37.888Z [ERROR] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: failed to get log: index=2 error="log not found" 2025-01-28T14:48:37.890Z [INFO] creating new snapshot: path=/tmp/raft1923064124/snapshots/2-102-1738075717890.tmp 2025-01-28T14:48:37.892Z [INFO] server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2025-01-28T14:48:37.897Z [INFO] server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb: copied to local snapshot: bytes=693 2025-01-28T14:48:37.899Z [INFO] server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb: snapshot restore progress: id=2-102-1738075717890 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2025-01-28T14:48:37.899Z [INFO] server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb: Installed remote snapshot 2025-01-28T14:48:37.900Z [WARN] server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb: skipping application of old log: index=102 2025-01-28T14:48:37.900Z [INFO] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: pipelining replication: peer="{Voter server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb}" 2025-01-28T14:48:37.903Z [INFO] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: aborting pipeline replication: peer="{Voter server-7bacb526-80a8-a9cf-06f9-21febbffef8e 7bacb526-80a8-a9cf-06f9-21febbffef8e}" 2025-01-28T14:48:37.903Z [INFO] server-316ada7c-e3c1-50a4-7107-315421fc3f4f: aborting pipeline replication: peer="{Voter server-36dfa77f-42ea-aecc-e97a-0fe8582e3fdb 36dfa77f-42ea-aecc-e97a-0fe8582e3fdb}" --- PASS: TestRaft_SendSnapshotFollower (0.38s) === RUN TestRaft_SendSnapshotAndLogsFollower 2025-01-28T14:48:37.916Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d1decea-a770-faf8-210c-86a6e5d06545 Address:4d1decea-a770-faf8-210c-86a6e5d06545} {Suffrage:Voter ID:server-031e1864-785b-cf2a-7814-174409746957 Address:031e1864-785b-cf2a-7814-174409746957} {Suffrage:Voter ID:server-fa678abe-f03d-7e8f-0bf3-453214a9dba1 Address:fa678abe-f03d-7e8f-0bf3-453214a9dba1}]" 2025-01-28T14:48:37.917Z [INFO] server-031e1864-785b-cf2a-7814-174409746957: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d1decea-a770-faf8-210c-86a6e5d06545 Address:4d1decea-a770-faf8-210c-86a6e5d06545} {Suffrage:Voter ID:server-031e1864-785b-cf2a-7814-174409746957 Address:031e1864-785b-cf2a-7814-174409746957} {Suffrage:Voter ID:server-fa678abe-f03d-7e8f-0bf3-453214a9dba1 Address:fa678abe-f03d-7e8f-0bf3-453214a9dba1}]" 2025-01-28T14:48:37.917Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: entering follower state: follower="Node at 4d1decea-a770-faf8-210c-86a6e5d06545 [Follower]" leader-address= leader-id= 2025-01-28T14:48:37.918Z [INFO] server-031e1864-785b-cf2a-7814-174409746957: entering follower state: follower="Node at 031e1864-785b-cf2a-7814-174409746957 [Follower]" leader-address= leader-id= 2025-01-28T14:48:37.918Z [INFO] server-fa678abe-f03d-7e8f-0bf3-453214a9dba1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d1decea-a770-faf8-210c-86a6e5d06545 Address:4d1decea-a770-faf8-210c-86a6e5d06545} {Suffrage:Voter ID:server-031e1864-785b-cf2a-7814-174409746957 Address:031e1864-785b-cf2a-7814-174409746957} {Suffrage:Voter ID:server-fa678abe-f03d-7e8f-0bf3-453214a9dba1 Address:fa678abe-f03d-7e8f-0bf3-453214a9dba1}]" 2025-01-28T14:48:37.918Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:37.918Z [INFO] server-fa678abe-f03d-7e8f-0bf3-453214a9dba1: entering follower state: follower="Node at fa678abe-f03d-7e8f-0bf3-453214a9dba1 [Follower]" leader-address= leader-id= 2025-01-28T14:48:37.976Z [WARN] server-4d1decea-a770-faf8-210c-86a6e5d06545: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:37.977Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: entering candidate state: node="Node at 4d1decea-a770-faf8-210c-86a6e5d06545 [Candidate]" term=2 2025-01-28T14:48:37.977Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: election won: term=2 tally=2 2025-01-28T14:48:37.977Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: entering leader state: leader="Node at 4d1decea-a770-faf8-210c-86a6e5d06545 [Leader]" 2025-01-28T14:48:37.977Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: added peer, starting replication: peer=server-031e1864-785b-cf2a-7814-174409746957 2025-01-28T14:48:37.978Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: added peer, starting replication: peer=server-fa678abe-f03d-7e8f-0bf3-453214a9dba1 2025-01-28T14:48:37.979Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: pipelining replication: peer="{Voter server-fa678abe-f03d-7e8f-0bf3-453214a9dba1 fa678abe-f03d-7e8f-0bf3-453214a9dba1}" 2025-01-28T14:48:37.979Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: pipelining replication: peer="{Voter server-031e1864-785b-cf2a-7814-174409746957 031e1864-785b-cf2a-7814-174409746957}" 2025-01-28T14:48:38.084Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:37.979269428 +0000 UTC m=+101.062674119 (2 nodes), 60.556131ms from start of poll, 63.337881ms from cluster start. Timeout at 2025-01-28 14:48:38.084279762 +0000 UTC m=+101.167684453, 105.010334ms after stability 2025-01-28T14:48:38.084Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:38.192Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:38.084873565 +0000 UTC m=+101.168278256 (1 nodes), 6.667µs from start of poll, 168.942018ms from cluster start. Timeout at 2025-01-28 14:48:38.190349912 +0000 UTC m=+101.273754936, 105.47668ms after stability 2025-01-28T14:48:38.193Z [ERROR] server-4d1decea-a770-faf8-210c-86a6e5d06545: failed to pipeline appendEntries: peer="{Voter server-031e1864-785b-cf2a-7814-174409746957 031e1864-785b-cf2a-7814-174409746957}" error="append pipeline closed" 2025-01-28T14:48:38.193Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: aborting pipeline replication: peer="{Voter server-031e1864-785b-cf2a-7814-174409746957 031e1864-785b-cf2a-7814-174409746957}" 2025-01-28T14:48:38.193Z [ERROR] server-4d1decea-a770-faf8-210c-86a6e5d06545: failed to appendEntries to: peer="{Voter server-031e1864-785b-cf2a-7814-174409746957 031e1864-785b-cf2a-7814-174409746957}" error="failed to connect to peer: 031e1864-785b-cf2a-7814-174409746957" 2025-01-28T14:48:38.194Z [ERROR] server-4d1decea-a770-faf8-210c-86a6e5d06545: failed to heartbeat to: peer=031e1864-785b-cf2a-7814-174409746957 backoff time=10ms error="failed to connect to peer: 031e1864-785b-cf2a-7814-174409746957" 2025-01-28T14:48:38.204Z [ERROR] server-4d1decea-a770-faf8-210c-86a6e5d06545: failed to appendEntries to: peer="{Voter server-031e1864-785b-cf2a-7814-174409746957 031e1864-785b-cf2a-7814-174409746957}" error="failed to connect to peer: 031e1864-785b-cf2a-7814-174409746957" 2025-01-28T14:48:38.211Z [ERROR] server-4d1decea-a770-faf8-210c-86a6e5d06545: failed to heartbeat to: peer=031e1864-785b-cf2a-7814-174409746957 backoff time=10ms error="failed to connect to peer: 031e1864-785b-cf2a-7814-174409746957" 2025-01-28T14:48:38.214Z [ERROR] server-4d1decea-a770-faf8-210c-86a6e5d06545: failed to appendEntries to: peer="{Voter server-031e1864-785b-cf2a-7814-174409746957 031e1864-785b-cf2a-7814-174409746957}" error="failed to connect to peer: 031e1864-785b-cf2a-7814-174409746957" raft_test.go:1669: [INFO] Finished apply without behind follower 2025-01-28T14:48:38.227Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: starting snapshot up to: index=102 2025-01-28T14:48:38.228Z [INFO] creating new snapshot: path=/tmp/raft1686706354/snapshots/2-102-1738075718228.tmp 2025-01-28T14:48:38.228Z [ERROR] server-4d1decea-a770-faf8-210c-86a6e5d06545: failed to heartbeat to: peer=031e1864-785b-cf2a-7814-174409746957 backoff time=10ms error="failed to connect to peer: 031e1864-785b-cf2a-7814-174409746957" 2025-01-28T14:48:38.234Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: compacting logs: from=1 to=92 2025-01-28T14:48:38.234Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: snapshot complete up to: index=102 2025-01-28T14:48:38.236Z [ERROR] server-4d1decea-a770-faf8-210c-86a6e5d06545: failed to get log: index=2 error="log not found" 2025-01-28T14:48:38.243Z [ERROR] server-031e1864-785b-cf2a-7814-174409746957: failed to take snapshot: error="nothing new to snapshot" 2025-01-28T14:48:38.244Z [ERROR] server-4d1decea-a770-faf8-210c-86a6e5d06545: failed to install snapshot: id=2-102-1738075718228 error="failed to connect to peer: 031e1864-785b-cf2a-7814-174409746957" 2025-01-28T14:48:38.245Z [ERROR] server-4d1decea-a770-faf8-210c-86a6e5d06545: failed to send snapshot to: peer="{Voter server-031e1864-785b-cf2a-7814-174409746957 031e1864-785b-cf2a-7814-174409746957}" error="failed to connect to peer: 031e1864-785b-cf2a-7814-174409746957" 2025-01-28T14:48:38.247Z [WARN] server-4d1decea-a770-faf8-210c-86a6e5d06545: failed to contact: server-id=server-031e1864-785b-cf2a-7814-174409746957 time=59.91178ms 2025-01-28T14:48:38.249Z [INFO] server-fa678abe-f03d-7e8f-0bf3-453214a9dba1: starting snapshot up to: index=102 2025-01-28T14:48:38.249Z [INFO] creating new snapshot: path=/tmp/raft956992359/snapshots/2-102-1738075718249.tmp 2025-01-28T14:48:38.253Z [INFO] server-fa678abe-f03d-7e8f-0bf3-453214a9dba1: compacting logs: from=1 to=92 2025-01-28T14:48:38.253Z [INFO] server-fa678abe-f03d-7e8f-0bf3-453214a9dba1: snapshot complete up to: index=102 2025-01-28T14:48:38.257Z [ERROR] server-4d1decea-a770-faf8-210c-86a6e5d06545: failed to heartbeat to: peer=031e1864-785b-cf2a-7814-174409746957 backoff time=20ms error="failed to connect to peer: 031e1864-785b-cf2a-7814-174409746957" 2025-01-28T14:48:38.264Z [WARN] server-031e1864-785b-cf2a-7814-174409746957: heartbeat timeout reached, starting election: last-leader-addr=4d1decea-a770-faf8-210c-86a6e5d06545 last-leader-id=server-4d1decea-a770-faf8-210c-86a6e5d06545 2025-01-28T14:48:38.265Z [INFO] server-031e1864-785b-cf2a-7814-174409746957: entering candidate state: node="Node at 031e1864-785b-cf2a-7814-174409746957 [Candidate]" term=3 2025-01-28T14:48:38.265Z [ERROR] server-031e1864-785b-cf2a-7814-174409746957: failed to make requestVote RPC: target="{Voter server-fa678abe-f03d-7e8f-0bf3-453214a9dba1 fa678abe-f03d-7e8f-0bf3-453214a9dba1}" error="failed to connect to peer: fa678abe-f03d-7e8f-0bf3-453214a9dba1" term=3 2025-01-28T14:48:38.265Z [ERROR] server-031e1864-785b-cf2a-7814-174409746957: failed to make requestVote RPC: target="{Voter server-4d1decea-a770-faf8-210c-86a6e5d06545 4d1decea-a770-faf8-210c-86a6e5d06545}" error="failed to connect to peer: 4d1decea-a770-faf8-210c-86a6e5d06545" term=3 raft_test.go:1690: [INFO] Finished apply without behind follower 2025-01-28T14:48:38.285Z [ERROR] server-4d1decea-a770-faf8-210c-86a6e5d06545: failed to get log: index=2 error="log not found" 2025-01-28T14:48:38.286Z [INFO] server-031e1864-785b-cf2a-7814-174409746957: ignoring installSnapshot request with older term than current term: request-term=2 current-term=3 2025-01-28T14:48:38.287Z [ERROR] server-4d1decea-a770-faf8-210c-86a6e5d06545: peer has newer term, stopping replication: peer="{Voter server-031e1864-785b-cf2a-7814-174409746957 031e1864-785b-cf2a-7814-174409746957}" 2025-01-28T14:48:38.287Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: entering follower state: follower="Node at 4d1decea-a770-faf8-210c-86a6e5d06545 [Follower]" leader-address= leader-id= 2025-01-28T14:48:38.287Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: aborting pipeline replication: peer="{Voter server-fa678abe-f03d-7e8f-0bf3-453214a9dba1 fa678abe-f03d-7e8f-0bf3-453214a9dba1}" 2025-01-28T14:48:38.359Z [WARN] server-fa678abe-f03d-7e8f-0bf3-453214a9dba1: heartbeat timeout reached, starting election: last-leader-addr=4d1decea-a770-faf8-210c-86a6e5d06545 last-leader-id=server-4d1decea-a770-faf8-210c-86a6e5d06545 2025-01-28T14:48:38.359Z [INFO] server-fa678abe-f03d-7e8f-0bf3-453214a9dba1: entering candidate state: node="Node at fa678abe-f03d-7e8f-0bf3-453214a9dba1 [Candidate]" term=3 2025-01-28T14:48:38.361Z [WARN] server-4d1decea-a770-faf8-210c-86a6e5d06545: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:38.362Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: entering candidate state: node="Node at 4d1decea-a770-faf8-210c-86a6e5d06545 [Candidate]" term=3 2025-01-28T14:48:38.362Z [INFO] server-fa678abe-f03d-7e8f-0bf3-453214a9dba1: duplicate requestVote for same term: term=3 2025-01-28T14:48:38.362Z [INFO] server-031e1864-785b-cf2a-7814-174409746957: duplicate requestVote for same term: term=3 2025-01-28T14:48:38.363Z [INFO] server-031e1864-785b-cf2a-7814-174409746957: duplicate requestVote for same term: term=3 2025-01-28T14:48:38.363Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: duplicate requestVote for same term: term=3 2025-01-28T14:48:38.364Z [WARN] server-031e1864-785b-cf2a-7814-174409746957: Election timeout reached, restarting election 2025-01-28T14:48:38.364Z [INFO] server-031e1864-785b-cf2a-7814-174409746957: entering candidate state: node="Node at 031e1864-785b-cf2a-7814-174409746957 [Candidate]" term=4 2025-01-28T14:48:38.365Z [WARN] server-fa678abe-f03d-7e8f-0bf3-453214a9dba1: rejecting vote request since our last index is greater: candidate=031e1864-785b-cf2a-7814-174409746957 last-index=202 last-candidate-index=2 2025-01-28T14:48:38.365Z [INFO] server-fa678abe-f03d-7e8f-0bf3-453214a9dba1: entering follower state: follower="Node at fa678abe-f03d-7e8f-0bf3-453214a9dba1 [Follower]" leader-address= leader-id= 2025-01-28T14:48:38.366Z [WARN] server-4d1decea-a770-faf8-210c-86a6e5d06545: rejecting vote request since our last index is greater: candidate=031e1864-785b-cf2a-7814-174409746957 last-index=202 last-candidate-index=2 2025-01-28T14:48:38.366Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: entering follower state: follower="Node at 4d1decea-a770-faf8-210c-86a6e5d06545 [Follower]" leader-address= leader-id= 2025-01-28T14:48:38.439Z [WARN] server-031e1864-785b-cf2a-7814-174409746957: Election timeout reached, restarting election 2025-01-28T14:48:38.439Z [INFO] server-031e1864-785b-cf2a-7814-174409746957: entering candidate state: node="Node at 031e1864-785b-cf2a-7814-174409746957 [Candidate]" term=5 2025-01-28T14:48:38.439Z [WARN] server-4d1decea-a770-faf8-210c-86a6e5d06545: rejecting vote request since our last index is greater: candidate=031e1864-785b-cf2a-7814-174409746957 last-index=202 last-candidate-index=2 2025-01-28T14:48:38.439Z [WARN] server-fa678abe-f03d-7e8f-0bf3-453214a9dba1: rejecting vote request since our last index is greater: candidate=031e1864-785b-cf2a-7814-174409746957 last-index=202 last-candidate-index=2 2025-01-28T14:48:38.457Z [WARN] server-4d1decea-a770-faf8-210c-86a6e5d06545: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:38.457Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: entering candidate state: node="Node at 4d1decea-a770-faf8-210c-86a6e5d06545 [Candidate]" term=6 2025-01-28T14:48:38.458Z [INFO] server-031e1864-785b-cf2a-7814-174409746957: entering follower state: follower="Node at 031e1864-785b-cf2a-7814-174409746957 [Follower]" leader-address= leader-id= 2025-01-28T14:48:38.459Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: election won: term=6 tally=2 2025-01-28T14:48:38.459Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: entering leader state: leader="Node at 4d1decea-a770-faf8-210c-86a6e5d06545 [Leader]" 2025-01-28T14:48:38.459Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: added peer, starting replication: peer=server-031e1864-785b-cf2a-7814-174409746957 2025-01-28T14:48:38.459Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: added peer, starting replication: peer=server-fa678abe-f03d-7e8f-0bf3-453214a9dba1 2025-01-28T14:48:38.460Z [WARN] server-031e1864-785b-cf2a-7814-174409746957: failed to get previous log: previous-index=202 last-index=2 error="log not found" 2025-01-28T14:48:38.460Z [WARN] server-4d1decea-a770-faf8-210c-86a6e5d06545: appendEntries rejected, sending older logs: peer="{Voter server-031e1864-785b-cf2a-7814-174409746957 031e1864-785b-cf2a-7814-174409746957}" next=3 2025-01-28T14:48:38.460Z [ERROR] server-4d1decea-a770-faf8-210c-86a6e5d06545: failed to get log: index=2 error="log not found" 2025-01-28T14:48:38.461Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: pipelining replication: peer="{Voter server-fa678abe-f03d-7e8f-0bf3-453214a9dba1 fa678abe-f03d-7e8f-0bf3-453214a9dba1}" 2025-01-28T14:48:38.463Z [INFO] creating new snapshot: path=/tmp/raft931015257/snapshots/2-102-1738075718463.tmp 2025-01-28T14:48:38.465Z [INFO] server-031e1864-785b-cf2a-7814-174409746957: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2025-01-28T14:48:38.467Z [INFO] server-031e1864-785b-cf2a-7814-174409746957: copied to local snapshot: bytes=693 2025-01-28T14:48:38.470Z [INFO] server-031e1864-785b-cf2a-7814-174409746957: snapshot restore progress: id=2-102-1738075718463 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2025-01-28T14:48:38.470Z [INFO] server-031e1864-785b-cf2a-7814-174409746957: Installed remote snapshot 2025-01-28T14:48:38.472Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: pipelining replication: peer="{Voter server-031e1864-785b-cf2a-7814-174409746957 031e1864-785b-cf2a-7814-174409746957}" 2025-01-28T14:48:38.476Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: aborting pipeline replication: peer="{Voter server-031e1864-785b-cf2a-7814-174409746957 031e1864-785b-cf2a-7814-174409746957}" 2025-01-28T14:48:38.477Z [INFO] server-4d1decea-a770-faf8-210c-86a6e5d06545: aborting pipeline replication: peer="{Voter server-fa678abe-f03d-7e8f-0bf3-453214a9dba1 fa678abe-f03d-7e8f-0bf3-453214a9dba1}" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.59s) === RUN TestRaft_ReJoinFollower 2025-01-28T14:48:38.503Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-becdd36e-e96c-187b-3eb7-8a992d4eedfb Address:becdd36e-e96c-187b-3eb7-8a992d4eedfb} {Suffrage:Voter ID:server-04a0601a-25ab-1658-3c9e-4426f4ec72e4 Address:04a0601a-25ab-1658-3c9e-4426f4ec72e4} {Suffrage:Voter ID:server-ac425c94-2ab9-dbef-5131-ef711499ccef Address:ac425c94-2ab9-dbef-5131-ef711499ccef}]" 2025-01-28T14:48:38.504Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: entering follower state: follower="Node at becdd36e-e96c-187b-3eb7-8a992d4eedfb [Follower]" leader-address= leader-id= 2025-01-28T14:48:38.504Z [INFO] server-04a0601a-25ab-1658-3c9e-4426f4ec72e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-becdd36e-e96c-187b-3eb7-8a992d4eedfb Address:becdd36e-e96c-187b-3eb7-8a992d4eedfb} {Suffrage:Voter ID:server-04a0601a-25ab-1658-3c9e-4426f4ec72e4 Address:04a0601a-25ab-1658-3c9e-4426f4ec72e4} {Suffrage:Voter ID:server-ac425c94-2ab9-dbef-5131-ef711499ccef Address:ac425c94-2ab9-dbef-5131-ef711499ccef}]" 2025-01-28T14:48:38.504Z [INFO] server-04a0601a-25ab-1658-3c9e-4426f4ec72e4: entering follower state: follower="Node at 04a0601a-25ab-1658-3c9e-4426f4ec72e4 [Follower]" leader-address= leader-id= 2025-01-28T14:48:38.505Z [INFO] server-ac425c94-2ab9-dbef-5131-ef711499ccef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-becdd36e-e96c-187b-3eb7-8a992d4eedfb Address:becdd36e-e96c-187b-3eb7-8a992d4eedfb} {Suffrage:Voter ID:server-04a0601a-25ab-1658-3c9e-4426f4ec72e4 Address:04a0601a-25ab-1658-3c9e-4426f4ec72e4} {Suffrage:Voter ID:server-ac425c94-2ab9-dbef-5131-ef711499ccef Address:ac425c94-2ab9-dbef-5131-ef711499ccef}]" 2025-01-28T14:48:38.505Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:38.505Z [INFO] server-ac425c94-2ab9-dbef-5131-ef711499ccef: entering follower state: follower="Node at ac425c94-2ab9-dbef-5131-ef711499ccef [Follower]" leader-address= leader-id= 2025-01-28T14:48:38.563Z [WARN] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:38.563Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: entering candidate state: node="Node at becdd36e-e96c-187b-3eb7-8a992d4eedfb [Candidate]" term=2 2025-01-28T14:48:38.563Z [WARN] server-04a0601a-25ab-1658-3c9e-4426f4ec72e4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:38.563Z [INFO] server-04a0601a-25ab-1658-3c9e-4426f4ec72e4: entering candidate state: node="Node at 04a0601a-25ab-1658-3c9e-4426f4ec72e4 [Candidate]" term=2 2025-01-28T14:48:38.564Z [INFO] server-04a0601a-25ab-1658-3c9e-4426f4ec72e4: duplicate requestVote for same term: term=2 2025-01-28T14:48:38.564Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: duplicate requestVote for same term: term=2 2025-01-28T14:48:38.564Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: election won: term=2 tally=2 2025-01-28T14:48:38.564Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: entering leader state: leader="Node at becdd36e-e96c-187b-3eb7-8a992d4eedfb [Leader]" 2025-01-28T14:48:38.565Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: added peer, starting replication: peer=server-04a0601a-25ab-1658-3c9e-4426f4ec72e4 2025-01-28T14:48:38.564Z [INFO] server-ac425c94-2ab9-dbef-5131-ef711499ccef: duplicate requestVote for same term: term=2 2025-01-28T14:48:38.565Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: added peer, starting replication: peer=server-ac425c94-2ab9-dbef-5131-ef711499ccef 2025-01-28T14:48:38.565Z [INFO] server-04a0601a-25ab-1658-3c9e-4426f4ec72e4: entering follower state: follower="Node at 04a0601a-25ab-1658-3c9e-4426f4ec72e4 [Follower]" leader-address=becdd36e-e96c-187b-3eb7-8a992d4eedfb leader-id=server-becdd36e-e96c-187b-3eb7-8a992d4eedfb 2025-01-28T14:48:38.566Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: pipelining replication: peer="{Voter server-ac425c94-2ab9-dbef-5131-ef711499ccef ac425c94-2ab9-dbef-5131-ef711499ccef}" 2025-01-28T14:48:38.566Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: pipelining replication: peer="{Voter server-04a0601a-25ab-1658-3c9e-4426f4ec72e4 04a0601a-25ab-1658-3c9e-4426f4ec72e4}" 2025-01-28T14:48:38.671Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:38.566404198 +0000 UTC m=+101.649809556 (1 nodes), 61.011145ms from start of poll, 63.362882ms from cluster start. Timeout at 2025-01-28 14:48:38.671416198 +0000 UTC m=+101.754821223, 105.011667ms after stability 2025-01-28T14:48:38.677Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:38.782Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:38.677168821 +0000 UTC m=+101.760573512 (2 nodes), 7µs from start of poll, 174.126838ms from cluster start. Timeout at 2025-01-28 14:48:38.782660835 +0000 UTC m=+101.866065859, 105.492347ms after stability 2025-01-28T14:48:38.783Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: updating configuration: command=RemoveServer server-id=server-04a0601a-25ab-1658-3c9e-4426f4ec72e4 server-addr= servers="[{Suffrage:Voter ID:server-becdd36e-e96c-187b-3eb7-8a992d4eedfb Address:becdd36e-e96c-187b-3eb7-8a992d4eedfb} {Suffrage:Voter ID:server-ac425c94-2ab9-dbef-5131-ef711499ccef Address:ac425c94-2ab9-dbef-5131-ef711499ccef}]" 2025-01-28T14:48:38.783Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: removed peer, stopping replication: peer=server-04a0601a-25ab-1658-3c9e-4426f4ec72e4 last-index=3 2025-01-28T14:48:38.783Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: aborting pipeline replication: peer="{Voter server-04a0601a-25ab-1658-3c9e-4426f4ec72e4 04a0601a-25ab-1658-3c9e-4426f4ec72e4}" 2025-01-28T14:48:38.871Z [WARN] server-04a0601a-25ab-1658-3c9e-4426f4ec72e4: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2025-01-28T14:48:38.891Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: updating configuration: command=AddVoter server-id=server-04a0601a-25ab-1658-3c9e-4426f4ec72e4 server-addr=04a0601a-25ab-1658-3c9e-4426f4ec72e4 servers="[{Suffrage:Voter ID:server-becdd36e-e96c-187b-3eb7-8a992d4eedfb Address:becdd36e-e96c-187b-3eb7-8a992d4eedfb} {Suffrage:Voter ID:server-ac425c94-2ab9-dbef-5131-ef711499ccef Address:ac425c94-2ab9-dbef-5131-ef711499ccef} {Suffrage:Voter ID:server-04a0601a-25ab-1658-3c9e-4426f4ec72e4 Address:04a0601a-25ab-1658-3c9e-4426f4ec72e4}]" 2025-01-28T14:48:38.891Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: added peer, starting replication: peer=server-04a0601a-25ab-1658-3c9e-4426f4ec72e4 2025-01-28T14:48:38.892Z [WARN] server-04a0601a-25ab-1658-3c9e-4426f4ec72e4: failed to get previous log: previous-index=4 last-index=3 error="log not found" 2025-01-28T14:48:38.892Z [WARN] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: appendEntries rejected, sending older logs: peer="{Voter server-04a0601a-25ab-1658-3c9e-4426f4ec72e4 04a0601a-25ab-1658-3c9e-4426f4ec72e4}" next=4 2025-01-28T14:48:38.892Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:38.892Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: pipelining replication: peer="{Voter server-04a0601a-25ab-1658-3c9e-4426f4ec72e4 04a0601a-25ab-1658-3c9e-4426f4ec72e4}" 2025-01-28T14:48:38.998Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:38.892517883 +0000 UTC m=+101.975922574 (1 nodes), 5.667µs from start of poll, 389.4759ms from cluster start. Timeout at 2025-01-28 14:48:38.998063232 +0000 UTC m=+102.081468590, 105.546016ms after stability 2025-01-28T14:48:38.998Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: aborting pipeline replication: peer="{Voter server-ac425c94-2ab9-dbef-5131-ef711499ccef ac425c94-2ab9-dbef-5131-ef711499ccef}" 2025-01-28T14:48:38.998Z [INFO] server-becdd36e-e96c-187b-3eb7-8a992d4eedfb: aborting pipeline replication: peer="{Voter server-04a0601a-25ab-1658-3c9e-4426f4ec72e4 04a0601a-25ab-1658-3c9e-4426f4ec72e4}" --- PASS: TestRaft_ReJoinFollower (0.50s) === RUN TestRaft_LeaderLeaseExpire 2025-01-28T14:48:39.006Z [INFO] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83d2b01a-b337-f10e-82b1-c0df4243cedb Address:83d2b01a-b337-f10e-82b1-c0df4243cedb} {Suffrage:Voter ID:server-6c4c558b-5e1d-a75b-a0c7-d2006e611008 Address:6c4c558b-5e1d-a75b-a0c7-d2006e611008}]" 2025-01-28T14:48:39.007Z [INFO] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: entering follower state: follower="Node at 83d2b01a-b337-f10e-82b1-c0df4243cedb [Follower]" leader-address= leader-id= 2025-01-28T14:48:39.007Z [INFO] server-6c4c558b-5e1d-a75b-a0c7-d2006e611008: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83d2b01a-b337-f10e-82b1-c0df4243cedb Address:83d2b01a-b337-f10e-82b1-c0df4243cedb} {Suffrage:Voter ID:server-6c4c558b-5e1d-a75b-a0c7-d2006e611008 Address:6c4c558b-5e1d-a75b-a0c7-d2006e611008}]" 2025-01-28T14:48:39.007Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:39.007Z [INFO] server-6c4c558b-5e1d-a75b-a0c7-d2006e611008: entering follower state: follower="Node at 6c4c558b-5e1d-a75b-a0c7-d2006e611008 [Follower]" leader-address= leader-id= 2025-01-28T14:48:39.077Z [WARN] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:39.077Z [INFO] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: entering candidate state: node="Node at 83d2b01a-b337-f10e-82b1-c0df4243cedb [Candidate]" term=2 2025-01-28T14:48:39.078Z [INFO] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: election won: term=2 tally=2 2025-01-28T14:48:39.078Z [INFO] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: entering leader state: leader="Node at 83d2b01a-b337-f10e-82b1-c0df4243cedb [Leader]" 2025-01-28T14:48:39.078Z [INFO] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: added peer, starting replication: peer=server-6c4c558b-5e1d-a75b-a0c7-d2006e611008 2025-01-28T14:48:39.079Z [INFO] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: pipelining replication: peer="{Voter server-6c4c558b-5e1d-a75b-a0c7-d2006e611008 6c4c558b-5e1d-a75b-a0c7-d2006e611008}" 2025-01-28T14:48:39.184Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:39.079322764 +0000 UTC m=+102.162727788 (1 nodes), 71.752797ms from start of poll, 73.226841ms from cluster start. Timeout at 2025-01-28 14:48:39.184343432 +0000 UTC m=+102.267748123, 105.020335ms after stability 2025-01-28T14:48:39.191Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:39.297Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:39.191318423 +0000 UTC m=+102.274723114 (1 nodes), 5.667µs from start of poll, 185.222167ms from cluster start. Timeout at 2025-01-28 14:48:39.297095446 +0000 UTC m=+102.380500470, 105.777356ms after stability raft_test.go:1799: [INFO] Disconnecting Node at 6c4c558b-5e1d-a75b-a0c7-d2006e611008 [Follower] 2025-01-28T14:48:39.298Z [ERROR] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: failed to pipeline appendEntries: peer="{Voter server-6c4c558b-5e1d-a75b-a0c7-d2006e611008 6c4c558b-5e1d-a75b-a0c7-d2006e611008}" error="append pipeline closed" 2025-01-28T14:48:39.298Z [INFO] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: aborting pipeline replication: peer="{Voter server-6c4c558b-5e1d-a75b-a0c7-d2006e611008 6c4c558b-5e1d-a75b-a0c7-d2006e611008}" 2025-01-28T14:48:39.303Z [ERROR] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: failed to heartbeat to: peer=6c4c558b-5e1d-a75b-a0c7-d2006e611008 backoff time=10ms error="failed to connect to peer: 6c4c558b-5e1d-a75b-a0c7-d2006e611008" 2025-01-28T14:48:39.307Z [ERROR] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: failed to appendEntries to: peer="{Voter server-6c4c558b-5e1d-a75b-a0c7-d2006e611008 6c4c558b-5e1d-a75b-a0c7-d2006e611008}" error="failed to connect to peer: 6c4c558b-5e1d-a75b-a0c7-d2006e611008" 2025-01-28T14:48:39.323Z [ERROR] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: failed to appendEntries to: peer="{Voter server-6c4c558b-5e1d-a75b-a0c7-d2006e611008 6c4c558b-5e1d-a75b-a0c7-d2006e611008}" error="failed to connect to peer: 6c4c558b-5e1d-a75b-a0c7-d2006e611008" 2025-01-28T14:48:39.325Z [ERROR] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: failed to heartbeat to: peer=6c4c558b-5e1d-a75b-a0c7-d2006e611008 backoff time=10ms error="failed to connect to peer: 6c4c558b-5e1d-a75b-a0c7-d2006e611008" 2025-01-28T14:48:39.342Z [ERROR] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: failed to appendEntries to: peer="{Voter server-6c4c558b-5e1d-a75b-a0c7-d2006e611008 6c4c558b-5e1d-a75b-a0c7-d2006e611008}" error="failed to connect to peer: 6c4c558b-5e1d-a75b-a0c7-d2006e611008" 2025-01-28T14:48:39.344Z [WARN] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: failed to contact: server-id=server-6c4c558b-5e1d-a75b-a0c7-d2006e611008 time=50.158823ms 2025-01-28T14:48:39.344Z [WARN] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: failed to contact quorum of nodes, stepping down 2025-01-28T14:48:39.344Z [INFO] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: entering follower state: follower="Node at 83d2b01a-b337-f10e-82b1-c0df4243cedb [Follower]" leader-address= leader-id= 2025-01-28T14:48:39.345Z [ERROR] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: failed to heartbeat to: peer=6c4c558b-5e1d-a75b-a0c7-d2006e611008 backoff time=10ms error="failed to connect to peer: 6c4c558b-5e1d-a75b-a0c7-d2006e611008" 2025-01-28T14:48:39.345Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:39.351Z [WARN] server-6c4c558b-5e1d-a75b-a0c7-d2006e611008: heartbeat timeout reached, starting election: last-leader-addr=83d2b01a-b337-f10e-82b1-c0df4243cedb last-leader-id=server-83d2b01a-b337-f10e-82b1-c0df4243cedb 2025-01-28T14:48:39.351Z [INFO] server-6c4c558b-5e1d-a75b-a0c7-d2006e611008: entering candidate state: node="Node at 6c4c558b-5e1d-a75b-a0c7-d2006e611008 [Candidate]" term=3 2025-01-28T14:48:39.352Z [ERROR] server-6c4c558b-5e1d-a75b-a0c7-d2006e611008: failed to make requestVote RPC: target="{Voter server-83d2b01a-b337-f10e-82b1-c0df4243cedb 83d2b01a-b337-f10e-82b1-c0df4243cedb}" error="failed to connect to peer: 83d2b01a-b337-f10e-82b1-c0df4243cedb" term=3 2025-01-28T14:48:39.400Z [WARN] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:39.400Z [INFO] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: entering candidate state: node="Node at 83d2b01a-b337-f10e-82b1-c0df4243cedb [Candidate]" term=3 2025-01-28T14:48:39.400Z [ERROR] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: failed to make requestVote RPC: target="{Voter server-6c4c558b-5e1d-a75b-a0c7-d2006e611008 6c4c558b-5e1d-a75b-a0c7-d2006e611008}" error="failed to connect to peer: 6c4c558b-5e1d-a75b-a0c7-d2006e611008" term=3 2025-01-28T14:48:39.443Z [WARN] server-6c4c558b-5e1d-a75b-a0c7-d2006e611008: Election timeout reached, restarting election 2025-01-28T14:48:39.443Z [INFO] server-6c4c558b-5e1d-a75b-a0c7-d2006e611008: entering candidate state: node="Node at 6c4c558b-5e1d-a75b-a0c7-d2006e611008 [Candidate]" term=4 2025-01-28T14:48:39.443Z [ERROR] server-6c4c558b-5e1d-a75b-a0c7-d2006e611008: failed to make requestVote RPC: target="{Voter server-83d2b01a-b337-f10e-82b1-c0df4243cedb 83d2b01a-b337-f10e-82b1-c0df4243cedb}" error="failed to connect to peer: 83d2b01a-b337-f10e-82b1-c0df4243cedb" term=4 2025-01-28T14:48:39.489Z [WARN] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: Election timeout reached, restarting election 2025-01-28T14:48:39.490Z [INFO] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: entering candidate state: node="Node at 83d2b01a-b337-f10e-82b1-c0df4243cedb [Candidate]" term=4 2025-01-28T14:48:39.490Z [ERROR] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: failed to make requestVote RPC: target="{Voter server-6c4c558b-5e1d-a75b-a0c7-d2006e611008 6c4c558b-5e1d-a75b-a0c7-d2006e611008}" error="failed to connect to peer: 6c4c558b-5e1d-a75b-a0c7-d2006e611008" term=4 2025-01-28T14:48:39.506Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:39.400769977 +0000 UTC m=+102.484175335 (0 nodes), 55.350644ms from start of poll, 394.674388ms from cluster start. Timeout at 2025-01-28 14:48:39.505800978 +0000 UTC m=+102.589236670, 105.061335ms after stability 2025-01-28T14:48:39.536Z [WARN] server-6c4c558b-5e1d-a75b-a0c7-d2006e611008: Election timeout reached, restarting election 2025-01-28T14:48:39.536Z [INFO] server-6c4c558b-5e1d-a75b-a0c7-d2006e611008: entering candidate state: node="Node at 6c4c558b-5e1d-a75b-a0c7-d2006e611008 [Candidate]" term=5 2025-01-28T14:48:39.537Z [ERROR] server-6c4c558b-5e1d-a75b-a0c7-d2006e611008: failed to make requestVote RPC: target="{Voter server-83d2b01a-b337-f10e-82b1-c0df4243cedb 83d2b01a-b337-f10e-82b1-c0df4243cedb}" error="failed to connect to peer: 83d2b01a-b337-f10e-82b1-c0df4243cedb" term=5 2025-01-28T14:48:39.552Z [WARN] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: Election timeout reached, restarting election 2025-01-28T14:48:39.553Z [INFO] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: entering candidate state: node="Node at 83d2b01a-b337-f10e-82b1-c0df4243cedb [Candidate]" term=5 2025-01-28T14:48:39.561Z [ERROR] server-83d2b01a-b337-f10e-82b1-c0df4243cedb: failed to make requestVote RPC: target="{Voter server-6c4c558b-5e1d-a75b-a0c7-d2006e611008 6c4c558b-5e1d-a75b-a0c7-d2006e611008}" error="failed to connect to peer: 6c4c558b-5e1d-a75b-a0c7-d2006e611008" term=5 --- PASS: TestRaft_LeaderLeaseExpire (0.61s) === RUN TestRaft_Barrier 2025-01-28T14:48:39.621Z [INFO] server-0e1a995e-8e97-c882-ff10-9cd227dcef60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e1a995e-8e97-c882-ff10-9cd227dcef60 Address:0e1a995e-8e97-c882-ff10-9cd227dcef60} {Suffrage:Voter ID:server-d15228fb-5ccc-b586-c2fc-2dd38233c8b3 Address:d15228fb-5ccc-b586-c2fc-2dd38233c8b3} {Suffrage:Voter ID:server-06136c5b-2af1-b22a-6c95-671429b8c757 Address:06136c5b-2af1-b22a-6c95-671429b8c757}]" 2025-01-28T14:48:39.621Z [INFO] server-0e1a995e-8e97-c882-ff10-9cd227dcef60: entering follower state: follower="Node at 0e1a995e-8e97-c882-ff10-9cd227dcef60 [Follower]" leader-address= leader-id= 2025-01-28T14:48:39.622Z [INFO] server-d15228fb-5ccc-b586-c2fc-2dd38233c8b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e1a995e-8e97-c882-ff10-9cd227dcef60 Address:0e1a995e-8e97-c882-ff10-9cd227dcef60} {Suffrage:Voter ID:server-d15228fb-5ccc-b586-c2fc-2dd38233c8b3 Address:d15228fb-5ccc-b586-c2fc-2dd38233c8b3} {Suffrage:Voter ID:server-06136c5b-2af1-b22a-6c95-671429b8c757 Address:06136c5b-2af1-b22a-6c95-671429b8c757}]" 2025-01-28T14:48:39.622Z [INFO] server-d15228fb-5ccc-b586-c2fc-2dd38233c8b3: entering follower state: follower="Node at d15228fb-5ccc-b586-c2fc-2dd38233c8b3 [Follower]" leader-address= leader-id= 2025-01-28T14:48:39.622Z [INFO] server-06136c5b-2af1-b22a-6c95-671429b8c757: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e1a995e-8e97-c882-ff10-9cd227dcef60 Address:0e1a995e-8e97-c882-ff10-9cd227dcef60} {Suffrage:Voter ID:server-d15228fb-5ccc-b586-c2fc-2dd38233c8b3 Address:d15228fb-5ccc-b586-c2fc-2dd38233c8b3} {Suffrage:Voter ID:server-06136c5b-2af1-b22a-6c95-671429b8c757 Address:06136c5b-2af1-b22a-6c95-671429b8c757}]" 2025-01-28T14:48:39.622Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:39.622Z [INFO] server-06136c5b-2af1-b22a-6c95-671429b8c757: entering follower state: follower="Node at 06136c5b-2af1-b22a-6c95-671429b8c757 [Follower]" leader-address= leader-id= 2025-01-28T14:48:39.677Z [WARN] server-d15228fb-5ccc-b586-c2fc-2dd38233c8b3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:39.677Z [INFO] server-d15228fb-5ccc-b586-c2fc-2dd38233c8b3: entering candidate state: node="Node at d15228fb-5ccc-b586-c2fc-2dd38233c8b3 [Candidate]" term=2 2025-01-28T14:48:39.678Z [INFO] server-d15228fb-5ccc-b586-c2fc-2dd38233c8b3: election won: term=2 tally=2 2025-01-28T14:48:39.678Z [INFO] server-d15228fb-5ccc-b586-c2fc-2dd38233c8b3: entering leader state: leader="Node at d15228fb-5ccc-b586-c2fc-2dd38233c8b3 [Leader]" 2025-01-28T14:48:39.678Z [INFO] server-d15228fb-5ccc-b586-c2fc-2dd38233c8b3: added peer, starting replication: peer=server-0e1a995e-8e97-c882-ff10-9cd227dcef60 2025-01-28T14:48:39.678Z [INFO] server-d15228fb-5ccc-b586-c2fc-2dd38233c8b3: added peer, starting replication: peer=server-06136c5b-2af1-b22a-6c95-671429b8c757 2025-01-28T14:48:39.679Z [INFO] server-d15228fb-5ccc-b586-c2fc-2dd38233c8b3: pipelining replication: peer="{Voter server-06136c5b-2af1-b22a-6c95-671429b8c757 06136c5b-2af1-b22a-6c95-671429b8c757}" 2025-01-28T14:48:39.679Z [INFO] server-d15228fb-5ccc-b586-c2fc-2dd38233c8b3: pipelining replication: peer="{Voter server-0e1a995e-8e97-c882-ff10-9cd227dcef60 0e1a995e-8e97-c882-ff10-9cd227dcef60}" 2025-01-28T14:48:39.784Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:39.679110576 +0000 UTC m=+102.762515934 (1 nodes), 56.106666ms from start of poll, 58.390734ms from cluster start. Timeout at 2025-01-28 14:48:39.784119577 +0000 UTC m=+102.867524268, 105.008334ms after stability 2025-01-28T14:48:39.851Z [INFO] server-d15228fb-5ccc-b586-c2fc-2dd38233c8b3: aborting pipeline replication: peer="{Voter server-06136c5b-2af1-b22a-6c95-671429b8c757 06136c5b-2af1-b22a-6c95-671429b8c757}" 2025-01-28T14:48:39.851Z [INFO] server-d15228fb-5ccc-b586-c2fc-2dd38233c8b3: aborting pipeline replication: peer="{Voter server-0e1a995e-8e97-c882-ff10-9cd227dcef60 0e1a995e-8e97-c882-ff10-9cd227dcef60}" --- PASS: TestRaft_Barrier (0.24s) === RUN TestRaft_VerifyLeader 2025-01-28T14:48:39.861Z [INFO] server-8d2aaa6e-a27f-f099-6d7b-5e48d001b477: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d2aaa6e-a27f-f099-6d7b-5e48d001b477 Address:8d2aaa6e-a27f-f099-6d7b-5e48d001b477} {Suffrage:Voter ID:server-d52a784d-5778-b9dc-aa2d-ecb714aaf07d Address:d52a784d-5778-b9dc-aa2d-ecb714aaf07d} {Suffrage:Voter ID:server-1d6b40aa-55f0-ff68-adb0-84e34ae7c65b Address:1d6b40aa-55f0-ff68-adb0-84e34ae7c65b}]" 2025-01-28T14:48:39.863Z [INFO] server-d52a784d-5778-b9dc-aa2d-ecb714aaf07d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d2aaa6e-a27f-f099-6d7b-5e48d001b477 Address:8d2aaa6e-a27f-f099-6d7b-5e48d001b477} {Suffrage:Voter ID:server-d52a784d-5778-b9dc-aa2d-ecb714aaf07d Address:d52a784d-5778-b9dc-aa2d-ecb714aaf07d} {Suffrage:Voter ID:server-1d6b40aa-55f0-ff68-adb0-84e34ae7c65b Address:1d6b40aa-55f0-ff68-adb0-84e34ae7c65b}]" 2025-01-28T14:48:39.864Z [INFO] server-8d2aaa6e-a27f-f099-6d7b-5e48d001b477: entering follower state: follower="Node at 8d2aaa6e-a27f-f099-6d7b-5e48d001b477 [Follower]" leader-address= leader-id= 2025-01-28T14:48:39.864Z [INFO] server-1d6b40aa-55f0-ff68-adb0-84e34ae7c65b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d2aaa6e-a27f-f099-6d7b-5e48d001b477 Address:8d2aaa6e-a27f-f099-6d7b-5e48d001b477} {Suffrage:Voter ID:server-d52a784d-5778-b9dc-aa2d-ecb714aaf07d Address:d52a784d-5778-b9dc-aa2d-ecb714aaf07d} {Suffrage:Voter ID:server-1d6b40aa-55f0-ff68-adb0-84e34ae7c65b Address:1d6b40aa-55f0-ff68-adb0-84e34ae7c65b}]" 2025-01-28T14:48:39.864Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:39.864Z [INFO] server-d52a784d-5778-b9dc-aa2d-ecb714aaf07d: entering follower state: follower="Node at d52a784d-5778-b9dc-aa2d-ecb714aaf07d [Follower]" leader-address= leader-id= 2025-01-28T14:48:39.865Z [INFO] server-1d6b40aa-55f0-ff68-adb0-84e34ae7c65b: entering follower state: follower="Node at 1d6b40aa-55f0-ff68-adb0-84e34ae7c65b [Follower]" leader-address= leader-id= 2025-01-28T14:48:39.921Z [WARN] server-1d6b40aa-55f0-ff68-adb0-84e34ae7c65b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:39.921Z [INFO] server-1d6b40aa-55f0-ff68-adb0-84e34ae7c65b: entering candidate state: node="Node at 1d6b40aa-55f0-ff68-adb0-84e34ae7c65b [Candidate]" term=2 2025-01-28T14:48:39.922Z [INFO] server-1d6b40aa-55f0-ff68-adb0-84e34ae7c65b: election won: term=2 tally=2 2025-01-28T14:48:39.922Z [INFO] server-1d6b40aa-55f0-ff68-adb0-84e34ae7c65b: entering leader state: leader="Node at 1d6b40aa-55f0-ff68-adb0-84e34ae7c65b [Leader]" 2025-01-28T14:48:39.922Z [INFO] server-1d6b40aa-55f0-ff68-adb0-84e34ae7c65b: added peer, starting replication: peer=server-8d2aaa6e-a27f-f099-6d7b-5e48d001b477 2025-01-28T14:48:39.922Z [INFO] server-1d6b40aa-55f0-ff68-adb0-84e34ae7c65b: added peer, starting replication: peer=server-d52a784d-5778-b9dc-aa2d-ecb714aaf07d 2025-01-28T14:48:39.923Z [INFO] server-1d6b40aa-55f0-ff68-adb0-84e34ae7c65b: pipelining replication: peer="{Voter server-8d2aaa6e-a27f-f099-6d7b-5e48d001b477 8d2aaa6e-a27f-f099-6d7b-5e48d001b477}" 2025-01-28T14:48:39.923Z [INFO] server-1d6b40aa-55f0-ff68-adb0-84e34ae7c65b: pipelining replication: peer="{Voter server-d52a784d-5778-b9dc-aa2d-ecb714aaf07d d52a784d-5778-b9dc-aa2d-ecb714aaf07d}" 2025-01-28T14:48:40.028Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:39.923062488 +0000 UTC m=+103.006467179 (1 nodes), 58.035724ms from start of poll, 61.809169ms from cluster start. Timeout at 2025-01-28 14:48:40.028071155 +0000 UTC m=+103.111475846, 105.008667ms after stability 2025-01-28T14:48:40.029Z [INFO] server-1d6b40aa-55f0-ff68-adb0-84e34ae7c65b: aborting pipeline replication: peer="{Voter server-d52a784d-5778-b9dc-aa2d-ecb714aaf07d d52a784d-5778-b9dc-aa2d-ecb714aaf07d}" 2025-01-28T14:48:40.029Z [INFO] server-1d6b40aa-55f0-ff68-adb0-84e34ae7c65b: aborting pipeline replication: peer="{Voter server-8d2aaa6e-a27f-f099-6d7b-5e48d001b477 8d2aaa6e-a27f-f099-6d7b-5e48d001b477}" --- PASS: TestRaft_VerifyLeader (0.18s) === RUN TestRaft_VerifyLeader_Single 2025-01-28T14:48:40.036Z [INFO] server-efb8567d-ccf9-42a4-7ff0-47c3785ed9ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-efb8567d-ccf9-42a4-7ff0-47c3785ed9ba Address:efb8567d-ccf9-42a4-7ff0-47c3785ed9ba}]" 2025-01-28T14:48:40.037Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:40.037Z [INFO] server-efb8567d-ccf9-42a4-7ff0-47c3785ed9ba: entering follower state: follower="Node at efb8567d-ccf9-42a4-7ff0-47c3785ed9ba [Follower]" leader-address= leader-id= 2025-01-28T14:48:40.128Z [WARN] server-efb8567d-ccf9-42a4-7ff0-47c3785ed9ba: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:40.129Z [INFO] server-efb8567d-ccf9-42a4-7ff0-47c3785ed9ba: entering candidate state: node="Node at efb8567d-ccf9-42a4-7ff0-47c3785ed9ba [Candidate]" term=2 2025-01-28T14:48:40.129Z [INFO] server-efb8567d-ccf9-42a4-7ff0-47c3785ed9ba: election won: term=2 tally=1 2025-01-28T14:48:40.129Z [INFO] server-efb8567d-ccf9-42a4-7ff0-47c3785ed9ba: entering leader state: leader="Node at efb8567d-ccf9-42a4-7ff0-47c3785ed9ba [Leader]" 2025-01-28T14:48:40.235Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:40.129933298 +0000 UTC m=+103.213338322 (1 nodes), 92.637751ms from start of poll, 93.917122ms from cluster start. Timeout at 2025-01-28 14:48:40.234967633 +0000 UTC m=+103.318372324, 105.034002ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.20s) === RUN TestRaft_VerifyLeader_Fail 2025-01-28T14:48:40.243Z [INFO] server-15b46014-ada1-d7f7-fd8c-74f125d62b49: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15b46014-ada1-d7f7-fd8c-74f125d62b49 Address:15b46014-ada1-d7f7-fd8c-74f125d62b49} {Suffrage:Voter ID:server-914e426a-804a-4c61-7a9b-fe9ce496db5b Address:914e426a-804a-4c61-7a9b-fe9ce496db5b}]" 2025-01-28T14:48:40.244Z [INFO] server-914e426a-804a-4c61-7a9b-fe9ce496db5b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15b46014-ada1-d7f7-fd8c-74f125d62b49 Address:15b46014-ada1-d7f7-fd8c-74f125d62b49} {Suffrage:Voter ID:server-914e426a-804a-4c61-7a9b-fe9ce496db5b Address:914e426a-804a-4c61-7a9b-fe9ce496db5b}]" 2025-01-28T14:48:40.244Z [INFO] server-15b46014-ada1-d7f7-fd8c-74f125d62b49: entering follower state: follower="Node at 15b46014-ada1-d7f7-fd8c-74f125d62b49 [Follower]" leader-address= leader-id= 2025-01-28T14:48:40.244Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:40.244Z [INFO] server-914e426a-804a-4c61-7a9b-fe9ce496db5b: entering follower state: follower="Node at 914e426a-804a-4c61-7a9b-fe9ce496db5b [Follower]" leader-address= leader-id= 2025-01-28T14:48:40.312Z [WARN] server-15b46014-ada1-d7f7-fd8c-74f125d62b49: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:40.312Z [INFO] server-15b46014-ada1-d7f7-fd8c-74f125d62b49: entering candidate state: node="Node at 15b46014-ada1-d7f7-fd8c-74f125d62b49 [Candidate]" term=2 2025-01-28T14:48:40.312Z [INFO] server-15b46014-ada1-d7f7-fd8c-74f125d62b49: election won: term=2 tally=2 2025-01-28T14:48:40.312Z [INFO] server-15b46014-ada1-d7f7-fd8c-74f125d62b49: entering leader state: leader="Node at 15b46014-ada1-d7f7-fd8c-74f125d62b49 [Leader]" 2025-01-28T14:48:40.313Z [INFO] server-15b46014-ada1-d7f7-fd8c-74f125d62b49: added peer, starting replication: peer=server-914e426a-804a-4c61-7a9b-fe9ce496db5b 2025-01-28T14:48:40.313Z [INFO] server-15b46014-ada1-d7f7-fd8c-74f125d62b49: pipelining replication: peer="{Voter server-914e426a-804a-4c61-7a9b-fe9ce496db5b 914e426a-804a-4c61-7a9b-fe9ce496db5b}" 2025-01-28T14:48:40.418Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:40.313200741 +0000 UTC m=+103.396605765 (1 nodes), 68.604037ms from start of poll, 70.327089ms from cluster start. Timeout at 2025-01-28 14:48:40.418210407 +0000 UTC m=+103.501615098, 105.009333ms after stability 2025-01-28T14:48:40.418Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:40.524Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:40.418696874 +0000 UTC m=+103.502101565 (1 nodes), 5.667µs from start of poll, 175.822889ms from cluster start. Timeout at 2025-01-28 14:48:40.524109552 +0000 UTC m=+103.607514577, 105.413012ms after stability 2025-01-28T14:48:40.531Z [ERROR] server-15b46014-ada1-d7f7-fd8c-74f125d62b49: peer has newer term, stopping replication: peer="{Voter server-914e426a-804a-4c61-7a9b-fe9ce496db5b 914e426a-804a-4c61-7a9b-fe9ce496db5b}" 2025-01-28T14:48:40.532Z [INFO] server-15b46014-ada1-d7f7-fd8c-74f125d62b49: aborting pipeline replication: peer="{Voter server-914e426a-804a-4c61-7a9b-fe9ce496db5b 914e426a-804a-4c61-7a9b-fe9ce496db5b}" 2025-01-28T14:48:40.532Z [INFO] server-15b46014-ada1-d7f7-fd8c-74f125d62b49: entering follower state: follower="Node at 15b46014-ada1-d7f7-fd8c-74f125d62b49 [Follower]" leader-address= leader-id= --- PASS: TestRaft_VerifyLeader_Fail (0.30s) === RUN TestRaft_VerifyLeader_PartialConnect 2025-01-28T14:48:40.542Z [INFO] server-0a048dae-4327-c16e-7a37-ed9e2e73814f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0a048dae-4327-c16e-7a37-ed9e2e73814f Address:0a048dae-4327-c16e-7a37-ed9e2e73814f} {Suffrage:Voter ID:server-fafe24b2-141e-b55c-c60b-afa8ac735a8f Address:fafe24b2-141e-b55c-c60b-afa8ac735a8f} {Suffrage:Voter ID:server-a215b2ae-1b73-b39c-cb9b-1774439b2582 Address:a215b2ae-1b73-b39c-cb9b-1774439b2582}]" 2025-01-28T14:48:40.542Z [INFO] server-0a048dae-4327-c16e-7a37-ed9e2e73814f: entering follower state: follower="Node at 0a048dae-4327-c16e-7a37-ed9e2e73814f [Follower]" leader-address= leader-id= 2025-01-28T14:48:40.543Z [INFO] server-fafe24b2-141e-b55c-c60b-afa8ac735a8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0a048dae-4327-c16e-7a37-ed9e2e73814f Address:0a048dae-4327-c16e-7a37-ed9e2e73814f} {Suffrage:Voter ID:server-fafe24b2-141e-b55c-c60b-afa8ac735a8f Address:fafe24b2-141e-b55c-c60b-afa8ac735a8f} {Suffrage:Voter ID:server-a215b2ae-1b73-b39c-cb9b-1774439b2582 Address:a215b2ae-1b73-b39c-cb9b-1774439b2582}]" 2025-01-28T14:48:40.543Z [INFO] server-fafe24b2-141e-b55c-c60b-afa8ac735a8f: entering follower state: follower="Node at fafe24b2-141e-b55c-c60b-afa8ac735a8f [Follower]" leader-address= leader-id= 2025-01-28T14:48:40.544Z [INFO] server-a215b2ae-1b73-b39c-cb9b-1774439b2582: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0a048dae-4327-c16e-7a37-ed9e2e73814f Address:0a048dae-4327-c16e-7a37-ed9e2e73814f} {Suffrage:Voter ID:server-fafe24b2-141e-b55c-c60b-afa8ac735a8f Address:fafe24b2-141e-b55c-c60b-afa8ac735a8f} {Suffrage:Voter ID:server-a215b2ae-1b73-b39c-cb9b-1774439b2582 Address:a215b2ae-1b73-b39c-cb9b-1774439b2582}]" 2025-01-28T14:48:40.544Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:40.544Z [INFO] server-a215b2ae-1b73-b39c-cb9b-1774439b2582: entering follower state: follower="Node at a215b2ae-1b73-b39c-cb9b-1774439b2582 [Follower]" leader-address= leader-id= 2025-01-28T14:48:40.600Z [WARN] server-0a048dae-4327-c16e-7a37-ed9e2e73814f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:40.600Z [INFO] server-0a048dae-4327-c16e-7a37-ed9e2e73814f: entering candidate state: node="Node at 0a048dae-4327-c16e-7a37-ed9e2e73814f [Candidate]" term=2 2025-01-28T14:48:40.601Z [INFO] server-0a048dae-4327-c16e-7a37-ed9e2e73814f: election won: term=2 tally=2 2025-01-28T14:48:40.601Z [INFO] server-0a048dae-4327-c16e-7a37-ed9e2e73814f: entering leader state: leader="Node at 0a048dae-4327-c16e-7a37-ed9e2e73814f [Leader]" 2025-01-28T14:48:40.601Z [INFO] server-0a048dae-4327-c16e-7a37-ed9e2e73814f: added peer, starting replication: peer=server-fafe24b2-141e-b55c-c60b-afa8ac735a8f 2025-01-28T14:48:40.601Z [INFO] server-0a048dae-4327-c16e-7a37-ed9e2e73814f: added peer, starting replication: peer=server-a215b2ae-1b73-b39c-cb9b-1774439b2582 2025-01-28T14:48:40.602Z [INFO] server-0a048dae-4327-c16e-7a37-ed9e2e73814f: pipelining replication: peer="{Voter server-fafe24b2-141e-b55c-c60b-afa8ac735a8f fafe24b2-141e-b55c-c60b-afa8ac735a8f}" 2025-01-28T14:48:40.602Z [INFO] server-0a048dae-4327-c16e-7a37-ed9e2e73814f: pipelining replication: peer="{Voter server-a215b2ae-1b73-b39c-cb9b-1774439b2582 a215b2ae-1b73-b39c-cb9b-1774439b2582}" 2025-01-28T14:48:40.707Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:40.601459634 +0000 UTC m=+103.684864659 (1 nodes), 56.759019ms from start of poll, 59.547435ms from cluster start. Timeout at 2025-01-28 14:48:40.706471635 +0000 UTC m=+103.789876659, 105.012ms after stability 2025-01-28T14:48:40.722Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:40.828Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:40.722617899 +0000 UTC m=+103.806022590 (2 nodes), 8.667µs from start of poll, 180.705366ms from cluster start. Timeout at 2025-01-28 14:48:40.828119248 +0000 UTC m=+103.911523939, 105.501349ms after stability raft_test.go:1968: [INFO] Disconnecting Node at fafe24b2-141e-b55c-c60b-afa8ac735a8f [Follower] 2025-01-28T14:48:40.829Z [ERROR] server-0a048dae-4327-c16e-7a37-ed9e2e73814f: failed to heartbeat to: peer=fafe24b2-141e-b55c-c60b-afa8ac735a8f backoff time=10ms error="failed to connect to peer: fafe24b2-141e-b55c-c60b-afa8ac735a8f" 2025-01-28T14:48:40.829Z [INFO] server-0a048dae-4327-c16e-7a37-ed9e2e73814f: aborting pipeline replication: peer="{Voter server-a215b2ae-1b73-b39c-cb9b-1774439b2582 a215b2ae-1b73-b39c-cb9b-1774439b2582}" 2025-01-28T14:48:40.829Z [INFO] server-0a048dae-4327-c16e-7a37-ed9e2e73814f: aborting pipeline replication: peer="{Voter server-fafe24b2-141e-b55c-c60b-afa8ac735a8f fafe24b2-141e-b55c-c60b-afa8ac735a8f}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.30s) === RUN TestRaft_NotifyCh 2025-01-28T14:48:40.836Z [INFO] server-eba94215-cb5f-1dfa-936d-423dc8ad1c7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eba94215-cb5f-1dfa-936d-423dc8ad1c7f Address:eba94215-cb5f-1dfa-936d-423dc8ad1c7f}]" 2025-01-28T14:48:40.836Z [INFO] server-eba94215-cb5f-1dfa-936d-423dc8ad1c7f: entering follower state: follower="Node at eba94215-cb5f-1dfa-936d-423dc8ad1c7f [Follower]" leader-address= leader-id= 2025-01-28T14:48:40.922Z [WARN] server-eba94215-cb5f-1dfa-936d-423dc8ad1c7f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:40.922Z [INFO] server-eba94215-cb5f-1dfa-936d-423dc8ad1c7f: entering candidate state: node="Node at eba94215-cb5f-1dfa-936d-423dc8ad1c7f [Candidate]" term=2 2025-01-28T14:48:40.922Z [INFO] server-eba94215-cb5f-1dfa-936d-423dc8ad1c7f: election won: term=2 tally=1 2025-01-28T14:48:40.922Z [INFO] server-eba94215-cb5f-1dfa-936d-423dc8ad1c7f: entering leader state: leader="Node at eba94215-cb5f-1dfa-936d-423dc8ad1c7f [Leader]" --- PASS: TestRaft_NotifyCh (0.09s) === RUN TestRaft_AppendEntry 2025-01-28T14:48:40.929Z [INFO] server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84 Address:1cb8c99c-787c-dfc2-f412-1a0e37c0ac84} {Suffrage:Voter ID:server-5a506305-9127-afcd-5100-8c0c2326fd47 Address:5a506305-9127-afcd-5100-8c0c2326fd47} {Suffrage:Voter ID:server-71610a26-48b6-e667-e077-ee4b48341756 Address:71610a26-48b6-e667-e077-ee4b48341756}]" 2025-01-28T14:48:40.930Z [INFO] server-5a506305-9127-afcd-5100-8c0c2326fd47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84 Address:1cb8c99c-787c-dfc2-f412-1a0e37c0ac84} {Suffrage:Voter ID:server-5a506305-9127-afcd-5100-8c0c2326fd47 Address:5a506305-9127-afcd-5100-8c0c2326fd47} {Suffrage:Voter ID:server-71610a26-48b6-e667-e077-ee4b48341756 Address:71610a26-48b6-e667-e077-ee4b48341756}]" 2025-01-28T14:48:40.931Z [INFO] server-71610a26-48b6-e667-e077-ee4b48341756: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84 Address:1cb8c99c-787c-dfc2-f412-1a0e37c0ac84} {Suffrage:Voter ID:server-5a506305-9127-afcd-5100-8c0c2326fd47 Address:5a506305-9127-afcd-5100-8c0c2326fd47} {Suffrage:Voter ID:server-71610a26-48b6-e667-e077-ee4b48341756 Address:71610a26-48b6-e667-e077-ee4b48341756}]" 2025-01-28T14:48:40.931Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:40.931Z [INFO] server-5a506305-9127-afcd-5100-8c0c2326fd47: entering follower state: follower="Node at 5a506305-9127-afcd-5100-8c0c2326fd47 [Follower]" leader-address= leader-id= 2025-01-28T14:48:40.932Z [INFO] server-71610a26-48b6-e667-e077-ee4b48341756: entering follower state: follower="Node at 71610a26-48b6-e667-e077-ee4b48341756 [Follower]" leader-address= leader-id= 2025-01-28T14:48:40.932Z [INFO] server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84: entering follower state: follower="Node at 1cb8c99c-787c-dfc2-f412-1a0e37c0ac84 [Follower]" leader-address= leader-id= 2025-01-28T14:48:40.984Z [WARN] server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:40.984Z [INFO] server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84: entering candidate state: node="Node at 1cb8c99c-787c-dfc2-f412-1a0e37c0ac84 [Candidate]" term=2 2025-01-28T14:48:40.984Z [INFO] server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84: election won: term=2 tally=2 2025-01-28T14:48:40.984Z [INFO] server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84: entering leader state: leader="Node at 1cb8c99c-787c-dfc2-f412-1a0e37c0ac84 [Leader]" 2025-01-28T14:48:40.985Z [INFO] server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84: added peer, starting replication: peer=server-5a506305-9127-afcd-5100-8c0c2326fd47 2025-01-28T14:48:40.985Z [INFO] server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84: added peer, starting replication: peer=server-71610a26-48b6-e667-e077-ee4b48341756 2025-01-28T14:48:40.985Z [WARN] server-71610a26-48b6-e667-e077-ee4b48341756: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:40.985Z [INFO] server-71610a26-48b6-e667-e077-ee4b48341756: entering candidate state: node="Node at 71610a26-48b6-e667-e077-ee4b48341756 [Candidate]" term=2 2025-01-28T14:48:40.985Z [INFO] server-71610a26-48b6-e667-e077-ee4b48341756: duplicate requestVote for same term: term=2 2025-01-28T14:48:40.985Z [WARN] server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84: rejecting vote request since we have a leader: from=71610a26-48b6-e667-e077-ee4b48341756 leader=1cb8c99c-787c-dfc2-f412-1a0e37c0ac84 leader-id=server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84 2025-01-28T14:48:40.985Z [WARN] server-5a506305-9127-afcd-5100-8c0c2326fd47: rejecting vote request since we have a leader: from=71610a26-48b6-e667-e077-ee4b48341756 leader=1cb8c99c-787c-dfc2-f412-1a0e37c0ac84 leader-id=server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84 2025-01-28T14:48:40.986Z [INFO] server-71610a26-48b6-e667-e077-ee4b48341756: entering follower state: follower="Node at 71610a26-48b6-e667-e077-ee4b48341756 [Follower]" leader-address=1cb8c99c-787c-dfc2-f412-1a0e37c0ac84 leader-id=server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84 2025-01-28T14:48:40.986Z [INFO] server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84: pipelining replication: peer="{Voter server-5a506305-9127-afcd-5100-8c0c2326fd47 5a506305-9127-afcd-5100-8c0c2326fd47}" 2025-01-28T14:48:40.986Z [INFO] server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84: pipelining replication: peer="{Voter server-71610a26-48b6-e667-e077-ee4b48341756 71610a26-48b6-e667-e077-ee4b48341756}" 2025-01-28T14:48:41.091Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:40.986386399 +0000 UTC m=+104.069791424 (2 nodes), 54.65329ms from start of poll, 57.425039ms from cluster start. Timeout at 2025-01-28 14:48:41.0914434 +0000 UTC m=+104.174848091, 105.056667ms after stability 2025-01-28T14:48:41.091Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:41.204Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:41.092046537 +0000 UTC m=+104.175451228 (1 nodes), 6.667µs from start of poll, 163.084843ms from cluster start. Timeout at 2025-01-28 14:48:41.197716223 +0000 UTC m=+104.281120914, 105.669686ms after stability 2025-01-28T14:48:41.204Z [WARN] server-5a506305-9127-afcd-5100-8c0c2326fd47: clearing log suffix: from=1 to=2 2025-01-28T14:48:41.204Z [WARN] server-5a506305-9127-afcd-5100-8c0c2326fd47: skipping application of old log: index=1 2025-01-28T14:48:41.204Z [WARN] server-5a506305-9127-afcd-5100-8c0c2326fd47: skipping application of old log: index=1 2025-01-28T14:48:41.207Z [INFO] server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84: aborting pipeline replication: peer="{Voter server-71610a26-48b6-e667-e077-ee4b48341756 71610a26-48b6-e667-e077-ee4b48341756}" 2025-01-28T14:48:41.207Z [INFO] server-1cb8c99c-787c-dfc2-f412-1a0e37c0ac84: aborting pipeline replication: peer="{Voter server-5a506305-9127-afcd-5100-8c0c2326fd47 5a506305-9127-afcd-5100-8c0c2326fd47}" --- PASS: TestRaft_AppendEntry (0.29s) === RUN TestRaft_VotingGrant_WhenLeaderAvailable 2025-01-28T14:48:41.219Z [INFO] server-a6cb8458-a62f-6562-de79-96841dc92ce3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6cb8458-a62f-6562-de79-96841dc92ce3 Address:a6cb8458-a62f-6562-de79-96841dc92ce3} {Suffrage:Voter ID:server-4bc584ce-f1f2-a354-7549-e4a7411870e4 Address:4bc584ce-f1f2-a354-7549-e4a7411870e4} {Suffrage:Voter ID:server-b67050a9-5a59-e238-2524-15516f7a1d21 Address:b67050a9-5a59-e238-2524-15516f7a1d21}]" 2025-01-28T14:48:41.219Z [INFO] server-a6cb8458-a62f-6562-de79-96841dc92ce3: entering follower state: follower="Node at a6cb8458-a62f-6562-de79-96841dc92ce3 [Follower]" leader-address= leader-id= 2025-01-28T14:48:41.220Z [INFO] server-4bc584ce-f1f2-a354-7549-e4a7411870e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6cb8458-a62f-6562-de79-96841dc92ce3 Address:a6cb8458-a62f-6562-de79-96841dc92ce3} {Suffrage:Voter ID:server-4bc584ce-f1f2-a354-7549-e4a7411870e4 Address:4bc584ce-f1f2-a354-7549-e4a7411870e4} {Suffrage:Voter ID:server-b67050a9-5a59-e238-2524-15516f7a1d21 Address:b67050a9-5a59-e238-2524-15516f7a1d21}]" 2025-01-28T14:48:41.220Z [INFO] server-4bc584ce-f1f2-a354-7549-e4a7411870e4: entering follower state: follower="Node at 4bc584ce-f1f2-a354-7549-e4a7411870e4 [Follower]" leader-address= leader-id= 2025-01-28T14:48:41.220Z [INFO] server-b67050a9-5a59-e238-2524-15516f7a1d21: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6cb8458-a62f-6562-de79-96841dc92ce3 Address:a6cb8458-a62f-6562-de79-96841dc92ce3} {Suffrage:Voter ID:server-4bc584ce-f1f2-a354-7549-e4a7411870e4 Address:4bc584ce-f1f2-a354-7549-e4a7411870e4} {Suffrage:Voter ID:server-b67050a9-5a59-e238-2524-15516f7a1d21 Address:b67050a9-5a59-e238-2524-15516f7a1d21}]" 2025-01-28T14:48:41.221Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:41.221Z [INFO] server-b67050a9-5a59-e238-2524-15516f7a1d21: entering follower state: follower="Node at b67050a9-5a59-e238-2524-15516f7a1d21 [Follower]" leader-address= leader-id= 2025-01-28T14:48:41.271Z [WARN] server-a6cb8458-a62f-6562-de79-96841dc92ce3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:41.271Z [INFO] server-a6cb8458-a62f-6562-de79-96841dc92ce3: entering candidate state: node="Node at a6cb8458-a62f-6562-de79-96841dc92ce3 [Candidate]" term=2 2025-01-28T14:48:41.272Z [INFO] server-a6cb8458-a62f-6562-de79-96841dc92ce3: election won: term=2 tally=2 2025-01-28T14:48:41.272Z [INFO] server-a6cb8458-a62f-6562-de79-96841dc92ce3: entering leader state: leader="Node at a6cb8458-a62f-6562-de79-96841dc92ce3 [Leader]" 2025-01-28T14:48:41.272Z [INFO] server-a6cb8458-a62f-6562-de79-96841dc92ce3: added peer, starting replication: peer=server-4bc584ce-f1f2-a354-7549-e4a7411870e4 2025-01-28T14:48:41.272Z [INFO] server-a6cb8458-a62f-6562-de79-96841dc92ce3: added peer, starting replication: peer=server-b67050a9-5a59-e238-2524-15516f7a1d21 2025-01-28T14:48:41.273Z [INFO] server-a6cb8458-a62f-6562-de79-96841dc92ce3: pipelining replication: peer="{Voter server-4bc584ce-f1f2-a354-7549-e4a7411870e4 4bc584ce-f1f2-a354-7549-e4a7411870e4}" 2025-01-28T14:48:41.273Z [INFO] server-a6cb8458-a62f-6562-de79-96841dc92ce3: pipelining replication: peer="{Voter server-b67050a9-5a59-e238-2524-15516f7a1d21 b67050a9-5a59-e238-2524-15516f7a1d21}" 2025-01-28T14:48:41.377Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:41.272552564 +0000 UTC m=+104.355957255 (2 nodes), 51.447194ms from start of poll, 54.156941ms from cluster start. Timeout at 2025-01-28 14:48:41.377563231 +0000 UTC m=+104.460968256, 105.011001ms after stability 2025-01-28T14:48:41.378Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:41.483Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:41.378098699 +0000 UTC m=+104.461503390 (1 nodes), 5.667µs from start of poll, 159.703076ms from cluster start. Timeout at 2025-01-28 14:48:41.483534378 +0000 UTC m=+104.566939736, 105.436346ms after stability 2025-01-28T14:48:41.484Z [WARN] server-b67050a9-5a59-e238-2524-15516f7a1d21: rejecting vote request since we have a leader: from=4bc584ce-f1f2-a354-7549-e4a7411870e4 leader=a6cb8458-a62f-6562-de79-96841dc92ce3 leader-id=server-a6cb8458-a62f-6562-de79-96841dc92ce3 2025-01-28T14:48:41.485Z [INFO] server-a6cb8458-a62f-6562-de79-96841dc92ce3: aborting pipeline replication: peer="{Voter server-4bc584ce-f1f2-a354-7549-e4a7411870e4 4bc584ce-f1f2-a354-7549-e4a7411870e4}" 2025-01-28T14:48:41.485Z [INFO] server-a6cb8458-a62f-6562-de79-96841dc92ce3: aborting pipeline replication: peer="{Voter server-b67050a9-5a59-e238-2524-15516f7a1d21 b67050a9-5a59-e238-2524-15516f7a1d21}" --- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.28s) === RUN TestRaft_ProtocolVersion_RejectRPC 2025-01-28T14:48:41.500Z [INFO] server-81272f9d-52cd-323c-d943-3d51c0be810e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81272f9d-52cd-323c-d943-3d51c0be810e Address:81272f9d-52cd-323c-d943-3d51c0be810e} {Suffrage:Voter ID:server-d3d508de-49e3-b3e4-33e1-5930ee54cf5b Address:d3d508de-49e3-b3e4-33e1-5930ee54cf5b} {Suffrage:Voter ID:server-11c2317d-1b03-ec9f-bce6-04efd5e1ef99 Address:11c2317d-1b03-ec9f-bce6-04efd5e1ef99}]" 2025-01-28T14:48:41.501Z [INFO] server-d3d508de-49e3-b3e4-33e1-5930ee54cf5b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81272f9d-52cd-323c-d943-3d51c0be810e Address:81272f9d-52cd-323c-d943-3d51c0be810e} {Suffrage:Voter ID:server-d3d508de-49e3-b3e4-33e1-5930ee54cf5b Address:d3d508de-49e3-b3e4-33e1-5930ee54cf5b} {Suffrage:Voter ID:server-11c2317d-1b03-ec9f-bce6-04efd5e1ef99 Address:11c2317d-1b03-ec9f-bce6-04efd5e1ef99}]" 2025-01-28T14:48:41.503Z [INFO] server-11c2317d-1b03-ec9f-bce6-04efd5e1ef99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81272f9d-52cd-323c-d943-3d51c0be810e Address:81272f9d-52cd-323c-d943-3d51c0be810e} {Suffrage:Voter ID:server-d3d508de-49e3-b3e4-33e1-5930ee54cf5b Address:d3d508de-49e3-b3e4-33e1-5930ee54cf5b} {Suffrage:Voter ID:server-11c2317d-1b03-ec9f-bce6-04efd5e1ef99 Address:11c2317d-1b03-ec9f-bce6-04efd5e1ef99}]" 2025-01-28T14:48:41.503Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:41.509Z [INFO] server-81272f9d-52cd-323c-d943-3d51c0be810e: entering follower state: follower="Node at 81272f9d-52cd-323c-d943-3d51c0be810e [Follower]" leader-address= leader-id= 2025-01-28T14:48:41.513Z [INFO] server-d3d508de-49e3-b3e4-33e1-5930ee54cf5b: entering follower state: follower="Node at d3d508de-49e3-b3e4-33e1-5930ee54cf5b [Follower]" leader-address= leader-id= 2025-01-28T14:48:41.515Z [INFO] server-11c2317d-1b03-ec9f-bce6-04efd5e1ef99: entering follower state: follower="Node at 11c2317d-1b03-ec9f-bce6-04efd5e1ef99 [Follower]" leader-address= leader-id= 2025-01-28T14:48:41.581Z [WARN] server-d3d508de-49e3-b3e4-33e1-5930ee54cf5b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:41.582Z [INFO] server-d3d508de-49e3-b3e4-33e1-5930ee54cf5b: entering candidate state: node="Node at d3d508de-49e3-b3e4-33e1-5930ee54cf5b [Candidate]" term=2 2025-01-28T14:48:41.582Z [INFO] server-d3d508de-49e3-b3e4-33e1-5930ee54cf5b: election won: term=2 tally=2 2025-01-28T14:48:41.582Z [INFO] server-d3d508de-49e3-b3e4-33e1-5930ee54cf5b: entering leader state: leader="Node at d3d508de-49e3-b3e4-33e1-5930ee54cf5b [Leader]" 2025-01-28T14:48:41.583Z [INFO] server-d3d508de-49e3-b3e4-33e1-5930ee54cf5b: added peer, starting replication: peer=server-81272f9d-52cd-323c-d943-3d51c0be810e 2025-01-28T14:48:41.583Z [INFO] server-d3d508de-49e3-b3e4-33e1-5930ee54cf5b: added peer, starting replication: peer=server-11c2317d-1b03-ec9f-bce6-04efd5e1ef99 2025-01-28T14:48:41.583Z [WARN] server-11c2317d-1b03-ec9f-bce6-04efd5e1ef99: rejecting vote request since our last term is greater: candidate=d3d508de-49e3-b3e4-33e1-5930ee54cf5b last-term=2 last-candidate-term=1 2025-01-28T14:48:41.583Z [INFO] server-d3d508de-49e3-b3e4-33e1-5930ee54cf5b: pipelining replication: peer="{Voter server-81272f9d-52cd-323c-d943-3d51c0be810e 81272f9d-52cd-323c-d943-3d51c0be810e}" 2025-01-28T14:48:41.584Z [INFO] server-d3d508de-49e3-b3e4-33e1-5930ee54cf5b: pipelining replication: peer="{Voter server-11c2317d-1b03-ec9f-bce6-04efd5e1ef99 11c2317d-1b03-ec9f-bce6-04efd5e1ef99}" 2025-01-28T14:48:41.691Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:41.584127484 +0000 UTC m=+104.667532509 (2 nodes), 80.335053ms from start of poll, 86.308897ms from cluster start. Timeout at 2025-01-28 14:48:41.689146818 +0000 UTC m=+104.772551510, 105.019001ms after stability 2025-01-28T14:48:41.691Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:41.796Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:41.691312001 +0000 UTC m=+104.774716692 (1 nodes), 5.667µs from start of poll, 193.49308ms from cluster start. Timeout at 2025-01-28 14:48:41.796761347 +0000 UTC m=+104.880166372, 105.44968ms after stability 2025-01-28T14:48:41.797Z [INFO] server-d3d508de-49e3-b3e4-33e1-5930ee54cf5b: aborting pipeline replication: peer="{Voter server-11c2317d-1b03-ec9f-bce6-04efd5e1ef99 11c2317d-1b03-ec9f-bce6-04efd5e1ef99}" 2025-01-28T14:48:41.797Z [INFO] server-d3d508de-49e3-b3e4-33e1-5930ee54cf5b: aborting pipeline replication: peer="{Voter server-81272f9d-52cd-323c-d943-3d51c0be810e 81272f9d-52cd-323c-d943-3d51c0be810e}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.31s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 raft_test.go:2151: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 raft_test.go:2196: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s) === RUN TestRaft_LeaderID_Propagated 2025-01-28T14:48:41.808Z [INFO] server-5fc5789d-ad75-ea7a-ce82-2306bbd06e97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fc5789d-ad75-ea7a-ce82-2306bbd06e97 Address:5fc5789d-ad75-ea7a-ce82-2306bbd06e97} {Suffrage:Voter ID:server-dbc2434c-2f9c-d5be-52f3-acdba4809358 Address:dbc2434c-2f9c-d5be-52f3-acdba4809358} {Suffrage:Voter ID:server-549a53a8-7c78-b20d-2257-020fdd6fde79 Address:549a53a8-7c78-b20d-2257-020fdd6fde79}]" 2025-01-28T14:48:41.812Z [INFO] server-dbc2434c-2f9c-d5be-52f3-acdba4809358: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fc5789d-ad75-ea7a-ce82-2306bbd06e97 Address:5fc5789d-ad75-ea7a-ce82-2306bbd06e97} {Suffrage:Voter ID:server-dbc2434c-2f9c-d5be-52f3-acdba4809358 Address:dbc2434c-2f9c-d5be-52f3-acdba4809358} {Suffrage:Voter ID:server-549a53a8-7c78-b20d-2257-020fdd6fde79 Address:549a53a8-7c78-b20d-2257-020fdd6fde79}]" 2025-01-28T14:48:41.813Z [INFO] server-549a53a8-7c78-b20d-2257-020fdd6fde79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fc5789d-ad75-ea7a-ce82-2306bbd06e97 Address:5fc5789d-ad75-ea7a-ce82-2306bbd06e97} {Suffrage:Voter ID:server-dbc2434c-2f9c-d5be-52f3-acdba4809358 Address:dbc2434c-2f9c-d5be-52f3-acdba4809358} {Suffrage:Voter ID:server-549a53a8-7c78-b20d-2257-020fdd6fde79 Address:549a53a8-7c78-b20d-2257-020fdd6fde79}]" 2025-01-28T14:48:41.813Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:41.817Z [INFO] server-5fc5789d-ad75-ea7a-ce82-2306bbd06e97: entering follower state: follower="Node at 5fc5789d-ad75-ea7a-ce82-2306bbd06e97 [Follower]" leader-address= leader-id= 2025-01-28T14:48:41.821Z [INFO] server-dbc2434c-2f9c-d5be-52f3-acdba4809358: entering follower state: follower="Node at dbc2434c-2f9c-d5be-52f3-acdba4809358 [Follower]" leader-address= leader-id= 2025-01-28T14:48:41.824Z [INFO] server-549a53a8-7c78-b20d-2257-020fdd6fde79: entering follower state: follower="Node at 549a53a8-7c78-b20d-2257-020fdd6fde79 [Follower]" leader-address= leader-id= 2025-01-28T14:48:41.876Z [WARN] server-5fc5789d-ad75-ea7a-ce82-2306bbd06e97: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:41.876Z [INFO] server-5fc5789d-ad75-ea7a-ce82-2306bbd06e97: entering candidate state: node="Node at 5fc5789d-ad75-ea7a-ce82-2306bbd06e97 [Candidate]" term=2 2025-01-28T14:48:41.877Z [INFO] server-5fc5789d-ad75-ea7a-ce82-2306bbd06e97: election won: term=2 tally=2 2025-01-28T14:48:41.877Z [INFO] server-5fc5789d-ad75-ea7a-ce82-2306bbd06e97: entering leader state: leader="Node at 5fc5789d-ad75-ea7a-ce82-2306bbd06e97 [Leader]" 2025-01-28T14:48:41.877Z [INFO] server-5fc5789d-ad75-ea7a-ce82-2306bbd06e97: added peer, starting replication: peer=server-dbc2434c-2f9c-d5be-52f3-acdba4809358 2025-01-28T14:48:41.878Z [INFO] server-5fc5789d-ad75-ea7a-ce82-2306bbd06e97: added peer, starting replication: peer=server-549a53a8-7c78-b20d-2257-020fdd6fde79 2025-01-28T14:48:41.879Z [INFO] server-5fc5789d-ad75-ea7a-ce82-2306bbd06e97: pipelining replication: peer="{Voter server-549a53a8-7c78-b20d-2257-020fdd6fde79 549a53a8-7c78-b20d-2257-020fdd6fde79}" 2025-01-28T14:48:41.879Z [INFO] server-5fc5789d-ad75-ea7a-ce82-2306bbd06e97: pipelining replication: peer="{Voter server-dbc2434c-2f9c-d5be-52f3-acdba4809358 dbc2434c-2f9c-d5be-52f3-acdba4809358}" 2025-01-28T14:48:41.983Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:41.878498893 +0000 UTC m=+104.961903917 (1 nodes), 64.936261ms from start of poll, 70.422424ms from cluster start. Timeout at 2025-01-28 14:48:41.983531561 +0000 UTC m=+105.066936585, 105.032668ms after stability 2025-01-28T14:48:41.984Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:42.090Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:41.984311369 +0000 UTC m=+105.067716060 (1 nodes), 6.334µs from start of poll, 176.234567ms from cluster start. Timeout at 2025-01-28 14:48:42.089950388 +0000 UTC m=+105.173355413, 105.639353ms after stability 2025-01-28T14:48:42.092Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:42.199Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:42.092464247 +0000 UTC m=+105.175868938 (1 nodes), 7.333µs from start of poll, 284.387445ms from cluster start. Timeout at 2025-01-28 14:48:42.197932262 +0000 UTC m=+105.281336953, 105.468015ms after stability 2025-01-28T14:48:42.201Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:42.307Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:42.202055835 +0000 UTC m=+105.285460526 (1 nodes), 5.666µs from start of poll, 393.979033ms from cluster start. Timeout at 2025-01-28 14:48:42.307527516 +0000 UTC m=+105.390932207, 105.471681ms after stability 2025-01-28T14:48:42.309Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:42.415Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:42.309350022 +0000 UTC m=+105.392754713 (1 nodes), 5.667µs from start of poll, 501.27322ms from cluster start. Timeout at 2025-01-28 14:48:42.414795368 +0000 UTC m=+105.498200059, 105.445346ms after stability 2025-01-28T14:48:42.416Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:42.523Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:42.41626653 +0000 UTC m=+105.499671888 (1 nodes), 6.334µs from start of poll, 608.190395ms from cluster start. Timeout at 2025-01-28 14:48:42.521819546 +0000 UTC m=+105.605224237, 105.552349ms after stability 2025-01-28T14:48:42.630Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:42.746Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:42.631104911 +0000 UTC m=+105.714509935 (1 nodes), 7.667µs from start of poll, 823.028442ms from cluster start. Timeout at 2025-01-28 14:48:42.736439254 +0000 UTC m=+105.819844278, 105.334343ms after stability 2025-01-28T14:48:42.747Z [INFO] server-5fc5789d-ad75-ea7a-ce82-2306bbd06e97: aborting pipeline replication: peer="{Voter server-549a53a8-7c78-b20d-2257-020fdd6fde79 549a53a8-7c78-b20d-2257-020fdd6fde79}" 2025-01-28T14:48:42.748Z [INFO] server-5fc5789d-ad75-ea7a-ce82-2306bbd06e97: aborting pipeline replication: peer="{Voter server-dbc2434c-2f9c-d5be-52f3-acdba4809358 dbc2434c-2f9c-d5be-52f3-acdba4809358}" --- PASS: TestRaft_LeaderID_Propagated (0.95s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer 2025-01-28T14:48:42.759Z [INFO] server-b607987d-b2f3-4c7d-28ea-d8df8113a996: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b607987d-b2f3-4c7d-28ea-d8df8113a996 Address:b607987d-b2f3-4c7d-28ea-d8df8113a996} {Suffrage:Voter ID:server-9c7d37f2-00e3-a392-65bb-e671d5b8b763 Address:9c7d37f2-00e3-a392-65bb-e671d5b8b763} {Suffrage:Voter ID:server-7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f Address:7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f}]" 2025-01-28T14:48:42.760Z [INFO] server-b607987d-b2f3-4c7d-28ea-d8df8113a996: entering follower state: follower="Node at b607987d-b2f3-4c7d-28ea-d8df8113a996 [Follower]" leader-address= leader-id= 2025-01-28T14:48:42.760Z [INFO] server-9c7d37f2-00e3-a392-65bb-e671d5b8b763: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b607987d-b2f3-4c7d-28ea-d8df8113a996 Address:b607987d-b2f3-4c7d-28ea-d8df8113a996} {Suffrage:Voter ID:server-9c7d37f2-00e3-a392-65bb-e671d5b8b763 Address:9c7d37f2-00e3-a392-65bb-e671d5b8b763} {Suffrage:Voter ID:server-7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f Address:7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f}]" 2025-01-28T14:48:42.760Z [INFO] server-9c7d37f2-00e3-a392-65bb-e671d5b8b763: entering follower state: follower="Node at 9c7d37f2-00e3-a392-65bb-e671d5b8b763 [Follower]" leader-address= leader-id= 2025-01-28T14:48:42.761Z [INFO] server-7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b607987d-b2f3-4c7d-28ea-d8df8113a996 Address:b607987d-b2f3-4c7d-28ea-d8df8113a996} {Suffrage:Voter ID:server-9c7d37f2-00e3-a392-65bb-e671d5b8b763 Address:9c7d37f2-00e3-a392-65bb-e671d5b8b763} {Suffrage:Voter ID:server-7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f Address:7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f}]" 2025-01-28T14:48:42.761Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:42.761Z [INFO] server-7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f: entering follower state: follower="Node at 7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f [Follower]" leader-address= leader-id= 2025-01-28T14:48:42.829Z [WARN] server-9c7d37f2-00e3-a392-65bb-e671d5b8b763: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:42.829Z [INFO] server-9c7d37f2-00e3-a392-65bb-e671d5b8b763: entering candidate state: node="Node at 9c7d37f2-00e3-a392-65bb-e671d5b8b763 [Candidate]" term=2 2025-01-28T14:48:42.830Z [INFO] server-9c7d37f2-00e3-a392-65bb-e671d5b8b763: election won: term=2 tally=2 2025-01-28T14:48:42.830Z [INFO] server-9c7d37f2-00e3-a392-65bb-e671d5b8b763: entering leader state: leader="Node at 9c7d37f2-00e3-a392-65bb-e671d5b8b763 [Leader]" 2025-01-28T14:48:42.830Z [INFO] server-9c7d37f2-00e3-a392-65bb-e671d5b8b763: added peer, starting replication: peer=server-b607987d-b2f3-4c7d-28ea-d8df8113a996 2025-01-28T14:48:42.830Z [INFO] server-9c7d37f2-00e3-a392-65bb-e671d5b8b763: added peer, starting replication: peer=server-7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f 2025-01-28T14:48:42.831Z [INFO] server-9c7d37f2-00e3-a392-65bb-e671d5b8b763: pipelining replication: peer="{Voter server-7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f 7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f}" 2025-01-28T14:48:42.831Z [INFO] server-9c7d37f2-00e3-a392-65bb-e671d5b8b763: pipelining replication: peer="{Voter server-b607987d-b2f3-4c7d-28ea-d8df8113a996 b607987d-b2f3-4c7d-28ea-d8df8113a996}" 2025-01-28T14:48:42.937Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:42.830568501 +0000 UTC m=+105.913973525 (1 nodes), 68.79971ms from start of poll, 71.48979ms from cluster start. Timeout at 2025-01-28 14:48:42.935579169 +0000 UTC m=+106.018983860, 105.010335ms after stability 2025-01-28T14:48:42.938Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:43.045Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:42.938142362 +0000 UTC m=+106.021547053 (1 nodes), 6.333µs from start of poll, 179.063318ms from cluster start. Timeout at 2025-01-28 14:48:43.043570709 +0000 UTC m=+106.126975733, 105.42868ms after stability 2025-01-28T14:48:43.051Z [INFO] server-b607987d-b2f3-4c7d-28ea-d8df8113a996: entering candidate state: node="Node at b607987d-b2f3-4c7d-28ea-d8df8113a996 [Candidate]" term=3 2025-01-28T14:48:43.053Z [INFO] server-9c7d37f2-00e3-a392-65bb-e671d5b8b763: entering follower state: follower="Node at 9c7d37f2-00e3-a392-65bb-e671d5b8b763 [Follower]" leader-address= leader-id= 2025-01-28T14:48:43.053Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:43.053Z [INFO] server-9c7d37f2-00e3-a392-65bb-e671d5b8b763: aborting pipeline replication: peer="{Voter server-7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f 7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f}" 2025-01-28T14:48:43.054Z [INFO] server-9c7d37f2-00e3-a392-65bb-e671d5b8b763: aborting pipeline replication: peer="{Voter server-b607987d-b2f3-4c7d-28ea-d8df8113a996 b607987d-b2f3-4c7d-28ea-d8df8113a996}" 2025-01-28T14:48:43.054Z [INFO] server-b607987d-b2f3-4c7d-28ea-d8df8113a996: election won: term=3 tally=2 2025-01-28T14:48:43.054Z [INFO] server-b607987d-b2f3-4c7d-28ea-d8df8113a996: entering leader state: leader="Node at b607987d-b2f3-4c7d-28ea-d8df8113a996 [Leader]" 2025-01-28T14:48:43.054Z [INFO] server-b607987d-b2f3-4c7d-28ea-d8df8113a996: added peer, starting replication: peer=server-9c7d37f2-00e3-a392-65bb-e671d5b8b763 2025-01-28T14:48:43.054Z [INFO] server-b607987d-b2f3-4c7d-28ea-d8df8113a996: added peer, starting replication: peer=server-7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f 2025-01-28T14:48:43.056Z [INFO] server-b607987d-b2f3-4c7d-28ea-d8df8113a996: pipelining replication: peer="{Voter server-9c7d37f2-00e3-a392-65bb-e671d5b8b763 9c7d37f2-00e3-a392-65bb-e671d5b8b763}" 2025-01-28T14:48:43.056Z [INFO] server-b607987d-b2f3-4c7d-28ea-d8df8113a996: pipelining replication: peer="{Voter server-7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f 7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f}" 2025-01-28T14:48:43.160Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:43.05546318 +0000 UTC m=+106.138868204 (1 nodes), 1.959058ms from start of poll, 296.384469ms from cluster start. Timeout at 2025-01-28 14:48:43.160505181 +0000 UTC m=+106.243909872, 105.041668ms after stability 2025-01-28T14:48:43.161Z [INFO] server-b607987d-b2f3-4c7d-28ea-d8df8113a996: aborting pipeline replication: peer="{Voter server-9c7d37f2-00e3-a392-65bb-e671d5b8b763 9c7d37f2-00e3-a392-65bb-e671d5b8b763}" 2025-01-28T14:48:43.161Z [INFO] server-b607987d-b2f3-4c7d-28ea-d8df8113a996: aborting pipeline replication: peer="{Voter server-7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f 7a8ff00a-a994-a0cc-dc84-bbb8febb5d8f}" --- PASS: TestRaft_LeadershipTransfer (0.41s) === RUN TestRaft_LeadershipTransferWithOneNode 2025-01-28T14:48:43.167Z [INFO] server-bcc368e0-c68a-7746-681c-a9ae5f00023d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcc368e0-c68a-7746-681c-a9ae5f00023d Address:bcc368e0-c68a-7746-681c-a9ae5f00023d}]" 2025-01-28T14:48:43.167Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:43.167Z [INFO] server-bcc368e0-c68a-7746-681c-a9ae5f00023d: entering follower state: follower="Node at bcc368e0-c68a-7746-681c-a9ae5f00023d [Follower]" leader-address= leader-id= 2025-01-28T14:48:43.226Z [WARN] server-bcc368e0-c68a-7746-681c-a9ae5f00023d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:43.226Z [INFO] server-bcc368e0-c68a-7746-681c-a9ae5f00023d: entering candidate state: node="Node at bcc368e0-c68a-7746-681c-a9ae5f00023d [Candidate]" term=2 2025-01-28T14:48:43.226Z [INFO] server-bcc368e0-c68a-7746-681c-a9ae5f00023d: election won: term=2 tally=1 2025-01-28T14:48:43.226Z [INFO] server-bcc368e0-c68a-7746-681c-a9ae5f00023d: entering leader state: leader="Node at bcc368e0-c68a-7746-681c-a9ae5f00023d [Leader]" 2025-01-28T14:48:43.333Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:43.227484622 +0000 UTC m=+106.310889313 (1 nodes), 59.736108ms from start of poll, 60.474796ms from cluster start. Timeout at 2025-01-28 14:48:43.332495955 +0000 UTC m=+106.415900980, 105.011667ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.17s) === RUN TestRaft_LeadershipTransferWithSevenNodes 2025-01-28T14:48:43.347Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af54d44b-015a-18e6-da85-46538a6c4186 Address:af54d44b-015a-18e6-da85-46538a6c4186} {Suffrage:Voter ID:server-995f613f-bb00-44bd-a796-36ce725eb03a Address:995f613f-bb00-44bd-a796-36ce725eb03a} {Suffrage:Voter ID:server-9b7543b8-bf54-9d8a-5f42-a571c32df417 Address:9b7543b8-bf54-9d8a-5f42-a571c32df417} {Suffrage:Voter ID:server-45375828-775a-7428-e34a-83c3dfd99ef9 Address:45375828-775a-7428-e34a-83c3dfd99ef9} {Suffrage:Voter ID:server-2577410f-84cf-07c4-74d3-e5d05021e9d7 Address:2577410f-84cf-07c4-74d3-e5d05021e9d7} {Suffrage:Voter ID:server-9fdd3a18-c17b-507e-24f3-d63df555b804 Address:9fdd3a18-c17b-507e-24f3-d63df555b804} {Suffrage:Voter ID:server-faea0b1f-5b46-7485-0d7f-f26a767ab98a Address:faea0b1f-5b46-7485-0d7f-f26a767ab98a}]" 2025-01-28T14:48:43.348Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: entering follower state: follower="Node at af54d44b-015a-18e6-da85-46538a6c4186 [Follower]" leader-address= leader-id= 2025-01-28T14:48:43.348Z [INFO] server-995f613f-bb00-44bd-a796-36ce725eb03a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af54d44b-015a-18e6-da85-46538a6c4186 Address:af54d44b-015a-18e6-da85-46538a6c4186} {Suffrage:Voter ID:server-995f613f-bb00-44bd-a796-36ce725eb03a Address:995f613f-bb00-44bd-a796-36ce725eb03a} {Suffrage:Voter ID:server-9b7543b8-bf54-9d8a-5f42-a571c32df417 Address:9b7543b8-bf54-9d8a-5f42-a571c32df417} {Suffrage:Voter ID:server-45375828-775a-7428-e34a-83c3dfd99ef9 Address:45375828-775a-7428-e34a-83c3dfd99ef9} {Suffrage:Voter ID:server-2577410f-84cf-07c4-74d3-e5d05021e9d7 Address:2577410f-84cf-07c4-74d3-e5d05021e9d7} {Suffrage:Voter ID:server-9fdd3a18-c17b-507e-24f3-d63df555b804 Address:9fdd3a18-c17b-507e-24f3-d63df555b804} {Suffrage:Voter ID:server-faea0b1f-5b46-7485-0d7f-f26a767ab98a Address:faea0b1f-5b46-7485-0d7f-f26a767ab98a}]" 2025-01-28T14:48:43.348Z [INFO] server-995f613f-bb00-44bd-a796-36ce725eb03a: entering follower state: follower="Node at 995f613f-bb00-44bd-a796-36ce725eb03a [Follower]" leader-address= leader-id= 2025-01-28T14:48:43.349Z [INFO] server-9b7543b8-bf54-9d8a-5f42-a571c32df417: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af54d44b-015a-18e6-da85-46538a6c4186 Address:af54d44b-015a-18e6-da85-46538a6c4186} {Suffrage:Voter ID:server-995f613f-bb00-44bd-a796-36ce725eb03a Address:995f613f-bb00-44bd-a796-36ce725eb03a} {Suffrage:Voter ID:server-9b7543b8-bf54-9d8a-5f42-a571c32df417 Address:9b7543b8-bf54-9d8a-5f42-a571c32df417} {Suffrage:Voter ID:server-45375828-775a-7428-e34a-83c3dfd99ef9 Address:45375828-775a-7428-e34a-83c3dfd99ef9} {Suffrage:Voter ID:server-2577410f-84cf-07c4-74d3-e5d05021e9d7 Address:2577410f-84cf-07c4-74d3-e5d05021e9d7} {Suffrage:Voter ID:server-9fdd3a18-c17b-507e-24f3-d63df555b804 Address:9fdd3a18-c17b-507e-24f3-d63df555b804} {Suffrage:Voter ID:server-faea0b1f-5b46-7485-0d7f-f26a767ab98a Address:faea0b1f-5b46-7485-0d7f-f26a767ab98a}]" 2025-01-28T14:48:43.350Z [INFO] server-9b7543b8-bf54-9d8a-5f42-a571c32df417: entering follower state: follower="Node at 9b7543b8-bf54-9d8a-5f42-a571c32df417 [Follower]" leader-address= leader-id= 2025-01-28T14:48:43.350Z [INFO] server-45375828-775a-7428-e34a-83c3dfd99ef9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af54d44b-015a-18e6-da85-46538a6c4186 Address:af54d44b-015a-18e6-da85-46538a6c4186} {Suffrage:Voter ID:server-995f613f-bb00-44bd-a796-36ce725eb03a Address:995f613f-bb00-44bd-a796-36ce725eb03a} {Suffrage:Voter ID:server-9b7543b8-bf54-9d8a-5f42-a571c32df417 Address:9b7543b8-bf54-9d8a-5f42-a571c32df417} {Suffrage:Voter ID:server-45375828-775a-7428-e34a-83c3dfd99ef9 Address:45375828-775a-7428-e34a-83c3dfd99ef9} {Suffrage:Voter ID:server-2577410f-84cf-07c4-74d3-e5d05021e9d7 Address:2577410f-84cf-07c4-74d3-e5d05021e9d7} {Suffrage:Voter ID:server-9fdd3a18-c17b-507e-24f3-d63df555b804 Address:9fdd3a18-c17b-507e-24f3-d63df555b804} {Suffrage:Voter ID:server-faea0b1f-5b46-7485-0d7f-f26a767ab98a Address:faea0b1f-5b46-7485-0d7f-f26a767ab98a}]" 2025-01-28T14:48:43.351Z [INFO] server-45375828-775a-7428-e34a-83c3dfd99ef9: entering follower state: follower="Node at 45375828-775a-7428-e34a-83c3dfd99ef9 [Follower]" leader-address= leader-id= 2025-01-28T14:48:43.351Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af54d44b-015a-18e6-da85-46538a6c4186 Address:af54d44b-015a-18e6-da85-46538a6c4186} {Suffrage:Voter ID:server-995f613f-bb00-44bd-a796-36ce725eb03a Address:995f613f-bb00-44bd-a796-36ce725eb03a} {Suffrage:Voter ID:server-9b7543b8-bf54-9d8a-5f42-a571c32df417 Address:9b7543b8-bf54-9d8a-5f42-a571c32df417} {Suffrage:Voter ID:server-45375828-775a-7428-e34a-83c3dfd99ef9 Address:45375828-775a-7428-e34a-83c3dfd99ef9} {Suffrage:Voter ID:server-2577410f-84cf-07c4-74d3-e5d05021e9d7 Address:2577410f-84cf-07c4-74d3-e5d05021e9d7} {Suffrage:Voter ID:server-9fdd3a18-c17b-507e-24f3-d63df555b804 Address:9fdd3a18-c17b-507e-24f3-d63df555b804} {Suffrage:Voter ID:server-faea0b1f-5b46-7485-0d7f-f26a767ab98a Address:faea0b1f-5b46-7485-0d7f-f26a767ab98a}]" 2025-01-28T14:48:43.352Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: entering follower state: follower="Node at 2577410f-84cf-07c4-74d3-e5d05021e9d7 [Follower]" leader-address= leader-id= 2025-01-28T14:48:43.355Z [INFO] server-9fdd3a18-c17b-507e-24f3-d63df555b804: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af54d44b-015a-18e6-da85-46538a6c4186 Address:af54d44b-015a-18e6-da85-46538a6c4186} {Suffrage:Voter ID:server-995f613f-bb00-44bd-a796-36ce725eb03a Address:995f613f-bb00-44bd-a796-36ce725eb03a} {Suffrage:Voter ID:server-9b7543b8-bf54-9d8a-5f42-a571c32df417 Address:9b7543b8-bf54-9d8a-5f42-a571c32df417} {Suffrage:Voter ID:server-45375828-775a-7428-e34a-83c3dfd99ef9 Address:45375828-775a-7428-e34a-83c3dfd99ef9} {Suffrage:Voter ID:server-2577410f-84cf-07c4-74d3-e5d05021e9d7 Address:2577410f-84cf-07c4-74d3-e5d05021e9d7} {Suffrage:Voter ID:server-9fdd3a18-c17b-507e-24f3-d63df555b804 Address:9fdd3a18-c17b-507e-24f3-d63df555b804} {Suffrage:Voter ID:server-faea0b1f-5b46-7485-0d7f-f26a767ab98a Address:faea0b1f-5b46-7485-0d7f-f26a767ab98a}]" 2025-01-28T14:48:43.356Z [INFO] server-faea0b1f-5b46-7485-0d7f-f26a767ab98a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af54d44b-015a-18e6-da85-46538a6c4186 Address:af54d44b-015a-18e6-da85-46538a6c4186} {Suffrage:Voter ID:server-995f613f-bb00-44bd-a796-36ce725eb03a Address:995f613f-bb00-44bd-a796-36ce725eb03a} {Suffrage:Voter ID:server-9b7543b8-bf54-9d8a-5f42-a571c32df417 Address:9b7543b8-bf54-9d8a-5f42-a571c32df417} {Suffrage:Voter ID:server-45375828-775a-7428-e34a-83c3dfd99ef9 Address:45375828-775a-7428-e34a-83c3dfd99ef9} {Suffrage:Voter ID:server-2577410f-84cf-07c4-74d3-e5d05021e9d7 Address:2577410f-84cf-07c4-74d3-e5d05021e9d7} {Suffrage:Voter ID:server-9fdd3a18-c17b-507e-24f3-d63df555b804 Address:9fdd3a18-c17b-507e-24f3-d63df555b804} {Suffrage:Voter ID:server-faea0b1f-5b46-7485-0d7f-f26a767ab98a Address:faea0b1f-5b46-7485-0d7f-f26a767ab98a}]" 2025-01-28T14:48:43.356Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:43.362Z [INFO] server-9fdd3a18-c17b-507e-24f3-d63df555b804: entering follower state: follower="Node at 9fdd3a18-c17b-507e-24f3-d63df555b804 [Follower]" leader-address= leader-id= 2025-01-28T14:48:43.365Z [INFO] server-faea0b1f-5b46-7485-0d7f-f26a767ab98a: entering follower state: follower="Node at faea0b1f-5b46-7485-0d7f-f26a767ab98a [Follower]" leader-address= leader-id= 2025-01-28T14:48:43.409Z [WARN] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:43.409Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: entering candidate state: node="Node at 2577410f-84cf-07c4-74d3-e5d05021e9d7 [Candidate]" term=2 2025-01-28T14:48:43.410Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: election won: term=2 tally=4 2025-01-28T14:48:43.410Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: entering leader state: leader="Node at 2577410f-84cf-07c4-74d3-e5d05021e9d7 [Leader]" 2025-01-28T14:48:43.410Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: added peer, starting replication: peer=server-af54d44b-015a-18e6-da85-46538a6c4186 2025-01-28T14:48:43.411Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: added peer, starting replication: peer=server-995f613f-bb00-44bd-a796-36ce725eb03a 2025-01-28T14:48:43.411Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: added peer, starting replication: peer=server-9b7543b8-bf54-9d8a-5f42-a571c32df417 2025-01-28T14:48:43.411Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: added peer, starting replication: peer=server-45375828-775a-7428-e34a-83c3dfd99ef9 2025-01-28T14:48:43.411Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: added peer, starting replication: peer=server-9fdd3a18-c17b-507e-24f3-d63df555b804 2025-01-28T14:48:43.411Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: added peer, starting replication: peer=server-faea0b1f-5b46-7485-0d7f-f26a767ab98a 2025-01-28T14:48:43.411Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: pipelining replication: peer="{Voter server-af54d44b-015a-18e6-da85-46538a6c4186 af54d44b-015a-18e6-da85-46538a6c4186}" 2025-01-28T14:48:43.412Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: pipelining replication: peer="{Voter server-995f613f-bb00-44bd-a796-36ce725eb03a 995f613f-bb00-44bd-a796-36ce725eb03a}" 2025-01-28T14:48:43.412Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: pipelining replication: peer="{Voter server-9fdd3a18-c17b-507e-24f3-d63df555b804 9fdd3a18-c17b-507e-24f3-d63df555b804}" 2025-01-28T14:48:43.412Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: pipelining replication: peer="{Voter server-faea0b1f-5b46-7485-0d7f-f26a767ab98a faea0b1f-5b46-7485-0d7f-f26a767ab98a}" 2025-01-28T14:48:43.413Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: pipelining replication: peer="{Voter server-45375828-775a-7428-e34a-83c3dfd99ef9 45375828-775a-7428-e34a-83c3dfd99ef9}" 2025-01-28T14:48:43.414Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: pipelining replication: peer="{Voter server-9b7543b8-bf54-9d8a-5f42-a571c32df417 9b7543b8-bf54-9d8a-5f42-a571c32df417}" 2025-01-28T14:48:43.516Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:43.410593726 +0000 UTC m=+106.493999084 (1 nodes), 53.572591ms from start of poll, 63.792228ms from cluster start. Timeout at 2025-01-28 14:48:43.51560306 +0000 UTC m=+106.599008085, 105.009001ms after stability 2025-01-28T14:48:43.516Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:43.623Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:43.516410869 +0000 UTC m=+106.599815560 (6 nodes), 11.334µs from start of poll, 169.608704ms from cluster start. Timeout at 2025-01-28 14:48:43.621950885 +0000 UTC m=+106.705355909, 105.540349ms after stability 2025-01-28T14:48:43.623Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:43.729Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:43.623658387 +0000 UTC m=+106.707063078 (1 nodes), 9µs from start of poll, 276.856222ms from cluster start. Timeout at 2025-01-28 14:48:43.729067066 +0000 UTC m=+106.812472090, 105.409012ms after stability 2025-01-28T14:48:43.729Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: entering candidate state: node="Node at af54d44b-015a-18e6-da85-46538a6c4186 [Candidate]" term=3 2025-01-28T14:48:43.731Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:43.731Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: election won: term=3 tally=4 2025-01-28T14:48:43.731Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: entering follower state: follower="Node at 2577410f-84cf-07c4-74d3-e5d05021e9d7 [Follower]" leader-address= leader-id= 2025-01-28T14:48:43.731Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: entering leader state: leader="Node at af54d44b-015a-18e6-da85-46538a6c4186 [Leader]" 2025-01-28T14:48:43.732Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: added peer, starting replication: peer=server-995f613f-bb00-44bd-a796-36ce725eb03a 2025-01-28T14:48:43.732Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: added peer, starting replication: peer=server-9b7543b8-bf54-9d8a-5f42-a571c32df417 2025-01-28T14:48:43.732Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: added peer, starting replication: peer=server-45375828-775a-7428-e34a-83c3dfd99ef9 2025-01-28T14:48:43.732Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: added peer, starting replication: peer=server-2577410f-84cf-07c4-74d3-e5d05021e9d7 2025-01-28T14:48:43.732Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: aborting pipeline replication: peer="{Voter server-9fdd3a18-c17b-507e-24f3-d63df555b804 9fdd3a18-c17b-507e-24f3-d63df555b804}" 2025-01-28T14:48:43.732Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: added peer, starting replication: peer=server-9fdd3a18-c17b-507e-24f3-d63df555b804 2025-01-28T14:48:43.732Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: added peer, starting replication: peer=server-faea0b1f-5b46-7485-0d7f-f26a767ab98a 2025-01-28T14:48:43.732Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: aborting pipeline replication: peer="{Voter server-9b7543b8-bf54-9d8a-5f42-a571c32df417 9b7543b8-bf54-9d8a-5f42-a571c32df417}" 2025-01-28T14:48:43.732Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: aborting pipeline replication: peer="{Voter server-45375828-775a-7428-e34a-83c3dfd99ef9 45375828-775a-7428-e34a-83c3dfd99ef9}" 2025-01-28T14:48:43.733Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: aborting pipeline replication: peer="{Voter server-faea0b1f-5b46-7485-0d7f-f26a767ab98a faea0b1f-5b46-7485-0d7f-f26a767ab98a}" 2025-01-28T14:48:43.733Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: aborting pipeline replication: peer="{Voter server-af54d44b-015a-18e6-da85-46538a6c4186 af54d44b-015a-18e6-da85-46538a6c4186}" 2025-01-28T14:48:43.733Z [INFO] server-2577410f-84cf-07c4-74d3-e5d05021e9d7: aborting pipeline replication: peer="{Voter server-995f613f-bb00-44bd-a796-36ce725eb03a 995f613f-bb00-44bd-a796-36ce725eb03a}" 2025-01-28T14:48:43.733Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: pipelining replication: peer="{Voter server-2577410f-84cf-07c4-74d3-e5d05021e9d7 2577410f-84cf-07c4-74d3-e5d05021e9d7}" 2025-01-28T14:48:43.733Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: pipelining replication: peer="{Voter server-995f613f-bb00-44bd-a796-36ce725eb03a 995f613f-bb00-44bd-a796-36ce725eb03a}" 2025-01-28T14:48:43.734Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: pipelining replication: peer="{Voter server-faea0b1f-5b46-7485-0d7f-f26a767ab98a faea0b1f-5b46-7485-0d7f-f26a767ab98a}" 2025-01-28T14:48:43.734Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: pipelining replication: peer="{Voter server-9fdd3a18-c17b-507e-24f3-d63df555b804 9fdd3a18-c17b-507e-24f3-d63df555b804}" 2025-01-28T14:48:43.734Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: pipelining replication: peer="{Voter server-9b7543b8-bf54-9d8a-5f42-a571c32df417 9b7543b8-bf54-9d8a-5f42-a571c32df417}" 2025-01-28T14:48:43.733Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: pipelining replication: peer="{Voter server-45375828-775a-7428-e34a-83c3dfd99ef9 45375828-775a-7428-e34a-83c3dfd99ef9}" 2025-01-28T14:48:43.840Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:43.734250005 +0000 UTC m=+106.817654696 (1 nodes), 2.908086ms from start of poll, 387.44784ms from cluster start. Timeout at 2025-01-28 14:48:43.839323674 +0000 UTC m=+106.922728365, 105.073669ms after stability 2025-01-28T14:48:43.841Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: aborting pipeline replication: peer="{Voter server-faea0b1f-5b46-7485-0d7f-f26a767ab98a faea0b1f-5b46-7485-0d7f-f26a767ab98a}" 2025-01-28T14:48:43.841Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: aborting pipeline replication: peer="{Voter server-45375828-775a-7428-e34a-83c3dfd99ef9 45375828-775a-7428-e34a-83c3dfd99ef9}" 2025-01-28T14:48:43.841Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: aborting pipeline replication: peer="{Voter server-995f613f-bb00-44bd-a796-36ce725eb03a 995f613f-bb00-44bd-a796-36ce725eb03a}" 2025-01-28T14:48:43.841Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: aborting pipeline replication: peer="{Voter server-9b7543b8-bf54-9d8a-5f42-a571c32df417 9b7543b8-bf54-9d8a-5f42-a571c32df417}" 2025-01-28T14:48:43.841Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: aborting pipeline replication: peer="{Voter server-9fdd3a18-c17b-507e-24f3-d63df555b804 9fdd3a18-c17b-507e-24f3-d63df555b804}" 2025-01-28T14:48:43.841Z [INFO] server-af54d44b-015a-18e6-da85-46538a6c4186: aborting pipeline replication: peer="{Voter server-2577410f-84cf-07c4-74d3-e5d05021e9d7 2577410f-84cf-07c4-74d3-e5d05021e9d7}" 2025-01-28T14:48:44.340Z [ERROR] server-af54d44b-015a-18e6-da85-46538a6c4186: failed to heartbeat to: peer=9b7543b8-bf54-9d8a-5f42-a571c32df417 backoff time=10ms error="command timed out" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (1.01s) === RUN TestRaft_LeadershipTransferToInvalidID 2025-01-28T14:48:44.354Z [INFO] server-bcce9e67-cdac-8446-03eb-ac75fde74318: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcce9e67-cdac-8446-03eb-ac75fde74318 Address:bcce9e67-cdac-8446-03eb-ac75fde74318} {Suffrage:Voter ID:server-71c1c501-e310-74cd-06da-c96f93de7aa7 Address:71c1c501-e310-74cd-06da-c96f93de7aa7} {Suffrage:Voter ID:server-edac89a2-dc6f-a176-fa2b-5420b288a2a9 Address:edac89a2-dc6f-a176-fa2b-5420b288a2a9}]" 2025-01-28T14:48:44.355Z [INFO] server-71c1c501-e310-74cd-06da-c96f93de7aa7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcce9e67-cdac-8446-03eb-ac75fde74318 Address:bcce9e67-cdac-8446-03eb-ac75fde74318} {Suffrage:Voter ID:server-71c1c501-e310-74cd-06da-c96f93de7aa7 Address:71c1c501-e310-74cd-06da-c96f93de7aa7} {Suffrage:Voter ID:server-edac89a2-dc6f-a176-fa2b-5420b288a2a9 Address:edac89a2-dc6f-a176-fa2b-5420b288a2a9}]" 2025-01-28T14:48:44.356Z [INFO] server-bcce9e67-cdac-8446-03eb-ac75fde74318: entering follower state: follower="Node at bcce9e67-cdac-8446-03eb-ac75fde74318 [Follower]" leader-address= leader-id= 2025-01-28T14:48:44.356Z [INFO] server-71c1c501-e310-74cd-06da-c96f93de7aa7: entering follower state: follower="Node at 71c1c501-e310-74cd-06da-c96f93de7aa7 [Follower]" leader-address= leader-id= 2025-01-28T14:48:44.356Z [INFO] server-edac89a2-dc6f-a176-fa2b-5420b288a2a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcce9e67-cdac-8446-03eb-ac75fde74318 Address:bcce9e67-cdac-8446-03eb-ac75fde74318} {Suffrage:Voter ID:server-71c1c501-e310-74cd-06da-c96f93de7aa7 Address:71c1c501-e310-74cd-06da-c96f93de7aa7} {Suffrage:Voter ID:server-edac89a2-dc6f-a176-fa2b-5420b288a2a9 Address:edac89a2-dc6f-a176-fa2b-5420b288a2a9}]" 2025-01-28T14:48:44.356Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:44.356Z [INFO] server-edac89a2-dc6f-a176-fa2b-5420b288a2a9: entering follower state: follower="Node at edac89a2-dc6f-a176-fa2b-5420b288a2a9 [Follower]" leader-address= leader-id= 2025-01-28T14:48:44.418Z [WARN] server-71c1c501-e310-74cd-06da-c96f93de7aa7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:44.418Z [INFO] server-71c1c501-e310-74cd-06da-c96f93de7aa7: entering candidate state: node="Node at 71c1c501-e310-74cd-06da-c96f93de7aa7 [Candidate]" term=2 2025-01-28T14:48:44.419Z [INFO] server-71c1c501-e310-74cd-06da-c96f93de7aa7: election won: term=2 tally=2 2025-01-28T14:48:44.419Z [INFO] server-71c1c501-e310-74cd-06da-c96f93de7aa7: entering leader state: leader="Node at 71c1c501-e310-74cd-06da-c96f93de7aa7 [Leader]" 2025-01-28T14:48:44.419Z [INFO] server-71c1c501-e310-74cd-06da-c96f93de7aa7: added peer, starting replication: peer=server-bcce9e67-cdac-8446-03eb-ac75fde74318 2025-01-28T14:48:44.419Z [INFO] server-71c1c501-e310-74cd-06da-c96f93de7aa7: added peer, starting replication: peer=server-edac89a2-dc6f-a176-fa2b-5420b288a2a9 2025-01-28T14:48:44.440Z [INFO] server-71c1c501-e310-74cd-06da-c96f93de7aa7: pipelining replication: peer="{Voter server-edac89a2-dc6f-a176-fa2b-5420b288a2a9 edac89a2-dc6f-a176-fa2b-5420b288a2a9}" 2025-01-28T14:48:44.441Z [INFO] server-71c1c501-e310-74cd-06da-c96f93de7aa7: pipelining replication: peer="{Voter server-bcce9e67-cdac-8446-03eb-ac75fde74318 bcce9e67-cdac-8446-03eb-ac75fde74318}" 2025-01-28T14:48:44.524Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:44.419552691 +0000 UTC m=+107.502957715 (1 nodes), 62.62186ms from start of poll, 65.332607ms from cluster start. Timeout at 2025-01-28 14:48:44.524562024 +0000 UTC m=+107.607966715, 105.009ms after stability 2025-01-28T14:48:44.525Z [INFO] server-71c1c501-e310-74cd-06da-c96f93de7aa7: aborting pipeline replication: peer="{Voter server-edac89a2-dc6f-a176-fa2b-5420b288a2a9 edac89a2-dc6f-a176-fa2b-5420b288a2a9}" 2025-01-28T14:48:44.525Z [INFO] server-71c1c501-e310-74cd-06da-c96f93de7aa7: aborting pipeline replication: peer="{Voter server-bcce9e67-cdac-8446-03eb-ac75fde74318 bcce9e67-cdac-8446-03eb-ac75fde74318}" 2025-01-28T14:48:45.025Z [ERROR] server-71c1c501-e310-74cd-06da-c96f93de7aa7: failed to heartbeat to: peer=bcce9e67-cdac-8446-03eb-ac75fde74318 backoff time=10ms error="command timed out" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.68s) === RUN TestRaft_LeadershipTransferToInvalidAddress 2025-01-28T14:48:45.036Z [INFO] server-b3939ce3-51d9-ea5f-5fd4-92ddc181e72d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3939ce3-51d9-ea5f-5fd4-92ddc181e72d Address:b3939ce3-51d9-ea5f-5fd4-92ddc181e72d} {Suffrage:Voter ID:server-4a904f83-b0b9-6414-5409-934f37263ec1 Address:4a904f83-b0b9-6414-5409-934f37263ec1} {Suffrage:Voter ID:server-925a0735-4560-c63f-8b19-0a1fc7efaae2 Address:925a0735-4560-c63f-8b19-0a1fc7efaae2}]" 2025-01-28T14:48:45.036Z [INFO] server-b3939ce3-51d9-ea5f-5fd4-92ddc181e72d: entering follower state: follower="Node at b3939ce3-51d9-ea5f-5fd4-92ddc181e72d [Follower]" leader-address= leader-id= 2025-01-28T14:48:45.037Z [INFO] server-4a904f83-b0b9-6414-5409-934f37263ec1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3939ce3-51d9-ea5f-5fd4-92ddc181e72d Address:b3939ce3-51d9-ea5f-5fd4-92ddc181e72d} {Suffrage:Voter ID:server-4a904f83-b0b9-6414-5409-934f37263ec1 Address:4a904f83-b0b9-6414-5409-934f37263ec1} {Suffrage:Voter ID:server-925a0735-4560-c63f-8b19-0a1fc7efaae2 Address:925a0735-4560-c63f-8b19-0a1fc7efaae2}]" 2025-01-28T14:48:45.037Z [INFO] server-4a904f83-b0b9-6414-5409-934f37263ec1: entering follower state: follower="Node at 4a904f83-b0b9-6414-5409-934f37263ec1 [Follower]" leader-address= leader-id= 2025-01-28T14:48:45.038Z [INFO] server-925a0735-4560-c63f-8b19-0a1fc7efaae2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3939ce3-51d9-ea5f-5fd4-92ddc181e72d Address:b3939ce3-51d9-ea5f-5fd4-92ddc181e72d} {Suffrage:Voter ID:server-4a904f83-b0b9-6414-5409-934f37263ec1 Address:4a904f83-b0b9-6414-5409-934f37263ec1} {Suffrage:Voter ID:server-925a0735-4560-c63f-8b19-0a1fc7efaae2 Address:925a0735-4560-c63f-8b19-0a1fc7efaae2}]" 2025-01-28T14:48:45.038Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:45.038Z [INFO] server-925a0735-4560-c63f-8b19-0a1fc7efaae2: entering follower state: follower="Node at 925a0735-4560-c63f-8b19-0a1fc7efaae2 [Follower]" leader-address= leader-id= 2025-01-28T14:48:45.100Z [WARN] server-b3939ce3-51d9-ea5f-5fd4-92ddc181e72d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:45.100Z [INFO] server-b3939ce3-51d9-ea5f-5fd4-92ddc181e72d: entering candidate state: node="Node at b3939ce3-51d9-ea5f-5fd4-92ddc181e72d [Candidate]" term=2 2025-01-28T14:48:45.101Z [INFO] server-b3939ce3-51d9-ea5f-5fd4-92ddc181e72d: election won: term=2 tally=2 2025-01-28T14:48:45.101Z [INFO] server-b3939ce3-51d9-ea5f-5fd4-92ddc181e72d: entering leader state: leader="Node at b3939ce3-51d9-ea5f-5fd4-92ddc181e72d [Leader]" 2025-01-28T14:48:45.101Z [INFO] server-b3939ce3-51d9-ea5f-5fd4-92ddc181e72d: added peer, starting replication: peer=server-4a904f83-b0b9-6414-5409-934f37263ec1 2025-01-28T14:48:45.101Z [INFO] server-b3939ce3-51d9-ea5f-5fd4-92ddc181e72d: added peer, starting replication: peer=server-925a0735-4560-c63f-8b19-0a1fc7efaae2 2025-01-28T14:48:45.103Z [INFO] server-b3939ce3-51d9-ea5f-5fd4-92ddc181e72d: pipelining replication: peer="{Voter server-925a0735-4560-c63f-8b19-0a1fc7efaae2 925a0735-4560-c63f-8b19-0a1fc7efaae2}" 2025-01-28T14:48:45.103Z [INFO] server-b3939ce3-51d9-ea5f-5fd4-92ddc181e72d: pipelining replication: peer="{Voter server-4a904f83-b0b9-6414-5409-934f37263ec1 4a904f83-b0b9-6414-5409-934f37263ec1}" 2025-01-28T14:48:45.206Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:45.101130265 +0000 UTC m=+108.184535623 (2 nodes), 62.469188ms from start of poll, 65.271605ms from cluster start. Timeout at 2025-01-28 14:48:45.206221268 +0000 UTC m=+108.289626293, 105.09067ms after stability 2025-01-28T14:48:45.206Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:45.312Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:45.206887073 +0000 UTC m=+108.290292097 (1 nodes), 7µs from start of poll, 171.028079ms from cluster start. Timeout at 2025-01-28 14:48:45.31236042 +0000 UTC m=+108.395765111, 105.473014ms after stability 2025-01-28T14:48:45.313Z [INFO] server-b3939ce3-51d9-ea5f-5fd4-92ddc181e72d: aborting pipeline replication: peer="{Voter server-925a0735-4560-c63f-8b19-0a1fc7efaae2 925a0735-4560-c63f-8b19-0a1fc7efaae2}" 2025-01-28T14:48:45.313Z [INFO] server-b3939ce3-51d9-ea5f-5fd4-92ddc181e72d: aborting pipeline replication: peer="{Voter server-4a904f83-b0b9-6414-5409-934f37263ec1 4a904f83-b0b9-6414-5409-934f37263ec1}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.29s) === RUN TestRaft_LeadershipTransferToBehindServer 2025-01-28T14:48:45.322Z [INFO] server-7588cb91-eaf2-3bec-10c7-b9c66d449fb5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7588cb91-eaf2-3bec-10c7-b9c66d449fb5 Address:7588cb91-eaf2-3bec-10c7-b9c66d449fb5} {Suffrage:Voter ID:server-27644e4f-7bc4-5771-7a43-b70690d4c570 Address:27644e4f-7bc4-5771-7a43-b70690d4c570} {Suffrage:Voter ID:server-f2e555fc-6442-fcb9-6d01-ef7562218184 Address:f2e555fc-6442-fcb9-6d01-ef7562218184}]" 2025-01-28T14:48:45.323Z [INFO] server-27644e4f-7bc4-5771-7a43-b70690d4c570: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7588cb91-eaf2-3bec-10c7-b9c66d449fb5 Address:7588cb91-eaf2-3bec-10c7-b9c66d449fb5} {Suffrage:Voter ID:server-27644e4f-7bc4-5771-7a43-b70690d4c570 Address:27644e4f-7bc4-5771-7a43-b70690d4c570} {Suffrage:Voter ID:server-f2e555fc-6442-fcb9-6d01-ef7562218184 Address:f2e555fc-6442-fcb9-6d01-ef7562218184}]" 2025-01-28T14:48:45.323Z [INFO] server-7588cb91-eaf2-3bec-10c7-b9c66d449fb5: entering follower state: follower="Node at 7588cb91-eaf2-3bec-10c7-b9c66d449fb5 [Follower]" leader-address= leader-id= 2025-01-28T14:48:45.323Z [INFO] server-27644e4f-7bc4-5771-7a43-b70690d4c570: entering follower state: follower="Node at 27644e4f-7bc4-5771-7a43-b70690d4c570 [Follower]" leader-address= leader-id= 2025-01-28T14:48:45.323Z [INFO] server-f2e555fc-6442-fcb9-6d01-ef7562218184: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7588cb91-eaf2-3bec-10c7-b9c66d449fb5 Address:7588cb91-eaf2-3bec-10c7-b9c66d449fb5} {Suffrage:Voter ID:server-27644e4f-7bc4-5771-7a43-b70690d4c570 Address:27644e4f-7bc4-5771-7a43-b70690d4c570} {Suffrage:Voter ID:server-f2e555fc-6442-fcb9-6d01-ef7562218184 Address:f2e555fc-6442-fcb9-6d01-ef7562218184}]" 2025-01-28T14:48:45.323Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:45.324Z [INFO] server-f2e555fc-6442-fcb9-6d01-ef7562218184: entering follower state: follower="Node at f2e555fc-6442-fcb9-6d01-ef7562218184 [Follower]" leader-address= leader-id= 2025-01-28T14:48:45.386Z [WARN] server-27644e4f-7bc4-5771-7a43-b70690d4c570: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:45.386Z [INFO] server-27644e4f-7bc4-5771-7a43-b70690d4c570: entering candidate state: node="Node at 27644e4f-7bc4-5771-7a43-b70690d4c570 [Candidate]" term=2 2025-01-28T14:48:45.387Z [INFO] server-27644e4f-7bc4-5771-7a43-b70690d4c570: election won: term=2 tally=2 2025-01-28T14:48:45.387Z [INFO] server-27644e4f-7bc4-5771-7a43-b70690d4c570: entering leader state: leader="Node at 27644e4f-7bc4-5771-7a43-b70690d4c570 [Leader]" 2025-01-28T14:48:45.387Z [INFO] server-27644e4f-7bc4-5771-7a43-b70690d4c570: added peer, starting replication: peer=server-7588cb91-eaf2-3bec-10c7-b9c66d449fb5 2025-01-28T14:48:45.387Z [INFO] server-27644e4f-7bc4-5771-7a43-b70690d4c570: added peer, starting replication: peer=server-f2e555fc-6442-fcb9-6d01-ef7562218184 2025-01-28T14:48:45.388Z [INFO] server-27644e4f-7bc4-5771-7a43-b70690d4c570: pipelining replication: peer="{Voter server-7588cb91-eaf2-3bec-10c7-b9c66d449fb5 7588cb91-eaf2-3bec-10c7-b9c66d449fb5}" 2025-01-28T14:48:45.389Z [INFO] server-27644e4f-7bc4-5771-7a43-b70690d4c570: pipelining replication: peer="{Voter server-f2e555fc-6442-fcb9-6d01-ef7562218184 f2e555fc-6442-fcb9-6d01-ef7562218184}" 2025-01-28T14:48:45.493Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:45.388313128 +0000 UTC m=+108.471718152 (1 nodes), 64.271575ms from start of poll, 66.392305ms from cluster start. Timeout at 2025-01-28 14:48:45.493334795 +0000 UTC m=+108.576739820, 105.021668ms after stability 2025-01-28T14:48:45.493Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:45.599Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:45.493831928 +0000 UTC m=+108.577236952 (2 nodes), 9.333µs from start of poll, 171.911105ms from cluster start. Timeout at 2025-01-28 14:48:45.599393945 +0000 UTC m=+108.682798969, 105.562017ms after stability 2025-01-28T14:48:46.078Z [INFO] server-7588cb91-eaf2-3bec-10c7-b9c66d449fb5: entering candidate state: node="Node at 7588cb91-eaf2-3bec-10c7-b9c66d449fb5 [Candidate]" term=3 2025-01-28T14:48:46.078Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:46.079Z [INFO] server-27644e4f-7bc4-5771-7a43-b70690d4c570: entering follower state: follower="Node at 27644e4f-7bc4-5771-7a43-b70690d4c570 [Follower]" leader-address= leader-id= 2025-01-28T14:48:46.079Z [INFO] server-7588cb91-eaf2-3bec-10c7-b9c66d449fb5: election won: term=3 tally=2 2025-01-28T14:48:46.079Z [INFO] server-7588cb91-eaf2-3bec-10c7-b9c66d449fb5: entering leader state: leader="Node at 7588cb91-eaf2-3bec-10c7-b9c66d449fb5 [Leader]" 2025-01-28T14:48:46.079Z [INFO] server-27644e4f-7bc4-5771-7a43-b70690d4c570: aborting pipeline replication: peer="{Voter server-f2e555fc-6442-fcb9-6d01-ef7562218184 f2e555fc-6442-fcb9-6d01-ef7562218184}" 2025-01-28T14:48:46.079Z [INFO] server-7588cb91-eaf2-3bec-10c7-b9c66d449fb5: added peer, starting replication: peer=server-27644e4f-7bc4-5771-7a43-b70690d4c570 2025-01-28T14:48:46.079Z [INFO] server-27644e4f-7bc4-5771-7a43-b70690d4c570: aborting pipeline replication: peer="{Voter server-7588cb91-eaf2-3bec-10c7-b9c66d449fb5 7588cb91-eaf2-3bec-10c7-b9c66d449fb5}" 2025-01-28T14:48:46.079Z [INFO] server-7588cb91-eaf2-3bec-10c7-b9c66d449fb5: added peer, starting replication: peer=server-f2e555fc-6442-fcb9-6d01-ef7562218184 2025-01-28T14:48:46.080Z [INFO] server-7588cb91-eaf2-3bec-10c7-b9c66d449fb5: pipelining replication: peer="{Voter server-f2e555fc-6442-fcb9-6d01-ef7562218184 f2e555fc-6442-fcb9-6d01-ef7562218184}" 2025-01-28T14:48:46.080Z [INFO] server-7588cb91-eaf2-3bec-10c7-b9c66d449fb5: pipelining replication: peer="{Voter server-27644e4f-7bc4-5771-7a43-b70690d4c570 27644e4f-7bc4-5771-7a43-b70690d4c570}" 2025-01-28T14:48:46.185Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:46.07957499 +0000 UTC m=+109.162979681 (1 nodes), 667.02µs from start of poll, 757.653834ms from cluster start. Timeout at 2025-01-28 14:48:46.184701661 +0000 UTC m=+109.268106686, 105.127005ms after stability 2025-01-28T14:48:46.185Z [INFO] server-7588cb91-eaf2-3bec-10c7-b9c66d449fb5: aborting pipeline replication: peer="{Voter server-f2e555fc-6442-fcb9-6d01-ef7562218184 f2e555fc-6442-fcb9-6d01-ef7562218184}" 2025-01-28T14:48:46.185Z [INFO] server-7588cb91-eaf2-3bec-10c7-b9c66d449fb5: aborting pipeline replication: peer="{Voter server-27644e4f-7bc4-5771-7a43-b70690d4c570 27644e4f-7bc4-5771-7a43-b70690d4c570}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.87s) === RUN TestRaft_LeadershipTransferToItself 2025-01-28T14:48:46.196Z [INFO] server-6bd40765-8b95-ac6c-7661-da9c1959975e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6bd40765-8b95-ac6c-7661-da9c1959975e Address:6bd40765-8b95-ac6c-7661-da9c1959975e} {Suffrage:Voter ID:server-98e2625f-f9f3-bb2f-af74-74bb426c2829 Address:98e2625f-f9f3-bb2f-af74-74bb426c2829} {Suffrage:Voter ID:server-fa269533-67e1-c719-201a-f91ad8d8b0b5 Address:fa269533-67e1-c719-201a-f91ad8d8b0b5}]" 2025-01-28T14:48:46.196Z [INFO] server-98e2625f-f9f3-bb2f-af74-74bb426c2829: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6bd40765-8b95-ac6c-7661-da9c1959975e Address:6bd40765-8b95-ac6c-7661-da9c1959975e} {Suffrage:Voter ID:server-98e2625f-f9f3-bb2f-af74-74bb426c2829 Address:98e2625f-f9f3-bb2f-af74-74bb426c2829} {Suffrage:Voter ID:server-fa269533-67e1-c719-201a-f91ad8d8b0b5 Address:fa269533-67e1-c719-201a-f91ad8d8b0b5}]" 2025-01-28T14:48:46.196Z [INFO] server-6bd40765-8b95-ac6c-7661-da9c1959975e: entering follower state: follower="Node at 6bd40765-8b95-ac6c-7661-da9c1959975e [Follower]" leader-address= leader-id= 2025-01-28T14:48:46.197Z [INFO] server-fa269533-67e1-c719-201a-f91ad8d8b0b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6bd40765-8b95-ac6c-7661-da9c1959975e Address:6bd40765-8b95-ac6c-7661-da9c1959975e} {Suffrage:Voter ID:server-98e2625f-f9f3-bb2f-af74-74bb426c2829 Address:98e2625f-f9f3-bb2f-af74-74bb426c2829} {Suffrage:Voter ID:server-fa269533-67e1-c719-201a-f91ad8d8b0b5 Address:fa269533-67e1-c719-201a-f91ad8d8b0b5}]" 2025-01-28T14:48:46.197Z [INFO] server-98e2625f-f9f3-bb2f-af74-74bb426c2829: entering follower state: follower="Node at 98e2625f-f9f3-bb2f-af74-74bb426c2829 [Follower]" leader-address= leader-id= 2025-01-28T14:48:46.197Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:46.197Z [INFO] server-fa269533-67e1-c719-201a-f91ad8d8b0b5: entering follower state: follower="Node at fa269533-67e1-c719-201a-f91ad8d8b0b5 [Follower]" leader-address= leader-id= 2025-01-28T14:48:46.258Z [WARN] server-fa269533-67e1-c719-201a-f91ad8d8b0b5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:46.258Z [INFO] server-fa269533-67e1-c719-201a-f91ad8d8b0b5: entering candidate state: node="Node at fa269533-67e1-c719-201a-f91ad8d8b0b5 [Candidate]" term=2 2025-01-28T14:48:46.258Z [INFO] server-fa269533-67e1-c719-201a-f91ad8d8b0b5: election won: term=2 tally=2 2025-01-28T14:48:46.258Z [INFO] server-fa269533-67e1-c719-201a-f91ad8d8b0b5: entering leader state: leader="Node at fa269533-67e1-c719-201a-f91ad8d8b0b5 [Leader]" 2025-01-28T14:48:46.259Z [INFO] server-fa269533-67e1-c719-201a-f91ad8d8b0b5: added peer, starting replication: peer=server-6bd40765-8b95-ac6c-7661-da9c1959975e 2025-01-28T14:48:46.259Z [INFO] server-fa269533-67e1-c719-201a-f91ad8d8b0b5: added peer, starting replication: peer=server-98e2625f-f9f3-bb2f-af74-74bb426c2829 2025-01-28T14:48:46.259Z [WARN] server-6bd40765-8b95-ac6c-7661-da9c1959975e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:46.259Z [INFO] server-6bd40765-8b95-ac6c-7661-da9c1959975e: entering candidate state: node="Node at 6bd40765-8b95-ac6c-7661-da9c1959975e [Candidate]" term=2 2025-01-28T14:48:46.260Z [INFO] server-6bd40765-8b95-ac6c-7661-da9c1959975e: entering follower state: follower="Node at 6bd40765-8b95-ac6c-7661-da9c1959975e [Follower]" leader-address=fa269533-67e1-c719-201a-f91ad8d8b0b5 leader-id=server-fa269533-67e1-c719-201a-f91ad8d8b0b5 2025-01-28T14:48:46.260Z [INFO] server-fa269533-67e1-c719-201a-f91ad8d8b0b5: pipelining replication: peer="{Voter server-98e2625f-f9f3-bb2f-af74-74bb426c2829 98e2625f-f9f3-bb2f-af74-74bb426c2829}" 2025-01-28T14:48:46.260Z [INFO] server-6bd40765-8b95-ac6c-7661-da9c1959975e: duplicate requestVote for same term: term=2 2025-01-28T14:48:46.260Z [WARN] server-98e2625f-f9f3-bb2f-af74-74bb426c2829: rejecting vote request since we have a leader: from=6bd40765-8b95-ac6c-7661-da9c1959975e leader=fa269533-67e1-c719-201a-f91ad8d8b0b5 leader-id=server-fa269533-67e1-c719-201a-f91ad8d8b0b5 2025-01-28T14:48:46.260Z [WARN] server-fa269533-67e1-c719-201a-f91ad8d8b0b5: rejecting vote request since we have a leader: from=6bd40765-8b95-ac6c-7661-da9c1959975e leader=fa269533-67e1-c719-201a-f91ad8d8b0b5 leader-id=server-fa269533-67e1-c719-201a-f91ad8d8b0b5 2025-01-28T14:48:46.260Z [INFO] server-fa269533-67e1-c719-201a-f91ad8d8b0b5: pipelining replication: peer="{Voter server-6bd40765-8b95-ac6c-7661-da9c1959975e 6bd40765-8b95-ac6c-7661-da9c1959975e}" 2025-01-28T14:48:46.366Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:46.260980711 +0000 UTC m=+109.344385735 (1 nodes), 63.189876ms from start of poll, 65.426609ms from cluster start. Timeout at 2025-01-28 14:48:46.36605138 +0000 UTC m=+109.449455738, 105.070003ms after stability 2025-01-28T14:48:46.366Z [INFO] server-fa269533-67e1-c719-201a-f91ad8d8b0b5: cannot transfer leadership to itself 2025-01-28T14:48:46.366Z [INFO] server-fa269533-67e1-c719-201a-f91ad8d8b0b5: aborting pipeline replication: peer="{Voter server-6bd40765-8b95-ac6c-7661-da9c1959975e 6bd40765-8b95-ac6c-7661-da9c1959975e}" 2025-01-28T14:48:46.366Z [INFO] server-fa269533-67e1-c719-201a-f91ad8d8b0b5: aborting pipeline replication: peer="{Voter server-98e2625f-f9f3-bb2f-af74-74bb426c2829 98e2625f-f9f3-bb2f-af74-74bb426c2829}" --- PASS: TestRaft_LeadershipTransferToItself (0.18s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests 2025-01-28T14:48:46.376Z [INFO] server-70e7cabe-d4e1-5dbd-abb8-7d0e330afb19: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70e7cabe-d4e1-5dbd-abb8-7d0e330afb19 Address:70e7cabe-d4e1-5dbd-abb8-7d0e330afb19} {Suffrage:Voter ID:server-83dd11f1-9edb-1cdd-a3c4-6ce7566542a0 Address:83dd11f1-9edb-1cdd-a3c4-6ce7566542a0} {Suffrage:Voter ID:server-f06a603f-d552-747c-f85d-f199b4b83304 Address:f06a603f-d552-747c-f85d-f199b4b83304}]" 2025-01-28T14:48:46.376Z [INFO] server-70e7cabe-d4e1-5dbd-abb8-7d0e330afb19: entering follower state: follower="Node at 70e7cabe-d4e1-5dbd-abb8-7d0e330afb19 [Follower]" leader-address= leader-id= 2025-01-28T14:48:46.376Z [INFO] server-83dd11f1-9edb-1cdd-a3c4-6ce7566542a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70e7cabe-d4e1-5dbd-abb8-7d0e330afb19 Address:70e7cabe-d4e1-5dbd-abb8-7d0e330afb19} {Suffrage:Voter ID:server-83dd11f1-9edb-1cdd-a3c4-6ce7566542a0 Address:83dd11f1-9edb-1cdd-a3c4-6ce7566542a0} {Suffrage:Voter ID:server-f06a603f-d552-747c-f85d-f199b4b83304 Address:f06a603f-d552-747c-f85d-f199b4b83304}]" 2025-01-28T14:48:46.377Z [INFO] server-83dd11f1-9edb-1cdd-a3c4-6ce7566542a0: entering follower state: follower="Node at 83dd11f1-9edb-1cdd-a3c4-6ce7566542a0 [Follower]" leader-address= leader-id= 2025-01-28T14:48:46.377Z [INFO] server-f06a603f-d552-747c-f85d-f199b4b83304: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70e7cabe-d4e1-5dbd-abb8-7d0e330afb19 Address:70e7cabe-d4e1-5dbd-abb8-7d0e330afb19} {Suffrage:Voter ID:server-83dd11f1-9edb-1cdd-a3c4-6ce7566542a0 Address:83dd11f1-9edb-1cdd-a3c4-6ce7566542a0} {Suffrage:Voter ID:server-f06a603f-d552-747c-f85d-f199b4b83304 Address:f06a603f-d552-747c-f85d-f199b4b83304}]" 2025-01-28T14:48:46.377Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:46.378Z [INFO] server-f06a603f-d552-747c-f85d-f199b4b83304: entering follower state: follower="Node at f06a603f-d552-747c-f85d-f199b4b83304 [Follower]" leader-address= leader-id= 2025-01-28T14:48:46.431Z [WARN] server-83dd11f1-9edb-1cdd-a3c4-6ce7566542a0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:46.431Z [INFO] server-83dd11f1-9edb-1cdd-a3c4-6ce7566542a0: entering candidate state: node="Node at 83dd11f1-9edb-1cdd-a3c4-6ce7566542a0 [Candidate]" term=2 2025-01-28T14:48:46.431Z [INFO] server-83dd11f1-9edb-1cdd-a3c4-6ce7566542a0: election won: term=2 tally=2 2025-01-28T14:48:46.431Z [INFO] server-83dd11f1-9edb-1cdd-a3c4-6ce7566542a0: entering leader state: leader="Node at 83dd11f1-9edb-1cdd-a3c4-6ce7566542a0 [Leader]" 2025-01-28T14:48:46.432Z [INFO] server-83dd11f1-9edb-1cdd-a3c4-6ce7566542a0: added peer, starting replication: peer=server-70e7cabe-d4e1-5dbd-abb8-7d0e330afb19 2025-01-28T14:48:46.432Z [INFO] server-83dd11f1-9edb-1cdd-a3c4-6ce7566542a0: added peer, starting replication: peer=server-f06a603f-d552-747c-f85d-f199b4b83304 2025-01-28T14:48:46.432Z [INFO] server-83dd11f1-9edb-1cdd-a3c4-6ce7566542a0: pipelining replication: peer="{Voter server-f06a603f-d552-747c-f85d-f199b4b83304 f06a603f-d552-747c-f85d-f199b4b83304}" 2025-01-28T14:48:46.432Z [INFO] server-83dd11f1-9edb-1cdd-a3c4-6ce7566542a0: pipelining replication: peer="{Voter server-70e7cabe-d4e1-5dbd-abb8-7d0e330afb19 70e7cabe-d4e1-5dbd-abb8-7d0e330afb19}" 2025-01-28T14:48:46.537Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:46.432207129 +0000 UTC m=+109.515611820 (1 nodes), 54.374281ms from start of poll, 56.712684ms from cluster start. Timeout at 2025-01-28 14:48:46.537226463 +0000 UTC m=+109.620631154, 105.019334ms after stability raft_test.go:2457: waiting on future 0 raft_test.go:2457: waiting on future 1 raft_test.go:2457: waiting on future 2 raft_test.go:2457: waiting on future 3 raft_test.go:2457: waiting on future 4 raft_test.go:2457: waiting on future 5 raft_test.go:2457: waiting on future 6 2025-01-28T14:48:46.539Z [INFO] server-83dd11f1-9edb-1cdd-a3c4-6ce7566542a0: aborting pipeline replication: peer="{Voter server-f06a603f-d552-747c-f85d-f199b4b83304 f06a603f-d552-747c-f85d-f199b4b83304}" 2025-01-28T14:48:46.539Z [INFO] server-83dd11f1-9edb-1cdd-a3c4-6ce7566542a0: aborting pipeline replication: peer="{Voter server-70e7cabe-d4e1-5dbd-abb8-7d0e330afb19 70e7cabe-d4e1-5dbd-abb8-7d0e330afb19}" --- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.18s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout 2025-01-28T14:48:46.554Z [INFO] server-9137fac4-d9ad-ab45-4d94-29a930dc9895: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9137fac4-d9ad-ab45-4d94-29a930dc9895 Address:9137fac4-d9ad-ab45-4d94-29a930dc9895} {Suffrage:Voter ID:server-47698f02-310c-8f90-8f68-f90971275a51 Address:47698f02-310c-8f90-8f68-f90971275a51} {Suffrage:Voter ID:server-1d3883c9-a248-4cb1-cab0-1b28ab999b37 Address:1d3883c9-a248-4cb1-cab0-1b28ab999b37}]" 2025-01-28T14:48:46.554Z [INFO] server-9137fac4-d9ad-ab45-4d94-29a930dc9895: entering follower state: follower="Node at 9137fac4-d9ad-ab45-4d94-29a930dc9895 [Follower]" leader-address= leader-id= 2025-01-28T14:48:46.554Z [INFO] server-47698f02-310c-8f90-8f68-f90971275a51: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9137fac4-d9ad-ab45-4d94-29a930dc9895 Address:9137fac4-d9ad-ab45-4d94-29a930dc9895} {Suffrage:Voter ID:server-47698f02-310c-8f90-8f68-f90971275a51 Address:47698f02-310c-8f90-8f68-f90971275a51} {Suffrage:Voter ID:server-1d3883c9-a248-4cb1-cab0-1b28ab999b37 Address:1d3883c9-a248-4cb1-cab0-1b28ab999b37}]" 2025-01-28T14:48:46.555Z [INFO] server-47698f02-310c-8f90-8f68-f90971275a51: entering follower state: follower="Node at 47698f02-310c-8f90-8f68-f90971275a51 [Follower]" leader-address= leader-id= 2025-01-28T14:48:46.555Z [INFO] server-1d3883c9-a248-4cb1-cab0-1b28ab999b37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9137fac4-d9ad-ab45-4d94-29a930dc9895 Address:9137fac4-d9ad-ab45-4d94-29a930dc9895} {Suffrage:Voter ID:server-47698f02-310c-8f90-8f68-f90971275a51 Address:47698f02-310c-8f90-8f68-f90971275a51} {Suffrage:Voter ID:server-1d3883c9-a248-4cb1-cab0-1b28ab999b37 Address:1d3883c9-a248-4cb1-cab0-1b28ab999b37}]" 2025-01-28T14:48:46.555Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:46.555Z [INFO] server-1d3883c9-a248-4cb1-cab0-1b28ab999b37: entering follower state: follower="Node at 1d3883c9-a248-4cb1-cab0-1b28ab999b37 [Follower]" leader-address= leader-id= 2025-01-28T14:48:46.611Z [WARN] server-1d3883c9-a248-4cb1-cab0-1b28ab999b37: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:46.612Z [INFO] server-1d3883c9-a248-4cb1-cab0-1b28ab999b37: entering candidate state: node="Node at 1d3883c9-a248-4cb1-cab0-1b28ab999b37 [Candidate]" term=2 2025-01-28T14:48:46.613Z [INFO] server-1d3883c9-a248-4cb1-cab0-1b28ab999b37: election won: term=2 tally=2 2025-01-28T14:48:46.613Z [INFO] server-1d3883c9-a248-4cb1-cab0-1b28ab999b37: entering leader state: leader="Node at 1d3883c9-a248-4cb1-cab0-1b28ab999b37 [Leader]" 2025-01-28T14:48:46.613Z [INFO] server-1d3883c9-a248-4cb1-cab0-1b28ab999b37: added peer, starting replication: peer=server-9137fac4-d9ad-ab45-4d94-29a930dc9895 2025-01-28T14:48:46.613Z [INFO] server-1d3883c9-a248-4cb1-cab0-1b28ab999b37: added peer, starting replication: peer=server-47698f02-310c-8f90-8f68-f90971275a51 2025-01-28T14:48:46.615Z [INFO] server-1d3883c9-a248-4cb1-cab0-1b28ab999b37: pipelining replication: peer="{Voter server-9137fac4-d9ad-ab45-4d94-29a930dc9895 9137fac4-d9ad-ab45-4d94-29a930dc9895}" 2025-01-28T14:48:46.615Z [INFO] server-1d3883c9-a248-4cb1-cab0-1b28ab999b37: pipelining replication: peer="{Voter server-47698f02-310c-8f90-8f68-f90971275a51 47698f02-310c-8f90-8f68-f90971275a51}" 2025-01-28T14:48:46.719Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:46.613781855 +0000 UTC m=+109.697187213 (1 nodes), 58.103726ms from start of poll, 60.379793ms from cluster start. Timeout at 2025-01-28 14:48:46.718807189 +0000 UTC m=+109.802212547, 105.025334ms after stability 2025-01-28T14:48:46.720Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:46.826Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:46.720225683 +0000 UTC m=+109.803630374 (2 nodes), 8.333µs from start of poll, 166.822954ms from cluster start. Timeout at 2025-01-28 14:48:46.825651029 +0000 UTC m=+109.909057720, 105.427346ms after stability 2025-01-28T14:48:51.812Z [INFO] server-1d3883c9-a248-4cb1-cab0-1b28ab999b37: aborting pipeline replication: peer="{Voter server-47698f02-310c-8f90-8f68-f90971275a51 47698f02-310c-8f90-8f68-f90971275a51}" 2025-01-28T14:48:51.812Z [INFO] server-1d3883c9-a248-4cb1-cab0-1b28ab999b37: aborting pipeline replication: peer="{Voter server-9137fac4-d9ad-ab45-4d94-29a930dc9895 9137fac4-d9ad-ab45-4d94-29a930dc9895}" --- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (5.27s) === RUN TestRaft_LeadershipTransferIgnoresNonvoters 2025-01-28T14:48:51.822Z [INFO] server-78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec Address:78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec} {Suffrage:Voter ID:server-3fa0a35f-4abe-899c-b34a-739c0a55fb0c Address:3fa0a35f-4abe-899c-b34a-739c0a55fb0c}]" 2025-01-28T14:48:51.822Z [INFO] server-3fa0a35f-4abe-899c-b34a-739c0a55fb0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec Address:78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec} {Suffrage:Voter ID:server-3fa0a35f-4abe-899c-b34a-739c0a55fb0c Address:3fa0a35f-4abe-899c-b34a-739c0a55fb0c}]" 2025-01-28T14:48:51.822Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:51.822Z [INFO] server-78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec: entering follower state: follower="Node at 78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec [Follower]" leader-address= leader-id= 2025-01-28T14:48:51.823Z [INFO] server-3fa0a35f-4abe-899c-b34a-739c0a55fb0c: entering follower state: follower="Node at 3fa0a35f-4abe-899c-b34a-739c0a55fb0c [Follower]" leader-address= leader-id= 2025-01-28T14:48:51.877Z [WARN] server-78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:51.878Z [INFO] server-78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec: entering candidate state: node="Node at 78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec [Candidate]" term=2 2025-01-28T14:48:51.878Z [INFO] server-78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec: election won: term=2 tally=2 2025-01-28T14:48:51.878Z [INFO] server-78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec: entering leader state: leader="Node at 78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec [Leader]" 2025-01-28T14:48:51.878Z [INFO] server-78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec: added peer, starting replication: peer=server-3fa0a35f-4abe-899c-b34a-739c0a55fb0c 2025-01-28T14:48:51.879Z [INFO] server-78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec: pipelining replication: peer="{Voter server-3fa0a35f-4abe-899c-b34a-739c0a55fb0c 3fa0a35f-4abe-899c-b34a-739c0a55fb0c}" 2025-01-28T14:48:51.984Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:51.878934556 +0000 UTC m=+114.962339581 (1 nodes), 55.964663ms from start of poll, 57.525709ms from cluster start. Timeout at 2025-01-28 14:48:51.98395589 +0000 UTC m=+115.067360581, 105.021ms after stability 2025-01-28T14:48:51.984Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:52.090Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:51.984421689 +0000 UTC m=+115.067826380 (1 nodes), 5.333µs from start of poll, 163.012508ms from cluster start. Timeout at 2025-01-28 14:48:52.090075042 +0000 UTC m=+115.173479733, 105.653353ms after stability 2025-01-28T14:48:52.091Z [INFO] server-78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec: updating configuration: command=DemoteVoter server-id=server-3fa0a35f-4abe-899c-b34a-739c0a55fb0c server-addr= servers="[{Suffrage:Voter ID:server-78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec Address:78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec} {Suffrage:Nonvoter ID:server-3fa0a35f-4abe-899c-b34a-739c0a55fb0c Address:3fa0a35f-4abe-899c-b34a-739c0a55fb0c}]" 2025-01-28T14:48:52.091Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:52.198Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:52.091933882 +0000 UTC m=+115.175338573 (1 nodes), 5µs from start of poll, 270.524701ms from cluster start. Timeout at 2025-01-28 14:48:52.197401896 +0000 UTC m=+115.280807254, 105.468681ms after stability 2025-01-28T14:48:52.198Z [INFO] server-78e2f2ac-9cce-1fa2-3b7d-cdd481acfcec: aborting pipeline replication: peer="{Voter server-3fa0a35f-4abe-899c-b34a-739c0a55fb0c 3fa0a35f-4abe-899c-b34a-739c0a55fb0c}" --- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.38s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap 2025-01-28T14:48:52.206Z [INFO] server-d699ea89-e89c-78b2-290e-53716f271be2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d699ea89-e89c-78b2-290e-53716f271be2 Address:d699ea89-e89c-78b2-290e-53716f271be2} {Suffrage:Voter ID:server-932cea43-eb53-8ead-42e5-80de86d72d1c Address:932cea43-eb53-8ead-42e5-80de86d72d1c}]" 2025-01-28T14:48:52.207Z [INFO] server-d699ea89-e89c-78b2-290e-53716f271be2: entering follower state: follower="Node at d699ea89-e89c-78b2-290e-53716f271be2 [Follower]" leader-address= leader-id= 2025-01-28T14:48:52.207Z [INFO] server-932cea43-eb53-8ead-42e5-80de86d72d1c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d699ea89-e89c-78b2-290e-53716f271be2 Address:d699ea89-e89c-78b2-290e-53716f271be2} {Suffrage:Voter ID:server-932cea43-eb53-8ead-42e5-80de86d72d1c Address:932cea43-eb53-8ead-42e5-80de86d72d1c}]" 2025-01-28T14:48:52.207Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:52.207Z [INFO] server-932cea43-eb53-8ead-42e5-80de86d72d1c: entering follower state: follower="Node at 932cea43-eb53-8ead-42e5-80de86d72d1c [Follower]" leader-address= leader-id= 2025-01-28T14:48:52.268Z [WARN] server-932cea43-eb53-8ead-42e5-80de86d72d1c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:52.269Z [INFO] server-932cea43-eb53-8ead-42e5-80de86d72d1c: entering candidate state: node="Node at 932cea43-eb53-8ead-42e5-80de86d72d1c [Candidate]" term=2 2025-01-28T14:48:52.269Z [INFO] server-932cea43-eb53-8ead-42e5-80de86d72d1c: election won: term=2 tally=2 2025-01-28T14:48:52.269Z [INFO] server-932cea43-eb53-8ead-42e5-80de86d72d1c: entering leader state: leader="Node at 932cea43-eb53-8ead-42e5-80de86d72d1c [Leader]" 2025-01-28T14:48:52.270Z [INFO] server-932cea43-eb53-8ead-42e5-80de86d72d1c: added peer, starting replication: peer=server-d699ea89-e89c-78b2-290e-53716f271be2 2025-01-28T14:48:52.271Z [INFO] server-932cea43-eb53-8ead-42e5-80de86d72d1c: pipelining replication: peer="{Voter server-d699ea89-e89c-78b2-290e-53716f271be2 d699ea89-e89c-78b2-290e-53716f271be2}" 2025-01-28T14:48:52.376Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:52.270692524 +0000 UTC m=+115.354097549 (1 nodes), 62.847533ms from start of poll, 64.516916ms from cluster start. Timeout at 2025-01-28 14:48:52.375727525 +0000 UTC m=+115.459132882, 105.035333ms after stability 2025-01-28T14:48:52.376Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:52.482Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:52.376289994 +0000 UTC m=+115.459694685 (1 nodes), 6.667µs from start of poll, 170.114052ms from cluster start. Timeout at 2025-01-28 14:48:52.481645004 +0000 UTC m=+115.565049695, 105.35501ms after stability 2025-01-28T14:48:52.495Z [INFO] server-932cea43-eb53-8ead-42e5-80de86d72d1c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d699ea89-e89c-78b2-290e-53716f271be2 Address:d699ea89-e89c-78b2-290e-53716f271be2} {Suffrage:Voter ID:server-932cea43-eb53-8ead-42e5-80de86d72d1c Address:932cea43-eb53-8ead-42e5-80de86d72d1c}]" 2025-01-28T14:48:52.496Z [INFO] server-932cea43-eb53-8ead-42e5-80de86d72d1c: aborting pipeline replication: peer="{Voter server-d699ea89-e89c-78b2-290e-53716f271be2 d699ea89-e89c-78b2-290e-53716f271be2}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.30s) === RUN TestRaft_LogStoreIsMonotonic 2025-01-28T14:48:52.502Z [INFO] server-2e4d42ae-cb6e-c1ac-6b30-c58d4fa0f40f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e4d42ae-cb6e-c1ac-6b30-c58d4fa0f40f Address:2e4d42ae-cb6e-c1ac-6b30-c58d4fa0f40f}]" 2025-01-28T14:48:52.503Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:52.503Z [INFO] server-2e4d42ae-cb6e-c1ac-6b30-c58d4fa0f40f: entering follower state: follower="Node at 2e4d42ae-cb6e-c1ac-6b30-c58d4fa0f40f [Follower]" leader-address= leader-id= 2025-01-28T14:48:52.582Z [WARN] server-2e4d42ae-cb6e-c1ac-6b30-c58d4fa0f40f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:52.582Z [INFO] server-2e4d42ae-cb6e-c1ac-6b30-c58d4fa0f40f: entering candidate state: node="Node at 2e4d42ae-cb6e-c1ac-6b30-c58d4fa0f40f [Candidate]" term=2 2025-01-28T14:48:52.582Z [INFO] server-2e4d42ae-cb6e-c1ac-6b30-c58d4fa0f40f: election won: term=2 tally=1 2025-01-28T14:48:52.582Z [INFO] server-2e4d42ae-cb6e-c1ac-6b30-c58d4fa0f40f: entering leader state: leader="Node at 2e4d42ae-cb6e-c1ac-6b30-c58d4fa0f40f [Leader]" 2025-01-28T14:48:52.688Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:52.582989466 +0000 UTC m=+115.666394490 (1 nodes), 79.737701ms from start of poll, 80.723064ms from cluster start. Timeout at 2025-01-28 14:48:52.6880098 +0000 UTC m=+115.771414825, 105.020335ms after stability --- PASS: TestRaft_LogStoreIsMonotonic (0.19s) === RUN TestRaft_CacheLogWithStoreError 2025-01-28T14:48:52.695Z [INFO] server-c1801c2c-c2cc-9592-1f1b-987e27cf26ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1801c2c-c2cc-9592-1f1b-987e27cf26ea Address:c1801c2c-c2cc-9592-1f1b-987e27cf26ea} {Suffrage:Voter ID:server-e5696284-fc7d-53cb-f50e-429cb2693fee Address:e5696284-fc7d-53cb-f50e-429cb2693fee}]" 2025-01-28T14:48:52.695Z [INFO] server-c1801c2c-c2cc-9592-1f1b-987e27cf26ea: entering follower state: follower="Node at c1801c2c-c2cc-9592-1f1b-987e27cf26ea [Follower]" leader-address= leader-id= 2025-01-28T14:48:52.696Z [INFO] server-e5696284-fc7d-53cb-f50e-429cb2693fee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1801c2c-c2cc-9592-1f1b-987e27cf26ea Address:c1801c2c-c2cc-9592-1f1b-987e27cf26ea} {Suffrage:Voter ID:server-e5696284-fc7d-53cb-f50e-429cb2693fee Address:e5696284-fc7d-53cb-f50e-429cb2693fee}]" 2025-01-28T14:48:52.696Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:52.696Z [INFO] server-e5696284-fc7d-53cb-f50e-429cb2693fee: entering follower state: follower="Node at e5696284-fc7d-53cb-f50e-429cb2693fee [Follower]" leader-address= leader-id= 2025-01-28T14:48:52.750Z [WARN] server-c1801c2c-c2cc-9592-1f1b-987e27cf26ea: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:52.750Z [INFO] server-c1801c2c-c2cc-9592-1f1b-987e27cf26ea: entering candidate state: node="Node at c1801c2c-c2cc-9592-1f1b-987e27cf26ea [Candidate]" term=2 2025-01-28T14:48:52.750Z [INFO] server-c1801c2c-c2cc-9592-1f1b-987e27cf26ea: election won: term=2 tally=2 2025-01-28T14:48:52.751Z [INFO] server-c1801c2c-c2cc-9592-1f1b-987e27cf26ea: entering leader state: leader="Node at c1801c2c-c2cc-9592-1f1b-987e27cf26ea [Leader]" 2025-01-28T14:48:52.751Z [INFO] server-c1801c2c-c2cc-9592-1f1b-987e27cf26ea: added peer, starting replication: peer=server-e5696284-fc7d-53cb-f50e-429cb2693fee 2025-01-28T14:48:52.752Z [INFO] server-c1801c2c-c2cc-9592-1f1b-987e27cf26ea: pipelining replication: peer="{Voter server-e5696284-fc7d-53cb-f50e-429cb2693fee e5696284-fc7d-53cb-f50e-429cb2693fee}" 2025-01-28T14:48:52.859Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:52.751090125 +0000 UTC m=+115.834495149 (1 nodes), 54.779293ms from start of poll, 56.184002ms from cluster start. Timeout at 2025-01-28 14:48:52.856102125 +0000 UTC m=+115.939506816, 105.011667ms after stability 2025-01-28T14:48:52.859Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:52.965Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:52.859643682 +0000 UTC m=+115.943048707 (1 nodes), 6.334µs from start of poll, 164.73756ms from cluster start. Timeout at 2025-01-28 14:48:52.96502036 +0000 UTC m=+116.048425051, 105.376344ms after stability 2025-01-28T14:48:52.970Z [ERROR] server-e5696284-fc7d-53cb-f50e-429cb2693fee: failed to append to logs: error="unable to store logs within log store, err: "some error"" 2025-01-28T14:48:52.970Z [WARN] server-e5696284-fc7d-53cb-f50e-429cb2693fee: failed to get previous log: previous-index=9 last-index=7 error="log not found" 2025-01-28T14:48:52.971Z [INFO] server-c1801c2c-c2cc-9592-1f1b-987e27cf26ea: aborting pipeline replication: peer="{Voter server-e5696284-fc7d-53cb-f50e-429cb2693fee e5696284-fc7d-53cb-f50e-429cb2693fee}" 2025-01-28T14:48:52.979Z [INFO] server-c1801c2c-c2cc-9592-1f1b-987e27cf26ea: pipelining replication: peer="{Voter server-e5696284-fc7d-53cb-f50e-429cb2693fee e5696284-fc7d-53cb-f50e-429cb2693fee}" 2025-01-28T14:48:52.980Z [INFO] server-e5696284-fc7d-53cb-f50e-429cb2693fee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1801c2c-c2cc-9592-1f1b-987e27cf26ea Address:c1801c2c-c2cc-9592-1f1b-987e27cf26ea} {Suffrage:Voter ID:server-e5696284-fc7d-53cb-f50e-429cb2693fee Address:e5696284-fc7d-53cb-f50e-429cb2693fee}]" 2025-01-28T14:48:52.980Z [INFO] server-c1801c2c-c2cc-9592-1f1b-987e27cf26ea: aborting pipeline replication: peer="{Voter server-e5696284-fc7d-53cb-f50e-429cb2693fee e5696284-fc7d-53cb-f50e-429cb2693fee}" --- PASS: TestRaft_CacheLogWithStoreError (0.29s) === RUN TestRaft_ReloadConfig 2025-01-28T14:48:52.986Z [INFO] server-928c4222-f6ba-02e3-7020-b64180a6433b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-928c4222-f6ba-02e3-7020-b64180a6433b Address:928c4222-f6ba-02e3-7020-b64180a6433b}]" --- PASS: TestRaft_ReloadConfig (0.00s) === RUN TestRaft_ReloadConfigValidates 2025-01-28T14:48:52.990Z [INFO] server-114adba3-5315-b185-90e8-8ebf73c56e6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-114adba3-5315-b185-90e8-8ebf73c56e6a Address:114adba3-5315-b185-90e8-8ebf73c56e6a}]" 2025-01-28T14:48:52.991Z [INFO] server-114adba3-5315-b185-90e8-8ebf73c56e6a: entering follower state: follower="Node at 114adba3-5315-b185-90e8-8ebf73c56e6a [Follower]" leader-address= leader-id= --- PASS: TestRaft_ReloadConfigValidates (0.00s) === RUN TestRaft_InstallSnapshot_InvalidPeers 2025-01-28T14:48:52.993Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)" --- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s) === RUN TestRaft_VoteNotGranted_WhenNodeNotInCluster 2025-01-28T14:48:52.998Z [INFO] server-88b88133-5891-83ce-40d5-3d3fe98b0de4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88b88133-5891-83ce-40d5-3d3fe98b0de4 Address:88b88133-5891-83ce-40d5-3d3fe98b0de4} {Suffrage:Voter ID:server-5c6b8d04-d082-09e9-06b2-3b08436b6107 Address:5c6b8d04-d082-09e9-06b2-3b08436b6107} {Suffrage:Voter ID:server-c192c0cc-d216-cd75-6587-0b16d94694e7 Address:c192c0cc-d216-cd75-6587-0b16d94694e7}]" 2025-01-28T14:48:52.999Z [INFO] server-88b88133-5891-83ce-40d5-3d3fe98b0de4: entering follower state: follower="Node at 88b88133-5891-83ce-40d5-3d3fe98b0de4 [Follower]" leader-address= leader-id= 2025-01-28T14:48:52.999Z [INFO] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88b88133-5891-83ce-40d5-3d3fe98b0de4 Address:88b88133-5891-83ce-40d5-3d3fe98b0de4} {Suffrage:Voter ID:server-5c6b8d04-d082-09e9-06b2-3b08436b6107 Address:5c6b8d04-d082-09e9-06b2-3b08436b6107} {Suffrage:Voter ID:server-c192c0cc-d216-cd75-6587-0b16d94694e7 Address:c192c0cc-d216-cd75-6587-0b16d94694e7}]" 2025-01-28T14:48:52.999Z [INFO] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: entering follower state: follower="Node at 5c6b8d04-d082-09e9-06b2-3b08436b6107 [Follower]" leader-address= leader-id= 2025-01-28T14:48:53.000Z [INFO] server-c192c0cc-d216-cd75-6587-0b16d94694e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88b88133-5891-83ce-40d5-3d3fe98b0de4 Address:88b88133-5891-83ce-40d5-3d3fe98b0de4} {Suffrage:Voter ID:server-5c6b8d04-d082-09e9-06b2-3b08436b6107 Address:5c6b8d04-d082-09e9-06b2-3b08436b6107} {Suffrage:Voter ID:server-c192c0cc-d216-cd75-6587-0b16d94694e7 Address:c192c0cc-d216-cd75-6587-0b16d94694e7}]" 2025-01-28T14:48:53.000Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:53.000Z [INFO] server-c192c0cc-d216-cd75-6587-0b16d94694e7: entering follower state: follower="Node at c192c0cc-d216-cd75-6587-0b16d94694e7 [Follower]" leader-address= leader-id= 2025-01-28T14:48:53.061Z [WARN] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:53.061Z [INFO] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: entering candidate state: node="Node at 5c6b8d04-d082-09e9-06b2-3b08436b6107 [Candidate]" term=2 2025-01-28T14:48:53.062Z [INFO] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: election won: term=2 tally=2 2025-01-28T14:48:53.062Z [INFO] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: entering leader state: leader="Node at 5c6b8d04-d082-09e9-06b2-3b08436b6107 [Leader]" 2025-01-28T14:48:53.062Z [INFO] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: added peer, starting replication: peer=server-88b88133-5891-83ce-40d5-3d3fe98b0de4 2025-01-28T14:48:53.062Z [INFO] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: added peer, starting replication: peer=server-c192c0cc-d216-cd75-6587-0b16d94694e7 2025-01-28T14:48:53.063Z [INFO] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: pipelining replication: peer="{Voter server-88b88133-5891-83ce-40d5-3d3fe98b0de4 88b88133-5891-83ce-40d5-3d3fe98b0de4}" 2025-01-28T14:48:53.063Z [INFO] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: pipelining replication: peer="{Voter server-c192c0cc-d216-cd75-6587-0b16d94694e7 c192c0cc-d216-cd75-6587-0b16d94694e7}" 2025-01-28T14:48:53.167Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:53.06261171 +0000 UTC m=+116.146016734 (1 nodes), 62.268182ms from start of poll, 64.514249ms from cluster start. Timeout at 2025-01-28 14:48:53.167675045 +0000 UTC m=+116.251080070, 105.063336ms after stability 2025-01-28T14:48:53.173Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:53.279Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:53.173319665 +0000 UTC m=+116.256724689 (2 nodes), 9µs from start of poll, 175.222204ms from cluster start. Timeout at 2025-01-28 14:48:53.27873601 +0000 UTC m=+116.362141035, 105.416346ms after stability 2025-01-28T14:48:53.279Z [INFO] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: updating configuration: command=RemoveServer server-id=server-88b88133-5891-83ce-40d5-3d3fe98b0de4 server-addr= servers="[{Suffrage:Voter ID:server-5c6b8d04-d082-09e9-06b2-3b08436b6107 Address:5c6b8d04-d082-09e9-06b2-3b08436b6107} {Suffrage:Voter ID:server-c192c0cc-d216-cd75-6587-0b16d94694e7 Address:c192c0cc-d216-cd75-6587-0b16d94694e7}]" 2025-01-28T14:48:53.279Z [INFO] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: removed peer, stopping replication: peer=server-88b88133-5891-83ce-40d5-3d3fe98b0de4 last-index=3 2025-01-28T14:48:53.280Z [INFO] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: aborting pipeline replication: peer="{Voter server-88b88133-5891-83ce-40d5-3d3fe98b0de4 88b88133-5891-83ce-40d5-3d3fe98b0de4}" 2025-01-28T14:48:53.365Z [WARN] server-88b88133-5891-83ce-40d5-3d3fe98b0de4: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2025-01-28T14:48:53.387Z [ERROR] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: failed to pipeline appendEntries: peer="{Voter server-c192c0cc-d216-cd75-6587-0b16d94694e7 c192c0cc-d216-cd75-6587-0b16d94694e7}" error="append pipeline closed" 2025-01-28T14:48:53.387Z [INFO] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: aborting pipeline replication: peer="{Voter server-c192c0cc-d216-cd75-6587-0b16d94694e7 c192c0cc-d216-cd75-6587-0b16d94694e7}" 2025-01-28T14:48:53.389Z [ERROR] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: failed to heartbeat to: peer=c192c0cc-d216-cd75-6587-0b16d94694e7 backoff time=10ms error="failed to connect to peer: c192c0cc-d216-cd75-6587-0b16d94694e7" 2025-01-28T14:48:53.393Z [ERROR] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: failed to appendEntries to: peer="{Voter server-c192c0cc-d216-cd75-6587-0b16d94694e7 c192c0cc-d216-cd75-6587-0b16d94694e7}" error="failed to connect to peer: c192c0cc-d216-cd75-6587-0b16d94694e7" 2025-01-28T14:48:53.406Z [ERROR] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: failed to heartbeat to: peer=c192c0cc-d216-cd75-6587-0b16d94694e7 backoff time=10ms error="failed to connect to peer: c192c0cc-d216-cd75-6587-0b16d94694e7" 2025-01-28T14:48:53.413Z [ERROR] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: failed to appendEntries to: peer="{Voter server-c192c0cc-d216-cd75-6587-0b16d94694e7 c192c0cc-d216-cd75-6587-0b16d94694e7}" error="failed to connect to peer: c192c0cc-d216-cd75-6587-0b16d94694e7" 2025-01-28T14:48:53.426Z [ERROR] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: failed to heartbeat to: peer=c192c0cc-d216-cd75-6587-0b16d94694e7 backoff time=10ms error="failed to connect to peer: c192c0cc-d216-cd75-6587-0b16d94694e7" 2025-01-28T14:48:53.430Z [WARN] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: failed to contact: server-id=server-c192c0cc-d216-cd75-6587-0b16d94694e7 time=50.196157ms 2025-01-28T14:48:53.430Z [WARN] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: failed to contact quorum of nodes, stepping down 2025-01-28T14:48:53.430Z [INFO] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: entering follower state: follower="Node at 5c6b8d04-d082-09e9-06b2-3b08436b6107 [Follower]" leader-address= leader-id= 2025-01-28T14:48:53.433Z [ERROR] server-5c6b8d04-d082-09e9-06b2-3b08436b6107: failed to appendEntries to: peer="{Voter server-c192c0cc-d216-cd75-6587-0b16d94694e7 c192c0cc-d216-cd75-6587-0b16d94694e7}" error="failed to connect to peer: c192c0cc-d216-cd75-6587-0b16d94694e7" 2025-01-28T14:48:53.482Z [WARN] server-c192c0cc-d216-cd75-6587-0b16d94694e7: heartbeat timeout reached, starting election: last-leader-addr=5c6b8d04-d082-09e9-06b2-3b08436b6107 last-leader-id=server-5c6b8d04-d082-09e9-06b2-3b08436b6107 2025-01-28T14:48:53.483Z [INFO] server-c192c0cc-d216-cd75-6587-0b16d94694e7: entering candidate state: node="Node at c192c0cc-d216-cd75-6587-0b16d94694e7 [Candidate]" term=3 2025-01-28T14:48:53.483Z [ERROR] server-c192c0cc-d216-cd75-6587-0b16d94694e7: failed to make requestVote RPC: target="{Voter server-5c6b8d04-d082-09e9-06b2-3b08436b6107 5c6b8d04-d082-09e9-06b2-3b08436b6107}" error="failed to connect to peer: 5c6b8d04-d082-09e9-06b2-3b08436b6107" term=3 2025-01-28T14:48:53.491Z [WARN] server-c192c0cc-d216-cd75-6587-0b16d94694e7: rejecting vote request since node is not in configuration: from=88b88133-5891-83ce-40d5-3d3fe98b0de4 --- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.50s) === RUN TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin 2025-01-28T14:48:53.503Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 Address:6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0} {Suffrage:Voter ID:server-a7075132-ba8a-d443-a1f5-6963bc4ad7af Address:a7075132-ba8a-d443-a1f5-6963bc4ad7af} {Suffrage:Voter ID:server-329b4e6c-bc90-2795-bd62-ace23e50051f Address:329b4e6c-bc90-2795-bd62-ace23e50051f}]" 2025-01-28T14:48:53.504Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: entering follower state: follower="Node at 6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 [Follower]" leader-address= leader-id= 2025-01-28T14:48:53.505Z [INFO] server-a7075132-ba8a-d443-a1f5-6963bc4ad7af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 Address:6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0} {Suffrage:Voter ID:server-a7075132-ba8a-d443-a1f5-6963bc4ad7af Address:a7075132-ba8a-d443-a1f5-6963bc4ad7af} {Suffrage:Voter ID:server-329b4e6c-bc90-2795-bd62-ace23e50051f Address:329b4e6c-bc90-2795-bd62-ace23e50051f}]" 2025-01-28T14:48:53.505Z [INFO] server-a7075132-ba8a-d443-a1f5-6963bc4ad7af: entering follower state: follower="Node at a7075132-ba8a-d443-a1f5-6963bc4ad7af [Follower]" leader-address= leader-id= 2025-01-28T14:48:53.505Z [INFO] server-329b4e6c-bc90-2795-bd62-ace23e50051f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 Address:6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0} {Suffrage:Voter ID:server-a7075132-ba8a-d443-a1f5-6963bc4ad7af Address:a7075132-ba8a-d443-a1f5-6963bc4ad7af} {Suffrage:Voter ID:server-329b4e6c-bc90-2795-bd62-ace23e50051f Address:329b4e6c-bc90-2795-bd62-ace23e50051f}]" 2025-01-28T14:48:53.506Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:53.506Z [INFO] server-329b4e6c-bc90-2795-bd62-ace23e50051f: entering follower state: follower="Node at 329b4e6c-bc90-2795-bd62-ace23e50051f [Follower]" leader-address= leader-id= 2025-01-28T14:48:53.561Z [WARN] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:53.561Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: entering candidate state: node="Node at 6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 [Candidate]" term=2 2025-01-28T14:48:53.562Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: election won: term=2 tally=2 2025-01-28T14:48:53.562Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: entering leader state: leader="Node at 6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 [Leader]" 2025-01-28T14:48:53.562Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: added peer, starting replication: peer=server-a7075132-ba8a-d443-a1f5-6963bc4ad7af 2025-01-28T14:48:53.563Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: added peer, starting replication: peer=server-329b4e6c-bc90-2795-bd62-ace23e50051f 2025-01-28T14:48:53.563Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: pipelining replication: peer="{Voter server-a7075132-ba8a-d443-a1f5-6963bc4ad7af a7075132-ba8a-d443-a1f5-6963bc4ad7af}" 2025-01-28T14:48:53.564Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: pipelining replication: peer="{Voter server-329b4e6c-bc90-2795-bd62-ace23e50051f 329b4e6c-bc90-2795-bd62-ace23e50051f}" 2025-01-28T14:48:53.636Z [WARN] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: failed to contact: server-id=server-a7075132-ba8a-d443-a1f5-6963bc4ad7af time=56.097333ms 2025-01-28T14:48:53.668Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:53.563345581 +0000 UTC m=+116.646750606 (1 nodes), 57.170031ms from start of poll, 60.074784ms from cluster start. Timeout at 2025-01-28 14:48:53.668362582 +0000 UTC m=+116.751767273, 105.016667ms after stability 2025-01-28T14:48:53.673Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:53.780Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:53.6739652 +0000 UTC m=+116.757370224 (2 nodes), 7.333µs from start of poll, 170.694402ms from cluster start. Timeout at 2025-01-28 14:48:53.779468881 +0000 UTC m=+116.862873906, 105.503682ms after stability 2025-01-28T14:48:53.780Z [ERROR] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: failed to pipeline appendEntries: peer="{Voter server-a7075132-ba8a-d443-a1f5-6963bc4ad7af a7075132-ba8a-d443-a1f5-6963bc4ad7af}" error="append pipeline closed" 2025-01-28T14:48:53.780Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: aborting pipeline replication: peer="{Voter server-a7075132-ba8a-d443-a1f5-6963bc4ad7af a7075132-ba8a-d443-a1f5-6963bc4ad7af}" 2025-01-28T14:48:53.782Z [ERROR] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: failed to heartbeat to: peer=a7075132-ba8a-d443-a1f5-6963bc4ad7af backoff time=10ms error="failed to connect to peer: a7075132-ba8a-d443-a1f5-6963bc4ad7af" 2025-01-28T14:48:53.788Z [ERROR] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: failed to appendEntries to: peer="{Voter server-a7075132-ba8a-d443-a1f5-6963bc4ad7af a7075132-ba8a-d443-a1f5-6963bc4ad7af}" error="failed to connect to peer: a7075132-ba8a-d443-a1f5-6963bc4ad7af" 2025-01-28T14:48:53.800Z [ERROR] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: failed to heartbeat to: peer=a7075132-ba8a-d443-a1f5-6963bc4ad7af backoff time=10ms error="failed to connect to peer: a7075132-ba8a-d443-a1f5-6963bc4ad7af" 2025-01-28T14:48:53.807Z [ERROR] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: failed to appendEntries to: peer="{Voter server-a7075132-ba8a-d443-a1f5-6963bc4ad7af a7075132-ba8a-d443-a1f5-6963bc4ad7af}" error="failed to connect to peer: a7075132-ba8a-d443-a1f5-6963bc4ad7af" 2025-01-28T14:48:53.818Z [ERROR] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: failed to heartbeat to: peer=a7075132-ba8a-d443-a1f5-6963bc4ad7af backoff time=10ms error="failed to connect to peer: a7075132-ba8a-d443-a1f5-6963bc4ad7af" 2025-01-28T14:48:53.824Z [ERROR] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: failed to appendEntries to: peer="{Voter server-a7075132-ba8a-d443-a1f5-6963bc4ad7af a7075132-ba8a-d443-a1f5-6963bc4ad7af}" error="failed to connect to peer: a7075132-ba8a-d443-a1f5-6963bc4ad7af" 2025-01-28T14:48:53.831Z [WARN] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: failed to contact: server-id=server-a7075132-ba8a-d443-a1f5-6963bc4ad7af time=57.107696ms 2025-01-28T14:48:53.840Z [ERROR] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: failed to heartbeat to: peer=a7075132-ba8a-d443-a1f5-6963bc4ad7af backoff time=20ms error="failed to connect to peer: a7075132-ba8a-d443-a1f5-6963bc4ad7af" 2025-01-28T14:48:53.849Z [WARN] server-a7075132-ba8a-d443-a1f5-6963bc4ad7af: heartbeat timeout reached, starting election: last-leader-addr=6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 last-leader-id=server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 2025-01-28T14:48:53.849Z [INFO] server-a7075132-ba8a-d443-a1f5-6963bc4ad7af: entering candidate state: node="Node at a7075132-ba8a-d443-a1f5-6963bc4ad7af [Candidate]" term=3 2025-01-28T14:48:53.849Z [ERROR] server-a7075132-ba8a-d443-a1f5-6963bc4ad7af: failed to make requestVote RPC: target="{Voter server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0}" error="failed to connect to peer: 6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0" term=3 2025-01-28T14:48:53.849Z [ERROR] server-a7075132-ba8a-d443-a1f5-6963bc4ad7af: failed to make requestVote RPC: target="{Voter server-329b4e6c-bc90-2795-bd62-ace23e50051f 329b4e6c-bc90-2795-bd62-ace23e50051f}" error="failed to connect to peer: 329b4e6c-bc90-2795-bd62-ace23e50051f" term=3 2025-01-28T14:48:53.851Z [ERROR] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: failed to appendEntries to: peer="{Voter server-a7075132-ba8a-d443-a1f5-6963bc4ad7af a7075132-ba8a-d443-a1f5-6963bc4ad7af}" error="failed to connect to peer: a7075132-ba8a-d443-a1f5-6963bc4ad7af" 2025-01-28T14:48:53.869Z [ERROR] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: failed to heartbeat to: peer=a7075132-ba8a-d443-a1f5-6963bc4ad7af backoff time=25ms error="failed to connect to peer: a7075132-ba8a-d443-a1f5-6963bc4ad7af" 2025-01-28T14:48:53.880Z [WARN] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: failed to contact: server-id=server-a7075132-ba8a-d443-a1f5-6963bc4ad7af time=106.352825ms 2025-01-28T14:48:53.885Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: updating configuration: command=RemoveServer server-id=server-a7075132-ba8a-d443-a1f5-6963bc4ad7af server-addr= servers="[{Suffrage:Voter ID:server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 Address:6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0} {Suffrage:Voter ID:server-329b4e6c-bc90-2795-bd62-ace23e50051f Address:329b4e6c-bc90-2795-bd62-ace23e50051f}]" 2025-01-28T14:48:53.886Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: removed peer, stopping replication: peer=server-a7075132-ba8a-d443-a1f5-6963bc4ad7af last-index=3 2025-01-28T14:48:53.887Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: updating configuration: command=AddNonvoter server-id=server-a7075132-ba8a-d443-a1f5-6963bc4ad7af server-addr=a7075132-ba8a-d443-a1f5-6963bc4ad7af servers="[{Suffrage:Voter ID:server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 Address:6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0} {Suffrage:Voter ID:server-329b4e6c-bc90-2795-bd62-ace23e50051f Address:329b4e6c-bc90-2795-bd62-ace23e50051f} {Suffrage:Nonvoter ID:server-a7075132-ba8a-d443-a1f5-6963bc4ad7af Address:a7075132-ba8a-d443-a1f5-6963bc4ad7af}]" 2025-01-28T14:48:53.888Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: added peer, starting replication: peer=server-a7075132-ba8a-d443-a1f5-6963bc4ad7af 2025-01-28T14:48:53.888Z [ERROR] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: failed to appendEntries to: peer="{Nonvoter server-a7075132-ba8a-d443-a1f5-6963bc4ad7af a7075132-ba8a-d443-a1f5-6963bc4ad7af}" error="failed to connect to peer: a7075132-ba8a-d443-a1f5-6963bc4ad7af" 2025-01-28T14:48:53.900Z [ERROR] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: peer has newer term, stopping replication: peer="{Voter server-a7075132-ba8a-d443-a1f5-6963bc4ad7af a7075132-ba8a-d443-a1f5-6963bc4ad7af}" 2025-01-28T14:48:53.900Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: entering follower state: follower="Node at 6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 [Follower]" leader-address= leader-id= 2025-01-28T14:48:53.900Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: aborting pipeline replication: peer="{Voter server-329b4e6c-bc90-2795-bd62-ace23e50051f 329b4e6c-bc90-2795-bd62-ace23e50051f}" 2025-01-28T14:48:53.906Z [ERROR] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: peer has newer term, stopping replication: peer="{Nonvoter server-a7075132-ba8a-d443-a1f5-6963bc4ad7af a7075132-ba8a-d443-a1f5-6963bc4ad7af}" 2025-01-28T14:48:53.923Z [WARN] server-a7075132-ba8a-d443-a1f5-6963bc4ad7af: Election timeout reached, restarting election 2025-01-28T14:48:53.923Z [INFO] server-a7075132-ba8a-d443-a1f5-6963bc4ad7af: entering candidate state: node="Node at a7075132-ba8a-d443-a1f5-6963bc4ad7af [Candidate]" term=23 2025-01-28T14:48:53.923Z [WARN] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: rejecting vote request since node is not a voter: from=a7075132-ba8a-d443-a1f5-6963bc4ad7af 2025-01-28T14:48:53.924Z [WARN] server-329b4e6c-bc90-2795-bd62-ace23e50051f: rejecting vote request since we have a leader: from=a7075132-ba8a-d443-a1f5-6963bc4ad7af leader=6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 leader-id=server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 2025-01-28T14:48:53.973Z [WARN] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:53.973Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: entering candidate state: node="Node at 6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 [Candidate]" term=24 2025-01-28T14:48:53.974Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: election won: term=24 tally=2 2025-01-28T14:48:53.974Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: entering leader state: leader="Node at 6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 [Leader]" 2025-01-28T14:48:53.974Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: added peer, starting replication: peer=server-329b4e6c-bc90-2795-bd62-ace23e50051f 2025-01-28T14:48:53.974Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: added peer, starting replication: peer=server-a7075132-ba8a-d443-a1f5-6963bc4ad7af 2025-01-28T14:48:53.975Z [WARN] server-a7075132-ba8a-d443-a1f5-6963bc4ad7af: failed to get previous log: previous-index=4 last-index=2 error="log not found" 2025-01-28T14:48:53.975Z [INFO] server-a7075132-ba8a-d443-a1f5-6963bc4ad7af: entering follower state: follower="Node at a7075132-ba8a-d443-a1f5-6963bc4ad7af [Follower]" leader-address=6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 leader-id=server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 2025-01-28T14:48:53.975Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: pipelining replication: peer="{Voter server-329b4e6c-bc90-2795-bd62-ace23e50051f 329b4e6c-bc90-2795-bd62-ace23e50051f}" 2025-01-28T14:48:53.975Z [WARN] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: appendEntries rejected, sending older logs: peer="{Nonvoter server-a7075132-ba8a-d443-a1f5-6963bc4ad7af a7075132-ba8a-d443-a1f5-6963bc4ad7af}" next=3 2025-01-28T14:48:53.976Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: pipelining replication: peer="{Nonvoter server-a7075132-ba8a-d443-a1f5-6963bc4ad7af a7075132-ba8a-d443-a1f5-6963bc4ad7af}" 2025-01-28T14:48:54.034Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:54.144Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:54.037902009 +0000 UTC m=+117.121307366 (1 nodes), 3.61544ms from start of poll, 534.631544ms from cluster start. Timeout at 2025-01-28 14:48:54.142919675 +0000 UTC m=+117.226324700, 105.017334ms after stability 2025-01-28T14:48:54.268Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: updating configuration: command=RemoveServer server-id=server-a7075132-ba8a-d443-a1f5-6963bc4ad7af server-addr= servers="[{Suffrage:Voter ID:server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 Address:6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0} {Suffrage:Voter ID:server-329b4e6c-bc90-2795-bd62-ace23e50051f Address:329b4e6c-bc90-2795-bd62-ace23e50051f}]" 2025-01-28T14:48:54.269Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: removed peer, stopping replication: peer=server-a7075132-ba8a-d443-a1f5-6963bc4ad7af last-index=106 2025-01-28T14:48:54.269Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: aborting pipeline replication: peer="{Nonvoter server-a7075132-ba8a-d443-a1f5-6963bc4ad7af a7075132-ba8a-d443-a1f5-6963bc4ad7af}" 2025-01-28T14:48:54.270Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: updating configuration: command=AddVoter server-id=server-a7075132-ba8a-d443-a1f5-6963bc4ad7af server-addr=a7075132-ba8a-d443-a1f5-6963bc4ad7af servers="[{Suffrage:Voter ID:server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0 Address:6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0} {Suffrage:Voter ID:server-329b4e6c-bc90-2795-bd62-ace23e50051f Address:329b4e6c-bc90-2795-bd62-ace23e50051f} {Suffrage:Voter ID:server-a7075132-ba8a-d443-a1f5-6963bc4ad7af Address:a7075132-ba8a-d443-a1f5-6963bc4ad7af}]" 2025-01-28T14:48:54.270Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: added peer, starting replication: peer=server-a7075132-ba8a-d443-a1f5-6963bc4ad7af 2025-01-28T14:48:54.270Z [WARN] server-a7075132-ba8a-d443-a1f5-6963bc4ad7af: failed to get previous log: previous-index=107 last-index=106 error="log not found" 2025-01-28T14:48:54.271Z [WARN] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: appendEntries rejected, sending older logs: peer="{Voter server-a7075132-ba8a-d443-a1f5-6963bc4ad7af a7075132-ba8a-d443-a1f5-6963bc4ad7af}" next=107 2025-01-28T14:48:54.271Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: pipelining replication: peer="{Voter server-a7075132-ba8a-d443-a1f5-6963bc4ad7af a7075132-ba8a-d443-a1f5-6963bc4ad7af}" 2025-01-28T14:48:55.480Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:55.586Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:55.481142205 +0000 UTC m=+118.564546896 (1 nodes), 6µs from start of poll, 1.977871074s from cluster start. Timeout at 2025-01-28 14:48:55.586518216 +0000 UTC m=+118.669923240, 105.376344ms after stability 2025-01-28T14:48:55.587Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: aborting pipeline replication: peer="{Voter server-a7075132-ba8a-d443-a1f5-6963bc4ad7af a7075132-ba8a-d443-a1f5-6963bc4ad7af}" 2025-01-28T14:48:55.587Z [INFO] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: aborting pipeline replication: peer="{Voter server-329b4e6c-bc90-2795-bd62-ace23e50051f 329b4e6c-bc90-2795-bd62-ace23e50051f}" 2025-01-28T14:48:56.087Z [ERROR] server-6544fdc7-bb64-0c48-3ee6-dcb05fc5dca0: failed to heartbeat to: peer=329b4e6c-bc90-2795-bd62-ace23e50051f backoff time=10ms error="command timed out" --- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (2.60s) === RUN TestRaft_FollowerRemovalNoElection 2025-01-28T14:48:56.099Z [INFO] server-1422e03a-af23-f87e-e595-2d8e1c9f2b0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1422e03a-af23-f87e-e595-2d8e1c9f2b0f Address:1422e03a-af23-f87e-e595-2d8e1c9f2b0f} {Suffrage:Voter ID:server-7c127b04-fbe8-a3b1-b64d-38f4c7d74e30 Address:7c127b04-fbe8-a3b1-b64d-38f4c7d74e30} {Suffrage:Voter ID:server-c7c160c6-db20-0af7-e998-5c658a7b2bef Address:c7c160c6-db20-0af7-e998-5c658a7b2bef}]" 2025-01-28T14:48:56.099Z [INFO] server-1422e03a-af23-f87e-e595-2d8e1c9f2b0f: entering follower state: follower="Node at 1422e03a-af23-f87e-e595-2d8e1c9f2b0f [Follower]" leader-address= leader-id= 2025-01-28T14:48:56.100Z [INFO] server-7c127b04-fbe8-a3b1-b64d-38f4c7d74e30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1422e03a-af23-f87e-e595-2d8e1c9f2b0f Address:1422e03a-af23-f87e-e595-2d8e1c9f2b0f} {Suffrage:Voter ID:server-7c127b04-fbe8-a3b1-b64d-38f4c7d74e30 Address:7c127b04-fbe8-a3b1-b64d-38f4c7d74e30} {Suffrage:Voter ID:server-c7c160c6-db20-0af7-e998-5c658a7b2bef Address:c7c160c6-db20-0af7-e998-5c658a7b2bef}]" 2025-01-28T14:48:56.100Z [INFO] server-7c127b04-fbe8-a3b1-b64d-38f4c7d74e30: entering follower state: follower="Node at 7c127b04-fbe8-a3b1-b64d-38f4c7d74e30 [Follower]" leader-address= leader-id= 2025-01-28T14:48:56.100Z [INFO] server-c7c160c6-db20-0af7-e998-5c658a7b2bef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1422e03a-af23-f87e-e595-2d8e1c9f2b0f Address:1422e03a-af23-f87e-e595-2d8e1c9f2b0f} {Suffrage:Voter ID:server-7c127b04-fbe8-a3b1-b64d-38f4c7d74e30 Address:7c127b04-fbe8-a3b1-b64d-38f4c7d74e30} {Suffrage:Voter ID:server-c7c160c6-db20-0af7-e998-5c658a7b2bef Address:c7c160c6-db20-0af7-e998-5c658a7b2bef}]" 2025-01-28T14:48:56.100Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:56.100Z [INFO] server-c7c160c6-db20-0af7-e998-5c658a7b2bef: entering follower state: follower="Node at c7c160c6-db20-0af7-e998-5c658a7b2bef [Follower]" leader-address= leader-id= 2025-01-28T14:48:56.243Z [WARN] server-c7c160c6-db20-0af7-e998-5c658a7b2bef: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:56.243Z [INFO] server-c7c160c6-db20-0af7-e998-5c658a7b2bef: entering candidate state: node="Node at c7c160c6-db20-0af7-e998-5c658a7b2bef [Candidate]" term=2 2025-01-28T14:48:56.244Z [INFO] server-c7c160c6-db20-0af7-e998-5c658a7b2bef: election won: term=2 tally=2 2025-01-28T14:48:56.245Z [INFO] server-c7c160c6-db20-0af7-e998-5c658a7b2bef: entering leader state: leader="Node at c7c160c6-db20-0af7-e998-5c658a7b2bef [Leader]" 2025-01-28T14:48:56.245Z [INFO] server-c7c160c6-db20-0af7-e998-5c658a7b2bef: added peer, starting replication: peer=server-1422e03a-af23-f87e-e595-2d8e1c9f2b0f 2025-01-28T14:48:56.245Z [INFO] server-c7c160c6-db20-0af7-e998-5c658a7b2bef: added peer, starting replication: peer=server-7c127b04-fbe8-a3b1-b64d-38f4c7d74e30 2025-01-28T14:48:56.246Z [INFO] server-c7c160c6-db20-0af7-e998-5c658a7b2bef: pipelining replication: peer="{Voter server-1422e03a-af23-f87e-e595-2d8e1c9f2b0f 1422e03a-af23-f87e-e595-2d8e1c9f2b0f}" 2025-01-28T14:48:56.246Z [INFO] server-c7c160c6-db20-0af7-e998-5c658a7b2bef: pipelining replication: peer="{Voter server-7c127b04-fbe8-a3b1-b64d-38f4c7d74e30 7c127b04-fbe8-a3b1-b64d-38f4c7d74e30}" 2025-01-28T14:48:56.451Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:56.245377569 +0000 UTC m=+119.328782593 (1 nodes), 144.437623ms from start of poll, 146.726691ms from cluster start. Timeout at 2025-01-28 14:48:56.450580575 +0000 UTC m=+119.533985599, 205.203006ms after stability 2025-01-28T14:48:56.452Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:56.659Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:56.452209712 +0000 UTC m=+119.535614403 (1 nodes), 7.334µs from start of poll, 353.558501ms from cluster start. Timeout at 2025-01-28 14:48:56.657607056 +0000 UTC m=+119.741012081, 205.397678ms after stability 2025-01-28T14:48:56.664Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:56.870Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:56.664853694 +0000 UTC m=+119.748258385 (2 nodes), 6.334µs from start of poll, 566.202483ms from cluster start. Timeout at 2025-01-28 14:48:56.870266039 +0000 UTC m=+119.953670730, 205.412345ms after stability 2025-01-28T14:48:56.870Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:57.076Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:56.870728141 +0000 UTC m=+119.954132832 (1 nodes), 5.333µs from start of poll, 772.07693ms from cluster start. Timeout at 2025-01-28 14:48:57.076183822 +0000 UTC m=+120.159589179, 205.456347ms after stability raft_test.go:2961: [INFO] restarting Node at 1422e03a-af23-f87e-e595-2d8e1c9f2b0f [Follower] 2025-01-28T14:48:57.077Z [INFO] server-1422e03a-af23-f87e-e595-2d8e1c9f2b0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1422e03a-af23-f87e-e595-2d8e1c9f2b0f Address:1422e03a-af23-f87e-e595-2d8e1c9f2b0f} {Suffrage:Voter ID:server-7c127b04-fbe8-a3b1-b64d-38f4c7d74e30 Address:7c127b04-fbe8-a3b1-b64d-38f4c7d74e30} {Suffrage:Voter ID:server-c7c160c6-db20-0af7-e998-5c658a7b2bef Address:c7c160c6-db20-0af7-e998-5c658a7b2bef}]" 2025-01-28T14:48:57.077Z [INFO] server-1422e03a-af23-f87e-e595-2d8e1c9f2b0f: entering follower state: follower="Node at 1422e03a-af23-f87e-e595-2d8e1c9f2b0f [Follower]" leader-address= leader-id= 2025-01-28T14:48:57.344Z [INFO] server-c7c160c6-db20-0af7-e998-5c658a7b2bef: aborting pipeline replication: peer="{Voter server-7c127b04-fbe8-a3b1-b64d-38f4c7d74e30 7c127b04-fbe8-a3b1-b64d-38f4c7d74e30}" 2025-01-28T14:48:57.726Z [ERROR] server-c7c160c6-db20-0af7-e998-5c658a7b2bef: failed to pipeline appendEntries: peer="{Voter server-1422e03a-af23-f87e-e595-2d8e1c9f2b0f 1422e03a-af23-f87e-e595-2d8e1c9f2b0f}" error="command enqueue timeout" 2025-01-28T14:48:57.726Z [INFO] server-c7c160c6-db20-0af7-e998-5c658a7b2bef: aborting pipeline replication: peer="{Voter server-1422e03a-af23-f87e-e595-2d8e1c9f2b0f 1422e03a-af23-f87e-e595-2d8e1c9f2b0f}" 2025-01-28T14:48:57.843Z [ERROR] server-c7c160c6-db20-0af7-e998-5c658a7b2bef: failed to heartbeat to: peer=7c127b04-fbe8-a3b1-b64d-38f4c7d74e30 backoff time=10ms error="command timed out" 2025-01-28T14:48:57.845Z [ERROR] server-c7c160c6-db20-0af7-e998-5c658a7b2bef: failed to heartbeat to: peer=1422e03a-af23-f87e-e595-2d8e1c9f2b0f backoff time=10ms error="command timed out" --- PASS: TestRaft_FollowerRemovalNoElection (1.76s) === RUN TestRaft_VoteWithNoIDNoAddr 2025-01-28T14:48:57.855Z [INFO] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb Address:1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb} {Suffrage:Voter ID:server-25f685d3-7df0-63c4-baa1-426fd42a906a Address:25f685d3-7df0-63c4-baa1-426fd42a906a} {Suffrage:Voter ID:server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b Address:a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b}]" 2025-01-28T14:48:57.856Z [INFO] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: entering follower state: follower="Node at 1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb [Follower]" leader-address= leader-id= 2025-01-28T14:48:57.856Z [INFO] server-25f685d3-7df0-63c4-baa1-426fd42a906a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb Address:1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb} {Suffrage:Voter ID:server-25f685d3-7df0-63c4-baa1-426fd42a906a Address:25f685d3-7df0-63c4-baa1-426fd42a906a} {Suffrage:Voter ID:server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b Address:a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b}]" 2025-01-28T14:48:57.857Z [INFO] server-25f685d3-7df0-63c4-baa1-426fd42a906a: entering follower state: follower="Node at 25f685d3-7df0-63c4-baa1-426fd42a906a [Follower]" leader-address= leader-id= 2025-01-28T14:48:57.857Z [INFO] server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb Address:1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb} {Suffrage:Voter ID:server-25f685d3-7df0-63c4-baa1-426fd42a906a Address:25f685d3-7df0-63c4-baa1-426fd42a906a} {Suffrage:Voter ID:server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b Address:a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b}]" 2025-01-28T14:48:57.857Z [INFO] server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b: entering follower state: follower="Node at a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b [Follower]" leader-address= leader-id= 2025-01-28T14:48:57.858Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:57.908Z [WARN] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2025-01-28T14:48:57.908Z [INFO] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: entering candidate state: node="Node at 1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb [Candidate]" term=2 2025-01-28T14:48:57.910Z [INFO] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: election won: term=2 tally=2 2025-01-28T14:48:57.910Z [INFO] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: entering leader state: leader="Node at 1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb [Leader]" 2025-01-28T14:48:57.910Z [INFO] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: added peer, starting replication: peer=server-25f685d3-7df0-63c4-baa1-426fd42a906a 2025-01-28T14:48:57.910Z [INFO] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: added peer, starting replication: peer=server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b 2025-01-28T14:48:57.910Z [INFO] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: pipelining replication: peer="{Voter server-25f685d3-7df0-63c4-baa1-426fd42a906a 25f685d3-7df0-63c4-baa1-426fd42a906a}" 2025-01-28T14:48:57.911Z [INFO] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: pipelining replication: peer="{Voter server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b}" 2025-01-28T14:48:58.015Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:57.910282015 +0000 UTC m=+120.993686707 (1 nodes), 51.984211ms from start of poll, 55.228974ms from cluster start. Timeout at 2025-01-28 14:48:58.015291015 +0000 UTC m=+121.098695706, 105.008999ms after stability 2025-01-28T14:48:58.015Z [INFO] cluster: starting stability test: raft-state=Leader 2025-01-28T14:48:58.122Z [INFO] cluster: stable state for Leader reached at 2025-01-28 14:48:58.015713813 +0000 UTC m=+121.099118504 (1 nodes), 6.333µs from start of poll, 160.660771ms from cluster start. Timeout at 2025-01-28 14:48:58.121138826 +0000 UTC m=+121.204544184, 105.42568ms after stability 2025-01-28T14:48:58.127Z [INFO] cluster: starting stability test: raft-state=Follower 2025-01-28T14:48:58.234Z [INFO] cluster: stable state for Follower reached at 2025-01-28 14:48:58.128024816 +0000 UTC m=+121.211429840 (2 nodes), 8.334µs from start of poll, 272.972107ms from cluster start. Timeout at 2025-01-28 14:48:58.233462494 +0000 UTC m=+121.316867519, 105.437679ms after stability 2025-01-28T14:48:58.235Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to heartbeat to: peer=a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b backoff time=10ms error="failed to connect to peer: a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b" 2025-01-28T14:48:58.235Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to heartbeat to: peer=25f685d3-7df0-63c4-baa1-426fd42a906a backoff time=10ms error="failed to connect to peer: 25f685d3-7df0-63c4-baa1-426fd42a906a" 2025-01-28T14:48:58.239Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to pipeline appendEntries: peer="{Voter server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b}" error="append pipeline closed" 2025-01-28T14:48:58.239Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to pipeline appendEntries: peer="{Voter server-25f685d3-7df0-63c4-baa1-426fd42a906a 25f685d3-7df0-63c4-baa1-426fd42a906a}" error="append pipeline closed" 2025-01-28T14:48:58.240Z [INFO] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: aborting pipeline replication: peer="{Voter server-25f685d3-7df0-63c4-baa1-426fd42a906a 25f685d3-7df0-63c4-baa1-426fd42a906a}" 2025-01-28T14:48:58.239Z [INFO] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: aborting pipeline replication: peer="{Voter server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b}" 2025-01-28T14:48:58.246Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to appendEntries to: peer="{Voter server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b}" error="failed to connect to peer: a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b" 2025-01-28T14:48:58.247Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to appendEntries to: peer="{Voter server-25f685d3-7df0-63c4-baa1-426fd42a906a 25f685d3-7df0-63c4-baa1-426fd42a906a}" error="failed to connect to peer: 25f685d3-7df0-63c4-baa1-426fd42a906a" 2025-01-28T14:48:58.253Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to heartbeat to: peer=a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b backoff time=10ms error="failed to connect to peer: a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b" 2025-01-28T14:48:58.255Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to heartbeat to: peer=25f685d3-7df0-63c4-baa1-426fd42a906a backoff time=10ms error="failed to connect to peer: 25f685d3-7df0-63c4-baa1-426fd42a906a" 2025-01-28T14:48:58.265Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to appendEntries to: peer="{Voter server-25f685d3-7df0-63c4-baa1-426fd42a906a 25f685d3-7df0-63c4-baa1-426fd42a906a}" error="failed to connect to peer: 25f685d3-7df0-63c4-baa1-426fd42a906a" 2025-01-28T14:48:58.267Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to appendEntries to: peer="{Voter server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b}" error="failed to connect to peer: a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b" 2025-01-28T14:48:58.273Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to heartbeat to: peer=a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b backoff time=10ms error="failed to connect to peer: a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b" 2025-01-28T14:48:58.273Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to heartbeat to: peer=25f685d3-7df0-63c4-baa1-426fd42a906a backoff time=10ms error="failed to connect to peer: 25f685d3-7df0-63c4-baa1-426fd42a906a" 2025-01-28T14:48:58.281Z [WARN] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to contact: server-id=server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b time=50.596836ms 2025-01-28T14:48:58.282Z [WARN] server-25f685d3-7df0-63c4-baa1-426fd42a906a: heartbeat timeout reached, starting election: last-leader-addr=1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb last-leader-id=server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb 2025-01-28T14:48:58.282Z [INFO] server-25f685d3-7df0-63c4-baa1-426fd42a906a: entering candidate state: node="Node at 25f685d3-7df0-63c4-baa1-426fd42a906a [Candidate]" term=3 2025-01-28T14:48:58.283Z [WARN] server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b: rejecting vote request since we have a leader: from=25f685d3-7df0-63c4-baa1-426fd42a906a leader=1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb leader-id=server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb 2025-01-28T14:48:58.283Z [ERROR] server-25f685d3-7df0-63c4-baa1-426fd42a906a: failed to make requestVote RPC: target="{Voter server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb 1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb}" error="failed to connect to peer: 1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb" term=3 2025-01-28T14:48:58.283Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to appendEntries to: peer="{Voter server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b}" error="failed to connect to peer: a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b" 2025-01-28T14:48:58.284Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to appendEntries to: peer="{Voter server-25f685d3-7df0-63c4-baa1-426fd42a906a 25f685d3-7df0-63c4-baa1-426fd42a906a}" error="failed to connect to peer: 25f685d3-7df0-63c4-baa1-426fd42a906a" 2025-01-28T14:48:58.291Z [WARN] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to contact: server-id=server-25f685d3-7df0-63c4-baa1-426fd42a906a time=59.326762ms 2025-01-28T14:48:58.291Z [WARN] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to contact: server-id=server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b time=60.889809ms 2025-01-28T14:48:58.291Z [WARN] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to contact quorum of nodes, stepping down 2025-01-28T14:48:58.292Z [INFO] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: entering follower state: follower="Node at 1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb [Follower]" leader-address= leader-id= 2025-01-28T14:48:58.291Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to heartbeat to: peer=25f685d3-7df0-63c4-baa1-426fd42a906a backoff time=20ms error="failed to connect to peer: 25f685d3-7df0-63c4-baa1-426fd42a906a" 2025-01-28T14:48:58.293Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to heartbeat to: peer=a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b backoff time=20ms error="failed to connect to peer: a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b" 2025-01-28T14:48:58.297Z [WARN] server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b: heartbeat timeout reached, starting election: last-leader-addr=1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb last-leader-id=server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb 2025-01-28T14:48:58.298Z [INFO] server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b: entering candidate state: node="Node at a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b [Candidate]" term=3 2025-01-28T14:48:58.298Z [ERROR] server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b: failed to make requestVote RPC: target="{Voter server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb 1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb}" error="failed to connect to peer: 1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb" term=3 2025-01-28T14:48:58.298Z [INFO] server-25f685d3-7df0-63c4-baa1-426fd42a906a: duplicate requestVote for same term: term=3 2025-01-28T14:48:58.311Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to appendEntries to: peer="{Voter server-a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b}" error="failed to connect to peer: a34baadf-8ed7-0dfd-6ba1-a1ebeca5aa7b" 2025-01-28T14:48:58.311Z [ERROR] server-1b49f7fa-dc5d-c142-fa88-f33ab1c3d9eb: failed to appendEntries to: peer="{Voter server-25f685d3-7df0-63c4-baa1-426fd42a906a 25f685d3-7df0-63c4-baa1-426fd42a906a}" error="failed to connect to peer: 25f685d3-7df0-63c4-baa1-426fd42a906a" 2025-01-28T14:48:58.339Z [INFO] server-25f685d3-7df0-63c4-baa1-426fd42a906a: entering follower state: follower="Node at 25f685d3-7df0-63c4-baa1-426fd42a906a [Follower]" leader-address= leader-id= --- PASS: TestRaft_VoteWithNoIDNoAddr (0.49s) === RUN TestRaft_runFollower_State_Transition === RUN TestRaft_runFollower_State_Transition/NonVoter 2025-01-28T14:48:58.346Z [INFO] 127.0.0.1:46455 :: starting node: addr=127.0.0.1:46455 2025-01-28T14:48:58.347Z [INFO] 127.0.0.1:46455 :: initial configuration: index=0 servers=[] 2025-01-28T14:48:58.347Z [INFO] 127.0.0.1:46455 :: entering follower state: follower="Node at 127.0.0.1:46455 [Follower]" leader-address= leader-id= 2025-01-28T14:48:58.408Z [WARN] 127.0.0.1:46455 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election === RUN TestRaft_runFollower_State_Transition/Voter 2025-01-28T14:48:58.500Z [INFO] 127.0.0.1:39569 :: starting node: addr=127.0.0.1:39569 2025-01-28T14:48:58.500Z [INFO] 127.0.0.1:39569 :: initial configuration: index=0 servers=[] 2025-01-28T14:48:58.500Z [INFO] 127.0.0.1:39569 :: entering follower state: follower="Node at 127.0.0.1:39569 [Follower]" leader-address= leader-id= 2025-01-28T14:48:58.573Z [WARN] 127.0.0.1:39569 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= === RUN TestRaft_runFollower_State_Transition/Not_in_Config 2025-01-28T14:48:58.653Z [INFO] 127.0.0.1:42957 :: starting node: addr=127.0.0.1:42957 2025-01-28T14:48:58.654Z [INFO] 127.0.0.1:42957 :: initial configuration: index=0 servers=[] 2025-01-28T14:48:58.654Z [INFO] 127.0.0.1:42957 :: entering follower state: follower="Node at 127.0.0.1:42957 [Follower]" leader-address= leader-id= 2025-01-28T14:48:58.714Z [WARN] 127.0.0.1:42957 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election --- PASS: TestRaft_runFollower_State_Transition (0.46s) --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s) === RUN TestRaft_runFollower_ReloadTimeoutConfigs 2025-01-28T14:48:58.807Z [INFO] 127.0.0.1:40327 :: starting node: addr=127.0.0.1:40327 2025-01-28T14:48:58.807Z [INFO] 127.0.0.1:40327 :: initial configuration: index=0 servers=[] 2025-01-28T14:48:58.807Z [INFO] 127.0.0.1:40327 :: entering follower state: follower="Node at 127.0.0.1:40327 [Follower]" leader-address= leader-id= 2025-01-28T14:48:58.808Z [WARN] 127.0.0.1:40327 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= --- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s) === RUN TestSaturationMetric === RUN TestSaturationMetric/without_smoothing --- PASS: TestSaturationMetric (0.00s) --- PASS: TestSaturationMetric/without_smoothing (0.00s) === RUN TestSaturationMetric_IncorrectUsage === RUN TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_first --- PASS: TestSaturationMetric_IncorrectUsage (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_EmptyAddr --- PASS: TestTCPTransport_EmptyAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.01s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.03s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) === RUN TestOverrideNotifyBool --- PASS: TestOverrideNotifyBool (0.00s) PASS ok github.com/hashicorp/raft 122.124s make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' create-stamp debian/debhelper-build-stamp dh_testroot -O--buildsystem=golang dh_prep -O--buildsystem=golang dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang dh_installdocs -O--buildsystem=golang dh_installchangelogs -O--buildsystem=golang dh_installsystemduser -O--buildsystem=golang dh_perl -O--buildsystem=golang dh_link -O--buildsystem=golang dh_strip_nondeterminism -O--buildsystem=golang dh_compress -O--buildsystem=golang dh_fixperms -O--buildsystem=golang dh_missing -O--buildsystem=golang dh_installdeb -O--buildsystem=golang dh_golang -O--buildsystem=golang dh_gencontrol -O--buildsystem=golang dh_md5sums -O--buildsystem=golang dh_builddeb -O--buildsystem=golang dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'. dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_armhf.buildinfo dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_armhf.changes dpkg-genchanges: info: binary-only upload (no source code included) dpkg-source --after-build . dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/5020 and its subdirectories I: Current time: Tue Jan 28 02:49:37 -12 2025 I: pbuilder-time-stamp: 1738075777