Diff of the two buildlogs: -- --- b1/build.log 2024-01-09 05:10:22.070257919 +0000 +++ b2/build.log 2024-01-09 05:11:29.728146061 +0000 @@ -1,6 +1,6 @@ I: pbuilder: network access will be disabled during build -I: Current time: Mon Jan 8 17:08:02 -12 2024 -I: pbuilder-time-stamp: 1704776882 +I: Current time: Tue Feb 11 01:33:23 +14 2025 +I: pbuilder-time-stamp: 1739187203 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/bullseye-reproducible-base.tgz] I: copying local configuration @@ -18,7 +18,7 @@ I: copying [./golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz] I: Extracting source gpgv: unknown type of key resource 'trustedkeys.kbx' -gpgv: keyblock resource '/tmp/dpkg-verify-sig.IocVDySR/trustedkeys.kbx': General error +gpgv: keyblock resource '/tmp/dpkg-verify-sig.CPYwhOi7/trustedkeys.kbx': General error gpgv: Signature made Sun Jan 19 07:24:28 2020 gpgv: using RSA key 50BC7CF939D20C272A6B065652B6BBD953968D1B gpgv: Can't check signature: No public key @@ -34,51 +34,82 @@ dpkg-source: info: applying t-skip--TestRaft_SnapshotRestore_PeerChange.patch I: using fakeroot in build. I: Installing the build-deps -I: user script /srv/workspace/pbuilder/904/tmp/hooks/D02_print_environment starting +I: user script /srv/workspace/pbuilder/63060/tmp/hooks/D01_modify_environment starting +debug: Running on ionos6-i386. +I: Changing host+domainname to test build reproducibility +I: Adding a custom variable just for the fun of it... +I: Changing /bin/sh to bash +'/bin/sh' -> '/bin/bash' +lrwxrwxrwx 1 root root 9 Feb 10 11:33 /bin/sh -> /bin/bash +I: Setting pbuilder2's login shell to /bin/bash +I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other +I: user script /srv/workspace/pbuilder/63060/tmp/hooks/D01_modify_environment finished +I: user script /srv/workspace/pbuilder/63060/tmp/hooks/D02_print_environment starting I: set - BUILDDIR='/build/reproducible-path' - BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' - BUILDUSERNAME='pbuilder1' - BUILD_ARCH='i386' - DEBIAN_FRONTEND='noninteractive' - DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all,-fixfilepath parallel=8 ' - DISTRIBUTION='bullseye' - HOME='/root' - HOST_ARCH='i386' + BASH=/bin/sh + BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath + BASH_ALIASES=() + BASH_ARGC=() + BASH_ARGV=() + BASH_CMDS=() + BASH_LINENO=([0]="12" [1]="0") + BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment") + BASH_VERSINFO=([0]="5" [1]="1" [2]="4" [3]="1" [4]="release" [5]="i686-pc-linux-gnu") + BASH_VERSION='5.1.4(1)-release' + BUILDDIR=/build/reproducible-path + BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other' + BUILDUSERNAME=pbuilder2 + BUILD_ARCH=i386 + DEBIAN_FRONTEND=noninteractive + DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all,-fixfilepath parallel=15 ' + DIRSTACK=() + DISTRIBUTION=bullseye + EUID=0 + FUNCNAME=([0]="Echo" [1]="main") + GROUPS=() + HOME=/root + HOSTNAME=i-capture-the-hostname + HOSTTYPE=i686 + HOST_ARCH=i386 IFS=' ' - INVOCATION_ID='c7bdbcbca84f40239dfe7a5fef4cb1a5' - LANG='C' - LANGUAGE='en_US:en' - LC_ALL='C' - LD_LIBRARY_PATH='/usr/lib/libeatmydata' - LD_PRELOAD='libeatmydata.so' - MAIL='/var/mail/root' - OPTIND='1' - PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' - PBCURRENTCOMMANDLINEOPERATION='build' - PBUILDER_OPERATION='build' - PBUILDER_PKGDATADIR='/usr/share/pbuilder' - PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' - PBUILDER_SYSCONFDIR='/etc' - PPID='904' - PS1='# ' - PS2='> ' + INVOCATION_ID=5529d8d1cd1b4a2a9b4eedde0fded182 + LANG=C + LANGUAGE=de_CH:de + LC_ALL=C + LD_LIBRARY_PATH=/usr/lib/libeatmydata + LD_PRELOAD=libeatmydata.so + MACHTYPE=i686-pc-linux-gnu + MAIL=/var/mail/root + OPTERR=1 + OPTIND=1 + OSTYPE=linux-gnu + PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path + PBCURRENTCOMMANDLINEOPERATION=build + PBUILDER_OPERATION=build + PBUILDER_PKGDATADIR=/usr/share/pbuilder + PBUILDER_PKGLIBDIR=/usr/lib/pbuilder + PBUILDER_SYSCONFDIR=/etc + PIPESTATUS=([0]="0") + POSIXLY_CORRECT=y + PPID=63060 PS4='+ ' - PWD='/' - SHELL='/bin/bash' - SHLVL='2' - SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.IdhfiL2y/pbuilderrc_H70d --distribution bullseye --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bullseye-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.IdhfiL2y/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.1.2-1.dsc' - SUDO_GID='112' - SUDO_UID='107' - SUDO_USER='jenkins' - TERM='unknown' - TZ='/usr/share/zoneinfo/Etc/GMT+12' - USER='root' - _='/usr/bin/systemd-run' - http_proxy='http://78.137.99.97:3128' + PWD=/ + SHELL=/bin/bash + SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix + SHLVL=3 + SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.IdhfiL2y/pbuilderrc_1JhH --distribution bullseye --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bullseye-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.IdhfiL2y/b2 --logfile b2/build.log golang-github-hashicorp-raft_1.1.2-1.dsc' + SUDO_GID=112 + SUDO_UID=107 + SUDO_USER=jenkins + TERM=unknown + TZ=/usr/share/zoneinfo/Etc/GMT-14 + UID=0 + USER=root + _='I: set' + http_proxy=http://85.184.249.68:3128 I: uname -a - Linux ionos2-i386 6.1.0-17-686-pae #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) i686 GNU/Linux + Linux i-capture-the-hostname 6.1.0-17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux I: ls -l /bin total 5776 -rwxr-xr-x 1 root root 1367848 Mar 27 2022 bash @@ -138,7 +169,7 @@ -rwxr-xr-x 1 root root 46984 Sep 22 2020 rmdir -rwxr-xr-x 1 root root 22292 Sep 27 2020 run-parts -rwxr-xr-x 1 root root 125036 Dec 22 2018 sed - lrwxrwxrwx 1 root root 4 Jan 7 09:24 sh -> dash + lrwxrwxrwx 1 root root 9 Feb 10 11:33 sh -> /bin/bash -rwxr-xr-x 1 root root 34696 Sep 22 2020 sleep -rwxr-xr-x 1 root root 83880 Sep 22 2020 stty -rwsr-xr-x 1 root root 79396 Jan 20 2022 su @@ -164,7 +195,7 @@ -rwxr-xr-x 1 root root 2206 Apr 10 2022 zless -rwxr-xr-x 1 root root 1842 Apr 10 2022 zmore -rwxr-xr-x 1 root root 4577 Apr 10 2022 znew -I: user script /srv/workspace/pbuilder/904/tmp/hooks/D02_print_environment finished +I: user script /srv/workspace/pbuilder/63060/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy @@ -323,7 +354,7 @@ Get: 111 http://deb.debian.org/debian bullseye/main i386 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 112 http://deb.debian.org/debian bullseye/main i386 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 113 http://deb.debian.org/debian bullseye/main i386 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] -Fetched 95.8 MB in 8s (12.5 MB/s) +Fetched 95.8 MB in 3s (31.4 MB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package bsdextrautils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 17763 files and directories currently installed.) @@ -796,7 +827,11 @@ fakeroot is already the newest version (1.25.3-1.1). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. I: Building the package -I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes +I: user script /srv/workspace/pbuilder/63060/tmp/hooks/A99_set_merged_usr starting +Not re-configuring usrmerge for bullseye +I: user script /srv/workspace/pbuilder/63060/tmp/hooks/A99_set_merged_usr finished +hostname: Name or service not known +I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.1.2-1 dpkg-buildpackage: info: source distribution unstable @@ -814,126 +849,126 @@ dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang - cd obj-i686-linux-gnu && go install -trimpath -v -p 8 github.com/hashicorp/raft github.com/hashicorp/raft/bench + cd obj-i686-linux-gnu && go install -trimpath -v -p 15 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/unsafeheader +crypto/subtle +crypto/internal/subtle +internal/race +encoding +internal/nettrace +container/list runtime/internal/atomic -unicode/utf8 +unicode/utf16 +math/bits internal/cpu -container/list -internal/race -runtime/internal/sys +unicode/utf8 sync/atomic unicode -crypto/internal/subtle -crypto/subtle -math/bits -encoding -internal/testlog -unicode/utf16 -internal/nettrace -runtime/internal/math +runtime/internal/sys vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/subtle golang.org/x/sys/internal/unsafeheader runtime/cgo -internal/bytealg +internal/testlog +runtime/internal/math math +internal/bytealg runtime internal/reflectlite sync -math/rand internal/singleflight +math/rand errors sort io internal/oserror -strconv github.com/hashicorp/golang-lru/simplelru vendor/golang.org/x/net/dns/dnsmessage +strconv syscall +hash +crypto/internal/randutil bytes strings -crypto/internal/randutil -hash -hash/crc32 crypto/hmac hash/crc64 +hash/crc32 vendor/golang.org/x/crypto/hkdf crypto crypto/rc4 reflect vendor/golang.org/x/text/transform -bufio path -regexp/syntax +bufio github.com/hashicorp/go-immutable-radix html +regexp/syntax internal/syscall/execenv internal/syscall/unix time regexp context internal/poll -encoding/binary internal/fmtsort +encoding/binary os -crypto/cipher -crypto/sha256 encoding/base64 +crypto/cipher +crypto/sha1 crypto/sha512 +crypto/sha256 crypto/ed25519/internal/edwards25519 -crypto/md5 -crypto/sha1 vendor/golang.org/x/crypto/poly1305 +crypto/md5 +golang.org/x/sys/unix encoding/pem crypto/aes crypto/des vendor/golang.org/x/crypto/chacha20 -golang.org/x/sys/unix -fmt -os/signal path/filepath +os/signal runtime/debug +fmt net vendor/golang.org/x/crypto/chacha20poly1305 io/ioutil -encoding/json log -encoding/hex net/url -compress/flate math/big +encoding/hex vendor/golang.org/x/crypto/curve25519 -vendor/golang.org/x/text/unicode/bidi -vendor/golang.org/x/text/unicode/norm -vendor/golang.org/x/net/http2/hpack -compress/gzip -mime mime/quotedprintable -vendor/golang.org/x/text/secure/bidirule +vendor/golang.org/x/net/http2/hpack net/http/internal -github.com/mattn/go-isatty +mime +compress/flate +encoding/json encoding/gob +vendor/golang.org/x/text/unicode/norm +vendor/golang.org/x/text/unicode/bidi go/token text/template/parse -github.com/mattn/go-colorable flag -github.com/fatih/color runtime/trace +testing +vendor/golang.org/x/text/secure/bidirule +github.com/mattn/go-isatty +compress/gzip +github.com/mattn/go-colorable +github.com/fatih/color +text/template +github.com/hashicorp/go-hclog vendor/golang.org/x/net/idna crypto/rand crypto/elliptic -encoding/asn1 crypto/dsa +encoding/asn1 crypto/ed25519 crypto/rsa -github.com/hashicorp/go-hclog -text/template -testing -crypto/x509/pkix +html/template vendor/golang.org/x/crypto/cryptobyte +crypto/x509/pkix crypto/ecdsa -html/template vendor/golang.org/x/net/http/httpproxy net/textproto crypto/x509 @@ -942,13 +977,13 @@ crypto/tls net/http/httptrace net/http -github.com/armon/go-metrics net/rpc +github.com/armon/go-metrics github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench dh_auto_test -O--buildsystem=golang - cd obj-i686-linux-gnu && go test -vet=off -v -p 8 github.com/hashicorp/raft github.com/hashicorp/raft/bench + cd obj-i686-linux-gnu && go test -vet=off -v -p 15 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max @@ -990,33 +1025,33 @@ === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir - testing.go:153: 2024-01-09T05:09:48.750Z [INFO] creating new snapshot: path=/tmp/raft828672824/raft927827767/snapshots/3-10-1704776988750.tmp ---- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.07s) + testing.go:153: 2025-02-10T11:34:03.633Z [INFO] creating new snapshot: path=/tmp/raft280901713/raft366091644/snapshots/3-10-1739187243633.tmp +--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s) === RUN TestFileSS_CreateSnapshot - testing.go:153: 2024-01-09T05:09:48.817Z [INFO] creating new snapshot: path=/tmp/raft266540842/snapshots/3-10-1704776988817.tmp - testing.go:153: 2024-01-09T05:09:48.831Z [WARN] found temporary snapshot: name=3-10-1704776988817.tmp ---- PASS: TestFileSS_CreateSnapshot (0.04s) + testing.go:153: 2025-02-10T11:34:03.637Z [INFO] creating new snapshot: path=/tmp/raft042644139/snapshots/3-10-1739187243637.tmp + testing.go:153: 2025-02-10T11:34:03.640Z [WARN] found temporary snapshot: name=3-10-1739187243637.tmp +--- PASS: TestFileSS_CreateSnapshot (0.00s) === RUN TestFileSS_CancelSnapshot - testing.go:153: 2024-01-09T05:09:48.855Z [INFO] creating new snapshot: path=/tmp/raft125259905/snapshots/3-10-1704776988855.tmp ---- PASS: TestFileSS_CancelSnapshot (0.01s) + testing.go:153: 2025-02-10T11:34:03.642Z [INFO] creating new snapshot: path=/tmp/raft367907342/snapshots/3-10-1739187243642.tmp +--- PASS: TestFileSS_CancelSnapshot (0.00s) === RUN TestFileSS_Retention - testing.go:153: 2024-01-09T05:09:48.863Z [INFO] creating new snapshot: path=/tmp/raft738891244/snapshots/3-10-1704776988863.tmp - testing.go:153: 2024-01-09T05:09:48.873Z [INFO] creating new snapshot: path=/tmp/raft738891244/snapshots/3-11-1704776988873.tmp - testing.go:153: 2024-01-09T05:09:48.891Z [INFO] creating new snapshot: path=/tmp/raft738891244/snapshots/3-12-1704776988891.tmp - testing.go:153: 2024-01-09T05:09:48.899Z [INFO] reaping snapshot: path=/tmp/raft738891244/snapshots/3-10-1704776988863 - testing.go:153: 2024-01-09T05:09:48.900Z [INFO] creating new snapshot: path=/tmp/raft738891244/snapshots/3-13-1704776988900.tmp - testing.go:153: 2024-01-09T05:09:48.919Z [INFO] reaping snapshot: path=/tmp/raft738891244/snapshots/3-11-1704776988873 - testing.go:153: 2024-01-09T05:09:48.919Z [INFO] creating new snapshot: path=/tmp/raft738891244/snapshots/3-14-1704776988919.tmp - testing.go:153: 2024-01-09T05:09:48.943Z [INFO] reaping snapshot: path=/tmp/raft738891244/snapshots/3-12-1704776988891 ---- PASS: TestFileSS_Retention (0.08s) + testing.go:153: 2025-02-10T11:34:03.644Z [INFO] creating new snapshot: path=/tmp/raft868915733/snapshots/3-10-1739187243644.tmp + testing.go:153: 2025-02-10T11:34:03.647Z [INFO] creating new snapshot: path=/tmp/raft868915733/snapshots/3-11-1739187243647.tmp + testing.go:153: 2025-02-10T11:34:03.650Z [INFO] creating new snapshot: path=/tmp/raft868915733/snapshots/3-12-1739187243650.tmp + testing.go:153: 2025-02-10T11:34:03.653Z [INFO] reaping snapshot: path=/tmp/raft868915733/snapshots/3-10-1739187243644 + testing.go:153: 2025-02-10T11:34:03.653Z [INFO] creating new snapshot: path=/tmp/raft868915733/snapshots/3-13-1739187243653.tmp + testing.go:153: 2025-02-10T11:34:03.656Z [INFO] reaping snapshot: path=/tmp/raft868915733/snapshots/3-11-1739187243647 + testing.go:153: 2025-02-10T11:34:03.656Z [INFO] creating new snapshot: path=/tmp/raft868915733/snapshots/3-14-1739187243656.tmp + testing.go:153: 2025-02-10T11:34:03.659Z [INFO] reaping snapshot: path=/tmp/raft868915733/snapshots/3-12-1739187243650 +--- PASS: TestFileSS_Retention (0.02s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering - testing.go:153: 2024-01-09T05:09:48.944Z [INFO] creating new snapshot: path=/tmp/raft180599231/snapshots/5-130350-1704776988944.tmp - testing.go:153: 2024-01-09T05:09:48.971Z [INFO] creating new snapshot: path=/tmp/raft180599231/snapshots/36-204917-1704776988971.tmp ---- PASS: TestFileSS_Ordering (0.05s) + testing.go:153: 2025-02-10T11:34:03.660Z [INFO] creating new snapshot: path=/tmp/raft471177380/snapshots/5-130350-1739187243660.tmp + testing.go:153: 2025-02-10T11:34:03.663Z [INFO] creating new snapshot: path=/tmp/raft471177380/snapshots/36-204917-1739187243663.tmp +--- PASS: TestFileSS_Ordering (0.01s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError @@ -1048,7 +1083,7 @@ === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline ---- PASS: TestNetworkTransport_AppendEntriesPipeline (0.01s) +--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams net_transport_test.go:339: Disabled unreliable test --- SKIP: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.00s) @@ -1061,31 +1096,31 @@ === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn ---- PASS: TestNetworkTransport_PooledConn (0.05s) +--- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff - net_transport_test.go:718: countingWriter: 2024-01-09T05:09:49.075Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2025-02-10T11:34:03.683Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-01-09T05:09:49.083Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2025-02-10T11:34:03.688Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-01-09T05:09:49.095Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2025-02-10T11:34:03.698Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-01-09T05:09:49.118Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2025-02-10T11:34:03.718Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-01-09T05:09:49.163Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2025-02-10T11:34:03.758Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-01-09T05:09:49.246Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2025-02-10T11:34:03.838Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-01-09T05:09:49.411Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2025-02-10T11:34:03.998Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-01-09T05:09:49.734Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2025-02-10T11:34:04.318Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-01-09T05:09:50.378Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2025-02-10T11:34:04.959Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-01-09T05:09:51.382Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2025-02-10T11:34:05.959Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-01-09T05:09:52.383Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2025-02-10T11:34:06.959Z [ERROR] test: failed to accept connection: error="intentional error in test" ---- PASS: TestNetworkTransport_ListenBackoff (4.01s) +--- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON @@ -1093,2875 +1128,2869 @@ === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop -testLoggerAdapter verbose: server-55abb527-eaf0-2b2a-e90d-2328d695ea45: 2024-01-09T05:09:53.084Z [INFO] server-55abb527-eaf0-2b2a-e90d-2328d695ea45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-55abb527-eaf0-2b2a-e90d-2328d695ea45 Address:55abb527-eaf0-2b2a-e90d-2328d695ea45}]" - testing.go:149: server-55abb527-eaf0-2b2a-e90d-2328d695ea45: 2024-01-09T05:09:53.084Z [INFO] server-55abb527-eaf0-2b2a-e90d-2328d695ea45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-55abb527-eaf0-2b2a-e90d-2328d695ea45 Address:55abb527-eaf0-2b2a-e90d-2328d695ea45}]" +testLoggerAdapter verbose: server-5a5ae570-a20c-77b7-62d3-7aaa51d80fc4: 2025-02-10T11:34:07.684Z [INFO] server-5a5ae570-a20c-77b7-62d3-7aaa51d80fc4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a5ae570-a20c-77b7-62d3-7aaa51d80fc4 Address:5a5ae570-a20c-77b7-62d3-7aaa51d80fc4}]" + testing.go:149: server-5a5ae570-a20c-77b7-62d3-7aaa51d80fc4: 2025-02-10T11:34:07.684Z [INFO] server-5a5ae570-a20c-77b7-62d3-7aaa51d80fc4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a5ae570-a20c-77b7-62d3-7aaa51d80fc4 Address:5a5ae570-a20c-77b7-62d3-7aaa51d80fc4}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown -testLoggerAdapter verbose: server-a997f564-6c22-dbcc-2988-3716bf7b1140: 2024-01-09T05:09:53.085Z [INFO] server-a997f564-6c22-dbcc-2988-3716bf7b1140: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a997f564-6c22-dbcc-2988-3716bf7b1140 Address:a997f564-6c22-dbcc-2988-3716bf7b1140}]" - testing.go:149: server-a997f564-6c22-dbcc-2988-3716bf7b1140: 2024-01-09T05:09:53.085Z [INFO] server-a997f564-6c22-dbcc-2988-3716bf7b1140: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a997f564-6c22-dbcc-2988-3716bf7b1140 Address:a997f564-6c22-dbcc-2988-3716bf7b1140}]" +testLoggerAdapter verbose: server-2919e3a4-7739-8c12-39fb-54ac5c5843f1: 2025-02-10T11:34:07.684Z [INFO] server-2919e3a4-7739-8c12-39fb-54ac5c5843f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2919e3a4-7739-8c12-39fb-54ac5c5843f1 Address:2919e3a4-7739-8c12-39fb-54ac5c5843f1}]" + testing.go:149: server-2919e3a4-7739-8c12-39fb-54ac5c5843f1: 2025-02-10T11:34:07.684Z [INFO] server-2919e3a4-7739-8c12-39fb-54ac5c5843f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2919e3a4-7739-8c12-39fb-54ac5c5843f1 Address:2919e3a4-7739-8c12-39fb-54ac5c5843f1}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap -testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.086Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: initial configuration: index=0 servers=[] - testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.086Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: 2024-01-09T05:09:53.086Z [INFO] server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: initial configuration: index=0 servers=[] - testing.go:149: server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: 2024-01-09T05:09:53.086Z [INFO] server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-656011a4-1086-512b-bfcf-946610766331: 2024-01-09T05:09:53.086Z [INFO] server-656011a4-1086-512b-bfcf-946610766331: initial configuration: index=0 servers=[] - testing.go:149: server-656011a4-1086-512b-bfcf-946610766331: 2024-01-09T05:09:53.086Z [INFO] server-656011a4-1086-512b-bfcf-946610766331: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.086Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: entering follower state: follower="Node at a507f1fe-3378-abb6-7909-9a89e5cc2e43 [Follower]" leader= - testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.086Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: entering follower state: follower="Node at a507f1fe-3378-abb6-7909-9a89e5cc2e43 [Follower]" leader= -testLoggerAdapter verbose: server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: 2024-01-09T05:09:53.086Z [INFO] server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: entering follower state: follower="Node at 1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 [Follower]" leader= - testing.go:149: server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: 2024-01-09T05:09:53.086Z [INFO] server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: entering follower state: follower="Node at 1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 [Follower]" leader= -testLoggerAdapter verbose: server-656011a4-1086-512b-bfcf-946610766331: 2024-01-09T05:09:53.086Z [INFO] server-656011a4-1086-512b-bfcf-946610766331: entering follower state: follower="Node at 656011a4-1086-512b-bfcf-946610766331 [Follower]" leader= - testing.go:149: server-656011a4-1086-512b-bfcf-946610766331: 2024-01-09T05:09:53.086Z [INFO] server-656011a4-1086-512b-bfcf-946610766331: entering follower state: follower="Node at 656011a4-1086-512b-bfcf-946610766331 [Follower]" leader= -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.086Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:09:53.086Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: 2024-01-09T05:09:53.155Z [WARN] server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: no known peers, aborting election - testing.go:149: server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: 2024-01-09T05:09:53.155Z [WARN] server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: no known peers, aborting election -testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [WARN] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [WARN] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: entering candidate state: node="Node at a507f1fe-3378-abb6-7909-9a89e5cc2e43 [Candidate]" term=2 - testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: entering candidate state: node="Node at a507f1fe-3378-abb6-7909-9a89e5cc2e43 [Candidate]" term=2 -testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: election won: tally=2 - testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: election won: tally=2 -testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: entering leader state: leader="Node at a507f1fe-3378-abb6-7909-9a89e5cc2e43 [Leader]" - testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: entering leader state: leader="Node at a507f1fe-3378-abb6-7909-9a89e5cc2e43 [Leader]" -testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: added peer, starting replication: peer=server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 - testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: added peer, starting replication: peer=server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 -testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: added peer, starting replication: peer=server-656011a4-1086-512b-bfcf-946610766331 - testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: added peer, starting replication: peer=server-656011a4-1086-512b-bfcf-946610766331 -testLoggerAdapter verbose: server-656011a4-1086-512b-bfcf-946610766331: 2024-01-09T05:09:53.181Z [WARN] server-656011a4-1086-512b-bfcf-946610766331: failed to get previous log: previous-index=1 last-index=0 error="log not found" - testing.go:149: server-656011a4-1086-512b-bfcf-946610766331: 2024-01-09T05:09:53.181Z [WARN] server-656011a4-1086-512b-bfcf-946610766331: failed to get previous log: previous-index=1 last-index=0 error="log not found" -testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.181Z [WARN] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: appendEntries rejected, sending older logs: peer="{Voter server-656011a4-1086-512b-bfcf-946610766331 656011a4-1086-512b-bfcf-946610766331}" next=1 - testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.181Z [WARN] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: appendEntries rejected, sending older logs: peer="{Voter server-656011a4-1086-512b-bfcf-946610766331 656011a4-1086-512b-bfcf-946610766331}" next=1 -testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.181Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: pipelining replication: peer="{Voter server-656011a4-1086-512b-bfcf-946610766331 656011a4-1086-512b-bfcf-946610766331}" - testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.181Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: pipelining replication: peer="{Voter server-656011a4-1086-512b-bfcf-946610766331 656011a4-1086-512b-bfcf-946610766331}" -testLoggerAdapter verbose: server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: 2024-01-09T05:09:53.181Z [WARN] server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: failed to get previous log: previous-index=1 last-index=0 error="log not found" - testing.go:149: server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: 2024-01-09T05:09:53.181Z [WARN] server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: failed to get previous log: previous-index=1 last-index=0 error="log not found" -testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.181Z [WARN] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: appendEntries rejected, sending older logs: peer="{Voter server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 1552d3c7-5e4f-7085-5532-50bfdf8f2ab5}" next=1 - testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.181Z [WARN] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: appendEntries rejected, sending older logs: peer="{Voter server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 1552d3c7-5e4f-7085-5532-50bfdf8f2ab5}" next=1 -testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.181Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: pipelining replication: peer="{Voter server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 1552d3c7-5e4f-7085-5532-50bfdf8f2ab5}" - testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.181Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: pipelining replication: peer="{Voter server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 1552d3c7-5e4f-7085-5532-50bfdf8f2ab5}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.286Z [INFO] stable state for Follower reached at 2024-01-09 05:09:53.181389363 +0000 UTC m=+4.434440333 (2 nodes), 94.771481ms from start of poll, 95.253921ms from cluster start. Timeout at 2024-01-09 05:09:53.286504464 +0000 UTC m=+4.539555593, 105.11526ms after stability - testing.go:149: cluster: 2024-01-09T05:09:53.286Z [INFO] stable state for Follower reached at 2024-01-09 05:09:53.181389363 +0000 UTC m=+4.434440333 (2 nodes), 94.771481ms from start of poll, 95.253921ms from cluster start. Timeout at 2024-01-09 05:09:53.286504464 +0000 UTC m=+4.539555593, 105.11526ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.286Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:53.286Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.391Z [INFO] stable state for Leader reached at 2024-01-09 05:09:53.286673581 +0000 UTC m=+4.539724578 (1 nodes), 2.159µs from start of poll, 200.538166ms from cluster start. Timeout at 2024-01-09 05:09:53.391729209 +0000 UTC m=+4.644780267, 105.055689ms after stability - testing.go:149: cluster: 2024-01-09T05:09:53.391Z [INFO] stable state for Leader reached at 2024-01-09 05:09:53.286673581 +0000 UTC m=+4.539724578 (1 nodes), 2.159µs from start of poll, 200.538166ms from cluster start. Timeout at 2024-01-09 05:09:53.391729209 +0000 UTC m=+4.644780267, 105.055689ms after stability -testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.402Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: aborting pipeline replication: peer="{Voter server-656011a4-1086-512b-bfcf-946610766331 656011a4-1086-512b-bfcf-946610766331}" - testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.402Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: aborting pipeline replication: peer="{Voter server-656011a4-1086-512b-bfcf-946610766331 656011a4-1086-512b-bfcf-946610766331}" -testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.402Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: aborting pipeline replication: peer="{Voter server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 1552d3c7-5e4f-7085-5532-50bfdf8f2ab5}" - testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.402Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: aborting pipeline replication: peer="{Voter server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 1552d3c7-5e4f-7085-5532-50bfdf8f2ab5}" ---- PASS: TestRaft_LiveBootstrap (0.32s) +testLoggerAdapter verbose: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.685Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: initial configuration: index=0 servers=[] + testing.go:149: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.685Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-196ee12f-d0a1-7443-78b4-23de8d3db955: 2025-02-10T11:34:07.685Z [INFO] server-196ee12f-d0a1-7443-78b4-23de8d3db955: initial configuration: index=0 servers=[] + testing.go:149: server-196ee12f-d0a1-7443-78b4-23de8d3db955: 2025-02-10T11:34:07.685Z [INFO] server-196ee12f-d0a1-7443-78b4-23de8d3db955: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.685Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: entering follower state: follower="Node at 20ccb388-7eac-7a49-4054-462a1a6abfba [Follower]" leader= +testLoggerAdapter verbose: server-bfd680a6-3138-b572-07f4-d76107385122: 2025-02-10T11:34:07.685Z [INFO] server-bfd680a6-3138-b572-07f4-d76107385122: initial configuration: index=0 servers=[] + testing.go:149: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.685Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: entering follower state: follower="Node at 20ccb388-7eac-7a49-4054-462a1a6abfba [Follower]" leader= + testing.go:149: server-bfd680a6-3138-b572-07f4-d76107385122: 2025-02-10T11:34:07.685Z [INFO] server-bfd680a6-3138-b572-07f4-d76107385122: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-bfd680a6-3138-b572-07f4-d76107385122: 2025-02-10T11:34:07.685Z [INFO] server-bfd680a6-3138-b572-07f4-d76107385122: entering follower state: follower="Node at bfd680a6-3138-b572-07f4-d76107385122 [Follower]" leader= + testing.go:149: server-bfd680a6-3138-b572-07f4-d76107385122: 2025-02-10T11:34:07.685Z [INFO] server-bfd680a6-3138-b572-07f4-d76107385122: entering follower state: follower="Node at bfd680a6-3138-b572-07f4-d76107385122 [Follower]" leader= +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:07.685Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:07.685Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-196ee12f-d0a1-7443-78b4-23de8d3db955: 2025-02-10T11:34:07.685Z [INFO] server-196ee12f-d0a1-7443-78b4-23de8d3db955: entering follower state: follower="Node at 196ee12f-d0a1-7443-78b4-23de8d3db955 [Follower]" leader= + testing.go:149: server-196ee12f-d0a1-7443-78b4-23de8d3db955: 2025-02-10T11:34:07.685Z [INFO] server-196ee12f-d0a1-7443-78b4-23de8d3db955: entering follower state: follower="Node at 196ee12f-d0a1-7443-78b4-23de8d3db955 [Follower]" leader= +testLoggerAdapter verbose: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [WARN] server-20ccb388-7eac-7a49-4054-462a1a6abfba: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [WARN] server-20ccb388-7eac-7a49-4054-462a1a6abfba: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: entering candidate state: node="Node at 20ccb388-7eac-7a49-4054-462a1a6abfba [Candidate]" term=2 + testing.go:149: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: entering candidate state: node="Node at 20ccb388-7eac-7a49-4054-462a1a6abfba [Candidate]" term=2 +testLoggerAdapter verbose: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: election won: tally=2 + testing.go:149: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: election won: tally=2 +testLoggerAdapter verbose: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: entering leader state: leader="Node at 20ccb388-7eac-7a49-4054-462a1a6abfba [Leader]" + testing.go:149: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: entering leader state: leader="Node at 20ccb388-7eac-7a49-4054-462a1a6abfba [Leader]" +testLoggerAdapter verbose: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: added peer, starting replication: peer=server-196ee12f-d0a1-7443-78b4-23de8d3db955 + testing.go:149: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: added peer, starting replication: peer=server-196ee12f-d0a1-7443-78b4-23de8d3db955 +testLoggerAdapter verbose: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: added peer, starting replication: peer=server-bfd680a6-3138-b572-07f4-d76107385122 + testing.go:149: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: added peer, starting replication: peer=server-bfd680a6-3138-b572-07f4-d76107385122 +testLoggerAdapter verbose: server-196ee12f-d0a1-7443-78b4-23de8d3db955: 2025-02-10T11:34:07.737Z [WARN] server-196ee12f-d0a1-7443-78b4-23de8d3db955: failed to get previous log: previous-index=1 last-index=0 error="log not found" +testLoggerAdapter verbose: server-bfd680a6-3138-b572-07f4-d76107385122: 2025-02-10T11:34:07.737Z [WARN] server-bfd680a6-3138-b572-07f4-d76107385122: failed to get previous log: previous-index=1 last-index=0 error="log not found" + testing.go:149: server-196ee12f-d0a1-7443-78b4-23de8d3db955: 2025-02-10T11:34:07.737Z [WARN] server-196ee12f-d0a1-7443-78b4-23de8d3db955: failed to get previous log: previous-index=1 last-index=0 error="log not found" +testLoggerAdapter verbose: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [WARN] server-20ccb388-7eac-7a49-4054-462a1a6abfba: appendEntries rejected, sending older logs: peer="{Voter server-196ee12f-d0a1-7443-78b4-23de8d3db955 196ee12f-d0a1-7443-78b4-23de8d3db955}" next=1 + testing.go:149: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [WARN] server-20ccb388-7eac-7a49-4054-462a1a6abfba: appendEntries rejected, sending older logs: peer="{Voter server-196ee12f-d0a1-7443-78b4-23de8d3db955 196ee12f-d0a1-7443-78b4-23de8d3db955}" next=1 + testing.go:149: server-bfd680a6-3138-b572-07f4-d76107385122: 2025-02-10T11:34:07.737Z [WARN] server-bfd680a6-3138-b572-07f4-d76107385122: failed to get previous log: previous-index=1 last-index=0 error="log not found" +testLoggerAdapter verbose: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: pipelining replication: peer="{Voter server-196ee12f-d0a1-7443-78b4-23de8d3db955 196ee12f-d0a1-7443-78b4-23de8d3db955}" + testing.go:149: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: pipelining replication: peer="{Voter server-196ee12f-d0a1-7443-78b4-23de8d3db955 196ee12f-d0a1-7443-78b4-23de8d3db955}" +testLoggerAdapter verbose: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [WARN] server-20ccb388-7eac-7a49-4054-462a1a6abfba: appendEntries rejected, sending older logs: peer="{Voter server-bfd680a6-3138-b572-07f4-d76107385122 bfd680a6-3138-b572-07f4-d76107385122}" next=1 + testing.go:149: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [WARN] server-20ccb388-7eac-7a49-4054-462a1a6abfba: appendEntries rejected, sending older logs: peer="{Voter server-bfd680a6-3138-b572-07f4-d76107385122 bfd680a6-3138-b572-07f4-d76107385122}" next=1 +testLoggerAdapter verbose: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: pipelining replication: peer="{Voter server-bfd680a6-3138-b572-07f4-d76107385122 bfd680a6-3138-b572-07f4-d76107385122}" + testing.go:149: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.737Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: pipelining replication: peer="{Voter server-bfd680a6-3138-b572-07f4-d76107385122 bfd680a6-3138-b572-07f4-d76107385122}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:07.842Z [INFO] stable state for Follower reached at 2025-02-10 11:34:07.737266215 +0000 UTC m=+4.106170527 (2 nodes), 51.261022ms from start of poll, 51.687875ms from cluster start. Timeout at 2025-02-10 11:34:07.842368191 +0000 UTC m=+4.211272488, 105.101961ms after stability + testing.go:149: cluster: 2025-02-10T11:34:07.842Z [INFO] stable state for Follower reached at 2025-02-10 11:34:07.737266215 +0000 UTC m=+4.106170527 (2 nodes), 51.261022ms from start of poll, 51.687875ms from cluster start. Timeout at 2025-02-10 11:34:07.842368191 +0000 UTC m=+4.211272488, 105.101961ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:07.842Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:07.842Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:07.947Z [INFO] stable state for Leader reached at 2025-02-10 11:34:07.842474736 +0000 UTC m=+4.211379035 (1 nodes), 396ns from start of poll, 156.896383ms from cluster start. Timeout at 2025-02-10 11:34:07.947549651 +0000 UTC m=+4.316453971, 105.074936ms after stability + testing.go:149: cluster: 2025-02-10T11:34:07.947Z [INFO] stable state for Leader reached at 2025-02-10 11:34:07.842474736 +0000 UTC m=+4.211379035 (1 nodes), 396ns from start of poll, 156.896383ms from cluster start. Timeout at 2025-02-10 11:34:07.947549651 +0000 UTC m=+4.316453971, 105.074936ms after stability +testLoggerAdapter verbose: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.957Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: aborting pipeline replication: peer="{Voter server-bfd680a6-3138-b572-07f4-d76107385122 bfd680a6-3138-b572-07f4-d76107385122}" + testing.go:149: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.957Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: aborting pipeline replication: peer="{Voter server-bfd680a6-3138-b572-07f4-d76107385122 bfd680a6-3138-b572-07f4-d76107385122}" +testLoggerAdapter verbose: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.957Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: aborting pipeline replication: peer="{Voter server-196ee12f-d0a1-7443-78b4-23de8d3db955 196ee12f-d0a1-7443-78b4-23de8d3db955}" + testing.go:149: server-20ccb388-7eac-7a49-4054-462a1a6abfba: 2025-02-10T11:34:07.957Z [INFO] server-20ccb388-7eac-7a49-4054-462a1a6abfba: aborting pipeline replication: peer="{Voter server-196ee12f-d0a1-7443-78b4-23de8d3db955 196ee12f-d0a1-7443-78b4-23de8d3db955}" +--- PASS: TestRaft_LiveBootstrap (0.27s) === RUN TestRaft_RecoverCluster_NoState -testLoggerAdapter verbose: server-7c429492-d4bd-b3ea-1a84-97cfd04660c0: 2024-01-09T05:09:53.403Z [INFO] server-7c429492-d4bd-b3ea-1a84-97cfd04660c0: initial configuration: index=0 servers=[] - testing.go:149: server-7c429492-d4bd-b3ea-1a84-97cfd04660c0: 2024-01-09T05:09:53.403Z [INFO] server-7c429492-d4bd-b3ea-1a84-97cfd04660c0: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-91d2ee6e-388a-0c83-9702-e7ff2dd0a1b8: 2025-02-10T11:34:07.958Z [INFO] server-91d2ee6e-388a-0c83-9702-e7ff2dd0a1b8: initial configuration: index=0 servers=[] + testing.go:149: server-91d2ee6e-388a-0c83-9702-e7ff2dd0a1b8: 2025-02-10T11:34:07.958Z [INFO] server-91d2ee6e-388a-0c83-9702-e7ff2dd0a1b8: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-91d2ee6e-388a-0c83-9702-e7ff2dd0a1b8: 2025-02-10T11:34:07.958Z [INFO] server-91d2ee6e-388a-0c83-9702-e7ff2dd0a1b8: entering follower state: follower="Node at 91d2ee6e-388a-0c83-9702-e7ff2dd0a1b8 [Follower]" leader= + testing.go:149: server-91d2ee6e-388a-0c83-9702-e7ff2dd0a1b8: 2025-02-10T11:34:07.958Z [INFO] server-91d2ee6e-388a-0c83-9702-e7ff2dd0a1b8: entering follower state: follower="Node at 91d2ee6e-388a-0c83-9702-e7ff2dd0a1b8 [Follower]" leader= --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster raft_test.go:119: Disabled unreliable test --- SKIP: TestRaft_RecoverCluster (0.00s) === RUN TestRaft_HasExistingState -testLoggerAdapter verbose: server-725cb9e4-97f5-9799-ba23-5965e058cf3b: 2024-01-09T05:09:53.405Z [INFO] server-725cb9e4-97f5-9799-ba23-5965e058cf3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725cb9e4-97f5-9799-ba23-5965e058cf3b Address:725cb9e4-97f5-9799-ba23-5965e058cf3b} {Suffrage:Voter ID:server-3cfec63e-0c1f-b6f9-9a35-287122a41b24 Address:3cfec63e-0c1f-b6f9-9a35-287122a41b24}]" - testing.go:149: server-725cb9e4-97f5-9799-ba23-5965e058cf3b: 2024-01-09T05:09:53.405Z [INFO] server-725cb9e4-97f5-9799-ba23-5965e058cf3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725cb9e4-97f5-9799-ba23-5965e058cf3b Address:725cb9e4-97f5-9799-ba23-5965e058cf3b} {Suffrage:Voter ID:server-3cfec63e-0c1f-b6f9-9a35-287122a41b24 Address:3cfec63e-0c1f-b6f9-9a35-287122a41b24}]" -testLoggerAdapter verbose: server-725cb9e4-97f5-9799-ba23-5965e058cf3b: 2024-01-09T05:09:53.405Z [INFO] server-725cb9e4-97f5-9799-ba23-5965e058cf3b: entering follower state: follower="Node at 725cb9e4-97f5-9799-ba23-5965e058cf3b [Follower]" leader= - testing.go:149: server-725cb9e4-97f5-9799-ba23-5965e058cf3b: 2024-01-09T05:09:53.405Z [INFO] server-725cb9e4-97f5-9799-ba23-5965e058cf3b: entering follower state: follower="Node at 725cb9e4-97f5-9799-ba23-5965e058cf3b [Follower]" leader= -testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.406Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725cb9e4-97f5-9799-ba23-5965e058cf3b Address:725cb9e4-97f5-9799-ba23-5965e058cf3b} {Suffrage:Voter ID:server-3cfec63e-0c1f-b6f9-9a35-287122a41b24 Address:3cfec63e-0c1f-b6f9-9a35-287122a41b24}]" - testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.406Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725cb9e4-97f5-9799-ba23-5965e058cf3b Address:725cb9e4-97f5-9799-ba23-5965e058cf3b} {Suffrage:Voter ID:server-3cfec63e-0c1f-b6f9-9a35-287122a41b24 Address:3cfec63e-0c1f-b6f9-9a35-287122a41b24}]" -testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.406Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: entering follower state: follower="Node at 3cfec63e-0c1f-b6f9-9a35-287122a41b24 [Follower]" leader= - testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.406Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: entering follower state: follower="Node at 3cfec63e-0c1f-b6f9-9a35-287122a41b24 [Follower]" leader= -testLoggerAdapter verbose: server-b758e9ef-de23-6e38-0468-f7931f6b8f05: 2024-01-09T05:09:53.423Z [INFO] server-b758e9ef-de23-6e38-0468-f7931f6b8f05: initial configuration: index=0 servers=[] - testing.go:149: server-b758e9ef-de23-6e38-0468-f7931f6b8f05: 2024-01-09T05:09:53.423Z [INFO] server-b758e9ef-de23-6e38-0468-f7931f6b8f05: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.423Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:53.423Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-b758e9ef-de23-6e38-0468-f7931f6b8f05: 2024-01-09T05:09:53.424Z [INFO] server-b758e9ef-de23-6e38-0468-f7931f6b8f05: entering follower state: follower="Node at b758e9ef-de23-6e38-0468-f7931f6b8f05 [Follower]" leader= - testing.go:149: server-b758e9ef-de23-6e38-0468-f7931f6b8f05: 2024-01-09T05:09:53.424Z [INFO] server-b758e9ef-de23-6e38-0468-f7931f6b8f05: entering follower state: follower="Node at b758e9ef-de23-6e38-0468-f7931f6b8f05 [Follower]" leader= -testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [WARN] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [WARN] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: entering candidate state: node="Node at 3cfec63e-0c1f-b6f9-9a35-287122a41b24 [Candidate]" term=2 - testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: entering candidate state: node="Node at 3cfec63e-0c1f-b6f9-9a35-287122a41b24 [Candidate]" term=2 -testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: election won: tally=2 - testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: election won: tally=2 -testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: entering leader state: leader="Node at 3cfec63e-0c1f-b6f9-9a35-287122a41b24 [Leader]" - testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: entering leader state: leader="Node at 3cfec63e-0c1f-b6f9-9a35-287122a41b24 [Leader]" -testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: added peer, starting replication: peer=server-725cb9e4-97f5-9799-ba23-5965e058cf3b - testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: added peer, starting replication: peer=server-725cb9e4-97f5-9799-ba23-5965e058cf3b -testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.483Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: pipelining replication: peer="{Voter server-725cb9e4-97f5-9799-ba23-5965e058cf3b 725cb9e4-97f5-9799-ba23-5965e058cf3b}" - testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.483Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: pipelining replication: peer="{Voter server-725cb9e4-97f5-9799-ba23-5965e058cf3b 725cb9e4-97f5-9799-ba23-5965e058cf3b}" -testLoggerAdapter verbose: server-b758e9ef-de23-6e38-0468-f7931f6b8f05: 2024-01-09T05:09:53.489Z [WARN] server-b758e9ef-de23-6e38-0468-f7931f6b8f05: no known peers, aborting election - testing.go:149: server-b758e9ef-de23-6e38-0468-f7931f6b8f05: 2024-01-09T05:09:53.489Z [WARN] server-b758e9ef-de23-6e38-0468-f7931f6b8f05: no known peers, aborting election -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.589Z [INFO] stable state for Leader reached at 2024-01-09 05:09:53.483301515 +0000 UTC m=+4.736352615 (1 nodes), 59.316167ms from start of poll, 78.12126ms from cluster start. Timeout at 2024-01-09 05:09:53.589315059 +0000 UTC m=+4.842366230, 106.013615ms after stability - testing.go:149: cluster: 2024-01-09T05:09:53.589Z [INFO] stable state for Leader reached at 2024-01-09 05:09:53.483301515 +0000 UTC m=+4.736352615 (1 nodes), 59.316167ms from start of poll, 78.12126ms from cluster start. Timeout at 2024-01-09 05:09:53.589315059 +0000 UTC m=+4.842366230, 106.013615ms after stability -testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.589Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: updating configuration: command=AddStaging server-id=server-b758e9ef-de23-6e38-0468-f7931f6b8f05 server-addr=b758e9ef-de23-6e38-0468-f7931f6b8f05 servers="[{Suffrage:Voter ID:server-725cb9e4-97f5-9799-ba23-5965e058cf3b Address:725cb9e4-97f5-9799-ba23-5965e058cf3b} {Suffrage:Voter ID:server-3cfec63e-0c1f-b6f9-9a35-287122a41b24 Address:3cfec63e-0c1f-b6f9-9a35-287122a41b24} {Suffrage:Voter ID:server-b758e9ef-de23-6e38-0468-f7931f6b8f05 Address:b758e9ef-de23-6e38-0468-f7931f6b8f05}]" - testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.589Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: updating configuration: command=AddStaging server-id=server-b758e9ef-de23-6e38-0468-f7931f6b8f05 server-addr=b758e9ef-de23-6e38-0468-f7931f6b8f05 servers="[{Suffrage:Voter ID:server-725cb9e4-97f5-9799-ba23-5965e058cf3b Address:725cb9e4-97f5-9799-ba23-5965e058cf3b} {Suffrage:Voter ID:server-3cfec63e-0c1f-b6f9-9a35-287122a41b24 Address:3cfec63e-0c1f-b6f9-9a35-287122a41b24} {Suffrage:Voter ID:server-b758e9ef-de23-6e38-0468-f7931f6b8f05 Address:b758e9ef-de23-6e38-0468-f7931f6b8f05}]" -testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.589Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: added peer, starting replication: peer=server-b758e9ef-de23-6e38-0468-f7931f6b8f05 - testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.589Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: added peer, starting replication: peer=server-b758e9ef-de23-6e38-0468-f7931f6b8f05 -testLoggerAdapter verbose: server-b758e9ef-de23-6e38-0468-f7931f6b8f05: 2024-01-09T05:09:53.589Z [WARN] server-b758e9ef-de23-6e38-0468-f7931f6b8f05: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-b758e9ef-de23-6e38-0468-f7931f6b8f05: 2024-01-09T05:09:53.589Z [WARN] server-b758e9ef-de23-6e38-0468-f7931f6b8f05: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.589Z [WARN] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: appendEntries rejected, sending older logs: peer="{Voter server-b758e9ef-de23-6e38-0468-f7931f6b8f05 b758e9ef-de23-6e38-0468-f7931f6b8f05}" next=1 - testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.589Z [WARN] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: appendEntries rejected, sending older logs: peer="{Voter server-b758e9ef-de23-6e38-0468-f7931f6b8f05 b758e9ef-de23-6e38-0468-f7931f6b8f05}" next=1 -testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.589Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: pipelining replication: peer="{Voter server-b758e9ef-de23-6e38-0468-f7931f6b8f05 b758e9ef-de23-6e38-0468-f7931f6b8f05}" - testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.589Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: pipelining replication: peer="{Voter server-b758e9ef-de23-6e38-0468-f7931f6b8f05 b758e9ef-de23-6e38-0468-f7931f6b8f05}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.590Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:53.590Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.695Z [INFO] stable state for Leader reached at 2024-01-09 05:09:53.590053259 +0000 UTC m=+4.843104338 (1 nodes), 2.388µs from start of poll, 184.872983ms from cluster start. Timeout at 2024-01-09 05:09:53.695062572 +0000 UTC m=+4.948113581, 105.009243ms after stability - testing.go:149: cluster: 2024-01-09T05:09:53.695Z [INFO] stable state for Leader reached at 2024-01-09 05:09:53.590053259 +0000 UTC m=+4.843104338 (1 nodes), 2.388µs from start of poll, 184.872983ms from cluster start. Timeout at 2024-01-09 05:09:53.695062572 +0000 UTC m=+4.948113581, 105.009243ms after stability -testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.695Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: aborting pipeline replication: peer="{Voter server-b758e9ef-de23-6e38-0468-f7931f6b8f05 b758e9ef-de23-6e38-0468-f7931f6b8f05}" - testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.695Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: aborting pipeline replication: peer="{Voter server-b758e9ef-de23-6e38-0468-f7931f6b8f05 b758e9ef-de23-6e38-0468-f7931f6b8f05}" -testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.695Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: aborting pipeline replication: peer="{Voter server-725cb9e4-97f5-9799-ba23-5965e058cf3b 725cb9e4-97f5-9799-ba23-5965e058cf3b}" - testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.695Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: aborting pipeline replication: peer="{Voter server-725cb9e4-97f5-9799-ba23-5965e058cf3b 725cb9e4-97f5-9799-ba23-5965e058cf3b}" ---- PASS: TestRaft_HasExistingState (0.29s) +testLoggerAdapter verbose: server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be: 2025-02-10T11:34:07.959Z [INFO] server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be Address:bc2b45ce-150b-fb9f-cf00-a71bad6c18be} {Suffrage:Voter ID:server-5fa4e35b-c9ad-3195-e842-b7efa886a761 Address:5fa4e35b-c9ad-3195-e842-b7efa886a761}]" + testing.go:149: server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be: 2025-02-10T11:34:07.959Z [INFO] server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be Address:bc2b45ce-150b-fb9f-cf00-a71bad6c18be} {Suffrage:Voter ID:server-5fa4e35b-c9ad-3195-e842-b7efa886a761 Address:5fa4e35b-c9ad-3195-e842-b7efa886a761}]" +testLoggerAdapter verbose: server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be: 2025-02-10T11:34:07.959Z [INFO] server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be: entering follower state: follower="Node at bc2b45ce-150b-fb9f-cf00-a71bad6c18be [Follower]" leader= + testing.go:149: server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be: 2025-02-10T11:34:07.959Z [INFO] server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be: entering follower state: follower="Node at bc2b45ce-150b-fb9f-cf00-a71bad6c18be [Follower]" leader= +testLoggerAdapter verbose: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:07.959Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be Address:bc2b45ce-150b-fb9f-cf00-a71bad6c18be} {Suffrage:Voter ID:server-5fa4e35b-c9ad-3195-e842-b7efa886a761 Address:5fa4e35b-c9ad-3195-e842-b7efa886a761}]" + testing.go:149: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:07.959Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be Address:bc2b45ce-150b-fb9f-cf00-a71bad6c18be} {Suffrage:Voter ID:server-5fa4e35b-c9ad-3195-e842-b7efa886a761 Address:5fa4e35b-c9ad-3195-e842-b7efa886a761}]" +testLoggerAdapter verbose: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:07.959Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: entering follower state: follower="Node at 5fa4e35b-c9ad-3195-e842-b7efa886a761 [Follower]" leader= + testing.go:149: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:07.959Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: entering follower state: follower="Node at 5fa4e35b-c9ad-3195-e842-b7efa886a761 [Follower]" leader= +testLoggerAdapter verbose: server-14fec058-869d-8ea4-9afe-9a1f734217c2: 2025-02-10T11:34:07.960Z [INFO] server-14fec058-869d-8ea4-9afe-9a1f734217c2: initial configuration: index=0 servers=[] + testing.go:149: server-14fec058-869d-8ea4-9afe-9a1f734217c2: 2025-02-10T11:34:07.960Z [INFO] server-14fec058-869d-8ea4-9afe-9a1f734217c2: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-14fec058-869d-8ea4-9afe-9a1f734217c2: 2025-02-10T11:34:07.960Z [INFO] server-14fec058-869d-8ea4-9afe-9a1f734217c2: entering follower state: follower="Node at 14fec058-869d-8ea4-9afe-9a1f734217c2 [Follower]" leader= + testing.go:149: server-14fec058-869d-8ea4-9afe-9a1f734217c2: 2025-02-10T11:34:07.960Z [INFO] server-14fec058-869d-8ea4-9afe-9a1f734217c2: entering follower state: follower="Node at 14fec058-869d-8ea4-9afe-9a1f734217c2 [Follower]" leader= +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:07.960Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:07.960Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.010Z [WARN] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.010Z [WARN] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.010Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: entering candidate state: node="Node at 5fa4e35b-c9ad-3195-e842-b7efa886a761 [Candidate]" term=2 + testing.go:149: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.010Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: entering candidate state: node="Node at 5fa4e35b-c9ad-3195-e842-b7efa886a761 [Candidate]" term=2 +testLoggerAdapter verbose: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.010Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: election won: tally=2 + testing.go:149: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.010Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: election won: tally=2 +testLoggerAdapter verbose: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.010Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: entering leader state: leader="Node at 5fa4e35b-c9ad-3195-e842-b7efa886a761 [Leader]" + testing.go:149: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.010Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: entering leader state: leader="Node at 5fa4e35b-c9ad-3195-e842-b7efa886a761 [Leader]" +testLoggerAdapter verbose: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.010Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: added peer, starting replication: peer=server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be + testing.go:149: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.010Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: added peer, starting replication: peer=server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be +testLoggerAdapter verbose: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.010Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: pipelining replication: peer="{Voter server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be bc2b45ce-150b-fb9f-cf00-a71bad6c18be}" + testing.go:149: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.010Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: pipelining replication: peer="{Voter server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be bc2b45ce-150b-fb9f-cf00-a71bad6c18be}" +testLoggerAdapter verbose: server-14fec058-869d-8ea4-9afe-9a1f734217c2: 2025-02-10T11:34:08.055Z [WARN] server-14fec058-869d-8ea4-9afe-9a1f734217c2: no known peers, aborting election + testing.go:149: server-14fec058-869d-8ea4-9afe-9a1f734217c2: 2025-02-10T11:34:08.055Z [WARN] server-14fec058-869d-8ea4-9afe-9a1f734217c2: no known peers, aborting election +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:08.115Z [INFO] stable state for Leader reached at 2025-02-10 11:34:08.010179043 +0000 UTC m=+4.379083342 (1 nodes), 49.941648ms from start of poll, 50.647713ms from cluster start. Timeout at 2025-02-10 11:34:08.115281511 +0000 UTC m=+4.484185861, 105.102519ms after stability + testing.go:149: cluster: 2025-02-10T11:34:08.115Z [INFO] stable state for Leader reached at 2025-02-10 11:34:08.010179043 +0000 UTC m=+4.379083342 (1 nodes), 49.941648ms from start of poll, 50.647713ms from cluster start. Timeout at 2025-02-10 11:34:08.115281511 +0000 UTC m=+4.484185861, 105.102519ms after stability +testLoggerAdapter verbose: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.115Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: updating configuration: command=AddStaging server-id=server-14fec058-869d-8ea4-9afe-9a1f734217c2 server-addr=14fec058-869d-8ea4-9afe-9a1f734217c2 servers="[{Suffrage:Voter ID:server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be Address:bc2b45ce-150b-fb9f-cf00-a71bad6c18be} {Suffrage:Voter ID:server-5fa4e35b-c9ad-3195-e842-b7efa886a761 Address:5fa4e35b-c9ad-3195-e842-b7efa886a761} {Suffrage:Voter ID:server-14fec058-869d-8ea4-9afe-9a1f734217c2 Address:14fec058-869d-8ea4-9afe-9a1f734217c2}]" + testing.go:149: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.115Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: updating configuration: command=AddStaging server-id=server-14fec058-869d-8ea4-9afe-9a1f734217c2 server-addr=14fec058-869d-8ea4-9afe-9a1f734217c2 servers="[{Suffrage:Voter ID:server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be Address:bc2b45ce-150b-fb9f-cf00-a71bad6c18be} {Suffrage:Voter ID:server-5fa4e35b-c9ad-3195-e842-b7efa886a761 Address:5fa4e35b-c9ad-3195-e842-b7efa886a761} {Suffrage:Voter ID:server-14fec058-869d-8ea4-9afe-9a1f734217c2 Address:14fec058-869d-8ea4-9afe-9a1f734217c2}]" +testLoggerAdapter verbose: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.115Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: added peer, starting replication: peer=server-14fec058-869d-8ea4-9afe-9a1f734217c2 + testing.go:149: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.115Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: added peer, starting replication: peer=server-14fec058-869d-8ea4-9afe-9a1f734217c2 +testLoggerAdapter verbose: server-14fec058-869d-8ea4-9afe-9a1f734217c2: 2025-02-10T11:34:08.115Z [WARN] server-14fec058-869d-8ea4-9afe-9a1f734217c2: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-14fec058-869d-8ea4-9afe-9a1f734217c2: 2025-02-10T11:34:08.115Z [WARN] server-14fec058-869d-8ea4-9afe-9a1f734217c2: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.115Z [WARN] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: appendEntries rejected, sending older logs: peer="{Voter server-14fec058-869d-8ea4-9afe-9a1f734217c2 14fec058-869d-8ea4-9afe-9a1f734217c2}" next=1 + testing.go:149: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.115Z [WARN] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: appendEntries rejected, sending older logs: peer="{Voter server-14fec058-869d-8ea4-9afe-9a1f734217c2 14fec058-869d-8ea4-9afe-9a1f734217c2}" next=1 +testLoggerAdapter verbose: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.115Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: pipelining replication: peer="{Voter server-14fec058-869d-8ea4-9afe-9a1f734217c2 14fec058-869d-8ea4-9afe-9a1f734217c2}" + testing.go:149: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.115Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: pipelining replication: peer="{Voter server-14fec058-869d-8ea4-9afe-9a1f734217c2 14fec058-869d-8ea4-9afe-9a1f734217c2}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:08.120Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:08.120Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:08.225Z [INFO] stable state for Leader reached at 2025-02-10 11:34:08.120778336 +0000 UTC m=+4.489682638 (1 nodes), 429ns from start of poll, 161.247009ms from cluster start. Timeout at 2025-02-10 11:34:08.225794342 +0000 UTC m=+4.594698658, 105.01602ms after stability + testing.go:149: cluster: 2025-02-10T11:34:08.225Z [INFO] stable state for Leader reached at 2025-02-10 11:34:08.120778336 +0000 UTC m=+4.489682638 (1 nodes), 429ns from start of poll, 161.247009ms from cluster start. Timeout at 2025-02-10 11:34:08.225794342 +0000 UTC m=+4.594698658, 105.01602ms after stability +testLoggerAdapter verbose: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.225Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: aborting pipeline replication: peer="{Voter server-14fec058-869d-8ea4-9afe-9a1f734217c2 14fec058-869d-8ea4-9afe-9a1f734217c2}" + testing.go:149: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.225Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: aborting pipeline replication: peer="{Voter server-14fec058-869d-8ea4-9afe-9a1f734217c2 14fec058-869d-8ea4-9afe-9a1f734217c2}" +testLoggerAdapter verbose: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.225Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: aborting pipeline replication: peer="{Voter server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be bc2b45ce-150b-fb9f-cf00-a71bad6c18be}" + testing.go:149: server-5fa4e35b-c9ad-3195-e842-b7efa886a761: 2025-02-10T11:34:08.225Z [INFO] server-5fa4e35b-c9ad-3195-e842-b7efa886a761: aborting pipeline replication: peer="{Voter server-bc2b45ce-150b-fb9f-cf00-a71bad6c18be bc2b45ce-150b-fb9f-cf00-a71bad6c18be}" +--- PASS: TestRaft_HasExistingState (0.27s) === RUN TestRaft_SingleNode -testLoggerAdapter verbose: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.696Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-71bee22d-37bd-2ab1-9807-7add819f0e21 Address:71bee22d-37bd-2ab1-9807-7add819f0e21}]" - testing.go:149: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.696Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-71bee22d-37bd-2ab1-9807-7add819f0e21 Address:71bee22d-37bd-2ab1-9807-7add819f0e21}]" -testLoggerAdapter verbose: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.696Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: entering follower state: follower="Node at 71bee22d-37bd-2ab1-9807-7add819f0e21 [Follower]" leader= - testing.go:149: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.696Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: entering follower state: follower="Node at 71bee22d-37bd-2ab1-9807-7add819f0e21 [Follower]" leader= -testLoggerAdapter verbose: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.757Z [WARN] server-71bee22d-37bd-2ab1-9807-7add819f0e21: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.757Z [WARN] server-71bee22d-37bd-2ab1-9807-7add819f0e21: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.757Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: entering candidate state: node="Node at 71bee22d-37bd-2ab1-9807-7add819f0e21 [Candidate]" term=2 - testing.go:149: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.757Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: entering candidate state: node="Node at 71bee22d-37bd-2ab1-9807-7add819f0e21 [Candidate]" term=2 -testLoggerAdapter verbose: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.757Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: election won: tally=1 - testing.go:149: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.757Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: election won: tally=1 -testLoggerAdapter verbose: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.757Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: entering leader state: leader="Node at 71bee22d-37bd-2ab1-9807-7add819f0e21 [Leader]" - testing.go:149: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.757Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: entering leader state: leader="Node at 71bee22d-37bd-2ab1-9807-7add819f0e21 [Leader]" ---- PASS: TestRaft_SingleNode (0.06s) +testLoggerAdapter verbose: server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: 2025-02-10T11:34:08.226Z [INFO] server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54a76141-f09f-18ca-f88a-fc3fb5f334ef Address:54a76141-f09f-18ca-f88a-fc3fb5f334ef}]" + testing.go:149: server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: 2025-02-10T11:34:08.226Z [INFO] server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54a76141-f09f-18ca-f88a-fc3fb5f334ef Address:54a76141-f09f-18ca-f88a-fc3fb5f334ef}]" +testLoggerAdapter verbose: server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: 2025-02-10T11:34:08.226Z [INFO] server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: entering follower state: follower="Node at 54a76141-f09f-18ca-f88a-fc3fb5f334ef [Follower]" leader= + testing.go:149: server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: 2025-02-10T11:34:08.226Z [INFO] server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: entering follower state: follower="Node at 54a76141-f09f-18ca-f88a-fc3fb5f334ef [Follower]" leader= +testLoggerAdapter verbose: server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: 2025-02-10T11:34:08.278Z [WARN] server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: 2025-02-10T11:34:08.278Z [WARN] server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: 2025-02-10T11:34:08.278Z [INFO] server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: entering candidate state: node="Node at 54a76141-f09f-18ca-f88a-fc3fb5f334ef [Candidate]" term=2 + testing.go:149: server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: 2025-02-10T11:34:08.278Z [INFO] server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: entering candidate state: node="Node at 54a76141-f09f-18ca-f88a-fc3fb5f334ef [Candidate]" term=2 +testLoggerAdapter verbose: server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: 2025-02-10T11:34:08.278Z [INFO] server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: election won: tally=1 + testing.go:149: server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: 2025-02-10T11:34:08.278Z [INFO] server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: election won: tally=1 +testLoggerAdapter verbose: server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: 2025-02-10T11:34:08.278Z [INFO] server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: entering leader state: leader="Node at 54a76141-f09f-18ca-f88a-fc3fb5f334ef [Leader]" + testing.go:149: server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: 2025-02-10T11:34:08.278Z [INFO] server-54a76141-f09f-18ca-f88a-fc3fb5f334ef: entering leader state: leader="Node at 54a76141-f09f-18ca-f88a-fc3fb5f334ef [Leader]" +--- PASS: TestRaft_SingleNode (0.05s) === RUN TestRaft_TripleNode -testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.758Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02569c5a-a550-e67a-f071-c8cfcd65d351 Address:02569c5a-a550-e67a-f071-c8cfcd65d351} {Suffrage:Voter ID:server-b7d18231-0a9e-a900-556c-207650145640 Address:b7d18231-0a9e-a900-556c-207650145640} {Suffrage:Voter ID:server-52e1b013-6bf0-0a59-529f-985d3543858f Address:52e1b013-6bf0-0a59-529f-985d3543858f}]" - testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.758Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02569c5a-a550-e67a-f071-c8cfcd65d351 Address:02569c5a-a550-e67a-f071-c8cfcd65d351} {Suffrage:Voter ID:server-b7d18231-0a9e-a900-556c-207650145640 Address:b7d18231-0a9e-a900-556c-207650145640} {Suffrage:Voter ID:server-52e1b013-6bf0-0a59-529f-985d3543858f Address:52e1b013-6bf0-0a59-529f-985d3543858f}]" -testLoggerAdapter verbose: server-b7d18231-0a9e-a900-556c-207650145640: 2024-01-09T05:09:53.758Z [INFO] server-b7d18231-0a9e-a900-556c-207650145640: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02569c5a-a550-e67a-f071-c8cfcd65d351 Address:02569c5a-a550-e67a-f071-c8cfcd65d351} {Suffrage:Voter ID:server-b7d18231-0a9e-a900-556c-207650145640 Address:b7d18231-0a9e-a900-556c-207650145640} {Suffrage:Voter ID:server-52e1b013-6bf0-0a59-529f-985d3543858f Address:52e1b013-6bf0-0a59-529f-985d3543858f}]" - testing.go:149: server-b7d18231-0a9e-a900-556c-207650145640: 2024-01-09T05:09:53.758Z [INFO] server-b7d18231-0a9e-a900-556c-207650145640: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02569c5a-a550-e67a-f071-c8cfcd65d351 Address:02569c5a-a550-e67a-f071-c8cfcd65d351} {Suffrage:Voter ID:server-b7d18231-0a9e-a900-556c-207650145640 Address:b7d18231-0a9e-a900-556c-207650145640} {Suffrage:Voter ID:server-52e1b013-6bf0-0a59-529f-985d3543858f Address:52e1b013-6bf0-0a59-529f-985d3543858f}]" -testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.758Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: entering follower state: follower="Node at 02569c5a-a550-e67a-f071-c8cfcd65d351 [Follower]" leader= - testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.758Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: entering follower state: follower="Node at 02569c5a-a550-e67a-f071-c8cfcd65d351 [Follower]" leader= -testLoggerAdapter verbose: server-b7d18231-0a9e-a900-556c-207650145640: 2024-01-09T05:09:53.759Z [INFO] server-b7d18231-0a9e-a900-556c-207650145640: entering follower state: follower="Node at b7d18231-0a9e-a900-556c-207650145640 [Follower]" leader= - testing.go:149: server-b7d18231-0a9e-a900-556c-207650145640: 2024-01-09T05:09:53.759Z [INFO] server-b7d18231-0a9e-a900-556c-207650145640: entering follower state: follower="Node at b7d18231-0a9e-a900-556c-207650145640 [Follower]" leader= -testLoggerAdapter verbose: server-52e1b013-6bf0-0a59-529f-985d3543858f: 2024-01-09T05:09:53.759Z [INFO] server-52e1b013-6bf0-0a59-529f-985d3543858f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02569c5a-a550-e67a-f071-c8cfcd65d351 Address:02569c5a-a550-e67a-f071-c8cfcd65d351} {Suffrage:Voter ID:server-b7d18231-0a9e-a900-556c-207650145640 Address:b7d18231-0a9e-a900-556c-207650145640} {Suffrage:Voter ID:server-52e1b013-6bf0-0a59-529f-985d3543858f Address:52e1b013-6bf0-0a59-529f-985d3543858f}]" - testing.go:149: server-52e1b013-6bf0-0a59-529f-985d3543858f: 2024-01-09T05:09:53.759Z [INFO] server-52e1b013-6bf0-0a59-529f-985d3543858f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02569c5a-a550-e67a-f071-c8cfcd65d351 Address:02569c5a-a550-e67a-f071-c8cfcd65d351} {Suffrage:Voter ID:server-b7d18231-0a9e-a900-556c-207650145640 Address:b7d18231-0a9e-a900-556c-207650145640} {Suffrage:Voter ID:server-52e1b013-6bf0-0a59-529f-985d3543858f Address:52e1b013-6bf0-0a59-529f-985d3543858f}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.759Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:09:53.759Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-52e1b013-6bf0-0a59-529f-985d3543858f: 2024-01-09T05:09:53.759Z [INFO] server-52e1b013-6bf0-0a59-529f-985d3543858f: entering follower state: follower="Node at 52e1b013-6bf0-0a59-529f-985d3543858f [Follower]" leader= - testing.go:149: server-52e1b013-6bf0-0a59-529f-985d3543858f: 2024-01-09T05:09:53.759Z [INFO] server-52e1b013-6bf0-0a59-529f-985d3543858f: entering follower state: follower="Node at 52e1b013-6bf0-0a59-529f-985d3543858f [Follower]" leader= -testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.812Z [WARN] server-02569c5a-a550-e67a-f071-c8cfcd65d351: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.812Z [WARN] server-02569c5a-a550-e67a-f071-c8cfcd65d351: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.812Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: entering candidate state: node="Node at 02569c5a-a550-e67a-f071-c8cfcd65d351 [Candidate]" term=2 - testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.812Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: entering candidate state: node="Node at 02569c5a-a550-e67a-f071-c8cfcd65d351 [Candidate]" term=2 -testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: election won: tally=2 - testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: election won: tally=2 -testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: entering leader state: leader="Node at 02569c5a-a550-e67a-f071-c8cfcd65d351 [Leader]" - testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: entering leader state: leader="Node at 02569c5a-a550-e67a-f071-c8cfcd65d351 [Leader]" -testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: added peer, starting replication: peer=server-b7d18231-0a9e-a900-556c-207650145640 - testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: added peer, starting replication: peer=server-b7d18231-0a9e-a900-556c-207650145640 -testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: added peer, starting replication: peer=server-52e1b013-6bf0-0a59-529f-985d3543858f - testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: added peer, starting replication: peer=server-52e1b013-6bf0-0a59-529f-985d3543858f -testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: pipelining replication: peer="{Voter server-b7d18231-0a9e-a900-556c-207650145640 b7d18231-0a9e-a900-556c-207650145640}" - testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: pipelining replication: peer="{Voter server-b7d18231-0a9e-a900-556c-207650145640 b7d18231-0a9e-a900-556c-207650145640}" -testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.814Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: pipelining replication: peer="{Voter server-52e1b013-6bf0-0a59-529f-985d3543858f 52e1b013-6bf0-0a59-529f-985d3543858f}" - testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.814Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: pipelining replication: peer="{Voter server-52e1b013-6bf0-0a59-529f-985d3543858f 52e1b013-6bf0-0a59-529f-985d3543858f}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.918Z [INFO] stable state for Follower reached at 2024-01-09 05:09:53.813778294 +0000 UTC m=+5.066829460 (2 nodes), 54.536048ms from start of poll, 55.13129ms from cluster start. Timeout at 2024-01-09 05:09:53.918842028 +0000 UTC m=+5.171893074, 105.063614ms after stability - testing.go:149: cluster: 2024-01-09T05:09:53.918Z [INFO] stable state for Follower reached at 2024-01-09 05:09:53.813778294 +0000 UTC m=+5.066829460 (2 nodes), 54.536048ms from start of poll, 55.13129ms from cluster start. Timeout at 2024-01-09 05:09:53.918842028 +0000 UTC m=+5.171893074, 105.063614ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.918Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:53.918Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.024Z [INFO] stable state for Leader reached at 2024-01-09 05:09:53.919018369 +0000 UTC m=+5.172069369 (1 nodes), 2.021µs from start of poll, 160.371199ms from cluster start. Timeout at 2024-01-09 05:09:54.02411078 +0000 UTC m=+5.277161843, 105.092474ms after stability - testing.go:149: cluster: 2024-01-09T05:09:54.024Z [INFO] stable state for Leader reached at 2024-01-09 05:09:53.919018369 +0000 UTC m=+5.172069369 (1 nodes), 2.021µs from start of poll, 160.371199ms from cluster start. Timeout at 2024-01-09 05:09:54.02411078 +0000 UTC m=+5.277161843, 105.092474ms after stability -testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:54.037Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: aborting pipeline replication: peer="{Voter server-52e1b013-6bf0-0a59-529f-985d3543858f 52e1b013-6bf0-0a59-529f-985d3543858f}" - testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:54.037Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: aborting pipeline replication: peer="{Voter server-52e1b013-6bf0-0a59-529f-985d3543858f 52e1b013-6bf0-0a59-529f-985d3543858f}" -testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:54.037Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: aborting pipeline replication: peer="{Voter server-b7d18231-0a9e-a900-556c-207650145640 b7d18231-0a9e-a900-556c-207650145640}" - testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:54.037Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: aborting pipeline replication: peer="{Voter server-b7d18231-0a9e-a900-556c-207650145640 b7d18231-0a9e-a900-556c-207650145640}" ---- PASS: TestRaft_TripleNode (0.28s) +testLoggerAdapter verbose: server-6629d9bc-8f11-04c3-94ff-d919015e30db: 2025-02-10T11:34:08.278Z [INFO] server-6629d9bc-8f11-04c3-94ff-d919015e30db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6629d9bc-8f11-04c3-94ff-d919015e30db Address:6629d9bc-8f11-04c3-94ff-d919015e30db} {Suffrage:Voter ID:server-9c1017c0-6e05-e7e8-2bd5-034147417eed Address:9c1017c0-6e05-e7e8-2bd5-034147417eed} {Suffrage:Voter ID:server-23f6fc87-8f95-c9e5-b119-f1456ede73b7 Address:23f6fc87-8f95-c9e5-b119-f1456ede73b7}]" + testing.go:149: server-6629d9bc-8f11-04c3-94ff-d919015e30db: 2025-02-10T11:34:08.278Z [INFO] server-6629d9bc-8f11-04c3-94ff-d919015e30db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6629d9bc-8f11-04c3-94ff-d919015e30db Address:6629d9bc-8f11-04c3-94ff-d919015e30db} {Suffrage:Voter ID:server-9c1017c0-6e05-e7e8-2bd5-034147417eed Address:9c1017c0-6e05-e7e8-2bd5-034147417eed} {Suffrage:Voter ID:server-23f6fc87-8f95-c9e5-b119-f1456ede73b7 Address:23f6fc87-8f95-c9e5-b119-f1456ede73b7}]" +testLoggerAdapter verbose: server-6629d9bc-8f11-04c3-94ff-d919015e30db: 2025-02-10T11:34:08.279Z [INFO] server-6629d9bc-8f11-04c3-94ff-d919015e30db: entering follower state: follower="Node at 6629d9bc-8f11-04c3-94ff-d919015e30db [Follower]" leader= + testing.go:149: server-6629d9bc-8f11-04c3-94ff-d919015e30db: 2025-02-10T11:34:08.279Z [INFO] server-6629d9bc-8f11-04c3-94ff-d919015e30db: entering follower state: follower="Node at 6629d9bc-8f11-04c3-94ff-d919015e30db [Follower]" leader= +testLoggerAdapter verbose: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.279Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6629d9bc-8f11-04c3-94ff-d919015e30db Address:6629d9bc-8f11-04c3-94ff-d919015e30db} {Suffrage:Voter ID:server-9c1017c0-6e05-e7e8-2bd5-034147417eed Address:9c1017c0-6e05-e7e8-2bd5-034147417eed} {Suffrage:Voter ID:server-23f6fc87-8f95-c9e5-b119-f1456ede73b7 Address:23f6fc87-8f95-c9e5-b119-f1456ede73b7}]" + testing.go:149: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.279Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6629d9bc-8f11-04c3-94ff-d919015e30db Address:6629d9bc-8f11-04c3-94ff-d919015e30db} {Suffrage:Voter ID:server-9c1017c0-6e05-e7e8-2bd5-034147417eed Address:9c1017c0-6e05-e7e8-2bd5-034147417eed} {Suffrage:Voter ID:server-23f6fc87-8f95-c9e5-b119-f1456ede73b7 Address:23f6fc87-8f95-c9e5-b119-f1456ede73b7}]" +testLoggerAdapter verbose: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.279Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: entering follower state: follower="Node at 9c1017c0-6e05-e7e8-2bd5-034147417eed [Follower]" leader= + testing.go:149: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.279Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: entering follower state: follower="Node at 9c1017c0-6e05-e7e8-2bd5-034147417eed [Follower]" leader= +testLoggerAdapter verbose: server-23f6fc87-8f95-c9e5-b119-f1456ede73b7: 2025-02-10T11:34:08.279Z [INFO] server-23f6fc87-8f95-c9e5-b119-f1456ede73b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6629d9bc-8f11-04c3-94ff-d919015e30db Address:6629d9bc-8f11-04c3-94ff-d919015e30db} {Suffrage:Voter ID:server-9c1017c0-6e05-e7e8-2bd5-034147417eed Address:9c1017c0-6e05-e7e8-2bd5-034147417eed} {Suffrage:Voter ID:server-23f6fc87-8f95-c9e5-b119-f1456ede73b7 Address:23f6fc87-8f95-c9e5-b119-f1456ede73b7}]" + testing.go:149: server-23f6fc87-8f95-c9e5-b119-f1456ede73b7: 2025-02-10T11:34:08.279Z [INFO] server-23f6fc87-8f95-c9e5-b119-f1456ede73b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6629d9bc-8f11-04c3-94ff-d919015e30db Address:6629d9bc-8f11-04c3-94ff-d919015e30db} {Suffrage:Voter ID:server-9c1017c0-6e05-e7e8-2bd5-034147417eed Address:9c1017c0-6e05-e7e8-2bd5-034147417eed} {Suffrage:Voter ID:server-23f6fc87-8f95-c9e5-b119-f1456ede73b7 Address:23f6fc87-8f95-c9e5-b119-f1456ede73b7}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:08.279Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:08.279Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-23f6fc87-8f95-c9e5-b119-f1456ede73b7: 2025-02-10T11:34:08.279Z [INFO] server-23f6fc87-8f95-c9e5-b119-f1456ede73b7: entering follower state: follower="Node at 23f6fc87-8f95-c9e5-b119-f1456ede73b7 [Follower]" leader= + testing.go:149: server-23f6fc87-8f95-c9e5-b119-f1456ede73b7: 2025-02-10T11:34:08.279Z [INFO] server-23f6fc87-8f95-c9e5-b119-f1456ede73b7: entering follower state: follower="Node at 23f6fc87-8f95-c9e5-b119-f1456ede73b7 [Follower]" leader= +testLoggerAdapter verbose: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.331Z [WARN] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.331Z [WARN] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.331Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: entering candidate state: node="Node at 9c1017c0-6e05-e7e8-2bd5-034147417eed [Candidate]" term=2 + testing.go:149: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.331Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: entering candidate state: node="Node at 9c1017c0-6e05-e7e8-2bd5-034147417eed [Candidate]" term=2 +testLoggerAdapter verbose: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.331Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: election won: tally=2 + testing.go:149: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.331Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: election won: tally=2 +testLoggerAdapter verbose: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.331Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: entering leader state: leader="Node at 9c1017c0-6e05-e7e8-2bd5-034147417eed [Leader]" + testing.go:149: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.331Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: entering leader state: leader="Node at 9c1017c0-6e05-e7e8-2bd5-034147417eed [Leader]" +testLoggerAdapter verbose: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.331Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: added peer, starting replication: peer=server-6629d9bc-8f11-04c3-94ff-d919015e30db + testing.go:149: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.331Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: added peer, starting replication: peer=server-6629d9bc-8f11-04c3-94ff-d919015e30db +testLoggerAdapter verbose: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.331Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: added peer, starting replication: peer=server-23f6fc87-8f95-c9e5-b119-f1456ede73b7 + testing.go:149: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.331Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: added peer, starting replication: peer=server-23f6fc87-8f95-c9e5-b119-f1456ede73b7 +testLoggerAdapter verbose: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.331Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: pipelining replication: peer="{Voter server-6629d9bc-8f11-04c3-94ff-d919015e30db 6629d9bc-8f11-04c3-94ff-d919015e30db}" + testing.go:149: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.331Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: pipelining replication: peer="{Voter server-6629d9bc-8f11-04c3-94ff-d919015e30db 6629d9bc-8f11-04c3-94ff-d919015e30db}" +testLoggerAdapter verbose: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.331Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: pipelining replication: peer="{Voter server-23f6fc87-8f95-c9e5-b119-f1456ede73b7 23f6fc87-8f95-c9e5-b119-f1456ede73b7}" + testing.go:149: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.331Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: pipelining replication: peer="{Voter server-23f6fc87-8f95-c9e5-b119-f1456ede73b7 23f6fc87-8f95-c9e5-b119-f1456ede73b7}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:08.436Z [INFO] stable state for Follower reached at 2025-02-10 11:34:08.33170517 +0000 UTC m=+4.700609476 (2 nodes), 52.444483ms from start of poll, 52.815261ms from cluster start. Timeout at 2025-02-10 11:34:08.436768206 +0000 UTC m=+4.805672502, 105.063026ms after stability + testing.go:149: cluster: 2025-02-10T11:34:08.436Z [INFO] stable state for Follower reached at 2025-02-10 11:34:08.33170517 +0000 UTC m=+4.700609476 (2 nodes), 52.444483ms from start of poll, 52.815261ms from cluster start. Timeout at 2025-02-10 11:34:08.436768206 +0000 UTC m=+4.805672502, 105.063026ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:08.436Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:08.436Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:08.541Z [INFO] stable state for Leader reached at 2025-02-10 11:34:08.436843117 +0000 UTC m=+4.805747416 (1 nodes), 448ns from start of poll, 157.953201ms from cluster start. Timeout at 2025-02-10 11:34:08.541919964 +0000 UTC m=+4.910824262, 105.076846ms after stability + testing.go:149: cluster: 2025-02-10T11:34:08.541Z [INFO] stable state for Leader reached at 2025-02-10 11:34:08.436843117 +0000 UTC m=+4.805747416 (1 nodes), 448ns from start of poll, 157.953201ms from cluster start. Timeout at 2025-02-10 11:34:08.541919964 +0000 UTC m=+4.910824262, 105.076846ms after stability +testLoggerAdapter verbose: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.552Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: aborting pipeline replication: peer="{Voter server-23f6fc87-8f95-c9e5-b119-f1456ede73b7 23f6fc87-8f95-c9e5-b119-f1456ede73b7}" + testing.go:149: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.552Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: aborting pipeline replication: peer="{Voter server-23f6fc87-8f95-c9e5-b119-f1456ede73b7 23f6fc87-8f95-c9e5-b119-f1456ede73b7}" +testLoggerAdapter verbose: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.552Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: aborting pipeline replication: peer="{Voter server-6629d9bc-8f11-04c3-94ff-d919015e30db 6629d9bc-8f11-04c3-94ff-d919015e30db}" + testing.go:149: server-9c1017c0-6e05-e7e8-2bd5-034147417eed: 2025-02-10T11:34:08.552Z [INFO] server-9c1017c0-6e05-e7e8-2bd5-034147417eed: aborting pipeline replication: peer="{Voter server-6629d9bc-8f11-04c3-94ff-d919015e30db 6629d9bc-8f11-04c3-94ff-d919015e30db}" +--- PASS: TestRaft_TripleNode (0.27s) === RUN TestRaft_LeaderFail -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.039Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d6d0910-06fa-854f-5c8a-d43091411912 Address:4d6d0910-06fa-854f-5c8a-d43091411912} {Suffrage:Voter ID:server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea Address:a20fdaeb-68ed-be57-a26d-07cd322fc2ea} {Suffrage:Voter ID:server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 Address:f251bea8-a509-cbe0-decd-8c0e1d364fa2}]" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.039Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d6d0910-06fa-854f-5c8a-d43091411912 Address:4d6d0910-06fa-854f-5c8a-d43091411912} {Suffrage:Voter ID:server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea Address:a20fdaeb-68ed-be57-a26d-07cd322fc2ea} {Suffrage:Voter ID:server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 Address:f251bea8-a509-cbe0-decd-8c0e1d364fa2}]" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.039Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering follower state: follower="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Follower]" leader= - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.039Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering follower state: follower="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Follower]" leader= -testLoggerAdapter verbose: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.039Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d6d0910-06fa-854f-5c8a-d43091411912 Address:4d6d0910-06fa-854f-5c8a-d43091411912} {Suffrage:Voter ID:server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea Address:a20fdaeb-68ed-be57-a26d-07cd322fc2ea} {Suffrage:Voter ID:server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 Address:f251bea8-a509-cbe0-decd-8c0e1d364fa2}]" - testing.go:149: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.039Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d6d0910-06fa-854f-5c8a-d43091411912 Address:4d6d0910-06fa-854f-5c8a-d43091411912} {Suffrage:Voter ID:server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea Address:a20fdaeb-68ed-be57-a26d-07cd322fc2ea} {Suffrage:Voter ID:server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 Address:f251bea8-a509-cbe0-decd-8c0e1d364fa2}]" -testLoggerAdapter verbose: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.039Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: entering follower state: follower="Node at a20fdaeb-68ed-be57-a26d-07cd322fc2ea [Follower]" leader= - testing.go:149: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.039Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: entering follower state: follower="Node at a20fdaeb-68ed-be57-a26d-07cd322fc2ea [Follower]" leader= -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.039Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d6d0910-06fa-854f-5c8a-d43091411912 Address:4d6d0910-06fa-854f-5c8a-d43091411912} {Suffrage:Voter ID:server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea Address:a20fdaeb-68ed-be57-a26d-07cd322fc2ea} {Suffrage:Voter ID:server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 Address:f251bea8-a509-cbe0-decd-8c0e1d364fa2}]" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.039Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d6d0910-06fa-854f-5c8a-d43091411912 Address:4d6d0910-06fa-854f-5c8a-d43091411912} {Suffrage:Voter ID:server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea Address:a20fdaeb-68ed-be57-a26d-07cd322fc2ea} {Suffrage:Voter ID:server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 Address:f251bea8-a509-cbe0-decd-8c0e1d364fa2}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.039Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:09:54.039Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.039Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering follower state: follower="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Follower]" leader= - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.039Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering follower state: follower="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Follower]" leader= -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=2 - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=2 -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: election won: tally=2 - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: election won: tally=2 -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering leader state: leader="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Leader]" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering leader state: leader="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Leader]" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: added peer, starting replication: peer=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: added peer, starting replication: peer=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.095Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: added peer, starting replication: peer=server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.095Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: added peer, starting replication: peer=server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.095Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: pipelining replication: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.095Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: pipelining replication: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.095Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: pipelining replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.095Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: pipelining replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.206Z [INFO] stable state for Follower reached at 2024-01-09 05:09:54.095399776 +0000 UTC m=+5.348451128 (2 nodes), 55.804982ms from start of poll, 56.360248ms from cluster start. Timeout at 2024-01-09 05:09:54.206333162 +0000 UTC m=+5.459384225, 110.933097ms after stability - testing.go:149: cluster: 2024-01-09T05:09:54.206Z [INFO] stable state for Follower reached at 2024-01-09 05:09:54.095399776 +0000 UTC m=+5.348451128 (2 nodes), 55.804982ms from start of poll, 56.360248ms from cluster start. Timeout at 2024-01-09 05:09:54.206333162 +0000 UTC m=+5.459384225, 110.933097ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.206Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:54.206Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.311Z [INFO] stable state for Leader reached at 2024-01-09 05:09:54.206507028 +0000 UTC m=+5.459558208 (1 nodes), 2.879µs from start of poll, 167.467328ms from cluster start. Timeout at 2024-01-09 05:09:54.311641412 +0000 UTC m=+5.564692589, 105.134381ms after stability - testing.go:149: cluster: 2024-01-09T05:09:54.311Z [INFO] stable state for Leader reached at 2024-01-09 05:09:54.206507028 +0000 UTC m=+5.459558208 (1 nodes), 2.879µs from start of poll, 167.467328ms from cluster start. Timeout at 2024-01-09 05:09:54.311641412 +0000 UTC m=+5.564692589, 105.134381ms after stability - raft_test.go:336: [INFO] Disconnecting Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Leader] -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.329Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=a20fdaeb-68ed-be57-a26d-07cd322fc2ea error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.329Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=a20fdaeb-68ed-be57-a26d-07cd322fc2ea error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.331Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to pipeline appendEntries: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="append pipeline closed" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.331Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to pipeline appendEntries: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="append pipeline closed" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.331Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: aborting pipeline replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.331Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: aborting pipeline replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.331Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=f251bea8-a509-cbe0-decd-8c0e1d364fa2 error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.331Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=f251bea8-a509-cbe0-decd-8c0e1d364fa2 error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.332Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to pipeline appendEntries: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="append pipeline closed" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.332Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to pipeline appendEntries: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="append pipeline closed" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.332Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: aborting pipeline replication: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.332Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: aborting pipeline replication: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.333Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:54.333Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.341Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.341Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.341Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.341Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.347Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=f251bea8-a509-cbe0-decd-8c0e1d364fa2 error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.347Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=f251bea8-a509-cbe0-decd-8c0e1d364fa2 error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.348Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=a20fdaeb-68ed-be57-a26d-07cd322fc2ea error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.348Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=a20fdaeb-68ed-be57-a26d-07cd322fc2ea error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.357Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.357Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.358Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.358Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.363Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=f251bea8-a509-cbe0-decd-8c0e1d364fa2 error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.363Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=f251bea8-a509-cbe0-decd-8c0e1d364fa2 error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.368Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=a20fdaeb-68ed-be57-a26d-07cd322fc2ea error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.368Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=a20fdaeb-68ed-be57-a26d-07cd322fc2ea error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.373Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to contact: server-id=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea time=50.064143ms - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.373Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to contact: server-id=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea time=50.064143ms -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.376Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.376Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.377Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.377Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.383Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to contact: server-id=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea time=60.37515ms - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.383Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to contact: server-id=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea time=60.37515ms -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.383Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to contact: server-id=server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 time=59.344272ms - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.383Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to contact: server-id=server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 time=59.344272ms -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.384Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to contact quorum of nodes, stepping down - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.384Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to contact quorum of nodes, stepping down -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.384Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering follower state: follower="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Follower]" leader= - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.384Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering follower state: follower="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Follower]" leader= -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.392Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=f251bea8-a509-cbe0-decd-8c0e1d364fa2 error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.392Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=f251bea8-a509-cbe0-decd-8c0e1d364fa2 error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" -testLoggerAdapter verbose: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.396Z [WARN] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: heartbeat timeout reached, starting election: last-leader=4d6d0910-06fa-854f-5c8a-d43091411912 - testing.go:149: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.396Z [WARN] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: heartbeat timeout reached, starting election: last-leader=4d6d0910-06fa-854f-5c8a-d43091411912 -testLoggerAdapter verbose: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.396Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: entering candidate state: node="Node at a20fdaeb-68ed-be57-a26d-07cd322fc2ea [Candidate]" term=3 - testing.go:149: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.396Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: entering candidate state: node="Node at a20fdaeb-68ed-be57-a26d-07cd322fc2ea [Candidate]" term=3 -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.396Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: rejecting vote request since we have a leader: from=a20fdaeb-68ed-be57-a26d-07cd322fc2ea leader=4d6d0910-06fa-854f-5c8a-d43091411912 - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.396Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: rejecting vote request since we have a leader: from=a20fdaeb-68ed-be57-a26d-07cd322fc2ea leader=4d6d0910-06fa-854f-5c8a-d43091411912 -testLoggerAdapter verbose: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.397Z [ERROR] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: failed to make requestVote RPC: target="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" - testing.go:149: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.397Z [ERROR] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: failed to make requestVote RPC: target="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.398Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=a20fdaeb-68ed-be57-a26d-07cd322fc2ea error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.398Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=a20fdaeb-68ed-be57-a26d-07cd322fc2ea error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.403Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.403Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.404Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.404Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.409Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: heartbeat timeout reached, starting election: last-leader=4d6d0910-06fa-854f-5c8a-d43091411912 - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.409Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: heartbeat timeout reached, starting election: last-leader=4d6d0910-06fa-854f-5c8a-d43091411912 -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.409Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering candidate state: node="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Candidate]" term=3 - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.409Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering candidate state: node="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Candidate]" term=3 -testLoggerAdapter verbose: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.409Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: duplicate requestVote for same term: term=3 - testing.go:149: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.409Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: duplicate requestVote for same term: term=3 -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.409Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to make requestVote RPC: target="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.409Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to make requestVote RPC: target="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.452Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.452Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.452Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=3 - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.452Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=3 -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.452Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.452Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.452Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.452Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: Election timeout reached, restarting election - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: Election timeout reached, restarting election -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering candidate state: node="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Candidate]" term=4 - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering candidate state: node="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Candidate]" term=4 -testLoggerAdapter verbose: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.473Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: entering follower state: follower="Node at a20fdaeb-68ed-be57-a26d-07cd322fc2ea [Follower]" leader= - testing.go:149: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.473Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: entering follower state: follower="Node at a20fdaeb-68ed-be57-a26d-07cd322fc2ea [Follower]" leader= -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: election won: tally=2 - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: election won: tally=2 -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering leader state: leader="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Leader]" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering leader state: leader="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Leader]" -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: added peer, starting replication: peer=server-4d6d0910-06fa-854f-5c8a-d43091411912 - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: added peer, starting replication: peer=server-4d6d0910-06fa-854f-5c8a-d43091411912 -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: added peer, starting replication: peer=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: added peer, starting replication: peer=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: pipelining replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: pipelining replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to make requestVote RPC: target="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to make requestVote RPC: target="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to appendEntries to: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to appendEntries to: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.482Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to heartbeat to: peer=4d6d0910-06fa-854f-5c8a-d43091411912 error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.482Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to heartbeat to: peer=4d6d0910-06fa-854f-5c8a-d43091411912 error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.489Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to appendEntries to: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.489Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to appendEntries to: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.502Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to heartbeat to: peer=4d6d0910-06fa-854f-5c8a-d43091411912 error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.502Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to heartbeat to: peer=4d6d0910-06fa-854f-5c8a-d43091411912 error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.507Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to appendEntries to: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.507Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to appendEntries to: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.520Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to heartbeat to: peer=4d6d0910-06fa-854f-5c8a-d43091411912 error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.520Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to heartbeat to: peer=4d6d0910-06fa-854f-5c8a-d43091411912 error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.521Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: Election timeout reached, restarting election - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.521Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: Election timeout reached, restarting election -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.521Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=4 - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.521Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=4 -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.521Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.521Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.521Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.521Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.523Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to contact: server-id=server-4d6d0910-06fa-854f-5c8a-d43091411912 time=50.175494ms - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.523Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to contact: server-id=server-4d6d0910-06fa-854f-5c8a-d43091411912 time=50.175494ms -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.533Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to appendEntries to: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.533Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to appendEntries to: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.550Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to heartbeat to: peer=4d6d0910-06fa-854f-5c8a-d43091411912 error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.550Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to heartbeat to: peer=4d6d0910-06fa-854f-5c8a-d43091411912 error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.571Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: Election timeout reached, restarting election - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.571Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: Election timeout reached, restarting election -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.571Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=5 - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.571Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=5 -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.571Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to contact: server-id=server-4d6d0910-06fa-854f-5c8a-d43091411912 time=98.233819ms - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.571Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to contact: server-id=server-4d6d0910-06fa-854f-5c8a-d43091411912 time=98.233819ms -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.571Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.571Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.572Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.572Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.578Z [INFO] stable state for Leader reached at 2024-01-09 05:09:54.473660943 +0000 UTC m=+5.726712308 (1 nodes), 140.223516ms from start of poll, 434.621428ms from cluster start. Timeout at 2024-01-09 05:09:54.578682913 +0000 UTC m=+5.831733967, 105.021659ms after stability - testing.go:149: cluster: 2024-01-09T05:09:54.578Z [INFO] stable state for Leader reached at 2024-01-09 05:09:54.473660943 +0000 UTC m=+5.726712308 (1 nodes), 140.223516ms from start of poll, 434.621428ms from cluster start. Timeout at 2024-01-09 05:09:54.578682913 +0000 UTC m=+5.831733967, 105.021659ms after stability - raft_test.go:371: [INFO] Reconnecting Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate] -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.580Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: peer has newer term, stopping replication: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.580Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: peer has newer term, stopping replication: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.580Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering follower state: follower="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Follower]" leader= - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.580Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering follower state: follower="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Follower]" leader= -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.580Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: aborting pipeline replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.580Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: aborting pipeline replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.621Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: Election timeout reached, restarting election - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.621Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: Election timeout reached, restarting election -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.621Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=6 - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.621Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=6 -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.622Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: rejecting vote request since our last term is greater: candidate=4d6d0910-06fa-854f-5c8a-d43091411912 last-term=4 last-candidate-term=2 - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.622Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: rejecting vote request since our last term is greater: candidate=4d6d0910-06fa-854f-5c8a-d43091411912 last-term=4 last-candidate-term=2 -testLoggerAdapter verbose: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.622Z [WARN] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: rejecting vote request since we have a leader: from=4d6d0910-06fa-854f-5c8a-d43091411912 leader=f251bea8-a509-cbe0-decd-8c0e1d364fa2 - testing.go:149: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.622Z [WARN] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: rejecting vote request since we have a leader: from=4d6d0910-06fa-854f-5c8a-d43091411912 leader=f251bea8-a509-cbe0-decd-8c0e1d364fa2 -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering candidate state: node="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Candidate]" term=7 - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering candidate state: node="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Candidate]" term=7 -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: election won: tally=2 - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: election won: tally=2 -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering leader state: leader="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Leader]" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering leader state: leader="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Leader]" -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: added peer, starting replication: peer=server-4d6d0910-06fa-854f-5c8a-d43091411912 - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: added peer, starting replication: peer=server-4d6d0910-06fa-854f-5c8a-d43091411912 -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: added peer, starting replication: peer=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: added peer, starting replication: peer=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.643Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: pipelining replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.643Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: pipelining replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.643Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering follower state: follower="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Follower]" leader= - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.643Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering follower state: follower="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Follower]" leader= -testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.643Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to get previous log: previous-index=5 last-index=3 error="log not found" - testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.643Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to get previous log: previous-index=5 last-index=3 error="log not found" -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.643Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: appendEntries rejected, sending older logs: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" next=4 - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.643Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: appendEntries rejected, sending older logs: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" next=4 -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.643Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: pipelining replication: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.643Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: pipelining replication: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.648Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: aborting pipeline replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.648Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: aborting pipeline replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" -testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.648Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: aborting pipeline replication: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" - testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.648Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: aborting pipeline replication: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" ---- PASS: TestRaft_LeaderFail (0.61s) +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:08.553Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 Address:6fe55f11-8fc1-2744-7e49-1a80f4465bb7} {Suffrage:Voter ID:server-d697a267-1dcb-1c34-3e2b-52f10a826d65 Address:d697a267-1dcb-1c34-3e2b-52f10a826d65} {Suffrage:Voter ID:server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 Address:f9c2c13d-e823-6e25-0441-a5b28ca599a7}]" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:08.553Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 Address:6fe55f11-8fc1-2744-7e49-1a80f4465bb7} {Suffrage:Voter ID:server-d697a267-1dcb-1c34-3e2b-52f10a826d65 Address:d697a267-1dcb-1c34-3e2b-52f10a826d65} {Suffrage:Voter ID:server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 Address:f9c2c13d-e823-6e25-0441-a5b28ca599a7}]" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:08.553Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: entering follower state: follower="Node at 6fe55f11-8fc1-2744-7e49-1a80f4465bb7 [Follower]" leader= + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:08.553Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: entering follower state: follower="Node at 6fe55f11-8fc1-2744-7e49-1a80f4465bb7 [Follower]" leader= +testLoggerAdapter verbose: server-d697a267-1dcb-1c34-3e2b-52f10a826d65: 2025-02-10T11:34:08.553Z [INFO] server-d697a267-1dcb-1c34-3e2b-52f10a826d65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 Address:6fe55f11-8fc1-2744-7e49-1a80f4465bb7} {Suffrage:Voter ID:server-d697a267-1dcb-1c34-3e2b-52f10a826d65 Address:d697a267-1dcb-1c34-3e2b-52f10a826d65} {Suffrage:Voter ID:server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 Address:f9c2c13d-e823-6e25-0441-a5b28ca599a7}]" + testing.go:149: server-d697a267-1dcb-1c34-3e2b-52f10a826d65: 2025-02-10T11:34:08.553Z [INFO] server-d697a267-1dcb-1c34-3e2b-52f10a826d65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 Address:6fe55f11-8fc1-2744-7e49-1a80f4465bb7} {Suffrage:Voter ID:server-d697a267-1dcb-1c34-3e2b-52f10a826d65 Address:d697a267-1dcb-1c34-3e2b-52f10a826d65} {Suffrage:Voter ID:server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 Address:f9c2c13d-e823-6e25-0441-a5b28ca599a7}]" +testLoggerAdapter verbose: server-d697a267-1dcb-1c34-3e2b-52f10a826d65: 2025-02-10T11:34:08.553Z [INFO] server-d697a267-1dcb-1c34-3e2b-52f10a826d65: entering follower state: follower="Node at d697a267-1dcb-1c34-3e2b-52f10a826d65 [Follower]" leader= + testing.go:149: server-d697a267-1dcb-1c34-3e2b-52f10a826d65: 2025-02-10T11:34:08.553Z [INFO] server-d697a267-1dcb-1c34-3e2b-52f10a826d65: entering follower state: follower="Node at d697a267-1dcb-1c34-3e2b-52f10a826d65 [Follower]" leader= +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.553Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 Address:6fe55f11-8fc1-2744-7e49-1a80f4465bb7} {Suffrage:Voter ID:server-d697a267-1dcb-1c34-3e2b-52f10a826d65 Address:d697a267-1dcb-1c34-3e2b-52f10a826d65} {Suffrage:Voter ID:server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 Address:f9c2c13d-e823-6e25-0441-a5b28ca599a7}]" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.553Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 Address:6fe55f11-8fc1-2744-7e49-1a80f4465bb7} {Suffrage:Voter ID:server-d697a267-1dcb-1c34-3e2b-52f10a826d65 Address:d697a267-1dcb-1c34-3e2b-52f10a826d65} {Suffrage:Voter ID:server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 Address:f9c2c13d-e823-6e25-0441-a5b28ca599a7}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:08.553Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:08.553Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.553Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering follower state: follower="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Follower]" leader= + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.553Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering follower state: follower="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Follower]" leader= +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.616Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.616Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.616Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering candidate state: node="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Candidate]" term=2 + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.616Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering candidate state: node="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Candidate]" term=2 +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.616Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: election won: tally=2 + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.616Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: election won: tally=2 +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.616Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering leader state: leader="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Leader]" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.616Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering leader state: leader="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Leader]" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.616Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: added peer, starting replication: peer=server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.616Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: added peer, starting replication: peer=server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.616Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: added peer, starting replication: peer=server-d697a267-1dcb-1c34-3e2b-52f10a826d65 + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.616Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: added peer, starting replication: peer=server-d697a267-1dcb-1c34-3e2b-52f10a826d65 +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.616Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: pipelining replication: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.616Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: pipelining replication: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.616Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: pipelining replication: peer="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.616Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: pipelining replication: peer="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:08.721Z [INFO] stable state for Follower reached at 2025-02-10 11:34:08.616705176 +0000 UTC m=+4.985609484 (2 nodes), 63.163551ms from start of poll, 63.54209ms from cluster start. Timeout at 2025-02-10 11:34:08.721782017 +0000 UTC m=+5.090686313, 105.076829ms after stability + testing.go:149: cluster: 2025-02-10T11:34:08.721Z [INFO] stable state for Follower reached at 2025-02-10 11:34:08.616705176 +0000 UTC m=+4.985609484 (2 nodes), 63.163551ms from start of poll, 63.54209ms from cluster start. Timeout at 2025-02-10 11:34:08.721782017 +0000 UTC m=+5.090686313, 105.076829ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:08.721Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:08.721Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:08.826Z [INFO] stable state for Leader reached at 2025-02-10 11:34:08.721846737 +0000 UTC m=+5.090751038 (1 nodes), 449ns from start of poll, 168.683644ms from cluster start. Timeout at 2025-02-10 11:34:08.82695071 +0000 UTC m=+5.195855060, 105.104022ms after stability + testing.go:149: cluster: 2025-02-10T11:34:08.826Z [INFO] stable state for Leader reached at 2025-02-10 11:34:08.721846737 +0000 UTC m=+5.090751038 (1 nodes), 449ns from start of poll, 168.683644ms from cluster start. Timeout at 2025-02-10 11:34:08.82695071 +0000 UTC m=+5.195855060, 105.104022ms after stability + raft_test.go:336: [INFO] Disconnecting Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Leader] +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.837Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to heartbeat to: peer=6fe55f11-8fc1-2744-7e49-1a80f4465bb7 error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.837Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to heartbeat to: peer=6fe55f11-8fc1-2744-7e49-1a80f4465bb7 error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.841Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to heartbeat to: peer=d697a267-1dcb-1c34-3e2b-52f10a826d65 error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.841Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to heartbeat to: peer=d697a267-1dcb-1c34-3e2b-52f10a826d65 error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.842Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to pipeline appendEntries: peer="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" error="append pipeline closed" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.842Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to pipeline appendEntries: peer="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" error="append pipeline closed" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.842Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: aborting pipeline replication: peer="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.842Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: aborting pipeline replication: peer="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:08.842Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:08.842Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.844Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to pipeline appendEntries: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" error="append pipeline closed" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.844Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to pipeline appendEntries: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" error="append pipeline closed" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.844Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: aborting pipeline replication: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.844Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: aborting pipeline replication: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.850Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to appendEntries to: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.850Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to appendEntries to: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.851Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to appendEntries to: peer="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.851Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to appendEntries to: peer="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.858Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to heartbeat to: peer=6fe55f11-8fc1-2744-7e49-1a80f4465bb7 error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.858Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to heartbeat to: peer=6fe55f11-8fc1-2744-7e49-1a80f4465bb7 error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.858Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to heartbeat to: peer=d697a267-1dcb-1c34-3e2b-52f10a826d65 error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.858Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to heartbeat to: peer=d697a267-1dcb-1c34-3e2b-52f10a826d65 error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.866Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to appendEntries to: peer="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.866Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to appendEntries to: peer="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.869Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to appendEntries to: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.869Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to appendEntries to: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.874Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to heartbeat to: peer=d697a267-1dcb-1c34-3e2b-52f10a826d65 error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.874Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to heartbeat to: peer=d697a267-1dcb-1c34-3e2b-52f10a826d65 error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.875Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to heartbeat to: peer=6fe55f11-8fc1-2744-7e49-1a80f4465bb7 error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.875Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to heartbeat to: peer=6fe55f11-8fc1-2744-7e49-1a80f4465bb7 error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.884Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to contact: server-id=server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 time=50.107459ms + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.884Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to contact: server-id=server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 time=50.107459ms +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.885Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to appendEntries to: peer="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.885Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to appendEntries to: peer="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.887Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to appendEntries to: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.887Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to appendEntries to: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.894Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to contact: server-id=server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 time=60.204678ms + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.894Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to contact: server-id=server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 time=60.204678ms +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.894Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to contact: server-id=server-d697a267-1dcb-1c34-3e2b-52f10a826d65 time=59.270563ms + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.894Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to contact: server-id=server-d697a267-1dcb-1c34-3e2b-52f10a826d65 time=59.270563ms +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.894Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to contact quorum of nodes, stepping down + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.894Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to contact quorum of nodes, stepping down +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.894Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering follower state: follower="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Follower]" leader= + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.894Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering follower state: follower="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Follower]" leader= +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.902Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to heartbeat to: peer=6fe55f11-8fc1-2744-7e49-1a80f4465bb7 error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.902Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to heartbeat to: peer=6fe55f11-8fc1-2744-7e49-1a80f4465bb7 error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.903Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to heartbeat to: peer=d697a267-1dcb-1c34-3e2b-52f10a826d65 error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.903Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to heartbeat to: peer=d697a267-1dcb-1c34-3e2b-52f10a826d65 error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.911Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to appendEntries to: peer="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.911Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to appendEntries to: peer="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.914Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to appendEntries to: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.914Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to appendEntries to: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:08.924Z [WARN] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: heartbeat timeout reached, starting election: last-leader=f9c2c13d-e823-6e25-0441-a5b28ca599a7 + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:08.924Z [WARN] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: heartbeat timeout reached, starting election: last-leader=f9c2c13d-e823-6e25-0441-a5b28ca599a7 +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:08.924Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: entering candidate state: node="Node at 6fe55f11-8fc1-2744-7e49-1a80f4465bb7 [Candidate]" term=3 + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:08.924Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: entering candidate state: node="Node at 6fe55f11-8fc1-2744-7e49-1a80f4465bb7 [Candidate]" term=3 +testLoggerAdapter verbose: server-d697a267-1dcb-1c34-3e2b-52f10a826d65: 2025-02-10T11:34:08.924Z [WARN] server-d697a267-1dcb-1c34-3e2b-52f10a826d65: rejecting vote request since we have a leader: from=6fe55f11-8fc1-2744-7e49-1a80f4465bb7 leader=f9c2c13d-e823-6e25-0441-a5b28ca599a7 +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:08.924Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to make requestVote RPC: target="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" + testing.go:149: server-d697a267-1dcb-1c34-3e2b-52f10a826d65: 2025-02-10T11:34:08.924Z [WARN] server-d697a267-1dcb-1c34-3e2b-52f10a826d65: rejecting vote request since we have a leader: from=6fe55f11-8fc1-2744-7e49-1a80f4465bb7 leader=f9c2c13d-e823-6e25-0441-a5b28ca599a7 + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:08.924Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to make requestVote RPC: target="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" +testLoggerAdapter verbose: server-d697a267-1dcb-1c34-3e2b-52f10a826d65: 2025-02-10T11:34:08.946Z [WARN] server-d697a267-1dcb-1c34-3e2b-52f10a826d65: heartbeat timeout reached, starting election: last-leader=f9c2c13d-e823-6e25-0441-a5b28ca599a7 + testing.go:149: server-d697a267-1dcb-1c34-3e2b-52f10a826d65: 2025-02-10T11:34:08.946Z [WARN] server-d697a267-1dcb-1c34-3e2b-52f10a826d65: heartbeat timeout reached, starting election: last-leader=f9c2c13d-e823-6e25-0441-a5b28ca599a7 +testLoggerAdapter verbose: server-d697a267-1dcb-1c34-3e2b-52f10a826d65: 2025-02-10T11:34:08.946Z [INFO] server-d697a267-1dcb-1c34-3e2b-52f10a826d65: entering candidate state: node="Node at d697a267-1dcb-1c34-3e2b-52f10a826d65 [Candidate]" term=3 + testing.go:149: server-d697a267-1dcb-1c34-3e2b-52f10a826d65: 2025-02-10T11:34:08.946Z [INFO] server-d697a267-1dcb-1c34-3e2b-52f10a826d65: entering candidate state: node="Node at d697a267-1dcb-1c34-3e2b-52f10a826d65 [Candidate]" term=3 +testLoggerAdapter verbose: server-d697a267-1dcb-1c34-3e2b-52f10a826d65: 2025-02-10T11:34:08.946Z [ERROR] server-d697a267-1dcb-1c34-3e2b-52f10a826d65: failed to make requestVote RPC: target="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" + testing.go:149: server-d697a267-1dcb-1c34-3e2b-52f10a826d65: 2025-02-10T11:34:08.946Z [ERROR] server-d697a267-1dcb-1c34-3e2b-52f10a826d65: failed to make requestVote RPC: target="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:08.946Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: duplicate requestVote for same term: term=3 + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:08.946Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: duplicate requestVote for same term: term=3 +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.954Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.954Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.954Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering candidate state: node="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Candidate]" term=3 + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.954Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering candidate state: node="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Candidate]" term=3 +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.954Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to make requestVote RPC: target="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.954Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to make requestVote RPC: target="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.954Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to make requestVote RPC: target="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:08.954Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to make requestVote RPC: target="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [WARN] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: Election timeout reached, restarting election + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [WARN] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: Election timeout reached, restarting election +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: entering candidate state: node="Node at 6fe55f11-8fc1-2744-7e49-1a80f4465bb7 [Candidate]" term=4 + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: entering candidate state: node="Node at 6fe55f11-8fc1-2744-7e49-1a80f4465bb7 [Candidate]" term=4 +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to make requestVote RPC: target="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to make requestVote RPC: target="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" +testLoggerAdapter verbose: server-d697a267-1dcb-1c34-3e2b-52f10a826d65: 2025-02-10T11:34:09.020Z [INFO] server-d697a267-1dcb-1c34-3e2b-52f10a826d65: entering follower state: follower="Node at d697a267-1dcb-1c34-3e2b-52f10a826d65 [Follower]" leader= +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: election won: tally=2 + testing.go:149: server-d697a267-1dcb-1c34-3e2b-52f10a826d65: 2025-02-10T11:34:09.020Z [INFO] server-d697a267-1dcb-1c34-3e2b-52f10a826d65: entering follower state: follower="Node at d697a267-1dcb-1c34-3e2b-52f10a826d65 [Follower]" leader= + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: election won: tally=2 +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: entering leader state: leader="Node at 6fe55f11-8fc1-2744-7e49-1a80f4465bb7 [Leader]" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: entering leader state: leader="Node at 6fe55f11-8fc1-2744-7e49-1a80f4465bb7 [Leader]" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: added peer, starting replication: peer=server-d697a267-1dcb-1c34-3e2b-52f10a826d65 + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: added peer, starting replication: peer=server-d697a267-1dcb-1c34-3e2b-52f10a826d65 +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: added peer, starting replication: peer=server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: added peer, starting replication: peer=server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to appendEntries to: peer="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to appendEntries to: peer="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: pipelining replication: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.020Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: pipelining replication: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.027Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to heartbeat to: peer=f9c2c13d-e823-6e25-0441-a5b28ca599a7 error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.027Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to heartbeat to: peer=f9c2c13d-e823-6e25-0441-a5b28ca599a7 error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.039Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to appendEntries to: peer="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.039Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to appendEntries to: peer="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.044Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to heartbeat to: peer=f9c2c13d-e823-6e25-0441-a5b28ca599a7 error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.044Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to heartbeat to: peer=f9c2c13d-e823-6e25-0441-a5b28ca599a7 error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.051Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: Election timeout reached, restarting election + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.051Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: Election timeout reached, restarting election +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.051Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering candidate state: node="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Candidate]" term=4 + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.051Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering candidate state: node="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Candidate]" term=4 +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.051Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to make requestVote RPC: target="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.051Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to make requestVote RPC: target="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.052Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to make requestVote RPC: target="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.052Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to make requestVote RPC: target="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.057Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to appendEntries to: peer="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.057Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to appendEntries to: peer="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.063Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to heartbeat to: peer=f9c2c13d-e823-6e25-0441-a5b28ca599a7 error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.063Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to heartbeat to: peer=f9c2c13d-e823-6e25-0441-a5b28ca599a7 error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.070Z [WARN] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to contact: server-id=server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 time=50.096341ms + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.070Z [WARN] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to contact: server-id=server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 time=50.096341ms +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.084Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to appendEntries to: peer="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.084Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to appendEntries to: peer="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.090Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to heartbeat to: peer=f9c2c13d-e823-6e25-0441-a5b28ca599a7 error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.090Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to heartbeat to: peer=f9c2c13d-e823-6e25-0441-a5b28ca599a7 error="failed to connect to peer: f9c2c13d-e823-6e25-0441-a5b28ca599a7" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.103Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: Election timeout reached, restarting election + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.103Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: Election timeout reached, restarting election +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.103Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering candidate state: node="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Candidate]" term=5 + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.103Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering candidate state: node="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Candidate]" term=5 +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.103Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to make requestVote RPC: target="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.103Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to make requestVote RPC: target="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" error="failed to connect to peer: d697a267-1dcb-1c34-3e2b-52f10a826d65" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.103Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to make requestVote RPC: target="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.103Z [ERROR] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to make requestVote RPC: target="{Voter server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7 6fe55f11-8fc1-2744-7e49-1a80f4465bb7}" error="failed to connect to peer: 6fe55f11-8fc1-2744-7e49-1a80f4465bb7" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.119Z [WARN] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to contact: server-id=server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 time=98.618623ms + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.119Z [WARN] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: failed to contact: server-id=server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 time=98.618623ms +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:09.126Z [INFO] stable state for Leader reached at 2025-02-10 11:34:09.020915639 +0000 UTC m=+5.389819941 (1 nodes), 178.306236ms from start of poll, 467.752547ms from cluster start. Timeout at 2025-02-10 11:34:09.126021829 +0000 UTC m=+5.494926182, 105.106241ms after stability + testing.go:149: cluster: 2025-02-10T11:34:09.126Z [INFO] stable state for Leader reached at 2025-02-10 11:34:09.020915639 +0000 UTC m=+5.389819941 (1 nodes), 178.306236ms from start of poll, 467.752547ms from cluster start. Timeout at 2025-02-10 11:34:09.126021829 +0000 UTC m=+5.494926182, 105.106241ms after stability + raft_test.go:371: [INFO] Reconnecting Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Candidate] +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.131Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: peer has newer term, stopping replication: peer="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.131Z [ERROR] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: peer has newer term, stopping replication: peer="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.131Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: entering follower state: follower="Node at 6fe55f11-8fc1-2744-7e49-1a80f4465bb7 [Follower]" leader= + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.131Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: entering follower state: follower="Node at 6fe55f11-8fc1-2744-7e49-1a80f4465bb7 [Follower]" leader= +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.131Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: aborting pipeline replication: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.131Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: aborting pipeline replication: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.163Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: Election timeout reached, restarting election + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.163Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: Election timeout reached, restarting election +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.163Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering candidate state: node="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Candidate]" term=6 + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.163Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering candidate state: node="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Candidate]" term=6 +testLoggerAdapter verbose: server-d697a267-1dcb-1c34-3e2b-52f10a826d65: 2025-02-10T11:34:09.163Z [WARN] server-d697a267-1dcb-1c34-3e2b-52f10a826d65: rejecting vote request since we have a leader: from=f9c2c13d-e823-6e25-0441-a5b28ca599a7 leader=6fe55f11-8fc1-2744-7e49-1a80f4465bb7 +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.163Z [WARN] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: rejecting vote request since our last term is greater: candidate=f9c2c13d-e823-6e25-0441-a5b28ca599a7 last-term=4 last-candidate-term=2 + testing.go:149: server-d697a267-1dcb-1c34-3e2b-52f10a826d65: 2025-02-10T11:34:09.163Z [WARN] server-d697a267-1dcb-1c34-3e2b-52f10a826d65: rejecting vote request since we have a leader: from=f9c2c13d-e823-6e25-0441-a5b28ca599a7 leader=6fe55f11-8fc1-2744-7e49-1a80f4465bb7 + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.163Z [WARN] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: rejecting vote request since our last term is greater: candidate=f9c2c13d-e823-6e25-0441-a5b28ca599a7 last-term=4 last-candidate-term=2 +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.204Z [WARN] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.204Z [WARN] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.204Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: entering candidate state: node="Node at 6fe55f11-8fc1-2744-7e49-1a80f4465bb7 [Candidate]" term=7 + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.204Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: entering candidate state: node="Node at 6fe55f11-8fc1-2744-7e49-1a80f4465bb7 [Candidate]" term=7 +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.204Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: election won: tally=2 + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.204Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: election won: tally=2 +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.204Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: entering leader state: leader="Node at 6fe55f11-8fc1-2744-7e49-1a80f4465bb7 [Leader]" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.204Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: entering leader state: leader="Node at 6fe55f11-8fc1-2744-7e49-1a80f4465bb7 [Leader]" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.204Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: added peer, starting replication: peer=server-d697a267-1dcb-1c34-3e2b-52f10a826d65 + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.204Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: added peer, starting replication: peer=server-d697a267-1dcb-1c34-3e2b-52f10a826d65 +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.204Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: added peer, starting replication: peer=server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.204Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering follower state: follower="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Follower]" leader= + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.204Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: added peer, starting replication: peer=server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.204Z [INFO] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: entering follower state: follower="Node at f9c2c13d-e823-6e25-0441-a5b28ca599a7 [Follower]" leader= +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.204Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: pipelining replication: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" +testLoggerAdapter verbose: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.204Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to get previous log: previous-index=5 last-index=3 error="log not found" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.204Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: pipelining replication: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" + testing.go:149: server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: 2025-02-10T11:34:09.204Z [WARN] server-f9c2c13d-e823-6e25-0441-a5b28ca599a7: failed to get previous log: previous-index=5 last-index=3 error="log not found" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.205Z [WARN] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: appendEntries rejected, sending older logs: peer="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" next=4 + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.205Z [WARN] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: appendEntries rejected, sending older logs: peer="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" next=4 +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.205Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: pipelining replication: peer="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.205Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: pipelining replication: peer="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.210Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: aborting pipeline replication: peer="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.210Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: aborting pipeline replication: peer="{Voter server-f9c2c13d-e823-6e25-0441-a5b28ca599a7 f9c2c13d-e823-6e25-0441-a5b28ca599a7}" +testLoggerAdapter verbose: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.210Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: aborting pipeline replication: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" + testing.go:149: server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: 2025-02-10T11:34:09.210Z [INFO] server-6fe55f11-8fc1-2744-7e49-1a80f4465bb7: aborting pipeline replication: peer="{Voter server-d697a267-1dcb-1c34-3e2b-52f10a826d65 d697a267-1dcb-1c34-3e2b-52f10a826d65}" +--- PASS: TestRaft_LeaderFail (0.66s) === RUN TestRaft_BehindFollower -testLoggerAdapter verbose: server-49c4da0a-d151-c41c-77a6-4eef96aaf71f: 2024-01-09T05:09:54.650Z [INFO] server-49c4da0a-d151-c41c-77a6-4eef96aaf71f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49c4da0a-d151-c41c-77a6-4eef96aaf71f Address:49c4da0a-d151-c41c-77a6-4eef96aaf71f} {Suffrage:Voter ID:server-27bbf3ad-bb6e-b691-d788-e28474238269 Address:27bbf3ad-bb6e-b691-d788-e28474238269} {Suffrage:Voter ID:server-63264def-4c66-c838-4ee0-f04dbd85cbd3 Address:63264def-4c66-c838-4ee0-f04dbd85cbd3}]" - testing.go:149: server-49c4da0a-d151-c41c-77a6-4eef96aaf71f: 2024-01-09T05:09:54.650Z [INFO] server-49c4da0a-d151-c41c-77a6-4eef96aaf71f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49c4da0a-d151-c41c-77a6-4eef96aaf71f Address:49c4da0a-d151-c41c-77a6-4eef96aaf71f} {Suffrage:Voter ID:server-27bbf3ad-bb6e-b691-d788-e28474238269 Address:27bbf3ad-bb6e-b691-d788-e28474238269} {Suffrage:Voter ID:server-63264def-4c66-c838-4ee0-f04dbd85cbd3 Address:63264def-4c66-c838-4ee0-f04dbd85cbd3}]" -testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.650Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49c4da0a-d151-c41c-77a6-4eef96aaf71f Address:49c4da0a-d151-c41c-77a6-4eef96aaf71f} {Suffrage:Voter ID:server-27bbf3ad-bb6e-b691-d788-e28474238269 Address:27bbf3ad-bb6e-b691-d788-e28474238269} {Suffrage:Voter ID:server-63264def-4c66-c838-4ee0-f04dbd85cbd3 Address:63264def-4c66-c838-4ee0-f04dbd85cbd3}]" - testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.650Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49c4da0a-d151-c41c-77a6-4eef96aaf71f Address:49c4da0a-d151-c41c-77a6-4eef96aaf71f} {Suffrage:Voter ID:server-27bbf3ad-bb6e-b691-d788-e28474238269 Address:27bbf3ad-bb6e-b691-d788-e28474238269} {Suffrage:Voter ID:server-63264def-4c66-c838-4ee0-f04dbd85cbd3 Address:63264def-4c66-c838-4ee0-f04dbd85cbd3}]" -testLoggerAdapter verbose: server-63264def-4c66-c838-4ee0-f04dbd85cbd3: 2024-01-09T05:09:54.650Z [INFO] server-63264def-4c66-c838-4ee0-f04dbd85cbd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49c4da0a-d151-c41c-77a6-4eef96aaf71f Address:49c4da0a-d151-c41c-77a6-4eef96aaf71f} {Suffrage:Voter ID:server-27bbf3ad-bb6e-b691-d788-e28474238269 Address:27bbf3ad-bb6e-b691-d788-e28474238269} {Suffrage:Voter ID:server-63264def-4c66-c838-4ee0-f04dbd85cbd3 Address:63264def-4c66-c838-4ee0-f04dbd85cbd3}]" - testing.go:149: server-63264def-4c66-c838-4ee0-f04dbd85cbd3: 2024-01-09T05:09:54.650Z [INFO] server-63264def-4c66-c838-4ee0-f04dbd85cbd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49c4da0a-d151-c41c-77a6-4eef96aaf71f Address:49c4da0a-d151-c41c-77a6-4eef96aaf71f} {Suffrage:Voter ID:server-27bbf3ad-bb6e-b691-d788-e28474238269 Address:27bbf3ad-bb6e-b691-d788-e28474238269} {Suffrage:Voter ID:server-63264def-4c66-c838-4ee0-f04dbd85cbd3 Address:63264def-4c66-c838-4ee0-f04dbd85cbd3}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.650Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:54.650Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-49c4da0a-d151-c41c-77a6-4eef96aaf71f: 2024-01-09T05:09:54.650Z [INFO] server-49c4da0a-d151-c41c-77a6-4eef96aaf71f: entering follower state: follower="Node at 49c4da0a-d151-c41c-77a6-4eef96aaf71f [Follower]" leader= - testing.go:149: server-49c4da0a-d151-c41c-77a6-4eef96aaf71f: 2024-01-09T05:09:54.650Z [INFO] server-49c4da0a-d151-c41c-77a6-4eef96aaf71f: entering follower state: follower="Node at 49c4da0a-d151-c41c-77a6-4eef96aaf71f [Follower]" leader= -testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.650Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: entering follower state: follower="Node at 27bbf3ad-bb6e-b691-d788-e28474238269 [Follower]" leader= - testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.650Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: entering follower state: follower="Node at 27bbf3ad-bb6e-b691-d788-e28474238269 [Follower]" leader= -testLoggerAdapter verbose: server-63264def-4c66-c838-4ee0-f04dbd85cbd3: 2024-01-09T05:09:54.650Z [INFO] server-63264def-4c66-c838-4ee0-f04dbd85cbd3: entering follower state: follower="Node at 63264def-4c66-c838-4ee0-f04dbd85cbd3 [Follower]" leader= - testing.go:149: server-63264def-4c66-c838-4ee0-f04dbd85cbd3: 2024-01-09T05:09:54.650Z [INFO] server-63264def-4c66-c838-4ee0-f04dbd85cbd3: entering follower state: follower="Node at 63264def-4c66-c838-4ee0-f04dbd85cbd3 [Follower]" leader= -testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [WARN] server-27bbf3ad-bb6e-b691-d788-e28474238269: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [WARN] server-27bbf3ad-bb6e-b691-d788-e28474238269: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: entering candidate state: node="Node at 27bbf3ad-bb6e-b691-d788-e28474238269 [Candidate]" term=2 - testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: entering candidate state: node="Node at 27bbf3ad-bb6e-b691-d788-e28474238269 [Candidate]" term=2 -testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: election won: tally=2 - testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: election won: tally=2 -testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: entering leader state: leader="Node at 27bbf3ad-bb6e-b691-d788-e28474238269 [Leader]" - testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: entering leader state: leader="Node at 27bbf3ad-bb6e-b691-d788-e28474238269 [Leader]" -testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: added peer, starting replication: peer=server-49c4da0a-d151-c41c-77a6-4eef96aaf71f - testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: added peer, starting replication: peer=server-49c4da0a-d151-c41c-77a6-4eef96aaf71f -testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: added peer, starting replication: peer=server-63264def-4c66-c838-4ee0-f04dbd85cbd3 - testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: added peer, starting replication: peer=server-63264def-4c66-c838-4ee0-f04dbd85cbd3 -testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: pipelining replication: peer="{Voter server-63264def-4c66-c838-4ee0-f04dbd85cbd3 63264def-4c66-c838-4ee0-f04dbd85cbd3}" - testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: pipelining replication: peer="{Voter server-63264def-4c66-c838-4ee0-f04dbd85cbd3 63264def-4c66-c838-4ee0-f04dbd85cbd3}" -testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: pipelining replication: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" - testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: pipelining replication: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.822Z [INFO] stable state for Leader reached at 2024-01-09 05:09:54.716838203 +0000 UTC m=+5.969889276 (1 nodes), 66.18785ms from start of poll, 66.668652ms from cluster start. Timeout at 2024-01-09 05:09:54.822185831 +0000 UTC m=+6.075236891, 105.347615ms after stability - testing.go:149: cluster: 2024-01-09T05:09:54.822Z [INFO] stable state for Leader reached at 2024-01-09 05:09:54.716838203 +0000 UTC m=+5.969889276 (1 nodes), 66.18785ms from start of poll, 66.668652ms from cluster start. Timeout at 2024-01-09 05:09:54.822185831 +0000 UTC m=+6.075236891, 105.347615ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.822Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:09:54.822Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.927Z [INFO] stable state for Follower reached at 2024-01-09 05:09:54.82236484 +0000 UTC m=+6.075415880 (2 nodes), 3.231µs from start of poll, 172.195256ms from cluster start. Timeout at 2024-01-09 05:09:54.927484484 +0000 UTC m=+6.180535517, 105.119637ms after stability - testing.go:149: cluster: 2024-01-09T05:09:54.927Z [INFO] stable state for Follower reached at 2024-01-09 05:09:54.82236484 +0000 UTC m=+6.075415880 (2 nodes), 3.231µs from start of poll, 172.195256ms from cluster start. Timeout at 2024-01-09 05:09:54.927484484 +0000 UTC m=+6.180535517, 105.119637ms after stability -testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.927Z [ERROR] server-27bbf3ad-bb6e-b691-d788-e28474238269: failed to pipeline appendEntries: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" error="append pipeline closed" - testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.927Z [ERROR] server-27bbf3ad-bb6e-b691-d788-e28474238269: failed to pipeline appendEntries: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" error="append pipeline closed" -testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.927Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: aborting pipeline replication: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" - testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.927Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: aborting pipeline replication: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" -testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.927Z [ERROR] server-27bbf3ad-bb6e-b691-d788-e28474238269: failed to appendEntries to: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" error="failed to connect to peer: 49c4da0a-d151-c41c-77a6-4eef96aaf71f" - testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.927Z [ERROR] server-27bbf3ad-bb6e-b691-d788-e28474238269: failed to appendEntries to: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" error="failed to connect to peer: 49c4da0a-d151-c41c-77a6-4eef96aaf71f" +testLoggerAdapter verbose: server-407227cb-679c-1722-1648-68ec0d01f300: 2025-02-10T11:34:09.212Z [INFO] server-407227cb-679c-1722-1648-68ec0d01f300: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-407227cb-679c-1722-1648-68ec0d01f300 Address:407227cb-679c-1722-1648-68ec0d01f300} {Suffrage:Voter ID:server-0034cf17-a28c-6483-becb-3395b0525af8 Address:0034cf17-a28c-6483-becb-3395b0525af8} {Suffrage:Voter ID:server-507d33e5-8d88-8ed8-e12f-2009e6bab547 Address:507d33e5-8d88-8ed8-e12f-2009e6bab547}]" + testing.go:149: server-407227cb-679c-1722-1648-68ec0d01f300: 2025-02-10T11:34:09.212Z [INFO] server-407227cb-679c-1722-1648-68ec0d01f300: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-407227cb-679c-1722-1648-68ec0d01f300 Address:407227cb-679c-1722-1648-68ec0d01f300} {Suffrage:Voter ID:server-0034cf17-a28c-6483-becb-3395b0525af8 Address:0034cf17-a28c-6483-becb-3395b0525af8} {Suffrage:Voter ID:server-507d33e5-8d88-8ed8-e12f-2009e6bab547 Address:507d33e5-8d88-8ed8-e12f-2009e6bab547}]" +testLoggerAdapter verbose: server-407227cb-679c-1722-1648-68ec0d01f300: 2025-02-10T11:34:09.212Z [INFO] server-407227cb-679c-1722-1648-68ec0d01f300: entering follower state: follower="Node at 407227cb-679c-1722-1648-68ec0d01f300 [Follower]" leader= + testing.go:149: server-407227cb-679c-1722-1648-68ec0d01f300: 2025-02-10T11:34:09.212Z [INFO] server-407227cb-679c-1722-1648-68ec0d01f300: entering follower state: follower="Node at 407227cb-679c-1722-1648-68ec0d01f300 [Follower]" leader= +testLoggerAdapter verbose: server-0034cf17-a28c-6483-becb-3395b0525af8: 2025-02-10T11:34:09.212Z [INFO] server-0034cf17-a28c-6483-becb-3395b0525af8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-407227cb-679c-1722-1648-68ec0d01f300 Address:407227cb-679c-1722-1648-68ec0d01f300} {Suffrage:Voter ID:server-0034cf17-a28c-6483-becb-3395b0525af8 Address:0034cf17-a28c-6483-becb-3395b0525af8} {Suffrage:Voter ID:server-507d33e5-8d88-8ed8-e12f-2009e6bab547 Address:507d33e5-8d88-8ed8-e12f-2009e6bab547}]" + testing.go:149: server-0034cf17-a28c-6483-becb-3395b0525af8: 2025-02-10T11:34:09.212Z [INFO] server-0034cf17-a28c-6483-becb-3395b0525af8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-407227cb-679c-1722-1648-68ec0d01f300 Address:407227cb-679c-1722-1648-68ec0d01f300} {Suffrage:Voter ID:server-0034cf17-a28c-6483-becb-3395b0525af8 Address:0034cf17-a28c-6483-becb-3395b0525af8} {Suffrage:Voter ID:server-507d33e5-8d88-8ed8-e12f-2009e6bab547 Address:507d33e5-8d88-8ed8-e12f-2009e6bab547}]" +testLoggerAdapter verbose: server-0034cf17-a28c-6483-becb-3395b0525af8: 2025-02-10T11:34:09.212Z [INFO] server-0034cf17-a28c-6483-becb-3395b0525af8: entering follower state: follower="Node at 0034cf17-a28c-6483-becb-3395b0525af8 [Follower]" leader= + testing.go:149: server-0034cf17-a28c-6483-becb-3395b0525af8: 2025-02-10T11:34:09.212Z [INFO] server-0034cf17-a28c-6483-becb-3395b0525af8: entering follower state: follower="Node at 0034cf17-a28c-6483-becb-3395b0525af8 [Follower]" leader= +testLoggerAdapter verbose: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.212Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-407227cb-679c-1722-1648-68ec0d01f300 Address:407227cb-679c-1722-1648-68ec0d01f300} {Suffrage:Voter ID:server-0034cf17-a28c-6483-becb-3395b0525af8 Address:0034cf17-a28c-6483-becb-3395b0525af8} {Suffrage:Voter ID:server-507d33e5-8d88-8ed8-e12f-2009e6bab547 Address:507d33e5-8d88-8ed8-e12f-2009e6bab547}]" + testing.go:149: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.212Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-407227cb-679c-1722-1648-68ec0d01f300 Address:407227cb-679c-1722-1648-68ec0d01f300} {Suffrage:Voter ID:server-0034cf17-a28c-6483-becb-3395b0525af8 Address:0034cf17-a28c-6483-becb-3395b0525af8} {Suffrage:Voter ID:server-507d33e5-8d88-8ed8-e12f-2009e6bab547 Address:507d33e5-8d88-8ed8-e12f-2009e6bab547}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:09.212Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:09.212Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.212Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: entering follower state: follower="Node at 507d33e5-8d88-8ed8-e12f-2009e6bab547 [Follower]" leader= + testing.go:149: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.212Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: entering follower state: follower="Node at 507d33e5-8d88-8ed8-e12f-2009e6bab547 [Follower]" leader= +testLoggerAdapter verbose: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.266Z [WARN] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.266Z [WARN] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.266Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: entering candidate state: node="Node at 507d33e5-8d88-8ed8-e12f-2009e6bab547 [Candidate]" term=2 + testing.go:149: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.266Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: entering candidate state: node="Node at 507d33e5-8d88-8ed8-e12f-2009e6bab547 [Candidate]" term=2 +testLoggerAdapter verbose: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.267Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: election won: tally=2 + testing.go:149: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.267Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: election won: tally=2 +testLoggerAdapter verbose: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.267Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: entering leader state: leader="Node at 507d33e5-8d88-8ed8-e12f-2009e6bab547 [Leader]" + testing.go:149: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.267Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: entering leader state: leader="Node at 507d33e5-8d88-8ed8-e12f-2009e6bab547 [Leader]" +testLoggerAdapter verbose: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.267Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: added peer, starting replication: peer=server-407227cb-679c-1722-1648-68ec0d01f300 + testing.go:149: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.267Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: added peer, starting replication: peer=server-407227cb-679c-1722-1648-68ec0d01f300 +testLoggerAdapter verbose: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.267Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: added peer, starting replication: peer=server-0034cf17-a28c-6483-becb-3395b0525af8 + testing.go:149: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.267Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: added peer, starting replication: peer=server-0034cf17-a28c-6483-becb-3395b0525af8 +testLoggerAdapter verbose: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.267Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: pipelining replication: peer="{Voter server-407227cb-679c-1722-1648-68ec0d01f300 407227cb-679c-1722-1648-68ec0d01f300}" + testing.go:149: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.267Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: pipelining replication: peer="{Voter server-407227cb-679c-1722-1648-68ec0d01f300 407227cb-679c-1722-1648-68ec0d01f300}" +testLoggerAdapter verbose: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.267Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: pipelining replication: peer="{Voter server-0034cf17-a28c-6483-becb-3395b0525af8 0034cf17-a28c-6483-becb-3395b0525af8}" + testing.go:149: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.267Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: pipelining replication: peer="{Voter server-0034cf17-a28c-6483-becb-3395b0525af8 0034cf17-a28c-6483-becb-3395b0525af8}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:09.372Z [INFO] stable state for Leader reached at 2025-02-10 11:34:09.267083972 +0000 UTC m=+5.635988274 (1 nodes), 54.165455ms from start of poll, 54.537884ms from cluster start. Timeout at 2025-02-10 11:34:09.37218561 +0000 UTC m=+5.741089960, 105.101686ms after stability + testing.go:149: cluster: 2025-02-10T11:34:09.372Z [INFO] stable state for Leader reached at 2025-02-10 11:34:09.267083972 +0000 UTC m=+5.635988274 (1 nodes), 54.165455ms from start of poll, 54.537884ms from cluster start. Timeout at 2025-02-10 11:34:09.37218561 +0000 UTC m=+5.741089960, 105.101686ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:09.372Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:09.372Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:09.477Z [INFO] stable state for Follower reached at 2025-02-10 11:34:09.372343832 +0000 UTC m=+5.741248130 (2 nodes), 705ns from start of poll, 159.79774ms from cluster start. Timeout at 2025-02-10 11:34:09.477422299 +0000 UTC m=+5.846326595, 105.078465ms after stability + testing.go:149: cluster: 2025-02-10T11:34:09.477Z [INFO] stable state for Follower reached at 2025-02-10 11:34:09.372343832 +0000 UTC m=+5.741248130 (2 nodes), 705ns from start of poll, 159.79774ms from cluster start. Timeout at 2025-02-10 11:34:09.477422299 +0000 UTC m=+5.846326595, 105.078465ms after stability +testLoggerAdapter verbose: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.477Z [ERROR] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: failed to pipeline appendEntries: peer="{Voter server-407227cb-679c-1722-1648-68ec0d01f300 407227cb-679c-1722-1648-68ec0d01f300}" error="append pipeline closed" + testing.go:149: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.477Z [ERROR] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: failed to pipeline appendEntries: peer="{Voter server-407227cb-679c-1722-1648-68ec0d01f300 407227cb-679c-1722-1648-68ec0d01f300}" error="append pipeline closed" +testLoggerAdapter verbose: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.477Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: aborting pipeline replication: peer="{Voter server-407227cb-679c-1722-1648-68ec0d01f300 407227cb-679c-1722-1648-68ec0d01f300}" + testing.go:149: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.477Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: aborting pipeline replication: peer="{Voter server-407227cb-679c-1722-1648-68ec0d01f300 407227cb-679c-1722-1648-68ec0d01f300}" +testLoggerAdapter verbose: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.477Z [ERROR] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: failed to appendEntries to: peer="{Voter server-407227cb-679c-1722-1648-68ec0d01f300 407227cb-679c-1722-1648-68ec0d01f300}" error="failed to connect to peer: 407227cb-679c-1722-1648-68ec0d01f300" + testing.go:149: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.477Z [ERROR] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: failed to appendEntries to: peer="{Voter server-407227cb-679c-1722-1648-68ec0d01f300 407227cb-679c-1722-1648-68ec0d01f300}" error="failed to connect to peer: 407227cb-679c-1722-1648-68ec0d01f300" raft_test.go:420: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.938Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: pipelining replication: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" - testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.938Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: pipelining replication: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.950Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:54.950Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:55.055Z [INFO] stable state for Leader reached at 2024-01-09 05:09:54.950409839 +0000 UTC m=+6.203460861 (1 nodes), 2.201µs from start of poll, 300.240237ms from cluster start. Timeout at 2024-01-09 05:09:55.05551688 +0000 UTC m=+6.308568127, 105.107266ms after stability - testing.go:149: cluster: 2024-01-09T05:09:55.055Z [INFO] stable state for Leader reached at 2024-01-09 05:09:54.950409839 +0000 UTC m=+6.203460861 (1 nodes), 2.201µs from start of poll, 300.240237ms from cluster start. Timeout at 2024-01-09 05:09:55.05551688 +0000 UTC m=+6.308568127, 105.107266ms after stability -testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:55.055Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: aborting pipeline replication: peer="{Voter server-63264def-4c66-c838-4ee0-f04dbd85cbd3 63264def-4c66-c838-4ee0-f04dbd85cbd3}" - testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:55.055Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: aborting pipeline replication: peer="{Voter server-63264def-4c66-c838-4ee0-f04dbd85cbd3 63264def-4c66-c838-4ee0-f04dbd85cbd3}" -testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:55.055Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: aborting pipeline replication: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" - testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:55.055Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: aborting pipeline replication: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" ---- PASS: TestRaft_BehindFollower (0.41s) +testLoggerAdapter verbose: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.487Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: pipelining replication: peer="{Voter server-407227cb-679c-1722-1648-68ec0d01f300 407227cb-679c-1722-1648-68ec0d01f300}" + testing.go:149: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.487Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: pipelining replication: peer="{Voter server-407227cb-679c-1722-1648-68ec0d01f300 407227cb-679c-1722-1648-68ec0d01f300}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:09.498Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:09.498Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:09.604Z [INFO] stable state for Leader reached at 2025-02-10 11:34:09.498957666 +0000 UTC m=+5.867861968 (1 nodes), 492ns from start of poll, 286.411578ms from cluster start. Timeout at 2025-02-10 11:34:09.604074949 +0000 UTC m=+5.972979246, 105.117278ms after stability + testing.go:149: cluster: 2025-02-10T11:34:09.604Z [INFO] stable state for Leader reached at 2025-02-10 11:34:09.498957666 +0000 UTC m=+5.867861968 (1 nodes), 492ns from start of poll, 286.411578ms from cluster start. Timeout at 2025-02-10 11:34:09.604074949 +0000 UTC m=+5.972979246, 105.117278ms after stability +testLoggerAdapter verbose: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.604Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: aborting pipeline replication: peer="{Voter server-0034cf17-a28c-6483-becb-3395b0525af8 0034cf17-a28c-6483-becb-3395b0525af8}" + testing.go:149: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.604Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: aborting pipeline replication: peer="{Voter server-0034cf17-a28c-6483-becb-3395b0525af8 0034cf17-a28c-6483-becb-3395b0525af8}" +testLoggerAdapter verbose: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.604Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: aborting pipeline replication: peer="{Voter server-407227cb-679c-1722-1648-68ec0d01f300 407227cb-679c-1722-1648-68ec0d01f300}" + testing.go:149: server-507d33e5-8d88-8ed8-e12f-2009e6bab547: 2025-02-10T11:34:09.604Z [INFO] server-507d33e5-8d88-8ed8-e12f-2009e6bab547: aborting pipeline replication: peer="{Voter server-407227cb-679c-1722-1648-68ec0d01f300 407227cb-679c-1722-1648-68ec0d01f300}" +--- PASS: TestRaft_BehindFollower (0.39s) === RUN TestRaft_ApplyNonLeader -testLoggerAdapter verbose: server-0b4ed842-f581-ab95-5247-4f33547ec339: 2024-01-09T05:09:55.058Z [INFO] server-0b4ed842-f581-ab95-5247-4f33547ec339: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b4ed842-f581-ab95-5247-4f33547ec339 Address:0b4ed842-f581-ab95-5247-4f33547ec339} {Suffrage:Voter ID:server-59592c2d-859d-7af8-e969-a9b189bef710 Address:59592c2d-859d-7af8-e969-a9b189bef710} {Suffrage:Voter ID:server-2e794893-82a8-228d-d44e-33ad22a55e36 Address:2e794893-82a8-228d-d44e-33ad22a55e36}]" - testing.go:149: server-0b4ed842-f581-ab95-5247-4f33547ec339: 2024-01-09T05:09:55.058Z [INFO] server-0b4ed842-f581-ab95-5247-4f33547ec339: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b4ed842-f581-ab95-5247-4f33547ec339 Address:0b4ed842-f581-ab95-5247-4f33547ec339} {Suffrage:Voter ID:server-59592c2d-859d-7af8-e969-a9b189bef710 Address:59592c2d-859d-7af8-e969-a9b189bef710} {Suffrage:Voter ID:server-2e794893-82a8-228d-d44e-33ad22a55e36 Address:2e794893-82a8-228d-d44e-33ad22a55e36}]" -testLoggerAdapter verbose: server-59592c2d-859d-7af8-e969-a9b189bef710: 2024-01-09T05:09:55.058Z [INFO] server-59592c2d-859d-7af8-e969-a9b189bef710: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b4ed842-f581-ab95-5247-4f33547ec339 Address:0b4ed842-f581-ab95-5247-4f33547ec339} {Suffrage:Voter ID:server-59592c2d-859d-7af8-e969-a9b189bef710 Address:59592c2d-859d-7af8-e969-a9b189bef710} {Suffrage:Voter ID:server-2e794893-82a8-228d-d44e-33ad22a55e36 Address:2e794893-82a8-228d-d44e-33ad22a55e36}]" - testing.go:149: server-59592c2d-859d-7af8-e969-a9b189bef710: 2024-01-09T05:09:55.058Z [INFO] server-59592c2d-859d-7af8-e969-a9b189bef710: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b4ed842-f581-ab95-5247-4f33547ec339 Address:0b4ed842-f581-ab95-5247-4f33547ec339} {Suffrage:Voter ID:server-59592c2d-859d-7af8-e969-a9b189bef710 Address:59592c2d-859d-7af8-e969-a9b189bef710} {Suffrage:Voter ID:server-2e794893-82a8-228d-d44e-33ad22a55e36 Address:2e794893-82a8-228d-d44e-33ad22a55e36}]" -testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.058Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b4ed842-f581-ab95-5247-4f33547ec339 Address:0b4ed842-f581-ab95-5247-4f33547ec339} {Suffrage:Voter ID:server-59592c2d-859d-7af8-e969-a9b189bef710 Address:59592c2d-859d-7af8-e969-a9b189bef710} {Suffrage:Voter ID:server-2e794893-82a8-228d-d44e-33ad22a55e36 Address:2e794893-82a8-228d-d44e-33ad22a55e36}]" - testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.058Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b4ed842-f581-ab95-5247-4f33547ec339 Address:0b4ed842-f581-ab95-5247-4f33547ec339} {Suffrage:Voter ID:server-59592c2d-859d-7af8-e969-a9b189bef710 Address:59592c2d-859d-7af8-e969-a9b189bef710} {Suffrage:Voter ID:server-2e794893-82a8-228d-d44e-33ad22a55e36 Address:2e794893-82a8-228d-d44e-33ad22a55e36}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:55.059Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-59592c2d-859d-7af8-e969-a9b189bef710: 2024-01-09T05:09:55.059Z [INFO] server-59592c2d-859d-7af8-e969-a9b189bef710: entering follower state: follower="Node at 59592c2d-859d-7af8-e969-a9b189bef710 [Follower]" leader= -testLoggerAdapter verbose: server-0b4ed842-f581-ab95-5247-4f33547ec339: 2024-01-09T05:09:55.059Z [INFO] server-0b4ed842-f581-ab95-5247-4f33547ec339: entering follower state: follower="Node at 0b4ed842-f581-ab95-5247-4f33547ec339 [Follower]" leader= -testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.059Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: entering follower state: follower="Node at 2e794893-82a8-228d-d44e-33ad22a55e36 [Follower]" leader= - testing.go:149: cluster: 2024-01-09T05:09:55.059Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-59592c2d-859d-7af8-e969-a9b189bef710: 2024-01-09T05:09:55.059Z [INFO] server-59592c2d-859d-7af8-e969-a9b189bef710: entering follower state: follower="Node at 59592c2d-859d-7af8-e969-a9b189bef710 [Follower]" leader= - testing.go:149: server-0b4ed842-f581-ab95-5247-4f33547ec339: 2024-01-09T05:09:55.059Z [INFO] server-0b4ed842-f581-ab95-5247-4f33547ec339: entering follower state: follower="Node at 0b4ed842-f581-ab95-5247-4f33547ec339 [Follower]" leader= - testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.059Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: entering follower state: follower="Node at 2e794893-82a8-228d-d44e-33ad22a55e36 [Follower]" leader= -testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [WARN] server-2e794893-82a8-228d-d44e-33ad22a55e36: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [WARN] server-2e794893-82a8-228d-d44e-33ad22a55e36: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: entering candidate state: node="Node at 2e794893-82a8-228d-d44e-33ad22a55e36 [Candidate]" term=2 - testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: entering candidate state: node="Node at 2e794893-82a8-228d-d44e-33ad22a55e36 [Candidate]" term=2 -testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: election won: tally=2 - testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: election won: tally=2 -testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: entering leader state: leader="Node at 2e794893-82a8-228d-d44e-33ad22a55e36 [Leader]" - testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: entering leader state: leader="Node at 2e794893-82a8-228d-d44e-33ad22a55e36 [Leader]" -testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: added peer, starting replication: peer=server-0b4ed842-f581-ab95-5247-4f33547ec339 - testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: added peer, starting replication: peer=server-0b4ed842-f581-ab95-5247-4f33547ec339 -testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: added peer, starting replication: peer=server-59592c2d-859d-7af8-e969-a9b189bef710 - testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: added peer, starting replication: peer=server-59592c2d-859d-7af8-e969-a9b189bef710 -testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: pipelining replication: peer="{Voter server-59592c2d-859d-7af8-e969-a9b189bef710 59592c2d-859d-7af8-e969-a9b189bef710}" - testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: pipelining replication: peer="{Voter server-59592c2d-859d-7af8-e969-a9b189bef710 59592c2d-859d-7af8-e969-a9b189bef710}" -testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: pipelining replication: peer="{Voter server-0b4ed842-f581-ab95-5247-4f33547ec339 0b4ed842-f581-ab95-5247-4f33547ec339}" - testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: pipelining replication: peer="{Voter server-0b4ed842-f581-ab95-5247-4f33547ec339 0b4ed842-f581-ab95-5247-4f33547ec339}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:55.230Z [INFO] stable state for Leader reached at 2024-01-09 05:09:55.125061163 +0000 UTC m=+6.378112175 (1 nodes), 65.523595ms from start of poll, 66.767561ms from cluster start. Timeout at 2024-01-09 05:09:55.230241081 +0000 UTC m=+6.483292171, 105.179996ms after stability - testing.go:149: cluster: 2024-01-09T05:09:55.230Z [INFO] stable state for Leader reached at 2024-01-09 05:09:55.125061163 +0000 UTC m=+6.378112175 (1 nodes), 65.523595ms from start of poll, 66.767561ms from cluster start. Timeout at 2024-01-09 05:09:55.230241081 +0000 UTC m=+6.483292171, 105.179996ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:55.230Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:09:55.230Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:55.336Z [INFO] stable state for Follower reached at 2024-01-09 05:09:55.231027263 +0000 UTC m=+6.484078289 (2 nodes), 2.875µs from start of poll, 172.733675ms from cluster start. Timeout at 2024-01-09 05:09:55.336115841 +0000 UTC m=+6.589166854, 105.088565ms after stability - testing.go:149: cluster: 2024-01-09T05:09:55.336Z [INFO] stable state for Follower reached at 2024-01-09 05:09:55.231027263 +0000 UTC m=+6.484078289 (2 nodes), 2.875µs from start of poll, 172.733675ms from cluster start. Timeout at 2024-01-09 05:09:55.336115841 +0000 UTC m=+6.589166854, 105.088565ms after stability -testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.336Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: aborting pipeline replication: peer="{Voter server-59592c2d-859d-7af8-e969-a9b189bef710 59592c2d-859d-7af8-e969-a9b189bef710}" - testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.336Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: aborting pipeline replication: peer="{Voter server-59592c2d-859d-7af8-e969-a9b189bef710 59592c2d-859d-7af8-e969-a9b189bef710}" -testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.336Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: aborting pipeline replication: peer="{Voter server-0b4ed842-f581-ab95-5247-4f33547ec339 0b4ed842-f581-ab95-5247-4f33547ec339}" - testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.336Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: aborting pipeline replication: peer="{Voter server-0b4ed842-f581-ab95-5247-4f33547ec339 0b4ed842-f581-ab95-5247-4f33547ec339}" ---- PASS: TestRaft_ApplyNonLeader (0.28s) +testLoggerAdapter verbose: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.605Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73 Address:7bbbcdca-ea19-b91e-9337-e10d3b5a4b73} {Suffrage:Voter ID:server-359cdbde-e211-094c-efe1-de0836c5a9ac Address:359cdbde-e211-094c-efe1-de0836c5a9ac} {Suffrage:Voter ID:server-6490b056-bba7-4169-5c01-b53c9086aac3 Address:6490b056-bba7-4169-5c01-b53c9086aac3}]" + testing.go:149: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.605Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73 Address:7bbbcdca-ea19-b91e-9337-e10d3b5a4b73} {Suffrage:Voter ID:server-359cdbde-e211-094c-efe1-de0836c5a9ac Address:359cdbde-e211-094c-efe1-de0836c5a9ac} {Suffrage:Voter ID:server-6490b056-bba7-4169-5c01-b53c9086aac3 Address:6490b056-bba7-4169-5c01-b53c9086aac3}]" +testLoggerAdapter verbose: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.605Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: entering follower state: follower="Node at 7bbbcdca-ea19-b91e-9337-e10d3b5a4b73 [Follower]" leader= + testing.go:149: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.605Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: entering follower state: follower="Node at 7bbbcdca-ea19-b91e-9337-e10d3b5a4b73 [Follower]" leader= +testLoggerAdapter verbose: server-359cdbde-e211-094c-efe1-de0836c5a9ac: 2025-02-10T11:34:09.605Z [INFO] server-359cdbde-e211-094c-efe1-de0836c5a9ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73 Address:7bbbcdca-ea19-b91e-9337-e10d3b5a4b73} {Suffrage:Voter ID:server-359cdbde-e211-094c-efe1-de0836c5a9ac Address:359cdbde-e211-094c-efe1-de0836c5a9ac} {Suffrage:Voter ID:server-6490b056-bba7-4169-5c01-b53c9086aac3 Address:6490b056-bba7-4169-5c01-b53c9086aac3}]" + testing.go:149: server-359cdbde-e211-094c-efe1-de0836c5a9ac: 2025-02-10T11:34:09.605Z [INFO] server-359cdbde-e211-094c-efe1-de0836c5a9ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73 Address:7bbbcdca-ea19-b91e-9337-e10d3b5a4b73} {Suffrage:Voter ID:server-359cdbde-e211-094c-efe1-de0836c5a9ac Address:359cdbde-e211-094c-efe1-de0836c5a9ac} {Suffrage:Voter ID:server-6490b056-bba7-4169-5c01-b53c9086aac3 Address:6490b056-bba7-4169-5c01-b53c9086aac3}]" +testLoggerAdapter verbose: server-359cdbde-e211-094c-efe1-de0836c5a9ac: 2025-02-10T11:34:09.605Z [INFO] server-359cdbde-e211-094c-efe1-de0836c5a9ac: entering follower state: follower="Node at 359cdbde-e211-094c-efe1-de0836c5a9ac [Follower]" leader= + testing.go:149: server-359cdbde-e211-094c-efe1-de0836c5a9ac: 2025-02-10T11:34:09.605Z [INFO] server-359cdbde-e211-094c-efe1-de0836c5a9ac: entering follower state: follower="Node at 359cdbde-e211-094c-efe1-de0836c5a9ac [Follower]" leader= +testLoggerAdapter verbose: server-6490b056-bba7-4169-5c01-b53c9086aac3: 2025-02-10T11:34:09.605Z [INFO] server-6490b056-bba7-4169-5c01-b53c9086aac3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73 Address:7bbbcdca-ea19-b91e-9337-e10d3b5a4b73} {Suffrage:Voter ID:server-359cdbde-e211-094c-efe1-de0836c5a9ac Address:359cdbde-e211-094c-efe1-de0836c5a9ac} {Suffrage:Voter ID:server-6490b056-bba7-4169-5c01-b53c9086aac3 Address:6490b056-bba7-4169-5c01-b53c9086aac3}]" + testing.go:149: server-6490b056-bba7-4169-5c01-b53c9086aac3: 2025-02-10T11:34:09.605Z [INFO] server-6490b056-bba7-4169-5c01-b53c9086aac3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73 Address:7bbbcdca-ea19-b91e-9337-e10d3b5a4b73} {Suffrage:Voter ID:server-359cdbde-e211-094c-efe1-de0836c5a9ac Address:359cdbde-e211-094c-efe1-de0836c5a9ac} {Suffrage:Voter ID:server-6490b056-bba7-4169-5c01-b53c9086aac3 Address:6490b056-bba7-4169-5c01-b53c9086aac3}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:09.605Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:09.605Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-6490b056-bba7-4169-5c01-b53c9086aac3: 2025-02-10T11:34:09.605Z [INFO] server-6490b056-bba7-4169-5c01-b53c9086aac3: entering follower state: follower="Node at 6490b056-bba7-4169-5c01-b53c9086aac3 [Follower]" leader= + testing.go:149: server-6490b056-bba7-4169-5c01-b53c9086aac3: 2025-02-10T11:34:09.605Z [INFO] server-6490b056-bba7-4169-5c01-b53c9086aac3: entering follower state: follower="Node at 6490b056-bba7-4169-5c01-b53c9086aac3 [Follower]" leader= +testLoggerAdapter verbose: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.667Z [WARN] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.667Z [WARN] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.667Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: entering candidate state: node="Node at 7bbbcdca-ea19-b91e-9337-e10d3b5a4b73 [Candidate]" term=2 + testing.go:149: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.667Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: entering candidate state: node="Node at 7bbbcdca-ea19-b91e-9337-e10d3b5a4b73 [Candidate]" term=2 +testLoggerAdapter verbose: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.667Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: election won: tally=2 + testing.go:149: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.667Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: election won: tally=2 +testLoggerAdapter verbose: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.667Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: entering leader state: leader="Node at 7bbbcdca-ea19-b91e-9337-e10d3b5a4b73 [Leader]" + testing.go:149: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.667Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: entering leader state: leader="Node at 7bbbcdca-ea19-b91e-9337-e10d3b5a4b73 [Leader]" +testLoggerAdapter verbose: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.667Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: added peer, starting replication: peer=server-359cdbde-e211-094c-efe1-de0836c5a9ac + testing.go:149: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.667Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: added peer, starting replication: peer=server-359cdbde-e211-094c-efe1-de0836c5a9ac +testLoggerAdapter verbose: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.667Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: added peer, starting replication: peer=server-6490b056-bba7-4169-5c01-b53c9086aac3 + testing.go:149: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.667Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: added peer, starting replication: peer=server-6490b056-bba7-4169-5c01-b53c9086aac3 +testLoggerAdapter verbose: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.667Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: pipelining replication: peer="{Voter server-6490b056-bba7-4169-5c01-b53c9086aac3 6490b056-bba7-4169-5c01-b53c9086aac3}" + testing.go:149: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.667Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: pipelining replication: peer="{Voter server-6490b056-bba7-4169-5c01-b53c9086aac3 6490b056-bba7-4169-5c01-b53c9086aac3}" +testLoggerAdapter verbose: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.667Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: pipelining replication: peer="{Voter server-359cdbde-e211-094c-efe1-de0836c5a9ac 359cdbde-e211-094c-efe1-de0836c5a9ac}" + testing.go:149: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.667Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: pipelining replication: peer="{Voter server-359cdbde-e211-094c-efe1-de0836c5a9ac 359cdbde-e211-094c-efe1-de0836c5a9ac}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:09.772Z [INFO] stable state for Leader reached at 2025-02-10 11:34:09.667647891 +0000 UTC m=+6.036552193 (1 nodes), 62.141844ms from start of poll, 62.565327ms from cluster start. Timeout at 2025-02-10 11:34:09.772720709 +0000 UTC m=+6.141625004, 105.072811ms after stability + testing.go:149: cluster: 2025-02-10T11:34:09.772Z [INFO] stable state for Leader reached at 2025-02-10 11:34:09.667647891 +0000 UTC m=+6.036552193 (1 nodes), 62.141844ms from start of poll, 62.565327ms from cluster start. Timeout at 2025-02-10 11:34:09.772720709 +0000 UTC m=+6.141625004, 105.072811ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:09.772Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:09.772Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:09.877Z [INFO] stable state for Follower reached at 2025-02-10 11:34:09.772796722 +0000 UTC m=+6.141701022 (2 nodes), 703ns from start of poll, 167.714156ms from cluster start. Timeout at 2025-02-10 11:34:09.877899954 +0000 UTC m=+6.246804304, 105.103282ms after stability + testing.go:149: cluster: 2025-02-10T11:34:09.877Z [INFO] stable state for Follower reached at 2025-02-10 11:34:09.772796722 +0000 UTC m=+6.141701022 (2 nodes), 703ns from start of poll, 167.714156ms from cluster start. Timeout at 2025-02-10 11:34:09.877899954 +0000 UTC m=+6.246804304, 105.103282ms after stability +testLoggerAdapter verbose: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.878Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: aborting pipeline replication: peer="{Voter server-6490b056-bba7-4169-5c01-b53c9086aac3 6490b056-bba7-4169-5c01-b53c9086aac3}" + testing.go:149: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.878Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: aborting pipeline replication: peer="{Voter server-6490b056-bba7-4169-5c01-b53c9086aac3 6490b056-bba7-4169-5c01-b53c9086aac3}" +testLoggerAdapter verbose: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.878Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: aborting pipeline replication: peer="{Voter server-359cdbde-e211-094c-efe1-de0836c5a9ac 359cdbde-e211-094c-efe1-de0836c5a9ac}" + testing.go:149: server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: 2025-02-10T11:34:09.878Z [INFO] server-7bbbcdca-ea19-b91e-9337-e10d3b5a4b73: aborting pipeline replication: peer="{Voter server-359cdbde-e211-094c-efe1-de0836c5a9ac 359cdbde-e211-094c-efe1-de0836c5a9ac}" +--- PASS: TestRaft_ApplyNonLeader (0.27s) === RUN TestRaft_ApplyConcurrent -testLoggerAdapter verbose: server-a39db28d-8f3a-024e-8ef6-61762d94d488: 2024-01-09T05:09:55.337Z [INFO] server-a39db28d-8f3a-024e-8ef6-61762d94d488: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a39db28d-8f3a-024e-8ef6-61762d94d488 Address:a39db28d-8f3a-024e-8ef6-61762d94d488} {Suffrage:Voter ID:server-ac2de96e-8850-c488-3df4-bc550f9a55ec Address:ac2de96e-8850-c488-3df4-bc550f9a55ec} {Suffrage:Voter ID:server-9d2584ff-67d8-dff1-816e-661ae3882d48 Address:9d2584ff-67d8-dff1-816e-661ae3882d48}]" - testing.go:149: server-a39db28d-8f3a-024e-8ef6-61762d94d488: 2024-01-09T05:09:55.337Z [INFO] server-a39db28d-8f3a-024e-8ef6-61762d94d488: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a39db28d-8f3a-024e-8ef6-61762d94d488 Address:a39db28d-8f3a-024e-8ef6-61762d94d488} {Suffrage:Voter ID:server-ac2de96e-8850-c488-3df4-bc550f9a55ec Address:ac2de96e-8850-c488-3df4-bc550f9a55ec} {Suffrage:Voter ID:server-9d2584ff-67d8-dff1-816e-661ae3882d48 Address:9d2584ff-67d8-dff1-816e-661ae3882d48}]" -testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.337Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a39db28d-8f3a-024e-8ef6-61762d94d488 Address:a39db28d-8f3a-024e-8ef6-61762d94d488} {Suffrage:Voter ID:server-ac2de96e-8850-c488-3df4-bc550f9a55ec Address:ac2de96e-8850-c488-3df4-bc550f9a55ec} {Suffrage:Voter ID:server-9d2584ff-67d8-dff1-816e-661ae3882d48 Address:9d2584ff-67d8-dff1-816e-661ae3882d48}]" - testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.337Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a39db28d-8f3a-024e-8ef6-61762d94d488 Address:a39db28d-8f3a-024e-8ef6-61762d94d488} {Suffrage:Voter ID:server-ac2de96e-8850-c488-3df4-bc550f9a55ec Address:ac2de96e-8850-c488-3df4-bc550f9a55ec} {Suffrage:Voter ID:server-9d2584ff-67d8-dff1-816e-661ae3882d48 Address:9d2584ff-67d8-dff1-816e-661ae3882d48}]" -testLoggerAdapter verbose: server-9d2584ff-67d8-dff1-816e-661ae3882d48: 2024-01-09T05:09:55.337Z [INFO] server-9d2584ff-67d8-dff1-816e-661ae3882d48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a39db28d-8f3a-024e-8ef6-61762d94d488 Address:a39db28d-8f3a-024e-8ef6-61762d94d488} {Suffrage:Voter ID:server-ac2de96e-8850-c488-3df4-bc550f9a55ec Address:ac2de96e-8850-c488-3df4-bc550f9a55ec} {Suffrage:Voter ID:server-9d2584ff-67d8-dff1-816e-661ae3882d48 Address:9d2584ff-67d8-dff1-816e-661ae3882d48}]" - testing.go:149: server-9d2584ff-67d8-dff1-816e-661ae3882d48: 2024-01-09T05:09:55.337Z [INFO] server-9d2584ff-67d8-dff1-816e-661ae3882d48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a39db28d-8f3a-024e-8ef6-61762d94d488 Address:a39db28d-8f3a-024e-8ef6-61762d94d488} {Suffrage:Voter ID:server-ac2de96e-8850-c488-3df4-bc550f9a55ec Address:ac2de96e-8850-c488-3df4-bc550f9a55ec} {Suffrage:Voter ID:server-9d2584ff-67d8-dff1-816e-661ae3882d48 Address:9d2584ff-67d8-dff1-816e-661ae3882d48}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:55.337Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:55.337Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-a39db28d-8f3a-024e-8ef6-61762d94d488: 2024-01-09T05:09:55.338Z [INFO] server-a39db28d-8f3a-024e-8ef6-61762d94d488: entering follower state: follower="Node at a39db28d-8f3a-024e-8ef6-61762d94d488 [Follower]" leader= - testing.go:149: server-a39db28d-8f3a-024e-8ef6-61762d94d488: 2024-01-09T05:09:55.338Z [INFO] server-a39db28d-8f3a-024e-8ef6-61762d94d488: entering follower state: follower="Node at a39db28d-8f3a-024e-8ef6-61762d94d488 [Follower]" leader= -testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.338Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: entering follower state: follower="Node at ac2de96e-8850-c488-3df4-bc550f9a55ec [Follower]" leader= - testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.338Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: entering follower state: follower="Node at ac2de96e-8850-c488-3df4-bc550f9a55ec [Follower]" leader= -testLoggerAdapter verbose: server-9d2584ff-67d8-dff1-816e-661ae3882d48: 2024-01-09T05:09:55.338Z [INFO] server-9d2584ff-67d8-dff1-816e-661ae3882d48: entering follower state: follower="Node at 9d2584ff-67d8-dff1-816e-661ae3882d48 [Follower]" leader= - testing.go:149: server-9d2584ff-67d8-dff1-816e-661ae3882d48: 2024-01-09T05:09:55.338Z [INFO] server-9d2584ff-67d8-dff1-816e-661ae3882d48: entering follower state: follower="Node at 9d2584ff-67d8-dff1-816e-661ae3882d48 [Follower]" leader= -testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.447Z [WARN] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.447Z [WARN] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.447Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: entering candidate state: node="Node at ac2de96e-8850-c488-3df4-bc550f9a55ec [Candidate]" term=2 - testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.447Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: entering candidate state: node="Node at ac2de96e-8850-c488-3df4-bc550f9a55ec [Candidate]" term=2 -testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.447Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: election won: tally=2 - testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.447Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: election won: tally=2 -testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.447Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: entering leader state: leader="Node at ac2de96e-8850-c488-3df4-bc550f9a55ec [Leader]" - testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.447Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: entering leader state: leader="Node at ac2de96e-8850-c488-3df4-bc550f9a55ec [Leader]" -testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.448Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: added peer, starting replication: peer=server-a39db28d-8f3a-024e-8ef6-61762d94d488 - testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.448Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: added peer, starting replication: peer=server-a39db28d-8f3a-024e-8ef6-61762d94d488 -testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.448Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: added peer, starting replication: peer=server-9d2584ff-67d8-dff1-816e-661ae3882d48 - testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.448Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: added peer, starting replication: peer=server-9d2584ff-67d8-dff1-816e-661ae3882d48 -testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.448Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: pipelining replication: peer="{Voter server-9d2584ff-67d8-dff1-816e-661ae3882d48 9d2584ff-67d8-dff1-816e-661ae3882d48}" - testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.448Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: pipelining replication: peer="{Voter server-9d2584ff-67d8-dff1-816e-661ae3882d48 9d2584ff-67d8-dff1-816e-661ae3882d48}" -testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.448Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: pipelining replication: peer="{Voter server-a39db28d-8f3a-024e-8ef6-61762d94d488 a39db28d-8f3a-024e-8ef6-61762d94d488}" - testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.448Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: pipelining replication: peer="{Voter server-a39db28d-8f3a-024e-8ef6-61762d94d488 a39db28d-8f3a-024e-8ef6-61762d94d488}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:55.653Z [INFO] stable state for Leader reached at 2024-01-09 05:09:55.448192383 +0000 UTC m=+6.701244041 (1 nodes), 110.207265ms from start of poll, 110.624866ms from cluster start. Timeout at 2024-01-09 05:09:55.653584874 +0000 UTC m=+6.906636046, 205.392005ms after stability - testing.go:149: cluster: 2024-01-09T05:09:55.653Z [INFO] stable state for Leader reached at 2024-01-09 05:09:55.448192383 +0000 UTC m=+6.701244041 (1 nodes), 110.207265ms from start of poll, 110.624866ms from cluster start. Timeout at 2024-01-09 05:09:55.653584874 +0000 UTC m=+6.906636046, 205.392005ms after stability -testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.674Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: aborting pipeline replication: peer="{Voter server-9d2584ff-67d8-dff1-816e-661ae3882d48 9d2584ff-67d8-dff1-816e-661ae3882d48}" - testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.674Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: aborting pipeline replication: peer="{Voter server-9d2584ff-67d8-dff1-816e-661ae3882d48 9d2584ff-67d8-dff1-816e-661ae3882d48}" -testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.674Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: aborting pipeline replication: peer="{Voter server-a39db28d-8f3a-024e-8ef6-61762d94d488 a39db28d-8f3a-024e-8ef6-61762d94d488}" - testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.674Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: aborting pipeline replication: peer="{Voter server-a39db28d-8f3a-024e-8ef6-61762d94d488 a39db28d-8f3a-024e-8ef6-61762d94d488}" ---- PASS: TestRaft_ApplyConcurrent (0.34s) +testLoggerAdapter verbose: server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0: 2025-02-10T11:34:09.879Z [INFO] server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0 Address:f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0} {Suffrage:Voter ID:server-d96165b3-918a-7fad-e52b-6753fdcccd11 Address:d96165b3-918a-7fad-e52b-6753fdcccd11} {Suffrage:Voter ID:server-89c99f67-25f9-8b4e-0263-5cb1ce6de255 Address:89c99f67-25f9-8b4e-0263-5cb1ce6de255}]" + testing.go:149: server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0: 2025-02-10T11:34:09.879Z [INFO] server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0 Address:f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0} {Suffrage:Voter ID:server-d96165b3-918a-7fad-e52b-6753fdcccd11 Address:d96165b3-918a-7fad-e52b-6753fdcccd11} {Suffrage:Voter ID:server-89c99f67-25f9-8b4e-0263-5cb1ce6de255 Address:89c99f67-25f9-8b4e-0263-5cb1ce6de255}]" +testLoggerAdapter verbose: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:09.879Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0 Address:f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0} {Suffrage:Voter ID:server-d96165b3-918a-7fad-e52b-6753fdcccd11 Address:d96165b3-918a-7fad-e52b-6753fdcccd11} {Suffrage:Voter ID:server-89c99f67-25f9-8b4e-0263-5cb1ce6de255 Address:89c99f67-25f9-8b4e-0263-5cb1ce6de255}]" + testing.go:149: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:09.879Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0 Address:f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0} {Suffrage:Voter ID:server-d96165b3-918a-7fad-e52b-6753fdcccd11 Address:d96165b3-918a-7fad-e52b-6753fdcccd11} {Suffrage:Voter ID:server-89c99f67-25f9-8b4e-0263-5cb1ce6de255 Address:89c99f67-25f9-8b4e-0263-5cb1ce6de255}]" +testLoggerAdapter verbose: server-89c99f67-25f9-8b4e-0263-5cb1ce6de255: 2025-02-10T11:34:09.879Z [INFO] server-89c99f67-25f9-8b4e-0263-5cb1ce6de255: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0 Address:f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0} {Suffrage:Voter ID:server-d96165b3-918a-7fad-e52b-6753fdcccd11 Address:d96165b3-918a-7fad-e52b-6753fdcccd11} {Suffrage:Voter ID:server-89c99f67-25f9-8b4e-0263-5cb1ce6de255 Address:89c99f67-25f9-8b4e-0263-5cb1ce6de255}]" + testing.go:149: server-89c99f67-25f9-8b4e-0263-5cb1ce6de255: 2025-02-10T11:34:09.879Z [INFO] server-89c99f67-25f9-8b4e-0263-5cb1ce6de255: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0 Address:f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0} {Suffrage:Voter ID:server-d96165b3-918a-7fad-e52b-6753fdcccd11 Address:d96165b3-918a-7fad-e52b-6753fdcccd11} {Suffrage:Voter ID:server-89c99f67-25f9-8b4e-0263-5cb1ce6de255 Address:89c99f67-25f9-8b4e-0263-5cb1ce6de255}]" +testLoggerAdapter verbose: server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0: 2025-02-10T11:34:09.879Z [INFO] server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0: entering follower state: follower="Node at f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0 [Follower]" leader= + testing.go:149: server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0: 2025-02-10T11:34:09.879Z [INFO] server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0: entering follower state: follower="Node at f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0 [Follower]" leader= +testLoggerAdapter verbose: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:09.879Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: entering follower state: follower="Node at d96165b3-918a-7fad-e52b-6753fdcccd11 [Follower]" leader= + testing.go:149: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:09.879Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: entering follower state: follower="Node at d96165b3-918a-7fad-e52b-6753fdcccd11 [Follower]" leader= +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:09.879Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:09.879Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-89c99f67-25f9-8b4e-0263-5cb1ce6de255: 2025-02-10T11:34:09.880Z [INFO] server-89c99f67-25f9-8b4e-0263-5cb1ce6de255: entering follower state: follower="Node at 89c99f67-25f9-8b4e-0263-5cb1ce6de255 [Follower]" leader= + testing.go:149: server-89c99f67-25f9-8b4e-0263-5cb1ce6de255: 2025-02-10T11:34:09.880Z [INFO] server-89c99f67-25f9-8b4e-0263-5cb1ce6de255: entering follower state: follower="Node at 89c99f67-25f9-8b4e-0263-5cb1ce6de255 [Follower]" leader= +testLoggerAdapter verbose: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.048Z [WARN] server-d96165b3-918a-7fad-e52b-6753fdcccd11: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.048Z [WARN] server-d96165b3-918a-7fad-e52b-6753fdcccd11: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.048Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: entering candidate state: node="Node at d96165b3-918a-7fad-e52b-6753fdcccd11 [Candidate]" term=2 + testing.go:149: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.048Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: entering candidate state: node="Node at d96165b3-918a-7fad-e52b-6753fdcccd11 [Candidate]" term=2 +testLoggerAdapter verbose: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.048Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: election won: tally=2 + testing.go:149: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.048Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: election won: tally=2 +testLoggerAdapter verbose: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.048Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: entering leader state: leader="Node at d96165b3-918a-7fad-e52b-6753fdcccd11 [Leader]" + testing.go:149: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.048Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: entering leader state: leader="Node at d96165b3-918a-7fad-e52b-6753fdcccd11 [Leader]" +testLoggerAdapter verbose: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.048Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: added peer, starting replication: peer=server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0 + testing.go:149: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.048Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: added peer, starting replication: peer=server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0 +testLoggerAdapter verbose: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.048Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: added peer, starting replication: peer=server-89c99f67-25f9-8b4e-0263-5cb1ce6de255 + testing.go:149: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.048Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: added peer, starting replication: peer=server-89c99f67-25f9-8b4e-0263-5cb1ce6de255 +testLoggerAdapter verbose: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.048Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: pipelining replication: peer="{Voter server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0 f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0}" + testing.go:149: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.048Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: pipelining replication: peer="{Voter server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0 f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0}" +testLoggerAdapter verbose: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.048Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: pipelining replication: peer="{Voter server-89c99f67-25f9-8b4e-0263-5cb1ce6de255 89c99f67-25f9-8b4e-0263-5cb1ce6de255}" + testing.go:149: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.048Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: pipelining replication: peer="{Voter server-89c99f67-25f9-8b4e-0263-5cb1ce6de255 89c99f67-25f9-8b4e-0263-5cb1ce6de255}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:10.253Z [INFO] stable state for Leader reached at 2025-02-10 11:34:10.048169172 +0000 UTC m=+6.417073472 (1 nodes), 168.124617ms from start of poll, 169.18335ms from cluster start. Timeout at 2025-02-10 11:34:10.253270139 +0000 UTC m=+6.622174492, 205.10102ms after stability + testing.go:149: cluster: 2025-02-10T11:34:10.253Z [INFO] stable state for Leader reached at 2025-02-10 11:34:10.048169172 +0000 UTC m=+6.417073472 (1 nodes), 168.124617ms from start of poll, 169.18335ms from cluster start. Timeout at 2025-02-10 11:34:10.253270139 +0000 UTC m=+6.622174492, 205.10102ms after stability +testLoggerAdapter verbose: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.264Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: aborting pipeline replication: peer="{Voter server-89c99f67-25f9-8b4e-0263-5cb1ce6de255 89c99f67-25f9-8b4e-0263-5cb1ce6de255}" + testing.go:149: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.264Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: aborting pipeline replication: peer="{Voter server-89c99f67-25f9-8b4e-0263-5cb1ce6de255 89c99f67-25f9-8b4e-0263-5cb1ce6de255}" +testLoggerAdapter verbose: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.264Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: aborting pipeline replication: peer="{Voter server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0 f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0}" + testing.go:149: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.264Z [INFO] server-d96165b3-918a-7fad-e52b-6753fdcccd11: aborting pipeline replication: peer="{Voter server-f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0 f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0}" +testLoggerAdapter verbose: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.315Z [ERROR] server-d96165b3-918a-7fad-e52b-6753fdcccd11: failed to heartbeat to: peer=f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0 error="command timed out" + testing.go:149: server-d96165b3-918a-7fad-e52b-6753fdcccd11: 2025-02-10T11:34:10.315Z [ERROR] server-d96165b3-918a-7fad-e52b-6753fdcccd11: failed to heartbeat to: peer=f80a1c08-7986-b2ef-56a9-bc6e00b4c5d0 error="command timed out" +--- PASS: TestRaft_ApplyConcurrent (0.44s) === RUN TestRaft_ApplyConcurrent_Timeout -testLoggerAdapter verbose: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.675Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ca408a2-f45d-b934-ce21-049cc0a2f197 Address:7ca408a2-f45d-b934-ce21-049cc0a2f197}]" - testing.go:149: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.675Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ca408a2-f45d-b934-ce21-049cc0a2f197 Address:7ca408a2-f45d-b934-ce21-049cc0a2f197}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:55.675Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:55.675Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.675Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: entering follower state: follower="Node at 7ca408a2-f45d-b934-ce21-049cc0a2f197 [Follower]" leader= - testing.go:149: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.675Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: entering follower state: follower="Node at 7ca408a2-f45d-b934-ce21-049cc0a2f197 [Follower]" leader= -testLoggerAdapter verbose: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.856Z [WARN] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.856Z [WARN] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.856Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: entering candidate state: node="Node at 7ca408a2-f45d-b934-ce21-049cc0a2f197 [Candidate]" term=2 - testing.go:149: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.856Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: entering candidate state: node="Node at 7ca408a2-f45d-b934-ce21-049cc0a2f197 [Candidate]" term=2 -testLoggerAdapter verbose: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.856Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: election won: tally=1 - testing.go:149: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.856Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: election won: tally=1 -testLoggerAdapter verbose: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.856Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: entering leader state: leader="Node at 7ca408a2-f45d-b934-ce21-049cc0a2f197 [Leader]" - testing.go:149: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.856Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: entering leader state: leader="Node at 7ca408a2-f45d-b934-ce21-049cc0a2f197 [Leader]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.057Z [INFO] stable state for Leader reached at 2024-01-09 05:09:55.856495202 +0000 UTC m=+7.109546247 (1 nodes), 180.709198ms from start of poll, 180.951814ms from cluster start. Timeout at 2024-01-09 05:09:56.057627124 +0000 UTC m=+7.310678365, 201.132118ms after stability - testing.go:149: cluster: 2024-01-09T05:09:56.057Z [INFO] stable state for Leader reached at 2024-01-09 05:09:55.856495202 +0000 UTC m=+7.109546247 (1 nodes), 180.709198ms from start of poll, 180.951814ms from cluster start. Timeout at 2024-01-09 05:09:56.057627124 +0000 UTC m=+7.310678365, 201.132118ms after stability ---- PASS: TestRaft_ApplyConcurrent_Timeout (0.38s) +testLoggerAdapter verbose: server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: 2025-02-10T11:34:10.315Z [INFO] server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d169bdb-3864-ac4f-f013-4c94a790a8a0 Address:2d169bdb-3864-ac4f-f013-4c94a790a8a0}]" + testing.go:149: server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: 2025-02-10T11:34:10.315Z [INFO] server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d169bdb-3864-ac4f-f013-4c94a790a8a0 Address:2d169bdb-3864-ac4f-f013-4c94a790a8a0}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:10.316Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:10.316Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: 2025-02-10T11:34:10.316Z [INFO] server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: entering follower state: follower="Node at 2d169bdb-3864-ac4f-f013-4c94a790a8a0 [Follower]" leader= + testing.go:149: server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: 2025-02-10T11:34:10.316Z [INFO] server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: entering follower state: follower="Node at 2d169bdb-3864-ac4f-f013-4c94a790a8a0 [Follower]" leader= +testLoggerAdapter verbose: server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: 2025-02-10T11:34:10.431Z [WARN] server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: 2025-02-10T11:34:10.431Z [WARN] server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: 2025-02-10T11:34:10.431Z [INFO] server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: entering candidate state: node="Node at 2d169bdb-3864-ac4f-f013-4c94a790a8a0 [Candidate]" term=2 + testing.go:149: server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: 2025-02-10T11:34:10.431Z [INFO] server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: entering candidate state: node="Node at 2d169bdb-3864-ac4f-f013-4c94a790a8a0 [Candidate]" term=2 +testLoggerAdapter verbose: server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: 2025-02-10T11:34:10.431Z [INFO] server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: election won: tally=1 + testing.go:149: server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: 2025-02-10T11:34:10.431Z [INFO] server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: election won: tally=1 +testLoggerAdapter verbose: server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: 2025-02-10T11:34:10.431Z [INFO] server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: entering leader state: leader="Node at 2d169bdb-3864-ac4f-f013-4c94a790a8a0 [Leader]" + testing.go:149: server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: 2025-02-10T11:34:10.431Z [INFO] server-2d169bdb-3864-ac4f-f013-4c94a790a8a0: entering leader state: leader="Node at 2d169bdb-3864-ac4f-f013-4c94a790a8a0 [Leader]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:10.632Z [INFO] stable state for Leader reached at 2025-02-10 11:34:10.431892344 +0000 UTC m=+6.800796710 (1 nodes), 115.843764ms from start of poll, 116.007607ms from cluster start. Timeout at 2025-02-10 11:34:10.632965142 +0000 UTC m=+7.001869448, 201.072738ms after stability + testing.go:149: cluster: 2025-02-10T11:34:10.632Z [INFO] stable state for Leader reached at 2025-02-10 11:34:10.431892344 +0000 UTC m=+6.800796710 (1 nodes), 115.843764ms from start of poll, 116.007607ms from cluster start. Timeout at 2025-02-10 11:34:10.632965142 +0000 UTC m=+7.001869448, 201.072738ms after stability +--- PASS: TestRaft_ApplyConcurrent_Timeout (0.32s) === RUN TestRaft_JoinNode -testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.060Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3147839-f31a-e42c-3c68-b2f2bf175735 Address:c3147839-f31a-e42c-3c68-b2f2bf175735} {Suffrage:Voter ID:server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 Address:c3131786-6d4c-2e2e-0788-7fd8fa0b9b89}]" - testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.060Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3147839-f31a-e42c-3c68-b2f2bf175735 Address:c3147839-f31a-e42c-3c68-b2f2bf175735} {Suffrage:Voter ID:server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 Address:c3131786-6d4c-2e2e-0788-7fd8fa0b9b89}]" -testLoggerAdapter verbose: server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89: 2024-01-09T05:09:56.060Z [INFO] server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3147839-f31a-e42c-3c68-b2f2bf175735 Address:c3147839-f31a-e42c-3c68-b2f2bf175735} {Suffrage:Voter ID:server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 Address:c3131786-6d4c-2e2e-0788-7fd8fa0b9b89}]" - testing.go:149: server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89: 2024-01-09T05:09:56.060Z [INFO] server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3147839-f31a-e42c-3c68-b2f2bf175735 Address:c3147839-f31a-e42c-3c68-b2f2bf175735} {Suffrage:Voter ID:server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 Address:c3131786-6d4c-2e2e-0788-7fd8fa0b9b89}]" -testLoggerAdapter verbose: server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: 2024-01-09T05:09:56.061Z [INFO] server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: initial configuration: index=0 servers=[] - testing.go:149: server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: 2024-01-09T05:09:56.061Z [INFO] server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.061Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:56.061Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.061Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: entering follower state: follower="Node at c3147839-f31a-e42c-3c68-b2f2bf175735 [Follower]" leader= - testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.061Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: entering follower state: follower="Node at c3147839-f31a-e42c-3c68-b2f2bf175735 [Follower]" leader= -testLoggerAdapter verbose: server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: 2024-01-09T05:09:56.061Z [INFO] server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: entering follower state: follower="Node at eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 [Follower]" leader= -testLoggerAdapter verbose: server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89: 2024-01-09T05:09:56.061Z [INFO] server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89: entering follower state: follower="Node at c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 [Follower]" leader= - testing.go:149: server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: 2024-01-09T05:09:56.061Z [INFO] server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: entering follower state: follower="Node at eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 [Follower]" leader= - testing.go:149: server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89: 2024-01-09T05:09:56.061Z [INFO] server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89: entering follower state: follower="Node at c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 [Follower]" leader= -testLoggerAdapter verbose: server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: 2024-01-09T05:09:56.116Z [WARN] server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: no known peers, aborting election - testing.go:149: server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: 2024-01-09T05:09:56.116Z [WARN] server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: no known peers, aborting election -testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.124Z [WARN] server-c3147839-f31a-e42c-3c68-b2f2bf175735: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.124Z [WARN] server-c3147839-f31a-e42c-3c68-b2f2bf175735: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.124Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: entering candidate state: node="Node at c3147839-f31a-e42c-3c68-b2f2bf175735 [Candidate]" term=2 - testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.124Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: entering candidate state: node="Node at c3147839-f31a-e42c-3c68-b2f2bf175735 [Candidate]" term=2 -testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.125Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: election won: tally=2 - testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.125Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: election won: tally=2 -testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.125Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: entering leader state: leader="Node at c3147839-f31a-e42c-3c68-b2f2bf175735 [Leader]" - testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.125Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: entering leader state: leader="Node at c3147839-f31a-e42c-3c68-b2f2bf175735 [Leader]" -testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.125Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: added peer, starting replication: peer=server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 - testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.125Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: added peer, starting replication: peer=server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 -testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.125Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: pipelining replication: peer="{Voter server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 c3131786-6d4c-2e2e-0788-7fd8fa0b9b89}" - testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.125Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: pipelining replication: peer="{Voter server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 c3131786-6d4c-2e2e-0788-7fd8fa0b9b89}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.235Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.125291561 +0000 UTC m=+7.378343029 (1 nodes), 63.809808ms from start of poll, 65.170298ms from cluster start. Timeout at 2024-01-09 05:09:56.235075666 +0000 UTC m=+7.488126682, 109.783653ms after stability - testing.go:149: cluster: 2024-01-09T05:09:56.235Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.125291561 +0000 UTC m=+7.378343029 (1 nodes), 63.809808ms from start of poll, 65.170298ms from cluster start. Timeout at 2024-01-09 05:09:56.235075666 +0000 UTC m=+7.488126682, 109.783653ms after stability -testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.235Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: updating configuration: command=AddStaging server-id=server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 server-addr=eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 servers="[{Suffrage:Voter ID:server-c3147839-f31a-e42c-3c68-b2f2bf175735 Address:c3147839-f31a-e42c-3c68-b2f2bf175735} {Suffrage:Voter ID:server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 Address:c3131786-6d4c-2e2e-0788-7fd8fa0b9b89} {Suffrage:Voter ID:server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 Address:eaf7e51b-eef0-35ee-2498-9c3d8f4366c1}]" - testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.235Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: updating configuration: command=AddStaging server-id=server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 server-addr=eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 servers="[{Suffrage:Voter ID:server-c3147839-f31a-e42c-3c68-b2f2bf175735 Address:c3147839-f31a-e42c-3c68-b2f2bf175735} {Suffrage:Voter ID:server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 Address:c3131786-6d4c-2e2e-0788-7fd8fa0b9b89} {Suffrage:Voter ID:server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 Address:eaf7e51b-eef0-35ee-2498-9c3d8f4366c1}]" -testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.235Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: added peer, starting replication: peer=server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 - testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.235Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: added peer, starting replication: peer=server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 -testLoggerAdapter verbose: server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: 2024-01-09T05:09:56.235Z [WARN] server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: 2024-01-09T05:09:56.235Z [WARN] server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.235Z [WARN] server-c3147839-f31a-e42c-3c68-b2f2bf175735: appendEntries rejected, sending older logs: peer="{Voter server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 eaf7e51b-eef0-35ee-2498-9c3d8f4366c1}" next=1 - testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.235Z [WARN] server-c3147839-f31a-e42c-3c68-b2f2bf175735: appendEntries rejected, sending older logs: peer="{Voter server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 eaf7e51b-eef0-35ee-2498-9c3d8f4366c1}" next=1 -testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.235Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: pipelining replication: peer="{Voter server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 eaf7e51b-eef0-35ee-2498-9c3d8f4366c1}" - testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.235Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: pipelining replication: peer="{Voter server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 eaf7e51b-eef0-35ee-2498-9c3d8f4366c1}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.235Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:56.235Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.341Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.235776046 +0000 UTC m=+7.488834643 (1 nodes), 9.779µs from start of poll, 175.661912ms from cluster start. Timeout at 2024-01-09 05:09:56.341143553 +0000 UTC m=+7.594194585, 105.359942ms after stability - testing.go:149: cluster: 2024-01-09T05:09:56.341Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.235776046 +0000 UTC m=+7.488834643 (1 nodes), 9.779µs from start of poll, 175.661912ms from cluster start. Timeout at 2024-01-09 05:09:56.341143553 +0000 UTC m=+7.594194585, 105.359942ms after stability -testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.341Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: aborting pipeline replication: peer="{Voter server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 eaf7e51b-eef0-35ee-2498-9c3d8f4366c1}" - testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.341Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: aborting pipeline replication: peer="{Voter server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 eaf7e51b-eef0-35ee-2498-9c3d8f4366c1}" -testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.341Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: aborting pipeline replication: peer="{Voter server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 c3131786-6d4c-2e2e-0788-7fd8fa0b9b89}" - testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.341Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: aborting pipeline replication: peer="{Voter server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 c3131786-6d4c-2e2e-0788-7fd8fa0b9b89}" ---- PASS: TestRaft_JoinNode (0.28s) +testLoggerAdapter verbose: server-543b908b-aed1-0232-f10d-2d27521cd86e: 2025-02-10T11:34:10.633Z [INFO] server-543b908b-aed1-0232-f10d-2d27521cd86e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-543b908b-aed1-0232-f10d-2d27521cd86e Address:543b908b-aed1-0232-f10d-2d27521cd86e} {Suffrage:Voter ID:server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e Address:bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e}]" + testing.go:149: server-543b908b-aed1-0232-f10d-2d27521cd86e: 2025-02-10T11:34:10.633Z [INFO] server-543b908b-aed1-0232-f10d-2d27521cd86e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-543b908b-aed1-0232-f10d-2d27521cd86e Address:543b908b-aed1-0232-f10d-2d27521cd86e} {Suffrage:Voter ID:server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e Address:bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e}]" +testLoggerAdapter verbose: server-543b908b-aed1-0232-f10d-2d27521cd86e: 2025-02-10T11:34:10.633Z [INFO] server-543b908b-aed1-0232-f10d-2d27521cd86e: entering follower state: follower="Node at 543b908b-aed1-0232-f10d-2d27521cd86e [Follower]" leader= + testing.go:149: server-543b908b-aed1-0232-f10d-2d27521cd86e: 2025-02-10T11:34:10.633Z [INFO] server-543b908b-aed1-0232-f10d-2d27521cd86e: entering follower state: follower="Node at 543b908b-aed1-0232-f10d-2d27521cd86e [Follower]" leader= +testLoggerAdapter verbose: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.633Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-543b908b-aed1-0232-f10d-2d27521cd86e Address:543b908b-aed1-0232-f10d-2d27521cd86e} {Suffrage:Voter ID:server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e Address:bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e}]" + testing.go:149: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.633Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-543b908b-aed1-0232-f10d-2d27521cd86e Address:543b908b-aed1-0232-f10d-2d27521cd86e} {Suffrage:Voter ID:server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e Address:bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e}]" +testLoggerAdapter verbose: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.633Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: entering follower state: follower="Node at bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e [Follower]" leader= + testing.go:149: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.633Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: entering follower state: follower="Node at bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e [Follower]" leader= +testLoggerAdapter verbose: server-a125ff63-dc6d-ddfe-592a-7f0b1854e881: 2025-02-10T11:34:10.634Z [INFO] server-a125ff63-dc6d-ddfe-592a-7f0b1854e881: initial configuration: index=0 servers=[] + testing.go:149: server-a125ff63-dc6d-ddfe-592a-7f0b1854e881: 2025-02-10T11:34:10.634Z [INFO] server-a125ff63-dc6d-ddfe-592a-7f0b1854e881: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:10.634Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-a125ff63-dc6d-ddfe-592a-7f0b1854e881: 2025-02-10T11:34:10.634Z [INFO] server-a125ff63-dc6d-ddfe-592a-7f0b1854e881: entering follower state: follower="Node at a125ff63-dc6d-ddfe-592a-7f0b1854e881 [Follower]" leader= + testing.go:149: cluster: 2025-02-10T11:34:10.634Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-a125ff63-dc6d-ddfe-592a-7f0b1854e881: 2025-02-10T11:34:10.634Z [INFO] server-a125ff63-dc6d-ddfe-592a-7f0b1854e881: entering follower state: follower="Node at a125ff63-dc6d-ddfe-592a-7f0b1854e881 [Follower]" leader= +testLoggerAdapter verbose: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.692Z [WARN] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.692Z [WARN] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.692Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: entering candidate state: node="Node at bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e [Candidate]" term=2 + testing.go:149: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.692Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: entering candidate state: node="Node at bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e [Candidate]" term=2 +testLoggerAdapter verbose: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.692Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: election won: tally=2 + testing.go:149: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.692Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: election won: tally=2 +testLoggerAdapter verbose: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.692Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: entering leader state: leader="Node at bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e [Leader]" + testing.go:149: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.692Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: entering leader state: leader="Node at bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e [Leader]" +testLoggerAdapter verbose: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.692Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: added peer, starting replication: peer=server-543b908b-aed1-0232-f10d-2d27521cd86e + testing.go:149: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.692Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: added peer, starting replication: peer=server-543b908b-aed1-0232-f10d-2d27521cd86e +testLoggerAdapter verbose: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.692Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: pipelining replication: peer="{Voter server-543b908b-aed1-0232-f10d-2d27521cd86e 543b908b-aed1-0232-f10d-2d27521cd86e}" + testing.go:149: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.692Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: pipelining replication: peer="{Voter server-543b908b-aed1-0232-f10d-2d27521cd86e 543b908b-aed1-0232-f10d-2d27521cd86e}" +testLoggerAdapter verbose: server-a125ff63-dc6d-ddfe-592a-7f0b1854e881: 2025-02-10T11:34:10.701Z [WARN] server-a125ff63-dc6d-ddfe-592a-7f0b1854e881: no known peers, aborting election + testing.go:149: server-a125ff63-dc6d-ddfe-592a-7f0b1854e881: 2025-02-10T11:34:10.701Z [WARN] server-a125ff63-dc6d-ddfe-592a-7f0b1854e881: no known peers, aborting election +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:10.797Z [INFO] stable state for Leader reached at 2025-02-10 11:34:10.692601454 +0000 UTC m=+7.061505758 (1 nodes), 58.466846ms from start of poll, 59.044767ms from cluster start. Timeout at 2025-02-10 11:34:10.797704606 +0000 UTC m=+7.166608952, 105.103194ms after stability + testing.go:149: cluster: 2025-02-10T11:34:10.797Z [INFO] stable state for Leader reached at 2025-02-10 11:34:10.692601454 +0000 UTC m=+7.061505758 (1 nodes), 58.466846ms from start of poll, 59.044767ms from cluster start. Timeout at 2025-02-10 11:34:10.797704606 +0000 UTC m=+7.166608952, 105.103194ms after stability +testLoggerAdapter verbose: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.797Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: updating configuration: command=AddStaging server-id=server-a125ff63-dc6d-ddfe-592a-7f0b1854e881 server-addr=a125ff63-dc6d-ddfe-592a-7f0b1854e881 servers="[{Suffrage:Voter ID:server-543b908b-aed1-0232-f10d-2d27521cd86e Address:543b908b-aed1-0232-f10d-2d27521cd86e} {Suffrage:Voter ID:server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e Address:bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e} {Suffrage:Voter ID:server-a125ff63-dc6d-ddfe-592a-7f0b1854e881 Address:a125ff63-dc6d-ddfe-592a-7f0b1854e881}]" + testing.go:149: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.797Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: updating configuration: command=AddStaging server-id=server-a125ff63-dc6d-ddfe-592a-7f0b1854e881 server-addr=a125ff63-dc6d-ddfe-592a-7f0b1854e881 servers="[{Suffrage:Voter ID:server-543b908b-aed1-0232-f10d-2d27521cd86e Address:543b908b-aed1-0232-f10d-2d27521cd86e} {Suffrage:Voter ID:server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e Address:bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e} {Suffrage:Voter ID:server-a125ff63-dc6d-ddfe-592a-7f0b1854e881 Address:a125ff63-dc6d-ddfe-592a-7f0b1854e881}]" +testLoggerAdapter verbose: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.798Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: added peer, starting replication: peer=server-a125ff63-dc6d-ddfe-592a-7f0b1854e881 + testing.go:149: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.798Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: added peer, starting replication: peer=server-a125ff63-dc6d-ddfe-592a-7f0b1854e881 +testLoggerAdapter verbose: server-a125ff63-dc6d-ddfe-592a-7f0b1854e881: 2025-02-10T11:34:10.798Z [WARN] server-a125ff63-dc6d-ddfe-592a-7f0b1854e881: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-a125ff63-dc6d-ddfe-592a-7f0b1854e881: 2025-02-10T11:34:10.798Z [WARN] server-a125ff63-dc6d-ddfe-592a-7f0b1854e881: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.798Z [WARN] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: appendEntries rejected, sending older logs: peer="{Voter server-a125ff63-dc6d-ddfe-592a-7f0b1854e881 a125ff63-dc6d-ddfe-592a-7f0b1854e881}" next=1 + testing.go:149: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.798Z [WARN] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: appendEntries rejected, sending older logs: peer="{Voter server-a125ff63-dc6d-ddfe-592a-7f0b1854e881 a125ff63-dc6d-ddfe-592a-7f0b1854e881}" next=1 +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:10.798Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:10.798Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.798Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: pipelining replication: peer="{Voter server-a125ff63-dc6d-ddfe-592a-7f0b1854e881 a125ff63-dc6d-ddfe-592a-7f0b1854e881}" + testing.go:149: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.798Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: pipelining replication: peer="{Voter server-a125ff63-dc6d-ddfe-592a-7f0b1854e881 a125ff63-dc6d-ddfe-592a-7f0b1854e881}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:10.903Z [INFO] stable state for Leader reached at 2025-02-10 11:34:10.798185654 +0000 UTC m=+7.167089955 (1 nodes), 378ns from start of poll, 164.628964ms from cluster start. Timeout at 2025-02-10 11:34:10.903258966 +0000 UTC m=+7.272163259, 105.073304ms after stability + testing.go:149: cluster: 2025-02-10T11:34:10.903Z [INFO] stable state for Leader reached at 2025-02-10 11:34:10.798185654 +0000 UTC m=+7.167089955 (1 nodes), 378ns from start of poll, 164.628964ms from cluster start. Timeout at 2025-02-10 11:34:10.903258966 +0000 UTC m=+7.272163259, 105.073304ms after stability +testLoggerAdapter verbose: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.903Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: aborting pipeline replication: peer="{Voter server-a125ff63-dc6d-ddfe-592a-7f0b1854e881 a125ff63-dc6d-ddfe-592a-7f0b1854e881}" + testing.go:149: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.903Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: aborting pipeline replication: peer="{Voter server-a125ff63-dc6d-ddfe-592a-7f0b1854e881 a125ff63-dc6d-ddfe-592a-7f0b1854e881}" +testLoggerAdapter verbose: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.903Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: aborting pipeline replication: peer="{Voter server-543b908b-aed1-0232-f10d-2d27521cd86e 543b908b-aed1-0232-f10d-2d27521cd86e}" + testing.go:149: server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: 2025-02-10T11:34:10.903Z [INFO] server-bcfc8e4a-05d0-5c4d-adce-b1a181e5dc9e: aborting pipeline replication: peer="{Voter server-543b908b-aed1-0232-f10d-2d27521cd86e 543b908b-aed1-0232-f10d-2d27521cd86e}" +--- PASS: TestRaft_JoinNode (0.27s) === RUN TestRaft_JoinNode_ConfigStore -testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.343Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97371e1f-22e0-76af-4a81-5afa5cfd8315 Address:97371e1f-22e0-76af-4a81-5afa5cfd8315}]" - testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.343Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97371e1f-22e0-76af-4a81-5afa5cfd8315 Address:97371e1f-22e0-76af-4a81-5afa5cfd8315}]" -testLoggerAdapter verbose: server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: 2024-01-09T05:09:56.343Z [INFO] server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: initial configuration: index=0 servers=[] - testing.go:149: server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: 2024-01-09T05:09:56.343Z [INFO] server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: 2024-01-09T05:09:56.343Z [INFO] server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: initial configuration: index=0 servers=[] - testing.go:149: server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: 2024-01-09T05:09:56.343Z [INFO] server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.343Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:56.343Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.343Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: entering follower state: follower="Node at 97371e1f-22e0-76af-4a81-5afa5cfd8315 [Follower]" leader= - testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.343Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: entering follower state: follower="Node at 97371e1f-22e0-76af-4a81-5afa5cfd8315 [Follower]" leader= -testLoggerAdapter verbose: server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: 2024-01-09T05:09:56.343Z [INFO] server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: entering follower state: follower="Node at b1d2a2c9-ea13-4c34-4d91-d26b22449f69 [Follower]" leader= - testing.go:149: server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: 2024-01-09T05:09:56.343Z [INFO] server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: entering follower state: follower="Node at b1d2a2c9-ea13-4c34-4d91-d26b22449f69 [Follower]" leader= -testLoggerAdapter verbose: server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: 2024-01-09T05:09:56.343Z [INFO] server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: entering follower state: follower="Node at cdb7c38b-a9bd-110e-c6fd-d856526f9d2f [Follower]" leader= - testing.go:149: server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: 2024-01-09T05:09:56.343Z [INFO] server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: entering follower state: follower="Node at cdb7c38b-a9bd-110e-c6fd-d856526f9d2f [Follower]" leader= -testLoggerAdapter verbose: server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: 2024-01-09T05:09:56.409Z [WARN] server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: no known peers, aborting election - testing.go:149: server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: 2024-01-09T05:09:56.409Z [WARN] server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: no known peers, aborting election -testLoggerAdapter verbose: server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: 2024-01-09T05:09:56.422Z [WARN] server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: no known peers, aborting election - testing.go:149: server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: 2024-01-09T05:09:56.422Z [WARN] server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: no known peers, aborting election -testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.433Z [WARN] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.433Z [WARN] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.433Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: entering candidate state: node="Node at 97371e1f-22e0-76af-4a81-5afa5cfd8315 [Candidate]" term=2 - testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.433Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: entering candidate state: node="Node at 97371e1f-22e0-76af-4a81-5afa5cfd8315 [Candidate]" term=2 -testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.433Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: election won: tally=1 - testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.433Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: election won: tally=1 -testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.433Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: entering leader state: leader="Node at 97371e1f-22e0-76af-4a81-5afa5cfd8315 [Leader]" - testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.433Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: entering leader state: leader="Node at 97371e1f-22e0-76af-4a81-5afa5cfd8315 [Leader]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.538Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.43364299 +0000 UTC m=+7.686694602 (1 nodes), 89.816074ms from start of poll, 90.73522ms from cluster start. Timeout at 2024-01-09 05:09:56.538766278 +0000 UTC m=+7.791817515, 105.122913ms after stability - testing.go:149: cluster: 2024-01-09T05:09:56.538Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.43364299 +0000 UTC m=+7.686694602 (1 nodes), 89.816074ms from start of poll, 90.73522ms from cluster start. Timeout at 2024-01-09 05:09:56.538766278 +0000 UTC m=+7.791817515, 105.122913ms after stability -testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.538Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: updating configuration: command=AddStaging server-id=server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 server-addr=b1d2a2c9-ea13-4c34-4d91-d26b22449f69 servers="[{Suffrage:Voter ID:server-97371e1f-22e0-76af-4a81-5afa5cfd8315 Address:97371e1f-22e0-76af-4a81-5afa5cfd8315} {Suffrage:Voter ID:server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 Address:b1d2a2c9-ea13-4c34-4d91-d26b22449f69}]" - testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.538Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: updating configuration: command=AddStaging server-id=server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 server-addr=b1d2a2c9-ea13-4c34-4d91-d26b22449f69 servers="[{Suffrage:Voter ID:server-97371e1f-22e0-76af-4a81-5afa5cfd8315 Address:97371e1f-22e0-76af-4a81-5afa5cfd8315} {Suffrage:Voter ID:server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 Address:b1d2a2c9-ea13-4c34-4d91-d26b22449f69}]" -testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.539Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: added peer, starting replication: peer=server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 - testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.539Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: added peer, starting replication: peer=server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.539Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:56.539Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: 2024-01-09T05:09:56.539Z [WARN] server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: 2024-01-09T05:09:56.539Z [WARN] server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.539Z [WARN] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: appendEntries rejected, sending older logs: peer="{Voter server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 b1d2a2c9-ea13-4c34-4d91-d26b22449f69}" next=1 - testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.539Z [WARN] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: appendEntries rejected, sending older logs: peer="{Voter server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 b1d2a2c9-ea13-4c34-4d91-d26b22449f69}" next=1 -testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.539Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: pipelining replication: peer="{Voter server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 b1d2a2c9-ea13-4c34-4d91-d26b22449f69}" - testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.539Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: pipelining replication: peer="{Voter server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 b1d2a2c9-ea13-4c34-4d91-d26b22449f69}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.644Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.539227581 +0000 UTC m=+7.792278707 (1 nodes), 3.752µs from start of poll, 196.319325ms from cluster start. Timeout at 2024-01-09 05:09:56.644333904 +0000 UTC m=+7.897385032, 105.106325ms after stability - testing.go:149: cluster: 2024-01-09T05:09:56.644Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.539227581 +0000 UTC m=+7.792278707 (1 nodes), 3.752µs from start of poll, 196.319325ms from cluster start. Timeout at 2024-01-09 05:09:56.644333904 +0000 UTC m=+7.897385032, 105.106325ms after stability -testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.644Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: updating configuration: command=AddStaging server-id=server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f server-addr=cdb7c38b-a9bd-110e-c6fd-d856526f9d2f servers="[{Suffrage:Voter ID:server-97371e1f-22e0-76af-4a81-5afa5cfd8315 Address:97371e1f-22e0-76af-4a81-5afa5cfd8315} {Suffrage:Voter ID:server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 Address:b1d2a2c9-ea13-4c34-4d91-d26b22449f69} {Suffrage:Voter ID:server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f Address:cdb7c38b-a9bd-110e-c6fd-d856526f9d2f}]" - testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.644Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: updating configuration: command=AddStaging server-id=server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f server-addr=cdb7c38b-a9bd-110e-c6fd-d856526f9d2f servers="[{Suffrage:Voter ID:server-97371e1f-22e0-76af-4a81-5afa5cfd8315 Address:97371e1f-22e0-76af-4a81-5afa5cfd8315} {Suffrage:Voter ID:server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 Address:b1d2a2c9-ea13-4c34-4d91-d26b22449f69} {Suffrage:Voter ID:server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f Address:cdb7c38b-a9bd-110e-c6fd-d856526f9d2f}]" -testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.644Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: added peer, starting replication: peer=server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f - testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.644Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: added peer, starting replication: peer=server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f -testLoggerAdapter verbose: server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: 2024-01-09T05:09:56.644Z [WARN] server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: failed to get previous log: previous-index=4 last-index=0 error="log not found" - testing.go:149: server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: 2024-01-09T05:09:56.644Z [WARN] server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: failed to get previous log: previous-index=4 last-index=0 error="log not found" -testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.644Z [WARN] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: appendEntries rejected, sending older logs: peer="{Voter server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f cdb7c38b-a9bd-110e-c6fd-d856526f9d2f}" next=1 - testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.644Z [WARN] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: appendEntries rejected, sending older logs: peer="{Voter server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f cdb7c38b-a9bd-110e-c6fd-d856526f9d2f}" next=1 -testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.644Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: pipelining replication: peer="{Voter server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f cdb7c38b-a9bd-110e-c6fd-d856526f9d2f}" - testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.644Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: pipelining replication: peer="{Voter server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f cdb7c38b-a9bd-110e-c6fd-d856526f9d2f}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.645Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:56.645Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.755Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.645113715 +0000 UTC m=+7.898164838 (1 nodes), 2.552µs from start of poll, 302.205456ms from cluster start. Timeout at 2024-01-09 05:09:56.755051989 +0000 UTC m=+8.008102985, 109.938147ms after stability - testing.go:149: cluster: 2024-01-09T05:09:56.755Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.645113715 +0000 UTC m=+7.898164838 (1 nodes), 2.552µs from start of poll, 302.205456ms from cluster start. Timeout at 2024-01-09 05:09:56.755051989 +0000 UTC m=+8.008102985, 109.938147ms after stability -testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.755Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: aborting pipeline replication: peer="{Voter server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f cdb7c38b-a9bd-110e-c6fd-d856526f9d2f}" - testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.755Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: aborting pipeline replication: peer="{Voter server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f cdb7c38b-a9bd-110e-c6fd-d856526f9d2f}" -testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.755Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: aborting pipeline replication: peer="{Voter server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 b1d2a2c9-ea13-4c34-4d91-d26b22449f69}" - testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.755Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: aborting pipeline replication: peer="{Voter server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 b1d2a2c9-ea13-4c34-4d91-d26b22449f69}" ---- PASS: TestRaft_JoinNode_ConfigStore (0.41s) +testLoggerAdapter verbose: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:10.904Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff6b7497-89ec-f2d7-c906-c5218d0fc273 Address:ff6b7497-89ec-f2d7-c906-c5218d0fc273}]" + testing.go:149: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:10.904Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff6b7497-89ec-f2d7-c906-c5218d0fc273 Address:ff6b7497-89ec-f2d7-c906-c5218d0fc273}]" +testLoggerAdapter verbose: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:10.904Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: entering follower state: follower="Node at ff6b7497-89ec-f2d7-c906-c5218d0fc273 [Follower]" leader= + testing.go:149: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:10.904Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: entering follower state: follower="Node at ff6b7497-89ec-f2d7-c906-c5218d0fc273 [Follower]" leader= +testLoggerAdapter verbose: server-011d2ffb-116f-044e-01a7-99293021b44a: 2025-02-10T11:34:10.904Z [INFO] server-011d2ffb-116f-044e-01a7-99293021b44a: initial configuration: index=0 servers=[] + testing.go:149: server-011d2ffb-116f-044e-01a7-99293021b44a: 2025-02-10T11:34:10.904Z [INFO] server-011d2ffb-116f-044e-01a7-99293021b44a: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-011d2ffb-116f-044e-01a7-99293021b44a: 2025-02-10T11:34:10.904Z [INFO] server-011d2ffb-116f-044e-01a7-99293021b44a: entering follower state: follower="Node at 011d2ffb-116f-044e-01a7-99293021b44a [Follower]" leader= + testing.go:149: server-011d2ffb-116f-044e-01a7-99293021b44a: 2025-02-10T11:34:10.904Z [INFO] server-011d2ffb-116f-044e-01a7-99293021b44a: entering follower state: follower="Node at 011d2ffb-116f-044e-01a7-99293021b44a [Follower]" leader= +testLoggerAdapter verbose: server-80c460e7-ade7-622a-cbb2-4ecf276387c2: 2025-02-10T11:34:10.904Z [INFO] server-80c460e7-ade7-622a-cbb2-4ecf276387c2: initial configuration: index=0 servers=[] + testing.go:149: server-80c460e7-ade7-622a-cbb2-4ecf276387c2: 2025-02-10T11:34:10.904Z [INFO] server-80c460e7-ade7-622a-cbb2-4ecf276387c2: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:10.904Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-80c460e7-ade7-622a-cbb2-4ecf276387c2: 2025-02-10T11:34:10.904Z [INFO] server-80c460e7-ade7-622a-cbb2-4ecf276387c2: entering follower state: follower="Node at 80c460e7-ade7-622a-cbb2-4ecf276387c2 [Follower]" leader= + testing.go:149: cluster: 2025-02-10T11:34:10.904Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-80c460e7-ade7-622a-cbb2-4ecf276387c2: 2025-02-10T11:34:10.904Z [INFO] server-80c460e7-ade7-622a-cbb2-4ecf276387c2: entering follower state: follower="Node at 80c460e7-ade7-622a-cbb2-4ecf276387c2 [Follower]" leader= +testLoggerAdapter verbose: server-80c460e7-ade7-622a-cbb2-4ecf276387c2: 2025-02-10T11:34:10.976Z [WARN] server-80c460e7-ade7-622a-cbb2-4ecf276387c2: no known peers, aborting election + testing.go:149: server-80c460e7-ade7-622a-cbb2-4ecf276387c2: 2025-02-10T11:34:10.976Z [WARN] server-80c460e7-ade7-622a-cbb2-4ecf276387c2: no known peers, aborting election +testLoggerAdapter verbose: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.003Z [WARN] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.003Z [WARN] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.003Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: entering candidate state: node="Node at ff6b7497-89ec-f2d7-c906-c5218d0fc273 [Candidate]" term=2 +testLoggerAdapter verbose: server-011d2ffb-116f-044e-01a7-99293021b44a: 2025-02-10T11:34:11.003Z [WARN] server-011d2ffb-116f-044e-01a7-99293021b44a: no known peers, aborting election + testing.go:149: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.003Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: entering candidate state: node="Node at ff6b7497-89ec-f2d7-c906-c5218d0fc273 [Candidate]" term=2 +testLoggerAdapter verbose: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.003Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: election won: tally=1 + testing.go:149: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.003Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: election won: tally=1 +testLoggerAdapter verbose: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.003Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: entering leader state: leader="Node at ff6b7497-89ec-f2d7-c906-c5218d0fc273 [Leader]" + testing.go:149: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.003Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: entering leader state: leader="Node at ff6b7497-89ec-f2d7-c906-c5218d0fc273 [Leader]" + testing.go:149: server-011d2ffb-116f-044e-01a7-99293021b44a: 2025-02-10T11:34:11.003Z [WARN] server-011d2ffb-116f-044e-01a7-99293021b44a: no known peers, aborting election +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:11.108Z [INFO] stable state for Leader reached at 2025-02-10 11:34:11.003244339 +0000 UTC m=+7.372148670 (1 nodes), 98.520108ms from start of poll, 99.127421ms from cluster start. Timeout at 2025-02-10 11:34:11.108322021 +0000 UTC m=+7.477226317, 105.077647ms after stability + testing.go:149: cluster: 2025-02-10T11:34:11.108Z [INFO] stable state for Leader reached at 2025-02-10 11:34:11.003244339 +0000 UTC m=+7.372148670 (1 nodes), 98.520108ms from start of poll, 99.127421ms from cluster start. Timeout at 2025-02-10 11:34:11.108322021 +0000 UTC m=+7.477226317, 105.077647ms after stability +testLoggerAdapter verbose: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.108Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: updating configuration: command=AddStaging server-id=server-011d2ffb-116f-044e-01a7-99293021b44a server-addr=011d2ffb-116f-044e-01a7-99293021b44a servers="[{Suffrage:Voter ID:server-ff6b7497-89ec-f2d7-c906-c5218d0fc273 Address:ff6b7497-89ec-f2d7-c906-c5218d0fc273} {Suffrage:Voter ID:server-011d2ffb-116f-044e-01a7-99293021b44a Address:011d2ffb-116f-044e-01a7-99293021b44a}]" + testing.go:149: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.108Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: updating configuration: command=AddStaging server-id=server-011d2ffb-116f-044e-01a7-99293021b44a server-addr=011d2ffb-116f-044e-01a7-99293021b44a servers="[{Suffrage:Voter ID:server-ff6b7497-89ec-f2d7-c906-c5218d0fc273 Address:ff6b7497-89ec-f2d7-c906-c5218d0fc273} {Suffrage:Voter ID:server-011d2ffb-116f-044e-01a7-99293021b44a Address:011d2ffb-116f-044e-01a7-99293021b44a}]" +testLoggerAdapter verbose: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.108Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: added peer, starting replication: peer=server-011d2ffb-116f-044e-01a7-99293021b44a + testing.go:149: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.108Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: added peer, starting replication: peer=server-011d2ffb-116f-044e-01a7-99293021b44a +testLoggerAdapter verbose: server-011d2ffb-116f-044e-01a7-99293021b44a: 2025-02-10T11:34:11.108Z [WARN] server-011d2ffb-116f-044e-01a7-99293021b44a: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-011d2ffb-116f-044e-01a7-99293021b44a: 2025-02-10T11:34:11.108Z [WARN] server-011d2ffb-116f-044e-01a7-99293021b44a: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.108Z [WARN] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: appendEntries rejected, sending older logs: peer="{Voter server-011d2ffb-116f-044e-01a7-99293021b44a 011d2ffb-116f-044e-01a7-99293021b44a}" next=1 + testing.go:149: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.108Z [WARN] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: appendEntries rejected, sending older logs: peer="{Voter server-011d2ffb-116f-044e-01a7-99293021b44a 011d2ffb-116f-044e-01a7-99293021b44a}" next=1 +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:11.108Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:11.108Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.108Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: pipelining replication: peer="{Voter server-011d2ffb-116f-044e-01a7-99293021b44a 011d2ffb-116f-044e-01a7-99293021b44a}" + testing.go:149: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.108Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: pipelining replication: peer="{Voter server-011d2ffb-116f-044e-01a7-99293021b44a 011d2ffb-116f-044e-01a7-99293021b44a}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:11.213Z [INFO] stable state for Leader reached at 2025-02-10 11:34:11.10865151 +0000 UTC m=+7.477555836 (1 nodes), 669ns from start of poll, 204.534587ms from cluster start. Timeout at 2025-02-10 11:34:11.213719642 +0000 UTC m=+7.582623939, 105.068103ms after stability + testing.go:149: cluster: 2025-02-10T11:34:11.213Z [INFO] stable state for Leader reached at 2025-02-10 11:34:11.10865151 +0000 UTC m=+7.477555836 (1 nodes), 669ns from start of poll, 204.534587ms from cluster start. Timeout at 2025-02-10 11:34:11.213719642 +0000 UTC m=+7.582623939, 105.068103ms after stability +testLoggerAdapter verbose: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.213Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: updating configuration: command=AddStaging server-id=server-80c460e7-ade7-622a-cbb2-4ecf276387c2 server-addr=80c460e7-ade7-622a-cbb2-4ecf276387c2 servers="[{Suffrage:Voter ID:server-ff6b7497-89ec-f2d7-c906-c5218d0fc273 Address:ff6b7497-89ec-f2d7-c906-c5218d0fc273} {Suffrage:Voter ID:server-011d2ffb-116f-044e-01a7-99293021b44a Address:011d2ffb-116f-044e-01a7-99293021b44a} {Suffrage:Voter ID:server-80c460e7-ade7-622a-cbb2-4ecf276387c2 Address:80c460e7-ade7-622a-cbb2-4ecf276387c2}]" + testing.go:149: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.213Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: updating configuration: command=AddStaging server-id=server-80c460e7-ade7-622a-cbb2-4ecf276387c2 server-addr=80c460e7-ade7-622a-cbb2-4ecf276387c2 servers="[{Suffrage:Voter ID:server-ff6b7497-89ec-f2d7-c906-c5218d0fc273 Address:ff6b7497-89ec-f2d7-c906-c5218d0fc273} {Suffrage:Voter ID:server-011d2ffb-116f-044e-01a7-99293021b44a Address:011d2ffb-116f-044e-01a7-99293021b44a} {Suffrage:Voter ID:server-80c460e7-ade7-622a-cbb2-4ecf276387c2 Address:80c460e7-ade7-622a-cbb2-4ecf276387c2}]" +testLoggerAdapter verbose: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.213Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: added peer, starting replication: peer=server-80c460e7-ade7-622a-cbb2-4ecf276387c2 + testing.go:149: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.213Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: added peer, starting replication: peer=server-80c460e7-ade7-622a-cbb2-4ecf276387c2 +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:11.214Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-80c460e7-ade7-622a-cbb2-4ecf276387c2: 2025-02-10T11:34:11.214Z [WARN] server-80c460e7-ade7-622a-cbb2-4ecf276387c2: failed to get previous log: previous-index=4 last-index=0 error="log not found" + testing.go:149: cluster: 2025-02-10T11:34:11.214Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-80c460e7-ade7-622a-cbb2-4ecf276387c2: 2025-02-10T11:34:11.214Z [WARN] server-80c460e7-ade7-622a-cbb2-4ecf276387c2: failed to get previous log: previous-index=4 last-index=0 error="log not found" +testLoggerAdapter verbose: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.214Z [WARN] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: appendEntries rejected, sending older logs: peer="{Voter server-80c460e7-ade7-622a-cbb2-4ecf276387c2 80c460e7-ade7-622a-cbb2-4ecf276387c2}" next=1 + testing.go:149: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.214Z [WARN] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: appendEntries rejected, sending older logs: peer="{Voter server-80c460e7-ade7-622a-cbb2-4ecf276387c2 80c460e7-ade7-622a-cbb2-4ecf276387c2}" next=1 +testLoggerAdapter verbose: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.214Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: pipelining replication: peer="{Voter server-80c460e7-ade7-622a-cbb2-4ecf276387c2 80c460e7-ade7-622a-cbb2-4ecf276387c2}" + testing.go:149: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.214Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: pipelining replication: peer="{Voter server-80c460e7-ade7-622a-cbb2-4ecf276387c2 80c460e7-ade7-622a-cbb2-4ecf276387c2}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:11.319Z [INFO] stable state for Leader reached at 2025-02-10 11:34:11.214047749 +0000 UTC m=+7.582952048 (1 nodes), 537ns from start of poll, 309.930799ms from cluster start. Timeout at 2025-02-10 11:34:11.319120564 +0000 UTC m=+7.688024859, 105.072811ms after stability + testing.go:149: cluster: 2025-02-10T11:34:11.319Z [INFO] stable state for Leader reached at 2025-02-10 11:34:11.214047749 +0000 UTC m=+7.582952048 (1 nodes), 537ns from start of poll, 309.930799ms from cluster start. Timeout at 2025-02-10 11:34:11.319120564 +0000 UTC m=+7.688024859, 105.072811ms after stability +testLoggerAdapter verbose: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.319Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: aborting pipeline replication: peer="{Voter server-80c460e7-ade7-622a-cbb2-4ecf276387c2 80c460e7-ade7-622a-cbb2-4ecf276387c2}" + testing.go:149: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.319Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: aborting pipeline replication: peer="{Voter server-80c460e7-ade7-622a-cbb2-4ecf276387c2 80c460e7-ade7-622a-cbb2-4ecf276387c2}" +testLoggerAdapter verbose: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.319Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: aborting pipeline replication: peer="{Voter server-011d2ffb-116f-044e-01a7-99293021b44a 011d2ffb-116f-044e-01a7-99293021b44a}" + testing.go:149: server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: 2025-02-10T11:34:11.319Z [INFO] server-ff6b7497-89ec-f2d7-c906-c5218d0fc273: aborting pipeline replication: peer="{Voter server-011d2ffb-116f-044e-01a7-99293021b44a 011d2ffb-116f-044e-01a7-99293021b44a}" +--- PASS: TestRaft_JoinNode_ConfigStore (0.42s) === RUN TestRaft_RemoveFollower -testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.757Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c93f94-7b17-9861-cbac-b88b06a3a827 Address:25c93f94-7b17-9861-cbac-b88b06a3a827} {Suffrage:Voter ID:server-e591c761-60ba-b3d1-bba1-3608c0fde319 Address:e591c761-60ba-b3d1-bba1-3608c0fde319} {Suffrage:Voter ID:server-a3ddce4b-7390-67a1-253c-807365b0096c Address:a3ddce4b-7390-67a1-253c-807365b0096c}]" - testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.757Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c93f94-7b17-9861-cbac-b88b06a3a827 Address:25c93f94-7b17-9861-cbac-b88b06a3a827} {Suffrage:Voter ID:server-e591c761-60ba-b3d1-bba1-3608c0fde319 Address:e591c761-60ba-b3d1-bba1-3608c0fde319} {Suffrage:Voter ID:server-a3ddce4b-7390-67a1-253c-807365b0096c Address:a3ddce4b-7390-67a1-253c-807365b0096c}]" -testLoggerAdapter verbose: server-e591c761-60ba-b3d1-bba1-3608c0fde319: 2024-01-09T05:09:56.757Z [INFO] server-e591c761-60ba-b3d1-bba1-3608c0fde319: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c93f94-7b17-9861-cbac-b88b06a3a827 Address:25c93f94-7b17-9861-cbac-b88b06a3a827} {Suffrage:Voter ID:server-e591c761-60ba-b3d1-bba1-3608c0fde319 Address:e591c761-60ba-b3d1-bba1-3608c0fde319} {Suffrage:Voter ID:server-a3ddce4b-7390-67a1-253c-807365b0096c Address:a3ddce4b-7390-67a1-253c-807365b0096c}]" - testing.go:149: server-e591c761-60ba-b3d1-bba1-3608c0fde319: 2024-01-09T05:09:56.757Z [INFO] server-e591c761-60ba-b3d1-bba1-3608c0fde319: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c93f94-7b17-9861-cbac-b88b06a3a827 Address:25c93f94-7b17-9861-cbac-b88b06a3a827} {Suffrage:Voter ID:server-e591c761-60ba-b3d1-bba1-3608c0fde319 Address:e591c761-60ba-b3d1-bba1-3608c0fde319} {Suffrage:Voter ID:server-a3ddce4b-7390-67a1-253c-807365b0096c Address:a3ddce4b-7390-67a1-253c-807365b0096c}]" -testLoggerAdapter verbose: server-a3ddce4b-7390-67a1-253c-807365b0096c: 2024-01-09T05:09:56.757Z [INFO] server-a3ddce4b-7390-67a1-253c-807365b0096c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c93f94-7b17-9861-cbac-b88b06a3a827 Address:25c93f94-7b17-9861-cbac-b88b06a3a827} {Suffrage:Voter ID:server-e591c761-60ba-b3d1-bba1-3608c0fde319 Address:e591c761-60ba-b3d1-bba1-3608c0fde319} {Suffrage:Voter ID:server-a3ddce4b-7390-67a1-253c-807365b0096c Address:a3ddce4b-7390-67a1-253c-807365b0096c}]" - testing.go:149: server-a3ddce4b-7390-67a1-253c-807365b0096c: 2024-01-09T05:09:56.757Z [INFO] server-a3ddce4b-7390-67a1-253c-807365b0096c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c93f94-7b17-9861-cbac-b88b06a3a827 Address:25c93f94-7b17-9861-cbac-b88b06a3a827} {Suffrage:Voter ID:server-e591c761-60ba-b3d1-bba1-3608c0fde319 Address:e591c761-60ba-b3d1-bba1-3608c0fde319} {Suffrage:Voter ID:server-a3ddce4b-7390-67a1-253c-807365b0096c Address:a3ddce4b-7390-67a1-253c-807365b0096c}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.757Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:56.757Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.757Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: entering follower state: follower="Node at 25c93f94-7b17-9861-cbac-b88b06a3a827 [Follower]" leader= - testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.757Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: entering follower state: follower="Node at 25c93f94-7b17-9861-cbac-b88b06a3a827 [Follower]" leader= -testLoggerAdapter verbose: server-e591c761-60ba-b3d1-bba1-3608c0fde319: 2024-01-09T05:09:56.757Z [INFO] server-e591c761-60ba-b3d1-bba1-3608c0fde319: entering follower state: follower="Node at e591c761-60ba-b3d1-bba1-3608c0fde319 [Follower]" leader= - testing.go:149: server-e591c761-60ba-b3d1-bba1-3608c0fde319: 2024-01-09T05:09:56.757Z [INFO] server-e591c761-60ba-b3d1-bba1-3608c0fde319: entering follower state: follower="Node at e591c761-60ba-b3d1-bba1-3608c0fde319 [Follower]" leader= -testLoggerAdapter verbose: server-a3ddce4b-7390-67a1-253c-807365b0096c: 2024-01-09T05:09:56.757Z [INFO] server-a3ddce4b-7390-67a1-253c-807365b0096c: entering follower state: follower="Node at a3ddce4b-7390-67a1-253c-807365b0096c [Follower]" leader= - testing.go:149: server-a3ddce4b-7390-67a1-253c-807365b0096c: 2024-01-09T05:09:56.757Z [INFO] server-a3ddce4b-7390-67a1-253c-807365b0096c: entering follower state: follower="Node at a3ddce4b-7390-67a1-253c-807365b0096c [Follower]" leader= -testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [WARN] server-25c93f94-7b17-9861-cbac-b88b06a3a827: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [WARN] server-25c93f94-7b17-9861-cbac-b88b06a3a827: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: entering candidate state: node="Node at 25c93f94-7b17-9861-cbac-b88b06a3a827 [Candidate]" term=2 - testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: entering candidate state: node="Node at 25c93f94-7b17-9861-cbac-b88b06a3a827 [Candidate]" term=2 -testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: election won: tally=2 - testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: election won: tally=2 -testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: entering leader state: leader="Node at 25c93f94-7b17-9861-cbac-b88b06a3a827 [Leader]" - testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: entering leader state: leader="Node at 25c93f94-7b17-9861-cbac-b88b06a3a827 [Leader]" -testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: added peer, starting replication: peer=server-e591c761-60ba-b3d1-bba1-3608c0fde319 - testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: added peer, starting replication: peer=server-e591c761-60ba-b3d1-bba1-3608c0fde319 -testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: added peer, starting replication: peer=server-a3ddce4b-7390-67a1-253c-807365b0096c - testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: added peer, starting replication: peer=server-a3ddce4b-7390-67a1-253c-807365b0096c -testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: pipelining replication: peer="{Voter server-a3ddce4b-7390-67a1-253c-807365b0096c a3ddce4b-7390-67a1-253c-807365b0096c}" - testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: pipelining replication: peer="{Voter server-a3ddce4b-7390-67a1-253c-807365b0096c a3ddce4b-7390-67a1-253c-807365b0096c}" -testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: pipelining replication: peer="{Voter server-e591c761-60ba-b3d1-bba1-3608c0fde319 e591c761-60ba-b3d1-bba1-3608c0fde319}" - testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: pipelining replication: peer="{Voter server-e591c761-60ba-b3d1-bba1-3608c0fde319 e591c761-60ba-b3d1-bba1-3608c0fde319}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.919Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.814098 +0000 UTC m=+8.067149020 (1 nodes), 56.63438ms from start of poll, 57.226071ms from cluster start. Timeout at 2024-01-09 05:09:56.919122922 +0000 UTC m=+8.172174016, 105.024996ms after stability - testing.go:149: cluster: 2024-01-09T05:09:56.919Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.814098 +0000 UTC m=+8.067149020 (1 nodes), 56.63438ms from start of poll, 57.226071ms from cluster start. Timeout at 2024-01-09 05:09:56.919122922 +0000 UTC m=+8.172174016, 105.024996ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.924Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:09:56.924Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.035Z [INFO] stable state for Follower reached at 2024-01-09 05:09:56.924618469 +0000 UTC m=+8.177669469 (2 nodes), 2.96µs from start of poll, 167.74652ms from cluster start. Timeout at 2024-01-09 05:09:57.035117491 +0000 UTC m=+8.288168508, 110.499039ms after stability - testing.go:149: cluster: 2024-01-09T05:09:57.035Z [INFO] stable state for Follower reached at 2024-01-09 05:09:56.924618469 +0000 UTC m=+8.177669469 (2 nodes), 2.96µs from start of poll, 167.74652ms from cluster start. Timeout at 2024-01-09 05:09:57.035117491 +0000 UTC m=+8.288168508, 110.499039ms after stability -testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.035Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: updating configuration: command=RemoveServer server-id=server-e591c761-60ba-b3d1-bba1-3608c0fde319 server-addr= servers="[{Suffrage:Voter ID:server-25c93f94-7b17-9861-cbac-b88b06a3a827 Address:25c93f94-7b17-9861-cbac-b88b06a3a827} {Suffrage:Voter ID:server-a3ddce4b-7390-67a1-253c-807365b0096c Address:a3ddce4b-7390-67a1-253c-807365b0096c}]" - testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.035Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: updating configuration: command=RemoveServer server-id=server-e591c761-60ba-b3d1-bba1-3608c0fde319 server-addr= servers="[{Suffrage:Voter ID:server-25c93f94-7b17-9861-cbac-b88b06a3a827 Address:25c93f94-7b17-9861-cbac-b88b06a3a827} {Suffrage:Voter ID:server-a3ddce4b-7390-67a1-253c-807365b0096c Address:a3ddce4b-7390-67a1-253c-807365b0096c}]" -testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.035Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: removed peer, stopping replication: peer=server-e591c761-60ba-b3d1-bba1-3608c0fde319 last-index=3 - testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.035Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: removed peer, stopping replication: peer=server-e591c761-60ba-b3d1-bba1-3608c0fde319 last-index=3 -testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.035Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: aborting pipeline replication: peer="{Voter server-e591c761-60ba-b3d1-bba1-3608c0fde319 e591c761-60ba-b3d1-bba1-3608c0fde319}" - testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.035Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: aborting pipeline replication: peer="{Voter server-e591c761-60ba-b3d1-bba1-3608c0fde319 e591c761-60ba-b3d1-bba1-3608c0fde319}" -testLoggerAdapter verbose: server-e591c761-60ba-b3d1-bba1-3608c0fde319: 2024-01-09T05:09:57.128Z [WARN] server-e591c761-60ba-b3d1-bba1-3608c0fde319: heartbeat timeout reached, starting election: last-leader=25c93f94-7b17-9861-cbac-b88b06a3a827 - testing.go:149: server-e591c761-60ba-b3d1-bba1-3608c0fde319: 2024-01-09T05:09:57.128Z [WARN] server-e591c761-60ba-b3d1-bba1-3608c0fde319: heartbeat timeout reached, starting election: last-leader=25c93f94-7b17-9861-cbac-b88b06a3a827 -testLoggerAdapter verbose: server-e591c761-60ba-b3d1-bba1-3608c0fde319: 2024-01-09T05:09:57.128Z [INFO] server-e591c761-60ba-b3d1-bba1-3608c0fde319: entering candidate state: node="Node at e591c761-60ba-b3d1-bba1-3608c0fde319 [Candidate]" term=3 - testing.go:149: server-e591c761-60ba-b3d1-bba1-3608c0fde319: 2024-01-09T05:09:57.128Z [INFO] server-e591c761-60ba-b3d1-bba1-3608c0fde319: entering candidate state: node="Node at e591c761-60ba-b3d1-bba1-3608c0fde319 [Candidate]" term=3 -testLoggerAdapter verbose: server-a3ddce4b-7390-67a1-253c-807365b0096c: 2024-01-09T05:09:57.128Z [WARN] server-a3ddce4b-7390-67a1-253c-807365b0096c: rejecting vote request since we have a leader: from=e591c761-60ba-b3d1-bba1-3608c0fde319 leader=25c93f94-7b17-9861-cbac-b88b06a3a827 - testing.go:149: server-a3ddce4b-7390-67a1-253c-807365b0096c: 2024-01-09T05:09:57.128Z [WARN] server-a3ddce4b-7390-67a1-253c-807365b0096c: rejecting vote request since we have a leader: from=e591c761-60ba-b3d1-bba1-3608c0fde319 leader=25c93f94-7b17-9861-cbac-b88b06a3a827 -testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.128Z [WARN] server-25c93f94-7b17-9861-cbac-b88b06a3a827: rejecting vote request since we have a leader: from=e591c761-60ba-b3d1-bba1-3608c0fde319 leader=25c93f94-7b17-9861-cbac-b88b06a3a827 - testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.128Z [WARN] server-25c93f94-7b17-9861-cbac-b88b06a3a827: rejecting vote request since we have a leader: from=e591c761-60ba-b3d1-bba1-3608c0fde319 leader=25c93f94-7b17-9861-cbac-b88b06a3a827 -testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.147Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: aborting pipeline replication: peer="{Voter server-a3ddce4b-7390-67a1-253c-807365b0096c a3ddce4b-7390-67a1-253c-807365b0096c}" - testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.147Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: aborting pipeline replication: peer="{Voter server-a3ddce4b-7390-67a1-253c-807365b0096c a3ddce4b-7390-67a1-253c-807365b0096c}" ---- PASS: TestRaft_RemoveFollower (0.39s) +testLoggerAdapter verbose: server-6575fec4-7ef2-093e-b4ed-8ee28300c899: 2025-02-10T11:34:11.320Z [INFO] server-6575fec4-7ef2-093e-b4ed-8ee28300c899: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6575fec4-7ef2-093e-b4ed-8ee28300c899 Address:6575fec4-7ef2-093e-b4ed-8ee28300c899} {Suffrage:Voter ID:server-69b264f1-d42c-ea84-71c4-5fdbd600eed1 Address:69b264f1-d42c-ea84-71c4-5fdbd600eed1} {Suffrage:Voter ID:server-ddef9a16-de0b-679b-739d-e326a2103501 Address:ddef9a16-de0b-679b-739d-e326a2103501}]" + testing.go:149: server-6575fec4-7ef2-093e-b4ed-8ee28300c899: 2025-02-10T11:34:11.320Z [INFO] server-6575fec4-7ef2-093e-b4ed-8ee28300c899: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6575fec4-7ef2-093e-b4ed-8ee28300c899 Address:6575fec4-7ef2-093e-b4ed-8ee28300c899} {Suffrage:Voter ID:server-69b264f1-d42c-ea84-71c4-5fdbd600eed1 Address:69b264f1-d42c-ea84-71c4-5fdbd600eed1} {Suffrage:Voter ID:server-ddef9a16-de0b-679b-739d-e326a2103501 Address:ddef9a16-de0b-679b-739d-e326a2103501}]" +testLoggerAdapter verbose: server-6575fec4-7ef2-093e-b4ed-8ee28300c899: 2025-02-10T11:34:11.320Z [INFO] server-6575fec4-7ef2-093e-b4ed-8ee28300c899: entering follower state: follower="Node at 6575fec4-7ef2-093e-b4ed-8ee28300c899 [Follower]" leader= + testing.go:149: server-6575fec4-7ef2-093e-b4ed-8ee28300c899: 2025-02-10T11:34:11.320Z [INFO] server-6575fec4-7ef2-093e-b4ed-8ee28300c899: entering follower state: follower="Node at 6575fec4-7ef2-093e-b4ed-8ee28300c899 [Follower]" leader= +testLoggerAdapter verbose: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.320Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6575fec4-7ef2-093e-b4ed-8ee28300c899 Address:6575fec4-7ef2-093e-b4ed-8ee28300c899} {Suffrage:Voter ID:server-69b264f1-d42c-ea84-71c4-5fdbd600eed1 Address:69b264f1-d42c-ea84-71c4-5fdbd600eed1} {Suffrage:Voter ID:server-ddef9a16-de0b-679b-739d-e326a2103501 Address:ddef9a16-de0b-679b-739d-e326a2103501}]" + testing.go:149: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.320Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6575fec4-7ef2-093e-b4ed-8ee28300c899 Address:6575fec4-7ef2-093e-b4ed-8ee28300c899} {Suffrage:Voter ID:server-69b264f1-d42c-ea84-71c4-5fdbd600eed1 Address:69b264f1-d42c-ea84-71c4-5fdbd600eed1} {Suffrage:Voter ID:server-ddef9a16-de0b-679b-739d-e326a2103501 Address:ddef9a16-de0b-679b-739d-e326a2103501}]" +testLoggerAdapter verbose: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.320Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: entering follower state: follower="Node at 69b264f1-d42c-ea84-71c4-5fdbd600eed1 [Follower]" leader= + testing.go:149: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.320Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: entering follower state: follower="Node at 69b264f1-d42c-ea84-71c4-5fdbd600eed1 [Follower]" leader= +testLoggerAdapter verbose: server-ddef9a16-de0b-679b-739d-e326a2103501: 2025-02-10T11:34:11.320Z [INFO] server-ddef9a16-de0b-679b-739d-e326a2103501: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6575fec4-7ef2-093e-b4ed-8ee28300c899 Address:6575fec4-7ef2-093e-b4ed-8ee28300c899} {Suffrage:Voter ID:server-69b264f1-d42c-ea84-71c4-5fdbd600eed1 Address:69b264f1-d42c-ea84-71c4-5fdbd600eed1} {Suffrage:Voter ID:server-ddef9a16-de0b-679b-739d-e326a2103501 Address:ddef9a16-de0b-679b-739d-e326a2103501}]" + testing.go:149: server-ddef9a16-de0b-679b-739d-e326a2103501: 2025-02-10T11:34:11.320Z [INFO] server-ddef9a16-de0b-679b-739d-e326a2103501: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6575fec4-7ef2-093e-b4ed-8ee28300c899 Address:6575fec4-7ef2-093e-b4ed-8ee28300c899} {Suffrage:Voter ID:server-69b264f1-d42c-ea84-71c4-5fdbd600eed1 Address:69b264f1-d42c-ea84-71c4-5fdbd600eed1} {Suffrage:Voter ID:server-ddef9a16-de0b-679b-739d-e326a2103501 Address:ddef9a16-de0b-679b-739d-e326a2103501}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:11.320Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:11.320Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-ddef9a16-de0b-679b-739d-e326a2103501: 2025-02-10T11:34:11.320Z [INFO] server-ddef9a16-de0b-679b-739d-e326a2103501: entering follower state: follower="Node at ddef9a16-de0b-679b-739d-e326a2103501 [Follower]" leader= + testing.go:149: server-ddef9a16-de0b-679b-739d-e326a2103501: 2025-02-10T11:34:11.320Z [INFO] server-ddef9a16-de0b-679b-739d-e326a2103501: entering follower state: follower="Node at ddef9a16-de0b-679b-739d-e326a2103501 [Follower]" leader= +testLoggerAdapter verbose: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.377Z [WARN] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.377Z [WARN] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.377Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: entering candidate state: node="Node at 69b264f1-d42c-ea84-71c4-5fdbd600eed1 [Candidate]" term=2 + testing.go:149: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.377Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: entering candidate state: node="Node at 69b264f1-d42c-ea84-71c4-5fdbd600eed1 [Candidate]" term=2 +testLoggerAdapter verbose: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.377Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: election won: tally=2 + testing.go:149: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.377Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: election won: tally=2 +testLoggerAdapter verbose: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.377Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: entering leader state: leader="Node at 69b264f1-d42c-ea84-71c4-5fdbd600eed1 [Leader]" + testing.go:149: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.377Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: entering leader state: leader="Node at 69b264f1-d42c-ea84-71c4-5fdbd600eed1 [Leader]" +testLoggerAdapter verbose: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.377Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: added peer, starting replication: peer=server-6575fec4-7ef2-093e-b4ed-8ee28300c899 + testing.go:149: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.377Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: added peer, starting replication: peer=server-6575fec4-7ef2-093e-b4ed-8ee28300c899 +testLoggerAdapter verbose: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.377Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: added peer, starting replication: peer=server-ddef9a16-de0b-679b-739d-e326a2103501 + testing.go:149: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.377Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: added peer, starting replication: peer=server-ddef9a16-de0b-679b-739d-e326a2103501 +testLoggerAdapter verbose: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.377Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: pipelining replication: peer="{Voter server-6575fec4-7ef2-093e-b4ed-8ee28300c899 6575fec4-7ef2-093e-b4ed-8ee28300c899}" + testing.go:149: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.377Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: pipelining replication: peer="{Voter server-6575fec4-7ef2-093e-b4ed-8ee28300c899 6575fec4-7ef2-093e-b4ed-8ee28300c899}" +testLoggerAdapter verbose: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.377Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: pipelining replication: peer="{Voter server-ddef9a16-de0b-679b-739d-e326a2103501 ddef9a16-de0b-679b-739d-e326a2103501}" + testing.go:149: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.377Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: pipelining replication: peer="{Voter server-ddef9a16-de0b-679b-739d-e326a2103501 ddef9a16-de0b-679b-739d-e326a2103501}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:11.482Z [INFO] stable state for Leader reached at 2025-02-10 11:34:11.377400848 +0000 UTC m=+7.746305151 (1 nodes), 56.76796ms from start of poll, 57.165786ms from cluster start. Timeout at 2025-02-10 11:34:11.482474141 +0000 UTC m=+7.851378437, 105.073286ms after stability + testing.go:149: cluster: 2025-02-10T11:34:11.482Z [INFO] stable state for Leader reached at 2025-02-10 11:34:11.377400848 +0000 UTC m=+7.746305151 (1 nodes), 56.76796ms from start of poll, 57.165786ms from cluster start. Timeout at 2025-02-10 11:34:11.482474141 +0000 UTC m=+7.851378437, 105.073286ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:11.487Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:11.487Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:11.592Z [INFO] stable state for Follower reached at 2025-02-10 11:34:11.487656043 +0000 UTC m=+7.856560342 (2 nodes), 837ns from start of poll, 167.420977ms from cluster start. Timeout at 2025-02-10 11:34:11.592728302 +0000 UTC m=+7.961632602, 105.07226ms after stability + testing.go:149: cluster: 2025-02-10T11:34:11.592Z [INFO] stable state for Follower reached at 2025-02-10 11:34:11.487656043 +0000 UTC m=+7.856560342 (2 nodes), 837ns from start of poll, 167.420977ms from cluster start. Timeout at 2025-02-10 11:34:11.592728302 +0000 UTC m=+7.961632602, 105.07226ms after stability +testLoggerAdapter verbose: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.592Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: updating configuration: command=RemoveServer server-id=server-6575fec4-7ef2-093e-b4ed-8ee28300c899 server-addr= servers="[{Suffrage:Voter ID:server-69b264f1-d42c-ea84-71c4-5fdbd600eed1 Address:69b264f1-d42c-ea84-71c4-5fdbd600eed1} {Suffrage:Voter ID:server-ddef9a16-de0b-679b-739d-e326a2103501 Address:ddef9a16-de0b-679b-739d-e326a2103501}]" + testing.go:149: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.592Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: updating configuration: command=RemoveServer server-id=server-6575fec4-7ef2-093e-b4ed-8ee28300c899 server-addr= servers="[{Suffrage:Voter ID:server-69b264f1-d42c-ea84-71c4-5fdbd600eed1 Address:69b264f1-d42c-ea84-71c4-5fdbd600eed1} {Suffrage:Voter ID:server-ddef9a16-de0b-679b-739d-e326a2103501 Address:ddef9a16-de0b-679b-739d-e326a2103501}]" +testLoggerAdapter verbose: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.592Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: removed peer, stopping replication: peer=server-6575fec4-7ef2-093e-b4ed-8ee28300c899 last-index=3 + testing.go:149: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.592Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: removed peer, stopping replication: peer=server-6575fec4-7ef2-093e-b4ed-8ee28300c899 last-index=3 +testLoggerAdapter verbose: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.592Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: aborting pipeline replication: peer="{Voter server-6575fec4-7ef2-093e-b4ed-8ee28300c899 6575fec4-7ef2-093e-b4ed-8ee28300c899}" + testing.go:149: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.592Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: aborting pipeline replication: peer="{Voter server-6575fec4-7ef2-093e-b4ed-8ee28300c899 6575fec4-7ef2-093e-b4ed-8ee28300c899}" +testLoggerAdapter verbose: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.698Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: aborting pipeline replication: peer="{Voter server-ddef9a16-de0b-679b-739d-e326a2103501 ddef9a16-de0b-679b-739d-e326a2103501}" + testing.go:149: server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: 2025-02-10T11:34:11.698Z [INFO] server-69b264f1-d42c-ea84-71c4-5fdbd600eed1: aborting pipeline replication: peer="{Voter server-ddef9a16-de0b-679b-739d-e326a2103501 ddef9a16-de0b-679b-739d-e326a2103501}" +--- PASS: TestRaft_RemoveFollower (0.38s) === RUN TestRaft_RemoveLeader -testLoggerAdapter verbose: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.148Z [INFO] server-c8a06ca4-2690-98b7-5baa-63759e639581: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8a06ca4-2690-98b7-5baa-63759e639581 Address:c8a06ca4-2690-98b7-5baa-63759e639581} {Suffrage:Voter ID:server-41223f4a-7243-9254-7581-ddae91d497b1 Address:41223f4a-7243-9254-7581-ddae91d497b1} {Suffrage:Voter ID:server-2c7c7829-a541-c68f-c891-302be9f19edd Address:2c7c7829-a541-c68f-c891-302be9f19edd}]" - testing.go:149: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.148Z [INFO] server-c8a06ca4-2690-98b7-5baa-63759e639581: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8a06ca4-2690-98b7-5baa-63759e639581 Address:c8a06ca4-2690-98b7-5baa-63759e639581} {Suffrage:Voter ID:server-41223f4a-7243-9254-7581-ddae91d497b1 Address:41223f4a-7243-9254-7581-ddae91d497b1} {Suffrage:Voter ID:server-2c7c7829-a541-c68f-c891-302be9f19edd Address:2c7c7829-a541-c68f-c891-302be9f19edd}]" -testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.148Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8a06ca4-2690-98b7-5baa-63759e639581 Address:c8a06ca4-2690-98b7-5baa-63759e639581} {Suffrage:Voter ID:server-41223f4a-7243-9254-7581-ddae91d497b1 Address:41223f4a-7243-9254-7581-ddae91d497b1} {Suffrage:Voter ID:server-2c7c7829-a541-c68f-c891-302be9f19edd Address:2c7c7829-a541-c68f-c891-302be9f19edd}]" - testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.148Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8a06ca4-2690-98b7-5baa-63759e639581 Address:c8a06ca4-2690-98b7-5baa-63759e639581} {Suffrage:Voter ID:server-41223f4a-7243-9254-7581-ddae91d497b1 Address:41223f4a-7243-9254-7581-ddae91d497b1} {Suffrage:Voter ID:server-2c7c7829-a541-c68f-c891-302be9f19edd Address:2c7c7829-a541-c68f-c891-302be9f19edd}]" -testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.149Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8a06ca4-2690-98b7-5baa-63759e639581 Address:c8a06ca4-2690-98b7-5baa-63759e639581} {Suffrage:Voter ID:server-41223f4a-7243-9254-7581-ddae91d497b1 Address:41223f4a-7243-9254-7581-ddae91d497b1} {Suffrage:Voter ID:server-2c7c7829-a541-c68f-c891-302be9f19edd Address:2c7c7829-a541-c68f-c891-302be9f19edd}]" - testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.149Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8a06ca4-2690-98b7-5baa-63759e639581 Address:c8a06ca4-2690-98b7-5baa-63759e639581} {Suffrage:Voter ID:server-41223f4a-7243-9254-7581-ddae91d497b1 Address:41223f4a-7243-9254-7581-ddae91d497b1} {Suffrage:Voter ID:server-2c7c7829-a541-c68f-c891-302be9f19edd Address:2c7c7829-a541-c68f-c891-302be9f19edd}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.149Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:57.149Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.149Z [INFO] server-c8a06ca4-2690-98b7-5baa-63759e639581: entering follower state: follower="Node at c8a06ca4-2690-98b7-5baa-63759e639581 [Follower]" leader= - testing.go:149: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.149Z [INFO] server-c8a06ca4-2690-98b7-5baa-63759e639581: entering follower state: follower="Node at c8a06ca4-2690-98b7-5baa-63759e639581 [Follower]" leader= -testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.149Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: entering follower state: follower="Node at 41223f4a-7243-9254-7581-ddae91d497b1 [Follower]" leader= - testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.149Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: entering follower state: follower="Node at 41223f4a-7243-9254-7581-ddae91d497b1 [Follower]" leader= -testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.149Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: entering follower state: follower="Node at 2c7c7829-a541-c68f-c891-302be9f19edd [Follower]" leader= - testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.149Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: entering follower state: follower="Node at 2c7c7829-a541-c68f-c891-302be9f19edd [Follower]" leader= -testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [WARN] server-2c7c7829-a541-c68f-c891-302be9f19edd: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [WARN] server-2c7c7829-a541-c68f-c891-302be9f19edd: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: entering candidate state: node="Node at 2c7c7829-a541-c68f-c891-302be9f19edd [Candidate]" term=2 - testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: entering candidate state: node="Node at 2c7c7829-a541-c68f-c891-302be9f19edd [Candidate]" term=2 -testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: election won: tally=2 - testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: election won: tally=2 -testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: entering leader state: leader="Node at 2c7c7829-a541-c68f-c891-302be9f19edd [Leader]" - testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: entering leader state: leader="Node at 2c7c7829-a541-c68f-c891-302be9f19edd [Leader]" -testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: added peer, starting replication: peer=server-c8a06ca4-2690-98b7-5baa-63759e639581 - testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: added peer, starting replication: peer=server-c8a06ca4-2690-98b7-5baa-63759e639581 -testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: added peer, starting replication: peer=server-41223f4a-7243-9254-7581-ddae91d497b1 - testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: added peer, starting replication: peer=server-41223f4a-7243-9254-7581-ddae91d497b1 -testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: pipelining replication: peer="{Voter server-41223f4a-7243-9254-7581-ddae91d497b1 41223f4a-7243-9254-7581-ddae91d497b1}" - testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: pipelining replication: peer="{Voter server-41223f4a-7243-9254-7581-ddae91d497b1 41223f4a-7243-9254-7581-ddae91d497b1}" -testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: pipelining replication: peer="{Voter server-c8a06ca4-2690-98b7-5baa-63759e639581 c8a06ca4-2690-98b7-5baa-63759e639581}" - testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: pipelining replication: peer="{Voter server-c8a06ca4-2690-98b7-5baa-63759e639581 c8a06ca4-2690-98b7-5baa-63759e639581}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.328Z [INFO] stable state for Leader reached at 2024-01-09 05:09:57.223803261 +0000 UTC m=+8.476854291 (1 nodes), 74.708753ms from start of poll, 75.185186ms from cluster start. Timeout at 2024-01-09 05:09:57.328909408 +0000 UTC m=+8.581960556, 105.106265ms after stability - testing.go:149: cluster: 2024-01-09T05:09:57.328Z [INFO] stable state for Leader reached at 2024-01-09 05:09:57.223803261 +0000 UTC m=+8.476854291 (1 nodes), 74.708753ms from start of poll, 75.185186ms from cluster start. Timeout at 2024-01-09 05:09:57.328909408 +0000 UTC m=+8.581960556, 105.106265ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.334Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:09:57.334Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.439Z [INFO] stable state for Follower reached at 2024-01-09 05:09:57.334251502 +0000 UTC m=+8.587302618 (2 nodes), 2.876µs from start of poll, 185.633513ms from cluster start. Timeout at 2024-01-09 05:09:57.439428118 +0000 UTC m=+8.692479116, 105.176498ms after stability - testing.go:149: cluster: 2024-01-09T05:09:57.439Z [INFO] stable state for Follower reached at 2024-01-09 05:09:57.334251502 +0000 UTC m=+8.587302618 (2 nodes), 2.876µs from start of poll, 185.633513ms from cluster start. Timeout at 2024-01-09 05:09:57.439428118 +0000 UTC m=+8.692479116, 105.176498ms after stability -testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.439Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: updating configuration: command=RemoveServer server-id=server-2c7c7829-a541-c68f-c891-302be9f19edd server-addr= servers="[{Suffrage:Voter ID:server-c8a06ca4-2690-98b7-5baa-63759e639581 Address:c8a06ca4-2690-98b7-5baa-63759e639581} {Suffrage:Voter ID:server-41223f4a-7243-9254-7581-ddae91d497b1 Address:41223f4a-7243-9254-7581-ddae91d497b1}]" - testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.439Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: updating configuration: command=RemoveServer server-id=server-2c7c7829-a541-c68f-c891-302be9f19edd server-addr= servers="[{Suffrage:Voter ID:server-c8a06ca4-2690-98b7-5baa-63759e639581 Address:c8a06ca4-2690-98b7-5baa-63759e639581} {Suffrage:Voter ID:server-41223f4a-7243-9254-7581-ddae91d497b1 Address:41223f4a-7243-9254-7581-ddae91d497b1}]" -testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.439Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: removed ourself, shutting down - testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.439Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: removed ourself, shutting down -testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.439Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: aborting pipeline replication: peer="{Voter server-41223f4a-7243-9254-7581-ddae91d497b1 41223f4a-7243-9254-7581-ddae91d497b1}" - testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.439Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: aborting pipeline replication: peer="{Voter server-41223f4a-7243-9254-7581-ddae91d497b1 41223f4a-7243-9254-7581-ddae91d497b1}" -testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.439Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: aborting pipeline replication: peer="{Voter server-c8a06ca4-2690-98b7-5baa-63759e639581 c8a06ca4-2690-98b7-5baa-63759e639581}" - testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.439Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: aborting pipeline replication: peer="{Voter server-c8a06ca4-2690-98b7-5baa-63759e639581 c8a06ca4-2690-98b7-5baa-63759e639581}" -testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.519Z [WARN] server-41223f4a-7243-9254-7581-ddae91d497b1: heartbeat timeout reached, starting election: last-leader=2c7c7829-a541-c68f-c891-302be9f19edd - testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.519Z [WARN] server-41223f4a-7243-9254-7581-ddae91d497b1: heartbeat timeout reached, starting election: last-leader=2c7c7829-a541-c68f-c891-302be9f19edd -testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.519Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: entering candidate state: node="Node at 41223f4a-7243-9254-7581-ddae91d497b1 [Candidate]" term=3 - testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.519Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: entering candidate state: node="Node at 41223f4a-7243-9254-7581-ddae91d497b1 [Candidate]" term=3 -testLoggerAdapter verbose: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.519Z [WARN] server-c8a06ca4-2690-98b7-5baa-63759e639581: rejecting vote request since we have a leader: from=41223f4a-7243-9254-7581-ddae91d497b1 leader=2c7c7829-a541-c68f-c891-302be9f19edd - testing.go:149: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.519Z [WARN] server-c8a06ca4-2690-98b7-5baa-63759e639581: rejecting vote request since we have a leader: from=41223f4a-7243-9254-7581-ddae91d497b1 leader=2c7c7829-a541-c68f-c891-302be9f19edd -testLoggerAdapter verbose: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.543Z [WARN] server-c8a06ca4-2690-98b7-5baa-63759e639581: heartbeat timeout reached, starting election: last-leader=2c7c7829-a541-c68f-c891-302be9f19edd - testing.go:149: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.543Z [WARN] server-c8a06ca4-2690-98b7-5baa-63759e639581: heartbeat timeout reached, starting election: last-leader=2c7c7829-a541-c68f-c891-302be9f19edd -testLoggerAdapter verbose: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.543Z [INFO] server-c8a06ca4-2690-98b7-5baa-63759e639581: entering candidate state: node="Node at c8a06ca4-2690-98b7-5baa-63759e639581 [Candidate]" term=3 - testing.go:149: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.543Z [INFO] server-c8a06ca4-2690-98b7-5baa-63759e639581: entering candidate state: node="Node at c8a06ca4-2690-98b7-5baa-63759e639581 [Candidate]" term=3 -testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.543Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: duplicate requestVote for same term: term=3 - testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.543Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: duplicate requestVote for same term: term=3 -testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.594Z [WARN] server-41223f4a-7243-9254-7581-ddae91d497b1: Election timeout reached, restarting election - testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.594Z [WARN] server-41223f4a-7243-9254-7581-ddae91d497b1: Election timeout reached, restarting election -testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: entering candidate state: node="Node at 41223f4a-7243-9254-7581-ddae91d497b1 [Candidate]" term=4 - testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: entering candidate state: node="Node at 41223f4a-7243-9254-7581-ddae91d497b1 [Candidate]" term=4 -testLoggerAdapter verbose: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.595Z [INFO] server-c8a06ca4-2690-98b7-5baa-63759e639581: entering follower state: follower="Node at c8a06ca4-2690-98b7-5baa-63759e639581 [Follower]" leader= - testing.go:149: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.595Z [INFO] server-c8a06ca4-2690-98b7-5baa-63759e639581: entering follower state: follower="Node at c8a06ca4-2690-98b7-5baa-63759e639581 [Follower]" leader= -testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: election won: tally=2 - testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: election won: tally=2 -testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: entering leader state: leader="Node at 41223f4a-7243-9254-7581-ddae91d497b1 [Leader]" - testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: entering leader state: leader="Node at 41223f4a-7243-9254-7581-ddae91d497b1 [Leader]" -testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: added peer, starting replication: peer=server-c8a06ca4-2690-98b7-5baa-63759e639581 - testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: added peer, starting replication: peer=server-c8a06ca4-2690-98b7-5baa-63759e639581 -testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: pipelining replication: peer="{Voter server-c8a06ca4-2690-98b7-5baa-63759e639581 c8a06ca4-2690-98b7-5baa-63759e639581}" - testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: pipelining replication: peer="{Voter server-c8a06ca4-2690-98b7-5baa-63759e639581 c8a06ca4-2690-98b7-5baa-63759e639581}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.676Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:57.676Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.781Z [INFO] stable state for Leader reached at 2024-01-09 05:09:57.676182452 +0000 UTC m=+8.929233462 (1 nodes), 54.813µs from start of poll, 527.564357ms from cluster start. Timeout at 2024-01-09 05:09:57.781301951 +0000 UTC m=+9.034353068, 105.119606ms after stability - testing.go:149: cluster: 2024-01-09T05:09:57.781Z [INFO] stable state for Leader reached at 2024-01-09 05:09:57.676182452 +0000 UTC m=+8.929233462 (1 nodes), 54.813µs from start of poll, 527.564357ms from cluster start. Timeout at 2024-01-09 05:09:57.781301951 +0000 UTC m=+9.034353068, 105.119606ms after stability -testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.781Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: aborting pipeline replication: peer="{Voter server-c8a06ca4-2690-98b7-5baa-63759e639581 c8a06ca4-2690-98b7-5baa-63759e639581}" - testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.781Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: aborting pipeline replication: peer="{Voter server-c8a06ca4-2690-98b7-5baa-63759e639581 c8a06ca4-2690-98b7-5baa-63759e639581}" ---- PASS: TestRaft_RemoveLeader (0.63s) +testLoggerAdapter verbose: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.699Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15effd90-5407-7b7f-7502-c7f4035cc10d Address:15effd90-5407-7b7f-7502-c7f4035cc10d} {Suffrage:Voter ID:server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 Address:e8a89705-86f4-a5a9-9a7b-ad099c23e969} {Suffrage:Voter ID:server-274c9ffb-318d-1625-25c6-f6e8d57b8223 Address:274c9ffb-318d-1625-25c6-f6e8d57b8223}]" + testing.go:149: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.699Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15effd90-5407-7b7f-7502-c7f4035cc10d Address:15effd90-5407-7b7f-7502-c7f4035cc10d} {Suffrage:Voter ID:server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 Address:e8a89705-86f4-a5a9-9a7b-ad099c23e969} {Suffrage:Voter ID:server-274c9ffb-318d-1625-25c6-f6e8d57b8223 Address:274c9ffb-318d-1625-25c6-f6e8d57b8223}]" +testLoggerAdapter verbose: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.699Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: entering follower state: follower="Node at 15effd90-5407-7b7f-7502-c7f4035cc10d [Follower]" leader= + testing.go:149: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.699Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: entering follower state: follower="Node at 15effd90-5407-7b7f-7502-c7f4035cc10d [Follower]" leader= +testLoggerAdapter verbose: server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: 2025-02-10T11:34:11.699Z [INFO] server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15effd90-5407-7b7f-7502-c7f4035cc10d Address:15effd90-5407-7b7f-7502-c7f4035cc10d} {Suffrage:Voter ID:server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 Address:e8a89705-86f4-a5a9-9a7b-ad099c23e969} {Suffrage:Voter ID:server-274c9ffb-318d-1625-25c6-f6e8d57b8223 Address:274c9ffb-318d-1625-25c6-f6e8d57b8223}]" + testing.go:149: server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: 2025-02-10T11:34:11.699Z [INFO] server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15effd90-5407-7b7f-7502-c7f4035cc10d Address:15effd90-5407-7b7f-7502-c7f4035cc10d} {Suffrage:Voter ID:server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 Address:e8a89705-86f4-a5a9-9a7b-ad099c23e969} {Suffrage:Voter ID:server-274c9ffb-318d-1625-25c6-f6e8d57b8223 Address:274c9ffb-318d-1625-25c6-f6e8d57b8223}]" +testLoggerAdapter verbose: server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: 2025-02-10T11:34:11.699Z [INFO] server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: entering follower state: follower="Node at e8a89705-86f4-a5a9-9a7b-ad099c23e969 [Follower]" leader= + testing.go:149: server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: 2025-02-10T11:34:11.699Z [INFO] server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: entering follower state: follower="Node at e8a89705-86f4-a5a9-9a7b-ad099c23e969 [Follower]" leader= +testLoggerAdapter verbose: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:11.699Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15effd90-5407-7b7f-7502-c7f4035cc10d Address:15effd90-5407-7b7f-7502-c7f4035cc10d} {Suffrage:Voter ID:server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 Address:e8a89705-86f4-a5a9-9a7b-ad099c23e969} {Suffrage:Voter ID:server-274c9ffb-318d-1625-25c6-f6e8d57b8223 Address:274c9ffb-318d-1625-25c6-f6e8d57b8223}]" + testing.go:149: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:11.699Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15effd90-5407-7b7f-7502-c7f4035cc10d Address:15effd90-5407-7b7f-7502-c7f4035cc10d} {Suffrage:Voter ID:server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 Address:e8a89705-86f4-a5a9-9a7b-ad099c23e969} {Suffrage:Voter ID:server-274c9ffb-318d-1625-25c6-f6e8d57b8223 Address:274c9ffb-318d-1625-25c6-f6e8d57b8223}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:11.699Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:11.699Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:11.699Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: entering follower state: follower="Node at 274c9ffb-318d-1625-25c6-f6e8d57b8223 [Follower]" leader= + testing.go:149: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:11.699Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: entering follower state: follower="Node at 274c9ffb-318d-1625-25c6-f6e8d57b8223 [Follower]" leader= +testLoggerAdapter verbose: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.752Z [WARN] server-15effd90-5407-7b7f-7502-c7f4035cc10d: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.752Z [WARN] server-15effd90-5407-7b7f-7502-c7f4035cc10d: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.752Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: entering candidate state: node="Node at 15effd90-5407-7b7f-7502-c7f4035cc10d [Candidate]" term=2 + testing.go:149: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.752Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: entering candidate state: node="Node at 15effd90-5407-7b7f-7502-c7f4035cc10d [Candidate]" term=2 +testLoggerAdapter verbose: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.752Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: election won: tally=2 + testing.go:149: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.752Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: election won: tally=2 +testLoggerAdapter verbose: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.752Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: entering leader state: leader="Node at 15effd90-5407-7b7f-7502-c7f4035cc10d [Leader]" + testing.go:149: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.752Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: entering leader state: leader="Node at 15effd90-5407-7b7f-7502-c7f4035cc10d [Leader]" +testLoggerAdapter verbose: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.752Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: added peer, starting replication: peer=server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 + testing.go:149: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.752Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: added peer, starting replication: peer=server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 +testLoggerAdapter verbose: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.752Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: added peer, starting replication: peer=server-274c9ffb-318d-1625-25c6-f6e8d57b8223 + testing.go:149: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.752Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: added peer, starting replication: peer=server-274c9ffb-318d-1625-25c6-f6e8d57b8223 +testLoggerAdapter verbose: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.752Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: pipelining replication: peer="{Voter server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 e8a89705-86f4-a5a9-9a7b-ad099c23e969}" + testing.go:149: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.752Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: pipelining replication: peer="{Voter server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 e8a89705-86f4-a5a9-9a7b-ad099c23e969}" +testLoggerAdapter verbose: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.752Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: pipelining replication: peer="{Voter server-274c9ffb-318d-1625-25c6-f6e8d57b8223 274c9ffb-318d-1625-25c6-f6e8d57b8223}" + testing.go:149: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.752Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: pipelining replication: peer="{Voter server-274c9ffb-318d-1625-25c6-f6e8d57b8223 274c9ffb-318d-1625-25c6-f6e8d57b8223}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:11.857Z [INFO] stable state for Leader reached at 2025-02-10 11:34:11.752470057 +0000 UTC m=+8.121374358 (1 nodes), 52.941826ms from start of poll, 53.340587ms from cluster start. Timeout at 2025-02-10 11:34:11.857558136 +0000 UTC m=+8.226462512, 105.088154ms after stability + testing.go:149: cluster: 2025-02-10T11:34:11.857Z [INFO] stable state for Leader reached at 2025-02-10 11:34:11.752470057 +0000 UTC m=+8.121374358 (1 nodes), 52.941826ms from start of poll, 53.340587ms from cluster start. Timeout at 2025-02-10 11:34:11.857558136 +0000 UTC m=+8.226462512, 105.088154ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:11.862Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:11.862Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:11.967Z [INFO] stable state for Follower reached at 2025-02-10 11:34:11.862811007 +0000 UTC m=+8.231715307 (2 nodes), 597ns from start of poll, 163.681536ms from cluster start. Timeout at 2025-02-10 11:34:11.967883422 +0000 UTC m=+8.336787713, 105.072406ms after stability + testing.go:149: cluster: 2025-02-10T11:34:11.967Z [INFO] stable state for Follower reached at 2025-02-10 11:34:11.862811007 +0000 UTC m=+8.231715307 (2 nodes), 597ns from start of poll, 163.681536ms from cluster start. Timeout at 2025-02-10 11:34:11.967883422 +0000 UTC m=+8.336787713, 105.072406ms after stability +testLoggerAdapter verbose: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.967Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: updating configuration: command=RemoveServer server-id=server-15effd90-5407-7b7f-7502-c7f4035cc10d server-addr= servers="[{Suffrage:Voter ID:server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 Address:e8a89705-86f4-a5a9-9a7b-ad099c23e969} {Suffrage:Voter ID:server-274c9ffb-318d-1625-25c6-f6e8d57b8223 Address:274c9ffb-318d-1625-25c6-f6e8d57b8223}]" + testing.go:149: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.967Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: updating configuration: command=RemoveServer server-id=server-15effd90-5407-7b7f-7502-c7f4035cc10d server-addr= servers="[{Suffrage:Voter ID:server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 Address:e8a89705-86f4-a5a9-9a7b-ad099c23e969} {Suffrage:Voter ID:server-274c9ffb-318d-1625-25c6-f6e8d57b8223 Address:274c9ffb-318d-1625-25c6-f6e8d57b8223}]" +testLoggerAdapter verbose: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.968Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: removed ourself, shutting down + testing.go:149: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.968Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: removed ourself, shutting down +testLoggerAdapter verbose: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.968Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: aborting pipeline replication: peer="{Voter server-274c9ffb-318d-1625-25c6-f6e8d57b8223 274c9ffb-318d-1625-25c6-f6e8d57b8223}" + testing.go:149: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.968Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: aborting pipeline replication: peer="{Voter server-274c9ffb-318d-1625-25c6-f6e8d57b8223 274c9ffb-318d-1625-25c6-f6e8d57b8223}" +testLoggerAdapter verbose: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.968Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: aborting pipeline replication: peer="{Voter server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 e8a89705-86f4-a5a9-9a7b-ad099c23e969}" + testing.go:149: server-15effd90-5407-7b7f-7502-c7f4035cc10d: 2025-02-10T11:34:11.968Z [INFO] server-15effd90-5407-7b7f-7502-c7f4035cc10d: aborting pipeline replication: peer="{Voter server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 e8a89705-86f4-a5a9-9a7b-ad099c23e969}" +testLoggerAdapter verbose: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.034Z [WARN] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: heartbeat timeout reached, starting election: last-leader=15effd90-5407-7b7f-7502-c7f4035cc10d + testing.go:149: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.034Z [WARN] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: heartbeat timeout reached, starting election: last-leader=15effd90-5407-7b7f-7502-c7f4035cc10d +testLoggerAdapter verbose: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.034Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: entering candidate state: node="Node at 274c9ffb-318d-1625-25c6-f6e8d57b8223 [Candidate]" term=3 + testing.go:149: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.034Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: entering candidate state: node="Node at 274c9ffb-318d-1625-25c6-f6e8d57b8223 [Candidate]" term=3 +testLoggerAdapter verbose: server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: 2025-02-10T11:34:12.034Z [WARN] server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: rejecting vote request since we have a leader: from=274c9ffb-318d-1625-25c6-f6e8d57b8223 leader=15effd90-5407-7b7f-7502-c7f4035cc10d + testing.go:149: server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: 2025-02-10T11:34:12.034Z [WARN] server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: rejecting vote request since we have a leader: from=274c9ffb-318d-1625-25c6-f6e8d57b8223 leader=15effd90-5407-7b7f-7502-c7f4035cc10d +testLoggerAdapter verbose: server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: 2025-02-10T11:34:12.076Z [WARN] server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: heartbeat timeout reached, starting election: last-leader=15effd90-5407-7b7f-7502-c7f4035cc10d + testing.go:149: server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: 2025-02-10T11:34:12.076Z [WARN] server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: heartbeat timeout reached, starting election: last-leader=15effd90-5407-7b7f-7502-c7f4035cc10d +testLoggerAdapter verbose: server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: 2025-02-10T11:34:12.077Z [INFO] server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: entering candidate state: node="Node at e8a89705-86f4-a5a9-9a7b-ad099c23e969 [Candidate]" term=3 + testing.go:149: server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: 2025-02-10T11:34:12.077Z [INFO] server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: entering candidate state: node="Node at e8a89705-86f4-a5a9-9a7b-ad099c23e969 [Candidate]" term=3 +testLoggerAdapter verbose: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.077Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: duplicate requestVote for same term: term=3 + testing.go:149: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.077Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: duplicate requestVote for same term: term=3 +testLoggerAdapter verbose: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.116Z [WARN] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: Election timeout reached, restarting election + testing.go:149: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.116Z [WARN] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: Election timeout reached, restarting election +testLoggerAdapter verbose: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.116Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: entering candidate state: node="Node at 274c9ffb-318d-1625-25c6-f6e8d57b8223 [Candidate]" term=4 + testing.go:149: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.116Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: entering candidate state: node="Node at 274c9ffb-318d-1625-25c6-f6e8d57b8223 [Candidate]" term=4 +testLoggerAdapter verbose: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.116Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: election won: tally=2 + testing.go:149: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.116Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: election won: tally=2 +testLoggerAdapter verbose: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.116Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: entering leader state: leader="Node at 274c9ffb-318d-1625-25c6-f6e8d57b8223 [Leader]" + testing.go:149: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.116Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: entering leader state: leader="Node at 274c9ffb-318d-1625-25c6-f6e8d57b8223 [Leader]" +testLoggerAdapter verbose: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.116Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: added peer, starting replication: peer=server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 +testLoggerAdapter verbose: server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: 2025-02-10T11:34:12.116Z [INFO] server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: entering follower state: follower="Node at e8a89705-86f4-a5a9-9a7b-ad099c23e969 [Follower]" leader= + testing.go:149: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.116Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: added peer, starting replication: peer=server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 + testing.go:149: server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: 2025-02-10T11:34:12.116Z [INFO] server-e8a89705-86f4-a5a9-9a7b-ad099c23e969: entering follower state: follower="Node at e8a89705-86f4-a5a9-9a7b-ad099c23e969 [Follower]" leader= +testLoggerAdapter verbose: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.116Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: pipelining replication: peer="{Voter server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 e8a89705-86f4-a5a9-9a7b-ad099c23e969}" + testing.go:149: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.116Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: pipelining replication: peer="{Voter server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 e8a89705-86f4-a5a9-9a7b-ad099c23e969}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:12.178Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:12.178Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:12.283Z [INFO] stable state for Leader reached at 2025-02-10 11:34:12.178291735 +0000 UTC m=+8.547196035 (1 nodes), 21.937µs from start of poll, 479.162264ms from cluster start. Timeout at 2025-02-10 11:34:12.283361249 +0000 UTC m=+8.652265545, 105.06951ms after stability + testing.go:149: cluster: 2025-02-10T11:34:12.283Z [INFO] stable state for Leader reached at 2025-02-10 11:34:12.178291735 +0000 UTC m=+8.547196035 (1 nodes), 21.937µs from start of poll, 479.162264ms from cluster start. Timeout at 2025-02-10 11:34:12.283361249 +0000 UTC m=+8.652265545, 105.06951ms after stability +testLoggerAdapter verbose: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.283Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: aborting pipeline replication: peer="{Voter server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 e8a89705-86f4-a5a9-9a7b-ad099c23e969}" + testing.go:149: server-274c9ffb-318d-1625-25c6-f6e8d57b8223: 2025-02-10T11:34:12.283Z [INFO] server-274c9ffb-318d-1625-25c6-f6e8d57b8223: aborting pipeline replication: peer="{Voter server-e8a89705-86f4-a5a9-9a7b-ad099c23e969 e8a89705-86f4-a5a9-9a7b-ad099c23e969}" +--- PASS: TestRaft_RemoveLeader (0.59s) === RUN TestRaft_RemoveLeader_NoShutdown -testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:57.783Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd Address:9a5bce64-da87-eed0-1c8c-d753d3ab43bd} {Suffrage:Voter ID:server-e8bf385c-490a-2395-38fd-ba0d812e9332 Address:e8bf385c-490a-2395-38fd-ba0d812e9332} {Suffrage:Voter ID:server-637e3d76-fedb-7ba5-2cc2-895de972afce Address:637e3d76-fedb-7ba5-2cc2-895de972afce}]" - testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:57.783Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd Address:9a5bce64-da87-eed0-1c8c-d753d3ab43bd} {Suffrage:Voter ID:server-e8bf385c-490a-2395-38fd-ba0d812e9332 Address:e8bf385c-490a-2395-38fd-ba0d812e9332} {Suffrage:Voter ID:server-637e3d76-fedb-7ba5-2cc2-895de972afce Address:637e3d76-fedb-7ba5-2cc2-895de972afce}]" -testLoggerAdapter verbose: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:57.783Z [INFO] server-e8bf385c-490a-2395-38fd-ba0d812e9332: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd Address:9a5bce64-da87-eed0-1c8c-d753d3ab43bd} {Suffrage:Voter ID:server-e8bf385c-490a-2395-38fd-ba0d812e9332 Address:e8bf385c-490a-2395-38fd-ba0d812e9332} {Suffrage:Voter ID:server-637e3d76-fedb-7ba5-2cc2-895de972afce Address:637e3d76-fedb-7ba5-2cc2-895de972afce}]" - testing.go:149: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:57.783Z [INFO] server-e8bf385c-490a-2395-38fd-ba0d812e9332: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd Address:9a5bce64-da87-eed0-1c8c-d753d3ab43bd} {Suffrage:Voter ID:server-e8bf385c-490a-2395-38fd-ba0d812e9332 Address:e8bf385c-490a-2395-38fd-ba0d812e9332} {Suffrage:Voter ID:server-637e3d76-fedb-7ba5-2cc2-895de972afce Address:637e3d76-fedb-7ba5-2cc2-895de972afce}]" -testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.783Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd Address:9a5bce64-da87-eed0-1c8c-d753d3ab43bd} {Suffrage:Voter ID:server-e8bf385c-490a-2395-38fd-ba0d812e9332 Address:e8bf385c-490a-2395-38fd-ba0d812e9332} {Suffrage:Voter ID:server-637e3d76-fedb-7ba5-2cc2-895de972afce Address:637e3d76-fedb-7ba5-2cc2-895de972afce}]" - testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.783Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd Address:9a5bce64-da87-eed0-1c8c-d753d3ab43bd} {Suffrage:Voter ID:server-e8bf385c-490a-2395-38fd-ba0d812e9332 Address:e8bf385c-490a-2395-38fd-ba0d812e9332} {Suffrage:Voter ID:server-637e3d76-fedb-7ba5-2cc2-895de972afce Address:637e3d76-fedb-7ba5-2cc2-895de972afce}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.783Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:09:57.783Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:57.783Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: entering follower state: follower="Node at 9a5bce64-da87-eed0-1c8c-d753d3ab43bd [Follower]" leader= - testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:57.783Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: entering follower state: follower="Node at 9a5bce64-da87-eed0-1c8c-d753d3ab43bd [Follower]" leader= -testLoggerAdapter verbose: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:57.783Z [INFO] server-e8bf385c-490a-2395-38fd-ba0d812e9332: entering follower state: follower="Node at e8bf385c-490a-2395-38fd-ba0d812e9332 [Follower]" leader= - testing.go:149: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:57.783Z [INFO] server-e8bf385c-490a-2395-38fd-ba0d812e9332: entering follower state: follower="Node at e8bf385c-490a-2395-38fd-ba0d812e9332 [Follower]" leader= -testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.783Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: entering follower state: follower="Node at 637e3d76-fedb-7ba5-2cc2-895de972afce [Follower]" leader= - testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.783Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: entering follower state: follower="Node at 637e3d76-fedb-7ba5-2cc2-895de972afce [Follower]" leader= -testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.848Z [WARN] server-637e3d76-fedb-7ba5-2cc2-895de972afce: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.848Z [WARN] server-637e3d76-fedb-7ba5-2cc2-895de972afce: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.848Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: entering candidate state: node="Node at 637e3d76-fedb-7ba5-2cc2-895de972afce [Candidate]" term=2 - testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.848Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: entering candidate state: node="Node at 637e3d76-fedb-7ba5-2cc2-895de972afce [Candidate]" term=2 -testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.848Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: election won: tally=2 - testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.848Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: election won: tally=2 -testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: entering leader state: leader="Node at 637e3d76-fedb-7ba5-2cc2-895de972afce [Leader]" - testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: entering leader state: leader="Node at 637e3d76-fedb-7ba5-2cc2-895de972afce [Leader]" -testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: added peer, starting replication: peer=server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd - testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: added peer, starting replication: peer=server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd -testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: added peer, starting replication: peer=server-e8bf385c-490a-2395-38fd-ba0d812e9332 - testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: added peer, starting replication: peer=server-e8bf385c-490a-2395-38fd-ba0d812e9332 -testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: pipelining replication: peer="{Voter server-e8bf385c-490a-2395-38fd-ba0d812e9332 e8bf385c-490a-2395-38fd-ba0d812e9332}" - testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: pipelining replication: peer="{Voter server-e8bf385c-490a-2395-38fd-ba0d812e9332 e8bf385c-490a-2395-38fd-ba0d812e9332}" -testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: pipelining replication: peer="{Voter server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd 9a5bce64-da87-eed0-1c8c-d753d3ab43bd}" - testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: pipelining replication: peer="{Voter server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd 9a5bce64-da87-eed0-1c8c-d753d3ab43bd}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.954Z [INFO] stable state for Follower reached at 2024-01-09 05:09:57.849456457 +0000 UTC m=+9.102507588 (2 nodes), 65.83085ms from start of poll, 66.473384ms from cluster start. Timeout at 2024-01-09 05:09:57.95455978 +0000 UTC m=+9.207610974, 105.103386ms after stability - testing.go:149: cluster: 2024-01-09T05:09:57.954Z [INFO] stable state for Follower reached at 2024-01-09 05:09:57.849456457 +0000 UTC m=+9.102507588 (2 nodes), 65.83085ms from start of poll, 66.473384ms from cluster start. Timeout at 2024-01-09 05:09:57.95455978 +0000 UTC m=+9.207610974, 105.103386ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.954Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:57.954Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:58.059Z [INFO] stable state for Leader reached at 2024-01-09 05:09:57.954774353 +0000 UTC m=+9.207825470 (1 nodes), 2.417µs from start of poll, 171.791266ms from cluster start. Timeout at 2024-01-09 05:09:58.0598681 +0000 UTC m=+9.312919240, 105.09377ms after stability - testing.go:149: cluster: 2024-01-09T05:09:58.059Z [INFO] stable state for Leader reached at 2024-01-09 05:09:57.954774353 +0000 UTC m=+9.207825470 (1 nodes), 2.417µs from start of poll, 171.791266ms from cluster start. Timeout at 2024-01-09 05:09:58.0598681 +0000 UTC m=+9.312919240, 105.09377ms after stability -testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.066Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: updating configuration: command=RemoveServer server-id=server-637e3d76-fedb-7ba5-2cc2-895de972afce server-addr= servers="[{Suffrage:Voter ID:server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd Address:9a5bce64-da87-eed0-1c8c-d753d3ab43bd} {Suffrage:Voter ID:server-e8bf385c-490a-2395-38fd-ba0d812e9332 Address:e8bf385c-490a-2395-38fd-ba0d812e9332}]" - testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.066Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: updating configuration: command=RemoveServer server-id=server-637e3d76-fedb-7ba5-2cc2-895de972afce server-addr= servers="[{Suffrage:Voter ID:server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd Address:9a5bce64-da87-eed0-1c8c-d753d3ab43bd} {Suffrage:Voter ID:server-e8bf385c-490a-2395-38fd-ba0d812e9332 Address:e8bf385c-490a-2395-38fd-ba0d812e9332}]" -testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.066Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: removed ourself, transitioning to follower - testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.066Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: removed ourself, transitioning to follower -testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.067Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: entering follower state: follower="Node at 637e3d76-fedb-7ba5-2cc2-895de972afce [Follower]" leader= - testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.067Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: entering follower state: follower="Node at 637e3d76-fedb-7ba5-2cc2-895de972afce [Follower]" leader= -testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.067Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: aborting pipeline replication: peer="{Voter server-e8bf385c-490a-2395-38fd-ba0d812e9332 e8bf385c-490a-2395-38fd-ba0d812e9332}" - testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.067Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: aborting pipeline replication: peer="{Voter server-e8bf385c-490a-2395-38fd-ba0d812e9332 e8bf385c-490a-2395-38fd-ba0d812e9332}" -testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.067Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: aborting pipeline replication: peer="{Voter server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd 9a5bce64-da87-eed0-1c8c-d753d3ab43bd}" - testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.067Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: aborting pipeline replication: peer="{Voter server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd 9a5bce64-da87-eed0-1c8c-d753d3ab43bd}" -testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.117Z [WARN] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: heartbeat timeout reached, starting election: last-leader=637e3d76-fedb-7ba5-2cc2-895de972afce - testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.117Z [WARN] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: heartbeat timeout reached, starting election: last-leader=637e3d76-fedb-7ba5-2cc2-895de972afce -testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.117Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: entering candidate state: node="Node at 9a5bce64-da87-eed0-1c8c-d753d3ab43bd [Candidate]" term=3 - testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.117Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: entering candidate state: node="Node at 9a5bce64-da87-eed0-1c8c-d753d3ab43bd [Candidate]" term=3 -testLoggerAdapter verbose: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:58.118Z [WARN] server-e8bf385c-490a-2395-38fd-ba0d812e9332: rejecting vote request since we have a leader: from=9a5bce64-da87-eed0-1c8c-d753d3ab43bd leader=637e3d76-fedb-7ba5-2cc2-895de972afce - testing.go:149: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:58.118Z [WARN] server-e8bf385c-490a-2395-38fd-ba0d812e9332: rejecting vote request since we have a leader: from=9a5bce64-da87-eed0-1c8c-d753d3ab43bd leader=637e3d76-fedb-7ba5-2cc2-895de972afce -testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.160Z [WARN] server-637e3d76-fedb-7ba5-2cc2-895de972afce: not part of stable configuration, aborting election - testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.160Z [WARN] server-637e3d76-fedb-7ba5-2cc2-895de972afce: not part of stable configuration, aborting election -testLoggerAdapter verbose: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:58.170Z [WARN] server-e8bf385c-490a-2395-38fd-ba0d812e9332: heartbeat timeout reached, starting election: last-leader=637e3d76-fedb-7ba5-2cc2-895de972afce - testing.go:149: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:58.170Z [WARN] server-e8bf385c-490a-2395-38fd-ba0d812e9332: heartbeat timeout reached, starting election: last-leader=637e3d76-fedb-7ba5-2cc2-895de972afce -testLoggerAdapter verbose: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:58.170Z [INFO] server-e8bf385c-490a-2395-38fd-ba0d812e9332: entering candidate state: node="Node at e8bf385c-490a-2395-38fd-ba0d812e9332 [Candidate]" term=3 - testing.go:149: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:58.170Z [INFO] server-e8bf385c-490a-2395-38fd-ba0d812e9332: entering candidate state: node="Node at e8bf385c-490a-2395-38fd-ba0d812e9332 [Candidate]" term=3 -testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.170Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: duplicate requestVote for same term: term=3 - testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.170Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: duplicate requestVote for same term: term=3 -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:58.173Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:58.173Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.216Z [WARN] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: Election timeout reached, restarting election - testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.216Z [WARN] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: Election timeout reached, restarting election -testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.216Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: entering candidate state: node="Node at 9a5bce64-da87-eed0-1c8c-d753d3ab43bd [Candidate]" term=4 - testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.216Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: entering candidate state: node="Node at 9a5bce64-da87-eed0-1c8c-d753d3ab43bd [Candidate]" term=4 -testLoggerAdapter verbose: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:58.216Z [INFO] server-e8bf385c-490a-2395-38fd-ba0d812e9332: entering follower state: follower="Node at e8bf385c-490a-2395-38fd-ba0d812e9332 [Follower]" leader= - testing.go:149: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:58.216Z [INFO] server-e8bf385c-490a-2395-38fd-ba0d812e9332: entering follower state: follower="Node at e8bf385c-490a-2395-38fd-ba0d812e9332 [Follower]" leader= -testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.216Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: election won: tally=2 - testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.216Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: election won: tally=2 -testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.216Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: entering leader state: leader="Node at 9a5bce64-da87-eed0-1c8c-d753d3ab43bd [Leader]" - testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.216Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: entering leader state: leader="Node at 9a5bce64-da87-eed0-1c8c-d753d3ab43bd [Leader]" -testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.217Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: added peer, starting replication: peer=server-e8bf385c-490a-2395-38fd-ba0d812e9332 - testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.217Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: added peer, starting replication: peer=server-e8bf385c-490a-2395-38fd-ba0d812e9332 -testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.217Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: pipelining replication: peer="{Voter server-e8bf385c-490a-2395-38fd-ba0d812e9332 e8bf385c-490a-2395-38fd-ba0d812e9332}" - testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.217Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: pipelining replication: peer="{Voter server-e8bf385c-490a-2395-38fd-ba0d812e9332 e8bf385c-490a-2395-38fd-ba0d812e9332}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:58.323Z [INFO] stable state for Leader reached at 2024-01-09 05:09:58.217332903 +0000 UTC m=+9.470384039 (1 nodes), 43.958323ms from start of poll, 434.349835ms from cluster start. Timeout at 2024-01-09 05:09:58.322960517 +0000 UTC m=+9.576011784, 105.627745ms after stability - testing.go:149: cluster: 2024-01-09T05:09:58.323Z [INFO] stable state for Leader reached at 2024-01-09 05:09:58.217332903 +0000 UTC m=+9.470384039 (1 nodes), 43.958323ms from start of poll, 434.349835ms from cluster start. Timeout at 2024-01-09 05:09:58.322960517 +0000 UTC m=+9.576011784, 105.627745ms after stability -testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.435Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: aborting pipeline replication: peer="{Voter server-e8bf385c-490a-2395-38fd-ba0d812e9332 e8bf385c-490a-2395-38fd-ba0d812e9332}" - testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.435Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: aborting pipeline replication: peer="{Voter server-e8bf385c-490a-2395-38fd-ba0d812e9332 e8bf385c-490a-2395-38fd-ba0d812e9332}" ---- PASS: TestRaft_RemoveLeader_NoShutdown (0.65s) +testLoggerAdapter verbose: server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: 2025-02-10T11:34:12.284Z [INFO] server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 Address:e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44} {Suffrage:Voter ID:server-65e22077-d0d6-575f-959b-7c218aa329fd Address:65e22077-d0d6-575f-959b-7c218aa329fd} {Suffrage:Voter ID:server-7387e957-3920-f465-8f70-f69cf7046b0c Address:7387e957-3920-f465-8f70-f69cf7046b0c}]" + testing.go:149: server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: 2025-02-10T11:34:12.284Z [INFO] server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 Address:e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44} {Suffrage:Voter ID:server-65e22077-d0d6-575f-959b-7c218aa329fd Address:65e22077-d0d6-575f-959b-7c218aa329fd} {Suffrage:Voter ID:server-7387e957-3920-f465-8f70-f69cf7046b0c Address:7387e957-3920-f465-8f70-f69cf7046b0c}]" +testLoggerAdapter verbose: server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: 2025-02-10T11:34:12.284Z [INFO] server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: entering follower state: follower="Node at e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 [Follower]" leader= + testing.go:149: server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: 2025-02-10T11:34:12.284Z [INFO] server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: entering follower state: follower="Node at e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 [Follower]" leader= +testLoggerAdapter verbose: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.284Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 Address:e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44} {Suffrage:Voter ID:server-65e22077-d0d6-575f-959b-7c218aa329fd Address:65e22077-d0d6-575f-959b-7c218aa329fd} {Suffrage:Voter ID:server-7387e957-3920-f465-8f70-f69cf7046b0c Address:7387e957-3920-f465-8f70-f69cf7046b0c}]" + testing.go:149: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.284Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 Address:e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44} {Suffrage:Voter ID:server-65e22077-d0d6-575f-959b-7c218aa329fd Address:65e22077-d0d6-575f-959b-7c218aa329fd} {Suffrage:Voter ID:server-7387e957-3920-f465-8f70-f69cf7046b0c Address:7387e957-3920-f465-8f70-f69cf7046b0c}]" +testLoggerAdapter verbose: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.284Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: entering follower state: follower="Node at 65e22077-d0d6-575f-959b-7c218aa329fd [Follower]" leader= + testing.go:149: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.284Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: entering follower state: follower="Node at 65e22077-d0d6-575f-959b-7c218aa329fd [Follower]" leader= +testLoggerAdapter verbose: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.284Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 Address:e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44} {Suffrage:Voter ID:server-65e22077-d0d6-575f-959b-7c218aa329fd Address:65e22077-d0d6-575f-959b-7c218aa329fd} {Suffrage:Voter ID:server-7387e957-3920-f465-8f70-f69cf7046b0c Address:7387e957-3920-f465-8f70-f69cf7046b0c}]" + testing.go:149: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.284Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 Address:e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44} {Suffrage:Voter ID:server-65e22077-d0d6-575f-959b-7c218aa329fd Address:65e22077-d0d6-575f-959b-7c218aa329fd} {Suffrage:Voter ID:server-7387e957-3920-f465-8f70-f69cf7046b0c Address:7387e957-3920-f465-8f70-f69cf7046b0c}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:12.284Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:12.284Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.284Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: entering follower state: follower="Node at 7387e957-3920-f465-8f70-f69cf7046b0c [Follower]" leader= + testing.go:149: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.284Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: entering follower state: follower="Node at 7387e957-3920-f465-8f70-f69cf7046b0c [Follower]" leader= +testLoggerAdapter verbose: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.346Z [WARN] server-7387e957-3920-f465-8f70-f69cf7046b0c: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.346Z [WARN] server-7387e957-3920-f465-8f70-f69cf7046b0c: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.346Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: entering candidate state: node="Node at 7387e957-3920-f465-8f70-f69cf7046b0c [Candidate]" term=2 + testing.go:149: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.346Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: entering candidate state: node="Node at 7387e957-3920-f465-8f70-f69cf7046b0c [Candidate]" term=2 +testLoggerAdapter verbose: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.346Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: election won: tally=2 + testing.go:149: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.346Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: election won: tally=2 +testLoggerAdapter verbose: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.346Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: entering leader state: leader="Node at 7387e957-3920-f465-8f70-f69cf7046b0c [Leader]" + testing.go:149: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.346Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: entering leader state: leader="Node at 7387e957-3920-f465-8f70-f69cf7046b0c [Leader]" +testLoggerAdapter verbose: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.346Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: added peer, starting replication: peer=server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 + testing.go:149: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.346Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: added peer, starting replication: peer=server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 +testLoggerAdapter verbose: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.346Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: added peer, starting replication: peer=server-65e22077-d0d6-575f-959b-7c218aa329fd + testing.go:149: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.346Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: added peer, starting replication: peer=server-65e22077-d0d6-575f-959b-7c218aa329fd +testLoggerAdapter verbose: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.346Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: pipelining replication: peer="{Voter server-65e22077-d0d6-575f-959b-7c218aa329fd 65e22077-d0d6-575f-959b-7c218aa329fd}" + testing.go:149: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.346Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: pipelining replication: peer="{Voter server-65e22077-d0d6-575f-959b-7c218aa329fd 65e22077-d0d6-575f-959b-7c218aa329fd}" +testLoggerAdapter verbose: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.346Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: pipelining replication: peer="{Voter server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44}" + testing.go:149: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.346Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: pipelining replication: peer="{Voter server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:12.451Z [INFO] stable state for Follower reached at 2025-02-10 11:34:12.346255138 +0000 UTC m=+8.715159439 (2 nodes), 61.331252ms from start of poll, 61.715667ms from cluster start. Timeout at 2025-02-10 11:34:12.451327724 +0000 UTC m=+8.820232018, 105.072579ms after stability + testing.go:149: cluster: 2025-02-10T11:34:12.451Z [INFO] stable state for Follower reached at 2025-02-10 11:34:12.346255138 +0000 UTC m=+8.715159439 (2 nodes), 61.331252ms from start of poll, 61.715667ms from cluster start. Timeout at 2025-02-10 11:34:12.451327724 +0000 UTC m=+8.820232018, 105.072579ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:12.451Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:12.451Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:12.556Z [INFO] stable state for Leader reached at 2025-02-10 11:34:12.451408698 +0000 UTC m=+8.820312998 (1 nodes), 492ns from start of poll, 166.869226ms from cluster start. Timeout at 2025-02-10 11:34:12.556488313 +0000 UTC m=+8.925392610, 105.079612ms after stability + testing.go:149: cluster: 2025-02-10T11:34:12.556Z [INFO] stable state for Leader reached at 2025-02-10 11:34:12.451408698 +0000 UTC m=+8.820312998 (1 nodes), 492ns from start of poll, 166.869226ms from cluster start. Timeout at 2025-02-10 11:34:12.556488313 +0000 UTC m=+8.925392610, 105.079612ms after stability +testLoggerAdapter verbose: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.557Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: updating configuration: command=RemoveServer server-id=server-7387e957-3920-f465-8f70-f69cf7046b0c server-addr= servers="[{Suffrage:Voter ID:server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 Address:e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44} {Suffrage:Voter ID:server-65e22077-d0d6-575f-959b-7c218aa329fd Address:65e22077-d0d6-575f-959b-7c218aa329fd}]" + testing.go:149: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.557Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: updating configuration: command=RemoveServer server-id=server-7387e957-3920-f465-8f70-f69cf7046b0c server-addr= servers="[{Suffrage:Voter ID:server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 Address:e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44} {Suffrage:Voter ID:server-65e22077-d0d6-575f-959b-7c218aa329fd Address:65e22077-d0d6-575f-959b-7c218aa329fd}]" +testLoggerAdapter verbose: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.557Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: removed ourself, transitioning to follower + testing.go:149: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.557Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: removed ourself, transitioning to follower +testLoggerAdapter verbose: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.557Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: entering follower state: follower="Node at 7387e957-3920-f465-8f70-f69cf7046b0c [Follower]" leader= + testing.go:149: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.557Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: entering follower state: follower="Node at 7387e957-3920-f465-8f70-f69cf7046b0c [Follower]" leader= +testLoggerAdapter verbose: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.557Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: aborting pipeline replication: peer="{Voter server-65e22077-d0d6-575f-959b-7c218aa329fd 65e22077-d0d6-575f-959b-7c218aa329fd}" + testing.go:149: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.557Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: aborting pipeline replication: peer="{Voter server-65e22077-d0d6-575f-959b-7c218aa329fd 65e22077-d0d6-575f-959b-7c218aa329fd}" +testLoggerAdapter verbose: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.557Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: aborting pipeline replication: peer="{Voter server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44}" + testing.go:149: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.557Z [INFO] server-7387e957-3920-f465-8f70-f69cf7046b0c: aborting pipeline replication: peer="{Voter server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44}" +testLoggerAdapter verbose: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.623Z [WARN] server-65e22077-d0d6-575f-959b-7c218aa329fd: heartbeat timeout reached, starting election: last-leader=7387e957-3920-f465-8f70-f69cf7046b0c + testing.go:149: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.623Z [WARN] server-65e22077-d0d6-575f-959b-7c218aa329fd: heartbeat timeout reached, starting election: last-leader=7387e957-3920-f465-8f70-f69cf7046b0c +testLoggerAdapter verbose: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.623Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: entering candidate state: node="Node at 65e22077-d0d6-575f-959b-7c218aa329fd [Candidate]" term=3 + testing.go:149: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.623Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: entering candidate state: node="Node at 65e22077-d0d6-575f-959b-7c218aa329fd [Candidate]" term=3 +testLoggerAdapter verbose: server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: 2025-02-10T11:34:12.623Z [WARN] server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: rejecting vote request since we have a leader: from=65e22077-d0d6-575f-959b-7c218aa329fd leader=7387e957-3920-f465-8f70-f69cf7046b0c + testing.go:149: server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: 2025-02-10T11:34:12.623Z [WARN] server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: rejecting vote request since we have a leader: from=65e22077-d0d6-575f-959b-7c218aa329fd leader=7387e957-3920-f465-8f70-f69cf7046b0c +testLoggerAdapter verbose: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.651Z [WARN] server-7387e957-3920-f465-8f70-f69cf7046b0c: not part of stable configuration, aborting election + testing.go:149: server-7387e957-3920-f465-8f70-f69cf7046b0c: 2025-02-10T11:34:12.651Z [WARN] server-7387e957-3920-f465-8f70-f69cf7046b0c: not part of stable configuration, aborting election +testLoggerAdapter verbose: server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: 2025-02-10T11:34:12.655Z [WARN] server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: heartbeat timeout reached, starting election: last-leader=7387e957-3920-f465-8f70-f69cf7046b0c + testing.go:149: server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: 2025-02-10T11:34:12.655Z [WARN] server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: heartbeat timeout reached, starting election: last-leader=7387e957-3920-f465-8f70-f69cf7046b0c +testLoggerAdapter verbose: server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: 2025-02-10T11:34:12.655Z [INFO] server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: entering candidate state: node="Node at e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 [Candidate]" term=3 + testing.go:149: server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: 2025-02-10T11:34:12.655Z [INFO] server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: entering candidate state: node="Node at e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 [Candidate]" term=3 +testLoggerAdapter verbose: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.655Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: duplicate requestVote for same term: term=3 + testing.go:149: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.655Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: duplicate requestVote for same term: term=3 +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:12.662Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:12.662Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.695Z [WARN] server-65e22077-d0d6-575f-959b-7c218aa329fd: Election timeout reached, restarting election + testing.go:149: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.695Z [WARN] server-65e22077-d0d6-575f-959b-7c218aa329fd: Election timeout reached, restarting election +testLoggerAdapter verbose: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.695Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: entering candidate state: node="Node at 65e22077-d0d6-575f-959b-7c218aa329fd [Candidate]" term=4 + testing.go:149: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.695Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: entering candidate state: node="Node at 65e22077-d0d6-575f-959b-7c218aa329fd [Candidate]" term=4 +testLoggerAdapter verbose: server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: 2025-02-10T11:34:12.695Z [INFO] server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: entering follower state: follower="Node at e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 [Follower]" leader= + testing.go:149: server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: 2025-02-10T11:34:12.695Z [INFO] server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44: entering follower state: follower="Node at e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 [Follower]" leader= +testLoggerAdapter verbose: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.695Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: election won: tally=2 + testing.go:149: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.695Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: election won: tally=2 +testLoggerAdapter verbose: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.695Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: entering leader state: leader="Node at 65e22077-d0d6-575f-959b-7c218aa329fd [Leader]" + testing.go:149: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.695Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: entering leader state: leader="Node at 65e22077-d0d6-575f-959b-7c218aa329fd [Leader]" +testLoggerAdapter verbose: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.695Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: added peer, starting replication: peer=server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 + testing.go:149: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.695Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: added peer, starting replication: peer=server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 +testLoggerAdapter verbose: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.695Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: pipelining replication: peer="{Voter server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44}" + testing.go:149: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.695Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: pipelining replication: peer="{Voter server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:12.801Z [INFO] stable state for Leader reached at 2025-02-10 11:34:12.695883237 +0000 UTC m=+9.064787555 (1 nodes), 33.261908ms from start of poll, 411.343783ms from cluster start. Timeout at 2025-02-10 11:34:12.80098997 +0000 UTC m=+9.169894323, 105.106768ms after stability + testing.go:149: cluster: 2025-02-10T11:34:12.801Z [INFO] stable state for Leader reached at 2025-02-10 11:34:12.695883237 +0000 UTC m=+9.064787555 (1 nodes), 33.261908ms from start of poll, 411.343783ms from cluster start. Timeout at 2025-02-10 11:34:12.80098997 +0000 UTC m=+9.169894323, 105.106768ms after stability +testLoggerAdapter verbose: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.906Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: aborting pipeline replication: peer="{Voter server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44}" + testing.go:149: server-65e22077-d0d6-575f-959b-7c218aa329fd: 2025-02-10T11:34:12.906Z [INFO] server-65e22077-d0d6-575f-959b-7c218aa329fd: aborting pipeline replication: peer="{Voter server-e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44 e1cc21d9-3c6a-ff00-3645-c3f4e3f73b44}" +--- PASS: TestRaft_RemoveLeader_NoShutdown (0.62s) === RUN TestRaft_RemoveFollower_SplitCluster -testLoggerAdapter verbose: server-8e63e08b-8779-5ecf-6579-0337bd0b7259: 2024-01-09T05:09:58.437Z [INFO] server-8e63e08b-8779-5ecf-6579-0337bd0b7259: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 Address:dfab7669-1d70-4fbc-1a53-ab8358b776d5} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" - testing.go:149: server-8e63e08b-8779-5ecf-6579-0337bd0b7259: 2024-01-09T05:09:58.437Z [INFO] server-8e63e08b-8779-5ecf-6579-0337bd0b7259: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 Address:dfab7669-1d70-4fbc-1a53-ab8358b776d5} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" -testLoggerAdapter verbose: server-5db5b671-affb-8f8c-8841-e12234d7bd6b: 2024-01-09T05:09:58.437Z [INFO] server-5db5b671-affb-8f8c-8841-e12234d7bd6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 Address:dfab7669-1d70-4fbc-1a53-ab8358b776d5} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" - testing.go:149: server-5db5b671-affb-8f8c-8841-e12234d7bd6b: 2024-01-09T05:09:58.437Z [INFO] server-5db5b671-affb-8f8c-8841-e12234d7bd6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 Address:dfab7669-1d70-4fbc-1a53-ab8358b776d5} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" -testLoggerAdapter verbose: server-8e63e08b-8779-5ecf-6579-0337bd0b7259: 2024-01-09T05:09:58.437Z [INFO] server-8e63e08b-8779-5ecf-6579-0337bd0b7259: entering follower state: follower="Node at 8e63e08b-8779-5ecf-6579-0337bd0b7259 [Follower]" leader= - testing.go:149: server-8e63e08b-8779-5ecf-6579-0337bd0b7259: 2024-01-09T05:09:58.437Z [INFO] server-8e63e08b-8779-5ecf-6579-0337bd0b7259: entering follower state: follower="Node at 8e63e08b-8779-5ecf-6579-0337bd0b7259 [Follower]" leader= -testLoggerAdapter verbose: server-5db5b671-affb-8f8c-8841-e12234d7bd6b: 2024-01-09T05:09:58.437Z [INFO] server-5db5b671-affb-8f8c-8841-e12234d7bd6b: entering follower state: follower="Node at 5db5b671-affb-8f8c-8841-e12234d7bd6b [Follower]" leader= - testing.go:149: server-5db5b671-affb-8f8c-8841-e12234d7bd6b: 2024-01-09T05:09:58.437Z [INFO] server-5db5b671-affb-8f8c-8841-e12234d7bd6b: entering follower state: follower="Node at 5db5b671-affb-8f8c-8841-e12234d7bd6b [Follower]" leader= -testLoggerAdapter verbose: server-dfab7669-1d70-4fbc-1a53-ab8358b776d5: 2024-01-09T05:09:58.437Z [INFO] server-dfab7669-1d70-4fbc-1a53-ab8358b776d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 Address:dfab7669-1d70-4fbc-1a53-ab8358b776d5} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" - testing.go:149: server-dfab7669-1d70-4fbc-1a53-ab8358b776d5: 2024-01-09T05:09:58.437Z [INFO] server-dfab7669-1d70-4fbc-1a53-ab8358b776d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 Address:dfab7669-1d70-4fbc-1a53-ab8358b776d5} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.437Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 Address:dfab7669-1d70-4fbc-1a53-ab8358b776d5} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.437Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 Address:dfab7669-1d70-4fbc-1a53-ab8358b776d5} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:58.437Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:58.437Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-dfab7669-1d70-4fbc-1a53-ab8358b776d5: 2024-01-09T05:09:58.437Z [INFO] server-dfab7669-1d70-4fbc-1a53-ab8358b776d5: entering follower state: follower="Node at dfab7669-1d70-4fbc-1a53-ab8358b776d5 [Follower]" leader= - testing.go:149: server-dfab7669-1d70-4fbc-1a53-ab8358b776d5: 2024-01-09T05:09:58.437Z [INFO] server-dfab7669-1d70-4fbc-1a53-ab8358b776d5: entering follower state: follower="Node at dfab7669-1d70-4fbc-1a53-ab8358b776d5 [Follower]" leader= -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.437Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: entering follower state: follower="Node at f82cb951-6b0e-596d-ad0a-c1c6c66295c0 [Follower]" leader= - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.437Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: entering follower state: follower="Node at f82cb951-6b0e-596d-ad0a-c1c6c66295c0 [Follower]" leader= -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: entering candidate state: node="Node at f82cb951-6b0e-596d-ad0a-c1c6c66295c0 [Candidate]" term=2 - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: entering candidate state: node="Node at f82cb951-6b0e-596d-ad0a-c1c6c66295c0 [Candidate]" term=2 -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: election won: tally=3 - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: election won: tally=3 -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: entering leader state: leader="Node at f82cb951-6b0e-596d-ad0a-c1c6c66295c0 [Leader]" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: entering leader state: leader="Node at f82cb951-6b0e-596d-ad0a-c1c6c66295c0 [Leader]" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: added peer, starting replication: peer=server-8e63e08b-8779-5ecf-6579-0337bd0b7259 - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: added peer, starting replication: peer=server-8e63e08b-8779-5ecf-6579-0337bd0b7259 -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: added peer, starting replication: peer=server-5db5b671-affb-8f8c-8841-e12234d7bd6b - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: added peer, starting replication: peer=server-5db5b671-affb-8f8c-8841-e12234d7bd6b -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: added peer, starting replication: peer=server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: added peer, starting replication: peer=server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: pipelining replication: peer="{Voter server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 dfab7669-1d70-4fbc-1a53-ab8358b776d5}" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: pipelining replication: peer="{Voter server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 dfab7669-1d70-4fbc-1a53-ab8358b776d5}" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: pipelining replication: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: pipelining replication: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.520Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: pipelining replication: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.520Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: pipelining replication: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:58.625Z [INFO] stable state for Leader reached at 2024-01-09 05:09:58.519887674 +0000 UTC m=+9.772938728 (1 nodes), 82.068495ms from start of poll, 82.860903ms from cluster start. Timeout at 2024-01-09 05:09:58.625002025 +0000 UTC m=+9.878053192, 105.114464ms after stability - testing.go:149: cluster: 2024-01-09T05:09:58.625Z [INFO] stable state for Leader reached at 2024-01-09 05:09:58.519887674 +0000 UTC m=+9.772938728 (1 nodes), 82.068495ms from start of poll, 82.860903ms from cluster start. Timeout at 2024-01-09 05:09:58.625002025 +0000 UTC m=+9.878053192, 105.114464ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:58.735Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:09:58.735Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:58.840Z [INFO] stable state for Follower reached at 2024-01-09 05:09:58.735221379 +0000 UTC m=+9.988272520 (3 nodes), 2.938µs from start of poll, 298.194695ms from cluster start. Timeout at 2024-01-09 05:09:58.840315884 +0000 UTC m=+10.093366986, 105.094466ms after stability - testing.go:149: cluster: 2024-01-09T05:09:58.840Z [INFO] stable state for Follower reached at 2024-01-09 05:09:58.735221379 +0000 UTC m=+9.988272520 (3 nodes), 2.938µs from start of poll, 298.194695ms from cluster start. Timeout at 2024-01-09 05:09:58.840315884 +0000 UTC m=+10.093366986, 105.094466ms after stability -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: updating configuration: command=RemoveServer server-id=server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 server-addr= servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: updating configuration: command=RemoveServer server-id=server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 server-addr= servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: removed peer, stopping replication: peer=server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 last-index=3 - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: removed peer, stopping replication: peer=server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 last-index=3 -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to pipeline appendEntries: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="append pipeline closed" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to pipeline appendEntries: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="append pipeline closed" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: aborting pipeline replication: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: aborting pipeline replication: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to pipeline appendEntries: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="append pipeline closed" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to pipeline appendEntries: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="append pipeline closed" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: aborting pipeline replication: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: aborting pipeline replication: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: aborting pipeline replication: peer="{Voter server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 dfab7669-1d70-4fbc-1a53-ab8358b776d5}" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: aborting pipeline replication: peer="{Voter server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 dfab7669-1d70-4fbc-1a53-ab8358b776d5}" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.841Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=5db5b671-affb-8f8c-8841-e12234d7bd6b error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.841Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=5db5b671-affb-8f8c-8841-e12234d7bd6b error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.843Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=8e63e08b-8779-5ecf-6579-0337bd0b7259 error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.843Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=8e63e08b-8779-5ecf-6579-0337bd0b7259 error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.848Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.848Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.850Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.850Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.859Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=8e63e08b-8779-5ecf-6579-0337bd0b7259 error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.859Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=8e63e08b-8779-5ecf-6579-0337bd0b7259 error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.860Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=5db5b671-affb-8f8c-8841-e12234d7bd6b error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.860Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=5db5b671-affb-8f8c-8841-e12234d7bd6b error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.868Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.868Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.868Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.868Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.877Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=8e63e08b-8779-5ecf-6579-0337bd0b7259 error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.877Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=8e63e08b-8779-5ecf-6579-0337bd0b7259 error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.879Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=5db5b671-affb-8f8c-8841-e12234d7bd6b error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.879Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=5db5b671-affb-8f8c-8841-e12234d7bd6b error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.886Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to contact: server-id=server-5db5b671-affb-8f8c-8841-e12234d7bd6b time=50.273422ms - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.886Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to contact: server-id=server-5db5b671-affb-8f8c-8841-e12234d7bd6b time=50.273422ms -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.886Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.886Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.887Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.887Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to contact: server-id=server-8e63e08b-8779-5ecf-6579-0337bd0b7259 time=59.445109ms - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to contact: server-id=server-8e63e08b-8779-5ecf-6579-0337bd0b7259 time=59.445109ms -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to contact: server-id=server-5db5b671-affb-8f8c-8841-e12234d7bd6b time=60.646174ms - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to contact: server-id=server-5db5b671-affb-8f8c-8841-e12234d7bd6b time=60.646174ms -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to contact quorum of nodes, stepping down - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to contact quorum of nodes, stepping down -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: entering follower state: follower="Node at f82cb951-6b0e-596d-ad0a-c1c6c66295c0 [Follower]" leader= - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: entering follower state: follower="Node at f82cb951-6b0e-596d-ad0a-c1c6c66295c0 [Follower]" leader= -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" -testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" - testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" ---- PASS: TestRaft_RemoveFollower_SplitCluster (0.46s) +testLoggerAdapter verbose: server-24c3003c-4c9c-025e-ff01-255bebc31000: 2025-02-10T11:34:12.907Z [INFO] server-24c3003c-4c9c-025e-ff01-255bebc31000: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24c3003c-4c9c-025e-ff01-255bebc31000 Address:24c3003c-4c9c-025e-ff01-255bebc31000} {Suffrage:Voter ID:server-60141061-f7a9-4547-c487-7876db3d60b5 Address:60141061-f7a9-4547-c487-7876db3d60b5} {Suffrage:Voter ID:server-b9fab61a-5861-d93f-09b6-920663dfde4d Address:b9fab61a-5861-d93f-09b6-920663dfde4d} {Suffrage:Voter ID:server-d687b9b3-9467-cd46-cea1-c8f0e48c612b Address:d687b9b3-9467-cd46-cea1-c8f0e48c612b}]" + testing.go:149: server-24c3003c-4c9c-025e-ff01-255bebc31000: 2025-02-10T11:34:12.907Z [INFO] server-24c3003c-4c9c-025e-ff01-255bebc31000: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24c3003c-4c9c-025e-ff01-255bebc31000 Address:24c3003c-4c9c-025e-ff01-255bebc31000} {Suffrage:Voter ID:server-60141061-f7a9-4547-c487-7876db3d60b5 Address:60141061-f7a9-4547-c487-7876db3d60b5} {Suffrage:Voter ID:server-b9fab61a-5861-d93f-09b6-920663dfde4d Address:b9fab61a-5861-d93f-09b6-920663dfde4d} {Suffrage:Voter ID:server-d687b9b3-9467-cd46-cea1-c8f0e48c612b Address:d687b9b3-9467-cd46-cea1-c8f0e48c612b}]" +testLoggerAdapter verbose: server-24c3003c-4c9c-025e-ff01-255bebc31000: 2025-02-10T11:34:12.907Z [INFO] server-24c3003c-4c9c-025e-ff01-255bebc31000: entering follower state: follower="Node at 24c3003c-4c9c-025e-ff01-255bebc31000 [Follower]" leader= + testing.go:149: server-24c3003c-4c9c-025e-ff01-255bebc31000: 2025-02-10T11:34:12.907Z [INFO] server-24c3003c-4c9c-025e-ff01-255bebc31000: entering follower state: follower="Node at 24c3003c-4c9c-025e-ff01-255bebc31000 [Follower]" leader= +testLoggerAdapter verbose: server-60141061-f7a9-4547-c487-7876db3d60b5: 2025-02-10T11:34:12.907Z [INFO] server-60141061-f7a9-4547-c487-7876db3d60b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24c3003c-4c9c-025e-ff01-255bebc31000 Address:24c3003c-4c9c-025e-ff01-255bebc31000} {Suffrage:Voter ID:server-60141061-f7a9-4547-c487-7876db3d60b5 Address:60141061-f7a9-4547-c487-7876db3d60b5} {Suffrage:Voter ID:server-b9fab61a-5861-d93f-09b6-920663dfde4d Address:b9fab61a-5861-d93f-09b6-920663dfde4d} {Suffrage:Voter ID:server-d687b9b3-9467-cd46-cea1-c8f0e48c612b Address:d687b9b3-9467-cd46-cea1-c8f0e48c612b}]" + testing.go:149: server-60141061-f7a9-4547-c487-7876db3d60b5: 2025-02-10T11:34:12.907Z [INFO] server-60141061-f7a9-4547-c487-7876db3d60b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24c3003c-4c9c-025e-ff01-255bebc31000 Address:24c3003c-4c9c-025e-ff01-255bebc31000} {Suffrage:Voter ID:server-60141061-f7a9-4547-c487-7876db3d60b5 Address:60141061-f7a9-4547-c487-7876db3d60b5} {Suffrage:Voter ID:server-b9fab61a-5861-d93f-09b6-920663dfde4d Address:b9fab61a-5861-d93f-09b6-920663dfde4d} {Suffrage:Voter ID:server-d687b9b3-9467-cd46-cea1-c8f0e48c612b Address:d687b9b3-9467-cd46-cea1-c8f0e48c612b}]" +testLoggerAdapter verbose: server-60141061-f7a9-4547-c487-7876db3d60b5: 2025-02-10T11:34:12.907Z [INFO] server-60141061-f7a9-4547-c487-7876db3d60b5: entering follower state: follower="Node at 60141061-f7a9-4547-c487-7876db3d60b5 [Follower]" leader= + testing.go:149: server-60141061-f7a9-4547-c487-7876db3d60b5: 2025-02-10T11:34:12.907Z [INFO] server-60141061-f7a9-4547-c487-7876db3d60b5: entering follower state: follower="Node at 60141061-f7a9-4547-c487-7876db3d60b5 [Follower]" leader= +testLoggerAdapter verbose: server-b9fab61a-5861-d93f-09b6-920663dfde4d: 2025-02-10T11:34:12.907Z [INFO] server-b9fab61a-5861-d93f-09b6-920663dfde4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24c3003c-4c9c-025e-ff01-255bebc31000 Address:24c3003c-4c9c-025e-ff01-255bebc31000} {Suffrage:Voter ID:server-60141061-f7a9-4547-c487-7876db3d60b5 Address:60141061-f7a9-4547-c487-7876db3d60b5} {Suffrage:Voter ID:server-b9fab61a-5861-d93f-09b6-920663dfde4d Address:b9fab61a-5861-d93f-09b6-920663dfde4d} {Suffrage:Voter ID:server-d687b9b3-9467-cd46-cea1-c8f0e48c612b Address:d687b9b3-9467-cd46-cea1-c8f0e48c612b}]" + testing.go:149: server-b9fab61a-5861-d93f-09b6-920663dfde4d: 2025-02-10T11:34:12.907Z [INFO] server-b9fab61a-5861-d93f-09b6-920663dfde4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24c3003c-4c9c-025e-ff01-255bebc31000 Address:24c3003c-4c9c-025e-ff01-255bebc31000} {Suffrage:Voter ID:server-60141061-f7a9-4547-c487-7876db3d60b5 Address:60141061-f7a9-4547-c487-7876db3d60b5} {Suffrage:Voter ID:server-b9fab61a-5861-d93f-09b6-920663dfde4d Address:b9fab61a-5861-d93f-09b6-920663dfde4d} {Suffrage:Voter ID:server-d687b9b3-9467-cd46-cea1-c8f0e48c612b Address:d687b9b3-9467-cd46-cea1-c8f0e48c612b}]" +testLoggerAdapter verbose: server-b9fab61a-5861-d93f-09b6-920663dfde4d: 2025-02-10T11:34:12.907Z [INFO] server-b9fab61a-5861-d93f-09b6-920663dfde4d: entering follower state: follower="Node at b9fab61a-5861-d93f-09b6-920663dfde4d [Follower]" leader= + testing.go:149: server-b9fab61a-5861-d93f-09b6-920663dfde4d: 2025-02-10T11:34:12.907Z [INFO] server-b9fab61a-5861-d93f-09b6-920663dfde4d: entering follower state: follower="Node at b9fab61a-5861-d93f-09b6-920663dfde4d [Follower]" leader= +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.907Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24c3003c-4c9c-025e-ff01-255bebc31000 Address:24c3003c-4c9c-025e-ff01-255bebc31000} {Suffrage:Voter ID:server-60141061-f7a9-4547-c487-7876db3d60b5 Address:60141061-f7a9-4547-c487-7876db3d60b5} {Suffrage:Voter ID:server-b9fab61a-5861-d93f-09b6-920663dfde4d Address:b9fab61a-5861-d93f-09b6-920663dfde4d} {Suffrage:Voter ID:server-d687b9b3-9467-cd46-cea1-c8f0e48c612b Address:d687b9b3-9467-cd46-cea1-c8f0e48c612b}]" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.907Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24c3003c-4c9c-025e-ff01-255bebc31000 Address:24c3003c-4c9c-025e-ff01-255bebc31000} {Suffrage:Voter ID:server-60141061-f7a9-4547-c487-7876db3d60b5 Address:60141061-f7a9-4547-c487-7876db3d60b5} {Suffrage:Voter ID:server-b9fab61a-5861-d93f-09b6-920663dfde4d Address:b9fab61a-5861-d93f-09b6-920663dfde4d} {Suffrage:Voter ID:server-d687b9b3-9467-cd46-cea1-c8f0e48c612b Address:d687b9b3-9467-cd46-cea1-c8f0e48c612b}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:12.907Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:12.907Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.907Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: entering follower state: follower="Node at d687b9b3-9467-cd46-cea1-c8f0e48c612b [Follower]" leader= + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.907Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: entering follower state: follower="Node at d687b9b3-9467-cd46-cea1-c8f0e48c612b [Follower]" leader= +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [WARN] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [WARN] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: entering candidate state: node="Node at d687b9b3-9467-cd46-cea1-c8f0e48c612b [Candidate]" term=2 + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: entering candidate state: node="Node at d687b9b3-9467-cd46-cea1-c8f0e48c612b [Candidate]" term=2 +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: election won: tally=3 + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: election won: tally=3 +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: entering leader state: leader="Node at d687b9b3-9467-cd46-cea1-c8f0e48c612b [Leader]" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: entering leader state: leader="Node at d687b9b3-9467-cd46-cea1-c8f0e48c612b [Leader]" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: added peer, starting replication: peer=server-24c3003c-4c9c-025e-ff01-255bebc31000 + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: added peer, starting replication: peer=server-24c3003c-4c9c-025e-ff01-255bebc31000 +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: added peer, starting replication: peer=server-60141061-f7a9-4547-c487-7876db3d60b5 + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: added peer, starting replication: peer=server-60141061-f7a9-4547-c487-7876db3d60b5 +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: added peer, starting replication: peer=server-b9fab61a-5861-d93f-09b6-920663dfde4d + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: added peer, starting replication: peer=server-b9fab61a-5861-d93f-09b6-920663dfde4d +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: pipelining replication: peer="{Voter server-24c3003c-4c9c-025e-ff01-255bebc31000 24c3003c-4c9c-025e-ff01-255bebc31000}" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: pipelining replication: peer="{Voter server-24c3003c-4c9c-025e-ff01-255bebc31000 24c3003c-4c9c-025e-ff01-255bebc31000}" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: pipelining replication: peer="{Voter server-60141061-f7a9-4547-c487-7876db3d60b5 60141061-f7a9-4547-c487-7876db3d60b5}" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: pipelining replication: peer="{Voter server-60141061-f7a9-4547-c487-7876db3d60b5 60141061-f7a9-4547-c487-7876db3d60b5}" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: pipelining replication: peer="{Voter server-b9fab61a-5861-d93f-09b6-920663dfde4d b9fab61a-5861-d93f-09b6-920663dfde4d}" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:12.968Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: pipelining replication: peer="{Voter server-b9fab61a-5861-d93f-09b6-920663dfde4d b9fab61a-5861-d93f-09b6-920663dfde4d}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:13.073Z [INFO] stable state for Leader reached at 2025-02-10 11:34:12.968328085 +0000 UTC m=+9.337232383 (1 nodes), 60.540029ms from start of poll, 61.052877ms from cluster start. Timeout at 2025-02-10 11:34:13.073436846 +0000 UTC m=+9.442341252, 105.108869ms after stability + testing.go:149: cluster: 2025-02-10T11:34:13.073Z [INFO] stable state for Leader reached at 2025-02-10 11:34:12.968328085 +0000 UTC m=+9.337232383 (1 nodes), 60.540029ms from start of poll, 61.052877ms from cluster start. Timeout at 2025-02-10 11:34:13.073436846 +0000 UTC m=+9.442341252, 105.108869ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:13.178Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:13.178Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:13.283Z [INFO] stable state for Follower reached at 2025-02-10 11:34:13.178653742 +0000 UTC m=+9.547558043 (3 nodes), 804ns from start of poll, 271.378537ms from cluster start. Timeout at 2025-02-10 11:34:13.283754621 +0000 UTC m=+9.652658973, 105.10093ms after stability + testing.go:149: cluster: 2025-02-10T11:34:13.283Z [INFO] stable state for Follower reached at 2025-02-10 11:34:13.178653742 +0000 UTC m=+9.547558043 (3 nodes), 804ns from start of poll, 271.378537ms from cluster start. Timeout at 2025-02-10 11:34:13.283754621 +0000 UTC m=+9.652658973, 105.10093ms after stability +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.283Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: updating configuration: command=RemoveServer server-id=server-b9fab61a-5861-d93f-09b6-920663dfde4d server-addr= servers="[{Suffrage:Voter ID:server-24c3003c-4c9c-025e-ff01-255bebc31000 Address:24c3003c-4c9c-025e-ff01-255bebc31000} {Suffrage:Voter ID:server-60141061-f7a9-4547-c487-7876db3d60b5 Address:60141061-f7a9-4547-c487-7876db3d60b5} {Suffrage:Voter ID:server-d687b9b3-9467-cd46-cea1-c8f0e48c612b Address:d687b9b3-9467-cd46-cea1-c8f0e48c612b}]" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.283Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: updating configuration: command=RemoveServer server-id=server-b9fab61a-5861-d93f-09b6-920663dfde4d server-addr= servers="[{Suffrage:Voter ID:server-24c3003c-4c9c-025e-ff01-255bebc31000 Address:24c3003c-4c9c-025e-ff01-255bebc31000} {Suffrage:Voter ID:server-60141061-f7a9-4547-c487-7876db3d60b5 Address:60141061-f7a9-4547-c487-7876db3d60b5} {Suffrage:Voter ID:server-d687b9b3-9467-cd46-cea1-c8f0e48c612b Address:d687b9b3-9467-cd46-cea1-c8f0e48c612b}]" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.284Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: removed peer, stopping replication: peer=server-b9fab61a-5861-d93f-09b6-920663dfde4d last-index=3 + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.284Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: removed peer, stopping replication: peer=server-b9fab61a-5861-d93f-09b6-920663dfde4d last-index=3 +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.284Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to pipeline appendEntries: peer="{Voter server-60141061-f7a9-4547-c487-7876db3d60b5 60141061-f7a9-4547-c487-7876db3d60b5}" error="append pipeline closed" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.284Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to pipeline appendEntries: peer="{Voter server-60141061-f7a9-4547-c487-7876db3d60b5 60141061-f7a9-4547-c487-7876db3d60b5}" error="append pipeline closed" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.284Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: aborting pipeline replication: peer="{Voter server-60141061-f7a9-4547-c487-7876db3d60b5 60141061-f7a9-4547-c487-7876db3d60b5}" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.284Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: aborting pipeline replication: peer="{Voter server-60141061-f7a9-4547-c487-7876db3d60b5 60141061-f7a9-4547-c487-7876db3d60b5}" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.284Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: aborting pipeline replication: peer="{Voter server-b9fab61a-5861-d93f-09b6-920663dfde4d b9fab61a-5861-d93f-09b6-920663dfde4d}" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.284Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: aborting pipeline replication: peer="{Voter server-b9fab61a-5861-d93f-09b6-920663dfde4d b9fab61a-5861-d93f-09b6-920663dfde4d}" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.284Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to pipeline appendEntries: peer="{Voter server-24c3003c-4c9c-025e-ff01-255bebc31000 24c3003c-4c9c-025e-ff01-255bebc31000}" error="append pipeline closed" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.284Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to pipeline appendEntries: peer="{Voter server-24c3003c-4c9c-025e-ff01-255bebc31000 24c3003c-4c9c-025e-ff01-255bebc31000}" error="append pipeline closed" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.284Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: aborting pipeline replication: peer="{Voter server-24c3003c-4c9c-025e-ff01-255bebc31000 24c3003c-4c9c-025e-ff01-255bebc31000}" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.284Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: aborting pipeline replication: peer="{Voter server-24c3003c-4c9c-025e-ff01-255bebc31000 24c3003c-4c9c-025e-ff01-255bebc31000}" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.290Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to appendEntries to: peer="{Voter server-24c3003c-4c9c-025e-ff01-255bebc31000 24c3003c-4c9c-025e-ff01-255bebc31000}" error="failed to connect to peer: 24c3003c-4c9c-025e-ff01-255bebc31000" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.290Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to appendEntries to: peer="{Voter server-24c3003c-4c9c-025e-ff01-255bebc31000 24c3003c-4c9c-025e-ff01-255bebc31000}" error="failed to connect to peer: 24c3003c-4c9c-025e-ff01-255bebc31000" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.291Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to heartbeat to: peer=24c3003c-4c9c-025e-ff01-255bebc31000 error="failed to connect to peer: 24c3003c-4c9c-025e-ff01-255bebc31000" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.291Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to heartbeat to: peer=24c3003c-4c9c-025e-ff01-255bebc31000 error="failed to connect to peer: 24c3003c-4c9c-025e-ff01-255bebc31000" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.291Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to heartbeat to: peer=60141061-f7a9-4547-c487-7876db3d60b5 error="failed to connect to peer: 60141061-f7a9-4547-c487-7876db3d60b5" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.291Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to heartbeat to: peer=60141061-f7a9-4547-c487-7876db3d60b5 error="failed to connect to peer: 60141061-f7a9-4547-c487-7876db3d60b5" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.291Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to appendEntries to: peer="{Voter server-60141061-f7a9-4547-c487-7876db3d60b5 60141061-f7a9-4547-c487-7876db3d60b5}" error="failed to connect to peer: 60141061-f7a9-4547-c487-7876db3d60b5" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.291Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to appendEntries to: peer="{Voter server-60141061-f7a9-4547-c487-7876db3d60b5 60141061-f7a9-4547-c487-7876db3d60b5}" error="failed to connect to peer: 60141061-f7a9-4547-c487-7876db3d60b5" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.306Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to heartbeat to: peer=60141061-f7a9-4547-c487-7876db3d60b5 error="failed to connect to peer: 60141061-f7a9-4547-c487-7876db3d60b5" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.306Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to heartbeat to: peer=60141061-f7a9-4547-c487-7876db3d60b5 error="failed to connect to peer: 60141061-f7a9-4547-c487-7876db3d60b5" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.307Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to appendEntries to: peer="{Voter server-60141061-f7a9-4547-c487-7876db3d60b5 60141061-f7a9-4547-c487-7876db3d60b5}" error="failed to connect to peer: 60141061-f7a9-4547-c487-7876db3d60b5" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.307Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to appendEntries to: peer="{Voter server-60141061-f7a9-4547-c487-7876db3d60b5 60141061-f7a9-4547-c487-7876db3d60b5}" error="failed to connect to peer: 60141061-f7a9-4547-c487-7876db3d60b5" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.307Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to heartbeat to: peer=24c3003c-4c9c-025e-ff01-255bebc31000 error="failed to connect to peer: 24c3003c-4c9c-025e-ff01-255bebc31000" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.307Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to heartbeat to: peer=24c3003c-4c9c-025e-ff01-255bebc31000 error="failed to connect to peer: 24c3003c-4c9c-025e-ff01-255bebc31000" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.309Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to appendEntries to: peer="{Voter server-24c3003c-4c9c-025e-ff01-255bebc31000 24c3003c-4c9c-025e-ff01-255bebc31000}" error="failed to connect to peer: 24c3003c-4c9c-025e-ff01-255bebc31000" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.309Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to appendEntries to: peer="{Voter server-24c3003c-4c9c-025e-ff01-255bebc31000 24c3003c-4c9c-025e-ff01-255bebc31000}" error="failed to connect to peer: 24c3003c-4c9c-025e-ff01-255bebc31000" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.323Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to heartbeat to: peer=60141061-f7a9-4547-c487-7876db3d60b5 error="failed to connect to peer: 60141061-f7a9-4547-c487-7876db3d60b5" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.323Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to heartbeat to: peer=60141061-f7a9-4547-c487-7876db3d60b5 error="failed to connect to peer: 60141061-f7a9-4547-c487-7876db3d60b5" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.325Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to heartbeat to: peer=24c3003c-4c9c-025e-ff01-255bebc31000 error="failed to connect to peer: 24c3003c-4c9c-025e-ff01-255bebc31000" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.325Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to heartbeat to: peer=24c3003c-4c9c-025e-ff01-255bebc31000 error="failed to connect to peer: 24c3003c-4c9c-025e-ff01-255bebc31000" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.326Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to appendEntries to: peer="{Voter server-24c3003c-4c9c-025e-ff01-255bebc31000 24c3003c-4c9c-025e-ff01-255bebc31000}" error="failed to connect to peer: 24c3003c-4c9c-025e-ff01-255bebc31000" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.326Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to appendEntries to: peer="{Voter server-24c3003c-4c9c-025e-ff01-255bebc31000 24c3003c-4c9c-025e-ff01-255bebc31000}" error="failed to connect to peer: 24c3003c-4c9c-025e-ff01-255bebc31000" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.327Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to appendEntries to: peer="{Voter server-60141061-f7a9-4547-c487-7876db3d60b5 60141061-f7a9-4547-c487-7876db3d60b5}" error="failed to connect to peer: 60141061-f7a9-4547-c487-7876db3d60b5" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.327Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to appendEntries to: peer="{Voter server-60141061-f7a9-4547-c487-7876db3d60b5 60141061-f7a9-4547-c487-7876db3d60b5}" error="failed to connect to peer: 60141061-f7a9-4547-c487-7876db3d60b5" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.331Z [WARN] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to contact: server-id=server-24c3003c-4c9c-025e-ff01-255bebc31000 time=50.064048ms + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.331Z [WARN] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to contact: server-id=server-24c3003c-4c9c-025e-ff01-255bebc31000 time=50.064048ms +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.342Z [WARN] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to contact: server-id=server-24c3003c-4c9c-025e-ff01-255bebc31000 time=60.213892ms + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.342Z [WARN] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to contact: server-id=server-24c3003c-4c9c-025e-ff01-255bebc31000 time=60.213892ms +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.342Z [WARN] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to contact: server-id=server-60141061-f7a9-4547-c487-7876db3d60b5 time=60.107557ms + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.342Z [WARN] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to contact: server-id=server-60141061-f7a9-4547-c487-7876db3d60b5 time=60.107557ms +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.342Z [WARN] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to contact quorum of nodes, stepping down + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.342Z [WARN] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to contact quorum of nodes, stepping down +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.342Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: entering follower state: follower="Node at d687b9b3-9467-cd46-cea1-c8f0e48c612b [Follower]" leader= + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.342Z [INFO] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: entering follower state: follower="Node at d687b9b3-9467-cd46-cea1-c8f0e48c612b [Follower]" leader= +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.342Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to appendEntries to: peer="{Voter server-24c3003c-4c9c-025e-ff01-255bebc31000 24c3003c-4c9c-025e-ff01-255bebc31000}" error="failed to connect to peer: 24c3003c-4c9c-025e-ff01-255bebc31000" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.342Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to appendEntries to: peer="{Voter server-24c3003c-4c9c-025e-ff01-255bebc31000 24c3003c-4c9c-025e-ff01-255bebc31000}" error="failed to connect to peer: 24c3003c-4c9c-025e-ff01-255bebc31000" +testLoggerAdapter verbose: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.342Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to appendEntries to: peer="{Voter server-60141061-f7a9-4547-c487-7876db3d60b5 60141061-f7a9-4547-c487-7876db3d60b5}" error="failed to connect to peer: 60141061-f7a9-4547-c487-7876db3d60b5" + testing.go:149: server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: 2025-02-10T11:34:13.342Z [ERROR] server-d687b9b3-9467-cd46-cea1-c8f0e48c612b: failed to appendEntries to: peer="{Voter server-60141061-f7a9-4547-c487-7876db3d60b5 60141061-f7a9-4547-c487-7876db3d60b5}" error="failed to connect to peer: 60141061-f7a9-4547-c487-7876db3d60b5" +--- PASS: TestRaft_RemoveFollower_SplitCluster (0.44s) === RUN TestRaft_AddKnownPeer -testLoggerAdapter verbose: server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75: 2024-01-09T05:09:58.898Z [INFO] server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 Address:37d7450a-7d4c-fb10-d0c8-2f2dfef49c75} {Suffrage:Voter ID:server-0d37ee14-011c-72df-7fd6-5553a7037a2b Address:0d37ee14-011c-72df-7fd6-5553a7037a2b} {Suffrage:Voter ID:server-483f3ce8-5fdf-56e2-efdf-e06b182e266f Address:483f3ce8-5fdf-56e2-efdf-e06b182e266f}]" - testing.go:149: server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75: 2024-01-09T05:09:58.898Z [INFO] server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 Address:37d7450a-7d4c-fb10-d0c8-2f2dfef49c75} {Suffrage:Voter ID:server-0d37ee14-011c-72df-7fd6-5553a7037a2b Address:0d37ee14-011c-72df-7fd6-5553a7037a2b} {Suffrage:Voter ID:server-483f3ce8-5fdf-56e2-efdf-e06b182e266f Address:483f3ce8-5fdf-56e2-efdf-e06b182e266f}]" -testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.898Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 Address:37d7450a-7d4c-fb10-d0c8-2f2dfef49c75} {Suffrage:Voter ID:server-0d37ee14-011c-72df-7fd6-5553a7037a2b Address:0d37ee14-011c-72df-7fd6-5553a7037a2b} {Suffrage:Voter ID:server-483f3ce8-5fdf-56e2-efdf-e06b182e266f Address:483f3ce8-5fdf-56e2-efdf-e06b182e266f}]" - testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.898Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 Address:37d7450a-7d4c-fb10-d0c8-2f2dfef49c75} {Suffrage:Voter ID:server-0d37ee14-011c-72df-7fd6-5553a7037a2b Address:0d37ee14-011c-72df-7fd6-5553a7037a2b} {Suffrage:Voter ID:server-483f3ce8-5fdf-56e2-efdf-e06b182e266f Address:483f3ce8-5fdf-56e2-efdf-e06b182e266f}]" -testLoggerAdapter verbose: server-483f3ce8-5fdf-56e2-efdf-e06b182e266f: 2024-01-09T05:09:58.898Z [INFO] server-483f3ce8-5fdf-56e2-efdf-e06b182e266f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 Address:37d7450a-7d4c-fb10-d0c8-2f2dfef49c75} {Suffrage:Voter ID:server-0d37ee14-011c-72df-7fd6-5553a7037a2b Address:0d37ee14-011c-72df-7fd6-5553a7037a2b} {Suffrage:Voter ID:server-483f3ce8-5fdf-56e2-efdf-e06b182e266f Address:483f3ce8-5fdf-56e2-efdf-e06b182e266f}]" - testing.go:149: server-483f3ce8-5fdf-56e2-efdf-e06b182e266f: 2024-01-09T05:09:58.898Z [INFO] server-483f3ce8-5fdf-56e2-efdf-e06b182e266f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 Address:37d7450a-7d4c-fb10-d0c8-2f2dfef49c75} {Suffrage:Voter ID:server-0d37ee14-011c-72df-7fd6-5553a7037a2b Address:0d37ee14-011c-72df-7fd6-5553a7037a2b} {Suffrage:Voter ID:server-483f3ce8-5fdf-56e2-efdf-e06b182e266f Address:483f3ce8-5fdf-56e2-efdf-e06b182e266f}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:58.898Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:58.898Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75: 2024-01-09T05:09:58.899Z [INFO] server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75: entering follower state: follower="Node at 37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 [Follower]" leader= - testing.go:149: server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75: 2024-01-09T05:09:58.899Z [INFO] server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75: entering follower state: follower="Node at 37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 [Follower]" leader= -testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.899Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: entering follower state: follower="Node at 0d37ee14-011c-72df-7fd6-5553a7037a2b [Follower]" leader= - testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.899Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: entering follower state: follower="Node at 0d37ee14-011c-72df-7fd6-5553a7037a2b [Follower]" leader= -testLoggerAdapter verbose: server-483f3ce8-5fdf-56e2-efdf-e06b182e266f: 2024-01-09T05:09:58.899Z [INFO] server-483f3ce8-5fdf-56e2-efdf-e06b182e266f: entering follower state: follower="Node at 483f3ce8-5fdf-56e2-efdf-e06b182e266f [Follower]" leader= - testing.go:149: server-483f3ce8-5fdf-56e2-efdf-e06b182e266f: 2024-01-09T05:09:58.899Z [INFO] server-483f3ce8-5fdf-56e2-efdf-e06b182e266f: entering follower state: follower="Node at 483f3ce8-5fdf-56e2-efdf-e06b182e266f [Follower]" leader= -testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.975Z [WARN] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.975Z [WARN] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: entering candidate state: node="Node at 0d37ee14-011c-72df-7fd6-5553a7037a2b [Candidate]" term=2 - testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: entering candidate state: node="Node at 0d37ee14-011c-72df-7fd6-5553a7037a2b [Candidate]" term=2 -testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: election won: tally=2 - testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: election won: tally=2 -testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: entering leader state: leader="Node at 0d37ee14-011c-72df-7fd6-5553a7037a2b [Leader]" - testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: entering leader state: leader="Node at 0d37ee14-011c-72df-7fd6-5553a7037a2b [Leader]" -testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: added peer, starting replication: peer=server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 - testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: added peer, starting replication: peer=server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 -testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: added peer, starting replication: peer=server-483f3ce8-5fdf-56e2-efdf-e06b182e266f - testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: added peer, starting replication: peer=server-483f3ce8-5fdf-56e2-efdf-e06b182e266f -testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: pipelining replication: peer="{Voter server-483f3ce8-5fdf-56e2-efdf-e06b182e266f 483f3ce8-5fdf-56e2-efdf-e06b182e266f}" - testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: pipelining replication: peer="{Voter server-483f3ce8-5fdf-56e2-efdf-e06b182e266f 483f3ce8-5fdf-56e2-efdf-e06b182e266f}" -testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: pipelining replication: peer="{Voter server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 37d7450a-7d4c-fb10-d0c8-2f2dfef49c75}" - testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: pipelining replication: peer="{Voter server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 37d7450a-7d4c-fb10-d0c8-2f2dfef49c75}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.081Z [INFO] stable state for Leader reached at 2024-01-09 05:09:58.976515905 +0000 UTC m=+10.229567139 (1 nodes), 77.48489ms from start of poll, 78.042221ms from cluster start. Timeout at 2024-01-09 05:09:59.081621458 +0000 UTC m=+10.334672589, 105.10545ms after stability - testing.go:149: cluster: 2024-01-09T05:09:59.081Z [INFO] stable state for Leader reached at 2024-01-09 05:09:58.976515905 +0000 UTC m=+10.229567139 (1 nodes), 77.48489ms from start of poll, 78.042221ms from cluster start. Timeout at 2024-01-09 05:09:59.081621458 +0000 UTC m=+10.334672589, 105.10545ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.081Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:09:59.081Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.187Z [INFO] stable state for Follower reached at 2024-01-09 05:09:59.081769751 +0000 UTC m=+10.334820785 (2 nodes), 2.547µs from start of poll, 183.295867ms from cluster start. Timeout at 2024-01-09 05:09:59.187393592 +0000 UTC m=+10.440444720, 105.623935ms after stability - testing.go:149: cluster: 2024-01-09T05:09:59.187Z [INFO] stable state for Follower reached at 2024-01-09 05:09:59.081769751 +0000 UTC m=+10.334820785 (2 nodes), 2.547µs from start of poll, 183.295867ms from cluster start. Timeout at 2024-01-09 05:09:59.187393592 +0000 UTC m=+10.440444720, 105.623935ms after stability -testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:59.187Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: updating configuration: command=AddStaging server-id=server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 server-addr=37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 servers="[{Suffrage:Voter ID:server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 Address:37d7450a-7d4c-fb10-d0c8-2f2dfef49c75} {Suffrage:Voter ID:server-0d37ee14-011c-72df-7fd6-5553a7037a2b Address:0d37ee14-011c-72df-7fd6-5553a7037a2b} {Suffrage:Voter ID:server-483f3ce8-5fdf-56e2-efdf-e06b182e266f Address:483f3ce8-5fdf-56e2-efdf-e06b182e266f}]" - testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:59.187Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: updating configuration: command=AddStaging server-id=server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 server-addr=37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 servers="[{Suffrage:Voter ID:server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 Address:37d7450a-7d4c-fb10-d0c8-2f2dfef49c75} {Suffrage:Voter ID:server-0d37ee14-011c-72df-7fd6-5553a7037a2b Address:0d37ee14-011c-72df-7fd6-5553a7037a2b} {Suffrage:Voter ID:server-483f3ce8-5fdf-56e2-efdf-e06b182e266f Address:483f3ce8-5fdf-56e2-efdf-e06b182e266f}]" -testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:59.190Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: aborting pipeline replication: peer="{Voter server-483f3ce8-5fdf-56e2-efdf-e06b182e266f 483f3ce8-5fdf-56e2-efdf-e06b182e266f}" - testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:59.190Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: aborting pipeline replication: peer="{Voter server-483f3ce8-5fdf-56e2-efdf-e06b182e266f 483f3ce8-5fdf-56e2-efdf-e06b182e266f}" -testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:59.190Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: aborting pipeline replication: peer="{Voter server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 37d7450a-7d4c-fb10-d0c8-2f2dfef49c75}" - testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:59.190Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: aborting pipeline replication: peer="{Voter server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 37d7450a-7d4c-fb10-d0c8-2f2dfef49c75}" ---- PASS: TestRaft_AddKnownPeer (0.30s) +testLoggerAdapter verbose: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.343Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acdc1783-00f7-1b47-6362-2ca2fa448643 Address:acdc1783-00f7-1b47-6362-2ca2fa448643} {Suffrage:Voter ID:server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 Address:5b6dfd96-4124-b06c-cfaa-25e1a189b1e3} {Suffrage:Voter ID:server-92905cb5-0269-dc14-2d78-8df63d45a327 Address:92905cb5-0269-dc14-2d78-8df63d45a327}]" + testing.go:149: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.343Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acdc1783-00f7-1b47-6362-2ca2fa448643 Address:acdc1783-00f7-1b47-6362-2ca2fa448643} {Suffrage:Voter ID:server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 Address:5b6dfd96-4124-b06c-cfaa-25e1a189b1e3} {Suffrage:Voter ID:server-92905cb5-0269-dc14-2d78-8df63d45a327 Address:92905cb5-0269-dc14-2d78-8df63d45a327}]" +testLoggerAdapter verbose: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.343Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: entering follower state: follower="Node at acdc1783-00f7-1b47-6362-2ca2fa448643 [Follower]" leader= + testing.go:149: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.343Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: entering follower state: follower="Node at acdc1783-00f7-1b47-6362-2ca2fa448643 [Follower]" leader= +testLoggerAdapter verbose: server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3: 2025-02-10T11:34:13.343Z [INFO] server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acdc1783-00f7-1b47-6362-2ca2fa448643 Address:acdc1783-00f7-1b47-6362-2ca2fa448643} {Suffrage:Voter ID:server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 Address:5b6dfd96-4124-b06c-cfaa-25e1a189b1e3} {Suffrage:Voter ID:server-92905cb5-0269-dc14-2d78-8df63d45a327 Address:92905cb5-0269-dc14-2d78-8df63d45a327}]" + testing.go:149: server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3: 2025-02-10T11:34:13.343Z [INFO] server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acdc1783-00f7-1b47-6362-2ca2fa448643 Address:acdc1783-00f7-1b47-6362-2ca2fa448643} {Suffrage:Voter ID:server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 Address:5b6dfd96-4124-b06c-cfaa-25e1a189b1e3} {Suffrage:Voter ID:server-92905cb5-0269-dc14-2d78-8df63d45a327 Address:92905cb5-0269-dc14-2d78-8df63d45a327}]" +testLoggerAdapter verbose: server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3: 2025-02-10T11:34:13.343Z [INFO] server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3: entering follower state: follower="Node at 5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 [Follower]" leader= + testing.go:149: server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3: 2025-02-10T11:34:13.343Z [INFO] server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3: entering follower state: follower="Node at 5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 [Follower]" leader= +testLoggerAdapter verbose: server-92905cb5-0269-dc14-2d78-8df63d45a327: 2025-02-10T11:34:13.343Z [INFO] server-92905cb5-0269-dc14-2d78-8df63d45a327: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acdc1783-00f7-1b47-6362-2ca2fa448643 Address:acdc1783-00f7-1b47-6362-2ca2fa448643} {Suffrage:Voter ID:server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 Address:5b6dfd96-4124-b06c-cfaa-25e1a189b1e3} {Suffrage:Voter ID:server-92905cb5-0269-dc14-2d78-8df63d45a327 Address:92905cb5-0269-dc14-2d78-8df63d45a327}]" + testing.go:149: server-92905cb5-0269-dc14-2d78-8df63d45a327: 2025-02-10T11:34:13.343Z [INFO] server-92905cb5-0269-dc14-2d78-8df63d45a327: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acdc1783-00f7-1b47-6362-2ca2fa448643 Address:acdc1783-00f7-1b47-6362-2ca2fa448643} {Suffrage:Voter ID:server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 Address:5b6dfd96-4124-b06c-cfaa-25e1a189b1e3} {Suffrage:Voter ID:server-92905cb5-0269-dc14-2d78-8df63d45a327 Address:92905cb5-0269-dc14-2d78-8df63d45a327}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:13.343Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:13.343Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-92905cb5-0269-dc14-2d78-8df63d45a327: 2025-02-10T11:34:13.343Z [INFO] server-92905cb5-0269-dc14-2d78-8df63d45a327: entering follower state: follower="Node at 92905cb5-0269-dc14-2d78-8df63d45a327 [Follower]" leader= + testing.go:149: server-92905cb5-0269-dc14-2d78-8df63d45a327: 2025-02-10T11:34:13.343Z [INFO] server-92905cb5-0269-dc14-2d78-8df63d45a327: entering follower state: follower="Node at 92905cb5-0269-dc14-2d78-8df63d45a327 [Follower]" leader= +testLoggerAdapter verbose: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.403Z [WARN] server-acdc1783-00f7-1b47-6362-2ca2fa448643: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.403Z [WARN] server-acdc1783-00f7-1b47-6362-2ca2fa448643: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.403Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: entering candidate state: node="Node at acdc1783-00f7-1b47-6362-2ca2fa448643 [Candidate]" term=2 + testing.go:149: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.403Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: entering candidate state: node="Node at acdc1783-00f7-1b47-6362-2ca2fa448643 [Candidate]" term=2 +testLoggerAdapter verbose: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.403Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: election won: tally=2 + testing.go:149: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.403Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: election won: tally=2 +testLoggerAdapter verbose: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.403Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: entering leader state: leader="Node at acdc1783-00f7-1b47-6362-2ca2fa448643 [Leader]" + testing.go:149: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.403Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: entering leader state: leader="Node at acdc1783-00f7-1b47-6362-2ca2fa448643 [Leader]" +testLoggerAdapter verbose: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.403Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: added peer, starting replication: peer=server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 + testing.go:149: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.403Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: added peer, starting replication: peer=server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 +testLoggerAdapter verbose: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.403Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: added peer, starting replication: peer=server-92905cb5-0269-dc14-2d78-8df63d45a327 + testing.go:149: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.403Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: added peer, starting replication: peer=server-92905cb5-0269-dc14-2d78-8df63d45a327 +testLoggerAdapter verbose: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.403Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: pipelining replication: peer="{Voter server-92905cb5-0269-dc14-2d78-8df63d45a327 92905cb5-0269-dc14-2d78-8df63d45a327}" + testing.go:149: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.403Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: pipelining replication: peer="{Voter server-92905cb5-0269-dc14-2d78-8df63d45a327 92905cb5-0269-dc14-2d78-8df63d45a327}" +testLoggerAdapter verbose: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.403Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: pipelining replication: peer="{Voter server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 5b6dfd96-4124-b06c-cfaa-25e1a189b1e3}" + testing.go:149: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.403Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: pipelining replication: peer="{Voter server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 5b6dfd96-4124-b06c-cfaa-25e1a189b1e3}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:13.508Z [INFO] stable state for Leader reached at 2025-02-10 11:34:13.403541963 +0000 UTC m=+9.772446264 (1 nodes), 59.655121ms from start of poll, 59.997976ms from cluster start. Timeout at 2025-02-10 11:34:13.508635198 +0000 UTC m=+9.877539555, 105.093291ms after stability + testing.go:149: cluster: 2025-02-10T11:34:13.508Z [INFO] stable state for Leader reached at 2025-02-10 11:34:13.403541963 +0000 UTC m=+9.772446264 (1 nodes), 59.655121ms from start of poll, 59.997976ms from cluster start. Timeout at 2025-02-10 11:34:13.508635198 +0000 UTC m=+9.877539555, 105.093291ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:13.508Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:13.508Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:13.613Z [INFO] stable state for Follower reached at 2025-02-10 11:34:13.508829808 +0000 UTC m=+9.877734171 (2 nodes), 1.484µs from start of poll, 165.285883ms from cluster start. Timeout at 2025-02-10 11:34:13.613903936 +0000 UTC m=+9.982808233, 105.074062ms after stability + testing.go:149: cluster: 2025-02-10T11:34:13.613Z [INFO] stable state for Follower reached at 2025-02-10 11:34:13.508829808 +0000 UTC m=+9.877734171 (2 nodes), 1.484µs from start of poll, 165.285883ms from cluster start. Timeout at 2025-02-10 11:34:13.613903936 +0000 UTC m=+9.982808233, 105.074062ms after stability +testLoggerAdapter verbose: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.613Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: updating configuration: command=AddStaging server-id=server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 server-addr=5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 servers="[{Suffrage:Voter ID:server-acdc1783-00f7-1b47-6362-2ca2fa448643 Address:acdc1783-00f7-1b47-6362-2ca2fa448643} {Suffrage:Voter ID:server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 Address:5b6dfd96-4124-b06c-cfaa-25e1a189b1e3} {Suffrage:Voter ID:server-92905cb5-0269-dc14-2d78-8df63d45a327 Address:92905cb5-0269-dc14-2d78-8df63d45a327}]" + testing.go:149: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.613Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: updating configuration: command=AddStaging server-id=server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 server-addr=5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 servers="[{Suffrage:Voter ID:server-acdc1783-00f7-1b47-6362-2ca2fa448643 Address:acdc1783-00f7-1b47-6362-2ca2fa448643} {Suffrage:Voter ID:server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 Address:5b6dfd96-4124-b06c-cfaa-25e1a189b1e3} {Suffrage:Voter ID:server-92905cb5-0269-dc14-2d78-8df63d45a327 Address:92905cb5-0269-dc14-2d78-8df63d45a327}]" +testLoggerAdapter verbose: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.614Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: aborting pipeline replication: peer="{Voter server-92905cb5-0269-dc14-2d78-8df63d45a327 92905cb5-0269-dc14-2d78-8df63d45a327}" + testing.go:149: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.614Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: aborting pipeline replication: peer="{Voter server-92905cb5-0269-dc14-2d78-8df63d45a327 92905cb5-0269-dc14-2d78-8df63d45a327}" +testLoggerAdapter verbose: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.614Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: aborting pipeline replication: peer="{Voter server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 5b6dfd96-4124-b06c-cfaa-25e1a189b1e3}" + testing.go:149: server-acdc1783-00f7-1b47-6362-2ca2fa448643: 2025-02-10T11:34:13.614Z [INFO] server-acdc1783-00f7-1b47-6362-2ca2fa448643: aborting pipeline replication: peer="{Voter server-5b6dfd96-4124-b06c-cfaa-25e1a189b1e3 5b6dfd96-4124-b06c-cfaa-25e1a189b1e3}" +--- PASS: TestRaft_AddKnownPeer (0.27s) === RUN TestRaft_RemoveUnknownPeer -testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.199Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b62f0521-7d78-49be-e421-c343bee52c3c Address:b62f0521-7d78-49be-e421-c343bee52c3c} {Suffrage:Voter ID:server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 Address:650c6d6f-3e9c-4281-fa05-7671ff6196c6} {Suffrage:Voter ID:server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f Address:8fd7c004-241a-b68b-3e7b-abd2b1e8006f}]" - testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.199Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b62f0521-7d78-49be-e421-c343bee52c3c Address:b62f0521-7d78-49be-e421-c343bee52c3c} {Suffrage:Voter ID:server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 Address:650c6d6f-3e9c-4281-fa05-7671ff6196c6} {Suffrage:Voter ID:server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f Address:8fd7c004-241a-b68b-3e7b-abd2b1e8006f}]" -testLoggerAdapter verbose: server-650c6d6f-3e9c-4281-fa05-7671ff6196c6: 2024-01-09T05:09:59.199Z [INFO] server-650c6d6f-3e9c-4281-fa05-7671ff6196c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b62f0521-7d78-49be-e421-c343bee52c3c Address:b62f0521-7d78-49be-e421-c343bee52c3c} {Suffrage:Voter ID:server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 Address:650c6d6f-3e9c-4281-fa05-7671ff6196c6} {Suffrage:Voter ID:server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f Address:8fd7c004-241a-b68b-3e7b-abd2b1e8006f}]" - testing.go:149: server-650c6d6f-3e9c-4281-fa05-7671ff6196c6: 2024-01-09T05:09:59.199Z [INFO] server-650c6d6f-3e9c-4281-fa05-7671ff6196c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b62f0521-7d78-49be-e421-c343bee52c3c Address:b62f0521-7d78-49be-e421-c343bee52c3c} {Suffrage:Voter ID:server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 Address:650c6d6f-3e9c-4281-fa05-7671ff6196c6} {Suffrage:Voter ID:server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f Address:8fd7c004-241a-b68b-3e7b-abd2b1e8006f}]" -testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.200Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: entering follower state: follower="Node at b62f0521-7d78-49be-e421-c343bee52c3c [Follower]" leader= - testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.200Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: entering follower state: follower="Node at b62f0521-7d78-49be-e421-c343bee52c3c [Follower]" leader= -testLoggerAdapter verbose: server-650c6d6f-3e9c-4281-fa05-7671ff6196c6: 2024-01-09T05:09:59.200Z [INFO] server-650c6d6f-3e9c-4281-fa05-7671ff6196c6: entering follower state: follower="Node at 650c6d6f-3e9c-4281-fa05-7671ff6196c6 [Follower]" leader= - testing.go:149: server-650c6d6f-3e9c-4281-fa05-7671ff6196c6: 2024-01-09T05:09:59.200Z [INFO] server-650c6d6f-3e9c-4281-fa05-7671ff6196c6: entering follower state: follower="Node at 650c6d6f-3e9c-4281-fa05-7671ff6196c6 [Follower]" leader= -testLoggerAdapter verbose: server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f: 2024-01-09T05:09:59.200Z [INFO] server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b62f0521-7d78-49be-e421-c343bee52c3c Address:b62f0521-7d78-49be-e421-c343bee52c3c} {Suffrage:Voter ID:server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 Address:650c6d6f-3e9c-4281-fa05-7671ff6196c6} {Suffrage:Voter ID:server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f Address:8fd7c004-241a-b68b-3e7b-abd2b1e8006f}]" - testing.go:149: server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f: 2024-01-09T05:09:59.200Z [INFO] server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b62f0521-7d78-49be-e421-c343bee52c3c Address:b62f0521-7d78-49be-e421-c343bee52c3c} {Suffrage:Voter ID:server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 Address:650c6d6f-3e9c-4281-fa05-7671ff6196c6} {Suffrage:Voter ID:server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f Address:8fd7c004-241a-b68b-3e7b-abd2b1e8006f}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.200Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:59.200Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f: 2024-01-09T05:09:59.200Z [INFO] server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f: entering follower state: follower="Node at 8fd7c004-241a-b68b-3e7b-abd2b1e8006f [Follower]" leader= - testing.go:149: server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f: 2024-01-09T05:09:59.200Z [INFO] server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f: entering follower state: follower="Node at 8fd7c004-241a-b68b-3e7b-abd2b1e8006f [Follower]" leader= -testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.251Z [WARN] server-b62f0521-7d78-49be-e421-c343bee52c3c: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.251Z [WARN] server-b62f0521-7d78-49be-e421-c343bee52c3c: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.251Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: entering candidate state: node="Node at b62f0521-7d78-49be-e421-c343bee52c3c [Candidate]" term=2 - testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.251Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: entering candidate state: node="Node at b62f0521-7d78-49be-e421-c343bee52c3c [Candidate]" term=2 -testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.253Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: election won: tally=2 - testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.253Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: election won: tally=2 -testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.253Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: entering leader state: leader="Node at b62f0521-7d78-49be-e421-c343bee52c3c [Leader]" - testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.253Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: entering leader state: leader="Node at b62f0521-7d78-49be-e421-c343bee52c3c [Leader]" -testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.253Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: added peer, starting replication: peer=server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 - testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.253Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: added peer, starting replication: peer=server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 -testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.254Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: added peer, starting replication: peer=server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f - testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.254Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: added peer, starting replication: peer=server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f -testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.254Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: pipelining replication: peer="{Voter server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f 8fd7c004-241a-b68b-3e7b-abd2b1e8006f}" - testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.254Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: pipelining replication: peer="{Voter server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f 8fd7c004-241a-b68b-3e7b-abd2b1e8006f}" -testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.255Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: pipelining replication: peer="{Voter server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 650c6d6f-3e9c-4281-fa05-7671ff6196c6}" - testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.255Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: pipelining replication: peer="{Voter server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 650c6d6f-3e9c-4281-fa05-7671ff6196c6}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.359Z [INFO] stable state for Leader reached at 2024-01-09 05:09:59.254617444 +0000 UTC m=+10.507668480 (1 nodes), 53.918384ms from start of poll, 55.593048ms from cluster start. Timeout at 2024-01-09 05:09:59.359698059 +0000 UTC m=+10.612749120, 105.08064ms after stability - testing.go:149: cluster: 2024-01-09T05:09:59.359Z [INFO] stable state for Leader reached at 2024-01-09 05:09:59.254617444 +0000 UTC m=+10.507668480 (1 nodes), 53.918384ms from start of poll, 55.593048ms from cluster start. Timeout at 2024-01-09 05:09:59.359698059 +0000 UTC m=+10.612749120, 105.08064ms after stability -testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.359Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: updating configuration: command=RemoveServer server-id=a98d063c-8213-edb6-fa4a-32463f278c19 server-addr= servers="[{Suffrage:Voter ID:server-b62f0521-7d78-49be-e421-c343bee52c3c Address:b62f0521-7d78-49be-e421-c343bee52c3c} {Suffrage:Voter ID:server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 Address:650c6d6f-3e9c-4281-fa05-7671ff6196c6} {Suffrage:Voter ID:server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f Address:8fd7c004-241a-b68b-3e7b-abd2b1e8006f}]" - testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.359Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: updating configuration: command=RemoveServer server-id=a98d063c-8213-edb6-fa4a-32463f278c19 server-addr= servers="[{Suffrage:Voter ID:server-b62f0521-7d78-49be-e421-c343bee52c3c Address:b62f0521-7d78-49be-e421-c343bee52c3c} {Suffrage:Voter ID:server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 Address:650c6d6f-3e9c-4281-fa05-7671ff6196c6} {Suffrage:Voter ID:server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f Address:8fd7c004-241a-b68b-3e7b-abd2b1e8006f}]" -testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.360Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: aborting pipeline replication: peer="{Voter server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f 8fd7c004-241a-b68b-3e7b-abd2b1e8006f}" - testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.360Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: aborting pipeline replication: peer="{Voter server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f 8fd7c004-241a-b68b-3e7b-abd2b1e8006f}" -testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.360Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: aborting pipeline replication: peer="{Voter server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 650c6d6f-3e9c-4281-fa05-7671ff6196c6}" - testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.360Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: aborting pipeline replication: peer="{Voter server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 650c6d6f-3e9c-4281-fa05-7671ff6196c6}" ---- PASS: TestRaft_RemoveUnknownPeer (0.16s) +testLoggerAdapter verbose: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.615Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48ffc5e2-1ffc-220c-f30d-2687d1530cee Address:48ffc5e2-1ffc-220c-f30d-2687d1530cee} {Suffrage:Voter ID:server-baee4eee-a15f-a1a6-ad90-e59dcde42748 Address:baee4eee-a15f-a1a6-ad90-e59dcde42748} {Suffrage:Voter ID:server-210968fe-7ad3-bad6-5d58-7b08e51e49eb Address:210968fe-7ad3-bad6-5d58-7b08e51e49eb}]" + testing.go:149: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.615Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48ffc5e2-1ffc-220c-f30d-2687d1530cee Address:48ffc5e2-1ffc-220c-f30d-2687d1530cee} {Suffrage:Voter ID:server-baee4eee-a15f-a1a6-ad90-e59dcde42748 Address:baee4eee-a15f-a1a6-ad90-e59dcde42748} {Suffrage:Voter ID:server-210968fe-7ad3-bad6-5d58-7b08e51e49eb Address:210968fe-7ad3-bad6-5d58-7b08e51e49eb}]" +testLoggerAdapter verbose: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.615Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: entering follower state: follower="Node at 48ffc5e2-1ffc-220c-f30d-2687d1530cee [Follower]" leader= + testing.go:149: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.615Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: entering follower state: follower="Node at 48ffc5e2-1ffc-220c-f30d-2687d1530cee [Follower]" leader= +testLoggerAdapter verbose: server-baee4eee-a15f-a1a6-ad90-e59dcde42748: 2025-02-10T11:34:13.615Z [INFO] server-baee4eee-a15f-a1a6-ad90-e59dcde42748: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48ffc5e2-1ffc-220c-f30d-2687d1530cee Address:48ffc5e2-1ffc-220c-f30d-2687d1530cee} {Suffrage:Voter ID:server-baee4eee-a15f-a1a6-ad90-e59dcde42748 Address:baee4eee-a15f-a1a6-ad90-e59dcde42748} {Suffrage:Voter ID:server-210968fe-7ad3-bad6-5d58-7b08e51e49eb Address:210968fe-7ad3-bad6-5d58-7b08e51e49eb}]" + testing.go:149: server-baee4eee-a15f-a1a6-ad90-e59dcde42748: 2025-02-10T11:34:13.615Z [INFO] server-baee4eee-a15f-a1a6-ad90-e59dcde42748: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48ffc5e2-1ffc-220c-f30d-2687d1530cee Address:48ffc5e2-1ffc-220c-f30d-2687d1530cee} {Suffrage:Voter ID:server-baee4eee-a15f-a1a6-ad90-e59dcde42748 Address:baee4eee-a15f-a1a6-ad90-e59dcde42748} {Suffrage:Voter ID:server-210968fe-7ad3-bad6-5d58-7b08e51e49eb Address:210968fe-7ad3-bad6-5d58-7b08e51e49eb}]" +testLoggerAdapter verbose: server-baee4eee-a15f-a1a6-ad90-e59dcde42748: 2025-02-10T11:34:13.615Z [INFO] server-baee4eee-a15f-a1a6-ad90-e59dcde42748: entering follower state: follower="Node at baee4eee-a15f-a1a6-ad90-e59dcde42748 [Follower]" leader= + testing.go:149: server-baee4eee-a15f-a1a6-ad90-e59dcde42748: 2025-02-10T11:34:13.615Z [INFO] server-baee4eee-a15f-a1a6-ad90-e59dcde42748: entering follower state: follower="Node at baee4eee-a15f-a1a6-ad90-e59dcde42748 [Follower]" leader= +testLoggerAdapter verbose: server-210968fe-7ad3-bad6-5d58-7b08e51e49eb: 2025-02-10T11:34:13.615Z [INFO] server-210968fe-7ad3-bad6-5d58-7b08e51e49eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48ffc5e2-1ffc-220c-f30d-2687d1530cee Address:48ffc5e2-1ffc-220c-f30d-2687d1530cee} {Suffrage:Voter ID:server-baee4eee-a15f-a1a6-ad90-e59dcde42748 Address:baee4eee-a15f-a1a6-ad90-e59dcde42748} {Suffrage:Voter ID:server-210968fe-7ad3-bad6-5d58-7b08e51e49eb Address:210968fe-7ad3-bad6-5d58-7b08e51e49eb}]" + testing.go:149: server-210968fe-7ad3-bad6-5d58-7b08e51e49eb: 2025-02-10T11:34:13.615Z [INFO] server-210968fe-7ad3-bad6-5d58-7b08e51e49eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48ffc5e2-1ffc-220c-f30d-2687d1530cee Address:48ffc5e2-1ffc-220c-f30d-2687d1530cee} {Suffrage:Voter ID:server-baee4eee-a15f-a1a6-ad90-e59dcde42748 Address:baee4eee-a15f-a1a6-ad90-e59dcde42748} {Suffrage:Voter ID:server-210968fe-7ad3-bad6-5d58-7b08e51e49eb Address:210968fe-7ad3-bad6-5d58-7b08e51e49eb}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:13.615Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:13.615Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-210968fe-7ad3-bad6-5d58-7b08e51e49eb: 2025-02-10T11:34:13.615Z [INFO] server-210968fe-7ad3-bad6-5d58-7b08e51e49eb: entering follower state: follower="Node at 210968fe-7ad3-bad6-5d58-7b08e51e49eb [Follower]" leader= + testing.go:149: server-210968fe-7ad3-bad6-5d58-7b08e51e49eb: 2025-02-10T11:34:13.615Z [INFO] server-210968fe-7ad3-bad6-5d58-7b08e51e49eb: entering follower state: follower="Node at 210968fe-7ad3-bad6-5d58-7b08e51e49eb [Follower]" leader= +testLoggerAdapter verbose: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.680Z [WARN] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.680Z [WARN] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.680Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: entering candidate state: node="Node at 48ffc5e2-1ffc-220c-f30d-2687d1530cee [Candidate]" term=2 + testing.go:149: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.680Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: entering candidate state: node="Node at 48ffc5e2-1ffc-220c-f30d-2687d1530cee [Candidate]" term=2 +testLoggerAdapter verbose: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.680Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: election won: tally=2 + testing.go:149: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.680Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: election won: tally=2 +testLoggerAdapter verbose: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.680Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: entering leader state: leader="Node at 48ffc5e2-1ffc-220c-f30d-2687d1530cee [Leader]" + testing.go:149: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.680Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: entering leader state: leader="Node at 48ffc5e2-1ffc-220c-f30d-2687d1530cee [Leader]" +testLoggerAdapter verbose: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.680Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: added peer, starting replication: peer=server-baee4eee-a15f-a1a6-ad90-e59dcde42748 + testing.go:149: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.680Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: added peer, starting replication: peer=server-baee4eee-a15f-a1a6-ad90-e59dcde42748 +testLoggerAdapter verbose: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.680Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: added peer, starting replication: peer=server-210968fe-7ad3-bad6-5d58-7b08e51e49eb + testing.go:149: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.680Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: added peer, starting replication: peer=server-210968fe-7ad3-bad6-5d58-7b08e51e49eb +testLoggerAdapter verbose: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.680Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: pipelining replication: peer="{Voter server-baee4eee-a15f-a1a6-ad90-e59dcde42748 baee4eee-a15f-a1a6-ad90-e59dcde42748}" + testing.go:149: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.680Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: pipelining replication: peer="{Voter server-baee4eee-a15f-a1a6-ad90-e59dcde42748 baee4eee-a15f-a1a6-ad90-e59dcde42748}" +testLoggerAdapter verbose: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.680Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: pipelining replication: peer="{Voter server-210968fe-7ad3-bad6-5d58-7b08e51e49eb 210968fe-7ad3-bad6-5d58-7b08e51e49eb}" + testing.go:149: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.680Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: pipelining replication: peer="{Voter server-210968fe-7ad3-bad6-5d58-7b08e51e49eb 210968fe-7ad3-bad6-5d58-7b08e51e49eb}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:13.785Z [INFO] stable state for Leader reached at 2025-02-10 11:34:13.680623236 +0000 UTC m=+10.049527602 (1 nodes), 65.154056ms from start of poll, 65.529091ms from cluster start. Timeout at 2025-02-10 11:34:13.785697837 +0000 UTC m=+10.154602134, 105.074532ms after stability + testing.go:149: cluster: 2025-02-10T11:34:13.785Z [INFO] stable state for Leader reached at 2025-02-10 11:34:13.680623236 +0000 UTC m=+10.049527602 (1 nodes), 65.154056ms from start of poll, 65.529091ms from cluster start. Timeout at 2025-02-10 11:34:13.785697837 +0000 UTC m=+10.154602134, 105.074532ms after stability +testLoggerAdapter verbose: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.785Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: updating configuration: command=RemoveServer server-id=abd03c33-e653-c776-70e3-9a7c59d3c0f1 server-addr= servers="[{Suffrage:Voter ID:server-48ffc5e2-1ffc-220c-f30d-2687d1530cee Address:48ffc5e2-1ffc-220c-f30d-2687d1530cee} {Suffrage:Voter ID:server-baee4eee-a15f-a1a6-ad90-e59dcde42748 Address:baee4eee-a15f-a1a6-ad90-e59dcde42748} {Suffrage:Voter ID:server-210968fe-7ad3-bad6-5d58-7b08e51e49eb Address:210968fe-7ad3-bad6-5d58-7b08e51e49eb}]" + testing.go:149: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.785Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: updating configuration: command=RemoveServer server-id=abd03c33-e653-c776-70e3-9a7c59d3c0f1 server-addr= servers="[{Suffrage:Voter ID:server-48ffc5e2-1ffc-220c-f30d-2687d1530cee Address:48ffc5e2-1ffc-220c-f30d-2687d1530cee} {Suffrage:Voter ID:server-baee4eee-a15f-a1a6-ad90-e59dcde42748 Address:baee4eee-a15f-a1a6-ad90-e59dcde42748} {Suffrage:Voter ID:server-210968fe-7ad3-bad6-5d58-7b08e51e49eb Address:210968fe-7ad3-bad6-5d58-7b08e51e49eb}]" +testLoggerAdapter verbose: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.785Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: aborting pipeline replication: peer="{Voter server-210968fe-7ad3-bad6-5d58-7b08e51e49eb 210968fe-7ad3-bad6-5d58-7b08e51e49eb}" + testing.go:149: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.785Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: aborting pipeline replication: peer="{Voter server-210968fe-7ad3-bad6-5d58-7b08e51e49eb 210968fe-7ad3-bad6-5d58-7b08e51e49eb}" +testLoggerAdapter verbose: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.785Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: aborting pipeline replication: peer="{Voter server-baee4eee-a15f-a1a6-ad90-e59dcde42748 baee4eee-a15f-a1a6-ad90-e59dcde42748}" + testing.go:149: server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: 2025-02-10T11:34:13.785Z [INFO] server-48ffc5e2-1ffc-220c-f30d-2687d1530cee: aborting pipeline replication: peer="{Voter server-baee4eee-a15f-a1a6-ad90-e59dcde42748 baee4eee-a15f-a1a6-ad90-e59dcde42748}" +--- PASS: TestRaft_RemoveUnknownPeer (0.17s) === RUN TestRaft_SnapshotRestore -testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.361Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31783a95-4ae1-8861-c5c6-d38a313ca10d Address:31783a95-4ae1-8861-c5c6-d38a313ca10d}]" - testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.361Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31783a95-4ae1-8861-c5c6-d38a313ca10d Address:31783a95-4ae1-8861-c5c6-d38a313ca10d}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.361Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:59.361Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.362Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: entering follower state: follower="Node at 31783a95-4ae1-8861-c5c6-d38a313ca10d [Follower]" leader= - testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.362Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: entering follower state: follower="Node at 31783a95-4ae1-8861-c5c6-d38a313ca10d [Follower]" leader= -testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.429Z [WARN] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.429Z [WARN] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.430Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: entering candidate state: node="Node at 31783a95-4ae1-8861-c5c6-d38a313ca10d [Candidate]" term=2 - testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.430Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: entering candidate state: node="Node at 31783a95-4ae1-8861-c5c6-d38a313ca10d [Candidate]" term=2 -testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.430Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: election won: tally=1 - testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.430Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: election won: tally=1 -testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.430Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: entering leader state: leader="Node at 31783a95-4ae1-8861-c5c6-d38a313ca10d [Leader]" - testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.430Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: entering leader state: leader="Node at 31783a95-4ae1-8861-c5c6-d38a313ca10d [Leader]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.535Z [INFO] stable state for Leader reached at 2024-01-09 05:09:59.430248822 +0000 UTC m=+10.683299941 (1 nodes), 68.225783ms from start of poll, 68.436468ms from cluster start. Timeout at 2024-01-09 05:09:59.535368354 +0000 UTC m=+10.788419555, 105.119614ms after stability - testing.go:149: cluster: 2024-01-09T05:09:59.535Z [INFO] stable state for Leader reached at 2024-01-09 05:09:59.430248822 +0000 UTC m=+10.683299941 (1 nodes), 68.225783ms from start of poll, 68.436468ms from cluster start. Timeout at 2024-01-09 05:09:59.535368354 +0000 UTC m=+10.788419555, 105.119614ms after stability -testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.538Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: starting snapshot up to: index=102 - testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.538Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: starting snapshot up to: index=102 - testing.go:153: 2024-01-09T05:09:59.538Z [INFO] creating new snapshot: path=/tmp/raft833869084/snapshots/2-102-1704776999538.tmp -testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.549Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: compacting logs: from=1 to=92 - testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.549Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.549Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: snapshot complete up to: index=102 - testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.549Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.549Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: restored from snapshot: id=2-102-1704776999538 - testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.549Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: restored from snapshot: id=2-102-1704776999538 -testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.549Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31783a95-4ae1-8861-c5c6-d38a313ca10d Address:31783a95-4ae1-8861-c5c6-d38a313ca10d}]" - testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.549Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31783a95-4ae1-8861-c5c6-d38a313ca10d Address:31783a95-4ae1-8861-c5c6-d38a313ca10d}]" ---- PASS: TestRaft_SnapshotRestore (0.19s) +testLoggerAdapter verbose: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.786Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-852cefc1-8fe8-1a27-ad1e-037457a37f9b Address:852cefc1-8fe8-1a27-ad1e-037457a37f9b}]" + testing.go:149: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.786Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-852cefc1-8fe8-1a27-ad1e-037457a37f9b Address:852cefc1-8fe8-1a27-ad1e-037457a37f9b}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:13.786Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:13.786Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.786Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: entering follower state: follower="Node at 852cefc1-8fe8-1a27-ad1e-037457a37f9b [Follower]" leader= + testing.go:149: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.786Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: entering follower state: follower="Node at 852cefc1-8fe8-1a27-ad1e-037457a37f9b [Follower]" leader= +testLoggerAdapter verbose: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.853Z [WARN] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.853Z [WARN] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.853Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: entering candidate state: node="Node at 852cefc1-8fe8-1a27-ad1e-037457a37f9b [Candidate]" term=2 + testing.go:149: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.853Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: entering candidate state: node="Node at 852cefc1-8fe8-1a27-ad1e-037457a37f9b [Candidate]" term=2 +testLoggerAdapter verbose: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.853Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: election won: tally=1 + testing.go:149: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.853Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: election won: tally=1 +testLoggerAdapter verbose: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.853Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: entering leader state: leader="Node at 852cefc1-8fe8-1a27-ad1e-037457a37f9b [Leader]" + testing.go:149: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.853Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: entering leader state: leader="Node at 852cefc1-8fe8-1a27-ad1e-037457a37f9b [Leader]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:13.958Z [INFO] stable state for Leader reached at 2025-02-10 11:34:13.853767807 +0000 UTC m=+10.222672108 (1 nodes), 66.965135ms from start of poll, 67.11233ms from cluster start. Timeout at 2025-02-10 11:34:13.958866691 +0000 UTC m=+10.327771077, 105.098969ms after stability + testing.go:149: cluster: 2025-02-10T11:34:13.958Z [INFO] stable state for Leader reached at 2025-02-10 11:34:13.853767807 +0000 UTC m=+10.222672108 (1 nodes), 66.965135ms from start of poll, 67.11233ms from cluster start. Timeout at 2025-02-10 11:34:13.958866691 +0000 UTC m=+10.327771077, 105.098969ms after stability +testLoggerAdapter verbose: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.959Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: starting snapshot up to: index=102 + testing.go:149: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.959Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: starting snapshot up to: index=102 + testing.go:153: 2025-02-10T11:34:13.959Z [INFO] creating new snapshot: path=/tmp/raft381616389/snapshots/2-102-1739187253959.tmp +testLoggerAdapter verbose: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.964Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: compacting logs: from=1 to=92 + testing.go:149: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.964Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.964Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: snapshot complete up to: index=102 + testing.go:149: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.964Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.965Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: restored from snapshot: id=2-102-1739187253959 + testing.go:149: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.965Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: restored from snapshot: id=2-102-1739187253959 +testLoggerAdapter verbose: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.965Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-852cefc1-8fe8-1a27-ad1e-037457a37f9b Address:852cefc1-8fe8-1a27-ad1e-037457a37f9b}]" + testing.go:149: server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: 2025-02-10T11:34:13.965Z [INFO] server-852cefc1-8fe8-1a27-ad1e-037457a37f9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-852cefc1-8fe8-1a27-ad1e-037457a37f9b Address:852cefc1-8fe8-1a27-ad1e-037457a37f9b}]" +--- PASS: TestRaft_SnapshotRestore (0.18s) === RUN TestRaft_NoRestoreOnStart -testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.550Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5bda9bf0-2bdf-11b5-a986-342347309877 Address:5bda9bf0-2bdf-11b5-a986-342347309877}]" - testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.550Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5bda9bf0-2bdf-11b5-a986-342347309877 Address:5bda9bf0-2bdf-11b5-a986-342347309877}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.550Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:59.550Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.550Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering follower state: follower="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Follower]" leader= - testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.550Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering follower state: follower="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Follower]" leader= -testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.647Z [WARN] server-5bda9bf0-2bdf-11b5-a986-342347309877: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.647Z [WARN] server-5bda9bf0-2bdf-11b5-a986-342347309877: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.647Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering candidate state: node="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Candidate]" term=2 - testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.647Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering candidate state: node="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Candidate]" term=2 -testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.648Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: election won: tally=1 - testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.648Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: election won: tally=1 -testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.648Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering leader state: leader="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Leader]" - testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.648Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering leader state: leader="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Leader]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.753Z [INFO] stable state for Leader reached at 2024-01-09 05:09:59.648225691 +0000 UTC m=+10.901276853 (1 nodes), 97.627937ms from start of poll, 97.811798ms from cluster start. Timeout at 2024-01-09 05:09:59.753301073 +0000 UTC m=+11.006352095, 105.075242ms after stability - testing.go:149: cluster: 2024-01-09T05:09:59.753Z [INFO] stable state for Leader reached at 2024-01-09 05:09:59.648225691 +0000 UTC m=+10.901276853 (1 nodes), 97.627937ms from start of poll, 97.811798ms from cluster start. Timeout at 2024-01-09 05:09:59.753301073 +0000 UTC m=+11.006352095, 105.075242ms after stability -testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.754Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: starting snapshot up to: index=102 - testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.754Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: starting snapshot up to: index=102 - testing.go:153: 2024-01-09T05:09:59.754Z [INFO] creating new snapshot: path=/tmp/raft460926126/snapshots/2-102-1704776999754.tmp -testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.762Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: compacting logs: from=1 to=92 - testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.762Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.762Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: snapshot complete up to: index=102 - testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.762Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.762Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5bda9bf0-2bdf-11b5-a986-342347309877 Address:5bda9bf0-2bdf-11b5-a986-342347309877}]" - testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.762Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5bda9bf0-2bdf-11b5-a986-342347309877 Address:5bda9bf0-2bdf-11b5-a986-342347309877}]" ---- PASS: TestRaft_NoRestoreOnStart (0.21s) +testLoggerAdapter verbose: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:13.965Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93cd1f0a-625e-f564-dce2-07799daf38b2 Address:93cd1f0a-625e-f564-dce2-07799daf38b2}]" + testing.go:149: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:13.965Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93cd1f0a-625e-f564-dce2-07799daf38b2 Address:93cd1f0a-625e-f564-dce2-07799daf38b2}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:13.965Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:13.965Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:13.965Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: entering follower state: follower="Node at 93cd1f0a-625e-f564-dce2-07799daf38b2 [Follower]" leader= + testing.go:149: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:13.965Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: entering follower state: follower="Node at 93cd1f0a-625e-f564-dce2-07799daf38b2 [Follower]" leader= +testLoggerAdapter verbose: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.034Z [WARN] server-93cd1f0a-625e-f564-dce2-07799daf38b2: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.034Z [WARN] server-93cd1f0a-625e-f564-dce2-07799daf38b2: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.034Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: entering candidate state: node="Node at 93cd1f0a-625e-f564-dce2-07799daf38b2 [Candidate]" term=2 + testing.go:149: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.034Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: entering candidate state: node="Node at 93cd1f0a-625e-f564-dce2-07799daf38b2 [Candidate]" term=2 +testLoggerAdapter verbose: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.034Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: election won: tally=1 + testing.go:149: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.034Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: election won: tally=1 +testLoggerAdapter verbose: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.034Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: entering leader state: leader="Node at 93cd1f0a-625e-f564-dce2-07799daf38b2 [Leader]" + testing.go:149: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.034Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: entering leader state: leader="Node at 93cd1f0a-625e-f564-dce2-07799daf38b2 [Leader]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:14.139Z [INFO] stable state for Leader reached at 2025-02-10 11:34:14.034202323 +0000 UTC m=+10.403106624 (1 nodes), 68.403527ms from start of poll, 68.545916ms from cluster start. Timeout at 2025-02-10 11:34:14.139279954 +0000 UTC m=+10.508184247, 105.077623ms after stability + testing.go:149: cluster: 2025-02-10T11:34:14.139Z [INFO] stable state for Leader reached at 2025-02-10 11:34:14.034202323 +0000 UTC m=+10.403106624 (1 nodes), 68.403527ms from start of poll, 68.545916ms from cluster start. Timeout at 2025-02-10 11:34:14.139279954 +0000 UTC m=+10.508184247, 105.077623ms after stability +testLoggerAdapter verbose: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.139Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: starting snapshot up to: index=102 + testing.go:149: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.139Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: starting snapshot up to: index=102 + testing.go:153: 2025-02-10T11:34:14.140Z [INFO] creating new snapshot: path=/tmp/raft315923839/snapshots/2-102-1739187254140.tmp +testLoggerAdapter verbose: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.144Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: compacting logs: from=1 to=92 + testing.go:149: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.144Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.144Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: snapshot complete up to: index=102 + testing.go:149: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.144Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.144Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93cd1f0a-625e-f564-dce2-07799daf38b2 Address:93cd1f0a-625e-f564-dce2-07799daf38b2}]" + testing.go:149: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.144Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93cd1f0a-625e-f564-dce2-07799daf38b2 Address:93cd1f0a-625e-f564-dce2-07799daf38b2}]" +--- PASS: TestRaft_NoRestoreOnStart (0.18s) === RUN TestRaft_SnapshotRestore_PeerChange raft_test.go:1026: Disabled unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot +testLoggerAdapter verbose: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.144Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: entering follower state: follower="Node at 93cd1f0a-625e-f564-dce2-07799daf38b2 [Follower]" leader= raft_test.go:1132: Disabled unreliable test --- SKIP: TestRaft_AutoSnapshot (0.00s) === RUN TestRaft_UserSnapshot -testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.763Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06 Address:b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06}]" - testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.763Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06 Address:b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.763Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:59.763Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.763Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering follower state: follower="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Follower]" leader= -testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.763Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: entering follower state: follower="Node at b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06 [Follower]" leader= - testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.763Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: entering follower state: follower="Node at b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06 [Follower]" leader= -testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.822Z [WARN] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.822Z [WARN] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.822Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: entering candidate state: node="Node at b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06 [Candidate]" term=2 - testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.822Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: entering candidate state: node="Node at b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06 [Candidate]" term=2 -testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.822Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: election won: tally=1 - testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.822Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: election won: tally=1 -testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.822Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: entering leader state: leader="Node at b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06 [Leader]" - testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.822Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: entering leader state: leader="Node at b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06 [Leader]" -testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.835Z [WARN] server-5bda9bf0-2bdf-11b5-a986-342347309877: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.835Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering candidate state: node="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Candidate]" term=3 -testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.835Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: election won: tally=1 -testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.835Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering leader state: leader="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Leader]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.927Z [INFO] stable state for Leader reached at 2024-01-09 05:09:59.822760144 +0000 UTC m=+11.075811216 (1 nodes), 59.254034ms from start of poll, 59.390684ms from cluster start. Timeout at 2024-01-09 05:09:59.927858419 +0000 UTC m=+11.180909549, 105.098333ms after stability - testing.go:149: cluster: 2024-01-09T05:09:59.927Z [INFO] stable state for Leader reached at 2024-01-09 05:09:59.822760144 +0000 UTC m=+11.075811216 (1 nodes), 59.254034ms from start of poll, 59.390684ms from cluster start. Timeout at 2024-01-09 05:09:59.927858419 +0000 UTC m=+11.180909549, 105.098333ms after stability -testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.928Z [ERROR] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: failed to take snapshot: error="nothing new to snapshot" - testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.928Z [ERROR] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: failed to take snapshot: error="nothing new to snapshot" -testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.928Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: starting snapshot up to: index=12 - testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.928Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: starting snapshot up to: index=12 - testing.go:153: 2024-01-09T05:09:59.928Z [INFO] creating new snapshot: path=/tmp/raft285144592/snapshots/2-12-1704776999928.tmp -testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.947Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: compacting logs: from=1 to=2 - testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.947Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: compacting logs: from=1 to=2 -testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.947Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: snapshot complete up to: index=12 - testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.947Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: snapshot complete up to: index=12 ---- PASS: TestRaft_UserSnapshot (0.19s) +testLoggerAdapter verbose: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.144Z [INFO] server-e603ad55-fc4f-b668-115f-550176ee85e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e603ad55-fc4f-b668-115f-550176ee85e2 Address:e603ad55-fc4f-b668-115f-550176ee85e2}]" + testing.go:149: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.144Z [INFO] server-e603ad55-fc4f-b668-115f-550176ee85e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e603ad55-fc4f-b668-115f-550176ee85e2 Address:e603ad55-fc4f-b668-115f-550176ee85e2}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:14.144Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:14.144Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.144Z [INFO] server-e603ad55-fc4f-b668-115f-550176ee85e2: entering follower state: follower="Node at e603ad55-fc4f-b668-115f-550176ee85e2 [Follower]" leader= + testing.go:149: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.144Z [INFO] server-e603ad55-fc4f-b668-115f-550176ee85e2: entering follower state: follower="Node at e603ad55-fc4f-b668-115f-550176ee85e2 [Follower]" leader= +testLoggerAdapter verbose: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.203Z [WARN] server-e603ad55-fc4f-b668-115f-550176ee85e2: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.203Z [WARN] server-e603ad55-fc4f-b668-115f-550176ee85e2: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.203Z [INFO] server-e603ad55-fc4f-b668-115f-550176ee85e2: entering candidate state: node="Node at e603ad55-fc4f-b668-115f-550176ee85e2 [Candidate]" term=2 + testing.go:149: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.203Z [INFO] server-e603ad55-fc4f-b668-115f-550176ee85e2: entering candidate state: node="Node at e603ad55-fc4f-b668-115f-550176ee85e2 [Candidate]" term=2 +testLoggerAdapter verbose: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.203Z [INFO] server-e603ad55-fc4f-b668-115f-550176ee85e2: election won: tally=1 + testing.go:149: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.203Z [INFO] server-e603ad55-fc4f-b668-115f-550176ee85e2: election won: tally=1 +testLoggerAdapter verbose: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.203Z [INFO] server-e603ad55-fc4f-b668-115f-550176ee85e2: entering leader state: leader="Node at e603ad55-fc4f-b668-115f-550176ee85e2 [Leader]" + testing.go:149: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.203Z [INFO] server-e603ad55-fc4f-b668-115f-550176ee85e2: entering leader state: leader="Node at e603ad55-fc4f-b668-115f-550176ee85e2 [Leader]" +testLoggerAdapter verbose: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.208Z [WARN] server-93cd1f0a-625e-f564-dce2-07799daf38b2: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.208Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: entering candidate state: node="Node at 93cd1f0a-625e-f564-dce2-07799daf38b2 [Candidate]" term=3 +testLoggerAdapter verbose: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.208Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: election won: tally=1 +testLoggerAdapter verbose: server-93cd1f0a-625e-f564-dce2-07799daf38b2: 2025-02-10T11:34:14.208Z [INFO] server-93cd1f0a-625e-f564-dce2-07799daf38b2: entering leader state: leader="Node at 93cd1f0a-625e-f564-dce2-07799daf38b2 [Leader]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:14.308Z [INFO] stable state for Leader reached at 2025-02-10 11:34:14.203387579 +0000 UTC m=+10.572291945 (1 nodes), 58.529487ms from start of poll, 58.685738ms from cluster start. Timeout at 2025-02-10 11:34:14.308406517 +0000 UTC m=+10.677310986, 105.019041ms after stability + testing.go:149: cluster: 2025-02-10T11:34:14.308Z [INFO] stable state for Leader reached at 2025-02-10 11:34:14.203387579 +0000 UTC m=+10.572291945 (1 nodes), 58.529487ms from start of poll, 58.685738ms from cluster start. Timeout at 2025-02-10 11:34:14.308406517 +0000 UTC m=+10.677310986, 105.019041ms after stability +testLoggerAdapter verbose: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.308Z [ERROR] server-e603ad55-fc4f-b668-115f-550176ee85e2: failed to take snapshot: error="nothing new to snapshot" + testing.go:149: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.308Z [ERROR] server-e603ad55-fc4f-b668-115f-550176ee85e2: failed to take snapshot: error="nothing new to snapshot" +testLoggerAdapter verbose: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.308Z [INFO] server-e603ad55-fc4f-b668-115f-550176ee85e2: starting snapshot up to: index=12 + testing.go:149: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.308Z [INFO] server-e603ad55-fc4f-b668-115f-550176ee85e2: starting snapshot up to: index=12 + testing.go:153: 2025-02-10T11:34:14.308Z [INFO] creating new snapshot: path=/tmp/raft019808265/snapshots/2-12-1739187254308.tmp +testLoggerAdapter verbose: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.312Z [INFO] server-e603ad55-fc4f-b668-115f-550176ee85e2: compacting logs: from=1 to=2 + testing.go:149: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.312Z [INFO] server-e603ad55-fc4f-b668-115f-550176ee85e2: compacting logs: from=1 to=2 +testLoggerAdapter verbose: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.312Z [INFO] server-e603ad55-fc4f-b668-115f-550176ee85e2: snapshot complete up to: index=12 + testing.go:149: server-e603ad55-fc4f-b668-115f-550176ee85e2: 2025-02-10T11:34:14.312Z [INFO] server-e603ad55-fc4f-b668-115f-550176ee85e2: snapshot complete up to: index=12 +--- PASS: TestRaft_UserSnapshot (0.17s) === RUN TestRaft_UserRestore -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:09:59.951Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a68b9448-5aac-228e-387f-635da2a03624 Address:a68b9448-5aac-228e-387f-635da2a03624} {Suffrage:Voter ID:server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 Address:39d4e4e1-1ef2-2bb4-7d89-481da9d138f9} {Suffrage:Voter ID:server-7b17c738-d926-4a54-385d-51c86c237c47 Address:7b17c738-d926-4a54-385d-51c86c237c47}]" - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:09:59.951Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a68b9448-5aac-228e-387f-635da2a03624 Address:a68b9448-5aac-228e-387f-635da2a03624} {Suffrage:Voter ID:server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 Address:39d4e4e1-1ef2-2bb4-7d89-481da9d138f9} {Suffrage:Voter ID:server-7b17c738-d926-4a54-385d-51c86c237c47 Address:7b17c738-d926-4a54-385d-51c86c237c47}]" -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:09:59.952Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: entering follower state: follower="Node at a68b9448-5aac-228e-387f-635da2a03624 [Follower]" leader= - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:09:59.952Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: entering follower state: follower="Node at a68b9448-5aac-228e-387f-635da2a03624 [Follower]" leader= -testLoggerAdapter verbose: server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: 2024-01-09T05:09:59.952Z [INFO] server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a68b9448-5aac-228e-387f-635da2a03624 Address:a68b9448-5aac-228e-387f-635da2a03624} {Suffrage:Voter ID:server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 Address:39d4e4e1-1ef2-2bb4-7d89-481da9d138f9} {Suffrage:Voter ID:server-7b17c738-d926-4a54-385d-51c86c237c47 Address:7b17c738-d926-4a54-385d-51c86c237c47}]" - testing.go:149: server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: 2024-01-09T05:09:59.952Z [INFO] server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a68b9448-5aac-228e-387f-635da2a03624 Address:a68b9448-5aac-228e-387f-635da2a03624} {Suffrage:Voter ID:server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 Address:39d4e4e1-1ef2-2bb4-7d89-481da9d138f9} {Suffrage:Voter ID:server-7b17c738-d926-4a54-385d-51c86c237c47 Address:7b17c738-d926-4a54-385d-51c86c237c47}]" -testLoggerAdapter verbose: server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: 2024-01-09T05:09:59.953Z [INFO] server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: entering follower state: follower="Node at 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 [Follower]" leader= - testing.go:149: server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: 2024-01-09T05:09:59.953Z [INFO] server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: entering follower state: follower="Node at 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 [Follower]" leader= -testLoggerAdapter verbose: server-7b17c738-d926-4a54-385d-51c86c237c47: 2024-01-09T05:09:59.954Z [INFO] server-7b17c738-d926-4a54-385d-51c86c237c47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a68b9448-5aac-228e-387f-635da2a03624 Address:a68b9448-5aac-228e-387f-635da2a03624} {Suffrage:Voter ID:server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 Address:39d4e4e1-1ef2-2bb4-7d89-481da9d138f9} {Suffrage:Voter ID:server-7b17c738-d926-4a54-385d-51c86c237c47 Address:7b17c738-d926-4a54-385d-51c86c237c47}]" - testing.go:149: server-7b17c738-d926-4a54-385d-51c86c237c47: 2024-01-09T05:09:59.954Z [INFO] server-7b17c738-d926-4a54-385d-51c86c237c47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a68b9448-5aac-228e-387f-635da2a03624 Address:a68b9448-5aac-228e-387f-635da2a03624} {Suffrage:Voter ID:server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 Address:39d4e4e1-1ef2-2bb4-7d89-481da9d138f9} {Suffrage:Voter ID:server-7b17c738-d926-4a54-385d-51c86c237c47 Address:7b17c738-d926-4a54-385d-51c86c237c47}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.954Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:09:59.954Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-7b17c738-d926-4a54-385d-51c86c237c47: 2024-01-09T05:09:59.954Z [INFO] server-7b17c738-d926-4a54-385d-51c86c237c47: entering follower state: follower="Node at 7b17c738-d926-4a54-385d-51c86c237c47 [Follower]" leader= - testing.go:149: server-7b17c738-d926-4a54-385d-51c86c237c47: 2024-01-09T05:09:59.954Z [INFO] server-7b17c738-d926-4a54-385d-51c86c237c47: entering follower state: follower="Node at 7b17c738-d926-4a54-385d-51c86c237c47 [Follower]" leader= -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.024Z [WARN] server-a68b9448-5aac-228e-387f-635da2a03624: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.024Z [WARN] server-a68b9448-5aac-228e-387f-635da2a03624: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.024Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: entering candidate state: node="Node at a68b9448-5aac-228e-387f-635da2a03624 [Candidate]" term=2 - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.024Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: entering candidate state: node="Node at a68b9448-5aac-228e-387f-635da2a03624 [Candidate]" term=2 -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.024Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: election won: tally=2 - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.024Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: election won: tally=2 -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.024Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: entering leader state: leader="Node at a68b9448-5aac-228e-387f-635da2a03624 [Leader]" - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.024Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: entering leader state: leader="Node at a68b9448-5aac-228e-387f-635da2a03624 [Leader]" -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.025Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: added peer, starting replication: peer=server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.025Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: added peer, starting replication: peer=server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.025Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: added peer, starting replication: peer=server-7b17c738-d926-4a54-385d-51c86c237c47 - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.025Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: added peer, starting replication: peer=server-7b17c738-d926-4a54-385d-51c86c237c47 -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.025Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: pipelining replication: peer="{Voter server-7b17c738-d926-4a54-385d-51c86c237c47 7b17c738-d926-4a54-385d-51c86c237c47}" - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.025Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: pipelining replication: peer="{Voter server-7b17c738-d926-4a54-385d-51c86c237c47 7b17c738-d926-4a54-385d-51c86c237c47}" -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.025Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: pipelining replication: peer="{Voter server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9}" - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.025Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: pipelining replication: peer="{Voter server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:00.130Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.025329694 +0000 UTC m=+11.278380755 (1 nodes), 71.166644ms from start of poll, 74.778508ms from cluster start. Timeout at 2024-01-09 05:10:00.130391623 +0000 UTC m=+11.383442610, 105.061855ms after stability - testing.go:149: cluster: 2024-01-09T05:10:00.130Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.025329694 +0000 UTC m=+11.278380755 (1 nodes), 71.166644ms from start of poll, 74.778508ms from cluster start. Timeout at 2024-01-09 05:10:00.130391623 +0000 UTC m=+11.383442610, 105.061855ms after stability -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.131Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: starting snapshot up to: index=12 - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.131Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: starting snapshot up to: index=12 - testing.go:153: 2024-01-09T05:10:00.131Z [INFO] creating new snapshot: path=/tmp/raft072134594/snapshots/2-12-1704777000131.tmp -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.138Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: snapshot complete up to: index=12 - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.138Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: snapshot complete up to: index=12 - testing.go:153: 2024-01-09T05:10:00.138Z [INFO] creating new snapshot: path=/tmp/raft072134594/snapshots/2-23-1704777000138.tmp -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: copied to local snapshot: bytes=71 - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: restored user snapshot: index=1 - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: restored user snapshot: index=1 -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [ERROR] server-a68b9448-5aac-228e-387f-635da2a03624: failed to get log: index=23 error="log not found" - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [ERROR] server-a68b9448-5aac-228e-387f-635da2a03624: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: aborting pipeline replication: peer="{Voter server-7b17c738-d926-4a54-385d-51c86c237c47 7b17c738-d926-4a54-385d-51c86c237c47}" - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: aborting pipeline replication: peer="{Voter server-7b17c738-d926-4a54-385d-51c86c237c47 7b17c738-d926-4a54-385d-51c86c237c47}" -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [ERROR] server-a68b9448-5aac-228e-387f-635da2a03624: failed to get log: index=23 error="log not found" - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [ERROR] server-a68b9448-5aac-228e-387f-635da2a03624: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: aborting pipeline replication: peer="{Voter server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9}" - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: aborting pipeline replication: peer="{Voter server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9}" -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.155Z [ERROR] server-a68b9448-5aac-228e-387f-635da2a03624: failed to get log: index=23 error="log not found" - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.155Z [ERROR] server-a68b9448-5aac-228e-387f-635da2a03624: failed to get log: index=23 error="log not found" - testing.go:153: 2024-01-09T05:10:00.156Z [INFO] creating new snapshot: path=/tmp/raft968478230/snapshots/2-23-1704777000156.tmp -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.158Z [ERROR] server-a68b9448-5aac-228e-387f-635da2a03624: failed to get log: index=23 error="log not found" - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.158Z [ERROR] server-a68b9448-5aac-228e-387f-635da2a03624: failed to get log: index=23 error="log not found" - testing.go:153: 2024-01-09T05:10:00.158Z [INFO] creating new snapshot: path=/tmp/raft182449732/snapshots/2-23-1704777000158.tmp -testLoggerAdapter verbose: server-7b17c738-d926-4a54-385d-51c86c237c47: 2024-01-09T05:10:00.183Z [INFO] server-7b17c738-d926-4a54-385d-51c86c237c47: copied to local snapshot: bytes=71 - testing.go:149: server-7b17c738-d926-4a54-385d-51c86c237c47: 2024-01-09T05:10:00.183Z [INFO] server-7b17c738-d926-4a54-385d-51c86c237c47: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-7b17c738-d926-4a54-385d-51c86c237c47: 2024-01-09T05:10:00.183Z [INFO] server-7b17c738-d926-4a54-385d-51c86c237c47: Installed remote snapshot - testing.go:149: server-7b17c738-d926-4a54-385d-51c86c237c47: 2024-01-09T05:10:00.183Z [INFO] server-7b17c738-d926-4a54-385d-51c86c237c47: Installed remote snapshot -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.184Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: pipelining replication: peer="{Voter server-7b17c738-d926-4a54-385d-51c86c237c47 7b17c738-d926-4a54-385d-51c86c237c47}" - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.184Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: pipelining replication: peer="{Voter server-7b17c738-d926-4a54-385d-51c86c237c47 7b17c738-d926-4a54-385d-51c86c237c47}" -testLoggerAdapter verbose: server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: 2024-01-09T05:10:00.184Z [INFO] server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: copied to local snapshot: bytes=71 - testing.go:149: server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: 2024-01-09T05:10:00.184Z [INFO] server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: 2024-01-09T05:10:00.185Z [INFO] server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: Installed remote snapshot - testing.go:149: server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: 2024-01-09T05:10:00.185Z [INFO] server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: Installed remote snapshot -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.185Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: pipelining replication: peer="{Voter server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9}" - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.185Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: pipelining replication: peer="{Voter server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9}" -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.200Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: aborting pipeline replication: peer="{Voter server-7b17c738-d926-4a54-385d-51c86c237c47 7b17c738-d926-4a54-385d-51c86c237c47}" - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.200Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: aborting pipeline replication: peer="{Voter server-7b17c738-d926-4a54-385d-51c86c237c47 7b17c738-d926-4a54-385d-51c86c237c47}" -testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.200Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: aborting pipeline replication: peer="{Voter server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9}" - testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.200Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: aborting pipeline replication: peer="{Voter server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9}" -testLoggerAdapter verbose: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.202Z [INFO] server-ec231c35-cbda-d116-0747-707f226cf159: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec231c35-cbda-d116-0747-707f226cf159 Address:ec231c35-cbda-d116-0747-707f226cf159} {Suffrage:Voter ID:server-bbfdd75e-6d44-738b-78d8-87f6425a0d08 Address:bbfdd75e-6d44-738b-78d8-87f6425a0d08} {Suffrage:Voter ID:server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 Address:bcaf3bbb-754c-2939-4a43-462cc5e29f87}]" - testing.go:149: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.202Z [INFO] server-ec231c35-cbda-d116-0747-707f226cf159: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec231c35-cbda-d116-0747-707f226cf159 Address:ec231c35-cbda-d116-0747-707f226cf159} {Suffrage:Voter ID:server-bbfdd75e-6d44-738b-78d8-87f6425a0d08 Address:bbfdd75e-6d44-738b-78d8-87f6425a0d08} {Suffrage:Voter ID:server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 Address:bcaf3bbb-754c-2939-4a43-462cc5e29f87}]" -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.202Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec231c35-cbda-d116-0747-707f226cf159 Address:ec231c35-cbda-d116-0747-707f226cf159} {Suffrage:Voter ID:server-bbfdd75e-6d44-738b-78d8-87f6425a0d08 Address:bbfdd75e-6d44-738b-78d8-87f6425a0d08} {Suffrage:Voter ID:server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 Address:bcaf3bbb-754c-2939-4a43-462cc5e29f87}]" -testLoggerAdapter verbose: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.202Z [INFO] server-ec231c35-cbda-d116-0747-707f226cf159: entering follower state: follower="Node at ec231c35-cbda-d116-0747-707f226cf159 [Follower]" leader= - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.202Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec231c35-cbda-d116-0747-707f226cf159 Address:ec231c35-cbda-d116-0747-707f226cf159} {Suffrage:Voter ID:server-bbfdd75e-6d44-738b-78d8-87f6425a0d08 Address:bbfdd75e-6d44-738b-78d8-87f6425a0d08} {Suffrage:Voter ID:server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 Address:bcaf3bbb-754c-2939-4a43-462cc5e29f87}]" -testLoggerAdapter verbose: server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: 2024-01-09T05:10:00.203Z [INFO] server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec231c35-cbda-d116-0747-707f226cf159 Address:ec231c35-cbda-d116-0747-707f226cf159} {Suffrage:Voter ID:server-bbfdd75e-6d44-738b-78d8-87f6425a0d08 Address:bbfdd75e-6d44-738b-78d8-87f6425a0d08} {Suffrage:Voter ID:server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 Address:bcaf3bbb-754c-2939-4a43-462cc5e29f87}]" - testing.go:149: server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: 2024-01-09T05:10:00.203Z [INFO] server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec231c35-cbda-d116-0747-707f226cf159 Address:ec231c35-cbda-d116-0747-707f226cf159} {Suffrage:Voter ID:server-bbfdd75e-6d44-738b-78d8-87f6425a0d08 Address:bbfdd75e-6d44-738b-78d8-87f6425a0d08} {Suffrage:Voter ID:server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 Address:bcaf3bbb-754c-2939-4a43-462cc5e29f87}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:00.203Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:00.203Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.202Z [INFO] server-ec231c35-cbda-d116-0747-707f226cf159: entering follower state: follower="Node at ec231c35-cbda-d116-0747-707f226cf159 [Follower]" leader= -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.203Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: entering follower state: follower="Node at bbfdd75e-6d44-738b-78d8-87f6425a0d08 [Follower]" leader= - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.203Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: entering follower state: follower="Node at bbfdd75e-6d44-738b-78d8-87f6425a0d08 [Follower]" leader= -testLoggerAdapter verbose: server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: 2024-01-09T05:10:00.203Z [INFO] server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: entering follower state: follower="Node at bcaf3bbb-754c-2939-4a43-462cc5e29f87 [Follower]" leader= - testing.go:149: server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: 2024-01-09T05:10:00.203Z [INFO] server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: entering follower state: follower="Node at bcaf3bbb-754c-2939-4a43-462cc5e29f87 [Follower]" leader= -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.257Z [WARN] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.257Z [WARN] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: entering candidate state: node="Node at bbfdd75e-6d44-738b-78d8-87f6425a0d08 [Candidate]" term=2 - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: entering candidate state: node="Node at bbfdd75e-6d44-738b-78d8-87f6425a0d08 [Candidate]" term=2 -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: election won: tally=2 - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: election won: tally=2 -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: entering leader state: leader="Node at bbfdd75e-6d44-738b-78d8-87f6425a0d08 [Leader]" - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: entering leader state: leader="Node at bbfdd75e-6d44-738b-78d8-87f6425a0d08 [Leader]" -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: added peer, starting replication: peer=server-ec231c35-cbda-d116-0747-707f226cf159 - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: added peer, starting replication: peer=server-ec231c35-cbda-d116-0747-707f226cf159 -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: added peer, starting replication: peer=server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: added peer, starting replication: peer=server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: pipelining replication: peer="{Voter server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 bcaf3bbb-754c-2939-4a43-462cc5e29f87}" - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: pipelining replication: peer="{Voter server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 bcaf3bbb-754c-2939-4a43-462cc5e29f87}" -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: pipelining replication: peer="{Voter server-ec231c35-cbda-d116-0747-707f226cf159 ec231c35-cbda-d116-0747-707f226cf159}" - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: pipelining replication: peer="{Voter server-ec231c35-cbda-d116-0747-707f226cf159 ec231c35-cbda-d116-0747-707f226cf159}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:00.363Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.258370623 +0000 UTC m=+11.511421812 (1 nodes), 54.669167ms from start of poll, 55.771458ms from cluster start. Timeout at 2024-01-09 05:10:00.363533126 +0000 UTC m=+11.616584229, 105.162417ms after stability - testing.go:149: cluster: 2024-01-09T05:10:00.363Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.258370623 +0000 UTC m=+11.511421812 (1 nodes), 54.669167ms from start of poll, 55.771458ms from cluster start. Timeout at 2024-01-09 05:10:00.363533126 +0000 UTC m=+11.616584229, 105.162417ms after stability -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.364Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: starting snapshot up to: index=12 - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.364Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: starting snapshot up to: index=12 - testing.go:153: 2024-01-09T05:10:00.364Z [INFO] creating new snapshot: path=/tmp/raft016711082/snapshots/2-12-1704777000364.tmp -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.371Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: snapshot complete up to: index=12 - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.371Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: snapshot complete up to: index=12 - testing.go:153: 2024-01-09T05:10:00.372Z [INFO] creating new snapshot: path=/tmp/raft016711082/snapshots/2-23-1704777000372.tmp -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: copied to local snapshot: bytes=71 - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: restored user snapshot: index=1 - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: restored user snapshot: index=1 -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [ERROR] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: failed to get log: index=23 error="log not found" - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [ERROR] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: aborting pipeline replication: peer="{Voter server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 bcaf3bbb-754c-2939-4a43-462cc5e29f87}" - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: aborting pipeline replication: peer="{Voter server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 bcaf3bbb-754c-2939-4a43-462cc5e29f87}" -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [ERROR] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: failed to get log: index=23 error="log not found" - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [ERROR] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: aborting pipeline replication: peer="{Voter server-ec231c35-cbda-d116-0747-707f226cf159 ec231c35-cbda-d116-0747-707f226cf159}" - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: aborting pipeline replication: peer="{Voter server-ec231c35-cbda-d116-0747-707f226cf159 ec231c35-cbda-d116-0747-707f226cf159}" -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.388Z [ERROR] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: failed to get log: index=23 error="log not found" - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.388Z [ERROR] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: failed to get log: index=23 error="log not found" - testing.go:153: 2024-01-09T05:10:00.388Z [INFO] creating new snapshot: path=/tmp/raft039767480/snapshots/2-23-1704777000388.tmp -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.392Z [ERROR] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: failed to get log: index=23 error="log not found" - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.392Z [ERROR] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: failed to get log: index=23 error="log not found" - testing.go:153: 2024-01-09T05:10:00.392Z [INFO] creating new snapshot: path=/tmp/raft246478444/snapshots/2-23-1704777000392.tmp -testLoggerAdapter verbose: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.400Z [INFO] server-ec231c35-cbda-d116-0747-707f226cf159: copied to local snapshot: bytes=71 - testing.go:149: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.400Z [INFO] server-ec231c35-cbda-d116-0747-707f226cf159: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.401Z [INFO] server-ec231c35-cbda-d116-0747-707f226cf159: Installed remote snapshot - testing.go:149: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.401Z [INFO] server-ec231c35-cbda-d116-0747-707f226cf159: Installed remote snapshot -testLoggerAdapter verbose: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.401Z [WARN] server-ec231c35-cbda-d116-0747-707f226cf159: skipping application of old log: index=22 - testing.go:149: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.401Z [WARN] server-ec231c35-cbda-d116-0747-707f226cf159: skipping application of old log: index=22 -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.401Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: pipelining replication: peer="{Voter server-ec231c35-cbda-d116-0747-707f226cf159 ec231c35-cbda-d116-0747-707f226cf159}" - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.401Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: pipelining replication: peer="{Voter server-ec231c35-cbda-d116-0747-707f226cf159 ec231c35-cbda-d116-0747-707f226cf159}" -testLoggerAdapter verbose: server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: 2024-01-09T05:10:00.402Z [INFO] server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: copied to local snapshot: bytes=71 - testing.go:149: server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: 2024-01-09T05:10:00.402Z [INFO] server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: 2024-01-09T05:10:00.403Z [INFO] server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: Installed remote snapshot - testing.go:149: server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: 2024-01-09T05:10:00.403Z [INFO] server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: Installed remote snapshot -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.403Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: pipelining replication: peer="{Voter server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 bcaf3bbb-754c-2939-4a43-462cc5e29f87}" - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.403Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: pipelining replication: peer="{Voter server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 bcaf3bbb-754c-2939-4a43-462cc5e29f87}" -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.420Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: aborting pipeline replication: peer="{Voter server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 bcaf3bbb-754c-2939-4a43-462cc5e29f87}" - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.420Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: aborting pipeline replication: peer="{Voter server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 bcaf3bbb-754c-2939-4a43-462cc5e29f87}" -testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.420Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: aborting pipeline replication: peer="{Voter server-ec231c35-cbda-d116-0747-707f226cf159 ec231c35-cbda-d116-0747-707f226cf159}" - testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.420Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: aborting pipeline replication: peer="{Voter server-ec231c35-cbda-d116-0747-707f226cf159 ec231c35-cbda-d116-0747-707f226cf159}" -testLoggerAdapter verbose: server-e64a76d6-af6a-9995-419b-a7cad3a65539: 2024-01-09T05:10:00.427Z [INFO] server-e64a76d6-af6a-9995-419b-a7cad3a65539: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e64a76d6-af6a-9995-419b-a7cad3a65539 Address:e64a76d6-af6a-9995-419b-a7cad3a65539} {Suffrage:Voter ID:server-b5e31636-90a1-b17c-5e9f-69b7b408058f Address:b5e31636-90a1-b17c-5e9f-69b7b408058f} {Suffrage:Voter ID:server-82fff768-5fd2-c8d5-68b3-669b8047ea37 Address:82fff768-5fd2-c8d5-68b3-669b8047ea37}]" - testing.go:149: server-e64a76d6-af6a-9995-419b-a7cad3a65539: 2024-01-09T05:10:00.427Z [INFO] server-e64a76d6-af6a-9995-419b-a7cad3a65539: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e64a76d6-af6a-9995-419b-a7cad3a65539 Address:e64a76d6-af6a-9995-419b-a7cad3a65539} {Suffrage:Voter ID:server-b5e31636-90a1-b17c-5e9f-69b7b408058f Address:b5e31636-90a1-b17c-5e9f-69b7b408058f} {Suffrage:Voter ID:server-82fff768-5fd2-c8d5-68b3-669b8047ea37 Address:82fff768-5fd2-c8d5-68b3-669b8047ea37}]" -testLoggerAdapter verbose: server-b5e31636-90a1-b17c-5e9f-69b7b408058f: 2024-01-09T05:10:00.427Z [INFO] server-b5e31636-90a1-b17c-5e9f-69b7b408058f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e64a76d6-af6a-9995-419b-a7cad3a65539 Address:e64a76d6-af6a-9995-419b-a7cad3a65539} {Suffrage:Voter ID:server-b5e31636-90a1-b17c-5e9f-69b7b408058f Address:b5e31636-90a1-b17c-5e9f-69b7b408058f} {Suffrage:Voter ID:server-82fff768-5fd2-c8d5-68b3-669b8047ea37 Address:82fff768-5fd2-c8d5-68b3-669b8047ea37}]" - testing.go:149: server-b5e31636-90a1-b17c-5e9f-69b7b408058f: 2024-01-09T05:10:00.427Z [INFO] server-b5e31636-90a1-b17c-5e9f-69b7b408058f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e64a76d6-af6a-9995-419b-a7cad3a65539 Address:e64a76d6-af6a-9995-419b-a7cad3a65539} {Suffrage:Voter ID:server-b5e31636-90a1-b17c-5e9f-69b7b408058f Address:b5e31636-90a1-b17c-5e9f-69b7b408058f} {Suffrage:Voter ID:server-82fff768-5fd2-c8d5-68b3-669b8047ea37 Address:82fff768-5fd2-c8d5-68b3-669b8047ea37}]" -testLoggerAdapter verbose: server-e64a76d6-af6a-9995-419b-a7cad3a65539: 2024-01-09T05:10:00.427Z [INFO] server-e64a76d6-af6a-9995-419b-a7cad3a65539: entering follower state: follower="Node at e64a76d6-af6a-9995-419b-a7cad3a65539 [Follower]" leader= - testing.go:149: server-e64a76d6-af6a-9995-419b-a7cad3a65539: 2024-01-09T05:10:00.427Z [INFO] server-e64a76d6-af6a-9995-419b-a7cad3a65539: entering follower state: follower="Node at e64a76d6-af6a-9995-419b-a7cad3a65539 [Follower]" leader= -testLoggerAdapter verbose: server-b5e31636-90a1-b17c-5e9f-69b7b408058f: 2024-01-09T05:10:00.427Z [INFO] server-b5e31636-90a1-b17c-5e9f-69b7b408058f: entering follower state: follower="Node at b5e31636-90a1-b17c-5e9f-69b7b408058f [Follower]" leader= - testing.go:149: server-b5e31636-90a1-b17c-5e9f-69b7b408058f: 2024-01-09T05:10:00.427Z [INFO] server-b5e31636-90a1-b17c-5e9f-69b7b408058f: entering follower state: follower="Node at b5e31636-90a1-b17c-5e9f-69b7b408058f [Follower]" leader= -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.428Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e64a76d6-af6a-9995-419b-a7cad3a65539 Address:e64a76d6-af6a-9995-419b-a7cad3a65539} {Suffrage:Voter ID:server-b5e31636-90a1-b17c-5e9f-69b7b408058f Address:b5e31636-90a1-b17c-5e9f-69b7b408058f} {Suffrage:Voter ID:server-82fff768-5fd2-c8d5-68b3-669b8047ea37 Address:82fff768-5fd2-c8d5-68b3-669b8047ea37}]" - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.428Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e64a76d6-af6a-9995-419b-a7cad3a65539 Address:e64a76d6-af6a-9995-419b-a7cad3a65539} {Suffrage:Voter ID:server-b5e31636-90a1-b17c-5e9f-69b7b408058f Address:b5e31636-90a1-b17c-5e9f-69b7b408058f} {Suffrage:Voter ID:server-82fff768-5fd2-c8d5-68b3-669b8047ea37 Address:82fff768-5fd2-c8d5-68b3-669b8047ea37}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:00.428Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:00.428Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.429Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: entering follower state: follower="Node at 82fff768-5fd2-c8d5-68b3-669b8047ea37 [Follower]" leader= - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.429Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: entering follower state: follower="Node at 82fff768-5fd2-c8d5-68b3-669b8047ea37 [Follower]" leader= -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [WARN] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [WARN] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: entering candidate state: node="Node at 82fff768-5fd2-c8d5-68b3-669b8047ea37 [Candidate]" term=2 - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: entering candidate state: node="Node at 82fff768-5fd2-c8d5-68b3-669b8047ea37 [Candidate]" term=2 -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: election won: tally=2 - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: election won: tally=2 -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: entering leader state: leader="Node at 82fff768-5fd2-c8d5-68b3-669b8047ea37 [Leader]" - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: entering leader state: leader="Node at 82fff768-5fd2-c8d5-68b3-669b8047ea37 [Leader]" -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: added peer, starting replication: peer=server-e64a76d6-af6a-9995-419b-a7cad3a65539 - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: added peer, starting replication: peer=server-e64a76d6-af6a-9995-419b-a7cad3a65539 -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: added peer, starting replication: peer=server-b5e31636-90a1-b17c-5e9f-69b7b408058f - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: added peer, starting replication: peer=server-b5e31636-90a1-b17c-5e9f-69b7b408058f -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: pipelining replication: peer="{Voter server-b5e31636-90a1-b17c-5e9f-69b7b408058f b5e31636-90a1-b17c-5e9f-69b7b408058f}" - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: pipelining replication: peer="{Voter server-b5e31636-90a1-b17c-5e9f-69b7b408058f b5e31636-90a1-b17c-5e9f-69b7b408058f}" -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: pipelining replication: peer="{Voter server-e64a76d6-af6a-9995-419b-a7cad3a65539 e64a76d6-af6a-9995-419b-a7cad3a65539}" - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: pipelining replication: peer="{Voter server-e64a76d6-af6a-9995-419b-a7cad3a65539 e64a76d6-af6a-9995-419b-a7cad3a65539}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:00.593Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.488034825 +0000 UTC m=+11.741085915 (1 nodes), 59.449482ms from start of poll, 61.518318ms from cluster start. Timeout at 2024-01-09 05:10:00.593602385 +0000 UTC m=+11.846653638, 105.567723ms after stability - testing.go:149: cluster: 2024-01-09T05:10:00.593Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.488034825 +0000 UTC m=+11.741085915 (1 nodes), 59.449482ms from start of poll, 61.518318ms from cluster start. Timeout at 2024-01-09 05:10:00.593602385 +0000 UTC m=+11.846653638, 105.567723ms after stability -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.594Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: starting snapshot up to: index=12 - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.594Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: starting snapshot up to: index=12 - testing.go:153: 2024-01-09T05:10:00.594Z [INFO] creating new snapshot: path=/tmp/raft209861266/snapshots/2-12-1704777000594.tmp -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.603Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: snapshot complete up to: index=12 - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.603Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: snapshot complete up to: index=12 - testing.go:153: 2024-01-09T05:10:00.603Z [INFO] creating new snapshot: path=/tmp/raft209861266/snapshots/2-23-1704777000603.tmp -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.611Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: copied to local snapshot: bytes=71 - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.611Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: restored user snapshot: index=1 - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: restored user snapshot: index=1 -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [ERROR] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: failed to get log: index=23 error="log not found" - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [ERROR] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: aborting pipeline replication: peer="{Voter server-e64a76d6-af6a-9995-419b-a7cad3a65539 e64a76d6-af6a-9995-419b-a7cad3a65539}" - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: aborting pipeline replication: peer="{Voter server-e64a76d6-af6a-9995-419b-a7cad3a65539 e64a76d6-af6a-9995-419b-a7cad3a65539}" -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [ERROR] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: failed to get log: index=23 error="log not found" - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [ERROR] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: aborting pipeline replication: peer="{Voter server-b5e31636-90a1-b17c-5e9f-69b7b408058f b5e31636-90a1-b17c-5e9f-69b7b408058f}" - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: aborting pipeline replication: peer="{Voter server-b5e31636-90a1-b17c-5e9f-69b7b408058f b5e31636-90a1-b17c-5e9f-69b7b408058f}" -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.619Z [ERROR] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: failed to get log: index=23 error="log not found" - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.619Z [ERROR] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: failed to get log: index=23 error="log not found" - testing.go:153: 2024-01-09T05:10:00.620Z [INFO] creating new snapshot: path=/tmp/raft489713248/snapshots/2-23-1704777000620.tmp -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.622Z [ERROR] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: failed to get log: index=23 error="log not found" - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.622Z [ERROR] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: failed to get log: index=23 error="log not found" - testing.go:153: 2024-01-09T05:10:00.623Z [INFO] creating new snapshot: path=/tmp/raft580825214/snapshots/2-23-1704777000623.tmp -testLoggerAdapter verbose: server-b5e31636-90a1-b17c-5e9f-69b7b408058f: 2024-01-09T05:10:00.632Z [INFO] server-b5e31636-90a1-b17c-5e9f-69b7b408058f: copied to local snapshot: bytes=71 - testing.go:149: server-b5e31636-90a1-b17c-5e9f-69b7b408058f: 2024-01-09T05:10:00.632Z [INFO] server-b5e31636-90a1-b17c-5e9f-69b7b408058f: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-e64a76d6-af6a-9995-419b-a7cad3a65539: 2024-01-09T05:10:00.633Z [INFO] server-e64a76d6-af6a-9995-419b-a7cad3a65539: copied to local snapshot: bytes=71 - testing.go:149: server-e64a76d6-af6a-9995-419b-a7cad3a65539: 2024-01-09T05:10:00.633Z [INFO] server-e64a76d6-af6a-9995-419b-a7cad3a65539: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-e64a76d6-af6a-9995-419b-a7cad3a65539: 2024-01-09T05:10:00.633Z [INFO] server-e64a76d6-af6a-9995-419b-a7cad3a65539: Installed remote snapshot - testing.go:149: server-e64a76d6-af6a-9995-419b-a7cad3a65539: 2024-01-09T05:10:00.633Z [INFO] server-e64a76d6-af6a-9995-419b-a7cad3a65539: Installed remote snapshot -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.633Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: pipelining replication: peer="{Voter server-e64a76d6-af6a-9995-419b-a7cad3a65539 e64a76d6-af6a-9995-419b-a7cad3a65539}" - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.633Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: pipelining replication: peer="{Voter server-e64a76d6-af6a-9995-419b-a7cad3a65539 e64a76d6-af6a-9995-419b-a7cad3a65539}" -testLoggerAdapter verbose: server-b5e31636-90a1-b17c-5e9f-69b7b408058f: 2024-01-09T05:10:00.633Z [INFO] server-b5e31636-90a1-b17c-5e9f-69b7b408058f: Installed remote snapshot - testing.go:149: server-b5e31636-90a1-b17c-5e9f-69b7b408058f: 2024-01-09T05:10:00.633Z [INFO] server-b5e31636-90a1-b17c-5e9f-69b7b408058f: Installed remote snapshot -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.633Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: pipelining replication: peer="{Voter server-b5e31636-90a1-b17c-5e9f-69b7b408058f b5e31636-90a1-b17c-5e9f-69b7b408058f}" - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.633Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: pipelining replication: peer="{Voter server-b5e31636-90a1-b17c-5e9f-69b7b408058f b5e31636-90a1-b17c-5e9f-69b7b408058f}" -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.651Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: aborting pipeline replication: peer="{Voter server-b5e31636-90a1-b17c-5e9f-69b7b408058f b5e31636-90a1-b17c-5e9f-69b7b408058f}" - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.651Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: aborting pipeline replication: peer="{Voter server-b5e31636-90a1-b17c-5e9f-69b7b408058f b5e31636-90a1-b17c-5e9f-69b7b408058f}" -testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.651Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: aborting pipeline replication: peer="{Voter server-e64a76d6-af6a-9995-419b-a7cad3a65539 e64a76d6-af6a-9995-419b-a7cad3a65539}" - testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.651Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: aborting pipeline replication: peer="{Voter server-e64a76d6-af6a-9995-419b-a7cad3a65539 e64a76d6-af6a-9995-419b-a7cad3a65539}" -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.653Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62b05e7b-869f-f84a-ab0d-250af0adb19a Address:62b05e7b-869f-f84a-ab0d-250af0adb19a} {Suffrage:Voter ID:server-f28f3437-45ec-cfb0-a8bb-291c891734ad Address:f28f3437-45ec-cfb0-a8bb-291c891734ad} {Suffrage:Voter ID:server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 Address:d64eed0d-cc3d-74e5-402b-904bfe0618d8}]" - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.653Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62b05e7b-869f-f84a-ab0d-250af0adb19a Address:62b05e7b-869f-f84a-ab0d-250af0adb19a} {Suffrage:Voter ID:server-f28f3437-45ec-cfb0-a8bb-291c891734ad Address:f28f3437-45ec-cfb0-a8bb-291c891734ad} {Suffrage:Voter ID:server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 Address:d64eed0d-cc3d-74e5-402b-904bfe0618d8}]" -testLoggerAdapter verbose: server-f28f3437-45ec-cfb0-a8bb-291c891734ad: 2024-01-09T05:10:00.653Z [INFO] server-f28f3437-45ec-cfb0-a8bb-291c891734ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62b05e7b-869f-f84a-ab0d-250af0adb19a Address:62b05e7b-869f-f84a-ab0d-250af0adb19a} {Suffrage:Voter ID:server-f28f3437-45ec-cfb0-a8bb-291c891734ad Address:f28f3437-45ec-cfb0-a8bb-291c891734ad} {Suffrage:Voter ID:server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 Address:d64eed0d-cc3d-74e5-402b-904bfe0618d8}]" - testing.go:149: server-f28f3437-45ec-cfb0-a8bb-291c891734ad: 2024-01-09T05:10:00.653Z [INFO] server-f28f3437-45ec-cfb0-a8bb-291c891734ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62b05e7b-869f-f84a-ab0d-250af0adb19a Address:62b05e7b-869f-f84a-ab0d-250af0adb19a} {Suffrage:Voter ID:server-f28f3437-45ec-cfb0-a8bb-291c891734ad Address:f28f3437-45ec-cfb0-a8bb-291c891734ad} {Suffrage:Voter ID:server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 Address:d64eed0d-cc3d-74e5-402b-904bfe0618d8}]" -testLoggerAdapter verbose: server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: 2024-01-09T05:10:00.654Z [INFO] server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62b05e7b-869f-f84a-ab0d-250af0adb19a Address:62b05e7b-869f-f84a-ab0d-250af0adb19a} {Suffrage:Voter ID:server-f28f3437-45ec-cfb0-a8bb-291c891734ad Address:f28f3437-45ec-cfb0-a8bb-291c891734ad} {Suffrage:Voter ID:server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 Address:d64eed0d-cc3d-74e5-402b-904bfe0618d8}]" - testing.go:149: server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: 2024-01-09T05:10:00.654Z [INFO] server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62b05e7b-869f-f84a-ab0d-250af0adb19a Address:62b05e7b-869f-f84a-ab0d-250af0adb19a} {Suffrage:Voter ID:server-f28f3437-45ec-cfb0-a8bb-291c891734ad Address:f28f3437-45ec-cfb0-a8bb-291c891734ad} {Suffrage:Voter ID:server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 Address:d64eed0d-cc3d-74e5-402b-904bfe0618d8}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:00.654Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:00.654Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.654Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: entering follower state: follower="Node at 62b05e7b-869f-f84a-ab0d-250af0adb19a [Follower]" leader= - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.654Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: entering follower state: follower="Node at 62b05e7b-869f-f84a-ab0d-250af0adb19a [Follower]" leader= -testLoggerAdapter verbose: server-f28f3437-45ec-cfb0-a8bb-291c891734ad: 2024-01-09T05:10:00.654Z [INFO] server-f28f3437-45ec-cfb0-a8bb-291c891734ad: entering follower state: follower="Node at f28f3437-45ec-cfb0-a8bb-291c891734ad [Follower]" leader= - testing.go:149: server-f28f3437-45ec-cfb0-a8bb-291c891734ad: 2024-01-09T05:10:00.654Z [INFO] server-f28f3437-45ec-cfb0-a8bb-291c891734ad: entering follower state: follower="Node at f28f3437-45ec-cfb0-a8bb-291c891734ad [Follower]" leader= -testLoggerAdapter verbose: server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: 2024-01-09T05:10:00.654Z [INFO] server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: entering follower state: follower="Node at d64eed0d-cc3d-74e5-402b-904bfe0618d8 [Follower]" leader= - testing.go:149: server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: 2024-01-09T05:10:00.654Z [INFO] server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: entering follower state: follower="Node at d64eed0d-cc3d-74e5-402b-904bfe0618d8 [Follower]" leader= -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [WARN] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [WARN] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: entering candidate state: node="Node at 62b05e7b-869f-f84a-ab0d-250af0adb19a [Candidate]" term=2 - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: entering candidate state: node="Node at 62b05e7b-869f-f84a-ab0d-250af0adb19a [Candidate]" term=2 -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: election won: tally=2 - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: election won: tally=2 -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: entering leader state: leader="Node at 62b05e7b-869f-f84a-ab0d-250af0adb19a [Leader]" - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: entering leader state: leader="Node at 62b05e7b-869f-f84a-ab0d-250af0adb19a [Leader]" -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: added peer, starting replication: peer=server-f28f3437-45ec-cfb0-a8bb-291c891734ad - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: added peer, starting replication: peer=server-f28f3437-45ec-cfb0-a8bb-291c891734ad -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: added peer, starting replication: peer=server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: added peer, starting replication: peer=server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: pipelining replication: peer="{Voter server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 d64eed0d-cc3d-74e5-402b-904bfe0618d8}" - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: pipelining replication: peer="{Voter server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 d64eed0d-cc3d-74e5-402b-904bfe0618d8}" -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: pipelining replication: peer="{Voter server-f28f3437-45ec-cfb0-a8bb-291c891734ad f28f3437-45ec-cfb0-a8bb-291c891734ad}" - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: pipelining replication: peer="{Voter server-f28f3437-45ec-cfb0-a8bb-291c891734ad f28f3437-45ec-cfb0-a8bb-291c891734ad}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:00.833Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.727874005 +0000 UTC m=+11.980925143 (1 nodes), 73.487736ms from start of poll, 74.292803ms from cluster start. Timeout at 2024-01-09 05:10:00.832977705 +0000 UTC m=+12.086028715, 105.103572ms after stability - testing.go:149: cluster: 2024-01-09T05:10:00.833Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.727874005 +0000 UTC m=+11.980925143 (1 nodes), 73.487736ms from start of poll, 74.292803ms from cluster start. Timeout at 2024-01-09 05:10:00.832977705 +0000 UTC m=+12.086028715, 105.103572ms after stability -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.833Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: starting snapshot up to: index=12 - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.833Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: starting snapshot up to: index=12 - testing.go:153: 2024-01-09T05:10:00.833Z [INFO] creating new snapshot: path=/tmp/raft145392020/snapshots/2-12-1704777000833.tmp -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.844Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: snapshot complete up to: index=12 - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.844Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: snapshot complete up to: index=12 - testing.go:153: 2024-01-09T05:10:00.845Z [INFO] creating new snapshot: path=/tmp/raft145392020/snapshots/2-113-1704777000845.tmp -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: copied to local snapshot: bytes=71 - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: restored user snapshot: index=1 - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: restored user snapshot: index=1 -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [ERROR] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: failed to get log: index=23 error="log not found" - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [ERROR] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: aborting pipeline replication: peer="{Voter server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 d64eed0d-cc3d-74e5-402b-904bfe0618d8}" - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: aborting pipeline replication: peer="{Voter server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 d64eed0d-cc3d-74e5-402b-904bfe0618d8}" -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [ERROR] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: failed to get log: index=23 error="log not found" - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [ERROR] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: aborting pipeline replication: peer="{Voter server-f28f3437-45ec-cfb0-a8bb-291c891734ad f28f3437-45ec-cfb0-a8bb-291c891734ad}" - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: aborting pipeline replication: peer="{Voter server-f28f3437-45ec-cfb0-a8bb-291c891734ad f28f3437-45ec-cfb0-a8bb-291c891734ad}" -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.864Z [ERROR] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: failed to get log: index=23 error="log not found" - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.864Z [ERROR] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: failed to get log: index=23 error="log not found" - testing.go:153: 2024-01-09T05:10:00.865Z [INFO] creating new snapshot: path=/tmp/raft395268104/snapshots/2-113-1704777000865.tmp -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.866Z [ERROR] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: failed to get log: index=23 error="log not found" - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.866Z [ERROR] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: failed to get log: index=23 error="log not found" - testing.go:153: 2024-01-09T05:10:00.867Z [INFO] creating new snapshot: path=/tmp/raft345521638/snapshots/2-113-1704777000867.tmp -testLoggerAdapter verbose: server-f28f3437-45ec-cfb0-a8bb-291c891734ad: 2024-01-09T05:10:00.884Z [INFO] server-f28f3437-45ec-cfb0-a8bb-291c891734ad: copied to local snapshot: bytes=71 - testing.go:149: server-f28f3437-45ec-cfb0-a8bb-291c891734ad: 2024-01-09T05:10:00.884Z [INFO] server-f28f3437-45ec-cfb0-a8bb-291c891734ad: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-f28f3437-45ec-cfb0-a8bb-291c891734ad: 2024-01-09T05:10:00.884Z [INFO] server-f28f3437-45ec-cfb0-a8bb-291c891734ad: Installed remote snapshot - testing.go:149: server-f28f3437-45ec-cfb0-a8bb-291c891734ad: 2024-01-09T05:10:00.884Z [INFO] server-f28f3437-45ec-cfb0-a8bb-291c891734ad: Installed remote snapshot -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.884Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: pipelining replication: peer="{Voter server-f28f3437-45ec-cfb0-a8bb-291c891734ad f28f3437-45ec-cfb0-a8bb-291c891734ad}" - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.884Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: pipelining replication: peer="{Voter server-f28f3437-45ec-cfb0-a8bb-291c891734ad f28f3437-45ec-cfb0-a8bb-291c891734ad}" -testLoggerAdapter verbose: server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: 2024-01-09T05:10:00.889Z [INFO] server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: copied to local snapshot: bytes=71 - testing.go:149: server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: 2024-01-09T05:10:00.889Z [INFO] server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: 2024-01-09T05:10:00.890Z [INFO] server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: Installed remote snapshot - testing.go:149: server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: 2024-01-09T05:10:00.890Z [INFO] server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: Installed remote snapshot -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.890Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: pipelining replication: peer="{Voter server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 d64eed0d-cc3d-74e5-402b-904bfe0618d8}" - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.890Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: pipelining replication: peer="{Voter server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 d64eed0d-cc3d-74e5-402b-904bfe0618d8}" -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.901Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: aborting pipeline replication: peer="{Voter server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 d64eed0d-cc3d-74e5-402b-904bfe0618d8}" - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.901Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: aborting pipeline replication: peer="{Voter server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 d64eed0d-cc3d-74e5-402b-904bfe0618d8}" -testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.901Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: aborting pipeline replication: peer="{Voter server-f28f3437-45ec-cfb0-a8bb-291c891734ad f28f3437-45ec-cfb0-a8bb-291c891734ad}" - testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.901Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: aborting pipeline replication: peer="{Voter server-f28f3437-45ec-cfb0-a8bb-291c891734ad f28f3437-45ec-cfb0-a8bb-291c891734ad}" -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.903Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b Address:9c29f5e2-1fc9-edd7-e97a-f9abc960667b} {Suffrage:Voter ID:server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 Address:f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7} {Suffrage:Voter ID:server-97d3eaf6-e477-f733-c692-4e3e39392f22 Address:97d3eaf6-e477-f733-c692-4e3e39392f22}]" - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.903Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b Address:9c29f5e2-1fc9-edd7-e97a-f9abc960667b} {Suffrage:Voter ID:server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 Address:f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7} {Suffrage:Voter ID:server-97d3eaf6-e477-f733-c692-4e3e39392f22 Address:97d3eaf6-e477-f733-c692-4e3e39392f22}]" -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.903Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: entering follower state: follower="Node at 9c29f5e2-1fc9-edd7-e97a-f9abc960667b [Follower]" leader= -testLoggerAdapter verbose: server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: 2024-01-09T05:10:00.903Z [INFO] server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b Address:9c29f5e2-1fc9-edd7-e97a-f9abc960667b} {Suffrage:Voter ID:server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 Address:f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7} {Suffrage:Voter ID:server-97d3eaf6-e477-f733-c692-4e3e39392f22 Address:97d3eaf6-e477-f733-c692-4e3e39392f22}]" - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.903Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: entering follower state: follower="Node at 9c29f5e2-1fc9-edd7-e97a-f9abc960667b [Follower]" leader= - testing.go:149: server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: 2024-01-09T05:10:00.903Z [INFO] server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b Address:9c29f5e2-1fc9-edd7-e97a-f9abc960667b} {Suffrage:Voter ID:server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 Address:f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7} {Suffrage:Voter ID:server-97d3eaf6-e477-f733-c692-4e3e39392f22 Address:97d3eaf6-e477-f733-c692-4e3e39392f22}]" -testLoggerAdapter verbose: server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: 2024-01-09T05:10:00.903Z [INFO] server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: entering follower state: follower="Node at f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 [Follower]" leader= - testing.go:149: server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: 2024-01-09T05:10:00.903Z [INFO] server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: entering follower state: follower="Node at f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 [Follower]" leader= -testLoggerAdapter verbose: server-97d3eaf6-e477-f733-c692-4e3e39392f22: 2024-01-09T05:10:00.903Z [INFO] server-97d3eaf6-e477-f733-c692-4e3e39392f22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b Address:9c29f5e2-1fc9-edd7-e97a-f9abc960667b} {Suffrage:Voter ID:server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 Address:f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7} {Suffrage:Voter ID:server-97d3eaf6-e477-f733-c692-4e3e39392f22 Address:97d3eaf6-e477-f733-c692-4e3e39392f22}]" - testing.go:149: server-97d3eaf6-e477-f733-c692-4e3e39392f22: 2024-01-09T05:10:00.903Z [INFO] server-97d3eaf6-e477-f733-c692-4e3e39392f22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b Address:9c29f5e2-1fc9-edd7-e97a-f9abc960667b} {Suffrage:Voter ID:server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 Address:f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7} {Suffrage:Voter ID:server-97d3eaf6-e477-f733-c692-4e3e39392f22 Address:97d3eaf6-e477-f733-c692-4e3e39392f22}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:00.903Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:00.903Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-97d3eaf6-e477-f733-c692-4e3e39392f22: 2024-01-09T05:10:00.903Z [INFO] server-97d3eaf6-e477-f733-c692-4e3e39392f22: entering follower state: follower="Node at 97d3eaf6-e477-f733-c692-4e3e39392f22 [Follower]" leader= - testing.go:149: server-97d3eaf6-e477-f733-c692-4e3e39392f22: 2024-01-09T05:10:00.903Z [INFO] server-97d3eaf6-e477-f733-c692-4e3e39392f22: entering follower state: follower="Node at 97d3eaf6-e477-f733-c692-4e3e39392f22 [Follower]" leader= -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.970Z [WARN] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.970Z [WARN] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: entering candidate state: node="Node at 9c29f5e2-1fc9-edd7-e97a-f9abc960667b [Candidate]" term=2 - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: entering candidate state: node="Node at 9c29f5e2-1fc9-edd7-e97a-f9abc960667b [Candidate]" term=2 -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: election won: tally=2 - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: election won: tally=2 -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: entering leader state: leader="Node at 9c29f5e2-1fc9-edd7-e97a-f9abc960667b [Leader]" - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: entering leader state: leader="Node at 9c29f5e2-1fc9-edd7-e97a-f9abc960667b [Leader]" -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: added peer, starting replication: peer=server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: added peer, starting replication: peer=server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: added peer, starting replication: peer=server-97d3eaf6-e477-f733-c692-4e3e39392f22 - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: added peer, starting replication: peer=server-97d3eaf6-e477-f733-c692-4e3e39392f22 -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: pipelining replication: peer="{Voter server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7}" - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: pipelining replication: peer="{Voter server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7}" -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: pipelining replication: peer="{Voter server-97d3eaf6-e477-f733-c692-4e3e39392f22 97d3eaf6-e477-f733-c692-4e3e39392f22}" - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: pipelining replication: peer="{Voter server-97d3eaf6-e477-f733-c692-4e3e39392f22 97d3eaf6-e477-f733-c692-4e3e39392f22}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.076Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.971380126 +0000 UTC m=+12.224431187 (1 nodes), 67.577002ms from start of poll, 68.185887ms from cluster start. Timeout at 2024-01-09 05:10:01.076474619 +0000 UTC m=+12.329525687, 105.0945ms after stability - testing.go:149: cluster: 2024-01-09T05:10:01.076Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.971380126 +0000 UTC m=+12.224431187 (1 nodes), 67.577002ms from start of poll, 68.185887ms from cluster start. Timeout at 2024-01-09 05:10:01.076474619 +0000 UTC m=+12.329525687, 105.0945ms after stability -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.077Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: starting snapshot up to: index=12 - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.077Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: starting snapshot up to: index=12 - testing.go:153: 2024-01-09T05:10:01.077Z [INFO] creating new snapshot: path=/tmp/raft214232186/snapshots/2-12-1704777001077.tmp -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.084Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: snapshot complete up to: index=12 - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.084Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: snapshot complete up to: index=12 - testing.go:153: 2024-01-09T05:10:01.084Z [INFO] creating new snapshot: path=/tmp/raft214232186/snapshots/2-1013-1704777001084.tmp -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.100Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: copied to local snapshot: bytes=71 - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.100Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.100Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: restored user snapshot: index=1 - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.100Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: restored user snapshot: index=1 -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.103Z [ERROR] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: failed to get log: index=23 error="log not found" - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.103Z [ERROR] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.105Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: aborting pipeline replication: peer="{Voter server-97d3eaf6-e477-f733-c692-4e3e39392f22 97d3eaf6-e477-f733-c692-4e3e39392f22}" - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.105Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: aborting pipeline replication: peer="{Voter server-97d3eaf6-e477-f733-c692-4e3e39392f22 97d3eaf6-e477-f733-c692-4e3e39392f22}" -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.103Z [ERROR] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: failed to get log: index=23 error="log not found" - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.103Z [ERROR] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.105Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: aborting pipeline replication: peer="{Voter server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7}" - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.105Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: aborting pipeline replication: peer="{Voter server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7}" -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.105Z [ERROR] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: failed to get log: index=23 error="log not found" - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.105Z [ERROR] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: failed to get log: index=23 error="log not found" - testing.go:153: 2024-01-09T05:10:01.105Z [INFO] creating new snapshot: path=/tmp/raft721165756/snapshots/2-1013-1704777001105.tmp -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.113Z [ERROR] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: failed to get log: index=23 error="log not found" - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.113Z [ERROR] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: failed to get log: index=23 error="log not found" - testing.go:153: 2024-01-09T05:10:01.113Z [INFO] creating new snapshot: path=/tmp/raft811189838/snapshots/2-1013-1704777001113.tmp -testLoggerAdapter verbose: server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: 2024-01-09T05:10:01.113Z [INFO] server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: copied to local snapshot: bytes=71 - testing.go:149: server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: 2024-01-09T05:10:01.113Z [INFO] server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: 2024-01-09T05:10:01.113Z [INFO] server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: Installed remote snapshot - testing.go:149: server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: 2024-01-09T05:10:01.113Z [INFO] server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: Installed remote snapshot -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.114Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: pipelining replication: peer="{Voter server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7}" - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.114Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: pipelining replication: peer="{Voter server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7}" -testLoggerAdapter verbose: server-97d3eaf6-e477-f733-c692-4e3e39392f22: 2024-01-09T05:10:01.123Z [INFO] server-97d3eaf6-e477-f733-c692-4e3e39392f22: copied to local snapshot: bytes=71 - testing.go:149: server-97d3eaf6-e477-f733-c692-4e3e39392f22: 2024-01-09T05:10:01.123Z [INFO] server-97d3eaf6-e477-f733-c692-4e3e39392f22: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-97d3eaf6-e477-f733-c692-4e3e39392f22: 2024-01-09T05:10:01.124Z [INFO] server-97d3eaf6-e477-f733-c692-4e3e39392f22: Installed remote snapshot - testing.go:149: server-97d3eaf6-e477-f733-c692-4e3e39392f22: 2024-01-09T05:10:01.124Z [INFO] server-97d3eaf6-e477-f733-c692-4e3e39392f22: Installed remote snapshot -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.124Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: pipelining replication: peer="{Voter server-97d3eaf6-e477-f733-c692-4e3e39392f22 97d3eaf6-e477-f733-c692-4e3e39392f22}" - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.124Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: pipelining replication: peer="{Voter server-97d3eaf6-e477-f733-c692-4e3e39392f22 97d3eaf6-e477-f733-c692-4e3e39392f22}" -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.135Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: aborting pipeline replication: peer="{Voter server-97d3eaf6-e477-f733-c692-4e3e39392f22 97d3eaf6-e477-f733-c692-4e3e39392f22}" - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.135Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: aborting pipeline replication: peer="{Voter server-97d3eaf6-e477-f733-c692-4e3e39392f22 97d3eaf6-e477-f733-c692-4e3e39392f22}" -testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.136Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: aborting pipeline replication: peer="{Voter server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7}" - testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.136Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: aborting pipeline replication: peer="{Voter server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7}" -testLoggerAdapter verbose: server-98380281-4484-82ac-6934-e75f6883688f: 2024-01-09T05:10:01.140Z [INFO] server-98380281-4484-82ac-6934-e75f6883688f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98380281-4484-82ac-6934-e75f6883688f Address:98380281-4484-82ac-6934-e75f6883688f} {Suffrage:Voter ID:server-9b6d9996-37d1-7f97-639b-f943fafa4261 Address:9b6d9996-37d1-7f97-639b-f943fafa4261} {Suffrage:Voter ID:server-591947dd-710c-7eb7-8b46-22a7944f4442 Address:591947dd-710c-7eb7-8b46-22a7944f4442}]" - testing.go:149: server-98380281-4484-82ac-6934-e75f6883688f: 2024-01-09T05:10:01.140Z [INFO] server-98380281-4484-82ac-6934-e75f6883688f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98380281-4484-82ac-6934-e75f6883688f Address:98380281-4484-82ac-6934-e75f6883688f} {Suffrage:Voter ID:server-9b6d9996-37d1-7f97-639b-f943fafa4261 Address:9b6d9996-37d1-7f97-639b-f943fafa4261} {Suffrage:Voter ID:server-591947dd-710c-7eb7-8b46-22a7944f4442 Address:591947dd-710c-7eb7-8b46-22a7944f4442}]" -testLoggerAdapter verbose: server-98380281-4484-82ac-6934-e75f6883688f: 2024-01-09T05:10:01.140Z [INFO] server-98380281-4484-82ac-6934-e75f6883688f: entering follower state: follower="Node at 98380281-4484-82ac-6934-e75f6883688f [Follower]" leader= - testing.go:149: server-98380281-4484-82ac-6934-e75f6883688f: 2024-01-09T05:10:01.140Z [INFO] server-98380281-4484-82ac-6934-e75f6883688f: entering follower state: follower="Node at 98380281-4484-82ac-6934-e75f6883688f [Follower]" leader= -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.141Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98380281-4484-82ac-6934-e75f6883688f Address:98380281-4484-82ac-6934-e75f6883688f} {Suffrage:Voter ID:server-9b6d9996-37d1-7f97-639b-f943fafa4261 Address:9b6d9996-37d1-7f97-639b-f943fafa4261} {Suffrage:Voter ID:server-591947dd-710c-7eb7-8b46-22a7944f4442 Address:591947dd-710c-7eb7-8b46-22a7944f4442}]" - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.141Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98380281-4484-82ac-6934-e75f6883688f Address:98380281-4484-82ac-6934-e75f6883688f} {Suffrage:Voter ID:server-9b6d9996-37d1-7f97-639b-f943fafa4261 Address:9b6d9996-37d1-7f97-639b-f943fafa4261} {Suffrage:Voter ID:server-591947dd-710c-7eb7-8b46-22a7944f4442 Address:591947dd-710c-7eb7-8b46-22a7944f4442}]" -testLoggerAdapter verbose: server-591947dd-710c-7eb7-8b46-22a7944f4442: 2024-01-09T05:10:01.141Z [INFO] server-591947dd-710c-7eb7-8b46-22a7944f4442: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98380281-4484-82ac-6934-e75f6883688f Address:98380281-4484-82ac-6934-e75f6883688f} {Suffrage:Voter ID:server-9b6d9996-37d1-7f97-639b-f943fafa4261 Address:9b6d9996-37d1-7f97-639b-f943fafa4261} {Suffrage:Voter ID:server-591947dd-710c-7eb7-8b46-22a7944f4442 Address:591947dd-710c-7eb7-8b46-22a7944f4442}]" - testing.go:149: server-591947dd-710c-7eb7-8b46-22a7944f4442: 2024-01-09T05:10:01.141Z [INFO] server-591947dd-710c-7eb7-8b46-22a7944f4442: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98380281-4484-82ac-6934-e75f6883688f Address:98380281-4484-82ac-6934-e75f6883688f} {Suffrage:Voter ID:server-9b6d9996-37d1-7f97-639b-f943fafa4261 Address:9b6d9996-37d1-7f97-639b-f943fafa4261} {Suffrage:Voter ID:server-591947dd-710c-7eb7-8b46-22a7944f4442 Address:591947dd-710c-7eb7-8b46-22a7944f4442}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.141Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:01.141Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.141Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: entering follower state: follower="Node at 9b6d9996-37d1-7f97-639b-f943fafa4261 [Follower]" leader= - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.141Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: entering follower state: follower="Node at 9b6d9996-37d1-7f97-639b-f943fafa4261 [Follower]" leader= -testLoggerAdapter verbose: server-591947dd-710c-7eb7-8b46-22a7944f4442: 2024-01-09T05:10:01.141Z [INFO] server-591947dd-710c-7eb7-8b46-22a7944f4442: entering follower state: follower="Node at 591947dd-710c-7eb7-8b46-22a7944f4442 [Follower]" leader= - testing.go:149: server-591947dd-710c-7eb7-8b46-22a7944f4442: 2024-01-09T05:10:01.141Z [INFO] server-591947dd-710c-7eb7-8b46-22a7944f4442: entering follower state: follower="Node at 591947dd-710c-7eb7-8b46-22a7944f4442 [Follower]" leader= -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.194Z [WARN] server-9b6d9996-37d1-7f97-639b-f943fafa4261: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.194Z [WARN] server-9b6d9996-37d1-7f97-639b-f943fafa4261: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: entering candidate state: node="Node at 9b6d9996-37d1-7f97-639b-f943fafa4261 [Candidate]" term=2 - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: entering candidate state: node="Node at 9b6d9996-37d1-7f97-639b-f943fafa4261 [Candidate]" term=2 -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: election won: tally=2 - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: election won: tally=2 -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: entering leader state: leader="Node at 9b6d9996-37d1-7f97-639b-f943fafa4261 [Leader]" - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: entering leader state: leader="Node at 9b6d9996-37d1-7f97-639b-f943fafa4261 [Leader]" -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: added peer, starting replication: peer=server-98380281-4484-82ac-6934-e75f6883688f - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: added peer, starting replication: peer=server-98380281-4484-82ac-6934-e75f6883688f -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: added peer, starting replication: peer=server-591947dd-710c-7eb7-8b46-22a7944f4442 - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: added peer, starting replication: peer=server-591947dd-710c-7eb7-8b46-22a7944f4442 -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: pipelining replication: peer="{Voter server-98380281-4484-82ac-6934-e75f6883688f 98380281-4484-82ac-6934-e75f6883688f}" - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: pipelining replication: peer="{Voter server-98380281-4484-82ac-6934-e75f6883688f 98380281-4484-82ac-6934-e75f6883688f}" -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: pipelining replication: peer="{Voter server-591947dd-710c-7eb7-8b46-22a7944f4442 591947dd-710c-7eb7-8b46-22a7944f4442}" - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: pipelining replication: peer="{Voter server-591947dd-710c-7eb7-8b46-22a7944f4442 591947dd-710c-7eb7-8b46-22a7944f4442}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.301Z [INFO] stable state for Leader reached at 2024-01-09 05:10:01.195422288 +0000 UTC m=+12.448473426 (1 nodes), 53.819278ms from start of poll, 55.104715ms from cluster start. Timeout at 2024-01-09 05:10:01.30112509 +0000 UTC m=+12.554176091, 105.702665ms after stability - testing.go:149: cluster: 2024-01-09T05:10:01.301Z [INFO] stable state for Leader reached at 2024-01-09 05:10:01.195422288 +0000 UTC m=+12.448473426 (1 nodes), 53.819278ms from start of poll, 55.104715ms from cluster start. Timeout at 2024-01-09 05:10:01.30112509 +0000 UTC m=+12.554176091, 105.702665ms after stability -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.301Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: starting snapshot up to: index=12 - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.301Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: starting snapshot up to: index=12 - testing.go:153: 2024-01-09T05:10:01.301Z [INFO] creating new snapshot: path=/tmp/raft518721890/snapshots/2-12-1704777001301.tmp -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.307Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: snapshot complete up to: index=12 - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.307Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: snapshot complete up to: index=12 - testing.go:153: 2024-01-09T05:10:01.309Z [INFO] creating new snapshot: path=/tmp/raft518721890/snapshots/2-10013-1704777001309.tmp -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.323Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: copied to local snapshot: bytes=71 - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.323Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: restored user snapshot: index=1 - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: restored user snapshot: index=1 -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [ERROR] server-9b6d9996-37d1-7f97-639b-f943fafa4261: failed to get log: index=23 error="log not found" - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [ERROR] server-9b6d9996-37d1-7f97-639b-f943fafa4261: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: aborting pipeline replication: peer="{Voter server-591947dd-710c-7eb7-8b46-22a7944f4442 591947dd-710c-7eb7-8b46-22a7944f4442}" - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: aborting pipeline replication: peer="{Voter server-591947dd-710c-7eb7-8b46-22a7944f4442 591947dd-710c-7eb7-8b46-22a7944f4442}" -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [ERROR] server-9b6d9996-37d1-7f97-639b-f943fafa4261: failed to get log: index=23 error="log not found" - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [ERROR] server-9b6d9996-37d1-7f97-639b-f943fafa4261: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: aborting pipeline replication: peer="{Voter server-98380281-4484-82ac-6934-e75f6883688f 98380281-4484-82ac-6934-e75f6883688f}" - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: aborting pipeline replication: peer="{Voter server-98380281-4484-82ac-6934-e75f6883688f 98380281-4484-82ac-6934-e75f6883688f}" -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.332Z [ERROR] server-9b6d9996-37d1-7f97-639b-f943fafa4261: failed to get log: index=23 error="log not found" - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.332Z [ERROR] server-9b6d9996-37d1-7f97-639b-f943fafa4261: failed to get log: index=23 error="log not found" - testing.go:153: 2024-01-09T05:10:01.332Z [INFO] creating new snapshot: path=/tmp/raft025824484/snapshots/2-10013-1704777001332.tmp -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.333Z [ERROR] server-9b6d9996-37d1-7f97-639b-f943fafa4261: failed to get log: index=23 error="log not found" - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.333Z [ERROR] server-9b6d9996-37d1-7f97-639b-f943fafa4261: failed to get log: index=23 error="log not found" - testing.go:153: 2024-01-09T05:10:01.334Z [INFO] creating new snapshot: path=/tmp/raft745983664/snapshots/2-10013-1704777001334.tmp -testLoggerAdapter verbose: server-98380281-4484-82ac-6934-e75f6883688f: 2024-01-09T05:10:01.353Z [INFO] server-98380281-4484-82ac-6934-e75f6883688f: copied to local snapshot: bytes=71 - testing.go:149: server-98380281-4484-82ac-6934-e75f6883688f: 2024-01-09T05:10:01.353Z [INFO] server-98380281-4484-82ac-6934-e75f6883688f: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-591947dd-710c-7eb7-8b46-22a7944f4442: 2024-01-09T05:10:01.354Z [INFO] server-591947dd-710c-7eb7-8b46-22a7944f4442: copied to local snapshot: bytes=71 - testing.go:149: server-591947dd-710c-7eb7-8b46-22a7944f4442: 2024-01-09T05:10:01.354Z [INFO] server-591947dd-710c-7eb7-8b46-22a7944f4442: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-591947dd-710c-7eb7-8b46-22a7944f4442: 2024-01-09T05:10:01.354Z [INFO] server-591947dd-710c-7eb7-8b46-22a7944f4442: Installed remote snapshot - testing.go:149: server-591947dd-710c-7eb7-8b46-22a7944f4442: 2024-01-09T05:10:01.354Z [INFO] server-591947dd-710c-7eb7-8b46-22a7944f4442: Installed remote snapshot -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.354Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: pipelining replication: peer="{Voter server-591947dd-710c-7eb7-8b46-22a7944f4442 591947dd-710c-7eb7-8b46-22a7944f4442}" - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.354Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: pipelining replication: peer="{Voter server-591947dd-710c-7eb7-8b46-22a7944f4442 591947dd-710c-7eb7-8b46-22a7944f4442}" -testLoggerAdapter verbose: server-98380281-4484-82ac-6934-e75f6883688f: 2024-01-09T05:10:01.355Z [INFO] server-98380281-4484-82ac-6934-e75f6883688f: Installed remote snapshot - testing.go:149: server-98380281-4484-82ac-6934-e75f6883688f: 2024-01-09T05:10:01.355Z [INFO] server-98380281-4484-82ac-6934-e75f6883688f: Installed remote snapshot -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.355Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: pipelining replication: peer="{Voter server-98380281-4484-82ac-6934-e75f6883688f 98380281-4484-82ac-6934-e75f6883688f}" - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.355Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: pipelining replication: peer="{Voter server-98380281-4484-82ac-6934-e75f6883688f 98380281-4484-82ac-6934-e75f6883688f}" -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.371Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: aborting pipeline replication: peer="{Voter server-591947dd-710c-7eb7-8b46-22a7944f4442 591947dd-710c-7eb7-8b46-22a7944f4442}" - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.371Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: aborting pipeline replication: peer="{Voter server-591947dd-710c-7eb7-8b46-22a7944f4442 591947dd-710c-7eb7-8b46-22a7944f4442}" -testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.371Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: aborting pipeline replication: peer="{Voter server-98380281-4484-82ac-6934-e75f6883688f 98380281-4484-82ac-6934-e75f6883688f}" - testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.371Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: aborting pipeline replication: peer="{Voter server-98380281-4484-82ac-6934-e75f6883688f 98380281-4484-82ac-6934-e75f6883688f}" ---- PASS: TestRaft_UserRestore (1.42s) +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.312Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e39bd5e-9334-fe43-6354-f78b50d41f05 Address:8e39bd5e-9334-fe43-6354-f78b50d41f05} {Suffrage:Voter ID:server-d8248c37-3322-d35d-4ed6-2d9614664faf Address:d8248c37-3322-d35d-4ed6-2d9614664faf} {Suffrage:Voter ID:server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb Address:b76869c8-0eeb-cb7f-c033-48ec73fbfeeb}]" + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.312Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e39bd5e-9334-fe43-6354-f78b50d41f05 Address:8e39bd5e-9334-fe43-6354-f78b50d41f05} {Suffrage:Voter ID:server-d8248c37-3322-d35d-4ed6-2d9614664faf Address:d8248c37-3322-d35d-4ed6-2d9614664faf} {Suffrage:Voter ID:server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb Address:b76869c8-0eeb-cb7f-c033-48ec73fbfeeb}]" +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.312Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: entering follower state: follower="Node at 8e39bd5e-9334-fe43-6354-f78b50d41f05 [Follower]" leader= + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.312Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: entering follower state: follower="Node at 8e39bd5e-9334-fe43-6354-f78b50d41f05 [Follower]" leader= +testLoggerAdapter verbose: server-d8248c37-3322-d35d-4ed6-2d9614664faf: 2025-02-10T11:34:14.313Z [INFO] server-d8248c37-3322-d35d-4ed6-2d9614664faf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e39bd5e-9334-fe43-6354-f78b50d41f05 Address:8e39bd5e-9334-fe43-6354-f78b50d41f05} {Suffrage:Voter ID:server-d8248c37-3322-d35d-4ed6-2d9614664faf Address:d8248c37-3322-d35d-4ed6-2d9614664faf} {Suffrage:Voter ID:server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb Address:b76869c8-0eeb-cb7f-c033-48ec73fbfeeb}]" + testing.go:149: server-d8248c37-3322-d35d-4ed6-2d9614664faf: 2025-02-10T11:34:14.313Z [INFO] server-d8248c37-3322-d35d-4ed6-2d9614664faf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e39bd5e-9334-fe43-6354-f78b50d41f05 Address:8e39bd5e-9334-fe43-6354-f78b50d41f05} {Suffrage:Voter ID:server-d8248c37-3322-d35d-4ed6-2d9614664faf Address:d8248c37-3322-d35d-4ed6-2d9614664faf} {Suffrage:Voter ID:server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb Address:b76869c8-0eeb-cb7f-c033-48ec73fbfeeb}]" +testLoggerAdapter verbose: server-d8248c37-3322-d35d-4ed6-2d9614664faf: 2025-02-10T11:34:14.313Z [INFO] server-d8248c37-3322-d35d-4ed6-2d9614664faf: entering follower state: follower="Node at d8248c37-3322-d35d-4ed6-2d9614664faf [Follower]" leader= + testing.go:149: server-d8248c37-3322-d35d-4ed6-2d9614664faf: 2025-02-10T11:34:14.313Z [INFO] server-d8248c37-3322-d35d-4ed6-2d9614664faf: entering follower state: follower="Node at d8248c37-3322-d35d-4ed6-2d9614664faf [Follower]" leader= +testLoggerAdapter verbose: server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb: 2025-02-10T11:34:14.313Z [INFO] server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e39bd5e-9334-fe43-6354-f78b50d41f05 Address:8e39bd5e-9334-fe43-6354-f78b50d41f05} {Suffrage:Voter ID:server-d8248c37-3322-d35d-4ed6-2d9614664faf Address:d8248c37-3322-d35d-4ed6-2d9614664faf} {Suffrage:Voter ID:server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb Address:b76869c8-0eeb-cb7f-c033-48ec73fbfeeb}]" + testing.go:149: server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb: 2025-02-10T11:34:14.313Z [INFO] server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e39bd5e-9334-fe43-6354-f78b50d41f05 Address:8e39bd5e-9334-fe43-6354-f78b50d41f05} {Suffrage:Voter ID:server-d8248c37-3322-d35d-4ed6-2d9614664faf Address:d8248c37-3322-d35d-4ed6-2d9614664faf} {Suffrage:Voter ID:server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb Address:b76869c8-0eeb-cb7f-c033-48ec73fbfeeb}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:14.313Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:14.313Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb: 2025-02-10T11:34:14.313Z [INFO] server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb: entering follower state: follower="Node at b76869c8-0eeb-cb7f-c033-48ec73fbfeeb [Follower]" leader= + testing.go:149: server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb: 2025-02-10T11:34:14.313Z [INFO] server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb: entering follower state: follower="Node at b76869c8-0eeb-cb7f-c033-48ec73fbfeeb [Follower]" leader= +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.375Z [WARN] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.375Z [WARN] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.375Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: entering candidate state: node="Node at 8e39bd5e-9334-fe43-6354-f78b50d41f05 [Candidate]" term=2 + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.375Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: entering candidate state: node="Node at 8e39bd5e-9334-fe43-6354-f78b50d41f05 [Candidate]" term=2 +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.375Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: election won: tally=2 + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.375Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: election won: tally=2 +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.375Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: entering leader state: leader="Node at 8e39bd5e-9334-fe43-6354-f78b50d41f05 [Leader]" + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.375Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: entering leader state: leader="Node at 8e39bd5e-9334-fe43-6354-f78b50d41f05 [Leader]" +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.375Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: added peer, starting replication: peer=server-d8248c37-3322-d35d-4ed6-2d9614664faf + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.375Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: added peer, starting replication: peer=server-d8248c37-3322-d35d-4ed6-2d9614664faf +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.375Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: added peer, starting replication: peer=server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.375Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: added peer, starting replication: peer=server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.375Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: pipelining replication: peer="{Voter server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb b76869c8-0eeb-cb7f-c033-48ec73fbfeeb}" + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.375Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: pipelining replication: peer="{Voter server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb b76869c8-0eeb-cb7f-c033-48ec73fbfeeb}" +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.375Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: pipelining replication: peer="{Voter server-d8248c37-3322-d35d-4ed6-2d9614664faf d8248c37-3322-d35d-4ed6-2d9614664faf}" + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.375Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: pipelining replication: peer="{Voter server-d8248c37-3322-d35d-4ed6-2d9614664faf d8248c37-3322-d35d-4ed6-2d9614664faf}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:14.480Z [INFO] stable state for Leader reached at 2025-02-10 11:34:14.375469175 +0000 UTC m=+10.744373476 (1 nodes), 62.252329ms from start of poll, 62.646306ms from cluster start. Timeout at 2025-02-10 11:34:14.480541566 +0000 UTC m=+10.849445886, 105.07241ms after stability + testing.go:149: cluster: 2025-02-10T11:34:14.480Z [INFO] stable state for Leader reached at 2025-02-10 11:34:14.375469175 +0000 UTC m=+10.744373476 (1 nodes), 62.252329ms from start of poll, 62.646306ms from cluster start. Timeout at 2025-02-10 11:34:14.480541566 +0000 UTC m=+10.849445886, 105.07241ms after stability +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.480Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: starting snapshot up to: index=12 + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.480Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: starting snapshot up to: index=12 + testing.go:153: 2025-02-10T11:34:14.480Z [INFO] creating new snapshot: path=/tmp/raft529662243/snapshots/2-12-1739187254480.tmp +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.484Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: snapshot complete up to: index=12 + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.484Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: snapshot complete up to: index=12 + testing.go:153: 2025-02-10T11:34:14.484Z [INFO] creating new snapshot: path=/tmp/raft529662243/snapshots/2-23-1739187254484.tmp +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.487Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: copied to local snapshot: bytes=71 + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.487Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.488Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: restored user snapshot: index=1 + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.488Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: restored user snapshot: index=1 +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.488Z [ERROR] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: failed to get log: index=23 error="log not found" + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.488Z [ERROR] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.488Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: aborting pipeline replication: peer="{Voter server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb b76869c8-0eeb-cb7f-c033-48ec73fbfeeb}" + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.488Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: aborting pipeline replication: peer="{Voter server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb b76869c8-0eeb-cb7f-c033-48ec73fbfeeb}" +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.488Z [ERROR] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: failed to get log: index=23 error="log not found" + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.488Z [ERROR] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.488Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: aborting pipeline replication: peer="{Voter server-d8248c37-3322-d35d-4ed6-2d9614664faf d8248c37-3322-d35d-4ed6-2d9614664faf}" + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.488Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: aborting pipeline replication: peer="{Voter server-d8248c37-3322-d35d-4ed6-2d9614664faf d8248c37-3322-d35d-4ed6-2d9614664faf}" +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.496Z [ERROR] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: failed to get log: index=23 error="log not found" + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.496Z [ERROR] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: failed to get log: index=23 error="log not found" + testing.go:153: 2025-02-10T11:34:14.496Z [INFO] creating new snapshot: path=/tmp/raft031717197/snapshots/2-23-1739187254496.tmp +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.497Z [ERROR] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: failed to get log: index=23 error="log not found" + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.497Z [ERROR] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: failed to get log: index=23 error="log not found" + testing.go:153: 2025-02-10T11:34:14.497Z [INFO] creating new snapshot: path=/tmp/raft407137543/snapshots/2-23-1739187254497.tmp +testLoggerAdapter verbose: server-d8248c37-3322-d35d-4ed6-2d9614664faf: 2025-02-10T11:34:14.501Z [INFO] server-d8248c37-3322-d35d-4ed6-2d9614664faf: copied to local snapshot: bytes=71 + testing.go:149: server-d8248c37-3322-d35d-4ed6-2d9614664faf: 2025-02-10T11:34:14.501Z [INFO] server-d8248c37-3322-d35d-4ed6-2d9614664faf: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-d8248c37-3322-d35d-4ed6-2d9614664faf: 2025-02-10T11:34:14.501Z [INFO] server-d8248c37-3322-d35d-4ed6-2d9614664faf: Installed remote snapshot + testing.go:149: server-d8248c37-3322-d35d-4ed6-2d9614664faf: 2025-02-10T11:34:14.501Z [INFO] server-d8248c37-3322-d35d-4ed6-2d9614664faf: Installed remote snapshot +testLoggerAdapter verbose: server-d8248c37-3322-d35d-4ed6-2d9614664faf: 2025-02-10T11:34:14.501Z [WARN] server-d8248c37-3322-d35d-4ed6-2d9614664faf: skipping application of old log: index=22 + testing.go:149: server-d8248c37-3322-d35d-4ed6-2d9614664faf: 2025-02-10T11:34:14.501Z [WARN] server-d8248c37-3322-d35d-4ed6-2d9614664faf: skipping application of old log: index=22 +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.501Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: pipelining replication: peer="{Voter server-d8248c37-3322-d35d-4ed6-2d9614664faf d8248c37-3322-d35d-4ed6-2d9614664faf}" + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.501Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: pipelining replication: peer="{Voter server-d8248c37-3322-d35d-4ed6-2d9614664faf d8248c37-3322-d35d-4ed6-2d9614664faf}" +testLoggerAdapter verbose: server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb: 2025-02-10T11:34:14.501Z [INFO] server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb: copied to local snapshot: bytes=71 + testing.go:149: server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb: 2025-02-10T11:34:14.501Z [INFO] server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb: 2025-02-10T11:34:14.501Z [INFO] server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb: Installed remote snapshot + testing.go:149: server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb: 2025-02-10T11:34:14.501Z [INFO] server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb: Installed remote snapshot +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.501Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: pipelining replication: peer="{Voter server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb b76869c8-0eeb-cb7f-c033-48ec73fbfeeb}" + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.501Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: pipelining replication: peer="{Voter server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb b76869c8-0eeb-cb7f-c033-48ec73fbfeeb}" +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.516Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: aborting pipeline replication: peer="{Voter server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb b76869c8-0eeb-cb7f-c033-48ec73fbfeeb}" + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.516Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: aborting pipeline replication: peer="{Voter server-b76869c8-0eeb-cb7f-c033-48ec73fbfeeb b76869c8-0eeb-cb7f-c033-48ec73fbfeeb}" +testLoggerAdapter verbose: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.516Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: aborting pipeline replication: peer="{Voter server-d8248c37-3322-d35d-4ed6-2d9614664faf d8248c37-3322-d35d-4ed6-2d9614664faf}" + testing.go:149: server-8e39bd5e-9334-fe43-6354-f78b50d41f05: 2025-02-10T11:34:14.516Z [INFO] server-8e39bd5e-9334-fe43-6354-f78b50d41f05: aborting pipeline replication: peer="{Voter server-d8248c37-3322-d35d-4ed6-2d9614664faf d8248c37-3322-d35d-4ed6-2d9614664faf}" +testLoggerAdapter verbose: server-77142aee-2979-634e-7a43-baea0fa60b72: 2025-02-10T11:34:14.518Z [INFO] server-77142aee-2979-634e-7a43-baea0fa60b72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77142aee-2979-634e-7a43-baea0fa60b72 Address:77142aee-2979-634e-7a43-baea0fa60b72} {Suffrage:Voter ID:server-cc1ddb75-ee67-a616-1d79-141e839e4263 Address:cc1ddb75-ee67-a616-1d79-141e839e4263} {Suffrage:Voter ID:server-7073b9b5-b424-61bd-522e-811dbc4994b5 Address:7073b9b5-b424-61bd-522e-811dbc4994b5}]" + testing.go:149: server-77142aee-2979-634e-7a43-baea0fa60b72: 2025-02-10T11:34:14.518Z [INFO] server-77142aee-2979-634e-7a43-baea0fa60b72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77142aee-2979-634e-7a43-baea0fa60b72 Address:77142aee-2979-634e-7a43-baea0fa60b72} {Suffrage:Voter ID:server-cc1ddb75-ee67-a616-1d79-141e839e4263 Address:cc1ddb75-ee67-a616-1d79-141e839e4263} {Suffrage:Voter ID:server-7073b9b5-b424-61bd-522e-811dbc4994b5 Address:7073b9b5-b424-61bd-522e-811dbc4994b5}]" +testLoggerAdapter verbose: server-77142aee-2979-634e-7a43-baea0fa60b72: 2025-02-10T11:34:14.518Z [INFO] server-77142aee-2979-634e-7a43-baea0fa60b72: entering follower state: follower="Node at 77142aee-2979-634e-7a43-baea0fa60b72 [Follower]" leader= + testing.go:149: server-77142aee-2979-634e-7a43-baea0fa60b72: 2025-02-10T11:34:14.518Z [INFO] server-77142aee-2979-634e-7a43-baea0fa60b72: entering follower state: follower="Node at 77142aee-2979-634e-7a43-baea0fa60b72 [Follower]" leader= +testLoggerAdapter verbose: server-cc1ddb75-ee67-a616-1d79-141e839e4263: 2025-02-10T11:34:14.518Z [INFO] server-cc1ddb75-ee67-a616-1d79-141e839e4263: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77142aee-2979-634e-7a43-baea0fa60b72 Address:77142aee-2979-634e-7a43-baea0fa60b72} {Suffrage:Voter ID:server-cc1ddb75-ee67-a616-1d79-141e839e4263 Address:cc1ddb75-ee67-a616-1d79-141e839e4263} {Suffrage:Voter ID:server-7073b9b5-b424-61bd-522e-811dbc4994b5 Address:7073b9b5-b424-61bd-522e-811dbc4994b5}]" + testing.go:149: server-cc1ddb75-ee67-a616-1d79-141e839e4263: 2025-02-10T11:34:14.518Z [INFO] server-cc1ddb75-ee67-a616-1d79-141e839e4263: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77142aee-2979-634e-7a43-baea0fa60b72 Address:77142aee-2979-634e-7a43-baea0fa60b72} {Suffrage:Voter ID:server-cc1ddb75-ee67-a616-1d79-141e839e4263 Address:cc1ddb75-ee67-a616-1d79-141e839e4263} {Suffrage:Voter ID:server-7073b9b5-b424-61bd-522e-811dbc4994b5 Address:7073b9b5-b424-61bd-522e-811dbc4994b5}]" +testLoggerAdapter verbose: server-cc1ddb75-ee67-a616-1d79-141e839e4263: 2025-02-10T11:34:14.518Z [INFO] server-cc1ddb75-ee67-a616-1d79-141e839e4263: entering follower state: follower="Node at cc1ddb75-ee67-a616-1d79-141e839e4263 [Follower]" leader= + testing.go:149: server-cc1ddb75-ee67-a616-1d79-141e839e4263: 2025-02-10T11:34:14.518Z [INFO] server-cc1ddb75-ee67-a616-1d79-141e839e4263: entering follower state: follower="Node at cc1ddb75-ee67-a616-1d79-141e839e4263 [Follower]" leader= +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.518Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77142aee-2979-634e-7a43-baea0fa60b72 Address:77142aee-2979-634e-7a43-baea0fa60b72} {Suffrage:Voter ID:server-cc1ddb75-ee67-a616-1d79-141e839e4263 Address:cc1ddb75-ee67-a616-1d79-141e839e4263} {Suffrage:Voter ID:server-7073b9b5-b424-61bd-522e-811dbc4994b5 Address:7073b9b5-b424-61bd-522e-811dbc4994b5}]" + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.518Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77142aee-2979-634e-7a43-baea0fa60b72 Address:77142aee-2979-634e-7a43-baea0fa60b72} {Suffrage:Voter ID:server-cc1ddb75-ee67-a616-1d79-141e839e4263 Address:cc1ddb75-ee67-a616-1d79-141e839e4263} {Suffrage:Voter ID:server-7073b9b5-b424-61bd-522e-811dbc4994b5 Address:7073b9b5-b424-61bd-522e-811dbc4994b5}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:14.518Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:14.518Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.518Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: entering follower state: follower="Node at 7073b9b5-b424-61bd-522e-811dbc4994b5 [Follower]" leader= + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.518Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: entering follower state: follower="Node at 7073b9b5-b424-61bd-522e-811dbc4994b5 [Follower]" leader= +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.576Z [WARN] server-7073b9b5-b424-61bd-522e-811dbc4994b5: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.576Z [WARN] server-7073b9b5-b424-61bd-522e-811dbc4994b5: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.576Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: entering candidate state: node="Node at 7073b9b5-b424-61bd-522e-811dbc4994b5 [Candidate]" term=2 + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.576Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: entering candidate state: node="Node at 7073b9b5-b424-61bd-522e-811dbc4994b5 [Candidate]" term=2 +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.576Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: election won: tally=2 + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.576Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: election won: tally=2 +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.576Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: entering leader state: leader="Node at 7073b9b5-b424-61bd-522e-811dbc4994b5 [Leader]" + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.576Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: entering leader state: leader="Node at 7073b9b5-b424-61bd-522e-811dbc4994b5 [Leader]" +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.576Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: added peer, starting replication: peer=server-77142aee-2979-634e-7a43-baea0fa60b72 + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.576Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: added peer, starting replication: peer=server-77142aee-2979-634e-7a43-baea0fa60b72 +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.576Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: added peer, starting replication: peer=server-cc1ddb75-ee67-a616-1d79-141e839e4263 + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.576Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: added peer, starting replication: peer=server-cc1ddb75-ee67-a616-1d79-141e839e4263 +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.576Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: pipelining replication: peer="{Voter server-cc1ddb75-ee67-a616-1d79-141e839e4263 cc1ddb75-ee67-a616-1d79-141e839e4263}" + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.576Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: pipelining replication: peer="{Voter server-cc1ddb75-ee67-a616-1d79-141e839e4263 cc1ddb75-ee67-a616-1d79-141e839e4263}" +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.576Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: pipelining replication: peer="{Voter server-77142aee-2979-634e-7a43-baea0fa60b72 77142aee-2979-634e-7a43-baea0fa60b72}" + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.576Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: pipelining replication: peer="{Voter server-77142aee-2979-634e-7a43-baea0fa60b72 77142aee-2979-634e-7a43-baea0fa60b72}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:14.681Z [INFO] stable state for Leader reached at 2025-02-10 11:34:14.576774433 +0000 UTC m=+10.945678732 (1 nodes), 58.284644ms from start of poll, 58.664008ms from cluster start. Timeout at 2025-02-10 11:34:14.681846899 +0000 UTC m=+11.050751195, 105.072463ms after stability + testing.go:149: cluster: 2025-02-10T11:34:14.681Z [INFO] stable state for Leader reached at 2025-02-10 11:34:14.576774433 +0000 UTC m=+10.945678732 (1 nodes), 58.284644ms from start of poll, 58.664008ms from cluster start. Timeout at 2025-02-10 11:34:14.681846899 +0000 UTC m=+11.050751195, 105.072463ms after stability +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.682Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: starting snapshot up to: index=12 + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.682Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: starting snapshot up to: index=12 + testing.go:153: 2025-02-10T11:34:14.682Z [INFO] creating new snapshot: path=/tmp/raft777606293/snapshots/2-12-1739187254682.tmp +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.686Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: snapshot complete up to: index=12 + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.686Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: snapshot complete up to: index=12 + testing.go:153: 2025-02-10T11:34:14.686Z [INFO] creating new snapshot: path=/tmp/raft777606293/snapshots/2-23-1739187254686.tmp +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.689Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: copied to local snapshot: bytes=71 + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.689Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.690Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: restored user snapshot: index=1 + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.690Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: restored user snapshot: index=1 +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.690Z [ERROR] server-7073b9b5-b424-61bd-522e-811dbc4994b5: failed to get log: index=23 error="log not found" + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.690Z [ERROR] server-7073b9b5-b424-61bd-522e-811dbc4994b5: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.690Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: aborting pipeline replication: peer="{Voter server-cc1ddb75-ee67-a616-1d79-141e839e4263 cc1ddb75-ee67-a616-1d79-141e839e4263}" + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.690Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: aborting pipeline replication: peer="{Voter server-cc1ddb75-ee67-a616-1d79-141e839e4263 cc1ddb75-ee67-a616-1d79-141e839e4263}" +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.690Z [ERROR] server-7073b9b5-b424-61bd-522e-811dbc4994b5: failed to get log: index=23 error="log not found" + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.690Z [ERROR] server-7073b9b5-b424-61bd-522e-811dbc4994b5: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.690Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: aborting pipeline replication: peer="{Voter server-77142aee-2979-634e-7a43-baea0fa60b72 77142aee-2979-634e-7a43-baea0fa60b72}" + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.690Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: aborting pipeline replication: peer="{Voter server-77142aee-2979-634e-7a43-baea0fa60b72 77142aee-2979-634e-7a43-baea0fa60b72}" +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.696Z [ERROR] server-7073b9b5-b424-61bd-522e-811dbc4994b5: failed to get log: index=23 error="log not found" + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.696Z [ERROR] server-7073b9b5-b424-61bd-522e-811dbc4994b5: failed to get log: index=23 error="log not found" + testing.go:153: 2025-02-10T11:34:14.696Z [INFO] creating new snapshot: path=/tmp/raft683169489/snapshots/2-23-1739187254696.tmp +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.697Z [ERROR] server-7073b9b5-b424-61bd-522e-811dbc4994b5: failed to get log: index=23 error="log not found" + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.697Z [ERROR] server-7073b9b5-b424-61bd-522e-811dbc4994b5: failed to get log: index=23 error="log not found" + testing.go:153: 2025-02-10T11:34:14.697Z [INFO] creating new snapshot: path=/tmp/raft856493867/snapshots/2-23-1739187254697.tmp +testLoggerAdapter verbose: server-77142aee-2979-634e-7a43-baea0fa60b72: 2025-02-10T11:34:14.700Z [INFO] server-77142aee-2979-634e-7a43-baea0fa60b72: copied to local snapshot: bytes=71 + testing.go:149: server-77142aee-2979-634e-7a43-baea0fa60b72: 2025-02-10T11:34:14.700Z [INFO] server-77142aee-2979-634e-7a43-baea0fa60b72: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-77142aee-2979-634e-7a43-baea0fa60b72: 2025-02-10T11:34:14.700Z [INFO] server-77142aee-2979-634e-7a43-baea0fa60b72: Installed remote snapshot + testing.go:149: server-77142aee-2979-634e-7a43-baea0fa60b72: 2025-02-10T11:34:14.700Z [INFO] server-77142aee-2979-634e-7a43-baea0fa60b72: Installed remote snapshot +testLoggerAdapter verbose: server-77142aee-2979-634e-7a43-baea0fa60b72: 2025-02-10T11:34:14.700Z [WARN] server-77142aee-2979-634e-7a43-baea0fa60b72: skipping application of old log: index=22 + testing.go:149: server-77142aee-2979-634e-7a43-baea0fa60b72: 2025-02-10T11:34:14.700Z [WARN] server-77142aee-2979-634e-7a43-baea0fa60b72: skipping application of old log: index=22 +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.700Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: pipelining replication: peer="{Voter server-77142aee-2979-634e-7a43-baea0fa60b72 77142aee-2979-634e-7a43-baea0fa60b72}" + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.700Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: pipelining replication: peer="{Voter server-77142aee-2979-634e-7a43-baea0fa60b72 77142aee-2979-634e-7a43-baea0fa60b72}" +testLoggerAdapter verbose: server-cc1ddb75-ee67-a616-1d79-141e839e4263: 2025-02-10T11:34:14.701Z [INFO] server-cc1ddb75-ee67-a616-1d79-141e839e4263: copied to local snapshot: bytes=71 + testing.go:149: server-cc1ddb75-ee67-a616-1d79-141e839e4263: 2025-02-10T11:34:14.701Z [INFO] server-cc1ddb75-ee67-a616-1d79-141e839e4263: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-cc1ddb75-ee67-a616-1d79-141e839e4263: 2025-02-10T11:34:14.701Z [INFO] server-cc1ddb75-ee67-a616-1d79-141e839e4263: Installed remote snapshot + testing.go:149: server-cc1ddb75-ee67-a616-1d79-141e839e4263: 2025-02-10T11:34:14.701Z [INFO] server-cc1ddb75-ee67-a616-1d79-141e839e4263: Installed remote snapshot +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.701Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: pipelining replication: peer="{Voter server-cc1ddb75-ee67-a616-1d79-141e839e4263 cc1ddb75-ee67-a616-1d79-141e839e4263}" + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.701Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: pipelining replication: peer="{Voter server-cc1ddb75-ee67-a616-1d79-141e839e4263 cc1ddb75-ee67-a616-1d79-141e839e4263}" +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.716Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: aborting pipeline replication: peer="{Voter server-cc1ddb75-ee67-a616-1d79-141e839e4263 cc1ddb75-ee67-a616-1d79-141e839e4263}" + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.716Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: aborting pipeline replication: peer="{Voter server-cc1ddb75-ee67-a616-1d79-141e839e4263 cc1ddb75-ee67-a616-1d79-141e839e4263}" +testLoggerAdapter verbose: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.716Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: aborting pipeline replication: peer="{Voter server-77142aee-2979-634e-7a43-baea0fa60b72 77142aee-2979-634e-7a43-baea0fa60b72}" + testing.go:149: server-7073b9b5-b424-61bd-522e-811dbc4994b5: 2025-02-10T11:34:14.716Z [INFO] server-7073b9b5-b424-61bd-522e-811dbc4994b5: aborting pipeline replication: peer="{Voter server-77142aee-2979-634e-7a43-baea0fa60b72 77142aee-2979-634e-7a43-baea0fa60b72}" +testLoggerAdapter verbose: server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: 2025-02-10T11:34:14.717Z [INFO] server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90c0e5c1-c290-77a3-9bb1-82eedf965d36 Address:90c0e5c1-c290-77a3-9bb1-82eedf965d36} {Suffrage:Voter ID:server-70230d0f-0411-338a-1dac-ec8f9531cd81 Address:70230d0f-0411-338a-1dac-ec8f9531cd81} {Suffrage:Voter ID:server-e84355b1-6bed-c60e-79ae-a49db53f455b Address:e84355b1-6bed-c60e-79ae-a49db53f455b}]" + testing.go:149: server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: 2025-02-10T11:34:14.717Z [INFO] server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90c0e5c1-c290-77a3-9bb1-82eedf965d36 Address:90c0e5c1-c290-77a3-9bb1-82eedf965d36} {Suffrage:Voter ID:server-70230d0f-0411-338a-1dac-ec8f9531cd81 Address:70230d0f-0411-338a-1dac-ec8f9531cd81} {Suffrage:Voter ID:server-e84355b1-6bed-c60e-79ae-a49db53f455b Address:e84355b1-6bed-c60e-79ae-a49db53f455b}]" +testLoggerAdapter verbose: server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: 2025-02-10T11:34:14.717Z [INFO] server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: entering follower state: follower="Node at 90c0e5c1-c290-77a3-9bb1-82eedf965d36 [Follower]" leader= + testing.go:149: server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: 2025-02-10T11:34:14.717Z [INFO] server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: entering follower state: follower="Node at 90c0e5c1-c290-77a3-9bb1-82eedf965d36 [Follower]" leader= +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.717Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90c0e5c1-c290-77a3-9bb1-82eedf965d36 Address:90c0e5c1-c290-77a3-9bb1-82eedf965d36} {Suffrage:Voter ID:server-70230d0f-0411-338a-1dac-ec8f9531cd81 Address:70230d0f-0411-338a-1dac-ec8f9531cd81} {Suffrage:Voter ID:server-e84355b1-6bed-c60e-79ae-a49db53f455b Address:e84355b1-6bed-c60e-79ae-a49db53f455b}]" + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.717Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90c0e5c1-c290-77a3-9bb1-82eedf965d36 Address:90c0e5c1-c290-77a3-9bb1-82eedf965d36} {Suffrage:Voter ID:server-70230d0f-0411-338a-1dac-ec8f9531cd81 Address:70230d0f-0411-338a-1dac-ec8f9531cd81} {Suffrage:Voter ID:server-e84355b1-6bed-c60e-79ae-a49db53f455b Address:e84355b1-6bed-c60e-79ae-a49db53f455b}]" +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.717Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: entering follower state: follower="Node at 70230d0f-0411-338a-1dac-ec8f9531cd81 [Follower]" leader= + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.717Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: entering follower state: follower="Node at 70230d0f-0411-338a-1dac-ec8f9531cd81 [Follower]" leader= +testLoggerAdapter verbose: server-e84355b1-6bed-c60e-79ae-a49db53f455b: 2025-02-10T11:34:14.717Z [INFO] server-e84355b1-6bed-c60e-79ae-a49db53f455b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90c0e5c1-c290-77a3-9bb1-82eedf965d36 Address:90c0e5c1-c290-77a3-9bb1-82eedf965d36} {Suffrage:Voter ID:server-70230d0f-0411-338a-1dac-ec8f9531cd81 Address:70230d0f-0411-338a-1dac-ec8f9531cd81} {Suffrage:Voter ID:server-e84355b1-6bed-c60e-79ae-a49db53f455b Address:e84355b1-6bed-c60e-79ae-a49db53f455b}]" + testing.go:149: server-e84355b1-6bed-c60e-79ae-a49db53f455b: 2025-02-10T11:34:14.717Z [INFO] server-e84355b1-6bed-c60e-79ae-a49db53f455b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90c0e5c1-c290-77a3-9bb1-82eedf965d36 Address:90c0e5c1-c290-77a3-9bb1-82eedf965d36} {Suffrage:Voter ID:server-70230d0f-0411-338a-1dac-ec8f9531cd81 Address:70230d0f-0411-338a-1dac-ec8f9531cd81} {Suffrage:Voter ID:server-e84355b1-6bed-c60e-79ae-a49db53f455b Address:e84355b1-6bed-c60e-79ae-a49db53f455b}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:14.717Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:14.717Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-e84355b1-6bed-c60e-79ae-a49db53f455b: 2025-02-10T11:34:14.717Z [INFO] server-e84355b1-6bed-c60e-79ae-a49db53f455b: entering follower state: follower="Node at e84355b1-6bed-c60e-79ae-a49db53f455b [Follower]" leader= + testing.go:149: server-e84355b1-6bed-c60e-79ae-a49db53f455b: 2025-02-10T11:34:14.717Z [INFO] server-e84355b1-6bed-c60e-79ae-a49db53f455b: entering follower state: follower="Node at e84355b1-6bed-c60e-79ae-a49db53f455b [Follower]" leader= +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.805Z [WARN] server-70230d0f-0411-338a-1dac-ec8f9531cd81: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.805Z [WARN] server-70230d0f-0411-338a-1dac-ec8f9531cd81: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.805Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: entering candidate state: node="Node at 70230d0f-0411-338a-1dac-ec8f9531cd81 [Candidate]" term=2 + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.805Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: entering candidate state: node="Node at 70230d0f-0411-338a-1dac-ec8f9531cd81 [Candidate]" term=2 +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.805Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: election won: tally=2 + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.805Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: election won: tally=2 +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.805Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: entering leader state: leader="Node at 70230d0f-0411-338a-1dac-ec8f9531cd81 [Leader]" + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.805Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: entering leader state: leader="Node at 70230d0f-0411-338a-1dac-ec8f9531cd81 [Leader]" +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.805Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: added peer, starting replication: peer=server-90c0e5c1-c290-77a3-9bb1-82eedf965d36 + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.805Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: added peer, starting replication: peer=server-90c0e5c1-c290-77a3-9bb1-82eedf965d36 +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.805Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: added peer, starting replication: peer=server-e84355b1-6bed-c60e-79ae-a49db53f455b + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.805Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: added peer, starting replication: peer=server-e84355b1-6bed-c60e-79ae-a49db53f455b +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.805Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: pipelining replication: peer="{Voter server-e84355b1-6bed-c60e-79ae-a49db53f455b e84355b1-6bed-c60e-79ae-a49db53f455b}" + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.805Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: pipelining replication: peer="{Voter server-e84355b1-6bed-c60e-79ae-a49db53f455b e84355b1-6bed-c60e-79ae-a49db53f455b}" +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.805Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: pipelining replication: peer="{Voter server-90c0e5c1-c290-77a3-9bb1-82eedf965d36 90c0e5c1-c290-77a3-9bb1-82eedf965d36}" + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.805Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: pipelining replication: peer="{Voter server-90c0e5c1-c290-77a3-9bb1-82eedf965d36 90c0e5c1-c290-77a3-9bb1-82eedf965d36}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:14.910Z [INFO] stable state for Leader reached at 2025-02-10 11:34:14.805272425 +0000 UTC m=+11.174176733 (1 nodes), 87.600354ms from start of poll, 87.975102ms from cluster start. Timeout at 2025-02-10 11:34:14.910348228 +0000 UTC m=+11.279252525, 105.075792ms after stability + testing.go:149: cluster: 2025-02-10T11:34:14.910Z [INFO] stable state for Leader reached at 2025-02-10 11:34:14.805272425 +0000 UTC m=+11.174176733 (1 nodes), 87.600354ms from start of poll, 87.975102ms from cluster start. Timeout at 2025-02-10 11:34:14.910348228 +0000 UTC m=+11.279252525, 105.075792ms after stability +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.910Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: starting snapshot up to: index=12 + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.910Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: starting snapshot up to: index=12 + testing.go:153: 2025-02-10T11:34:14.910Z [INFO] creating new snapshot: path=/tmp/raft193024153/snapshots/2-12-1739187254910.tmp +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.914Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: snapshot complete up to: index=12 + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.914Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: snapshot complete up to: index=12 + testing.go:153: 2025-02-10T11:34:14.914Z [INFO] creating new snapshot: path=/tmp/raft193024153/snapshots/2-23-1739187254914.tmp +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.918Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: copied to local snapshot: bytes=71 + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.918Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.918Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: restored user snapshot: index=1 + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.918Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: restored user snapshot: index=1 +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.918Z [ERROR] server-70230d0f-0411-338a-1dac-ec8f9531cd81: failed to get log: index=23 error="log not found" + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.918Z [ERROR] server-70230d0f-0411-338a-1dac-ec8f9531cd81: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.918Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: aborting pipeline replication: peer="{Voter server-e84355b1-6bed-c60e-79ae-a49db53f455b e84355b1-6bed-c60e-79ae-a49db53f455b}" + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.918Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: aborting pipeline replication: peer="{Voter server-e84355b1-6bed-c60e-79ae-a49db53f455b e84355b1-6bed-c60e-79ae-a49db53f455b}" +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.918Z [ERROR] server-70230d0f-0411-338a-1dac-ec8f9531cd81: failed to get log: index=23 error="log not found" + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.918Z [ERROR] server-70230d0f-0411-338a-1dac-ec8f9531cd81: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.918Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: aborting pipeline replication: peer="{Voter server-90c0e5c1-c290-77a3-9bb1-82eedf965d36 90c0e5c1-c290-77a3-9bb1-82eedf965d36}" + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.918Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: aborting pipeline replication: peer="{Voter server-90c0e5c1-c290-77a3-9bb1-82eedf965d36 90c0e5c1-c290-77a3-9bb1-82eedf965d36}" +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.925Z [ERROR] server-70230d0f-0411-338a-1dac-ec8f9531cd81: failed to get log: index=23 error="log not found" + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.925Z [ERROR] server-70230d0f-0411-338a-1dac-ec8f9531cd81: failed to get log: index=23 error="log not found" + testing.go:153: 2025-02-10T11:34:14.925Z [INFO] creating new snapshot: path=/tmp/raft729165199/snapshots/2-23-1739187254925.tmp +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.927Z [ERROR] server-70230d0f-0411-338a-1dac-ec8f9531cd81: failed to get log: index=23 error="log not found" + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.927Z [ERROR] server-70230d0f-0411-338a-1dac-ec8f9531cd81: failed to get log: index=23 error="log not found" + testing.go:153: 2025-02-10T11:34:14.928Z [INFO] creating new snapshot: path=/tmp/raft913532979/snapshots/2-23-1739187254928.tmp +testLoggerAdapter verbose: server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: 2025-02-10T11:34:14.928Z [INFO] server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: copied to local snapshot: bytes=71 + testing.go:149: server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: 2025-02-10T11:34:14.928Z [INFO] server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: 2025-02-10T11:34:14.928Z [INFO] server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: Installed remote snapshot + testing.go:149: server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: 2025-02-10T11:34:14.928Z [INFO] server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: Installed remote snapshot +testLoggerAdapter verbose: server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: 2025-02-10T11:34:14.928Z [WARN] server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: skipping application of old log: index=22 + testing.go:149: server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: 2025-02-10T11:34:14.928Z [WARN] server-90c0e5c1-c290-77a3-9bb1-82eedf965d36: skipping application of old log: index=22 +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.928Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: pipelining replication: peer="{Voter server-90c0e5c1-c290-77a3-9bb1-82eedf965d36 90c0e5c1-c290-77a3-9bb1-82eedf965d36}" + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.928Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: pipelining replication: peer="{Voter server-90c0e5c1-c290-77a3-9bb1-82eedf965d36 90c0e5c1-c290-77a3-9bb1-82eedf965d36}" +testLoggerAdapter verbose: server-e84355b1-6bed-c60e-79ae-a49db53f455b: 2025-02-10T11:34:14.931Z [INFO] server-e84355b1-6bed-c60e-79ae-a49db53f455b: copied to local snapshot: bytes=71 + testing.go:149: server-e84355b1-6bed-c60e-79ae-a49db53f455b: 2025-02-10T11:34:14.931Z [INFO] server-e84355b1-6bed-c60e-79ae-a49db53f455b: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-e84355b1-6bed-c60e-79ae-a49db53f455b: 2025-02-10T11:34:14.931Z [INFO] server-e84355b1-6bed-c60e-79ae-a49db53f455b: Installed remote snapshot + testing.go:149: server-e84355b1-6bed-c60e-79ae-a49db53f455b: 2025-02-10T11:34:14.931Z [INFO] server-e84355b1-6bed-c60e-79ae-a49db53f455b: Installed remote snapshot +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.931Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: pipelining replication: peer="{Voter server-e84355b1-6bed-c60e-79ae-a49db53f455b e84355b1-6bed-c60e-79ae-a49db53f455b}" + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.931Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: pipelining replication: peer="{Voter server-e84355b1-6bed-c60e-79ae-a49db53f455b e84355b1-6bed-c60e-79ae-a49db53f455b}" +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.944Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: aborting pipeline replication: peer="{Voter server-e84355b1-6bed-c60e-79ae-a49db53f455b e84355b1-6bed-c60e-79ae-a49db53f455b}" + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.944Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: aborting pipeline replication: peer="{Voter server-e84355b1-6bed-c60e-79ae-a49db53f455b e84355b1-6bed-c60e-79ae-a49db53f455b}" +testLoggerAdapter verbose: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.944Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: aborting pipeline replication: peer="{Voter server-90c0e5c1-c290-77a3-9bb1-82eedf965d36 90c0e5c1-c290-77a3-9bb1-82eedf965d36}" + testing.go:149: server-70230d0f-0411-338a-1dac-ec8f9531cd81: 2025-02-10T11:34:14.944Z [INFO] server-70230d0f-0411-338a-1dac-ec8f9531cd81: aborting pipeline replication: peer="{Voter server-90c0e5c1-c290-77a3-9bb1-82eedf965d36 90c0e5c1-c290-77a3-9bb1-82eedf965d36}" +testLoggerAdapter verbose: server-c47cd624-b2c4-794d-fb56-76673ad2ec12: 2025-02-10T11:34:14.945Z [INFO] server-c47cd624-b2c4-794d-fb56-76673ad2ec12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c47cd624-b2c4-794d-fb56-76673ad2ec12 Address:c47cd624-b2c4-794d-fb56-76673ad2ec12} {Suffrage:Voter ID:server-5e1ed750-a47f-50bf-4865-fd3b3e23a726 Address:5e1ed750-a47f-50bf-4865-fd3b3e23a726} {Suffrage:Voter ID:server-8fb74d80-05b8-90f9-e4e9-a17f5500abee Address:8fb74d80-05b8-90f9-e4e9-a17f5500abee}]" + testing.go:149: server-c47cd624-b2c4-794d-fb56-76673ad2ec12: 2025-02-10T11:34:14.945Z [INFO] server-c47cd624-b2c4-794d-fb56-76673ad2ec12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c47cd624-b2c4-794d-fb56-76673ad2ec12 Address:c47cd624-b2c4-794d-fb56-76673ad2ec12} {Suffrage:Voter ID:server-5e1ed750-a47f-50bf-4865-fd3b3e23a726 Address:5e1ed750-a47f-50bf-4865-fd3b3e23a726} {Suffrage:Voter ID:server-8fb74d80-05b8-90f9-e4e9-a17f5500abee Address:8fb74d80-05b8-90f9-e4e9-a17f5500abee}]" +testLoggerAdapter verbose: server-c47cd624-b2c4-794d-fb56-76673ad2ec12: 2025-02-10T11:34:14.945Z [INFO] server-c47cd624-b2c4-794d-fb56-76673ad2ec12: entering follower state: follower="Node at c47cd624-b2c4-794d-fb56-76673ad2ec12 [Follower]" leader= + testing.go:149: server-c47cd624-b2c4-794d-fb56-76673ad2ec12: 2025-02-10T11:34:14.945Z [INFO] server-c47cd624-b2c4-794d-fb56-76673ad2ec12: entering follower state: follower="Node at c47cd624-b2c4-794d-fb56-76673ad2ec12 [Follower]" leader= +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:14.945Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c47cd624-b2c4-794d-fb56-76673ad2ec12 Address:c47cd624-b2c4-794d-fb56-76673ad2ec12} {Suffrage:Voter ID:server-5e1ed750-a47f-50bf-4865-fd3b3e23a726 Address:5e1ed750-a47f-50bf-4865-fd3b3e23a726} {Suffrage:Voter ID:server-8fb74d80-05b8-90f9-e4e9-a17f5500abee Address:8fb74d80-05b8-90f9-e4e9-a17f5500abee}]" + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:14.945Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c47cd624-b2c4-794d-fb56-76673ad2ec12 Address:c47cd624-b2c4-794d-fb56-76673ad2ec12} {Suffrage:Voter ID:server-5e1ed750-a47f-50bf-4865-fd3b3e23a726 Address:5e1ed750-a47f-50bf-4865-fd3b3e23a726} {Suffrage:Voter ID:server-8fb74d80-05b8-90f9-e4e9-a17f5500abee Address:8fb74d80-05b8-90f9-e4e9-a17f5500abee}]" +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:14.945Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: entering follower state: follower="Node at 5e1ed750-a47f-50bf-4865-fd3b3e23a726 [Follower]" leader= + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:14.945Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: entering follower state: follower="Node at 5e1ed750-a47f-50bf-4865-fd3b3e23a726 [Follower]" leader= +testLoggerAdapter verbose: server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: 2025-02-10T11:34:14.945Z [INFO] server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c47cd624-b2c4-794d-fb56-76673ad2ec12 Address:c47cd624-b2c4-794d-fb56-76673ad2ec12} {Suffrage:Voter ID:server-5e1ed750-a47f-50bf-4865-fd3b3e23a726 Address:5e1ed750-a47f-50bf-4865-fd3b3e23a726} {Suffrage:Voter ID:server-8fb74d80-05b8-90f9-e4e9-a17f5500abee Address:8fb74d80-05b8-90f9-e4e9-a17f5500abee}]" + testing.go:149: server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: 2025-02-10T11:34:14.945Z [INFO] server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c47cd624-b2c4-794d-fb56-76673ad2ec12 Address:c47cd624-b2c4-794d-fb56-76673ad2ec12} {Suffrage:Voter ID:server-5e1ed750-a47f-50bf-4865-fd3b3e23a726 Address:5e1ed750-a47f-50bf-4865-fd3b3e23a726} {Suffrage:Voter ID:server-8fb74d80-05b8-90f9-e4e9-a17f5500abee Address:8fb74d80-05b8-90f9-e4e9-a17f5500abee}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:14.945Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:14.945Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: 2025-02-10T11:34:14.945Z [INFO] server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: entering follower state: follower="Node at 8fb74d80-05b8-90f9-e4e9-a17f5500abee [Follower]" leader= + testing.go:149: server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: 2025-02-10T11:34:14.945Z [INFO] server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: entering follower state: follower="Node at 8fb74d80-05b8-90f9-e4e9-a17f5500abee [Follower]" leader= +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.028Z [WARN] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.028Z [WARN] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.028Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: entering candidate state: node="Node at 5e1ed750-a47f-50bf-4865-fd3b3e23a726 [Candidate]" term=2 + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.028Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: entering candidate state: node="Node at 5e1ed750-a47f-50bf-4865-fd3b3e23a726 [Candidate]" term=2 +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.028Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: election won: tally=2 + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.028Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: election won: tally=2 +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.028Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: entering leader state: leader="Node at 5e1ed750-a47f-50bf-4865-fd3b3e23a726 [Leader]" + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.028Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: entering leader state: leader="Node at 5e1ed750-a47f-50bf-4865-fd3b3e23a726 [Leader]" +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.028Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: added peer, starting replication: peer=server-c47cd624-b2c4-794d-fb56-76673ad2ec12 + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.028Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: added peer, starting replication: peer=server-c47cd624-b2c4-794d-fb56-76673ad2ec12 +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.028Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: added peer, starting replication: peer=server-8fb74d80-05b8-90f9-e4e9-a17f5500abee + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.028Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: added peer, starting replication: peer=server-8fb74d80-05b8-90f9-e4e9-a17f5500abee +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.028Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: pipelining replication: peer="{Voter server-8fb74d80-05b8-90f9-e4e9-a17f5500abee 8fb74d80-05b8-90f9-e4e9-a17f5500abee}" + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.028Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: pipelining replication: peer="{Voter server-8fb74d80-05b8-90f9-e4e9-a17f5500abee 8fb74d80-05b8-90f9-e4e9-a17f5500abee}" +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.028Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: pipelining replication: peer="{Voter server-c47cd624-b2c4-794d-fb56-76673ad2ec12 c47cd624-b2c4-794d-fb56-76673ad2ec12}" + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.028Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: pipelining replication: peer="{Voter server-c47cd624-b2c4-794d-fb56-76673ad2ec12 c47cd624-b2c4-794d-fb56-76673ad2ec12}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:15.133Z [INFO] stable state for Leader reached at 2025-02-10 11:34:15.028574236 +0000 UTC m=+11.397478553 (1 nodes), 82.727978ms from start of poll, 83.110354ms from cluster start. Timeout at 2025-02-10 11:34:15.133646678 +0000 UTC m=+11.502550974, 105.072421ms after stability + testing.go:149: cluster: 2025-02-10T11:34:15.133Z [INFO] stable state for Leader reached at 2025-02-10 11:34:15.028574236 +0000 UTC m=+11.397478553 (1 nodes), 82.727978ms from start of poll, 83.110354ms from cluster start. Timeout at 2025-02-10 11:34:15.133646678 +0000 UTC m=+11.502550974, 105.072421ms after stability +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.133Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: starting snapshot up to: index=12 + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.133Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: starting snapshot up to: index=12 + testing.go:153: 2025-02-10T11:34:15.133Z [INFO] creating new snapshot: path=/tmp/raft960678167/snapshots/2-12-1739187255133.tmp +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.138Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: snapshot complete up to: index=12 + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.138Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: snapshot complete up to: index=12 + testing.go:153: 2025-02-10T11:34:15.138Z [INFO] creating new snapshot: path=/tmp/raft960678167/snapshots/2-113-1739187255138.tmp +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.141Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: copied to local snapshot: bytes=71 + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.141Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.141Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: restored user snapshot: index=1 + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.141Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: restored user snapshot: index=1 +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.141Z [ERROR] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: failed to get log: index=23 error="log not found" + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.141Z [ERROR] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.141Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: aborting pipeline replication: peer="{Voter server-8fb74d80-05b8-90f9-e4e9-a17f5500abee 8fb74d80-05b8-90f9-e4e9-a17f5500abee}" + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.141Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: aborting pipeline replication: peer="{Voter server-8fb74d80-05b8-90f9-e4e9-a17f5500abee 8fb74d80-05b8-90f9-e4e9-a17f5500abee}" +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.141Z [ERROR] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: failed to get log: index=23 error="log not found" + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.141Z [ERROR] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.141Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: aborting pipeline replication: peer="{Voter server-c47cd624-b2c4-794d-fb56-76673ad2ec12 c47cd624-b2c4-794d-fb56-76673ad2ec12}" + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.141Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: aborting pipeline replication: peer="{Voter server-c47cd624-b2c4-794d-fb56-76673ad2ec12 c47cd624-b2c4-794d-fb56-76673ad2ec12}" +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.147Z [ERROR] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: failed to get log: index=23 error="log not found" + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.147Z [ERROR] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: failed to get log: index=23 error="log not found" + testing.go:153: 2025-02-10T11:34:15.147Z [INFO] creating new snapshot: path=/tmp/raft984029025/snapshots/2-113-1739187255147.tmp +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.150Z [ERROR] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: failed to get log: index=23 error="log not found" + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.150Z [ERROR] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: failed to get log: index=23 error="log not found" + testing.go:153: 2025-02-10T11:34:15.150Z [INFO] creating new snapshot: path=/tmp/raft320489181/snapshots/2-113-1739187255150.tmp +testLoggerAdapter verbose: server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: 2025-02-10T11:34:15.150Z [INFO] server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: copied to local snapshot: bytes=71 + testing.go:149: server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: 2025-02-10T11:34:15.150Z [INFO] server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: 2025-02-10T11:34:15.150Z [INFO] server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: Installed remote snapshot + testing.go:149: server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: 2025-02-10T11:34:15.150Z [INFO] server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: Installed remote snapshot +testLoggerAdapter verbose: server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: 2025-02-10T11:34:15.150Z [WARN] server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: skipping application of old log: index=22 + testing.go:149: server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: 2025-02-10T11:34:15.150Z [WARN] server-8fb74d80-05b8-90f9-e4e9-a17f5500abee: skipping application of old log: index=22 +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.150Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: pipelining replication: peer="{Voter server-8fb74d80-05b8-90f9-e4e9-a17f5500abee 8fb74d80-05b8-90f9-e4e9-a17f5500abee}" + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.150Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: pipelining replication: peer="{Voter server-8fb74d80-05b8-90f9-e4e9-a17f5500abee 8fb74d80-05b8-90f9-e4e9-a17f5500abee}" +testLoggerAdapter verbose: server-c47cd624-b2c4-794d-fb56-76673ad2ec12: 2025-02-10T11:34:15.153Z [INFO] server-c47cd624-b2c4-794d-fb56-76673ad2ec12: copied to local snapshot: bytes=71 + testing.go:149: server-c47cd624-b2c4-794d-fb56-76673ad2ec12: 2025-02-10T11:34:15.153Z [INFO] server-c47cd624-b2c4-794d-fb56-76673ad2ec12: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-c47cd624-b2c4-794d-fb56-76673ad2ec12: 2025-02-10T11:34:15.153Z [INFO] server-c47cd624-b2c4-794d-fb56-76673ad2ec12: Installed remote snapshot + testing.go:149: server-c47cd624-b2c4-794d-fb56-76673ad2ec12: 2025-02-10T11:34:15.153Z [INFO] server-c47cd624-b2c4-794d-fb56-76673ad2ec12: Installed remote snapshot +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.153Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: pipelining replication: peer="{Voter server-c47cd624-b2c4-794d-fb56-76673ad2ec12 c47cd624-b2c4-794d-fb56-76673ad2ec12}" + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.153Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: pipelining replication: peer="{Voter server-c47cd624-b2c4-794d-fb56-76673ad2ec12 c47cd624-b2c4-794d-fb56-76673ad2ec12}" +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.166Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: aborting pipeline replication: peer="{Voter server-8fb74d80-05b8-90f9-e4e9-a17f5500abee 8fb74d80-05b8-90f9-e4e9-a17f5500abee}" + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.166Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: aborting pipeline replication: peer="{Voter server-8fb74d80-05b8-90f9-e4e9-a17f5500abee 8fb74d80-05b8-90f9-e4e9-a17f5500abee}" +testLoggerAdapter verbose: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.166Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: aborting pipeline replication: peer="{Voter server-c47cd624-b2c4-794d-fb56-76673ad2ec12 c47cd624-b2c4-794d-fb56-76673ad2ec12}" + testing.go:149: server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: 2025-02-10T11:34:15.166Z [INFO] server-5e1ed750-a47f-50bf-4865-fd3b3e23a726: aborting pipeline replication: peer="{Voter server-c47cd624-b2c4-794d-fb56-76673ad2ec12 c47cd624-b2c4-794d-fb56-76673ad2ec12}" +testLoggerAdapter verbose: server-e61a1425-2820-0d76-ef22-b9252688f802: 2025-02-10T11:34:15.167Z [INFO] server-e61a1425-2820-0d76-ef22-b9252688f802: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e61a1425-2820-0d76-ef22-b9252688f802 Address:e61a1425-2820-0d76-ef22-b9252688f802} {Suffrage:Voter ID:server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a Address:0e8b1fa8-63d7-7647-a9bb-65a1b951d56a} {Suffrage:Voter ID:server-27d2e67c-b3a2-be67-1885-adfbaf815739 Address:27d2e67c-b3a2-be67-1885-adfbaf815739}]" + testing.go:149: server-e61a1425-2820-0d76-ef22-b9252688f802: 2025-02-10T11:34:15.167Z [INFO] server-e61a1425-2820-0d76-ef22-b9252688f802: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e61a1425-2820-0d76-ef22-b9252688f802 Address:e61a1425-2820-0d76-ef22-b9252688f802} {Suffrage:Voter ID:server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a Address:0e8b1fa8-63d7-7647-a9bb-65a1b951d56a} {Suffrage:Voter ID:server-27d2e67c-b3a2-be67-1885-adfbaf815739 Address:27d2e67c-b3a2-be67-1885-adfbaf815739}]" +testLoggerAdapter verbose: server-e61a1425-2820-0d76-ef22-b9252688f802: 2025-02-10T11:34:15.167Z [INFO] server-e61a1425-2820-0d76-ef22-b9252688f802: entering follower state: follower="Node at e61a1425-2820-0d76-ef22-b9252688f802 [Follower]" leader= + testing.go:149: server-e61a1425-2820-0d76-ef22-b9252688f802: 2025-02-10T11:34:15.167Z [INFO] server-e61a1425-2820-0d76-ef22-b9252688f802: entering follower state: follower="Node at e61a1425-2820-0d76-ef22-b9252688f802 [Follower]" leader= +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.168Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e61a1425-2820-0d76-ef22-b9252688f802 Address:e61a1425-2820-0d76-ef22-b9252688f802} {Suffrage:Voter ID:server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a Address:0e8b1fa8-63d7-7647-a9bb-65a1b951d56a} {Suffrage:Voter ID:server-27d2e67c-b3a2-be67-1885-adfbaf815739 Address:27d2e67c-b3a2-be67-1885-adfbaf815739}]" + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.168Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e61a1425-2820-0d76-ef22-b9252688f802 Address:e61a1425-2820-0d76-ef22-b9252688f802} {Suffrage:Voter ID:server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a Address:0e8b1fa8-63d7-7647-a9bb-65a1b951d56a} {Suffrage:Voter ID:server-27d2e67c-b3a2-be67-1885-adfbaf815739 Address:27d2e67c-b3a2-be67-1885-adfbaf815739}]" +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.168Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: entering follower state: follower="Node at 0e8b1fa8-63d7-7647-a9bb-65a1b951d56a [Follower]" leader= + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.168Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: entering follower state: follower="Node at 0e8b1fa8-63d7-7647-a9bb-65a1b951d56a [Follower]" leader= +testLoggerAdapter verbose: server-27d2e67c-b3a2-be67-1885-adfbaf815739: 2025-02-10T11:34:15.168Z [INFO] server-27d2e67c-b3a2-be67-1885-adfbaf815739: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e61a1425-2820-0d76-ef22-b9252688f802 Address:e61a1425-2820-0d76-ef22-b9252688f802} {Suffrage:Voter ID:server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a Address:0e8b1fa8-63d7-7647-a9bb-65a1b951d56a} {Suffrage:Voter ID:server-27d2e67c-b3a2-be67-1885-adfbaf815739 Address:27d2e67c-b3a2-be67-1885-adfbaf815739}]" + testing.go:149: server-27d2e67c-b3a2-be67-1885-adfbaf815739: 2025-02-10T11:34:15.168Z [INFO] server-27d2e67c-b3a2-be67-1885-adfbaf815739: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e61a1425-2820-0d76-ef22-b9252688f802 Address:e61a1425-2820-0d76-ef22-b9252688f802} {Suffrage:Voter ID:server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a Address:0e8b1fa8-63d7-7647-a9bb-65a1b951d56a} {Suffrage:Voter ID:server-27d2e67c-b3a2-be67-1885-adfbaf815739 Address:27d2e67c-b3a2-be67-1885-adfbaf815739}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:15.168Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:15.168Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-27d2e67c-b3a2-be67-1885-adfbaf815739: 2025-02-10T11:34:15.168Z [INFO] server-27d2e67c-b3a2-be67-1885-adfbaf815739: entering follower state: follower="Node at 27d2e67c-b3a2-be67-1885-adfbaf815739 [Follower]" leader= + testing.go:149: server-27d2e67c-b3a2-be67-1885-adfbaf815739: 2025-02-10T11:34:15.168Z [INFO] server-27d2e67c-b3a2-be67-1885-adfbaf815739: entering follower state: follower="Node at 27d2e67c-b3a2-be67-1885-adfbaf815739 [Follower]" leader= +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.233Z [WARN] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.233Z [WARN] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.233Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: entering candidate state: node="Node at 0e8b1fa8-63d7-7647-a9bb-65a1b951d56a [Candidate]" term=2 + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.233Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: entering candidate state: node="Node at 0e8b1fa8-63d7-7647-a9bb-65a1b951d56a [Candidate]" term=2 +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.233Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: election won: tally=2 + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.233Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: election won: tally=2 +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.233Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: entering leader state: leader="Node at 0e8b1fa8-63d7-7647-a9bb-65a1b951d56a [Leader]" + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.233Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: entering leader state: leader="Node at 0e8b1fa8-63d7-7647-a9bb-65a1b951d56a [Leader]" +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.233Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: added peer, starting replication: peer=server-e61a1425-2820-0d76-ef22-b9252688f802 + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.233Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: added peer, starting replication: peer=server-e61a1425-2820-0d76-ef22-b9252688f802 +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.233Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: added peer, starting replication: peer=server-27d2e67c-b3a2-be67-1885-adfbaf815739 + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.233Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: added peer, starting replication: peer=server-27d2e67c-b3a2-be67-1885-adfbaf815739 +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.233Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: pipelining replication: peer="{Voter server-e61a1425-2820-0d76-ef22-b9252688f802 e61a1425-2820-0d76-ef22-b9252688f802}" + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.233Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: pipelining replication: peer="{Voter server-e61a1425-2820-0d76-ef22-b9252688f802 e61a1425-2820-0d76-ef22-b9252688f802}" +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.233Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: pipelining replication: peer="{Voter server-27d2e67c-b3a2-be67-1885-adfbaf815739 27d2e67c-b3a2-be67-1885-adfbaf815739}" + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.233Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: pipelining replication: peer="{Voter server-27d2e67c-b3a2-be67-1885-adfbaf815739 27d2e67c-b3a2-be67-1885-adfbaf815739}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:15.338Z [INFO] stable state for Leader reached at 2025-02-10 11:34:15.233682874 +0000 UTC m=+11.602587174 (1 nodes), 65.439297ms from start of poll, 65.91897ms from cluster start. Timeout at 2025-02-10 11:34:15.338757709 +0000 UTC m=+11.707662006, 105.074832ms after stability + testing.go:149: cluster: 2025-02-10T11:34:15.338Z [INFO] stable state for Leader reached at 2025-02-10 11:34:15.233682874 +0000 UTC m=+11.602587174 (1 nodes), 65.439297ms from start of poll, 65.91897ms from cluster start. Timeout at 2025-02-10 11:34:15.338757709 +0000 UTC m=+11.707662006, 105.074832ms after stability +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.338Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: starting snapshot up to: index=12 + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.338Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: starting snapshot up to: index=12 + testing.go:153: 2025-02-10T11:34:15.338Z [INFO] creating new snapshot: path=/tmp/raft142705189/snapshots/2-12-1739187255338.tmp +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.342Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: snapshot complete up to: index=12 + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.342Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: snapshot complete up to: index=12 + testing.go:153: 2025-02-10T11:34:15.343Z [INFO] creating new snapshot: path=/tmp/raft142705189/snapshots/2-1013-1739187255343.tmp +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.346Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: copied to local snapshot: bytes=71 + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.346Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.346Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: restored user snapshot: index=1 + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.346Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: restored user snapshot: index=1 +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.347Z [ERROR] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: failed to get log: index=23 error="log not found" + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.347Z [ERROR] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.347Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: aborting pipeline replication: peer="{Voter server-27d2e67c-b3a2-be67-1885-adfbaf815739 27d2e67c-b3a2-be67-1885-adfbaf815739}" + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.347Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: aborting pipeline replication: peer="{Voter server-27d2e67c-b3a2-be67-1885-adfbaf815739 27d2e67c-b3a2-be67-1885-adfbaf815739}" +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.347Z [ERROR] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: failed to get log: index=23 error="log not found" + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.347Z [ERROR] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.347Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: aborting pipeline replication: peer="{Voter server-e61a1425-2820-0d76-ef22-b9252688f802 e61a1425-2820-0d76-ef22-b9252688f802}" + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.347Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: aborting pipeline replication: peer="{Voter server-e61a1425-2820-0d76-ef22-b9252688f802 e61a1425-2820-0d76-ef22-b9252688f802}" +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.352Z [ERROR] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: failed to get log: index=23 error="log not found" + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.352Z [ERROR] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: failed to get log: index=23 error="log not found" + testing.go:153: 2025-02-10T11:34:15.353Z [INFO] creating new snapshot: path=/tmp/raft168087099/snapshots/2-1013-1739187255353.tmp +testLoggerAdapter verbose: server-e61a1425-2820-0d76-ef22-b9252688f802: 2025-02-10T11:34:15.356Z [INFO] server-e61a1425-2820-0d76-ef22-b9252688f802: copied to local snapshot: bytes=71 + testing.go:149: server-e61a1425-2820-0d76-ef22-b9252688f802: 2025-02-10T11:34:15.356Z [INFO] server-e61a1425-2820-0d76-ef22-b9252688f802: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-e61a1425-2820-0d76-ef22-b9252688f802: 2025-02-10T11:34:15.356Z [INFO] server-e61a1425-2820-0d76-ef22-b9252688f802: Installed remote snapshot + testing.go:149: server-e61a1425-2820-0d76-ef22-b9252688f802: 2025-02-10T11:34:15.356Z [INFO] server-e61a1425-2820-0d76-ef22-b9252688f802: Installed remote snapshot +testLoggerAdapter verbose: server-e61a1425-2820-0d76-ef22-b9252688f802: 2025-02-10T11:34:15.356Z [WARN] server-e61a1425-2820-0d76-ef22-b9252688f802: skipping application of old log: index=1013 + testing.go:149: server-e61a1425-2820-0d76-ef22-b9252688f802: 2025-02-10T11:34:15.356Z [WARN] server-e61a1425-2820-0d76-ef22-b9252688f802: skipping application of old log: index=1013 +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.356Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: pipelining replication: peer="{Voter server-e61a1425-2820-0d76-ef22-b9252688f802 e61a1425-2820-0d76-ef22-b9252688f802}" + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.356Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: pipelining replication: peer="{Voter server-e61a1425-2820-0d76-ef22-b9252688f802 e61a1425-2820-0d76-ef22-b9252688f802}" +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.356Z [ERROR] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: failed to get log: index=23 error="log not found" + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.356Z [ERROR] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: failed to get log: index=23 error="log not found" + testing.go:153: 2025-02-10T11:34:15.357Z [INFO] creating new snapshot: path=/tmp/raft984375199/snapshots/2-1013-1739187255357.tmp +testLoggerAdapter verbose: server-27d2e67c-b3a2-be67-1885-adfbaf815739: 2025-02-10T11:34:15.360Z [INFO] server-27d2e67c-b3a2-be67-1885-adfbaf815739: copied to local snapshot: bytes=71 + testing.go:149: server-27d2e67c-b3a2-be67-1885-adfbaf815739: 2025-02-10T11:34:15.360Z [INFO] server-27d2e67c-b3a2-be67-1885-adfbaf815739: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-27d2e67c-b3a2-be67-1885-adfbaf815739: 2025-02-10T11:34:15.360Z [INFO] server-27d2e67c-b3a2-be67-1885-adfbaf815739: Installed remote snapshot + testing.go:149: server-27d2e67c-b3a2-be67-1885-adfbaf815739: 2025-02-10T11:34:15.360Z [INFO] server-27d2e67c-b3a2-be67-1885-adfbaf815739: Installed remote snapshot +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.360Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: pipelining replication: peer="{Voter server-27d2e67c-b3a2-be67-1885-adfbaf815739 27d2e67c-b3a2-be67-1885-adfbaf815739}" + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.360Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: pipelining replication: peer="{Voter server-27d2e67c-b3a2-be67-1885-adfbaf815739 27d2e67c-b3a2-be67-1885-adfbaf815739}" +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.372Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: aborting pipeline replication: peer="{Voter server-27d2e67c-b3a2-be67-1885-adfbaf815739 27d2e67c-b3a2-be67-1885-adfbaf815739}" + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.372Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: aborting pipeline replication: peer="{Voter server-27d2e67c-b3a2-be67-1885-adfbaf815739 27d2e67c-b3a2-be67-1885-adfbaf815739}" +testLoggerAdapter verbose: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.372Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: aborting pipeline replication: peer="{Voter server-e61a1425-2820-0d76-ef22-b9252688f802 e61a1425-2820-0d76-ef22-b9252688f802}" + testing.go:149: server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: 2025-02-10T11:34:15.372Z [INFO] server-0e8b1fa8-63d7-7647-a9bb-65a1b951d56a: aborting pipeline replication: peer="{Voter server-e61a1425-2820-0d76-ef22-b9252688f802 e61a1425-2820-0d76-ef22-b9252688f802}" +testLoggerAdapter verbose: server-9460d8c4-1a97-026d-d752-fe77c5640676: 2025-02-10T11:34:15.373Z [INFO] server-9460d8c4-1a97-026d-d752-fe77c5640676: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9460d8c4-1a97-026d-d752-fe77c5640676 Address:9460d8c4-1a97-026d-d752-fe77c5640676} {Suffrage:Voter ID:server-f3e90ecb-61fa-597c-f955-b9aa253138ca Address:f3e90ecb-61fa-597c-f955-b9aa253138ca} {Suffrage:Voter ID:server-e25eecab-c323-3c4f-b52e-031f09fec682 Address:e25eecab-c323-3c4f-b52e-031f09fec682}]" + testing.go:149: server-9460d8c4-1a97-026d-d752-fe77c5640676: 2025-02-10T11:34:15.373Z [INFO] server-9460d8c4-1a97-026d-d752-fe77c5640676: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9460d8c4-1a97-026d-d752-fe77c5640676 Address:9460d8c4-1a97-026d-d752-fe77c5640676} {Suffrage:Voter ID:server-f3e90ecb-61fa-597c-f955-b9aa253138ca Address:f3e90ecb-61fa-597c-f955-b9aa253138ca} {Suffrage:Voter ID:server-e25eecab-c323-3c4f-b52e-031f09fec682 Address:e25eecab-c323-3c4f-b52e-031f09fec682}]" +testLoggerAdapter verbose: server-9460d8c4-1a97-026d-d752-fe77c5640676: 2025-02-10T11:34:15.373Z [INFO] server-9460d8c4-1a97-026d-d752-fe77c5640676: entering follower state: follower="Node at 9460d8c4-1a97-026d-d752-fe77c5640676 [Follower]" leader= + testing.go:149: server-9460d8c4-1a97-026d-d752-fe77c5640676: 2025-02-10T11:34:15.373Z [INFO] server-9460d8c4-1a97-026d-d752-fe77c5640676: entering follower state: follower="Node at 9460d8c4-1a97-026d-d752-fe77c5640676 [Follower]" leader= +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.373Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9460d8c4-1a97-026d-d752-fe77c5640676 Address:9460d8c4-1a97-026d-d752-fe77c5640676} {Suffrage:Voter ID:server-f3e90ecb-61fa-597c-f955-b9aa253138ca Address:f3e90ecb-61fa-597c-f955-b9aa253138ca} {Suffrage:Voter ID:server-e25eecab-c323-3c4f-b52e-031f09fec682 Address:e25eecab-c323-3c4f-b52e-031f09fec682}]" + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.373Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9460d8c4-1a97-026d-d752-fe77c5640676 Address:9460d8c4-1a97-026d-d752-fe77c5640676} {Suffrage:Voter ID:server-f3e90ecb-61fa-597c-f955-b9aa253138ca Address:f3e90ecb-61fa-597c-f955-b9aa253138ca} {Suffrage:Voter ID:server-e25eecab-c323-3c4f-b52e-031f09fec682 Address:e25eecab-c323-3c4f-b52e-031f09fec682}]" +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.373Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: entering follower state: follower="Node at f3e90ecb-61fa-597c-f955-b9aa253138ca [Follower]" leader= + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.373Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: entering follower state: follower="Node at f3e90ecb-61fa-597c-f955-b9aa253138ca [Follower]" leader= +testLoggerAdapter verbose: server-e25eecab-c323-3c4f-b52e-031f09fec682: 2025-02-10T11:34:15.373Z [INFO] server-e25eecab-c323-3c4f-b52e-031f09fec682: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9460d8c4-1a97-026d-d752-fe77c5640676 Address:9460d8c4-1a97-026d-d752-fe77c5640676} {Suffrage:Voter ID:server-f3e90ecb-61fa-597c-f955-b9aa253138ca Address:f3e90ecb-61fa-597c-f955-b9aa253138ca} {Suffrage:Voter ID:server-e25eecab-c323-3c4f-b52e-031f09fec682 Address:e25eecab-c323-3c4f-b52e-031f09fec682}]" + testing.go:149: server-e25eecab-c323-3c4f-b52e-031f09fec682: 2025-02-10T11:34:15.373Z [INFO] server-e25eecab-c323-3c4f-b52e-031f09fec682: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9460d8c4-1a97-026d-d752-fe77c5640676 Address:9460d8c4-1a97-026d-d752-fe77c5640676} {Suffrage:Voter ID:server-f3e90ecb-61fa-597c-f955-b9aa253138ca Address:f3e90ecb-61fa-597c-f955-b9aa253138ca} {Suffrage:Voter ID:server-e25eecab-c323-3c4f-b52e-031f09fec682 Address:e25eecab-c323-3c4f-b52e-031f09fec682}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:15.373Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:15.373Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-e25eecab-c323-3c4f-b52e-031f09fec682: 2025-02-10T11:34:15.373Z [INFO] server-e25eecab-c323-3c4f-b52e-031f09fec682: entering follower state: follower="Node at e25eecab-c323-3c4f-b52e-031f09fec682 [Follower]" leader= + testing.go:149: server-e25eecab-c323-3c4f-b52e-031f09fec682: 2025-02-10T11:34:15.373Z [INFO] server-e25eecab-c323-3c4f-b52e-031f09fec682: entering follower state: follower="Node at e25eecab-c323-3c4f-b52e-031f09fec682 [Follower]" leader= +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.425Z [WARN] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.425Z [WARN] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.425Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: entering candidate state: node="Node at f3e90ecb-61fa-597c-f955-b9aa253138ca [Candidate]" term=2 + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.425Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: entering candidate state: node="Node at f3e90ecb-61fa-597c-f955-b9aa253138ca [Candidate]" term=2 +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.425Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: election won: tally=2 + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.425Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: election won: tally=2 +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.425Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: entering leader state: leader="Node at f3e90ecb-61fa-597c-f955-b9aa253138ca [Leader]" + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.425Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: entering leader state: leader="Node at f3e90ecb-61fa-597c-f955-b9aa253138ca [Leader]" +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.425Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: added peer, starting replication: peer=server-9460d8c4-1a97-026d-d752-fe77c5640676 + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.425Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: added peer, starting replication: peer=server-9460d8c4-1a97-026d-d752-fe77c5640676 +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.425Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: added peer, starting replication: peer=server-e25eecab-c323-3c4f-b52e-031f09fec682 + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.425Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: added peer, starting replication: peer=server-e25eecab-c323-3c4f-b52e-031f09fec682 +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.425Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: pipelining replication: peer="{Voter server-e25eecab-c323-3c4f-b52e-031f09fec682 e25eecab-c323-3c4f-b52e-031f09fec682}" + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.425Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: pipelining replication: peer="{Voter server-e25eecab-c323-3c4f-b52e-031f09fec682 e25eecab-c323-3c4f-b52e-031f09fec682}" +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.425Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: pipelining replication: peer="{Voter server-9460d8c4-1a97-026d-d752-fe77c5640676 9460d8c4-1a97-026d-d752-fe77c5640676}" + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.425Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: pipelining replication: peer="{Voter server-9460d8c4-1a97-026d-d752-fe77c5640676 9460d8c4-1a97-026d-d752-fe77c5640676}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:15.530Z [INFO] stable state for Leader reached at 2025-02-10 11:34:15.425722993 +0000 UTC m=+11.794627294 (1 nodes), 52.122775ms from start of poll, 52.48789ms from cluster start. Timeout at 2025-02-10 11:34:15.530744918 +0000 UTC m=+11.899649216, 105.021922ms after stability + testing.go:149: cluster: 2025-02-10T11:34:15.530Z [INFO] stable state for Leader reached at 2025-02-10 11:34:15.425722993 +0000 UTC m=+11.794627294 (1 nodes), 52.122775ms from start of poll, 52.48789ms from cluster start. Timeout at 2025-02-10 11:34:15.530744918 +0000 UTC m=+11.899649216, 105.021922ms after stability +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.530Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: starting snapshot up to: index=12 + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.530Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: starting snapshot up to: index=12 + testing.go:153: 2025-02-10T11:34:15.531Z [INFO] creating new snapshot: path=/tmp/raft973796675/snapshots/2-12-1739187255531.tmp +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.534Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: snapshot complete up to: index=12 + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.534Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: snapshot complete up to: index=12 + testing.go:153: 2025-02-10T11:34:15.535Z [INFO] creating new snapshot: path=/tmp/raft973796675/snapshots/2-10013-1739187255535.tmp +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.538Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: copied to local snapshot: bytes=71 + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.538Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.538Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: restored user snapshot: index=1 + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.538Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: restored user snapshot: index=1 +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.538Z [ERROR] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: failed to get log: index=23 error="log not found" + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.538Z [ERROR] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.538Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: aborting pipeline replication: peer="{Voter server-e25eecab-c323-3c4f-b52e-031f09fec682 e25eecab-c323-3c4f-b52e-031f09fec682}" + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.538Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: aborting pipeline replication: peer="{Voter server-e25eecab-c323-3c4f-b52e-031f09fec682 e25eecab-c323-3c4f-b52e-031f09fec682}" +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.538Z [ERROR] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: failed to get log: index=23 error="log not found" + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.538Z [ERROR] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.538Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: aborting pipeline replication: peer="{Voter server-9460d8c4-1a97-026d-d752-fe77c5640676 9460d8c4-1a97-026d-d752-fe77c5640676}" + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.538Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: aborting pipeline replication: peer="{Voter server-9460d8c4-1a97-026d-d752-fe77c5640676 9460d8c4-1a97-026d-d752-fe77c5640676}" +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.546Z [ERROR] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: failed to get log: index=23 error="log not found" + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.546Z [ERROR] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: failed to get log: index=23 error="log not found" + testing.go:153: 2025-02-10T11:34:15.546Z [INFO] creating new snapshot: path=/tmp/raft692088105/snapshots/2-10013-1739187255546.tmp +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.547Z [ERROR] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: failed to get log: index=23 error="log not found" + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.547Z [ERROR] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: failed to get log: index=23 error="log not found" + testing.go:153: 2025-02-10T11:34:15.547Z [INFO] creating new snapshot: path=/tmp/raft878192237/snapshots/2-10013-1739187255547.tmp +testLoggerAdapter verbose: server-9460d8c4-1a97-026d-d752-fe77c5640676: 2025-02-10T11:34:15.551Z [INFO] server-9460d8c4-1a97-026d-d752-fe77c5640676: copied to local snapshot: bytes=71 + testing.go:149: server-9460d8c4-1a97-026d-d752-fe77c5640676: 2025-02-10T11:34:15.551Z [INFO] server-9460d8c4-1a97-026d-d752-fe77c5640676: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-9460d8c4-1a97-026d-d752-fe77c5640676: 2025-02-10T11:34:15.551Z [INFO] server-9460d8c4-1a97-026d-d752-fe77c5640676: Installed remote snapshot + testing.go:149: server-9460d8c4-1a97-026d-d752-fe77c5640676: 2025-02-10T11:34:15.551Z [INFO] server-9460d8c4-1a97-026d-d752-fe77c5640676: Installed remote snapshot +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.551Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: pipelining replication: peer="{Voter server-9460d8c4-1a97-026d-d752-fe77c5640676 9460d8c4-1a97-026d-d752-fe77c5640676}" + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.551Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: pipelining replication: peer="{Voter server-9460d8c4-1a97-026d-d752-fe77c5640676 9460d8c4-1a97-026d-d752-fe77c5640676}" +testLoggerAdapter verbose: server-e25eecab-c323-3c4f-b52e-031f09fec682: 2025-02-10T11:34:15.551Z [INFO] server-e25eecab-c323-3c4f-b52e-031f09fec682: copied to local snapshot: bytes=71 + testing.go:149: server-e25eecab-c323-3c4f-b52e-031f09fec682: 2025-02-10T11:34:15.551Z [INFO] server-e25eecab-c323-3c4f-b52e-031f09fec682: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-e25eecab-c323-3c4f-b52e-031f09fec682: 2025-02-10T11:34:15.552Z [INFO] server-e25eecab-c323-3c4f-b52e-031f09fec682: Installed remote snapshot + testing.go:149: server-e25eecab-c323-3c4f-b52e-031f09fec682: 2025-02-10T11:34:15.552Z [INFO] server-e25eecab-c323-3c4f-b52e-031f09fec682: Installed remote snapshot +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.552Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: pipelining replication: peer="{Voter server-e25eecab-c323-3c4f-b52e-031f09fec682 e25eecab-c323-3c4f-b52e-031f09fec682}" + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.552Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: pipelining replication: peer="{Voter server-e25eecab-c323-3c4f-b52e-031f09fec682 e25eecab-c323-3c4f-b52e-031f09fec682}" +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.567Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: aborting pipeline replication: peer="{Voter server-e25eecab-c323-3c4f-b52e-031f09fec682 e25eecab-c323-3c4f-b52e-031f09fec682}" + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.567Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: aborting pipeline replication: peer="{Voter server-e25eecab-c323-3c4f-b52e-031f09fec682 e25eecab-c323-3c4f-b52e-031f09fec682}" +testLoggerAdapter verbose: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.567Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: aborting pipeline replication: peer="{Voter server-9460d8c4-1a97-026d-d752-fe77c5640676 9460d8c4-1a97-026d-d752-fe77c5640676}" + testing.go:149: server-f3e90ecb-61fa-597c-f955-b9aa253138ca: 2025-02-10T11:34:15.567Z [INFO] server-f3e90ecb-61fa-597c-f955-b9aa253138ca: aborting pipeline replication: peer="{Voter server-9460d8c4-1a97-026d-d752-fe77c5640676 9460d8c4-1a97-026d-d752-fe77c5640676}" +--- PASS: TestRaft_UserRestore (1.26s) === RUN TestRaft_SendSnapshotFollower -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.374Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138c9853-4660-c7de-1ab8-eed6328689ca Address:138c9853-4660-c7de-1ab8-eed6328689ca} {Suffrage:Voter ID:server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc Address:cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc} {Suffrage:Voter ID:server-cc3c790b-3981-510f-d759-353e8ab43bff Address:cc3c790b-3981-510f-d759-353e8ab43bff}]" - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.374Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138c9853-4660-c7de-1ab8-eed6328689ca Address:138c9853-4660-c7de-1ab8-eed6328689ca} {Suffrage:Voter ID:server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc Address:cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc} {Suffrage:Voter ID:server-cc3c790b-3981-510f-d759-353e8ab43bff Address:cc3c790b-3981-510f-d759-353e8ab43bff}]" -testLoggerAdapter verbose: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.374Z [INFO] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138c9853-4660-c7de-1ab8-eed6328689ca Address:138c9853-4660-c7de-1ab8-eed6328689ca} {Suffrage:Voter ID:server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc Address:cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc} {Suffrage:Voter ID:server-cc3c790b-3981-510f-d759-353e8ab43bff Address:cc3c790b-3981-510f-d759-353e8ab43bff}]" - testing.go:149: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.374Z [INFO] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138c9853-4660-c7de-1ab8-eed6328689ca Address:138c9853-4660-c7de-1ab8-eed6328689ca} {Suffrage:Voter ID:server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc Address:cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc} {Suffrage:Voter ID:server-cc3c790b-3981-510f-d759-353e8ab43bff Address:cc3c790b-3981-510f-d759-353e8ab43bff}]" -testLoggerAdapter verbose: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.374Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138c9853-4660-c7de-1ab8-eed6328689ca Address:138c9853-4660-c7de-1ab8-eed6328689ca} {Suffrage:Voter ID:server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc Address:cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc} {Suffrage:Voter ID:server-cc3c790b-3981-510f-d759-353e8ab43bff Address:cc3c790b-3981-510f-d759-353e8ab43bff}]" - testing.go:149: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.374Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138c9853-4660-c7de-1ab8-eed6328689ca Address:138c9853-4660-c7de-1ab8-eed6328689ca} {Suffrage:Voter ID:server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc Address:cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc} {Suffrage:Voter ID:server-cc3c790b-3981-510f-d759-353e8ab43bff Address:cc3c790b-3981-510f-d759-353e8ab43bff}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.374Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:10:01.374Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.374Z [INFO] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: entering follower state: follower="Node at cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc [Follower]" leader= - testing.go:149: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.374Z [INFO] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: entering follower state: follower="Node at cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc [Follower]" leader= -testLoggerAdapter verbose: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.374Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: entering follower state: follower="Node at cc3c790b-3981-510f-d759-353e8ab43bff [Follower]" leader= - testing.go:149: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.374Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: entering follower state: follower="Node at cc3c790b-3981-510f-d759-353e8ab43bff [Follower]" leader= -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.374Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: entering follower state: follower="Node at 138c9853-4660-c7de-1ab8-eed6328689ca [Follower]" leader= - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.374Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: entering follower state: follower="Node at 138c9853-4660-c7de-1ab8-eed6328689ca [Follower]" leader= -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [WARN] server-138c9853-4660-c7de-1ab8-eed6328689ca: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [WARN] server-138c9853-4660-c7de-1ab8-eed6328689ca: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: entering candidate state: node="Node at 138c9853-4660-c7de-1ab8-eed6328689ca [Candidate]" term=2 - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: entering candidate state: node="Node at 138c9853-4660-c7de-1ab8-eed6328689ca [Candidate]" term=2 -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: election won: tally=2 - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: election won: tally=2 -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: entering leader state: leader="Node at 138c9853-4660-c7de-1ab8-eed6328689ca [Leader]" - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: entering leader state: leader="Node at 138c9853-4660-c7de-1ab8-eed6328689ca [Leader]" -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: added peer, starting replication: peer=server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: added peer, starting replication: peer=server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.452Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: added peer, starting replication: peer=server-cc3c790b-3981-510f-d759-353e8ab43bff - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.452Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: added peer, starting replication: peer=server-cc3c790b-3981-510f-d759-353e8ab43bff -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.452Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: pipelining replication: peer="{Voter server-cc3c790b-3981-510f-d759-353e8ab43bff cc3c790b-3981-510f-d759-353e8ab43bff}" - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.452Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: pipelining replication: peer="{Voter server-cc3c790b-3981-510f-d759-353e8ab43bff cc3c790b-3981-510f-d759-353e8ab43bff}" -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.452Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: pipelining replication: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.452Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: pipelining replication: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.557Z [INFO] stable state for Follower reached at 2024-01-09 05:10:01.452263018 +0000 UTC m=+12.705313991 (2 nodes), 77.905776ms from start of poll, 78.631545ms from cluster start. Timeout at 2024-01-09 05:10:01.557352629 +0000 UTC m=+12.810403601, 105.08961ms after stability - testing.go:149: cluster: 2024-01-09T05:10:01.557Z [INFO] stable state for Follower reached at 2024-01-09 05:10:01.452263018 +0000 UTC m=+12.705313991 (2 nodes), 77.905776ms from start of poll, 78.631545ms from cluster start. Timeout at 2024-01-09 05:10:01.557352629 +0000 UTC m=+12.810403601, 105.08961ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.557Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:01.557Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.662Z [INFO] stable state for Leader reached at 2024-01-09 05:10:01.557521543 +0000 UTC m=+12.810572501 (1 nodes), 1.867µs from start of poll, 183.890055ms from cluster start. Timeout at 2024-01-09 05:10:01.662672805 +0000 UTC m=+12.915723785, 105.151284ms after stability - testing.go:149: cluster: 2024-01-09T05:10:01.662Z [INFO] stable state for Leader reached at 2024-01-09 05:10:01.557521543 +0000 UTC m=+12.810572501 (1 nodes), 1.867µs from start of poll, 183.890055ms from cluster start. Timeout at 2024-01-09 05:10:01.662672805 +0000 UTC m=+12.915723785, 105.151284ms after stability -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.662Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to pipeline appendEntries: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="append pipeline closed" - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.662Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to pipeline appendEntries: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="append pipeline closed" -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.662Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: aborting pipeline replication: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.662Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: aborting pipeline replication: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.662Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to appendEntries to: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.662Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to appendEntries to: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.668Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to heartbeat to: peer=cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.668Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to heartbeat to: peer=cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.673Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to appendEntries to: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.673Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to appendEntries to: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" +testLoggerAdapter verbose: server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: 2025-02-10T11:34:15.568Z [INFO] server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db9355e2-3f83-6a49-22a3-3b602f6e05ba Address:db9355e2-3f83-6a49-22a3-3b602f6e05ba} {Suffrage:Voter ID:server-cf86f427-6a60-2994-9d2a-f75de2e456ad Address:cf86f427-6a60-2994-9d2a-f75de2e456ad} {Suffrage:Voter ID:server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47 Address:2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47}]" + testing.go:149: server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: 2025-02-10T11:34:15.568Z [INFO] server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db9355e2-3f83-6a49-22a3-3b602f6e05ba Address:db9355e2-3f83-6a49-22a3-3b602f6e05ba} {Suffrage:Voter ID:server-cf86f427-6a60-2994-9d2a-f75de2e456ad Address:cf86f427-6a60-2994-9d2a-f75de2e456ad} {Suffrage:Voter ID:server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47 Address:2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47}]" +testLoggerAdapter verbose: server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: 2025-02-10T11:34:15.568Z [INFO] server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: entering follower state: follower="Node at db9355e2-3f83-6a49-22a3-3b602f6e05ba [Follower]" leader= + testing.go:149: server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: 2025-02-10T11:34:15.568Z [INFO] server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: entering follower state: follower="Node at db9355e2-3f83-6a49-22a3-3b602f6e05ba [Follower]" leader= +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.568Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db9355e2-3f83-6a49-22a3-3b602f6e05ba Address:db9355e2-3f83-6a49-22a3-3b602f6e05ba} {Suffrage:Voter ID:server-cf86f427-6a60-2994-9d2a-f75de2e456ad Address:cf86f427-6a60-2994-9d2a-f75de2e456ad} {Suffrage:Voter ID:server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47 Address:2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47}]" + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.568Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db9355e2-3f83-6a49-22a3-3b602f6e05ba Address:db9355e2-3f83-6a49-22a3-3b602f6e05ba} {Suffrage:Voter ID:server-cf86f427-6a60-2994-9d2a-f75de2e456ad Address:cf86f427-6a60-2994-9d2a-f75de2e456ad} {Suffrage:Voter ID:server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47 Address:2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47}]" +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.568Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: entering follower state: follower="Node at cf86f427-6a60-2994-9d2a-f75de2e456ad [Follower]" leader= + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.568Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: entering follower state: follower="Node at cf86f427-6a60-2994-9d2a-f75de2e456ad [Follower]" leader= +testLoggerAdapter verbose: server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: 2025-02-10T11:34:15.568Z [INFO] server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db9355e2-3f83-6a49-22a3-3b602f6e05ba Address:db9355e2-3f83-6a49-22a3-3b602f6e05ba} {Suffrage:Voter ID:server-cf86f427-6a60-2994-9d2a-f75de2e456ad Address:cf86f427-6a60-2994-9d2a-f75de2e456ad} {Suffrage:Voter ID:server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47 Address:2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47}]" + testing.go:149: server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: 2025-02-10T11:34:15.568Z [INFO] server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db9355e2-3f83-6a49-22a3-3b602f6e05ba Address:db9355e2-3f83-6a49-22a3-3b602f6e05ba} {Suffrage:Voter ID:server-cf86f427-6a60-2994-9d2a-f75de2e456ad Address:cf86f427-6a60-2994-9d2a-f75de2e456ad} {Suffrage:Voter ID:server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47 Address:2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:15.568Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:15.568Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: 2025-02-10T11:34:15.568Z [INFO] server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: entering follower state: follower="Node at 2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47 [Follower]" leader= + testing.go:149: server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: 2025-02-10T11:34:15.568Z [INFO] server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: entering follower state: follower="Node at 2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47 [Follower]" leader= +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.636Z [WARN] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.636Z [WARN] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.636Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: entering candidate state: node="Node at cf86f427-6a60-2994-9d2a-f75de2e456ad [Candidate]" term=2 + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.636Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: entering candidate state: node="Node at cf86f427-6a60-2994-9d2a-f75de2e456ad [Candidate]" term=2 +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.636Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: election won: tally=2 + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.636Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: election won: tally=2 +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.636Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: entering leader state: leader="Node at cf86f427-6a60-2994-9d2a-f75de2e456ad [Leader]" + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.636Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: entering leader state: leader="Node at cf86f427-6a60-2994-9d2a-f75de2e456ad [Leader]" +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.636Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: added peer, starting replication: peer=server-db9355e2-3f83-6a49-22a3-3b602f6e05ba + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.636Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: added peer, starting replication: peer=server-db9355e2-3f83-6a49-22a3-3b602f6e05ba +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.636Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: added peer, starting replication: peer=server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47 + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.636Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: added peer, starting replication: peer=server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47 +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.636Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: pipelining replication: peer="{Voter server-db9355e2-3f83-6a49-22a3-3b602f6e05ba db9355e2-3f83-6a49-22a3-3b602f6e05ba}" + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.636Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: pipelining replication: peer="{Voter server-db9355e2-3f83-6a49-22a3-3b602f6e05ba db9355e2-3f83-6a49-22a3-3b602f6e05ba}" +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.636Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: pipelining replication: peer="{Voter server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47 2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47}" + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.636Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: pipelining replication: peer="{Voter server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47 2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:15.741Z [INFO] stable state for Follower reached at 2025-02-10 11:34:15.636228634 +0000 UTC m=+12.005132950 (2 nodes), 67.272781ms from start of poll, 67.652741ms from cluster start. Timeout at 2025-02-10 11:34:15.741305528 +0000 UTC m=+12.110209822, 105.076872ms after stability + testing.go:149: cluster: 2025-02-10T11:34:15.741Z [INFO] stable state for Follower reached at 2025-02-10 11:34:15.636228634 +0000 UTC m=+12.005132950 (2 nodes), 67.272781ms from start of poll, 67.652741ms from cluster start. Timeout at 2025-02-10 11:34:15.741305528 +0000 UTC m=+12.110209822, 105.076872ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:15.741Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:15.741Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:15.846Z [INFO] stable state for Leader reached at 2025-02-10 11:34:15.741376153 +0000 UTC m=+12.110280458 (1 nodes), 461ns from start of poll, 172.800249ms from cluster start. Timeout at 2025-02-10 11:34:15.846451158 +0000 UTC m=+12.215355454, 105.074996ms after stability + testing.go:149: cluster: 2025-02-10T11:34:15.846Z [INFO] stable state for Leader reached at 2025-02-10 11:34:15.741376153 +0000 UTC m=+12.110280458 (1 nodes), 461ns from start of poll, 172.800249ms from cluster start. Timeout at 2025-02-10 11:34:15.846451158 +0000 UTC m=+12.215355454, 105.074996ms after stability +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.846Z [ERROR] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to pipeline appendEntries: peer="{Voter server-db9355e2-3f83-6a49-22a3-3b602f6e05ba db9355e2-3f83-6a49-22a3-3b602f6e05ba}" error="append pipeline closed" + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.846Z [ERROR] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to pipeline appendEntries: peer="{Voter server-db9355e2-3f83-6a49-22a3-3b602f6e05ba db9355e2-3f83-6a49-22a3-3b602f6e05ba}" error="append pipeline closed" +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.846Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: aborting pipeline replication: peer="{Voter server-db9355e2-3f83-6a49-22a3-3b602f6e05ba db9355e2-3f83-6a49-22a3-3b602f6e05ba}" + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.846Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: aborting pipeline replication: peer="{Voter server-db9355e2-3f83-6a49-22a3-3b602f6e05ba db9355e2-3f83-6a49-22a3-3b602f6e05ba}" +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.846Z [ERROR] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to appendEntries to: peer="{Voter server-db9355e2-3f83-6a49-22a3-3b602f6e05ba db9355e2-3f83-6a49-22a3-3b602f6e05ba}" error="failed to connect to peer: db9355e2-3f83-6a49-22a3-3b602f6e05ba" + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.846Z [ERROR] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to appendEntries to: peer="{Voter server-db9355e2-3f83-6a49-22a3-3b602f6e05ba db9355e2-3f83-6a49-22a3-3b602f6e05ba}" error="failed to connect to peer: db9355e2-3f83-6a49-22a3-3b602f6e05ba" raft_test.go:1320: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.676Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: starting snapshot up to: index=102 - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.676Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: starting snapshot up to: index=102 - testing.go:153: 2024-01-09T05:10:01.676Z [INFO] creating new snapshot: path=/tmp/raft708945334/snapshots/2-102-1704777001676.tmp -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.683Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to appendEntries to: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.683Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to appendEntries to: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.689Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to heartbeat to: peer=cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.689Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to heartbeat to: peer=cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.706Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to heartbeat to: peer=cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.706Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to heartbeat to: peer=cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.708Z [WARN] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to contact: server-id=server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc time=50.064496ms - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.708Z [WARN] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to contact: server-id=server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc time=50.064496ms -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.712Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to appendEntries to: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.712Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to appendEntries to: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.718Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: compacting logs: from=1 to=92 - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.718Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.718Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: snapshot complete up to: index=102 - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.718Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.718Z [ERROR] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: failed to take snapshot: error="nothing new to snapshot" - testing.go:149: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.718Z [ERROR] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: failed to take snapshot: error="nothing new to snapshot" -testLoggerAdapter verbose: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.718Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: starting snapshot up to: index=102 - testing.go:149: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.718Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: starting snapshot up to: index=102 - testing.go:153: 2024-01-09T05:10:01.718Z [INFO] creating new snapshot: path=/tmp/raft250731850/snapshots/2-102-1704777001718.tmp -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.736Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to heartbeat to: peer=cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.736Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to heartbeat to: peer=cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" -testLoggerAdapter verbose: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.756Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: compacting logs: from=1 to=92 - testing.go:149: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.756Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.756Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: snapshot complete up to: index=102 - testing.go:149: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.756Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.756Z [WARN] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to contact: server-id=server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc time=97.870203ms - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.756Z [WARN] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to contact: server-id=server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc time=97.870203ms -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.761Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to get log: index=2 error="log not found" - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.761Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to get log: index=2 error="log not found" - testing.go:153: 2024-01-09T05:10:01.761Z [INFO] creating new snapshot: path=/tmp/raft680109656/snapshots/2-102-1704777001761.tmp -testLoggerAdapter verbose: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.769Z [INFO] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: copied to local snapshot: bytes=693 - testing.go:149: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.769Z [INFO] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: copied to local snapshot: bytes=693 -testLoggerAdapter verbose: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.769Z [INFO] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: Installed remote snapshot - testing.go:149: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.769Z [INFO] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: Installed remote snapshot -testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.771Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: aborting pipeline replication: peer="{Voter server-cc3c790b-3981-510f-d759-353e8ab43bff cc3c790b-3981-510f-d759-353e8ab43bff}" - testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.771Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: aborting pipeline replication: peer="{Voter server-cc3c790b-3981-510f-d759-353e8ab43bff cc3c790b-3981-510f-d759-353e8ab43bff}" ---- PASS: TestRaft_SendSnapshotFollower (0.40s) +testLoggerAdapter verbose: server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: 2025-02-10T11:34:15.847Z [ERROR] server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: failed to take snapshot: error="nothing new to snapshot" + testing.go:149: server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: 2025-02-10T11:34:15.847Z [ERROR] server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: failed to take snapshot: error="nothing new to snapshot" +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.847Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: starting snapshot up to: index=102 + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.847Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: starting snapshot up to: index=102 + testing.go:153: 2025-02-10T11:34:15.847Z [INFO] creating new snapshot: path=/tmp/raft659035121/snapshots/2-102-1739187255847.tmp +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.851Z [ERROR] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to heartbeat to: peer=db9355e2-3f83-6a49-22a3-3b602f6e05ba error="failed to connect to peer: db9355e2-3f83-6a49-22a3-3b602f6e05ba" + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.851Z [ERROR] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to heartbeat to: peer=db9355e2-3f83-6a49-22a3-3b602f6e05ba error="failed to connect to peer: db9355e2-3f83-6a49-22a3-3b602f6e05ba" +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.857Z [ERROR] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to appendEntries to: peer="{Voter server-db9355e2-3f83-6a49-22a3-3b602f6e05ba db9355e2-3f83-6a49-22a3-3b602f6e05ba}" error="failed to connect to peer: db9355e2-3f83-6a49-22a3-3b602f6e05ba" + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.857Z [ERROR] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to appendEntries to: peer="{Voter server-db9355e2-3f83-6a49-22a3-3b602f6e05ba db9355e2-3f83-6a49-22a3-3b602f6e05ba}" error="failed to connect to peer: db9355e2-3f83-6a49-22a3-3b602f6e05ba" +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.867Z [ERROR] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to appendEntries to: peer="{Voter server-db9355e2-3f83-6a49-22a3-3b602f6e05ba db9355e2-3f83-6a49-22a3-3b602f6e05ba}" error="failed to connect to peer: db9355e2-3f83-6a49-22a3-3b602f6e05ba" + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.867Z [ERROR] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to appendEntries to: peer="{Voter server-db9355e2-3f83-6a49-22a3-3b602f6e05ba db9355e2-3f83-6a49-22a3-3b602f6e05ba}" error="failed to connect to peer: db9355e2-3f83-6a49-22a3-3b602f6e05ba" +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.871Z [ERROR] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to heartbeat to: peer=db9355e2-3f83-6a49-22a3-3b602f6e05ba error="failed to connect to peer: db9355e2-3f83-6a49-22a3-3b602f6e05ba" + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.871Z [ERROR] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to heartbeat to: peer=db9355e2-3f83-6a49-22a3-3b602f6e05ba error="failed to connect to peer: db9355e2-3f83-6a49-22a3-3b602f6e05ba" +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.887Z [ERROR] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to heartbeat to: peer=db9355e2-3f83-6a49-22a3-3b602f6e05ba error="failed to connect to peer: db9355e2-3f83-6a49-22a3-3b602f6e05ba" + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.887Z [ERROR] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to heartbeat to: peer=db9355e2-3f83-6a49-22a3-3b602f6e05ba error="failed to connect to peer: db9355e2-3f83-6a49-22a3-3b602f6e05ba" +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.890Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: compacting logs: from=1 to=92 + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.890Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.890Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: snapshot complete up to: index=102 + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.890Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: 2025-02-10T11:34:15.890Z [INFO] server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: starting snapshot up to: index=102 + testing.go:149: server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: 2025-02-10T11:34:15.890Z [INFO] server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: starting snapshot up to: index=102 + testing.go:153: 2025-02-10T11:34:15.890Z [INFO] creating new snapshot: path=/tmp/raft652765515/snapshots/2-102-1739187255890.tmp +testLoggerAdapter verbose: server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: 2025-02-10T11:34:15.893Z [INFO] server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: compacting logs: from=1 to=92 + testing.go:149: server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: 2025-02-10T11:34:15.893Z [INFO] server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: 2025-02-10T11:34:15.893Z [INFO] server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: snapshot complete up to: index=102 + testing.go:149: server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: 2025-02-10T11:34:15.893Z [INFO] server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.896Z [WARN] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to contact: server-id=server-db9355e2-3f83-6a49-22a3-3b602f6e05ba time=50.10504ms + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.896Z [WARN] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to contact: server-id=server-db9355e2-3f83-6a49-22a3-3b602f6e05ba time=50.10504ms +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.897Z [ERROR] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to get log: index=2 error="log not found" + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.897Z [ERROR] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: failed to get log: index=2 error="log not found" + testing.go:153: 2025-02-10T11:34:15.898Z [INFO] creating new snapshot: path=/tmp/raft056120615/snapshots/2-102-1739187255898.tmp +testLoggerAdapter verbose: server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: 2025-02-10T11:34:15.901Z [INFO] server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: copied to local snapshot: bytes=693 + testing.go:149: server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: 2025-02-10T11:34:15.901Z [INFO] server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: copied to local snapshot: bytes=693 +testLoggerAdapter verbose: server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: 2025-02-10T11:34:15.901Z [INFO] server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: Installed remote snapshot + testing.go:149: server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: 2025-02-10T11:34:15.901Z [INFO] server-db9355e2-3f83-6a49-22a3-3b602f6e05ba: Installed remote snapshot +testLoggerAdapter verbose: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.903Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: aborting pipeline replication: peer="{Voter server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47 2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47}" + testing.go:149: server-cf86f427-6a60-2994-9d2a-f75de2e456ad: 2025-02-10T11:34:15.903Z [INFO] server-cf86f427-6a60-2994-9d2a-f75de2e456ad: aborting pipeline replication: peer="{Voter server-2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47 2cc1a1cc-cc0e-e376-718a-d54a1d9c7d47}" +--- PASS: TestRaft_SendSnapshotFollower (0.34s) === RUN TestRaft_SendSnapshotAndLogsFollower -testLoggerAdapter verbose: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:01.775Z [INFO] server-019f2655-7706-c399-525b-39ceb5b19abb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-019f2655-7706-c399-525b-39ceb5b19abb Address:019f2655-7706-c399-525b-39ceb5b19abb} {Suffrage:Voter ID:server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6 Address:2a03ca26-444b-58cf-c7ba-203a0d05a1b6} {Suffrage:Voter ID:server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 Address:cf5bea6e-9ea4-4c40-0729-d455d24e67c8}]" - testing.go:149: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:01.775Z [INFO] server-019f2655-7706-c399-525b-39ceb5b19abb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-019f2655-7706-c399-525b-39ceb5b19abb Address:019f2655-7706-c399-525b-39ceb5b19abb} {Suffrage:Voter ID:server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6 Address:2a03ca26-444b-58cf-c7ba-203a0d05a1b6} {Suffrage:Voter ID:server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 Address:cf5bea6e-9ea4-4c40-0729-d455d24e67c8}]" -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.776Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-019f2655-7706-c399-525b-39ceb5b19abb Address:019f2655-7706-c399-525b-39ceb5b19abb} {Suffrage:Voter ID:server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6 Address:2a03ca26-444b-58cf-c7ba-203a0d05a1b6} {Suffrage:Voter ID:server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 Address:cf5bea6e-9ea4-4c40-0729-d455d24e67c8}]" - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.776Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-019f2655-7706-c399-525b-39ceb5b19abb Address:019f2655-7706-c399-525b-39ceb5b19abb} {Suffrage:Voter ID:server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6 Address:2a03ca26-444b-58cf-c7ba-203a0d05a1b6} {Suffrage:Voter ID:server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 Address:cf5bea6e-9ea4-4c40-0729-d455d24e67c8}]" -testLoggerAdapter verbose: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:01.776Z [INFO] server-019f2655-7706-c399-525b-39ceb5b19abb: entering follower state: follower="Node at 019f2655-7706-c399-525b-39ceb5b19abb [Follower]" leader= -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.776Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: entering follower state: follower="Node at 2a03ca26-444b-58cf-c7ba-203a0d05a1b6 [Follower]" leader= - testing.go:149: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:01.776Z [INFO] server-019f2655-7706-c399-525b-39ceb5b19abb: entering follower state: follower="Node at 019f2655-7706-c399-525b-39ceb5b19abb [Follower]" leader= - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.776Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: entering follower state: follower="Node at 2a03ca26-444b-58cf-c7ba-203a0d05a1b6 [Follower]" leader= -testLoggerAdapter verbose: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:01.776Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-019f2655-7706-c399-525b-39ceb5b19abb Address:019f2655-7706-c399-525b-39ceb5b19abb} {Suffrage:Voter ID:server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6 Address:2a03ca26-444b-58cf-c7ba-203a0d05a1b6} {Suffrage:Voter ID:server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 Address:cf5bea6e-9ea4-4c40-0729-d455d24e67c8}]" - testing.go:149: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:01.776Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-019f2655-7706-c399-525b-39ceb5b19abb Address:019f2655-7706-c399-525b-39ceb5b19abb} {Suffrage:Voter ID:server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6 Address:2a03ca26-444b-58cf-c7ba-203a0d05a1b6} {Suffrage:Voter ID:server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 Address:cf5bea6e-9ea4-4c40-0729-d455d24e67c8}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.777Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:01.777Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: entering follower state: follower="Node at cf5bea6e-9ea4-4c40-0729-d455d24e67c8 [Follower]" leader= - testing.go:149: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:01.777Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: entering follower state: follower="Node at cf5bea6e-9ea4-4c40-0729-d455d24e67c8 [Follower]" leader= - testing.go:149: cluster: 2024-01-09T05:10:01.777Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [WARN] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [WARN] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: entering candidate state: node="Node at 2a03ca26-444b-58cf-c7ba-203a0d05a1b6 [Candidate]" term=2 - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: entering candidate state: node="Node at 2a03ca26-444b-58cf-c7ba-203a0d05a1b6 [Candidate]" term=2 -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: election won: tally=2 - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: election won: tally=2 -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: entering leader state: leader="Node at 2a03ca26-444b-58cf-c7ba-203a0d05a1b6 [Leader]" - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: entering leader state: leader="Node at 2a03ca26-444b-58cf-c7ba-203a0d05a1b6 [Leader]" -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: added peer, starting replication: peer=server-019f2655-7706-c399-525b-39ceb5b19abb - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: added peer, starting replication: peer=server-019f2655-7706-c399-525b-39ceb5b19abb -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: added peer, starting replication: peer=server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: added peer, starting replication: peer=server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: pipelining replication: peer="{Voter server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 cf5bea6e-9ea4-4c40-0729-d455d24e67c8}" - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: pipelining replication: peer="{Voter server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 cf5bea6e-9ea4-4c40-0729-d455d24e67c8}" -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.855Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: pipelining replication: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.855Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: pipelining replication: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.960Z [INFO] stable state for Follower reached at 2024-01-09 05:10:01.855223132 +0000 UTC m=+13.108274199 (2 nodes), 77.892793ms from start of poll, 79.728102ms from cluster start. Timeout at 2024-01-09 05:10:01.960328179 +0000 UTC m=+13.213379289, 105.10509ms after stability - testing.go:149: cluster: 2024-01-09T05:10:01.960Z [INFO] stable state for Follower reached at 2024-01-09 05:10:01.855223132 +0000 UTC m=+13.108274199 (2 nodes), 77.892793ms from start of poll, 79.728102ms from cluster start. Timeout at 2024-01-09 05:10:01.960328179 +0000 UTC m=+13.213379289, 105.10509ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.960Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:01.960Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.065Z [INFO] stable state for Leader reached at 2024-01-09 05:10:01.960492513 +0000 UTC m=+13.213543557 (1 nodes), 2.405µs from start of poll, 184.99746ms from cluster start. Timeout at 2024-01-09 05:10:02.065598472 +0000 UTC m=+13.318649667, 105.10611ms after stability - testing.go:149: cluster: 2024-01-09T05:10:02.065Z [INFO] stable state for Leader reached at 2024-01-09 05:10:01.960492513 +0000 UTC m=+13.213543557 (1 nodes), 2.405µs from start of poll, 184.99746ms from cluster start. Timeout at 2024-01-09 05:10:02.065598472 +0000 UTC m=+13.318649667, 105.10611ms after stability -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.065Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to pipeline appendEntries: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" error="append pipeline closed" - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.065Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to pipeline appendEntries: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" error="append pipeline closed" -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.065Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: aborting pipeline replication: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.065Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: aborting pipeline replication: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.065Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to appendEntries to: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.065Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to appendEntries to: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.070Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to heartbeat to: peer=019f2655-7706-c399-525b-39ceb5b19abb error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.070Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to heartbeat to: peer=019f2655-7706-c399-525b-39ceb5b19abb error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.076Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to appendEntries to: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.076Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to appendEntries to: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.905Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180b40c2-ef69-a3ce-ffbb-997856cc17cc Address:180b40c2-ef69-a3ce-ffbb-997856cc17cc} {Suffrage:Voter ID:server-a80fefcc-48e7-f68a-31a7-2757e3746482 Address:a80fefcc-48e7-f68a-31a7-2757e3746482} {Suffrage:Voter ID:server-030319e1-fd9e-1785-6008-7d4008060bfe Address:030319e1-fd9e-1785-6008-7d4008060bfe}]" + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.905Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180b40c2-ef69-a3ce-ffbb-997856cc17cc Address:180b40c2-ef69-a3ce-ffbb-997856cc17cc} {Suffrage:Voter ID:server-a80fefcc-48e7-f68a-31a7-2757e3746482 Address:a80fefcc-48e7-f68a-31a7-2757e3746482} {Suffrage:Voter ID:server-030319e1-fd9e-1785-6008-7d4008060bfe Address:030319e1-fd9e-1785-6008-7d4008060bfe}]" +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.905Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: entering follower state: follower="Node at 180b40c2-ef69-a3ce-ffbb-997856cc17cc [Follower]" leader= + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.905Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: entering follower state: follower="Node at 180b40c2-ef69-a3ce-ffbb-997856cc17cc [Follower]" leader= +testLoggerAdapter verbose: server-a80fefcc-48e7-f68a-31a7-2757e3746482: 2025-02-10T11:34:15.905Z [INFO] server-a80fefcc-48e7-f68a-31a7-2757e3746482: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180b40c2-ef69-a3ce-ffbb-997856cc17cc Address:180b40c2-ef69-a3ce-ffbb-997856cc17cc} {Suffrage:Voter ID:server-a80fefcc-48e7-f68a-31a7-2757e3746482 Address:a80fefcc-48e7-f68a-31a7-2757e3746482} {Suffrage:Voter ID:server-030319e1-fd9e-1785-6008-7d4008060bfe Address:030319e1-fd9e-1785-6008-7d4008060bfe}]" + testing.go:149: server-a80fefcc-48e7-f68a-31a7-2757e3746482: 2025-02-10T11:34:15.905Z [INFO] server-a80fefcc-48e7-f68a-31a7-2757e3746482: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180b40c2-ef69-a3ce-ffbb-997856cc17cc Address:180b40c2-ef69-a3ce-ffbb-997856cc17cc} {Suffrage:Voter ID:server-a80fefcc-48e7-f68a-31a7-2757e3746482 Address:a80fefcc-48e7-f68a-31a7-2757e3746482} {Suffrage:Voter ID:server-030319e1-fd9e-1785-6008-7d4008060bfe Address:030319e1-fd9e-1785-6008-7d4008060bfe}]" +testLoggerAdapter verbose: server-a80fefcc-48e7-f68a-31a7-2757e3746482: 2025-02-10T11:34:15.905Z [INFO] server-a80fefcc-48e7-f68a-31a7-2757e3746482: entering follower state: follower="Node at a80fefcc-48e7-f68a-31a7-2757e3746482 [Follower]" leader= +testLoggerAdapter verbose: server-030319e1-fd9e-1785-6008-7d4008060bfe: 2025-02-10T11:34:15.905Z [INFO] server-030319e1-fd9e-1785-6008-7d4008060bfe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180b40c2-ef69-a3ce-ffbb-997856cc17cc Address:180b40c2-ef69-a3ce-ffbb-997856cc17cc} {Suffrage:Voter ID:server-a80fefcc-48e7-f68a-31a7-2757e3746482 Address:a80fefcc-48e7-f68a-31a7-2757e3746482} {Suffrage:Voter ID:server-030319e1-fd9e-1785-6008-7d4008060bfe Address:030319e1-fd9e-1785-6008-7d4008060bfe}]" + testing.go:149: server-a80fefcc-48e7-f68a-31a7-2757e3746482: 2025-02-10T11:34:15.905Z [INFO] server-a80fefcc-48e7-f68a-31a7-2757e3746482: entering follower state: follower="Node at a80fefcc-48e7-f68a-31a7-2757e3746482 [Follower]" leader= + testing.go:149: server-030319e1-fd9e-1785-6008-7d4008060bfe: 2025-02-10T11:34:15.905Z [INFO] server-030319e1-fd9e-1785-6008-7d4008060bfe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180b40c2-ef69-a3ce-ffbb-997856cc17cc Address:180b40c2-ef69-a3ce-ffbb-997856cc17cc} {Suffrage:Voter ID:server-a80fefcc-48e7-f68a-31a7-2757e3746482 Address:a80fefcc-48e7-f68a-31a7-2757e3746482} {Suffrage:Voter ID:server-030319e1-fd9e-1785-6008-7d4008060bfe Address:030319e1-fd9e-1785-6008-7d4008060bfe}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:15.906Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:15.906Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-030319e1-fd9e-1785-6008-7d4008060bfe: 2025-02-10T11:34:15.906Z [INFO] server-030319e1-fd9e-1785-6008-7d4008060bfe: entering follower state: follower="Node at 030319e1-fd9e-1785-6008-7d4008060bfe [Follower]" leader= + testing.go:149: server-030319e1-fd9e-1785-6008-7d4008060bfe: 2025-02-10T11:34:15.906Z [INFO] server-030319e1-fd9e-1785-6008-7d4008060bfe: entering follower state: follower="Node at 030319e1-fd9e-1785-6008-7d4008060bfe [Follower]" leader= +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.957Z [WARN] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.957Z [WARN] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.957Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: entering candidate state: node="Node at 180b40c2-ef69-a3ce-ffbb-997856cc17cc [Candidate]" term=2 + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.957Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: entering candidate state: node="Node at 180b40c2-ef69-a3ce-ffbb-997856cc17cc [Candidate]" term=2 +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.957Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: election won: tally=2 + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.957Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: election won: tally=2 +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.957Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: entering leader state: leader="Node at 180b40c2-ef69-a3ce-ffbb-997856cc17cc [Leader]" + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.957Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: entering leader state: leader="Node at 180b40c2-ef69-a3ce-ffbb-997856cc17cc [Leader]" +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.957Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: added peer, starting replication: peer=server-a80fefcc-48e7-f68a-31a7-2757e3746482 + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.957Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: added peer, starting replication: peer=server-a80fefcc-48e7-f68a-31a7-2757e3746482 +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.957Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: added peer, starting replication: peer=server-030319e1-fd9e-1785-6008-7d4008060bfe + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.957Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: added peer, starting replication: peer=server-030319e1-fd9e-1785-6008-7d4008060bfe +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.957Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: pipelining replication: peer="{Voter server-a80fefcc-48e7-f68a-31a7-2757e3746482 a80fefcc-48e7-f68a-31a7-2757e3746482}" + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.957Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: pipelining replication: peer="{Voter server-a80fefcc-48e7-f68a-31a7-2757e3746482 a80fefcc-48e7-f68a-31a7-2757e3746482}" +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.957Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: pipelining replication: peer="{Voter server-030319e1-fd9e-1785-6008-7d4008060bfe 030319e1-fd9e-1785-6008-7d4008060bfe}" + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:15.957Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: pipelining replication: peer="{Voter server-030319e1-fd9e-1785-6008-7d4008060bfe 030319e1-fd9e-1785-6008-7d4008060bfe}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:16.062Z [INFO] stable state for Follower reached at 2025-02-10 11:34:15.957408874 +0000 UTC m=+12.326313201 (2 nodes), 51.349421ms from start of poll, 52.032449ms from cluster start. Timeout at 2025-02-10 11:34:16.06248323 +0000 UTC m=+12.431387526, 105.074325ms after stability + testing.go:149: cluster: 2025-02-10T11:34:16.062Z [INFO] stable state for Follower reached at 2025-02-10 11:34:15.957408874 +0000 UTC m=+12.326313201 (2 nodes), 51.349421ms from start of poll, 52.032449ms from cluster start. Timeout at 2025-02-10 11:34:16.06248323 +0000 UTC m=+12.431387526, 105.074325ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:16.062Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:16.062Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:16.167Z [INFO] stable state for Leader reached at 2025-02-10 11:34:16.062560361 +0000 UTC m=+12.431464670 (1 nodes), 537ns from start of poll, 157.183918ms from cluster start. Timeout at 2025-02-10 11:34:16.167663939 +0000 UTC m=+12.536568292, 105.103622ms after stability + testing.go:149: cluster: 2025-02-10T11:34:16.167Z [INFO] stable state for Leader reached at 2025-02-10 11:34:16.062560361 +0000 UTC m=+12.431464670 (1 nodes), 537ns from start of poll, 157.183918ms from cluster start. Timeout at 2025-02-10 11:34:16.167663939 +0000 UTC m=+12.536568292, 105.103622ms after stability +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.167Z [ERROR] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: failed to pipeline appendEntries: peer="{Voter server-a80fefcc-48e7-f68a-31a7-2757e3746482 a80fefcc-48e7-f68a-31a7-2757e3746482}" error="append pipeline closed" + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.167Z [ERROR] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: failed to pipeline appendEntries: peer="{Voter server-a80fefcc-48e7-f68a-31a7-2757e3746482 a80fefcc-48e7-f68a-31a7-2757e3746482}" error="append pipeline closed" +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.168Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: aborting pipeline replication: peer="{Voter server-a80fefcc-48e7-f68a-31a7-2757e3746482 a80fefcc-48e7-f68a-31a7-2757e3746482}" + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.168Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: aborting pipeline replication: peer="{Voter server-a80fefcc-48e7-f68a-31a7-2757e3746482 a80fefcc-48e7-f68a-31a7-2757e3746482}" +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.168Z [ERROR] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: failed to appendEntries to: peer="{Voter server-a80fefcc-48e7-f68a-31a7-2757e3746482 a80fefcc-48e7-f68a-31a7-2757e3746482}" error="failed to connect to peer: a80fefcc-48e7-f68a-31a7-2757e3746482" + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.168Z [ERROR] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: failed to appendEntries to: peer="{Voter server-a80fefcc-48e7-f68a-31a7-2757e3746482 a80fefcc-48e7-f68a-31a7-2757e3746482}" error="failed to connect to peer: a80fefcc-48e7-f68a-31a7-2757e3746482" +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.168Z [ERROR] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: failed to heartbeat to: peer=a80fefcc-48e7-f68a-31a7-2757e3746482 error="failed to connect to peer: a80fefcc-48e7-f68a-31a7-2757e3746482" + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.168Z [ERROR] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: failed to heartbeat to: peer=a80fefcc-48e7-f68a-31a7-2757e3746482 error="failed to connect to peer: a80fefcc-48e7-f68a-31a7-2757e3746482" raft_test.go:1362: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:02.076Z [ERROR] server-019f2655-7706-c399-525b-39ceb5b19abb: failed to take snapshot: error="nothing new to snapshot" - testing.go:149: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:02.076Z [ERROR] server-019f2655-7706-c399-525b-39ceb5b19abb: failed to take snapshot: error="nothing new to snapshot" -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.076Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: starting snapshot up to: index=102 - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.076Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: starting snapshot up to: index=102 - testing.go:153: 2024-01-09T05:10:02.076Z [INFO] creating new snapshot: path=/tmp/raft073156126/snapshots/2-102-1704777002076.tmp -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.083Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: compacting logs: from=1 to=92 - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.083Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.083Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: snapshot complete up to: index=102 - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.083Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:02.083Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: starting snapshot up to: index=87 - testing.go:149: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:02.083Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: starting snapshot up to: index=87 - testing.go:153: 2024-01-09T05:10:02.083Z [INFO] creating new snapshot: path=/tmp/raft911629056/snapshots/2-87-1704777002083.tmp -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.086Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to get log: index=2 error="log not found" - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.086Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to get log: index=2 error="log not found" -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.087Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to install snapshot: id=2-102-1704777002076 error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.087Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to install snapshot: id=2-102-1704777002076 error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.088Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to send snapshot to: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.088Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to send snapshot to: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.089Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to heartbeat to: peer=019f2655-7706-c399-525b-39ceb5b19abb error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.089Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to heartbeat to: peer=019f2655-7706-c399-525b-39ceb5b19abb error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" -testLoggerAdapter verbose: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:02.091Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: compacting logs: from=1 to=87 - testing.go:149: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:02.091Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: compacting logs: from=1 to=87 -testLoggerAdapter verbose: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:02.091Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: snapshot complete up to: index=87 - testing.go:149: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:02.091Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: snapshot complete up to: index=87 +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.168Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: starting snapshot up to: index=102 + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.168Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: starting snapshot up to: index=102 + testing.go:153: 2025-02-10T11:34:16.168Z [INFO] creating new snapshot: path=/tmp/raft508936117/snapshots/2-102-1739187256168.tmp +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.172Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: compacting logs: from=1 to=92 + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.172Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.172Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: snapshot complete up to: index=102 + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.172Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-a80fefcc-48e7-f68a-31a7-2757e3746482: 2025-02-10T11:34:16.172Z [ERROR] server-a80fefcc-48e7-f68a-31a7-2757e3746482: failed to take snapshot: error="nothing new to snapshot" + testing.go:149: server-a80fefcc-48e7-f68a-31a7-2757e3746482: 2025-02-10T11:34:16.172Z [ERROR] server-a80fefcc-48e7-f68a-31a7-2757e3746482: failed to take snapshot: error="nothing new to snapshot" +testLoggerAdapter verbose: server-030319e1-fd9e-1785-6008-7d4008060bfe: 2025-02-10T11:34:16.172Z [INFO] server-030319e1-fd9e-1785-6008-7d4008060bfe: starting snapshot up to: index=99 + testing.go:149: server-030319e1-fd9e-1785-6008-7d4008060bfe: 2025-02-10T11:34:16.172Z [INFO] server-030319e1-fd9e-1785-6008-7d4008060bfe: starting snapshot up to: index=99 + testing.go:153: 2025-02-10T11:34:16.172Z [INFO] creating new snapshot: path=/tmp/raft083431353/snapshots/2-99-1739187256172.tmp +testLoggerAdapter verbose: server-030319e1-fd9e-1785-6008-7d4008060bfe: 2025-02-10T11:34:16.175Z [INFO] server-030319e1-fd9e-1785-6008-7d4008060bfe: compacting logs: from=1 to=92 + testing.go:149: server-030319e1-fd9e-1785-6008-7d4008060bfe: 2025-02-10T11:34:16.175Z [INFO] server-030319e1-fd9e-1785-6008-7d4008060bfe: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-030319e1-fd9e-1785-6008-7d4008060bfe: 2025-02-10T11:34:16.175Z [INFO] server-030319e1-fd9e-1785-6008-7d4008060bfe: snapshot complete up to: index=99 + testing.go:149: server-030319e1-fd9e-1785-6008-7d4008060bfe: 2025-02-10T11:34:16.175Z [INFO] server-030319e1-fd9e-1785-6008-7d4008060bfe: snapshot complete up to: index=99 raft_test.go:1383: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.111Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to get log: index=2 error="log not found" - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.111Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to get log: index=2 error="log not found" - testing.go:153: 2024-01-09T05:10:02.112Z [INFO] creating new snapshot: path=/tmp/raft215901964/snapshots/2-102-1704777002112.tmp -testLoggerAdapter verbose: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:02.118Z [INFO] server-019f2655-7706-c399-525b-39ceb5b19abb: copied to local snapshot: bytes=693 - testing.go:149: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:02.118Z [INFO] server-019f2655-7706-c399-525b-39ceb5b19abb: copied to local snapshot: bytes=693 -testLoggerAdapter verbose: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:02.118Z [INFO] server-019f2655-7706-c399-525b-39ceb5b19abb: Installed remote snapshot - testing.go:149: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:02.118Z [INFO] server-019f2655-7706-c399-525b-39ceb5b19abb: Installed remote snapshot -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.118Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: pipelining replication: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.118Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: pipelining replication: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.131Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: aborting pipeline replication: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.131Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: aborting pipeline replication: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" -testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.131Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: aborting pipeline replication: peer="{Voter server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 cf5bea6e-9ea4-4c40-0729-d455d24e67c8}" - testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.131Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: aborting pipeline replication: peer="{Voter server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 cf5bea6e-9ea4-4c40-0729-d455d24e67c8}" ---- PASS: TestRaft_SendSnapshotAndLogsFollower (0.36s) +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.178Z [ERROR] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: failed to get log: index=2 error="log not found" + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.178Z [ERROR] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: failed to get log: index=2 error="log not found" + testing.go:153: 2025-02-10T11:34:16.178Z [INFO] creating new snapshot: path=/tmp/raft767681455/snapshots/2-102-1739187256178.tmp +testLoggerAdapter verbose: server-a80fefcc-48e7-f68a-31a7-2757e3746482: 2025-02-10T11:34:16.181Z [INFO] server-a80fefcc-48e7-f68a-31a7-2757e3746482: copied to local snapshot: bytes=693 + testing.go:149: server-a80fefcc-48e7-f68a-31a7-2757e3746482: 2025-02-10T11:34:16.181Z [INFO] server-a80fefcc-48e7-f68a-31a7-2757e3746482: copied to local snapshot: bytes=693 +testLoggerAdapter verbose: server-a80fefcc-48e7-f68a-31a7-2757e3746482: 2025-02-10T11:34:16.181Z [INFO] server-a80fefcc-48e7-f68a-31a7-2757e3746482: Installed remote snapshot + testing.go:149: server-a80fefcc-48e7-f68a-31a7-2757e3746482: 2025-02-10T11:34:16.181Z [INFO] server-a80fefcc-48e7-f68a-31a7-2757e3746482: Installed remote snapshot +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.182Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: pipelining replication: peer="{Voter server-a80fefcc-48e7-f68a-31a7-2757e3746482 a80fefcc-48e7-f68a-31a7-2757e3746482}" + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.182Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: pipelining replication: peer="{Voter server-a80fefcc-48e7-f68a-31a7-2757e3746482 a80fefcc-48e7-f68a-31a7-2757e3746482}" +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.187Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: aborting pipeline replication: peer="{Voter server-030319e1-fd9e-1785-6008-7d4008060bfe 030319e1-fd9e-1785-6008-7d4008060bfe}" + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.187Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: aborting pipeline replication: peer="{Voter server-030319e1-fd9e-1785-6008-7d4008060bfe 030319e1-fd9e-1785-6008-7d4008060bfe}" +testLoggerAdapter verbose: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.187Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: aborting pipeline replication: peer="{Voter server-a80fefcc-48e7-f68a-31a7-2757e3746482 a80fefcc-48e7-f68a-31a7-2757e3746482}" + testing.go:149: server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: 2025-02-10T11:34:16.187Z [INFO] server-180b40c2-ef69-a3ce-ffbb-997856cc17cc: aborting pipeline replication: peer="{Voter server-a80fefcc-48e7-f68a-31a7-2757e3746482 a80fefcc-48e7-f68a-31a7-2757e3746482}" +--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.28s) === RUN TestRaft_ReJoinFollower -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.133Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 Address:22f7ee5a-2e3f-78b5-a317-00ccd58a6c30} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1}]" - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.133Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 Address:22f7ee5a-2e3f-78b5-a317-00ccd58a6c30} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1}]" -testLoggerAdapter verbose: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.133Z [INFO] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 Address:22f7ee5a-2e3f-78b5-a317-00ccd58a6c30} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1}]" - testing.go:149: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.133Z [INFO] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 Address:22f7ee5a-2e3f-78b5-a317-00ccd58a6c30} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1}]" -testLoggerAdapter verbose: server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: 2024-01-09T05:10:02.133Z [INFO] server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 Address:22f7ee5a-2e3f-78b5-a317-00ccd58a6c30} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1}]" - testing.go:149: server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: 2024-01-09T05:10:02.133Z [INFO] server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 Address:22f7ee5a-2e3f-78b5-a317-00ccd58a6c30} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.133Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:02.133Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.133Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering follower state: follower="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Follower]" leader= - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.133Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering follower state: follower="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Follower]" leader= -testLoggerAdapter verbose: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.133Z [INFO] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: entering follower state: follower="Node at 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 [Follower]" leader= - testing.go:149: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.133Z [INFO] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: entering follower state: follower="Node at 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 [Follower]" leader= -testLoggerAdapter verbose: server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: 2024-01-09T05:10:02.133Z [INFO] server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: entering follower state: follower="Node at 005b5ae9-219d-6c40-ef9c-25f3ac9723e1 [Follower]" leader= - testing.go:149: server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: 2024-01-09T05:10:02.133Z [INFO] server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: entering follower state: follower="Node at 005b5ae9-219d-6c40-ef9c-25f3ac9723e1 [Follower]" leader= -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [WARN] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [WARN] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering candidate state: node="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Candidate]" term=2 - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering candidate state: node="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Candidate]" term=2 -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: election won: tally=2 - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: election won: tally=2 -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering leader state: leader="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Leader]" - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering leader state: leader="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Leader]" -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.203Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: pipelining replication: peer="{Voter server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 005b5ae9-219d-6c40-ef9c-25f3ac9723e1}" - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.203Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: pipelining replication: peer="{Voter server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 005b5ae9-219d-6c40-ef9c-25f3ac9723e1}" -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.203Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: pipelining replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.203Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: pipelining replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.308Z [INFO] stable state for Leader reached at 2024-01-09 05:10:02.203852697 +0000 UTC m=+13.456903662 (1 nodes), 70.195837ms from start of poll, 70.609195ms from cluster start. Timeout at 2024-01-09 05:10:02.308937202 +0000 UTC m=+13.561988189, 105.084527ms after stability - testing.go:149: cluster: 2024-01-09T05:10:02.308Z [INFO] stable state for Leader reached at 2024-01-09 05:10:02.203852697 +0000 UTC m=+13.456903662 (1 nodes), 70.195837ms from start of poll, 70.609195ms from cluster start. Timeout at 2024-01-09 05:10:02.308937202 +0000 UTC m=+13.561988189, 105.084527ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.314Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:10:02.314Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.419Z [INFO] stable state for Follower reached at 2024-01-09 05:10:02.314317619 +0000 UTC m=+13.567368576 (2 nodes), 2.362µs from start of poll, 181.074109ms from cluster start. Timeout at 2024-01-09 05:10:02.419325728 +0000 UTC m=+13.672376772, 105.008196ms after stability - testing.go:149: cluster: 2024-01-09T05:10:02.419Z [INFO] stable state for Follower reached at 2024-01-09 05:10:02.314317619 +0000 UTC m=+13.567368576 (2 nodes), 2.362µs from start of poll, 181.074109ms from cluster start. Timeout at 2024-01-09 05:10:02.419325728 +0000 UTC m=+13.672376772, 105.008196ms after stability -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.419Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: updating configuration: command=RemoveServer server-id=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 server-addr= servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1}]" - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.419Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: updating configuration: command=RemoveServer server-id=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 server-addr= servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1}]" -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.419Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: removed peer, stopping replication: peer=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 last-index=3 - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.419Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: removed peer, stopping replication: peer=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 last-index=3 -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.419Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: aborting pipeline replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.419Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: aborting pipeline replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" -testLoggerAdapter verbose: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.508Z [WARN] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: heartbeat timeout reached, starting election: last-leader=0dc470d6-a011-e4fd-046f-e7ec51b30d4b - testing.go:149: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.508Z [WARN] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: heartbeat timeout reached, starting election: last-leader=0dc470d6-a011-e4fd-046f-e7ec51b30d4b -testLoggerAdapter verbose: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.508Z [INFO] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: entering candidate state: node="Node at 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 [Candidate]" term=3 - testing.go:149: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.508Z [INFO] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: entering candidate state: node="Node at 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 [Candidate]" term=3 -testLoggerAdapter verbose: server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: 2024-01-09T05:10:02.508Z [WARN] server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: rejecting vote request since we have a leader: from=22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 leader=0dc470d6-a011-e4fd-046f-e7ec51b30d4b - testing.go:149: server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: 2024-01-09T05:10:02.508Z [WARN] server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: rejecting vote request since we have a leader: from=22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 leader=0dc470d6-a011-e4fd-046f-e7ec51b30d4b -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.508Z [WARN] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: rejecting vote request since we have a leader: from=22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 leader=0dc470d6-a011-e4fd-046f-e7ec51b30d4b - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.508Z [WARN] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: rejecting vote request since we have a leader: from=22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 leader=0dc470d6-a011-e4fd-046f-e7ec51b30d4b -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.525Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: updating configuration: command=AddStaging server-id=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 server-addr=22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1} {Suffrage:Voter ID:server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 Address:22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}]" - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.525Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: updating configuration: command=AddStaging server-id=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 server-addr=22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1} {Suffrage:Voter ID:server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 Address:22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}]" -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.525Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.525Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.526Z [ERROR] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: peer has newer term, stopping replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.526Z [ERROR] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: peer has newer term, stopping replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.526Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering follower state: follower="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Follower]" leader= - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.526Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering follower state: follower="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Follower]" leader= -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.526Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:02.526Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.526Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: aborting pipeline replication: peer="{Voter server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 005b5ae9-219d-6c40-ef9c-25f3ac9723e1}" - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.526Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: aborting pipeline replication: peer="{Voter server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 005b5ae9-219d-6c40-ef9c-25f3ac9723e1}" -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [WARN] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [WARN] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering candidate state: node="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Candidate]" term=3 - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering candidate state: node="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Candidate]" term=3 -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: election won: tally=2 - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: election won: tally=2 -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering leader state: leader="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Leader]" - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering leader state: leader="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Leader]" -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 -testLoggerAdapter verbose: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.591Z [WARN] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: failed to get previous log: previous-index=4 last-index=3 error="log not found" - testing.go:149: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.591Z [WARN] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: failed to get previous log: previous-index=4 last-index=3 error="log not found" -testLoggerAdapter verbose: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.591Z [INFO] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: entering follower state: follower="Node at 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 [Follower]" leader=0dc470d6-a011-e4fd-046f-e7ec51b30d4b - testing.go:149: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.591Z [INFO] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: entering follower state: follower="Node at 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 [Follower]" leader=0dc470d6-a011-e4fd-046f-e7ec51b30d4b -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [WARN] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: appendEntries rejected, sending older logs: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" next=4 - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [WARN] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: appendEntries rejected, sending older logs: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" next=4 -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: pipelining replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: pipelining replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: pipelining replication: peer="{Voter server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 005b5ae9-219d-6c40-ef9c-25f3ac9723e1}" - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: pipelining replication: peer="{Voter server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 005b5ae9-219d-6c40-ef9c-25f3ac9723e1}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.696Z [INFO] stable state for Leader reached at 2024-01-09 05:10:02.59167866 +0000 UTC m=+13.844729648 (1 nodes), 65.55793ms from start of poll, 458.435181ms from cluster start. Timeout at 2024-01-09 05:10:02.696696391 +0000 UTC m=+13.949747444, 105.017796ms after stability - testing.go:149: cluster: 2024-01-09T05:10:02.696Z [INFO] stable state for Leader reached at 2024-01-09 05:10:02.59167866 +0000 UTC m=+13.844729648 (1 nodes), 65.55793ms from start of poll, 458.435181ms from cluster start. Timeout at 2024-01-09 05:10:02.696696391 +0000 UTC m=+13.949747444, 105.017796ms after stability -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.696Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: aborting pipeline replication: peer="{Voter server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 005b5ae9-219d-6c40-ef9c-25f3ac9723e1}" - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.696Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: aborting pipeline replication: peer="{Voter server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 005b5ae9-219d-6c40-ef9c-25f3ac9723e1}" -testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.697Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: aborting pipeline replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" - testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.697Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: aborting pipeline replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" ---- PASS: TestRaft_ReJoinFollower (0.57s) +testLoggerAdapter verbose: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.188Z [INFO] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 Address:d8405245-e81a-0c2e-0b48-5f081bfa1a50} {Suffrage:Voter ID:server-7554165b-63d8-d697-3e13-234c07743bdf Address:7554165b-63d8-d697-3e13-234c07743bdf} {Suffrage:Voter ID:server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 Address:1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26}]" + testing.go:149: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.188Z [INFO] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 Address:d8405245-e81a-0c2e-0b48-5f081bfa1a50} {Suffrage:Voter ID:server-7554165b-63d8-d697-3e13-234c07743bdf Address:7554165b-63d8-d697-3e13-234c07743bdf} {Suffrage:Voter ID:server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 Address:1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26}]" +testLoggerAdapter verbose: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.188Z [INFO] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: entering follower state: follower="Node at d8405245-e81a-0c2e-0b48-5f081bfa1a50 [Follower]" leader= + testing.go:149: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.188Z [INFO] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: entering follower state: follower="Node at d8405245-e81a-0c2e-0b48-5f081bfa1a50 [Follower]" leader= +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.188Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 Address:d8405245-e81a-0c2e-0b48-5f081bfa1a50} {Suffrage:Voter ID:server-7554165b-63d8-d697-3e13-234c07743bdf Address:7554165b-63d8-d697-3e13-234c07743bdf} {Suffrage:Voter ID:server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 Address:1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26}]" + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.188Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 Address:d8405245-e81a-0c2e-0b48-5f081bfa1a50} {Suffrage:Voter ID:server-7554165b-63d8-d697-3e13-234c07743bdf Address:7554165b-63d8-d697-3e13-234c07743bdf} {Suffrage:Voter ID:server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 Address:1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26}]" +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.188Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: entering follower state: follower="Node at 7554165b-63d8-d697-3e13-234c07743bdf [Follower]" leader= + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.188Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: entering follower state: follower="Node at 7554165b-63d8-d697-3e13-234c07743bdf [Follower]" leader= +testLoggerAdapter verbose: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.188Z [INFO] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 Address:d8405245-e81a-0c2e-0b48-5f081bfa1a50} {Suffrage:Voter ID:server-7554165b-63d8-d697-3e13-234c07743bdf Address:7554165b-63d8-d697-3e13-234c07743bdf} {Suffrage:Voter ID:server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 Address:1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26}]" + testing.go:149: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.188Z [INFO] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 Address:d8405245-e81a-0c2e-0b48-5f081bfa1a50} {Suffrage:Voter ID:server-7554165b-63d8-d697-3e13-234c07743bdf Address:7554165b-63d8-d697-3e13-234c07743bdf} {Suffrage:Voter ID:server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 Address:1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:16.188Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:16.188Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.188Z [INFO] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: entering follower state: follower="Node at 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 [Follower]" leader= + testing.go:149: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.188Z [INFO] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: entering follower state: follower="Node at 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 [Follower]" leader= +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.274Z [WARN] server-7554165b-63d8-d697-3e13-234c07743bdf: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.274Z [WARN] server-7554165b-63d8-d697-3e13-234c07743bdf: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.274Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: entering candidate state: node="Node at 7554165b-63d8-d697-3e13-234c07743bdf [Candidate]" term=2 + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.274Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: entering candidate state: node="Node at 7554165b-63d8-d697-3e13-234c07743bdf [Candidate]" term=2 +testLoggerAdapter verbose: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.274Z [WARN] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.274Z [WARN] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.274Z [INFO] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: entering candidate state: node="Node at 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 [Candidate]" term=2 +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.275Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: election won: tally=2 + testing.go:149: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.274Z [INFO] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: entering candidate state: node="Node at 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 [Candidate]" term=2 +testLoggerAdapter verbose: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.275Z [INFO] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: duplicate requestVote for same term: term=2 + testing.go:149: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.275Z [INFO] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: duplicate requestVote for same term: term=2 + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.275Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: election won: tally=2 +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.275Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: entering leader state: leader="Node at 7554165b-63d8-d697-3e13-234c07743bdf [Leader]" + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.275Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: entering leader state: leader="Node at 7554165b-63d8-d697-3e13-234c07743bdf [Leader]" +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.275Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: added peer, starting replication: peer=server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 +testLoggerAdapter verbose: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.275Z [INFO] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: duplicate requestVote for same term: term=2 + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.275Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: added peer, starting replication: peer=server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.275Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: added peer, starting replication: peer=server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 + testing.go:149: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.275Z [INFO] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: duplicate requestVote for same term: term=2 + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.275Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: added peer, starting replication: peer=server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.275Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: pipelining replication: peer="{Voter server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 d8405245-e81a-0c2e-0b48-5f081bfa1a50}" + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.275Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: pipelining replication: peer="{Voter server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 d8405245-e81a-0c2e-0b48-5f081bfa1a50}" +testLoggerAdapter verbose: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.275Z [INFO] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: entering follower state: follower="Node at 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 [Follower]" leader=7554165b-63d8-d697-3e13-234c07743bdf +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.275Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: pipelining replication: peer="{Voter server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26}" + testing.go:149: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.275Z [INFO] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: entering follower state: follower="Node at 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 [Follower]" leader=7554165b-63d8-d697-3e13-234c07743bdf + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.275Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: pipelining replication: peer="{Voter server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26}" +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.275Z [WARN] server-7554165b-63d8-d697-3e13-234c07743bdf: rejecting vote request since we have a leader: from=1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 leader=7554165b-63d8-d697-3e13-234c07743bdf + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.275Z [WARN] server-7554165b-63d8-d697-3e13-234c07743bdf: rejecting vote request since we have a leader: from=1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 leader=7554165b-63d8-d697-3e13-234c07743bdf +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:16.380Z [INFO] stable state for Leader reached at 2025-02-10 11:34:16.275226613 +0000 UTC m=+12.644130916 (1 nodes), 86.38433ms from start of poll, 86.932269ms from cluster start. Timeout at 2025-02-10 11:34:16.380323459 +0000 UTC m=+12.749227755, 105.096839ms after stability + testing.go:149: cluster: 2025-02-10T11:34:16.380Z [INFO] stable state for Leader reached at 2025-02-10 11:34:16.275226613 +0000 UTC m=+12.644130916 (1 nodes), 86.38433ms from start of poll, 86.932269ms from cluster start. Timeout at 2025-02-10 11:34:16.380323459 +0000 UTC m=+12.749227755, 105.096839ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:16.385Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:16.385Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:16.490Z [INFO] stable state for Follower reached at 2025-02-10 11:34:16.38542618 +0000 UTC m=+12.754330483 (2 nodes), 609ns from start of poll, 197.131836ms from cluster start. Timeout at 2025-02-10 11:34:16.490498135 +0000 UTC m=+12.859402432, 105.071949ms after stability + testing.go:149: cluster: 2025-02-10T11:34:16.490Z [INFO] stable state for Follower reached at 2025-02-10 11:34:16.38542618 +0000 UTC m=+12.754330483 (2 nodes), 609ns from start of poll, 197.131836ms from cluster start. Timeout at 2025-02-10 11:34:16.490498135 +0000 UTC m=+12.859402432, 105.071949ms after stability +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.490Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: updating configuration: command=RemoveServer server-id=server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 server-addr= servers="[{Suffrage:Voter ID:server-7554165b-63d8-d697-3e13-234c07743bdf Address:7554165b-63d8-d697-3e13-234c07743bdf} {Suffrage:Voter ID:server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 Address:1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26}]" + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.490Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: updating configuration: command=RemoveServer server-id=server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 server-addr= servers="[{Suffrage:Voter ID:server-7554165b-63d8-d697-3e13-234c07743bdf Address:7554165b-63d8-d697-3e13-234c07743bdf} {Suffrage:Voter ID:server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 Address:1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26}]" +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.490Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: removed peer, stopping replication: peer=server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 last-index=3 + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.490Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: removed peer, stopping replication: peer=server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 last-index=3 +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.490Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: aborting pipeline replication: peer="{Voter server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 d8405245-e81a-0c2e-0b48-5f081bfa1a50}" + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.490Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: aborting pipeline replication: peer="{Voter server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 d8405245-e81a-0c2e-0b48-5f081bfa1a50}" +testLoggerAdapter verbose: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.545Z [WARN] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: heartbeat timeout reached, starting election: last-leader=7554165b-63d8-d697-3e13-234c07743bdf + testing.go:149: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.545Z [WARN] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: heartbeat timeout reached, starting election: last-leader=7554165b-63d8-d697-3e13-234c07743bdf +testLoggerAdapter verbose: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.545Z [INFO] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: entering candidate state: node="Node at d8405245-e81a-0c2e-0b48-5f081bfa1a50 [Candidate]" term=3 + testing.go:149: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.545Z [INFO] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: entering candidate state: node="Node at d8405245-e81a-0c2e-0b48-5f081bfa1a50 [Candidate]" term=3 +testLoggerAdapter verbose: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.545Z [WARN] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: rejecting vote request since we have a leader: from=d8405245-e81a-0c2e-0b48-5f081bfa1a50 leader=7554165b-63d8-d697-3e13-234c07743bdf + testing.go:149: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.545Z [WARN] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: rejecting vote request since we have a leader: from=d8405245-e81a-0c2e-0b48-5f081bfa1a50 leader=7554165b-63d8-d697-3e13-234c07743bdf +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.545Z [WARN] server-7554165b-63d8-d697-3e13-234c07743bdf: rejecting vote request since we have a leader: from=d8405245-e81a-0c2e-0b48-5f081bfa1a50 leader=7554165b-63d8-d697-3e13-234c07743bdf + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.545Z [WARN] server-7554165b-63d8-d697-3e13-234c07743bdf: rejecting vote request since we have a leader: from=d8405245-e81a-0c2e-0b48-5f081bfa1a50 leader=7554165b-63d8-d697-3e13-234c07743bdf +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.595Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: updating configuration: command=AddStaging server-id=server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 server-addr=d8405245-e81a-0c2e-0b48-5f081bfa1a50 servers="[{Suffrage:Voter ID:server-7554165b-63d8-d697-3e13-234c07743bdf Address:7554165b-63d8-d697-3e13-234c07743bdf} {Suffrage:Voter ID:server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 Address:1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26} {Suffrage:Voter ID:server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 Address:d8405245-e81a-0c2e-0b48-5f081bfa1a50}]" + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.595Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: updating configuration: command=AddStaging server-id=server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 server-addr=d8405245-e81a-0c2e-0b48-5f081bfa1a50 servers="[{Suffrage:Voter ID:server-7554165b-63d8-d697-3e13-234c07743bdf Address:7554165b-63d8-d697-3e13-234c07743bdf} {Suffrage:Voter ID:server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 Address:1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26} {Suffrage:Voter ID:server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 Address:d8405245-e81a-0c2e-0b48-5f081bfa1a50}]" +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.595Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: added peer, starting replication: peer=server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.595Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: added peer, starting replication: peer=server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.595Z [ERROR] server-7554165b-63d8-d697-3e13-234c07743bdf: peer has newer term, stopping replication: peer="{Voter server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 d8405245-e81a-0c2e-0b48-5f081bfa1a50}" + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.595Z [ERROR] server-7554165b-63d8-d697-3e13-234c07743bdf: peer has newer term, stopping replication: peer="{Voter server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 d8405245-e81a-0c2e-0b48-5f081bfa1a50}" +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.595Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: entering follower state: follower="Node at 7554165b-63d8-d697-3e13-234c07743bdf [Follower]" leader= + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.595Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: entering follower state: follower="Node at 7554165b-63d8-d697-3e13-234c07743bdf [Follower]" leader= +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:16.595Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:16.595Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.595Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: aborting pipeline replication: peer="{Voter server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26}" + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.595Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: aborting pipeline replication: peer="{Voter server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26}" +testLoggerAdapter verbose: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.643Z [WARN] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: Election timeout reached, restarting election + testing.go:149: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.643Z [WARN] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: Election timeout reached, restarting election +testLoggerAdapter verbose: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.643Z [INFO] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: entering candidate state: node="Node at d8405245-e81a-0c2e-0b48-5f081bfa1a50 [Candidate]" term=4 + testing.go:149: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.643Z [INFO] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: entering candidate state: node="Node at d8405245-e81a-0c2e-0b48-5f081bfa1a50 [Candidate]" term=4 +testLoggerAdapter verbose: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.643Z [WARN] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: rejecting vote request since we have a leader: from=d8405245-e81a-0c2e-0b48-5f081bfa1a50 leader=7554165b-63d8-d697-3e13-234c07743bdf +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.643Z [WARN] server-7554165b-63d8-d697-3e13-234c07743bdf: rejecting vote request since our last index is greater: candidate=d8405245-e81a-0c2e-0b48-5f081bfa1a50 last-index=4 last-candidate-index=3 + testing.go:149: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.643Z [WARN] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: rejecting vote request since we have a leader: from=d8405245-e81a-0c2e-0b48-5f081bfa1a50 leader=7554165b-63d8-d697-3e13-234c07743bdf + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.643Z [WARN] server-7554165b-63d8-d697-3e13-234c07743bdf: rejecting vote request since our last index is greater: candidate=d8405245-e81a-0c2e-0b48-5f081bfa1a50 last-index=4 last-candidate-index=3 +testLoggerAdapter verbose: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.649Z [WARN] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: heartbeat timeout reached, starting election: last-leader=7554165b-63d8-d697-3e13-234c07743bdf + testing.go:149: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.649Z [WARN] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: heartbeat timeout reached, starting election: last-leader=7554165b-63d8-d697-3e13-234c07743bdf +testLoggerAdapter verbose: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.649Z [INFO] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: entering candidate state: node="Node at 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 [Candidate]" term=3 + testing.go:149: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.649Z [INFO] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: entering candidate state: node="Node at 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 [Candidate]" term=3 +testLoggerAdapter verbose: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.649Z [INFO] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: entering follower state: follower="Node at 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 [Follower]" leader= + testing.go:149: server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: 2025-02-10T11:34:16.649Z [INFO] server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26: entering follower state: follower="Node at 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 [Follower]" leader= +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [WARN] server-7554165b-63d8-d697-3e13-234c07743bdf: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [WARN] server-7554165b-63d8-d697-3e13-234c07743bdf: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: entering candidate state: node="Node at 7554165b-63d8-d697-3e13-234c07743bdf [Candidate]" term=5 + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: entering candidate state: node="Node at 7554165b-63d8-d697-3e13-234c07743bdf [Candidate]" term=5 +testLoggerAdapter verbose: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.659Z [INFO] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: entering follower state: follower="Node at d8405245-e81a-0c2e-0b48-5f081bfa1a50 [Follower]" leader= + testing.go:149: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.659Z [INFO] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: entering follower state: follower="Node at d8405245-e81a-0c2e-0b48-5f081bfa1a50 [Follower]" leader= +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: election won: tally=2 + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: election won: tally=2 +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: entering leader state: leader="Node at 7554165b-63d8-d697-3e13-234c07743bdf [Leader]" + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: entering leader state: leader="Node at 7554165b-63d8-d697-3e13-234c07743bdf [Leader]" +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: added peer, starting replication: peer=server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: added peer, starting replication: peer=server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: added peer, starting replication: peer=server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: added peer, starting replication: peer=server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 +testLoggerAdapter verbose: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.659Z [WARN] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: failed to get previous log: previous-index=4 last-index=3 error="log not found" +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: pipelining replication: peer="{Voter server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26}" + testing.go:149: server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: 2025-02-10T11:34:16.659Z [WARN] server-d8405245-e81a-0c2e-0b48-5f081bfa1a50: failed to get previous log: previous-index=4 last-index=3 error="log not found" + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: pipelining replication: peer="{Voter server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26}" +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [WARN] server-7554165b-63d8-d697-3e13-234c07743bdf: appendEntries rejected, sending older logs: peer="{Voter server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 d8405245-e81a-0c2e-0b48-5f081bfa1a50}" next=4 + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [WARN] server-7554165b-63d8-d697-3e13-234c07743bdf: appendEntries rejected, sending older logs: peer="{Voter server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 d8405245-e81a-0c2e-0b48-5f081bfa1a50}" next=4 +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: pipelining replication: peer="{Voter server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 d8405245-e81a-0c2e-0b48-5f081bfa1a50}" + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.659Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: pipelining replication: peer="{Voter server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 d8405245-e81a-0c2e-0b48-5f081bfa1a50}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:16.764Z [INFO] stable state for Leader reached at 2025-02-10 11:34:16.659408581 +0000 UTC m=+13.028313000 (1 nodes), 63.468318ms from start of poll, 471.114353ms from cluster start. Timeout at 2025-02-10 11:34:16.76446748 +0000 UTC m=+13.133371833, 105.058833ms after stability + testing.go:149: cluster: 2025-02-10T11:34:16.764Z [INFO] stable state for Leader reached at 2025-02-10 11:34:16.659408581 +0000 UTC m=+13.028313000 (1 nodes), 63.468318ms from start of poll, 471.114353ms from cluster start. Timeout at 2025-02-10 11:34:16.76446748 +0000 UTC m=+13.133371833, 105.058833ms after stability +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.764Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: aborting pipeline replication: peer="{Voter server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 d8405245-e81a-0c2e-0b48-5f081bfa1a50}" + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.764Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: aborting pipeline replication: peer="{Voter server-d8405245-e81a-0c2e-0b48-5f081bfa1a50 d8405245-e81a-0c2e-0b48-5f081bfa1a50}" +testLoggerAdapter verbose: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.764Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: aborting pipeline replication: peer="{Voter server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26}" + testing.go:149: server-7554165b-63d8-d697-3e13-234c07743bdf: 2025-02-10T11:34:16.764Z [INFO] server-7554165b-63d8-d697-3e13-234c07743bdf: aborting pipeline replication: peer="{Voter server-1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26 1fdd7c5e-b0eb-47ab-6d87-3cfcffbaed26}" +--- PASS: TestRaft_ReJoinFollower (0.58s) === RUN TestRaft_LeaderLeaseExpire -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.699Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 Address:8617de21-0aaf-40bb-57ea-6dd62fb443e3} {Suffrage:Voter ID:server-709f3e06-aa57-e0d5-215e-a9989d22e81c Address:709f3e06-aa57-e0d5-215e-a9989d22e81c}]" - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.699Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 Address:8617de21-0aaf-40bb-57ea-6dd62fb443e3} {Suffrage:Voter ID:server-709f3e06-aa57-e0d5-215e-a9989d22e81c Address:709f3e06-aa57-e0d5-215e-a9989d22e81c}]" -testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:02.699Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 Address:8617de21-0aaf-40bb-57ea-6dd62fb443e3} {Suffrage:Voter ID:server-709f3e06-aa57-e0d5-215e-a9989d22e81c Address:709f3e06-aa57-e0d5-215e-a9989d22e81c}]" - testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:02.699Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 Address:8617de21-0aaf-40bb-57ea-6dd62fb443e3} {Suffrage:Voter ID:server-709f3e06-aa57-e0d5-215e-a9989d22e81c Address:709f3e06-aa57-e0d5-215e-a9989d22e81c}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.699Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:02.699Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:02.699Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: entering follower state: follower="Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Follower]" leader= - testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:02.699Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: entering follower state: follower="Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Follower]" leader= -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.699Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering follower state: follower="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Follower]" leader= - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.699Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering follower state: follower="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Follower]" leader= -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.750Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.750Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering candidate state: node="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Candidate]" term=2 - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering candidate state: node="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Candidate]" term=2 -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: election won: tally=2 - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: election won: tally=2 -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering leader state: leader="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Leader]" - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering leader state: leader="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Leader]" -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: added peer, starting replication: peer=server-709f3e06-aa57-e0d5-215e-a9989d22e81c - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: added peer, starting replication: peer=server-709f3e06-aa57-e0d5-215e-a9989d22e81c -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: pipelining replication: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: pipelining replication: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.856Z [INFO] stable state for Leader reached at 2024-01-09 05:10:02.751340387 +0000 UTC m=+14.004391361 (1 nodes), 52.063478ms from start of poll, 52.390421ms from cluster start. Timeout at 2024-01-09 05:10:02.856582914 +0000 UTC m=+14.109633895, 105.242534ms after stability - testing.go:149: cluster: 2024-01-09T05:10:02.856Z [INFO] stable state for Leader reached at 2024-01-09 05:10:02.751340387 +0000 UTC m=+14.004391361 (1 nodes), 52.063478ms from start of poll, 52.390421ms from cluster start. Timeout at 2024-01-09 05:10:02.856582914 +0000 UTC m=+14.109633895, 105.242534ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.861Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:10:02.861Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.967Z [INFO] stable state for Follower reached at 2024-01-09 05:10:02.86185176 +0000 UTC m=+14.114902728 (1 nodes), 2.075µs from start of poll, 162.901788ms from cluster start. Timeout at 2024-01-09 05:10:02.967508918 +0000 UTC m=+14.220559947, 105.657219ms after stability - testing.go:149: cluster: 2024-01-09T05:10:02.967Z [INFO] stable state for Follower reached at 2024-01-09 05:10:02.86185176 +0000 UTC m=+14.114902728 (1 nodes), 2.075µs from start of poll, 162.901788ms from cluster start. Timeout at 2024-01-09 05:10:02.967508918 +0000 UTC m=+14.220559947, 105.657219ms after stability - raft_test.go:1492: [INFO] Disconnecting Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Follower] -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.969Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to heartbeat to: peer=709f3e06-aa57-e0d5-215e-a9989d22e81c error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.969Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to heartbeat to: peer=709f3e06-aa57-e0d5-215e-a9989d22e81c error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.970Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to pipeline appendEntries: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="append pipeline closed" - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.970Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to pipeline appendEntries: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="append pipeline closed" -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.970Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: aborting pipeline replication: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.970Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: aborting pipeline replication: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.978Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to appendEntries to: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.978Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to appendEntries to: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.987Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to heartbeat to: peer=709f3e06-aa57-e0d5-215e-a9989d22e81c error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.987Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to heartbeat to: peer=709f3e06-aa57-e0d5-215e-a9989d22e81c error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.995Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to appendEntries to: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.995Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to appendEntries to: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.008Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to heartbeat to: peer=709f3e06-aa57-e0d5-215e-a9989d22e81c error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.008Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to heartbeat to: peer=709f3e06-aa57-e0d5-215e-a9989d22e81c error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.013Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to appendEntries to: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.013Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to appendEntries to: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.015Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to contact: server-id=server-709f3e06-aa57-e0d5-215e-a9989d22e81c time=50.092686ms - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.015Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to contact: server-id=server-709f3e06-aa57-e0d5-215e-a9989d22e81c time=50.092686ms -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.015Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to contact quorum of nodes, stepping down - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.015Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to contact quorum of nodes, stepping down -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.015Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering follower state: follower="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Follower]" leader= - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.015Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering follower state: follower="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Follower]" leader= -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.015Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:03.015Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.095Z [WARN] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: heartbeat timeout reached, starting election: last-leader=8617de21-0aaf-40bb-57ea-6dd62fb443e3 - testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.095Z [WARN] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: heartbeat timeout reached, starting election: last-leader=8617de21-0aaf-40bb-57ea-6dd62fb443e3 -testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.095Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: entering candidate state: node="Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Candidate]" term=3 - testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.095Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: entering candidate state: node="Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Candidate]" term=3 -testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.095Z [ERROR] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: failed to make requestVote RPC: target="{Voter server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 8617de21-0aaf-40bb-57ea-6dd62fb443e3}" error="failed to connect to peer: 8617de21-0aaf-40bb-57ea-6dd62fb443e3" - testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.095Z [ERROR] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: failed to make requestVote RPC: target="{Voter server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 8617de21-0aaf-40bb-57ea-6dd62fb443e3}" error="failed to connect to peer: 8617de21-0aaf-40bb-57ea-6dd62fb443e3" -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.097Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.097Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.097Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering candidate state: node="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Candidate]" term=3 - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.097Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering candidate state: node="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Candidate]" term=3 -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.098Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to make requestVote RPC: target="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.098Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to make requestVote RPC: target="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" -testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.164Z [WARN] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: Election timeout reached, restarting election - testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.164Z [WARN] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: Election timeout reached, restarting election -testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.164Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: entering candidate state: node="Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Candidate]" term=4 - testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.164Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: entering candidate state: node="Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Candidate]" term=4 -testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.164Z [ERROR] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: failed to make requestVote RPC: target="{Voter server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 8617de21-0aaf-40bb-57ea-6dd62fb443e3}" error="failed to connect to peer: 8617de21-0aaf-40bb-57ea-6dd62fb443e3" - testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.164Z [ERROR] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: failed to make requestVote RPC: target="{Voter server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 8617de21-0aaf-40bb-57ea-6dd62fb443e3}" error="failed to connect to peer: 8617de21-0aaf-40bb-57ea-6dd62fb443e3" -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.190Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: Election timeout reached, restarting election - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.190Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: Election timeout reached, restarting election -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.191Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering candidate state: node="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Candidate]" term=4 - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.191Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering candidate state: node="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Candidate]" term=4 -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.191Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to make requestVote RPC: target="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.191Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to make requestVote RPC: target="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.200Z [INFO] stable state for Leader reached at 2024-01-09 05:10:03.095257568 +0000 UTC m=+14.348308620 (0 nodes), 79.274905ms from start of poll, 396.30768ms from cluster start. Timeout at 2024-01-09 05:10:03.20036977 +0000 UTC m=+14.453420837, 105.112217ms after stability - testing.go:149: cluster: 2024-01-09T05:10:03.200Z [INFO] stable state for Leader reached at 2024-01-09 05:10:03.095257568 +0000 UTC m=+14.348308620 (0 nodes), 79.274905ms from start of poll, 396.30768ms from cluster start. Timeout at 2024-01-09 05:10:03.20036977 +0000 UTC m=+14.453420837, 105.112217ms after stability -testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.243Z [WARN] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: Election timeout reached, restarting election - testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.243Z [WARN] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: Election timeout reached, restarting election -testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.243Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: entering candidate state: node="Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Candidate]" term=5 - testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.243Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: entering candidate state: node="Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Candidate]" term=5 -testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.243Z [ERROR] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: failed to make requestVote RPC: target="{Voter server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 8617de21-0aaf-40bb-57ea-6dd62fb443e3}" error="failed to connect to peer: 8617de21-0aaf-40bb-57ea-6dd62fb443e3" - testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.243Z [ERROR] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: failed to make requestVote RPC: target="{Voter server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 8617de21-0aaf-40bb-57ea-6dd62fb443e3}" error="failed to connect to peer: 8617de21-0aaf-40bb-57ea-6dd62fb443e3" -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.273Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: Election timeout reached, restarting election - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.273Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: Election timeout reached, restarting election -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.273Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering candidate state: node="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Candidate]" term=5 - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.273Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering candidate state: node="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Candidate]" term=5 -testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.273Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to make requestVote RPC: target="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" - testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.273Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to make requestVote RPC: target="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" ---- PASS: TestRaft_LeaderLeaseExpire (0.61s) +testLoggerAdapter verbose: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:16.765Z [INFO] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 Address:d1441b8d-1ba0-5689-291b-e3dfd1fc3f53} {Suffrage:Voter ID:server-ecf585ab-2037-e4fe-34a0-12acb18a618e Address:ecf585ab-2037-e4fe-34a0-12acb18a618e}]" + testing.go:149: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:16.765Z [INFO] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 Address:d1441b8d-1ba0-5689-291b-e3dfd1fc3f53} {Suffrage:Voter ID:server-ecf585ab-2037-e4fe-34a0-12acb18a618e Address:ecf585ab-2037-e4fe-34a0-12acb18a618e}]" +testLoggerAdapter verbose: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:16.765Z [INFO] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: entering follower state: follower="Node at d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 [Follower]" leader= + testing.go:149: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:16.765Z [INFO] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: entering follower state: follower="Node at d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 [Follower]" leader= +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:16.765Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 Address:d1441b8d-1ba0-5689-291b-e3dfd1fc3f53} {Suffrage:Voter ID:server-ecf585ab-2037-e4fe-34a0-12acb18a618e Address:ecf585ab-2037-e4fe-34a0-12acb18a618e}]" + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:16.765Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 Address:d1441b8d-1ba0-5689-291b-e3dfd1fc3f53} {Suffrage:Voter ID:server-ecf585ab-2037-e4fe-34a0-12acb18a618e Address:ecf585ab-2037-e4fe-34a0-12acb18a618e}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:16.766Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:16.766Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:16.766Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: entering follower state: follower="Node at ecf585ab-2037-e4fe-34a0-12acb18a618e [Follower]" leader= + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:16.766Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: entering follower state: follower="Node at ecf585ab-2037-e4fe-34a0-12acb18a618e [Follower]" leader= +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:16.826Z [WARN] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:16.826Z [WARN] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:16.826Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: entering candidate state: node="Node at ecf585ab-2037-e4fe-34a0-12acb18a618e [Candidate]" term=2 + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:16.826Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: entering candidate state: node="Node at ecf585ab-2037-e4fe-34a0-12acb18a618e [Candidate]" term=2 +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:16.826Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: election won: tally=2 + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:16.826Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: election won: tally=2 +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:16.826Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: entering leader state: leader="Node at ecf585ab-2037-e4fe-34a0-12acb18a618e [Leader]" + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:16.826Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: entering leader state: leader="Node at ecf585ab-2037-e4fe-34a0-12acb18a618e [Leader]" +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:16.826Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: added peer, starting replication: peer=server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:16.826Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: added peer, starting replication: peer=server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:16.826Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: pipelining replication: peer="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:16.826Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: pipelining replication: peer="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:16.931Z [INFO] stable state for Leader reached at 2025-02-10 11:34:16.826303756 +0000 UTC m=+13.195208079 (1 nodes), 60.228856ms from start of poll, 60.579536ms from cluster start. Timeout at 2025-02-10 11:34:16.93140396 +0000 UTC m=+13.300308313, 105.100234ms after stability + testing.go:149: cluster: 2025-02-10T11:34:16.931Z [INFO] stable state for Leader reached at 2025-02-10 11:34:16.826303756 +0000 UTC m=+13.195208079 (1 nodes), 60.228856ms from start of poll, 60.579536ms from cluster start. Timeout at 2025-02-10 11:34:16.93140396 +0000 UTC m=+13.300308313, 105.100234ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:16.936Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:16.936Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:17.041Z [INFO] stable state for Follower reached at 2025-02-10 11:34:16.936643466 +0000 UTC m=+13.305547766 (1 nodes), 406ns from start of poll, 170.919223ms from cluster start. Timeout at 2025-02-10 11:34:17.041718669 +0000 UTC m=+13.410622973, 105.075207ms after stability + testing.go:149: cluster: 2025-02-10T11:34:17.041Z [INFO] stable state for Follower reached at 2025-02-10 11:34:16.936643466 +0000 UTC m=+13.305547766 (1 nodes), 406ns from start of poll, 170.919223ms from cluster start. Timeout at 2025-02-10 11:34:17.041718669 +0000 UTC m=+13.410622973, 105.075207ms after stability + raft_test.go:1492: [INFO] Disconnecting Node at d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 [Follower] +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.044Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to heartbeat to: peer=d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.044Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to heartbeat to: peer=d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.047Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to pipeline appendEntries: peer="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" error="append pipeline closed" + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.047Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to pipeline appendEntries: peer="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" error="append pipeline closed" +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.047Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: aborting pipeline replication: peer="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.047Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: aborting pipeline replication: peer="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.054Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to appendEntries to: peer="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.054Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to appendEntries to: peer="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.061Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to heartbeat to: peer=d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.061Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to heartbeat to: peer=d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.073Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to appendEntries to: peer="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.073Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to appendEntries to: peer="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.077Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to heartbeat to: peer=d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.077Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to heartbeat to: peer=d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.089Z [WARN] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to contact: server-id=server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 time=50.069392ms + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.089Z [WARN] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to contact: server-id=server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 time=50.069392ms +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.089Z [WARN] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to contact quorum of nodes, stepping down + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.089Z [WARN] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to contact quorum of nodes, stepping down +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.089Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: entering follower state: follower="Node at ecf585ab-2037-e4fe-34a0-12acb18a618e [Follower]" leader= + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.089Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: entering follower state: follower="Node at ecf585ab-2037-e4fe-34a0-12acb18a618e [Follower]" leader= +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:17.089Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:17.089Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.091Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to appendEntries to: peer="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.091Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to appendEntries to: peer="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" +testLoggerAdapter verbose: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.102Z [WARN] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: heartbeat timeout reached, starting election: last-leader=ecf585ab-2037-e4fe-34a0-12acb18a618e + testing.go:149: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.102Z [WARN] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: heartbeat timeout reached, starting election: last-leader=ecf585ab-2037-e4fe-34a0-12acb18a618e +testLoggerAdapter verbose: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.102Z [INFO] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: entering candidate state: node="Node at d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 [Candidate]" term=3 + testing.go:149: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.102Z [INFO] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: entering candidate state: node="Node at d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 [Candidate]" term=3 +testLoggerAdapter verbose: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.102Z [ERROR] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: failed to make requestVote RPC: target="{Voter server-ecf585ab-2037-e4fe-34a0-12acb18a618e ecf585ab-2037-e4fe-34a0-12acb18a618e}" error="failed to connect to peer: ecf585ab-2037-e4fe-34a0-12acb18a618e" + testing.go:149: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.102Z [ERROR] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: failed to make requestVote RPC: target="{Voter server-ecf585ab-2037-e4fe-34a0-12acb18a618e ecf585ab-2037-e4fe-34a0-12acb18a618e}" error="failed to connect to peer: ecf585ab-2037-e4fe-34a0-12acb18a618e" +testLoggerAdapter verbose: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.168Z [WARN] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: Election timeout reached, restarting election + testing.go:149: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.168Z [WARN] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: Election timeout reached, restarting election +testLoggerAdapter verbose: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.168Z [INFO] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: entering candidate state: node="Node at d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 [Candidate]" term=4 + testing.go:149: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.168Z [INFO] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: entering candidate state: node="Node at d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 [Candidate]" term=4 +testLoggerAdapter verbose: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.168Z [ERROR] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: failed to make requestVote RPC: target="{Voter server-ecf585ab-2037-e4fe-34a0-12acb18a618e ecf585ab-2037-e4fe-34a0-12acb18a618e}" error="failed to connect to peer: ecf585ab-2037-e4fe-34a0-12acb18a618e" + testing.go:149: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.168Z [ERROR] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: failed to make requestVote RPC: target="{Voter server-ecf585ab-2037-e4fe-34a0-12acb18a618e ecf585ab-2037-e4fe-34a0-12acb18a618e}" error="failed to connect to peer: ecf585ab-2037-e4fe-34a0-12acb18a618e" +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.185Z [WARN] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.185Z [WARN] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.186Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: entering candidate state: node="Node at ecf585ab-2037-e4fe-34a0-12acb18a618e [Candidate]" term=3 + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.186Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: entering candidate state: node="Node at ecf585ab-2037-e4fe-34a0-12acb18a618e [Candidate]" term=3 +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.186Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to make requestVote RPC: target="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.186Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to make requestVote RPC: target="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" +testLoggerAdapter verbose: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.219Z [WARN] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: Election timeout reached, restarting election + testing.go:149: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.219Z [WARN] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: Election timeout reached, restarting election +testLoggerAdapter verbose: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.219Z [INFO] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: entering candidate state: node="Node at d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 [Candidate]" term=5 + testing.go:149: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.219Z [INFO] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: entering candidate state: node="Node at d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 [Candidate]" term=5 +testLoggerAdapter verbose: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.219Z [ERROR] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: failed to make requestVote RPC: target="{Voter server-ecf585ab-2037-e4fe-34a0-12acb18a618e ecf585ab-2037-e4fe-34a0-12acb18a618e}" error="failed to connect to peer: ecf585ab-2037-e4fe-34a0-12acb18a618e" + testing.go:149: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.219Z [ERROR] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: failed to make requestVote RPC: target="{Voter server-ecf585ab-2037-e4fe-34a0-12acb18a618e ecf585ab-2037-e4fe-34a0-12acb18a618e}" error="failed to connect to peer: ecf585ab-2037-e4fe-34a0-12acb18a618e" +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.241Z [WARN] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: Election timeout reached, restarting election + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.241Z [WARN] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: Election timeout reached, restarting election +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.241Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: entering candidate state: node="Node at ecf585ab-2037-e4fe-34a0-12acb18a618e [Candidate]" term=4 + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.241Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: entering candidate state: node="Node at ecf585ab-2037-e4fe-34a0-12acb18a618e [Candidate]" term=4 +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.241Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to make requestVote RPC: target="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.241Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to make requestVote RPC: target="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:17.291Z [INFO] stable state for Leader reached at 2025-02-10 11:34:17.186134319 +0000 UTC m=+13.555038623 (0 nodes), 96.515538ms from start of poll, 420.41008ms from cluster start. Timeout at 2025-02-10 11:34:17.291234745 +0000 UTC m=+13.660139041, 105.100418ms after stability + testing.go:149: cluster: 2025-02-10T11:34:17.291Z [INFO] stable state for Leader reached at 2025-02-10 11:34:17.186134319 +0000 UTC m=+13.555038623 (0 nodes), 96.515538ms from start of poll, 420.41008ms from cluster start. Timeout at 2025-02-10 11:34:17.291234745 +0000 UTC m=+13.660139041, 105.100418ms after stability +testLoggerAdapter verbose: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.309Z [WARN] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: Election timeout reached, restarting election + testing.go:149: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.309Z [WARN] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: Election timeout reached, restarting election +testLoggerAdapter verbose: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.309Z [INFO] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: entering candidate state: node="Node at d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 [Candidate]" term=6 + testing.go:149: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.309Z [INFO] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: entering candidate state: node="Node at d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 [Candidate]" term=6 +testLoggerAdapter verbose: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.309Z [ERROR] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: failed to make requestVote RPC: target="{Voter server-ecf585ab-2037-e4fe-34a0-12acb18a618e ecf585ab-2037-e4fe-34a0-12acb18a618e}" error="failed to connect to peer: ecf585ab-2037-e4fe-34a0-12acb18a618e" + testing.go:149: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.309Z [ERROR] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: failed to make requestVote RPC: target="{Voter server-ecf585ab-2037-e4fe-34a0-12acb18a618e ecf585ab-2037-e4fe-34a0-12acb18a618e}" error="failed to connect to peer: ecf585ab-2037-e4fe-34a0-12acb18a618e" +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.319Z [WARN] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: Election timeout reached, restarting election + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.319Z [WARN] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: Election timeout reached, restarting election +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.319Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: entering candidate state: node="Node at ecf585ab-2037-e4fe-34a0-12acb18a618e [Candidate]" term=5 + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.319Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: entering candidate state: node="Node at ecf585ab-2037-e4fe-34a0-12acb18a618e [Candidate]" term=5 +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.319Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to make requestVote RPC: target="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.319Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to make requestVote RPC: target="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" +testLoggerAdapter verbose: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.362Z [WARN] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: Election timeout reached, restarting election + testing.go:149: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.362Z [WARN] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: Election timeout reached, restarting election +testLoggerAdapter verbose: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.362Z [INFO] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: entering candidate state: node="Node at d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 [Candidate]" term=7 + testing.go:149: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.362Z [INFO] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: entering candidate state: node="Node at d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 [Candidate]" term=7 +testLoggerAdapter verbose: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.362Z [ERROR] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: failed to make requestVote RPC: target="{Voter server-ecf585ab-2037-e4fe-34a0-12acb18a618e ecf585ab-2037-e4fe-34a0-12acb18a618e}" error="failed to connect to peer: ecf585ab-2037-e4fe-34a0-12acb18a618e" + testing.go:149: server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: 2025-02-10T11:34:17.362Z [ERROR] server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53: failed to make requestVote RPC: target="{Voter server-ecf585ab-2037-e4fe-34a0-12acb18a618e ecf585ab-2037-e4fe-34a0-12acb18a618e}" error="failed to connect to peer: ecf585ab-2037-e4fe-34a0-12acb18a618e" +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.375Z [WARN] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: Election timeout reached, restarting election + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.375Z [WARN] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: Election timeout reached, restarting election +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.375Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: entering candidate state: node="Node at ecf585ab-2037-e4fe-34a0-12acb18a618e [Candidate]" term=6 + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.375Z [INFO] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: entering candidate state: node="Node at ecf585ab-2037-e4fe-34a0-12acb18a618e [Candidate]" term=6 +testLoggerAdapter verbose: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.375Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to make requestVote RPC: target="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" + testing.go:149: server-ecf585ab-2037-e4fe-34a0-12acb18a618e: 2025-02-10T11:34:17.375Z [ERROR] server-ecf585ab-2037-e4fe-34a0-12acb18a618e: failed to make requestVote RPC: target="{Voter server-d1441b8d-1ba0-5689-291b-e3dfd1fc3f53 d1441b8d-1ba0-5689-291b-e3dfd1fc3f53}" error="failed to connect to peer: d1441b8d-1ba0-5689-291b-e3dfd1fc3f53" +--- PASS: TestRaft_LeaderLeaseExpire (0.63s) === RUN TestRaft_Barrier -testLoggerAdapter verbose: server-9a014d33-ecd4-ff65-9072-f7ec18431f63: 2024-01-09T05:10:03.308Z [INFO] server-9a014d33-ecd4-ff65-9072-f7ec18431f63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a014d33-ecd4-ff65-9072-f7ec18431f63 Address:9a014d33-ecd4-ff65-9072-f7ec18431f63} {Suffrage:Voter ID:server-2488494b-ec57-e41d-646d-b34026aff9e6 Address:2488494b-ec57-e41d-646d-b34026aff9e6} {Suffrage:Voter ID:server-bebf1eda-9b5a-62fc-4805-da04bddb943a Address:bebf1eda-9b5a-62fc-4805-da04bddb943a}]" - testing.go:149: server-9a014d33-ecd4-ff65-9072-f7ec18431f63: 2024-01-09T05:10:03.308Z [INFO] server-9a014d33-ecd4-ff65-9072-f7ec18431f63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a014d33-ecd4-ff65-9072-f7ec18431f63 Address:9a014d33-ecd4-ff65-9072-f7ec18431f63} {Suffrage:Voter ID:server-2488494b-ec57-e41d-646d-b34026aff9e6 Address:2488494b-ec57-e41d-646d-b34026aff9e6} {Suffrage:Voter ID:server-bebf1eda-9b5a-62fc-4805-da04bddb943a Address:bebf1eda-9b5a-62fc-4805-da04bddb943a}]" -testLoggerAdapter verbose: server-2488494b-ec57-e41d-646d-b34026aff9e6: 2024-01-09T05:10:03.309Z [INFO] server-2488494b-ec57-e41d-646d-b34026aff9e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a014d33-ecd4-ff65-9072-f7ec18431f63 Address:9a014d33-ecd4-ff65-9072-f7ec18431f63} {Suffrage:Voter ID:server-2488494b-ec57-e41d-646d-b34026aff9e6 Address:2488494b-ec57-e41d-646d-b34026aff9e6} {Suffrage:Voter ID:server-bebf1eda-9b5a-62fc-4805-da04bddb943a Address:bebf1eda-9b5a-62fc-4805-da04bddb943a}]" - testing.go:149: server-2488494b-ec57-e41d-646d-b34026aff9e6: 2024-01-09T05:10:03.309Z [INFO] server-2488494b-ec57-e41d-646d-b34026aff9e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a014d33-ecd4-ff65-9072-f7ec18431f63 Address:9a014d33-ecd4-ff65-9072-f7ec18431f63} {Suffrage:Voter ID:server-2488494b-ec57-e41d-646d-b34026aff9e6 Address:2488494b-ec57-e41d-646d-b34026aff9e6} {Suffrage:Voter ID:server-bebf1eda-9b5a-62fc-4805-da04bddb943a Address:bebf1eda-9b5a-62fc-4805-da04bddb943a}]" -testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.309Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a014d33-ecd4-ff65-9072-f7ec18431f63 Address:9a014d33-ecd4-ff65-9072-f7ec18431f63} {Suffrage:Voter ID:server-2488494b-ec57-e41d-646d-b34026aff9e6 Address:2488494b-ec57-e41d-646d-b34026aff9e6} {Suffrage:Voter ID:server-bebf1eda-9b5a-62fc-4805-da04bddb943a Address:bebf1eda-9b5a-62fc-4805-da04bddb943a}]" - testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.309Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a014d33-ecd4-ff65-9072-f7ec18431f63 Address:9a014d33-ecd4-ff65-9072-f7ec18431f63} {Suffrage:Voter ID:server-2488494b-ec57-e41d-646d-b34026aff9e6 Address:2488494b-ec57-e41d-646d-b34026aff9e6} {Suffrage:Voter ID:server-bebf1eda-9b5a-62fc-4805-da04bddb943a Address:bebf1eda-9b5a-62fc-4805-da04bddb943a}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.309Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:03.309Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-9a014d33-ecd4-ff65-9072-f7ec18431f63: 2024-01-09T05:10:03.309Z [INFO] server-9a014d33-ecd4-ff65-9072-f7ec18431f63: entering follower state: follower="Node at 9a014d33-ecd4-ff65-9072-f7ec18431f63 [Follower]" leader= - testing.go:149: server-9a014d33-ecd4-ff65-9072-f7ec18431f63: 2024-01-09T05:10:03.309Z [INFO] server-9a014d33-ecd4-ff65-9072-f7ec18431f63: entering follower state: follower="Node at 9a014d33-ecd4-ff65-9072-f7ec18431f63 [Follower]" leader= -testLoggerAdapter verbose: server-2488494b-ec57-e41d-646d-b34026aff9e6: 2024-01-09T05:10:03.310Z [INFO] server-2488494b-ec57-e41d-646d-b34026aff9e6: entering follower state: follower="Node at 2488494b-ec57-e41d-646d-b34026aff9e6 [Follower]" leader= - testing.go:149: server-2488494b-ec57-e41d-646d-b34026aff9e6: 2024-01-09T05:10:03.310Z [INFO] server-2488494b-ec57-e41d-646d-b34026aff9e6: entering follower state: follower="Node at 2488494b-ec57-e41d-646d-b34026aff9e6 [Follower]" leader= -testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.310Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: entering follower state: follower="Node at bebf1eda-9b5a-62fc-4805-da04bddb943a [Follower]" leader= - testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.310Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: entering follower state: follower="Node at bebf1eda-9b5a-62fc-4805-da04bddb943a [Follower]" leader= -testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [WARN] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [WARN] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: entering candidate state: node="Node at bebf1eda-9b5a-62fc-4805-da04bddb943a [Candidate]" term=2 - testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: entering candidate state: node="Node at bebf1eda-9b5a-62fc-4805-da04bddb943a [Candidate]" term=2 -testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: election won: tally=2 - testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: election won: tally=2 -testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: entering leader state: leader="Node at bebf1eda-9b5a-62fc-4805-da04bddb943a [Leader]" - testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: entering leader state: leader="Node at bebf1eda-9b5a-62fc-4805-da04bddb943a [Leader]" -testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: added peer, starting replication: peer=server-9a014d33-ecd4-ff65-9072-f7ec18431f63 - testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: added peer, starting replication: peer=server-9a014d33-ecd4-ff65-9072-f7ec18431f63 -testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: added peer, starting replication: peer=server-2488494b-ec57-e41d-646d-b34026aff9e6 - testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: added peer, starting replication: peer=server-2488494b-ec57-e41d-646d-b34026aff9e6 -testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: pipelining replication: peer="{Voter server-9a014d33-ecd4-ff65-9072-f7ec18431f63 9a014d33-ecd4-ff65-9072-f7ec18431f63}" - testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: pipelining replication: peer="{Voter server-9a014d33-ecd4-ff65-9072-f7ec18431f63 9a014d33-ecd4-ff65-9072-f7ec18431f63}" -testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: pipelining replication: peer="{Voter server-2488494b-ec57-e41d-646d-b34026aff9e6 2488494b-ec57-e41d-646d-b34026aff9e6}" - testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: pipelining replication: peer="{Voter server-2488494b-ec57-e41d-646d-b34026aff9e6 2488494b-ec57-e41d-646d-b34026aff9e6}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.499Z [INFO] stable state for Leader reached at 2024-01-09 05:10:03.393609191 +0000 UTC m=+14.646660196 (1 nodes), 84.200457ms from start of poll, 84.800943ms from cluster start. Timeout at 2024-01-09 05:10:03.498963557 +0000 UTC m=+14.752014791, 105.354595ms after stability - testing.go:149: cluster: 2024-01-09T05:10:03.499Z [INFO] stable state for Leader reached at 2024-01-09 05:10:03.393609191 +0000 UTC m=+14.646660196 (1 nodes), 84.200457ms from start of poll, 84.800943ms from cluster start. Timeout at 2024-01-09 05:10:03.498963557 +0000 UTC m=+14.752014791, 105.354595ms after stability -testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.517Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: aborting pipeline replication: peer="{Voter server-9a014d33-ecd4-ff65-9072-f7ec18431f63 9a014d33-ecd4-ff65-9072-f7ec18431f63}" - testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.517Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: aborting pipeline replication: peer="{Voter server-9a014d33-ecd4-ff65-9072-f7ec18431f63 9a014d33-ecd4-ff65-9072-f7ec18431f63}" -testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.517Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: aborting pipeline replication: peer="{Voter server-2488494b-ec57-e41d-646d-b34026aff9e6 2488494b-ec57-e41d-646d-b34026aff9e6}" - testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.517Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: aborting pipeline replication: peer="{Voter server-2488494b-ec57-e41d-646d-b34026aff9e6 2488494b-ec57-e41d-646d-b34026aff9e6}" ---- PASS: TestRaft_Barrier (0.21s) +testLoggerAdapter verbose: server-4ff9cfaa-097a-6628-a236-b6bde823ae5d: 2025-02-10T11:34:17.397Z [INFO] server-4ff9cfaa-097a-6628-a236-b6bde823ae5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ff9cfaa-097a-6628-a236-b6bde823ae5d Address:4ff9cfaa-097a-6628-a236-b6bde823ae5d} {Suffrage:Voter ID:server-b8eecf14-3b74-dcb4-d192-467613a45d7a Address:b8eecf14-3b74-dcb4-d192-467613a45d7a} {Suffrage:Voter ID:server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036 Address:486d0fd4-852d-d6fb-50fb-1b52d4fb0036}]" + testing.go:149: server-4ff9cfaa-097a-6628-a236-b6bde823ae5d: 2025-02-10T11:34:17.397Z [INFO] server-4ff9cfaa-097a-6628-a236-b6bde823ae5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ff9cfaa-097a-6628-a236-b6bde823ae5d Address:4ff9cfaa-097a-6628-a236-b6bde823ae5d} {Suffrage:Voter ID:server-b8eecf14-3b74-dcb4-d192-467613a45d7a Address:b8eecf14-3b74-dcb4-d192-467613a45d7a} {Suffrage:Voter ID:server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036 Address:486d0fd4-852d-d6fb-50fb-1b52d4fb0036}]" +testLoggerAdapter verbose: server-4ff9cfaa-097a-6628-a236-b6bde823ae5d: 2025-02-10T11:34:17.397Z [INFO] server-4ff9cfaa-097a-6628-a236-b6bde823ae5d: entering follower state: follower="Node at 4ff9cfaa-097a-6628-a236-b6bde823ae5d [Follower]" leader= + testing.go:149: server-4ff9cfaa-097a-6628-a236-b6bde823ae5d: 2025-02-10T11:34:17.397Z [INFO] server-4ff9cfaa-097a-6628-a236-b6bde823ae5d: entering follower state: follower="Node at 4ff9cfaa-097a-6628-a236-b6bde823ae5d [Follower]" leader= +testLoggerAdapter verbose: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.397Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ff9cfaa-097a-6628-a236-b6bde823ae5d Address:4ff9cfaa-097a-6628-a236-b6bde823ae5d} {Suffrage:Voter ID:server-b8eecf14-3b74-dcb4-d192-467613a45d7a Address:b8eecf14-3b74-dcb4-d192-467613a45d7a} {Suffrage:Voter ID:server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036 Address:486d0fd4-852d-d6fb-50fb-1b52d4fb0036}]" + testing.go:149: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.397Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ff9cfaa-097a-6628-a236-b6bde823ae5d Address:4ff9cfaa-097a-6628-a236-b6bde823ae5d} {Suffrage:Voter ID:server-b8eecf14-3b74-dcb4-d192-467613a45d7a Address:b8eecf14-3b74-dcb4-d192-467613a45d7a} {Suffrage:Voter ID:server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036 Address:486d0fd4-852d-d6fb-50fb-1b52d4fb0036}]" +testLoggerAdapter verbose: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.397Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: entering follower state: follower="Node at b8eecf14-3b74-dcb4-d192-467613a45d7a [Follower]" leader= +testLoggerAdapter verbose: server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036: 2025-02-10T11:34:17.397Z [INFO] server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ff9cfaa-097a-6628-a236-b6bde823ae5d Address:4ff9cfaa-097a-6628-a236-b6bde823ae5d} {Suffrage:Voter ID:server-b8eecf14-3b74-dcb4-d192-467613a45d7a Address:b8eecf14-3b74-dcb4-d192-467613a45d7a} {Suffrage:Voter ID:server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036 Address:486d0fd4-852d-d6fb-50fb-1b52d4fb0036}]" + testing.go:149: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.397Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: entering follower state: follower="Node at b8eecf14-3b74-dcb4-d192-467613a45d7a [Follower]" leader= + testing.go:149: server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036: 2025-02-10T11:34:17.397Z [INFO] server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ff9cfaa-097a-6628-a236-b6bde823ae5d Address:4ff9cfaa-097a-6628-a236-b6bde823ae5d} {Suffrage:Voter ID:server-b8eecf14-3b74-dcb4-d192-467613a45d7a Address:b8eecf14-3b74-dcb4-d192-467613a45d7a} {Suffrage:Voter ID:server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036 Address:486d0fd4-852d-d6fb-50fb-1b52d4fb0036}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:17.397Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:17.397Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036: 2025-02-10T11:34:17.397Z [INFO] server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036: entering follower state: follower="Node at 486d0fd4-852d-d6fb-50fb-1b52d4fb0036 [Follower]" leader= + testing.go:149: server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036: 2025-02-10T11:34:17.397Z [INFO] server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036: entering follower state: follower="Node at 486d0fd4-852d-d6fb-50fb-1b52d4fb0036 [Follower]" leader= +testLoggerAdapter verbose: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.452Z [WARN] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.452Z [WARN] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.452Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: entering candidate state: node="Node at b8eecf14-3b74-dcb4-d192-467613a45d7a [Candidate]" term=2 + testing.go:149: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.452Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: entering candidate state: node="Node at b8eecf14-3b74-dcb4-d192-467613a45d7a [Candidate]" term=2 +testLoggerAdapter verbose: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.452Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: election won: tally=2 + testing.go:149: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.452Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: election won: tally=2 +testLoggerAdapter verbose: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.452Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: entering leader state: leader="Node at b8eecf14-3b74-dcb4-d192-467613a45d7a [Leader]" + testing.go:149: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.452Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: entering leader state: leader="Node at b8eecf14-3b74-dcb4-d192-467613a45d7a [Leader]" +testLoggerAdapter verbose: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.452Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: added peer, starting replication: peer=server-4ff9cfaa-097a-6628-a236-b6bde823ae5d + testing.go:149: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.452Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: added peer, starting replication: peer=server-4ff9cfaa-097a-6628-a236-b6bde823ae5d +testLoggerAdapter verbose: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.452Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: added peer, starting replication: peer=server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036 + testing.go:149: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.452Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: added peer, starting replication: peer=server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036 +testLoggerAdapter verbose: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.452Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: pipelining replication: peer="{Voter server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036 486d0fd4-852d-d6fb-50fb-1b52d4fb0036}" + testing.go:149: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.452Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: pipelining replication: peer="{Voter server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036 486d0fd4-852d-d6fb-50fb-1b52d4fb0036}" +testLoggerAdapter verbose: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.452Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: pipelining replication: peer="{Voter server-4ff9cfaa-097a-6628-a236-b6bde823ae5d 4ff9cfaa-097a-6628-a236-b6bde823ae5d}" + testing.go:149: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.452Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: pipelining replication: peer="{Voter server-4ff9cfaa-097a-6628-a236-b6bde823ae5d 4ff9cfaa-097a-6628-a236-b6bde823ae5d}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:17.557Z [INFO] stable state for Leader reached at 2025-02-10 11:34:17.452459672 +0000 UTC m=+13.821364052 (1 nodes), 54.846404ms from start of poll, 55.236084ms from cluster start. Timeout at 2025-02-10 11:34:17.557586813 +0000 UTC m=+13.926491223, 105.127171ms after stability + testing.go:149: cluster: 2025-02-10T11:34:17.557Z [INFO] stable state for Leader reached at 2025-02-10 11:34:17.452459672 +0000 UTC m=+13.821364052 (1 nodes), 54.846404ms from start of poll, 55.236084ms from cluster start. Timeout at 2025-02-10 11:34:17.557586813 +0000 UTC m=+13.926491223, 105.127171ms after stability +testLoggerAdapter verbose: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.570Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: aborting pipeline replication: peer="{Voter server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036 486d0fd4-852d-d6fb-50fb-1b52d4fb0036}" + testing.go:149: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.570Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: aborting pipeline replication: peer="{Voter server-486d0fd4-852d-d6fb-50fb-1b52d4fb0036 486d0fd4-852d-d6fb-50fb-1b52d4fb0036}" +testLoggerAdapter verbose: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.570Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: aborting pipeline replication: peer="{Voter server-4ff9cfaa-097a-6628-a236-b6bde823ae5d 4ff9cfaa-097a-6628-a236-b6bde823ae5d}" + testing.go:149: server-b8eecf14-3b74-dcb4-d192-467613a45d7a: 2025-02-10T11:34:17.570Z [INFO] server-b8eecf14-3b74-dcb4-d192-467613a45d7a: aborting pipeline replication: peer="{Voter server-4ff9cfaa-097a-6628-a236-b6bde823ae5d 4ff9cfaa-097a-6628-a236-b6bde823ae5d}" +--- PASS: TestRaft_Barrier (0.17s) === RUN TestRaft_VerifyLeader -testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.519Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c7c37b4-200b-3db9-5c3a-b14689d5edba Address:1c7c37b4-200b-3db9-5c3a-b14689d5edba} {Suffrage:Voter ID:server-3920bd6f-c487-705e-b37a-27e1c98948b6 Address:3920bd6f-c487-705e-b37a-27e1c98948b6} {Suffrage:Voter ID:server-c982249b-8fe4-c5b8-b954-c74b18753511 Address:c982249b-8fe4-c5b8-b954-c74b18753511}]" - testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.519Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c7c37b4-200b-3db9-5c3a-b14689d5edba Address:1c7c37b4-200b-3db9-5c3a-b14689d5edba} {Suffrage:Voter ID:server-3920bd6f-c487-705e-b37a-27e1c98948b6 Address:3920bd6f-c487-705e-b37a-27e1c98948b6} {Suffrage:Voter ID:server-c982249b-8fe4-c5b8-b954-c74b18753511 Address:c982249b-8fe4-c5b8-b954-c74b18753511}]" -testLoggerAdapter verbose: server-3920bd6f-c487-705e-b37a-27e1c98948b6: 2024-01-09T05:10:03.519Z [INFO] server-3920bd6f-c487-705e-b37a-27e1c98948b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c7c37b4-200b-3db9-5c3a-b14689d5edba Address:1c7c37b4-200b-3db9-5c3a-b14689d5edba} {Suffrage:Voter ID:server-3920bd6f-c487-705e-b37a-27e1c98948b6 Address:3920bd6f-c487-705e-b37a-27e1c98948b6} {Suffrage:Voter ID:server-c982249b-8fe4-c5b8-b954-c74b18753511 Address:c982249b-8fe4-c5b8-b954-c74b18753511}]" - testing.go:149: server-3920bd6f-c487-705e-b37a-27e1c98948b6: 2024-01-09T05:10:03.519Z [INFO] server-3920bd6f-c487-705e-b37a-27e1c98948b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c7c37b4-200b-3db9-5c3a-b14689d5edba Address:1c7c37b4-200b-3db9-5c3a-b14689d5edba} {Suffrage:Voter ID:server-3920bd6f-c487-705e-b37a-27e1c98948b6 Address:3920bd6f-c487-705e-b37a-27e1c98948b6} {Suffrage:Voter ID:server-c982249b-8fe4-c5b8-b954-c74b18753511 Address:c982249b-8fe4-c5b8-b954-c74b18753511}]" -testLoggerAdapter verbose: server-c982249b-8fe4-c5b8-b954-c74b18753511: 2024-01-09T05:10:03.519Z [INFO] server-c982249b-8fe4-c5b8-b954-c74b18753511: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c7c37b4-200b-3db9-5c3a-b14689d5edba Address:1c7c37b4-200b-3db9-5c3a-b14689d5edba} {Suffrage:Voter ID:server-3920bd6f-c487-705e-b37a-27e1c98948b6 Address:3920bd6f-c487-705e-b37a-27e1c98948b6} {Suffrage:Voter ID:server-c982249b-8fe4-c5b8-b954-c74b18753511 Address:c982249b-8fe4-c5b8-b954-c74b18753511}]" - testing.go:149: server-c982249b-8fe4-c5b8-b954-c74b18753511: 2024-01-09T05:10:03.519Z [INFO] server-c982249b-8fe4-c5b8-b954-c74b18753511: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c7c37b4-200b-3db9-5c3a-b14689d5edba Address:1c7c37b4-200b-3db9-5c3a-b14689d5edba} {Suffrage:Voter ID:server-3920bd6f-c487-705e-b37a-27e1c98948b6 Address:3920bd6f-c487-705e-b37a-27e1c98948b6} {Suffrage:Voter ID:server-c982249b-8fe4-c5b8-b954-c74b18753511 Address:c982249b-8fe4-c5b8-b954-c74b18753511}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.519Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:03.519Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.519Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: entering follower state: follower="Node at 1c7c37b4-200b-3db9-5c3a-b14689d5edba [Follower]" leader= - testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.519Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: entering follower state: follower="Node at 1c7c37b4-200b-3db9-5c3a-b14689d5edba [Follower]" leader= -testLoggerAdapter verbose: server-3920bd6f-c487-705e-b37a-27e1c98948b6: 2024-01-09T05:10:03.519Z [INFO] server-3920bd6f-c487-705e-b37a-27e1c98948b6: entering follower state: follower="Node at 3920bd6f-c487-705e-b37a-27e1c98948b6 [Follower]" leader= - testing.go:149: server-3920bd6f-c487-705e-b37a-27e1c98948b6: 2024-01-09T05:10:03.519Z [INFO] server-3920bd6f-c487-705e-b37a-27e1c98948b6: entering follower state: follower="Node at 3920bd6f-c487-705e-b37a-27e1c98948b6 [Follower]" leader= -testLoggerAdapter verbose: server-c982249b-8fe4-c5b8-b954-c74b18753511: 2024-01-09T05:10:03.520Z [INFO] server-c982249b-8fe4-c5b8-b954-c74b18753511: entering follower state: follower="Node at c982249b-8fe4-c5b8-b954-c74b18753511 [Follower]" leader= - testing.go:149: server-c982249b-8fe4-c5b8-b954-c74b18753511: 2024-01-09T05:10:03.520Z [INFO] server-c982249b-8fe4-c5b8-b954-c74b18753511: entering follower state: follower="Node at c982249b-8fe4-c5b8-b954-c74b18753511 [Follower]" leader= -testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [WARN] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [WARN] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: entering candidate state: node="Node at 1c7c37b4-200b-3db9-5c3a-b14689d5edba [Candidate]" term=2 - testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: entering candidate state: node="Node at 1c7c37b4-200b-3db9-5c3a-b14689d5edba [Candidate]" term=2 -testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: election won: tally=2 - testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: election won: tally=2 -testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: entering leader state: leader="Node at 1c7c37b4-200b-3db9-5c3a-b14689d5edba [Leader]" - testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: entering leader state: leader="Node at 1c7c37b4-200b-3db9-5c3a-b14689d5edba [Leader]" -testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: added peer, starting replication: peer=server-3920bd6f-c487-705e-b37a-27e1c98948b6 - testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: added peer, starting replication: peer=server-3920bd6f-c487-705e-b37a-27e1c98948b6 -testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: added peer, starting replication: peer=server-c982249b-8fe4-c5b8-b954-c74b18753511 - testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: added peer, starting replication: peer=server-c982249b-8fe4-c5b8-b954-c74b18753511 -testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: pipelining replication: peer="{Voter server-3920bd6f-c487-705e-b37a-27e1c98948b6 3920bd6f-c487-705e-b37a-27e1c98948b6}" - testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: pipelining replication: peer="{Voter server-3920bd6f-c487-705e-b37a-27e1c98948b6 3920bd6f-c487-705e-b37a-27e1c98948b6}" -testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: pipelining replication: peer="{Voter server-c982249b-8fe4-c5b8-b954-c74b18753511 c982249b-8fe4-c5b8-b954-c74b18753511}" - testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: pipelining replication: peer="{Voter server-c982249b-8fe4-c5b8-b954-c74b18753511 c982249b-8fe4-c5b8-b954-c74b18753511}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.677Z [INFO] stable state for Leader reached at 2024-01-09 05:10:03.571655419 +0000 UTC m=+14.824706500 (1 nodes), 51.773319ms from start of poll, 52.327196ms from cluster start. Timeout at 2024-01-09 05:10:03.67708264 +0000 UTC m=+14.930133622, 105.427122ms after stability - testing.go:149: cluster: 2024-01-09T05:10:03.677Z [INFO] stable state for Leader reached at 2024-01-09 05:10:03.571655419 +0000 UTC m=+14.824706500 (1 nodes), 51.773319ms from start of poll, 52.327196ms from cluster start. Timeout at 2024-01-09 05:10:03.67708264 +0000 UTC m=+14.930133622, 105.427122ms after stability -testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.677Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: aborting pipeline replication: peer="{Voter server-c982249b-8fe4-c5b8-b954-c74b18753511 c982249b-8fe4-c5b8-b954-c74b18753511}" - testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.677Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: aborting pipeline replication: peer="{Voter server-c982249b-8fe4-c5b8-b954-c74b18753511 c982249b-8fe4-c5b8-b954-c74b18753511}" -testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.677Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: aborting pipeline replication: peer="{Voter server-3920bd6f-c487-705e-b37a-27e1c98948b6 3920bd6f-c487-705e-b37a-27e1c98948b6}" - testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.677Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: aborting pipeline replication: peer="{Voter server-3920bd6f-c487-705e-b37a-27e1c98948b6 3920bd6f-c487-705e-b37a-27e1c98948b6}" -testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.727Z [ERROR] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: failed to heartbeat to: peer=3920bd6f-c487-705e-b37a-27e1c98948b6 error="command timed out" - testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.727Z [ERROR] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: failed to heartbeat to: peer=3920bd6f-c487-705e-b37a-27e1c98948b6 error="command timed out" ---- PASS: TestRaft_VerifyLeader (0.21s) +testLoggerAdapter verbose: server-637376c3-5987-6205-0524-1df5e1074f91: 2025-02-10T11:34:17.571Z [INFO] server-637376c3-5987-6205-0524-1df5e1074f91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-637376c3-5987-6205-0524-1df5e1074f91 Address:637376c3-5987-6205-0524-1df5e1074f91} {Suffrage:Voter ID:server-07c83c14-1bc4-c7cc-280d-08154b7a3beb Address:07c83c14-1bc4-c7cc-280d-08154b7a3beb} {Suffrage:Voter ID:server-449d221e-17ce-46a1-5002-49732756b151 Address:449d221e-17ce-46a1-5002-49732756b151}]" + testing.go:149: server-637376c3-5987-6205-0524-1df5e1074f91: 2025-02-10T11:34:17.571Z [INFO] server-637376c3-5987-6205-0524-1df5e1074f91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-637376c3-5987-6205-0524-1df5e1074f91 Address:637376c3-5987-6205-0524-1df5e1074f91} {Suffrage:Voter ID:server-07c83c14-1bc4-c7cc-280d-08154b7a3beb Address:07c83c14-1bc4-c7cc-280d-08154b7a3beb} {Suffrage:Voter ID:server-449d221e-17ce-46a1-5002-49732756b151 Address:449d221e-17ce-46a1-5002-49732756b151}]" +testLoggerAdapter verbose: server-637376c3-5987-6205-0524-1df5e1074f91: 2025-02-10T11:34:17.571Z [INFO] server-637376c3-5987-6205-0524-1df5e1074f91: entering follower state: follower="Node at 637376c3-5987-6205-0524-1df5e1074f91 [Follower]" leader= + testing.go:149: server-637376c3-5987-6205-0524-1df5e1074f91: 2025-02-10T11:34:17.571Z [INFO] server-637376c3-5987-6205-0524-1df5e1074f91: entering follower state: follower="Node at 637376c3-5987-6205-0524-1df5e1074f91 [Follower]" leader= +testLoggerAdapter verbose: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.571Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-637376c3-5987-6205-0524-1df5e1074f91 Address:637376c3-5987-6205-0524-1df5e1074f91} {Suffrage:Voter ID:server-07c83c14-1bc4-c7cc-280d-08154b7a3beb Address:07c83c14-1bc4-c7cc-280d-08154b7a3beb} {Suffrage:Voter ID:server-449d221e-17ce-46a1-5002-49732756b151 Address:449d221e-17ce-46a1-5002-49732756b151}]" + testing.go:149: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.571Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-637376c3-5987-6205-0524-1df5e1074f91 Address:637376c3-5987-6205-0524-1df5e1074f91} {Suffrage:Voter ID:server-07c83c14-1bc4-c7cc-280d-08154b7a3beb Address:07c83c14-1bc4-c7cc-280d-08154b7a3beb} {Suffrage:Voter ID:server-449d221e-17ce-46a1-5002-49732756b151 Address:449d221e-17ce-46a1-5002-49732756b151}]" +testLoggerAdapter verbose: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.571Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: entering follower state: follower="Node at 07c83c14-1bc4-c7cc-280d-08154b7a3beb [Follower]" leader= + testing.go:149: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.571Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: entering follower state: follower="Node at 07c83c14-1bc4-c7cc-280d-08154b7a3beb [Follower]" leader= +testLoggerAdapter verbose: server-449d221e-17ce-46a1-5002-49732756b151: 2025-02-10T11:34:17.571Z [INFO] server-449d221e-17ce-46a1-5002-49732756b151: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-637376c3-5987-6205-0524-1df5e1074f91 Address:637376c3-5987-6205-0524-1df5e1074f91} {Suffrage:Voter ID:server-07c83c14-1bc4-c7cc-280d-08154b7a3beb Address:07c83c14-1bc4-c7cc-280d-08154b7a3beb} {Suffrage:Voter ID:server-449d221e-17ce-46a1-5002-49732756b151 Address:449d221e-17ce-46a1-5002-49732756b151}]" + testing.go:149: server-449d221e-17ce-46a1-5002-49732756b151: 2025-02-10T11:34:17.571Z [INFO] server-449d221e-17ce-46a1-5002-49732756b151: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-637376c3-5987-6205-0524-1df5e1074f91 Address:637376c3-5987-6205-0524-1df5e1074f91} {Suffrage:Voter ID:server-07c83c14-1bc4-c7cc-280d-08154b7a3beb Address:07c83c14-1bc4-c7cc-280d-08154b7a3beb} {Suffrage:Voter ID:server-449d221e-17ce-46a1-5002-49732756b151 Address:449d221e-17ce-46a1-5002-49732756b151}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:17.571Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:17.571Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-449d221e-17ce-46a1-5002-49732756b151: 2025-02-10T11:34:17.571Z [INFO] server-449d221e-17ce-46a1-5002-49732756b151: entering follower state: follower="Node at 449d221e-17ce-46a1-5002-49732756b151 [Follower]" leader= + testing.go:149: server-449d221e-17ce-46a1-5002-49732756b151: 2025-02-10T11:34:17.571Z [INFO] server-449d221e-17ce-46a1-5002-49732756b151: entering follower state: follower="Node at 449d221e-17ce-46a1-5002-49732756b151 [Follower]" leader= +testLoggerAdapter verbose: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.623Z [WARN] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.623Z [WARN] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.623Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: entering candidate state: node="Node at 07c83c14-1bc4-c7cc-280d-08154b7a3beb [Candidate]" term=2 + testing.go:149: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.623Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: entering candidate state: node="Node at 07c83c14-1bc4-c7cc-280d-08154b7a3beb [Candidate]" term=2 +testLoggerAdapter verbose: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.623Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: election won: tally=2 + testing.go:149: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.623Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: election won: tally=2 +testLoggerAdapter verbose: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.623Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: entering leader state: leader="Node at 07c83c14-1bc4-c7cc-280d-08154b7a3beb [Leader]" + testing.go:149: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.623Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: entering leader state: leader="Node at 07c83c14-1bc4-c7cc-280d-08154b7a3beb [Leader]" +testLoggerAdapter verbose: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.623Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: added peer, starting replication: peer=server-637376c3-5987-6205-0524-1df5e1074f91 + testing.go:149: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.623Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: added peer, starting replication: peer=server-637376c3-5987-6205-0524-1df5e1074f91 +testLoggerAdapter verbose: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.623Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: added peer, starting replication: peer=server-449d221e-17ce-46a1-5002-49732756b151 + testing.go:149: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.623Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: added peer, starting replication: peer=server-449d221e-17ce-46a1-5002-49732756b151 +testLoggerAdapter verbose: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.623Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: pipelining replication: peer="{Voter server-449d221e-17ce-46a1-5002-49732756b151 449d221e-17ce-46a1-5002-49732756b151}" + testing.go:149: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.623Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: pipelining replication: peer="{Voter server-449d221e-17ce-46a1-5002-49732756b151 449d221e-17ce-46a1-5002-49732756b151}" +testLoggerAdapter verbose: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.623Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: pipelining replication: peer="{Voter server-637376c3-5987-6205-0524-1df5e1074f91 637376c3-5987-6205-0524-1df5e1074f91}" + testing.go:149: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.623Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: pipelining replication: peer="{Voter server-637376c3-5987-6205-0524-1df5e1074f91 637376c3-5987-6205-0524-1df5e1074f91}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:17.728Z [INFO] stable state for Leader reached at 2025-02-10 11:34:17.623304925 +0000 UTC m=+13.992209225 (1 nodes), 51.907423ms from start of poll, 52.339024ms from cluster start. Timeout at 2025-02-10 11:34:17.728376301 +0000 UTC m=+14.097280597, 105.071372ms after stability + testing.go:149: cluster: 2025-02-10T11:34:17.728Z [INFO] stable state for Leader reached at 2025-02-10 11:34:17.623304925 +0000 UTC m=+13.992209225 (1 nodes), 51.907423ms from start of poll, 52.339024ms from cluster start. Timeout at 2025-02-10 11:34:17.728376301 +0000 UTC m=+14.097280597, 105.071372ms after stability +testLoggerAdapter verbose: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.728Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: aborting pipeline replication: peer="{Voter server-637376c3-5987-6205-0524-1df5e1074f91 637376c3-5987-6205-0524-1df5e1074f91}" + testing.go:149: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.728Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: aborting pipeline replication: peer="{Voter server-637376c3-5987-6205-0524-1df5e1074f91 637376c3-5987-6205-0524-1df5e1074f91}" +testLoggerAdapter verbose: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.728Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: aborting pipeline replication: peer="{Voter server-449d221e-17ce-46a1-5002-49732756b151 449d221e-17ce-46a1-5002-49732756b151}" + testing.go:149: server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: 2025-02-10T11:34:17.728Z [INFO] server-07c83c14-1bc4-c7cc-280d-08154b7a3beb: aborting pipeline replication: peer="{Voter server-449d221e-17ce-46a1-5002-49732756b151 449d221e-17ce-46a1-5002-49732756b151}" +--- PASS: TestRaft_VerifyLeader (0.16s) === RUN TestRaft_VerifyLeader_Single -testLoggerAdapter verbose: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.728Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e68225b-b745-0c7c-2387-ad9ed2c891b1 Address:3e68225b-b745-0c7c-2387-ad9ed2c891b1}]" - testing.go:149: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.728Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e68225b-b745-0c7c-2387-ad9ed2c891b1 Address:3e68225b-b745-0c7c-2387-ad9ed2c891b1}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.728Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:03.728Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.728Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: entering follower state: follower="Node at 3e68225b-b745-0c7c-2387-ad9ed2c891b1 [Follower]" leader= - testing.go:149: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.728Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: entering follower state: follower="Node at 3e68225b-b745-0c7c-2387-ad9ed2c891b1 [Follower]" leader= -testLoggerAdapter verbose: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.830Z [WARN] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.830Z [WARN] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.831Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: entering candidate state: node="Node at 3e68225b-b745-0c7c-2387-ad9ed2c891b1 [Candidate]" term=2 - testing.go:149: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.831Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: entering candidate state: node="Node at 3e68225b-b745-0c7c-2387-ad9ed2c891b1 [Candidate]" term=2 -testLoggerAdapter verbose: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.831Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: election won: tally=1 - testing.go:149: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.831Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: election won: tally=1 -testLoggerAdapter verbose: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.831Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: entering leader state: leader="Node at 3e68225b-b745-0c7c-2387-ad9ed2c891b1 [Leader]" - testing.go:149: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.831Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: entering leader state: leader="Node at 3e68225b-b745-0c7c-2387-ad9ed2c891b1 [Leader]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.936Z [INFO] stable state for Leader reached at 2024-01-09 05:10:03.831268687 +0000 UTC m=+15.084319711 (1 nodes), 102.548012ms from start of poll, 102.755078ms from cluster start. Timeout at 2024-01-09 05:10:03.936411313 +0000 UTC m=+15.189462630, 105.142919ms after stability - testing.go:149: cluster: 2024-01-09T05:10:03.936Z [INFO] stable state for Leader reached at 2024-01-09 05:10:03.831268687 +0000 UTC m=+15.084319711 (1 nodes), 102.548012ms from start of poll, 102.755078ms from cluster start. Timeout at 2024-01-09 05:10:03.936411313 +0000 UTC m=+15.189462630, 105.142919ms after stability ---- PASS: TestRaft_VerifyLeader_Single (0.21s) +testLoggerAdapter verbose: server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: 2025-02-10T11:34:17.729Z [INFO] server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb5183fb-121f-cc46-6595-e7bd8f536dc0 Address:fb5183fb-121f-cc46-6595-e7bd8f536dc0}]" + testing.go:149: server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: 2025-02-10T11:34:17.729Z [INFO] server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb5183fb-121f-cc46-6595-e7bd8f536dc0 Address:fb5183fb-121f-cc46-6595-e7bd8f536dc0}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:17.729Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:17.729Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: 2025-02-10T11:34:17.729Z [INFO] server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: entering follower state: follower="Node at fb5183fb-121f-cc46-6595-e7bd8f536dc0 [Follower]" leader= + testing.go:149: server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: 2025-02-10T11:34:17.729Z [INFO] server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: entering follower state: follower="Node at fb5183fb-121f-cc46-6595-e7bd8f536dc0 [Follower]" leader= +testLoggerAdapter verbose: server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: 2025-02-10T11:34:17.793Z [WARN] server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: 2025-02-10T11:34:17.793Z [WARN] server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: 2025-02-10T11:34:17.793Z [INFO] server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: entering candidate state: node="Node at fb5183fb-121f-cc46-6595-e7bd8f536dc0 [Candidate]" term=2 + testing.go:149: server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: 2025-02-10T11:34:17.793Z [INFO] server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: entering candidate state: node="Node at fb5183fb-121f-cc46-6595-e7bd8f536dc0 [Candidate]" term=2 +testLoggerAdapter verbose: server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: 2025-02-10T11:34:17.793Z [INFO] server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: election won: tally=1 + testing.go:149: server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: 2025-02-10T11:34:17.793Z [INFO] server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: election won: tally=1 +testLoggerAdapter verbose: server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: 2025-02-10T11:34:17.793Z [INFO] server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: entering leader state: leader="Node at fb5183fb-121f-cc46-6595-e7bd8f536dc0 [Leader]" + testing.go:149: server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: 2025-02-10T11:34:17.793Z [INFO] server-fb5183fb-121f-cc46-6595-e7bd8f536dc0: entering leader state: leader="Node at fb5183fb-121f-cc46-6595-e7bd8f536dc0 [Leader]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:17.898Z [INFO] stable state for Leader reached at 2025-02-10 11:34:17.793507339 +0000 UTC m=+14.162411638 (1 nodes), 64.19744ms from start of poll, 64.353711ms from cluster start. Timeout at 2025-02-10 11:34:17.898583979 +0000 UTC m=+14.267488276, 105.076638ms after stability + testing.go:149: cluster: 2025-02-10T11:34:17.898Z [INFO] stable state for Leader reached at 2025-02-10 11:34:17.793507339 +0000 UTC m=+14.162411638 (1 nodes), 64.19744ms from start of poll, 64.353711ms from cluster start. Timeout at 2025-02-10 11:34:17.898583979 +0000 UTC m=+14.267488276, 105.076638ms after stability +--- PASS: TestRaft_VerifyLeader_Single (0.17s) === RUN TestRaft_VerifyLeader_Fail -testLoggerAdapter verbose: server-1e63e209-64c4-2b05-52c3-18126e9220e0: 2024-01-09T05:10:03.937Z [INFO] server-1e63e209-64c4-2b05-52c3-18126e9220e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e63e209-64c4-2b05-52c3-18126e9220e0 Address:1e63e209-64c4-2b05-52c3-18126e9220e0} {Suffrage:Voter ID:server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd Address:ed1a0628-4a30-7e8e-ce29-f1b582d055fd}]" - testing.go:149: server-1e63e209-64c4-2b05-52c3-18126e9220e0: 2024-01-09T05:10:03.937Z [INFO] server-1e63e209-64c4-2b05-52c3-18126e9220e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e63e209-64c4-2b05-52c3-18126e9220e0 Address:1e63e209-64c4-2b05-52c3-18126e9220e0} {Suffrage:Voter ID:server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd Address:ed1a0628-4a30-7e8e-ce29-f1b582d055fd}]" -testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:03.937Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e63e209-64c4-2b05-52c3-18126e9220e0 Address:1e63e209-64c4-2b05-52c3-18126e9220e0} {Suffrage:Voter ID:server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd Address:ed1a0628-4a30-7e8e-ce29-f1b582d055fd}]" - testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:03.937Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e63e209-64c4-2b05-52c3-18126e9220e0 Address:1e63e209-64c4-2b05-52c3-18126e9220e0} {Suffrage:Voter ID:server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd Address:ed1a0628-4a30-7e8e-ce29-f1b582d055fd}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.937Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:03.937Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-1e63e209-64c4-2b05-52c3-18126e9220e0: 2024-01-09T05:10:03.937Z [INFO] server-1e63e209-64c4-2b05-52c3-18126e9220e0: entering follower state: follower="Node at 1e63e209-64c4-2b05-52c3-18126e9220e0 [Follower]" leader= - testing.go:149: server-1e63e209-64c4-2b05-52c3-18126e9220e0: 2024-01-09T05:10:03.937Z [INFO] server-1e63e209-64c4-2b05-52c3-18126e9220e0: entering follower state: follower="Node at 1e63e209-64c4-2b05-52c3-18126e9220e0 [Follower]" leader= -testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:03.937Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: entering follower state: follower="Node at ed1a0628-4a30-7e8e-ce29-f1b582d055fd [Follower]" leader= - testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:03.937Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: entering follower state: follower="Node at ed1a0628-4a30-7e8e-ce29-f1b582d055fd [Follower]" leader= -testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [WARN] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [WARN] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: entering candidate state: node="Node at ed1a0628-4a30-7e8e-ce29-f1b582d055fd [Candidate]" term=2 - testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: entering candidate state: node="Node at ed1a0628-4a30-7e8e-ce29-f1b582d055fd [Candidate]" term=2 -testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: election won: tally=2 - testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: election won: tally=2 -testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: entering leader state: leader="Node at ed1a0628-4a30-7e8e-ce29-f1b582d055fd [Leader]" - testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: entering leader state: leader="Node at ed1a0628-4a30-7e8e-ce29-f1b582d055fd [Leader]" -testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: added peer, starting replication: peer=server-1e63e209-64c4-2b05-52c3-18126e9220e0 - testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: added peer, starting replication: peer=server-1e63e209-64c4-2b05-52c3-18126e9220e0 -testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: pipelining replication: peer="{Voter server-1e63e209-64c4-2b05-52c3-18126e9220e0 1e63e209-64c4-2b05-52c3-18126e9220e0}" - testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: pipelining replication: peer="{Voter server-1e63e209-64c4-2b05-52c3-18126e9220e0 1e63e209-64c4-2b05-52c3-18126e9220e0}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.111Z [INFO] stable state for Leader reached at 2024-01-09 05:10:04.006601085 +0000 UTC m=+15.259652265 (1 nodes), 68.881808ms from start of poll, 69.230638ms from cluster start. Timeout at 2024-01-09 05:10:04.111632579 +0000 UTC m=+15.364683629, 105.031364ms after stability - testing.go:149: cluster: 2024-01-09T05:10:04.111Z [INFO] stable state for Leader reached at 2024-01-09 05:10:04.006601085 +0000 UTC m=+15.259652265 (1 nodes), 68.881808ms from start of poll, 69.230638ms from cluster start. Timeout at 2024-01-09 05:10:04.111632579 +0000 UTC m=+15.364683629, 105.031364ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.111Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:10:04.111Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.217Z [INFO] stable state for Follower reached at 2024-01-09 05:10:04.111827989 +0000 UTC m=+15.364878993 (1 nodes), 2.149µs from start of poll, 174.457366ms from cluster start. Timeout at 2024-01-09 05:10:04.217572196 +0000 UTC m=+15.470623402, 105.744409ms after stability - testing.go:149: cluster: 2024-01-09T05:10:04.217Z [INFO] stable state for Follower reached at 2024-01-09 05:10:04.111827989 +0000 UTC m=+15.364878993 (1 nodes), 2.149µs from start of poll, 174.457366ms from cluster start. Timeout at 2024-01-09 05:10:04.217572196 +0000 UTC m=+15.470623402, 105.744409ms after stability -testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.217Z [WARN] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: new leader elected, stepping down - testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.217Z [WARN] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: new leader elected, stepping down -testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.217Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: entering follower state: follower="Node at ed1a0628-4a30-7e8e-ce29-f1b582d055fd [Follower]" leader= - testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.217Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: entering follower state: follower="Node at ed1a0628-4a30-7e8e-ce29-f1b582d055fd [Follower]" leader= -testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.217Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: aborting pipeline replication: peer="{Voter server-1e63e209-64c4-2b05-52c3-18126e9220e0 1e63e209-64c4-2b05-52c3-18126e9220e0}" - testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.217Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: aborting pipeline replication: peer="{Voter server-1e63e209-64c4-2b05-52c3-18126e9220e0 1e63e209-64c4-2b05-52c3-18126e9220e0}" +testLoggerAdapter verbose: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:17.899Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5df9c90-71f4-faab-2144-f384ec8b5b64 Address:d5df9c90-71f4-faab-2144-f384ec8b5b64} {Suffrage:Voter ID:server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52 Address:47ce0eb2-eeda-d98d-aac1-2bd025bddc52}]" + testing.go:149: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:17.899Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5df9c90-71f4-faab-2144-f384ec8b5b64 Address:d5df9c90-71f4-faab-2144-f384ec8b5b64} {Suffrage:Voter ID:server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52 Address:47ce0eb2-eeda-d98d-aac1-2bd025bddc52}]" +testLoggerAdapter verbose: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:17.899Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: entering follower state: follower="Node at d5df9c90-71f4-faab-2144-f384ec8b5b64 [Follower]" leader= + testing.go:149: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:17.899Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: entering follower state: follower="Node at d5df9c90-71f4-faab-2144-f384ec8b5b64 [Follower]" leader= +testLoggerAdapter verbose: server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52: 2025-02-10T11:34:17.899Z [INFO] server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5df9c90-71f4-faab-2144-f384ec8b5b64 Address:d5df9c90-71f4-faab-2144-f384ec8b5b64} {Suffrage:Voter ID:server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52 Address:47ce0eb2-eeda-d98d-aac1-2bd025bddc52}]" + testing.go:149: server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52: 2025-02-10T11:34:17.899Z [INFO] server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5df9c90-71f4-faab-2144-f384ec8b5b64 Address:d5df9c90-71f4-faab-2144-f384ec8b5b64} {Suffrage:Voter ID:server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52 Address:47ce0eb2-eeda-d98d-aac1-2bd025bddc52}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:17.899Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:17.899Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52: 2025-02-10T11:34:17.899Z [INFO] server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52: entering follower state: follower="Node at 47ce0eb2-eeda-d98d-aac1-2bd025bddc52 [Follower]" leader= + testing.go:149: server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52: 2025-02-10T11:34:17.899Z [INFO] server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52: entering follower state: follower="Node at 47ce0eb2-eeda-d98d-aac1-2bd025bddc52 [Follower]" leader= +testLoggerAdapter verbose: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:17.965Z [WARN] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:17.965Z [WARN] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:17.965Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: entering candidate state: node="Node at d5df9c90-71f4-faab-2144-f384ec8b5b64 [Candidate]" term=2 + testing.go:149: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:17.965Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: entering candidate state: node="Node at d5df9c90-71f4-faab-2144-f384ec8b5b64 [Candidate]" term=2 +testLoggerAdapter verbose: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:17.965Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: election won: tally=2 + testing.go:149: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:17.965Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: election won: tally=2 +testLoggerAdapter verbose: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:17.965Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: entering leader state: leader="Node at d5df9c90-71f4-faab-2144-f384ec8b5b64 [Leader]" + testing.go:149: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:17.965Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: entering leader state: leader="Node at d5df9c90-71f4-faab-2144-f384ec8b5b64 [Leader]" +testLoggerAdapter verbose: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:17.965Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: added peer, starting replication: peer=server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52 + testing.go:149: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:17.965Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: added peer, starting replication: peer=server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52 +testLoggerAdapter verbose: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:17.965Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: pipelining replication: peer="{Voter server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52 47ce0eb2-eeda-d98d-aac1-2bd025bddc52}" + testing.go:149: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:17.965Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: pipelining replication: peer="{Voter server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52 47ce0eb2-eeda-d98d-aac1-2bd025bddc52}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:18.070Z [INFO] stable state for Leader reached at 2025-02-10 11:34:17.96558081 +0000 UTC m=+14.334485112 (1 nodes), 66.10447ms from start of poll, 66.38718ms from cluster start. Timeout at 2025-02-10 11:34:18.070655788 +0000 UTC m=+14.439560106, 105.074994ms after stability + testing.go:149: cluster: 2025-02-10T11:34:18.070Z [INFO] stable state for Leader reached at 2025-02-10 11:34:17.96558081 +0000 UTC m=+14.334485112 (1 nodes), 66.10447ms from start of poll, 66.38718ms from cluster start. Timeout at 2025-02-10 11:34:18.070655788 +0000 UTC m=+14.439560106, 105.074994ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:18.070Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:18.070Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:18.175Z [INFO] stable state for Follower reached at 2025-02-10 11:34:18.070765975 +0000 UTC m=+14.439670279 (1 nodes), 554ns from start of poll, 171.572347ms from cluster start. Timeout at 2025-02-10 11:34:18.175868455 +0000 UTC m=+14.544772804, 105.102525ms after stability + testing.go:149: cluster: 2025-02-10T11:34:18.175Z [INFO] stable state for Follower reached at 2025-02-10 11:34:18.070765975 +0000 UTC m=+14.439670279 (1 nodes), 554ns from start of poll, 171.572347ms from cluster start. Timeout at 2025-02-10 11:34:18.175868455 +0000 UTC m=+14.544772804, 105.102525ms after stability +testLoggerAdapter verbose: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:18.176Z [WARN] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: new leader elected, stepping down + testing.go:149: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:18.176Z [WARN] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: new leader elected, stepping down +testLoggerAdapter verbose: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:18.176Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: entering follower state: follower="Node at d5df9c90-71f4-faab-2144-f384ec8b5b64 [Follower]" leader= + testing.go:149: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:18.176Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: entering follower state: follower="Node at d5df9c90-71f4-faab-2144-f384ec8b5b64 [Follower]" leader= +testLoggerAdapter verbose: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:18.176Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: aborting pipeline replication: peer="{Voter server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52 47ce0eb2-eeda-d98d-aac1-2bd025bddc52}" + testing.go:149: server-d5df9c90-71f4-faab-2144-f384ec8b5b64: 2025-02-10T11:34:18.176Z [INFO] server-d5df9c90-71f4-faab-2144-f384ec8b5b64: aborting pipeline replication: peer="{Voter server-47ce0eb2-eeda-d98d-aac1-2bd025bddc52 47ce0eb2-eeda-d98d-aac1-2bd025bddc52}" --- PASS: TestRaft_VerifyLeader_Fail (0.28s) === RUN TestRaft_VerifyLeader_PartialConnect -testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.219Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-261bdfd8-57b8-4b39-9ec0-85027b02a719 Address:261bdfd8-57b8-4b39-9ec0-85027b02a719} {Suffrage:Voter ID:server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba Address:dd4c5831-2bac-9bc1-3abf-300dec9f0eba} {Suffrage:Voter ID:server-6d20a6da-abe8-43f1-271c-64588c3397d2 Address:6d20a6da-abe8-43f1-271c-64588c3397d2}]" - testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.219Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-261bdfd8-57b8-4b39-9ec0-85027b02a719 Address:261bdfd8-57b8-4b39-9ec0-85027b02a719} {Suffrage:Voter ID:server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba Address:dd4c5831-2bac-9bc1-3abf-300dec9f0eba} {Suffrage:Voter ID:server-6d20a6da-abe8-43f1-271c-64588c3397d2 Address:6d20a6da-abe8-43f1-271c-64588c3397d2}]" -testLoggerAdapter verbose: server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba: 2024-01-09T05:10:04.219Z [INFO] server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-261bdfd8-57b8-4b39-9ec0-85027b02a719 Address:261bdfd8-57b8-4b39-9ec0-85027b02a719} {Suffrage:Voter ID:server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba Address:dd4c5831-2bac-9bc1-3abf-300dec9f0eba} {Suffrage:Voter ID:server-6d20a6da-abe8-43f1-271c-64588c3397d2 Address:6d20a6da-abe8-43f1-271c-64588c3397d2}]" - testing.go:149: server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba: 2024-01-09T05:10:04.219Z [INFO] server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-261bdfd8-57b8-4b39-9ec0-85027b02a719 Address:261bdfd8-57b8-4b39-9ec0-85027b02a719} {Suffrage:Voter ID:server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba Address:dd4c5831-2bac-9bc1-3abf-300dec9f0eba} {Suffrage:Voter ID:server-6d20a6da-abe8-43f1-271c-64588c3397d2 Address:6d20a6da-abe8-43f1-271c-64588c3397d2}]" -testLoggerAdapter verbose: server-6d20a6da-abe8-43f1-271c-64588c3397d2: 2024-01-09T05:10:04.219Z [INFO] server-6d20a6da-abe8-43f1-271c-64588c3397d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-261bdfd8-57b8-4b39-9ec0-85027b02a719 Address:261bdfd8-57b8-4b39-9ec0-85027b02a719} {Suffrage:Voter ID:server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba Address:dd4c5831-2bac-9bc1-3abf-300dec9f0eba} {Suffrage:Voter ID:server-6d20a6da-abe8-43f1-271c-64588c3397d2 Address:6d20a6da-abe8-43f1-271c-64588c3397d2}]" - testing.go:149: server-6d20a6da-abe8-43f1-271c-64588c3397d2: 2024-01-09T05:10:04.219Z [INFO] server-6d20a6da-abe8-43f1-271c-64588c3397d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-261bdfd8-57b8-4b39-9ec0-85027b02a719 Address:261bdfd8-57b8-4b39-9ec0-85027b02a719} {Suffrage:Voter ID:server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba Address:dd4c5831-2bac-9bc1-3abf-300dec9f0eba} {Suffrage:Voter ID:server-6d20a6da-abe8-43f1-271c-64588c3397d2 Address:6d20a6da-abe8-43f1-271c-64588c3397d2}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.219Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:04.219Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.219Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: entering follower state: follower="Node at 261bdfd8-57b8-4b39-9ec0-85027b02a719 [Follower]" leader= - testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.219Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: entering follower state: follower="Node at 261bdfd8-57b8-4b39-9ec0-85027b02a719 [Follower]" leader= -testLoggerAdapter verbose: server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba: 2024-01-09T05:10:04.220Z [INFO] server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba: entering follower state: follower="Node at dd4c5831-2bac-9bc1-3abf-300dec9f0eba [Follower]" leader= - testing.go:149: server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba: 2024-01-09T05:10:04.220Z [INFO] server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba: entering follower state: follower="Node at dd4c5831-2bac-9bc1-3abf-300dec9f0eba [Follower]" leader= -testLoggerAdapter verbose: server-6d20a6da-abe8-43f1-271c-64588c3397d2: 2024-01-09T05:10:04.220Z [INFO] server-6d20a6da-abe8-43f1-271c-64588c3397d2: entering follower state: follower="Node at 6d20a6da-abe8-43f1-271c-64588c3397d2 [Follower]" leader= - testing.go:149: server-6d20a6da-abe8-43f1-271c-64588c3397d2: 2024-01-09T05:10:04.220Z [INFO] server-6d20a6da-abe8-43f1-271c-64588c3397d2: entering follower state: follower="Node at 6d20a6da-abe8-43f1-271c-64588c3397d2 [Follower]" leader= -testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [WARN] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [WARN] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: entering candidate state: node="Node at 261bdfd8-57b8-4b39-9ec0-85027b02a719 [Candidate]" term=2 - testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: entering candidate state: node="Node at 261bdfd8-57b8-4b39-9ec0-85027b02a719 [Candidate]" term=2 -testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: election won: tally=2 - testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: election won: tally=2 -testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: entering leader state: leader="Node at 261bdfd8-57b8-4b39-9ec0-85027b02a719 [Leader]" - testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: entering leader state: leader="Node at 261bdfd8-57b8-4b39-9ec0-85027b02a719 [Leader]" -testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: added peer, starting replication: peer=server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba - testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: added peer, starting replication: peer=server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba -testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: added peer, starting replication: peer=server-6d20a6da-abe8-43f1-271c-64588c3397d2 - testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: added peer, starting replication: peer=server-6d20a6da-abe8-43f1-271c-64588c3397d2 -testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: pipelining replication: peer="{Voter server-6d20a6da-abe8-43f1-271c-64588c3397d2 6d20a6da-abe8-43f1-271c-64588c3397d2}" - testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: pipelining replication: peer="{Voter server-6d20a6da-abe8-43f1-271c-64588c3397d2 6d20a6da-abe8-43f1-271c-64588c3397d2}" -testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: pipelining replication: peer="{Voter server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba dd4c5831-2bac-9bc1-3abf-300dec9f0eba}" - testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: pipelining replication: peer="{Voter server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba dd4c5831-2bac-9bc1-3abf-300dec9f0eba}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.398Z [INFO] stable state for Leader reached at 2024-01-09 05:10:04.292872676 +0000 UTC m=+15.545923742 (1 nodes), 72.910326ms from start of poll, 73.459003ms from cluster start. Timeout at 2024-01-09 05:10:04.398089643 +0000 UTC m=+15.651148361, 105.224619ms after stability - testing.go:149: cluster: 2024-01-09T05:10:04.398Z [INFO] stable state for Leader reached at 2024-01-09 05:10:04.292872676 +0000 UTC m=+15.545923742 (1 nodes), 72.910326ms from start of poll, 73.459003ms from cluster start. Timeout at 2024-01-09 05:10:04.398089643 +0000 UTC m=+15.651148361, 105.224619ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.403Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:10:04.403Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.509Z [INFO] stable state for Follower reached at 2024-01-09 05:10:04.40409861 +0000 UTC m=+15.657149745 (2 nodes), 3.019µs from start of poll, 184.685006ms from cluster start. Timeout at 2024-01-09 05:10:04.509273419 +0000 UTC m=+15.762324500, 105.174755ms after stability - testing.go:149: cluster: 2024-01-09T05:10:04.509Z [INFO] stable state for Follower reached at 2024-01-09 05:10:04.40409861 +0000 UTC m=+15.657149745 (2 nodes), 3.019µs from start of poll, 184.685006ms from cluster start. Timeout at 2024-01-09 05:10:04.509273419 +0000 UTC m=+15.762324500, 105.174755ms after stability - raft_test.go:1647: [INFO] Disconnecting Node at dd4c5831-2bac-9bc1-3abf-300dec9f0eba [Follower] -testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.509Z [ERROR] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: failed to heartbeat to: peer=dd4c5831-2bac-9bc1-3abf-300dec9f0eba error="failed to connect to peer: dd4c5831-2bac-9bc1-3abf-300dec9f0eba" - testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.509Z [ERROR] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: failed to heartbeat to: peer=dd4c5831-2bac-9bc1-3abf-300dec9f0eba error="failed to connect to peer: dd4c5831-2bac-9bc1-3abf-300dec9f0eba" -testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.509Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: aborting pipeline replication: peer="{Voter server-6d20a6da-abe8-43f1-271c-64588c3397d2 6d20a6da-abe8-43f1-271c-64588c3397d2}" - testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.509Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: aborting pipeline replication: peer="{Voter server-6d20a6da-abe8-43f1-271c-64588c3397d2 6d20a6da-abe8-43f1-271c-64588c3397d2}" -testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.510Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: aborting pipeline replication: peer="{Voter server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba dd4c5831-2bac-9bc1-3abf-300dec9f0eba}" - testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.510Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: aborting pipeline replication: peer="{Voter server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba dd4c5831-2bac-9bc1-3abf-300dec9f0eba}" ---- PASS: TestRaft_VerifyLeader_PartialConnect (0.29s) +testLoggerAdapter verbose: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.176Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f Address:bac5c5bb-1eac-a02d-69eb-ea4fc942051f} {Suffrage:Voter ID:server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd Address:03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd} {Suffrage:Voter ID:server-12ed9356-7797-de21-5e72-dd20b1a0314b Address:12ed9356-7797-de21-5e72-dd20b1a0314b}]" + testing.go:149: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.176Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f Address:bac5c5bb-1eac-a02d-69eb-ea4fc942051f} {Suffrage:Voter ID:server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd Address:03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd} {Suffrage:Voter ID:server-12ed9356-7797-de21-5e72-dd20b1a0314b Address:12ed9356-7797-de21-5e72-dd20b1a0314b}]" +testLoggerAdapter verbose: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.177Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: entering follower state: follower="Node at bac5c5bb-1eac-a02d-69eb-ea4fc942051f [Follower]" leader= + testing.go:149: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.177Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: entering follower state: follower="Node at bac5c5bb-1eac-a02d-69eb-ea4fc942051f [Follower]" leader= +testLoggerAdapter verbose: server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd: 2025-02-10T11:34:18.177Z [INFO] server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f Address:bac5c5bb-1eac-a02d-69eb-ea4fc942051f} {Suffrage:Voter ID:server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd Address:03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd} {Suffrage:Voter ID:server-12ed9356-7797-de21-5e72-dd20b1a0314b Address:12ed9356-7797-de21-5e72-dd20b1a0314b}]" + testing.go:149: server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd: 2025-02-10T11:34:18.177Z [INFO] server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f Address:bac5c5bb-1eac-a02d-69eb-ea4fc942051f} {Suffrage:Voter ID:server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd Address:03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd} {Suffrage:Voter ID:server-12ed9356-7797-de21-5e72-dd20b1a0314b Address:12ed9356-7797-de21-5e72-dd20b1a0314b}]" +testLoggerAdapter verbose: server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd: 2025-02-10T11:34:18.177Z [INFO] server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd: entering follower state: follower="Node at 03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd [Follower]" leader= + testing.go:149: server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd: 2025-02-10T11:34:18.177Z [INFO] server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd: entering follower state: follower="Node at 03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd [Follower]" leader= +testLoggerAdapter verbose: server-12ed9356-7797-de21-5e72-dd20b1a0314b: 2025-02-10T11:34:18.177Z [INFO] server-12ed9356-7797-de21-5e72-dd20b1a0314b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f Address:bac5c5bb-1eac-a02d-69eb-ea4fc942051f} {Suffrage:Voter ID:server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd Address:03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd} {Suffrage:Voter ID:server-12ed9356-7797-de21-5e72-dd20b1a0314b Address:12ed9356-7797-de21-5e72-dd20b1a0314b}]" + testing.go:149: server-12ed9356-7797-de21-5e72-dd20b1a0314b: 2025-02-10T11:34:18.177Z [INFO] server-12ed9356-7797-de21-5e72-dd20b1a0314b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f Address:bac5c5bb-1eac-a02d-69eb-ea4fc942051f} {Suffrage:Voter ID:server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd Address:03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd} {Suffrage:Voter ID:server-12ed9356-7797-de21-5e72-dd20b1a0314b Address:12ed9356-7797-de21-5e72-dd20b1a0314b}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:18.177Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:18.177Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-12ed9356-7797-de21-5e72-dd20b1a0314b: 2025-02-10T11:34:18.177Z [INFO] server-12ed9356-7797-de21-5e72-dd20b1a0314b: entering follower state: follower="Node at 12ed9356-7797-de21-5e72-dd20b1a0314b [Follower]" leader= + testing.go:149: server-12ed9356-7797-de21-5e72-dd20b1a0314b: 2025-02-10T11:34:18.177Z [INFO] server-12ed9356-7797-de21-5e72-dd20b1a0314b: entering follower state: follower="Node at 12ed9356-7797-de21-5e72-dd20b1a0314b [Follower]" leader= +testLoggerAdapter verbose: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.230Z [WARN] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.230Z [WARN] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.230Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: entering candidate state: node="Node at bac5c5bb-1eac-a02d-69eb-ea4fc942051f [Candidate]" term=2 + testing.go:149: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.230Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: entering candidate state: node="Node at bac5c5bb-1eac-a02d-69eb-ea4fc942051f [Candidate]" term=2 +testLoggerAdapter verbose: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.230Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: election won: tally=2 + testing.go:149: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.230Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: election won: tally=2 +testLoggerAdapter verbose: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.230Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: entering leader state: leader="Node at bac5c5bb-1eac-a02d-69eb-ea4fc942051f [Leader]" + testing.go:149: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.230Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: entering leader state: leader="Node at bac5c5bb-1eac-a02d-69eb-ea4fc942051f [Leader]" +testLoggerAdapter verbose: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.230Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: added peer, starting replication: peer=server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd + testing.go:149: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.230Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: added peer, starting replication: peer=server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd +testLoggerAdapter verbose: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.230Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: added peer, starting replication: peer=server-12ed9356-7797-de21-5e72-dd20b1a0314b + testing.go:149: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.230Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: added peer, starting replication: peer=server-12ed9356-7797-de21-5e72-dd20b1a0314b +testLoggerAdapter verbose: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.230Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: pipelining replication: peer="{Voter server-12ed9356-7797-de21-5e72-dd20b1a0314b 12ed9356-7797-de21-5e72-dd20b1a0314b}" + testing.go:149: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.230Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: pipelining replication: peer="{Voter server-12ed9356-7797-de21-5e72-dd20b1a0314b 12ed9356-7797-de21-5e72-dd20b1a0314b}" +testLoggerAdapter verbose: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.230Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: pipelining replication: peer="{Voter server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd 03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd}" + testing.go:149: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.230Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: pipelining replication: peer="{Voter server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd 03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:18.335Z [INFO] stable state for Leader reached at 2025-02-10 11:34:18.2306041 +0000 UTC m=+14.599508400 (1 nodes), 53.351796ms from start of poll, 53.721178ms from cluster start. Timeout at 2025-02-10 11:34:18.335707291 +0000 UTC m=+14.704611587, 105.103187ms after stability + testing.go:149: cluster: 2025-02-10T11:34:18.335Z [INFO] stable state for Leader reached at 2025-02-10 11:34:18.2306041 +0000 UTC m=+14.599508400 (1 nodes), 53.351796ms from start of poll, 53.721178ms from cluster start. Timeout at 2025-02-10 11:34:18.335707291 +0000 UTC m=+14.704611587, 105.103187ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:18.340Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:18.340Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:18.446Z [INFO] stable state for Follower reached at 2025-02-10 11:34:18.340941489 +0000 UTC m=+14.709845855 (2 nodes), 1.53µs from start of poll, 164.058633ms from cluster start. Timeout at 2025-02-10 11:34:18.446005357 +0000 UTC m=+14.814909710, 105.063855ms after stability + testing.go:149: cluster: 2025-02-10T11:34:18.446Z [INFO] stable state for Follower reached at 2025-02-10 11:34:18.340941489 +0000 UTC m=+14.709845855 (2 nodes), 1.53µs from start of poll, 164.058633ms from cluster start. Timeout at 2025-02-10 11:34:18.446005357 +0000 UTC m=+14.814909710, 105.063855ms after stability + raft_test.go:1647: [INFO] Disconnecting Node at 03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd [Follower] +testLoggerAdapter verbose: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.446Z [ERROR] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: failed to heartbeat to: peer=03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd error="failed to connect to peer: 03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd" + testing.go:149: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.446Z [ERROR] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: failed to heartbeat to: peer=03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd error="failed to connect to peer: 03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd" +testLoggerAdapter verbose: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.446Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: aborting pipeline replication: peer="{Voter server-12ed9356-7797-de21-5e72-dd20b1a0314b 12ed9356-7797-de21-5e72-dd20b1a0314b}" + testing.go:149: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.446Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: aborting pipeline replication: peer="{Voter server-12ed9356-7797-de21-5e72-dd20b1a0314b 12ed9356-7797-de21-5e72-dd20b1a0314b}" +testLoggerAdapter verbose: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.446Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: aborting pipeline replication: peer="{Voter server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd 03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd}" + testing.go:149: server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: 2025-02-10T11:34:18.446Z [INFO] server-bac5c5bb-1eac-a02d-69eb-ea4fc942051f: aborting pipeline replication: peer="{Voter server-03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd 03a0d760-2d05-e76c-b1b7-d8b35d1fa7bd}" +--- PASS: TestRaft_VerifyLeader_PartialConnect (0.27s) === RUN TestRaft_StartAsLeader -testLoggerAdapter verbose: server-2b62827e-6ae8-577c-5095-5baed64efa70: 2024-01-09T05:10:04.512Z [INFO] server-2b62827e-6ae8-577c-5095-5baed64efa70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b62827e-6ae8-577c-5095-5baed64efa70 Address:2b62827e-6ae8-577c-5095-5baed64efa70}]" - testing.go:149: server-2b62827e-6ae8-577c-5095-5baed64efa70: 2024-01-09T05:10:04.512Z [INFO] server-2b62827e-6ae8-577c-5095-5baed64efa70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b62827e-6ae8-577c-5095-5baed64efa70 Address:2b62827e-6ae8-577c-5095-5baed64efa70}]" -testLoggerAdapter verbose: server-2b62827e-6ae8-577c-5095-5baed64efa70: 2024-01-09T05:10:04.512Z [INFO] server-2b62827e-6ae8-577c-5095-5baed64efa70: entering leader state: leader="Node at 2b62827e-6ae8-577c-5095-5baed64efa70 [Leader]" - testing.go:149: server-2b62827e-6ae8-577c-5095-5baed64efa70: 2024-01-09T05:10:04.512Z [INFO] server-2b62827e-6ae8-577c-5095-5baed64efa70: entering leader state: leader="Node at 2b62827e-6ae8-577c-5095-5baed64efa70 [Leader]" +testLoggerAdapter verbose: server-b88fa5e6-0613-426c-d773-9351a28558dd: 2025-02-10T11:34:18.446Z [INFO] server-b88fa5e6-0613-426c-d773-9351a28558dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b88fa5e6-0613-426c-d773-9351a28558dd Address:b88fa5e6-0613-426c-d773-9351a28558dd}]" + testing.go:149: server-b88fa5e6-0613-426c-d773-9351a28558dd: 2025-02-10T11:34:18.446Z [INFO] server-b88fa5e6-0613-426c-d773-9351a28558dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b88fa5e6-0613-426c-d773-9351a28558dd Address:b88fa5e6-0613-426c-d773-9351a28558dd}]" +testLoggerAdapter verbose: server-b88fa5e6-0613-426c-d773-9351a28558dd: 2025-02-10T11:34:18.446Z [INFO] server-b88fa5e6-0613-426c-d773-9351a28558dd: entering leader state: leader="Node at b88fa5e6-0613-426c-d773-9351a28558dd [Leader]" + testing.go:149: server-b88fa5e6-0613-426c-d773-9351a28558dd: 2025-02-10T11:34:18.446Z [INFO] server-b88fa5e6-0613-426c-d773-9351a28558dd: entering leader state: leader="Node at b88fa5e6-0613-426c-d773-9351a28558dd [Leader]" --- PASS: TestRaft_StartAsLeader (0.00s) === RUN TestRaft_NotifyCh -testLoggerAdapter verbose: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.514Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da418d31-c1b2-2c51-7e6e-31147f72440d Address:da418d31-c1b2-2c51-7e6e-31147f72440d}]" - testing.go:149: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.514Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da418d31-c1b2-2c51-7e6e-31147f72440d Address:da418d31-c1b2-2c51-7e6e-31147f72440d}]" -testLoggerAdapter verbose: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.514Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: entering follower state: follower="Node at da418d31-c1b2-2c51-7e6e-31147f72440d [Follower]" leader= - testing.go:149: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.514Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: entering follower state: follower="Node at da418d31-c1b2-2c51-7e6e-31147f72440d [Follower]" leader= -testLoggerAdapter verbose: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.578Z [WARN] server-da418d31-c1b2-2c51-7e6e-31147f72440d: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.578Z [WARN] server-da418d31-c1b2-2c51-7e6e-31147f72440d: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.578Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: entering candidate state: node="Node at da418d31-c1b2-2c51-7e6e-31147f72440d [Candidate]" term=2 - testing.go:149: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.578Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: entering candidate state: node="Node at da418d31-c1b2-2c51-7e6e-31147f72440d [Candidate]" term=2 -testLoggerAdapter verbose: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.578Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: election won: tally=1 - testing.go:149: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.578Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: election won: tally=1 -testLoggerAdapter verbose: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.578Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: entering leader state: leader="Node at da418d31-c1b2-2c51-7e6e-31147f72440d [Leader]" - testing.go:149: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.578Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: entering leader state: leader="Node at da418d31-c1b2-2c51-7e6e-31147f72440d [Leader]" ---- PASS: TestRaft_NotifyCh (0.07s) +testLoggerAdapter verbose: server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: 2025-02-10T11:34:18.447Z [INFO] server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98ed08f6-7bd1-9066-db4d-500934aeb0f4 Address:98ed08f6-7bd1-9066-db4d-500934aeb0f4}]" + testing.go:149: server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: 2025-02-10T11:34:18.447Z [INFO] server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98ed08f6-7bd1-9066-db4d-500934aeb0f4 Address:98ed08f6-7bd1-9066-db4d-500934aeb0f4}]" +testLoggerAdapter verbose: server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: 2025-02-10T11:34:18.447Z [INFO] server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: entering follower state: follower="Node at 98ed08f6-7bd1-9066-db4d-500934aeb0f4 [Follower]" leader= + testing.go:149: server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: 2025-02-10T11:34:18.447Z [INFO] server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: entering follower state: follower="Node at 98ed08f6-7bd1-9066-db4d-500934aeb0f4 [Follower]" leader= +testLoggerAdapter verbose: server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: 2025-02-10T11:34:18.504Z [WARN] server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: 2025-02-10T11:34:18.504Z [WARN] server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: 2025-02-10T11:34:18.504Z [INFO] server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: entering candidate state: node="Node at 98ed08f6-7bd1-9066-db4d-500934aeb0f4 [Candidate]" term=2 + testing.go:149: server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: 2025-02-10T11:34:18.504Z [INFO] server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: entering candidate state: node="Node at 98ed08f6-7bd1-9066-db4d-500934aeb0f4 [Candidate]" term=2 +testLoggerAdapter verbose: server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: 2025-02-10T11:34:18.504Z [INFO] server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: election won: tally=1 + testing.go:149: server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: 2025-02-10T11:34:18.504Z [INFO] server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: election won: tally=1 +testLoggerAdapter verbose: server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: 2025-02-10T11:34:18.504Z [INFO] server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: entering leader state: leader="Node at 98ed08f6-7bd1-9066-db4d-500934aeb0f4 [Leader]" + testing.go:149: server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: 2025-02-10T11:34:18.504Z [INFO] server-98ed08f6-7bd1-9066-db4d-500934aeb0f4: entering leader state: leader="Node at 98ed08f6-7bd1-9066-db4d-500934aeb0f4 [Leader]" +--- PASS: TestRaft_NotifyCh (0.06s) === RUN TestRaft_Voting -testLoggerAdapter verbose: server-3620caaf-fe14-274d-b295-300856ae8562: 2024-01-09T05:10:04.580Z [INFO] server-3620caaf-fe14-274d-b295-300856ae8562: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3620caaf-fe14-274d-b295-300856ae8562 Address:3620caaf-fe14-274d-b295-300856ae8562} {Suffrage:Voter ID:server-40772aff-3515-f043-13c1-d269661cbc18 Address:40772aff-3515-f043-13c1-d269661cbc18} {Suffrage:Voter ID:server-537d52a1-1d79-f546-03cd-2e13d8290052 Address:537d52a1-1d79-f546-03cd-2e13d8290052}]" - testing.go:149: server-3620caaf-fe14-274d-b295-300856ae8562: 2024-01-09T05:10:04.580Z [INFO] server-3620caaf-fe14-274d-b295-300856ae8562: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3620caaf-fe14-274d-b295-300856ae8562 Address:3620caaf-fe14-274d-b295-300856ae8562} {Suffrage:Voter ID:server-40772aff-3515-f043-13c1-d269661cbc18 Address:40772aff-3515-f043-13c1-d269661cbc18} {Suffrage:Voter ID:server-537d52a1-1d79-f546-03cd-2e13d8290052 Address:537d52a1-1d79-f546-03cd-2e13d8290052}]" -testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.580Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3620caaf-fe14-274d-b295-300856ae8562 Address:3620caaf-fe14-274d-b295-300856ae8562} {Suffrage:Voter ID:server-40772aff-3515-f043-13c1-d269661cbc18 Address:40772aff-3515-f043-13c1-d269661cbc18} {Suffrage:Voter ID:server-537d52a1-1d79-f546-03cd-2e13d8290052 Address:537d52a1-1d79-f546-03cd-2e13d8290052}]" - testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.580Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3620caaf-fe14-274d-b295-300856ae8562 Address:3620caaf-fe14-274d-b295-300856ae8562} {Suffrage:Voter ID:server-40772aff-3515-f043-13c1-d269661cbc18 Address:40772aff-3515-f043-13c1-d269661cbc18} {Suffrage:Voter ID:server-537d52a1-1d79-f546-03cd-2e13d8290052 Address:537d52a1-1d79-f546-03cd-2e13d8290052}]" -testLoggerAdapter verbose: server-537d52a1-1d79-f546-03cd-2e13d8290052: 2024-01-09T05:10:04.580Z [INFO] server-537d52a1-1d79-f546-03cd-2e13d8290052: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3620caaf-fe14-274d-b295-300856ae8562 Address:3620caaf-fe14-274d-b295-300856ae8562} {Suffrage:Voter ID:server-40772aff-3515-f043-13c1-d269661cbc18 Address:40772aff-3515-f043-13c1-d269661cbc18} {Suffrage:Voter ID:server-537d52a1-1d79-f546-03cd-2e13d8290052 Address:537d52a1-1d79-f546-03cd-2e13d8290052}]" - testing.go:149: server-537d52a1-1d79-f546-03cd-2e13d8290052: 2024-01-09T05:10:04.580Z [INFO] server-537d52a1-1d79-f546-03cd-2e13d8290052: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3620caaf-fe14-274d-b295-300856ae8562 Address:3620caaf-fe14-274d-b295-300856ae8562} {Suffrage:Voter ID:server-40772aff-3515-f043-13c1-d269661cbc18 Address:40772aff-3515-f043-13c1-d269661cbc18} {Suffrage:Voter ID:server-537d52a1-1d79-f546-03cd-2e13d8290052 Address:537d52a1-1d79-f546-03cd-2e13d8290052}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.580Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:10:04.580Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-3620caaf-fe14-274d-b295-300856ae8562: 2024-01-09T05:10:04.580Z [INFO] server-3620caaf-fe14-274d-b295-300856ae8562: entering follower state: follower="Node at 3620caaf-fe14-274d-b295-300856ae8562 [Follower]" leader= - testing.go:149: server-3620caaf-fe14-274d-b295-300856ae8562: 2024-01-09T05:10:04.580Z [INFO] server-3620caaf-fe14-274d-b295-300856ae8562: entering follower state: follower="Node at 3620caaf-fe14-274d-b295-300856ae8562 [Follower]" leader= -testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.580Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: entering follower state: follower="Node at 40772aff-3515-f043-13c1-d269661cbc18 [Follower]" leader= - testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.580Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: entering follower state: follower="Node at 40772aff-3515-f043-13c1-d269661cbc18 [Follower]" leader= -testLoggerAdapter verbose: server-537d52a1-1d79-f546-03cd-2e13d8290052: 2024-01-09T05:10:04.580Z [INFO] server-537d52a1-1d79-f546-03cd-2e13d8290052: entering follower state: follower="Node at 537d52a1-1d79-f546-03cd-2e13d8290052 [Follower]" leader= - testing.go:149: server-537d52a1-1d79-f546-03cd-2e13d8290052: 2024-01-09T05:10:04.580Z [INFO] server-537d52a1-1d79-f546-03cd-2e13d8290052: entering follower state: follower="Node at 537d52a1-1d79-f546-03cd-2e13d8290052 [Follower]" leader= -testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [WARN] server-40772aff-3515-f043-13c1-d269661cbc18: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [WARN] server-40772aff-3515-f043-13c1-d269661cbc18: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: entering candidate state: node="Node at 40772aff-3515-f043-13c1-d269661cbc18 [Candidate]" term=2 - testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: entering candidate state: node="Node at 40772aff-3515-f043-13c1-d269661cbc18 [Candidate]" term=2 -testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: election won: tally=2 - testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: election won: tally=2 -testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: entering leader state: leader="Node at 40772aff-3515-f043-13c1-d269661cbc18 [Leader]" - testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: entering leader state: leader="Node at 40772aff-3515-f043-13c1-d269661cbc18 [Leader]" -testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: added peer, starting replication: peer=server-3620caaf-fe14-274d-b295-300856ae8562 - testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: added peer, starting replication: peer=server-3620caaf-fe14-274d-b295-300856ae8562 -testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: added peer, starting replication: peer=server-537d52a1-1d79-f546-03cd-2e13d8290052 - testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: added peer, starting replication: peer=server-537d52a1-1d79-f546-03cd-2e13d8290052 -testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: pipelining replication: peer="{Voter server-537d52a1-1d79-f546-03cd-2e13d8290052 537d52a1-1d79-f546-03cd-2e13d8290052}" - testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: pipelining replication: peer="{Voter server-537d52a1-1d79-f546-03cd-2e13d8290052 537d52a1-1d79-f546-03cd-2e13d8290052}" -testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: pipelining replication: peer="{Voter server-3620caaf-fe14-274d-b295-300856ae8562 3620caaf-fe14-274d-b295-300856ae8562}" - testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: pipelining replication: peer="{Voter server-3620caaf-fe14-274d-b295-300856ae8562 3620caaf-fe14-274d-b295-300856ae8562}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.766Z [INFO] stable state for Follower reached at 2024-01-09 05:10:04.660952146 +0000 UTC m=+15.914003288 (2 nodes), 80.371784ms from start of poll, 80.882219ms from cluster start. Timeout at 2024-01-09 05:10:04.766178865 +0000 UTC m=+16.019229914, 105.226626ms after stability - testing.go:149: cluster: 2024-01-09T05:10:04.766Z [INFO] stable state for Follower reached at 2024-01-09 05:10:04.660952146 +0000 UTC m=+15.914003288 (2 nodes), 80.371784ms from start of poll, 80.882219ms from cluster start. Timeout at 2024-01-09 05:10:04.766178865 +0000 UTC m=+16.019229914, 105.226626ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.766Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:04.766Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.872Z [INFO] stable state for Leader reached at 2024-01-09 05:10:04.76745132 +0000 UTC m=+16.020502450 (1 nodes), 3.18µs from start of poll, 187.381381ms from cluster start. Timeout at 2024-01-09 05:10:04.872564895 +0000 UTC m=+16.125616108, 105.113658ms after stability - testing.go:149: cluster: 2024-01-09T05:10:04.872Z [INFO] stable state for Leader reached at 2024-01-09 05:10:04.76745132 +0000 UTC m=+16.020502450 (1 nodes), 3.18µs from start of poll, 187.381381ms from cluster start. Timeout at 2024-01-09 05:10:04.872564895 +0000 UTC m=+16.125616108, 105.113658ms after stability -testLoggerAdapter verbose: server-537d52a1-1d79-f546-03cd-2e13d8290052: 2024-01-09T05:10:04.873Z [WARN] server-537d52a1-1d79-f546-03cd-2e13d8290052: rejecting vote request since we have a leader: from=3620caaf-fe14-274d-b295-300856ae8562 leader=40772aff-3515-f043-13c1-d269661cbc18 - testing.go:149: server-537d52a1-1d79-f546-03cd-2e13d8290052: 2024-01-09T05:10:04.873Z [WARN] server-537d52a1-1d79-f546-03cd-2e13d8290052: rejecting vote request since we have a leader: from=3620caaf-fe14-274d-b295-300856ae8562 leader=40772aff-3515-f043-13c1-d269661cbc18 -testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.873Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: aborting pipeline replication: peer="{Voter server-537d52a1-1d79-f546-03cd-2e13d8290052 537d52a1-1d79-f546-03cd-2e13d8290052}" - testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.873Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: aborting pipeline replication: peer="{Voter server-537d52a1-1d79-f546-03cd-2e13d8290052 537d52a1-1d79-f546-03cd-2e13d8290052}" -testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.873Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: aborting pipeline replication: peer="{Voter server-3620caaf-fe14-274d-b295-300856ae8562 3620caaf-fe14-274d-b295-300856ae8562}" - testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.873Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: aborting pipeline replication: peer="{Voter server-3620caaf-fe14-274d-b295-300856ae8562 3620caaf-fe14-274d-b295-300856ae8562}" ---- PASS: TestRaft_Voting (0.30s) +testLoggerAdapter verbose: server-5538d946-b962-50c5-417b-f59a15c4777e: 2025-02-10T11:34:18.505Z [INFO] server-5538d946-b962-50c5-417b-f59a15c4777e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5538d946-b962-50c5-417b-f59a15c4777e Address:5538d946-b962-50c5-417b-f59a15c4777e} {Suffrage:Voter ID:server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60 Address:73b68a81-53cb-35bf-6a1b-4f62aad6ca60} {Suffrage:Voter ID:server-09601f2e-2db5-c268-1618-90a2f3c92d1e Address:09601f2e-2db5-c268-1618-90a2f3c92d1e}]" + testing.go:149: server-5538d946-b962-50c5-417b-f59a15c4777e: 2025-02-10T11:34:18.505Z [INFO] server-5538d946-b962-50c5-417b-f59a15c4777e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5538d946-b962-50c5-417b-f59a15c4777e Address:5538d946-b962-50c5-417b-f59a15c4777e} {Suffrage:Voter ID:server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60 Address:73b68a81-53cb-35bf-6a1b-4f62aad6ca60} {Suffrage:Voter ID:server-09601f2e-2db5-c268-1618-90a2f3c92d1e Address:09601f2e-2db5-c268-1618-90a2f3c92d1e}]" +testLoggerAdapter verbose: server-5538d946-b962-50c5-417b-f59a15c4777e: 2025-02-10T11:34:18.505Z [INFO] server-5538d946-b962-50c5-417b-f59a15c4777e: entering follower state: follower="Node at 5538d946-b962-50c5-417b-f59a15c4777e [Follower]" leader= + testing.go:149: server-5538d946-b962-50c5-417b-f59a15c4777e: 2025-02-10T11:34:18.505Z [INFO] server-5538d946-b962-50c5-417b-f59a15c4777e: entering follower state: follower="Node at 5538d946-b962-50c5-417b-f59a15c4777e [Follower]" leader= +testLoggerAdapter verbose: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.505Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5538d946-b962-50c5-417b-f59a15c4777e Address:5538d946-b962-50c5-417b-f59a15c4777e} {Suffrage:Voter ID:server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60 Address:73b68a81-53cb-35bf-6a1b-4f62aad6ca60} {Suffrage:Voter ID:server-09601f2e-2db5-c268-1618-90a2f3c92d1e Address:09601f2e-2db5-c268-1618-90a2f3c92d1e}]" + testing.go:149: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.505Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5538d946-b962-50c5-417b-f59a15c4777e Address:5538d946-b962-50c5-417b-f59a15c4777e} {Suffrage:Voter ID:server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60 Address:73b68a81-53cb-35bf-6a1b-4f62aad6ca60} {Suffrage:Voter ID:server-09601f2e-2db5-c268-1618-90a2f3c92d1e Address:09601f2e-2db5-c268-1618-90a2f3c92d1e}]" +testLoggerAdapter verbose: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.505Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: entering follower state: follower="Node at 73b68a81-53cb-35bf-6a1b-4f62aad6ca60 [Follower]" leader= + testing.go:149: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.505Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: entering follower state: follower="Node at 73b68a81-53cb-35bf-6a1b-4f62aad6ca60 [Follower]" leader= +testLoggerAdapter verbose: server-09601f2e-2db5-c268-1618-90a2f3c92d1e: 2025-02-10T11:34:18.505Z [INFO] server-09601f2e-2db5-c268-1618-90a2f3c92d1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5538d946-b962-50c5-417b-f59a15c4777e Address:5538d946-b962-50c5-417b-f59a15c4777e} {Suffrage:Voter ID:server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60 Address:73b68a81-53cb-35bf-6a1b-4f62aad6ca60} {Suffrage:Voter ID:server-09601f2e-2db5-c268-1618-90a2f3c92d1e Address:09601f2e-2db5-c268-1618-90a2f3c92d1e}]" + testing.go:149: server-09601f2e-2db5-c268-1618-90a2f3c92d1e: 2025-02-10T11:34:18.505Z [INFO] server-09601f2e-2db5-c268-1618-90a2f3c92d1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5538d946-b962-50c5-417b-f59a15c4777e Address:5538d946-b962-50c5-417b-f59a15c4777e} {Suffrage:Voter ID:server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60 Address:73b68a81-53cb-35bf-6a1b-4f62aad6ca60} {Suffrage:Voter ID:server-09601f2e-2db5-c268-1618-90a2f3c92d1e Address:09601f2e-2db5-c268-1618-90a2f3c92d1e}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:18.505Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:18.505Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-09601f2e-2db5-c268-1618-90a2f3c92d1e: 2025-02-10T11:34:18.505Z [INFO] server-09601f2e-2db5-c268-1618-90a2f3c92d1e: entering follower state: follower="Node at 09601f2e-2db5-c268-1618-90a2f3c92d1e [Follower]" leader= + testing.go:149: server-09601f2e-2db5-c268-1618-90a2f3c92d1e: 2025-02-10T11:34:18.505Z [INFO] server-09601f2e-2db5-c268-1618-90a2f3c92d1e: entering follower state: follower="Node at 09601f2e-2db5-c268-1618-90a2f3c92d1e [Follower]" leader= +testLoggerAdapter verbose: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.570Z [WARN] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.570Z [WARN] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.570Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: entering candidate state: node="Node at 73b68a81-53cb-35bf-6a1b-4f62aad6ca60 [Candidate]" term=2 + testing.go:149: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.570Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: entering candidate state: node="Node at 73b68a81-53cb-35bf-6a1b-4f62aad6ca60 [Candidate]" term=2 +testLoggerAdapter verbose: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.570Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: election won: tally=2 + testing.go:149: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.570Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: election won: tally=2 +testLoggerAdapter verbose: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.570Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: entering leader state: leader="Node at 73b68a81-53cb-35bf-6a1b-4f62aad6ca60 [Leader]" + testing.go:149: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.570Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: entering leader state: leader="Node at 73b68a81-53cb-35bf-6a1b-4f62aad6ca60 [Leader]" +testLoggerAdapter verbose: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.570Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: added peer, starting replication: peer=server-5538d946-b962-50c5-417b-f59a15c4777e + testing.go:149: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.570Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: added peer, starting replication: peer=server-5538d946-b962-50c5-417b-f59a15c4777e +testLoggerAdapter verbose: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.570Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: added peer, starting replication: peer=server-09601f2e-2db5-c268-1618-90a2f3c92d1e + testing.go:149: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.570Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: added peer, starting replication: peer=server-09601f2e-2db5-c268-1618-90a2f3c92d1e +testLoggerAdapter verbose: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.570Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: pipelining replication: peer="{Voter server-5538d946-b962-50c5-417b-f59a15c4777e 5538d946-b962-50c5-417b-f59a15c4777e}" + testing.go:149: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.570Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: pipelining replication: peer="{Voter server-5538d946-b962-50c5-417b-f59a15c4777e 5538d946-b962-50c5-417b-f59a15c4777e}" +testLoggerAdapter verbose: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.570Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: pipelining replication: peer="{Voter server-09601f2e-2db5-c268-1618-90a2f3c92d1e 09601f2e-2db5-c268-1618-90a2f3c92d1e}" + testing.go:149: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.570Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: pipelining replication: peer="{Voter server-09601f2e-2db5-c268-1618-90a2f3c92d1e 09601f2e-2db5-c268-1618-90a2f3c92d1e}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:18.675Z [INFO] stable state for Follower reached at 2025-02-10 11:34:18.570658868 +0000 UTC m=+14.939563168 (2 nodes), 64.742183ms from start of poll, 65.137762ms from cluster start. Timeout at 2025-02-10 11:34:18.675752776 +0000 UTC m=+15.044657215, 105.094047ms after stability + testing.go:149: cluster: 2025-02-10T11:34:18.675Z [INFO] stable state for Follower reached at 2025-02-10 11:34:18.570658868 +0000 UTC m=+14.939563168 (2 nodes), 64.742183ms from start of poll, 65.137762ms from cluster start. Timeout at 2025-02-10 11:34:18.675752776 +0000 UTC m=+15.044657215, 105.094047ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:18.675Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:18.675Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:18.780Z [INFO] stable state for Leader reached at 2025-02-10 11:34:18.675893497 +0000 UTC m=+15.044797864 (1 nodes), 919ns from start of poll, 170.372458ms from cluster start. Timeout at 2025-02-10 11:34:18.780971934 +0000 UTC m=+15.149876225, 105.078361ms after stability + testing.go:149: cluster: 2025-02-10T11:34:18.780Z [INFO] stable state for Leader reached at 2025-02-10 11:34:18.675893497 +0000 UTC m=+15.044797864 (1 nodes), 919ns from start of poll, 170.372458ms from cluster start. Timeout at 2025-02-10 11:34:18.780971934 +0000 UTC m=+15.149876225, 105.078361ms after stability +testLoggerAdapter verbose: server-09601f2e-2db5-c268-1618-90a2f3c92d1e: 2025-02-10T11:34:18.781Z [WARN] server-09601f2e-2db5-c268-1618-90a2f3c92d1e: rejecting vote request since we have a leader: from=5538d946-b962-50c5-417b-f59a15c4777e leader=73b68a81-53cb-35bf-6a1b-4f62aad6ca60 + testing.go:149: server-09601f2e-2db5-c268-1618-90a2f3c92d1e: 2025-02-10T11:34:18.781Z [WARN] server-09601f2e-2db5-c268-1618-90a2f3c92d1e: rejecting vote request since we have a leader: from=5538d946-b962-50c5-417b-f59a15c4777e leader=73b68a81-53cb-35bf-6a1b-4f62aad6ca60 +testLoggerAdapter verbose: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.781Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: aborting pipeline replication: peer="{Voter server-09601f2e-2db5-c268-1618-90a2f3c92d1e 09601f2e-2db5-c268-1618-90a2f3c92d1e}" + testing.go:149: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.781Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: aborting pipeline replication: peer="{Voter server-09601f2e-2db5-c268-1618-90a2f3c92d1e 09601f2e-2db5-c268-1618-90a2f3c92d1e}" +testLoggerAdapter verbose: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.781Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: aborting pipeline replication: peer="{Voter server-5538d946-b962-50c5-417b-f59a15c4777e 5538d946-b962-50c5-417b-f59a15c4777e}" + testing.go:149: server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: 2025-02-10T11:34:18.781Z [INFO] server-73b68a81-53cb-35bf-6a1b-4f62aad6ca60: aborting pipeline replication: peer="{Voter server-5538d946-b962-50c5-417b-f59a15c4777e 5538d946-b962-50c5-417b-f59a15c4777e}" +--- PASS: TestRaft_Voting (0.28s) === RUN TestRaft_ProtocolVersion_RejectRPC -testLoggerAdapter verbose: server-8b6a6558-ed08-a050-331e-a0a39f8a19c6: 2024-01-09T05:10:04.877Z [INFO] server-8b6a6558-ed08-a050-331e-a0a39f8a19c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 Address:8b6a6558-ed08-a050-331e-a0a39f8a19c6} {Suffrage:Voter ID:server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac Address:5463e2ba-ed6b-6c8c-f662-21da3fb822ac} {Suffrage:Voter ID:server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b Address:e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b}]" - testing.go:149: server-8b6a6558-ed08-a050-331e-a0a39f8a19c6: 2024-01-09T05:10:04.877Z [INFO] server-8b6a6558-ed08-a050-331e-a0a39f8a19c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 Address:8b6a6558-ed08-a050-331e-a0a39f8a19c6} {Suffrage:Voter ID:server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac Address:5463e2ba-ed6b-6c8c-f662-21da3fb822ac} {Suffrage:Voter ID:server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b Address:e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b}]" -testLoggerAdapter verbose: server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac: 2024-01-09T05:10:04.878Z [INFO] server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 Address:8b6a6558-ed08-a050-331e-a0a39f8a19c6} {Suffrage:Voter ID:server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac Address:5463e2ba-ed6b-6c8c-f662-21da3fb822ac} {Suffrage:Voter ID:server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b Address:e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b}]" - testing.go:149: server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac: 2024-01-09T05:10:04.878Z [INFO] server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 Address:8b6a6558-ed08-a050-331e-a0a39f8a19c6} {Suffrage:Voter ID:server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac Address:5463e2ba-ed6b-6c8c-f662-21da3fb822ac} {Suffrage:Voter ID:server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b Address:e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b}]" -testLoggerAdapter verbose: server-8b6a6558-ed08-a050-331e-a0a39f8a19c6: 2024-01-09T05:10:04.878Z [INFO] server-8b6a6558-ed08-a050-331e-a0a39f8a19c6: entering follower state: follower="Node at 8b6a6558-ed08-a050-331e-a0a39f8a19c6 [Follower]" leader= - testing.go:149: server-8b6a6558-ed08-a050-331e-a0a39f8a19c6: 2024-01-09T05:10:04.878Z [INFO] server-8b6a6558-ed08-a050-331e-a0a39f8a19c6: entering follower state: follower="Node at 8b6a6558-ed08-a050-331e-a0a39f8a19c6 [Follower]" leader= -testLoggerAdapter verbose: server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac: 2024-01-09T05:10:04.878Z [INFO] server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac: entering follower state: follower="Node at 5463e2ba-ed6b-6c8c-f662-21da3fb822ac [Follower]" leader= - testing.go:149: server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac: 2024-01-09T05:10:04.878Z [INFO] server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac: entering follower state: follower="Node at 5463e2ba-ed6b-6c8c-f662-21da3fb822ac [Follower]" leader= -testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.879Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 Address:8b6a6558-ed08-a050-331e-a0a39f8a19c6} {Suffrage:Voter ID:server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac Address:5463e2ba-ed6b-6c8c-f662-21da3fb822ac} {Suffrage:Voter ID:server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b Address:e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b}]" - testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.879Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 Address:8b6a6558-ed08-a050-331e-a0a39f8a19c6} {Suffrage:Voter ID:server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac Address:5463e2ba-ed6b-6c8c-f662-21da3fb822ac} {Suffrage:Voter ID:server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b Address:e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.879Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:10:04.879Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.879Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: entering follower state: follower="Node at e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b [Follower]" leader= - testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.879Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: entering follower state: follower="Node at e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b [Follower]" leader= -testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.931Z [WARN] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.931Z [WARN] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: entering candidate state: node="Node at e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b [Candidate]" term=2 - testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: entering candidate state: node="Node at e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b [Candidate]" term=2 -testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: election won: tally=2 - testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: election won: tally=2 -testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: entering leader state: leader="Node at e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b [Leader]" - testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: entering leader state: leader="Node at e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b [Leader]" -testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: added peer, starting replication: peer=server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 - testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: added peer, starting replication: peer=server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 -testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: added peer, starting replication: peer=server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac - testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: added peer, starting replication: peer=server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac -testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: pipelining replication: peer="{Voter server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac 5463e2ba-ed6b-6c8c-f662-21da3fb822ac}" - testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: pipelining replication: peer="{Voter server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac 5463e2ba-ed6b-6c8c-f662-21da3fb822ac}" -testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: pipelining replication: peer="{Voter server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 8b6a6558-ed08-a050-331e-a0a39f8a19c6}" - testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: pipelining replication: peer="{Voter server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 8b6a6558-ed08-a050-331e-a0a39f8a19c6}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.038Z [INFO] stable state for Follower reached at 2024-01-09 05:10:04.932925112 +0000 UTC m=+16.185976253 (2 nodes), 53.543674ms from start of poll, 55.847804ms from cluster start. Timeout at 2024-01-09 05:10:05.038041127 +0000 UTC m=+16.291092273, 105.11602ms after stability - testing.go:149: cluster: 2024-01-09T05:10:05.038Z [INFO] stable state for Follower reached at 2024-01-09 05:10:04.932925112 +0000 UTC m=+16.185976253 (2 nodes), 53.543674ms from start of poll, 55.847804ms from cluster start. Timeout at 2024-01-09 05:10:05.038041127 +0000 UTC m=+16.291092273, 105.11602ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.038Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:05.038Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.143Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.038190268 +0000 UTC m=+16.291241410 (1 nodes), 2.573µs from start of poll, 161.112961ms from cluster start. Timeout at 2024-01-09 05:10:05.143296571 +0000 UTC m=+16.396347663, 105.106253ms after stability - testing.go:149: cluster: 2024-01-09T05:10:05.143Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.038190268 +0000 UTC m=+16.291241410 (1 nodes), 2.573µs from start of poll, 161.112961ms from cluster start. Timeout at 2024-01-09 05:10:05.143296571 +0000 UTC m=+16.396347663, 105.106253ms after stability -testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:05.143Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: aborting pipeline replication: peer="{Voter server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac 5463e2ba-ed6b-6c8c-f662-21da3fb822ac}" - testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:05.143Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: aborting pipeline replication: peer="{Voter server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac 5463e2ba-ed6b-6c8c-f662-21da3fb822ac}" -testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:05.143Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: aborting pipeline replication: peer="{Voter server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 8b6a6558-ed08-a050-331e-a0a39f8a19c6}" - testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:05.143Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: aborting pipeline replication: peer="{Voter server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 8b6a6558-ed08-a050-331e-a0a39f8a19c6}" ---- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s) +testLoggerAdapter verbose: server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad: 2025-02-10T11:34:18.782Z [INFO] server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad Address:e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad} {Suffrage:Voter ID:server-fab3e159-c600-8d9b-2dae-4217b3a13060 Address:fab3e159-c600-8d9b-2dae-4217b3a13060} {Suffrage:Voter ID:server-651b1828-d980-261c-fd3f-9fe7a58fc860 Address:651b1828-d980-261c-fd3f-9fe7a58fc860}]" + testing.go:149: server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad: 2025-02-10T11:34:18.782Z [INFO] server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad Address:e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad} {Suffrage:Voter ID:server-fab3e159-c600-8d9b-2dae-4217b3a13060 Address:fab3e159-c600-8d9b-2dae-4217b3a13060} {Suffrage:Voter ID:server-651b1828-d980-261c-fd3f-9fe7a58fc860 Address:651b1828-d980-261c-fd3f-9fe7a58fc860}]" +testLoggerAdapter verbose: server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad: 2025-02-10T11:34:18.782Z [INFO] server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad: entering follower state: follower="Node at e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad [Follower]" leader= + testing.go:149: server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad: 2025-02-10T11:34:18.782Z [INFO] server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad: entering follower state: follower="Node at e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad [Follower]" leader= +testLoggerAdapter verbose: server-fab3e159-c600-8d9b-2dae-4217b3a13060: 2025-02-10T11:34:18.782Z [INFO] server-fab3e159-c600-8d9b-2dae-4217b3a13060: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad Address:e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad} {Suffrage:Voter ID:server-fab3e159-c600-8d9b-2dae-4217b3a13060 Address:fab3e159-c600-8d9b-2dae-4217b3a13060} {Suffrage:Voter ID:server-651b1828-d980-261c-fd3f-9fe7a58fc860 Address:651b1828-d980-261c-fd3f-9fe7a58fc860}]" + testing.go:149: server-fab3e159-c600-8d9b-2dae-4217b3a13060: 2025-02-10T11:34:18.782Z [INFO] server-fab3e159-c600-8d9b-2dae-4217b3a13060: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad Address:e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad} {Suffrage:Voter ID:server-fab3e159-c600-8d9b-2dae-4217b3a13060 Address:fab3e159-c600-8d9b-2dae-4217b3a13060} {Suffrage:Voter ID:server-651b1828-d980-261c-fd3f-9fe7a58fc860 Address:651b1828-d980-261c-fd3f-9fe7a58fc860}]" +testLoggerAdapter verbose: server-fab3e159-c600-8d9b-2dae-4217b3a13060: 2025-02-10T11:34:18.782Z [INFO] server-fab3e159-c600-8d9b-2dae-4217b3a13060: entering follower state: follower="Node at fab3e159-c600-8d9b-2dae-4217b3a13060 [Follower]" leader= + testing.go:149: server-fab3e159-c600-8d9b-2dae-4217b3a13060: 2025-02-10T11:34:18.782Z [INFO] server-fab3e159-c600-8d9b-2dae-4217b3a13060: entering follower state: follower="Node at fab3e159-c600-8d9b-2dae-4217b3a13060 [Follower]" leader= +testLoggerAdapter verbose: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.782Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad Address:e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad} {Suffrage:Voter ID:server-fab3e159-c600-8d9b-2dae-4217b3a13060 Address:fab3e159-c600-8d9b-2dae-4217b3a13060} {Suffrage:Voter ID:server-651b1828-d980-261c-fd3f-9fe7a58fc860 Address:651b1828-d980-261c-fd3f-9fe7a58fc860}]" + testing.go:149: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.782Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad Address:e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad} {Suffrage:Voter ID:server-fab3e159-c600-8d9b-2dae-4217b3a13060 Address:fab3e159-c600-8d9b-2dae-4217b3a13060} {Suffrage:Voter ID:server-651b1828-d980-261c-fd3f-9fe7a58fc860 Address:651b1828-d980-261c-fd3f-9fe7a58fc860}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:18.782Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:18.782Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.782Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: entering follower state: follower="Node at 651b1828-d980-261c-fd3f-9fe7a58fc860 [Follower]" leader= + testing.go:149: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.782Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: entering follower state: follower="Node at 651b1828-d980-261c-fd3f-9fe7a58fc860 [Follower]" leader= +testLoggerAdapter verbose: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.862Z [WARN] server-651b1828-d980-261c-fd3f-9fe7a58fc860: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.862Z [WARN] server-651b1828-d980-261c-fd3f-9fe7a58fc860: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.862Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: entering candidate state: node="Node at 651b1828-d980-261c-fd3f-9fe7a58fc860 [Candidate]" term=2 + testing.go:149: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.862Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: entering candidate state: node="Node at 651b1828-d980-261c-fd3f-9fe7a58fc860 [Candidate]" term=2 +testLoggerAdapter verbose: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.862Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: election won: tally=2 + testing.go:149: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.862Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: election won: tally=2 +testLoggerAdapter verbose: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.862Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: entering leader state: leader="Node at 651b1828-d980-261c-fd3f-9fe7a58fc860 [Leader]" + testing.go:149: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.862Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: entering leader state: leader="Node at 651b1828-d980-261c-fd3f-9fe7a58fc860 [Leader]" +testLoggerAdapter verbose: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.862Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: added peer, starting replication: peer=server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad + testing.go:149: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.862Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: added peer, starting replication: peer=server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad +testLoggerAdapter verbose: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.862Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: added peer, starting replication: peer=server-fab3e159-c600-8d9b-2dae-4217b3a13060 + testing.go:149: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.862Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: added peer, starting replication: peer=server-fab3e159-c600-8d9b-2dae-4217b3a13060 +testLoggerAdapter verbose: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.862Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: pipelining replication: peer="{Voter server-fab3e159-c600-8d9b-2dae-4217b3a13060 fab3e159-c600-8d9b-2dae-4217b3a13060}" + testing.go:149: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.862Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: pipelining replication: peer="{Voter server-fab3e159-c600-8d9b-2dae-4217b3a13060 fab3e159-c600-8d9b-2dae-4217b3a13060}" +testLoggerAdapter verbose: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.862Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: pipelining replication: peer="{Voter server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad}" + testing.go:149: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:18.862Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: pipelining replication: peer="{Voter server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:18.967Z [INFO] stable state for Follower reached at 2025-02-10 11:34:18.862506272 +0000 UTC m=+15.231410573 (2 nodes), 80.062498ms from start of poll, 80.460056ms from cluster start. Timeout at 2025-02-10 11:34:18.967629312 +0000 UTC m=+15.336533668, 105.123095ms after stability + testing.go:149: cluster: 2025-02-10T11:34:18.967Z [INFO] stable state for Follower reached at 2025-02-10 11:34:18.862506272 +0000 UTC m=+15.231410573 (2 nodes), 80.062498ms from start of poll, 80.460056ms from cluster start. Timeout at 2025-02-10 11:34:18.967629312 +0000 UTC m=+15.336533668, 105.123095ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:18.967Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:18.967Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:19.072Z [INFO] stable state for Leader reached at 2025-02-10 11:34:18.967715384 +0000 UTC m=+15.336619689 (1 nodes), 480ns from start of poll, 185.669172ms from cluster start. Timeout at 2025-02-10 11:34:19.072818767 +0000 UTC m=+15.441723116, 105.103427ms after stability + testing.go:149: cluster: 2025-02-10T11:34:19.072Z [INFO] stable state for Leader reached at 2025-02-10 11:34:18.967715384 +0000 UTC m=+15.336619689 (1 nodes), 480ns from start of poll, 185.669172ms from cluster start. Timeout at 2025-02-10 11:34:19.072818767 +0000 UTC m=+15.441723116, 105.103427ms after stability +testLoggerAdapter verbose: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:19.072Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: aborting pipeline replication: peer="{Voter server-fab3e159-c600-8d9b-2dae-4217b3a13060 fab3e159-c600-8d9b-2dae-4217b3a13060}" + testing.go:149: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:19.072Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: aborting pipeline replication: peer="{Voter server-fab3e159-c600-8d9b-2dae-4217b3a13060 fab3e159-c600-8d9b-2dae-4217b3a13060}" +testLoggerAdapter verbose: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:19.073Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: aborting pipeline replication: peer="{Voter server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad}" + testing.go:149: server-651b1828-d980-261c-fd3f-9fe7a58fc860: 2025-02-10T11:34:19.073Z [INFO] server-651b1828-d980-261c-fd3f-9fe7a58fc860: aborting pipeline replication: peer="{Voter server-e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad e0367ca5-f0bf-cf68-dd2b-aca7e4a8b8ad}" +--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.29s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 -testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.151Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: initial configuration: index=1 servers="[{Suffrage:Voter ID:a0ea6c91-e115-e43e-e35e-3e180635831d Address:a0ea6c91-e115-e43e-e35e-3e180635831d} {Suffrage:Voter ID:a02787f6-97d3-11ae-ae8a-eff88e75b250 Address:a02787f6-97d3-11ae-ae8a-eff88e75b250}]" - testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.151Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: initial configuration: index=1 servers="[{Suffrage:Voter ID:a0ea6c91-e115-e43e-e35e-3e180635831d Address:a0ea6c91-e115-e43e-e35e-3e180635831d} {Suffrage:Voter ID:a02787f6-97d3-11ae-ae8a-eff88e75b250 Address:a02787f6-97d3-11ae-ae8a-eff88e75b250}]" -testLoggerAdapter verbose: a02787f6-97d3-11ae-ae8a-eff88e75b250: 2024-01-09T05:10:05.151Z [INFO] a02787f6-97d3-11ae-ae8a-eff88e75b250: initial configuration: index=1 servers="[{Suffrage:Voter ID:a0ea6c91-e115-e43e-e35e-3e180635831d Address:a0ea6c91-e115-e43e-e35e-3e180635831d} {Suffrage:Voter ID:a02787f6-97d3-11ae-ae8a-eff88e75b250 Address:a02787f6-97d3-11ae-ae8a-eff88e75b250}]" - testing.go:149: a02787f6-97d3-11ae-ae8a-eff88e75b250: 2024-01-09T05:10:05.151Z [INFO] a02787f6-97d3-11ae-ae8a-eff88e75b250: initial configuration: index=1 servers="[{Suffrage:Voter ID:a0ea6c91-e115-e43e-e35e-3e180635831d Address:a0ea6c91-e115-e43e-e35e-3e180635831d} {Suffrage:Voter ID:a02787f6-97d3-11ae-ae8a-eff88e75b250 Address:a02787f6-97d3-11ae-ae8a-eff88e75b250}]" -testLoggerAdapter verbose: aef09d44-1ee3-903b-a640-190442a2e065: 2024-01-09T05:10:05.152Z [INFO] aef09d44-1ee3-903b-a640-190442a2e065: initial configuration: index=0 servers=[] - testing.go:149: aef09d44-1ee3-903b-a640-190442a2e065: 2024-01-09T05:10:05.152Z [INFO] aef09d44-1ee3-903b-a640-190442a2e065: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.152Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:05.152Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.152Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: entering follower state: follower="Node at a0ea6c91-e115-e43e-e35e-3e180635831d [Follower]" leader= - testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.152Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: entering follower state: follower="Node at a0ea6c91-e115-e43e-e35e-3e180635831d [Follower]" leader= -testLoggerAdapter verbose: a02787f6-97d3-11ae-ae8a-eff88e75b250: 2024-01-09T05:10:05.153Z [INFO] a02787f6-97d3-11ae-ae8a-eff88e75b250: entering follower state: follower="Node at a02787f6-97d3-11ae-ae8a-eff88e75b250 [Follower]" leader= - testing.go:149: a02787f6-97d3-11ae-ae8a-eff88e75b250: 2024-01-09T05:10:05.153Z [INFO] a02787f6-97d3-11ae-ae8a-eff88e75b250: entering follower state: follower="Node at a02787f6-97d3-11ae-ae8a-eff88e75b250 [Follower]" leader= -testLoggerAdapter verbose: aef09d44-1ee3-903b-a640-190442a2e065: 2024-01-09T05:10:05.153Z [INFO] aef09d44-1ee3-903b-a640-190442a2e065: entering follower state: follower="Node at aef09d44-1ee3-903b-a640-190442a2e065 [Follower]" leader= - testing.go:149: aef09d44-1ee3-903b-a640-190442a2e065: 2024-01-09T05:10:05.153Z [INFO] aef09d44-1ee3-903b-a640-190442a2e065: entering follower state: follower="Node at aef09d44-1ee3-903b-a640-190442a2e065 [Follower]" leader= -testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [WARN] a0ea6c91-e115-e43e-e35e-3e180635831d: heartbeat timeout reached, starting election: last-leader= - testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [WARN] a0ea6c91-e115-e43e-e35e-3e180635831d: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: entering candidate state: node="Node at a0ea6c91-e115-e43e-e35e-3e180635831d [Candidate]" term=2 - testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: entering candidate state: node="Node at a0ea6c91-e115-e43e-e35e-3e180635831d [Candidate]" term=2 -testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: election won: tally=2 - testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: election won: tally=2 -testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: entering leader state: leader="Node at a0ea6c91-e115-e43e-e35e-3e180635831d [Leader]" - testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: entering leader state: leader="Node at a0ea6c91-e115-e43e-e35e-3e180635831d [Leader]" -testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: added peer, starting replication: peer=a02787f6-97d3-11ae-ae8a-eff88e75b250 - testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: added peer, starting replication: peer=a02787f6-97d3-11ae-ae8a-eff88e75b250 -testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: pipelining replication: peer="{Voter a02787f6-97d3-11ae-ae8a-eff88e75b250 a02787f6-97d3-11ae-ae8a-eff88e75b250}" - testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: pipelining replication: peer="{Voter a02787f6-97d3-11ae-ae8a-eff88e75b250 a02787f6-97d3-11ae-ae8a-eff88e75b250}" -testLoggerAdapter verbose: aef09d44-1ee3-903b-a640-190442a2e065: 2024-01-09T05:10:05.253Z [WARN] aef09d44-1ee3-903b-a640-190442a2e065: no known peers, aborting election - testing.go:149: aef09d44-1ee3-903b-a640-190442a2e065: 2024-01-09T05:10:05.253Z [WARN] aef09d44-1ee3-903b-a640-190442a2e065: no known peers, aborting election -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.330Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.224727087 +0000 UTC m=+16.477778132 (1 nodes), 72.288579ms from start of poll, 73.157815ms from cluster start. Timeout at 2024-01-09 05:10:05.330416078 +0000 UTC m=+16.583467148, 105.689016ms after stability - testing.go:149: cluster: 2024-01-09T05:10:05.330Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.224727087 +0000 UTC m=+16.477778132 (1 nodes), 72.288579ms from start of poll, 73.157815ms from cluster start. Timeout at 2024-01-09 05:10:05.330416078 +0000 UTC m=+16.583467148, 105.689016ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.330Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:05.330Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.435Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.330613883 +0000 UTC m=+16.583664974 (1 nodes), 3.116µs from start of poll, 179.044657ms from cluster start. Timeout at 2024-01-09 05:10:05.435731646 +0000 UTC m=+16.688782753, 105.117779ms after stability - testing.go:149: cluster: 2024-01-09T05:10:05.435Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.330613883 +0000 UTC m=+16.583664974 (1 nodes), 3.116µs from start of poll, 179.044657ms from cluster start. Timeout at 2024-01-09 05:10:05.435731646 +0000 UTC m=+16.688782753, 105.117779ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.435Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:05.435Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.541Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.435978659 +0000 UTC m=+16.689029766 (1 nodes), 2.359µs from start of poll, 284.409449ms from cluster start. Timeout at 2024-01-09 05:10:05.541090086 +0000 UTC m=+16.794141161, 105.111395ms after stability - testing.go:149: cluster: 2024-01-09T05:10:05.541Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.435978659 +0000 UTC m=+16.689029766 (1 nodes), 2.359µs from start of poll, 284.409449ms from cluster start. Timeout at 2024-01-09 05:10:05.541090086 +0000 UTC m=+16.794141161, 105.111395ms after stability -testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.541Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: updating configuration: command=AddStaging server-id=aef09d44-1ee3-903b-a640-190442a2e065 server-addr=aef09d44-1ee3-903b-a640-190442a2e065 servers="[{Suffrage:Voter ID:a0ea6c91-e115-e43e-e35e-3e180635831d Address:a0ea6c91-e115-e43e-e35e-3e180635831d} {Suffrage:Voter ID:a02787f6-97d3-11ae-ae8a-eff88e75b250 Address:a02787f6-97d3-11ae-ae8a-eff88e75b250} {Suffrage:Voter ID:aef09d44-1ee3-903b-a640-190442a2e065 Address:aef09d44-1ee3-903b-a640-190442a2e065}]" - testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.541Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: updating configuration: command=AddStaging server-id=aef09d44-1ee3-903b-a640-190442a2e065 server-addr=aef09d44-1ee3-903b-a640-190442a2e065 servers="[{Suffrage:Voter ID:a0ea6c91-e115-e43e-e35e-3e180635831d Address:a0ea6c91-e115-e43e-e35e-3e180635831d} {Suffrage:Voter ID:a02787f6-97d3-11ae-ae8a-eff88e75b250 Address:a02787f6-97d3-11ae-ae8a-eff88e75b250} {Suffrage:Voter ID:aef09d44-1ee3-903b-a640-190442a2e065 Address:aef09d44-1ee3-903b-a640-190442a2e065}]" -testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.541Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: added peer, starting replication: peer=aef09d44-1ee3-903b-a640-190442a2e065 - testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.541Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: added peer, starting replication: peer=aef09d44-1ee3-903b-a640-190442a2e065 -testLoggerAdapter verbose: aef09d44-1ee3-903b-a640-190442a2e065: 2024-01-09T05:10:05.541Z [WARN] aef09d44-1ee3-903b-a640-190442a2e065: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: aef09d44-1ee3-903b-a640-190442a2e065: 2024-01-09T05:10:05.541Z [WARN] aef09d44-1ee3-903b-a640-190442a2e065: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.541Z [WARN] a0ea6c91-e115-e43e-e35e-3e180635831d: appendEntries rejected, sending older logs: peer="{Voter aef09d44-1ee3-903b-a640-190442a2e065 aef09d44-1ee3-903b-a640-190442a2e065}" next=1 - testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.541Z [WARN] a0ea6c91-e115-e43e-e35e-3e180635831d: appendEntries rejected, sending older logs: peer="{Voter aef09d44-1ee3-903b-a640-190442a2e065 aef09d44-1ee3-903b-a640-190442a2e065}" next=1 -testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.541Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: pipelining replication: peer="{Voter aef09d44-1ee3-903b-a640-190442a2e065 aef09d44-1ee3-903b-a640-190442a2e065}" - testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.541Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: pipelining replication: peer="{Voter aef09d44-1ee3-903b-a640-190442a2e065 aef09d44-1ee3-903b-a640-190442a2e065}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.546Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:05.546Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.652Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.54707786 +0000 UTC m=+16.800128902 (1 nodes), 2.54µs from start of poll, 395.508585ms from cluster start. Timeout at 2024-01-09 05:10:05.652168671 +0000 UTC m=+16.905219675, 105.090773ms after stability - testing.go:149: cluster: 2024-01-09T05:10:05.652Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.54707786 +0000 UTC m=+16.800128902 (1 nodes), 2.54µs from start of poll, 395.508585ms from cluster start. Timeout at 2024-01-09 05:10:05.652168671 +0000 UTC m=+16.905219675, 105.090773ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.652Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:05.652Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.757Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.652355379 +0000 UTC m=+16.905406372 (1 nodes), 2.051µs from start of poll, 500.786055ms from cluster start. Timeout at 2024-01-09 05:10:05.757380951 +0000 UTC m=+17.010432061, 105.025689ms after stability - testing.go:149: cluster: 2024-01-09T05:10:05.757Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.652355379 +0000 UTC m=+16.905406372 (1 nodes), 2.051µs from start of poll, 500.786055ms from cluster start. Timeout at 2024-01-09 05:10:05.757380951 +0000 UTC m=+17.010432061, 105.025689ms after stability -testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.757Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: updating configuration: command=RemoveServer server-id=aef09d44-1ee3-903b-a640-190442a2e065 server-addr= servers="[{Suffrage:Voter ID:a0ea6c91-e115-e43e-e35e-3e180635831d Address:a0ea6c91-e115-e43e-e35e-3e180635831d} {Suffrage:Voter ID:a02787f6-97d3-11ae-ae8a-eff88e75b250 Address:a02787f6-97d3-11ae-ae8a-eff88e75b250}]" - testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.757Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: updating configuration: command=RemoveServer server-id=aef09d44-1ee3-903b-a640-190442a2e065 server-addr= servers="[{Suffrage:Voter ID:a0ea6c91-e115-e43e-e35e-3e180635831d Address:a0ea6c91-e115-e43e-e35e-3e180635831d} {Suffrage:Voter ID:a02787f6-97d3-11ae-ae8a-eff88e75b250 Address:a02787f6-97d3-11ae-ae8a-eff88e75b250}]" -testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.757Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: removed peer, stopping replication: peer=aef09d44-1ee3-903b-a640-190442a2e065 last-index=4 - testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.757Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: removed peer, stopping replication: peer=aef09d44-1ee3-903b-a640-190442a2e065 last-index=4 -testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.757Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: aborting pipeline replication: peer="{Voter aef09d44-1ee3-903b-a640-190442a2e065 aef09d44-1ee3-903b-a640-190442a2e065}" - testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.757Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: aborting pipeline replication: peer="{Voter aef09d44-1ee3-903b-a640-190442a2e065 aef09d44-1ee3-903b-a640-190442a2e065}" -testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.757Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: aborting pipeline replication: peer="{Voter a02787f6-97d3-11ae-ae8a-eff88e75b250 a02787f6-97d3-11ae-ae8a-eff88e75b250}" - testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.757Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: aborting pipeline replication: peer="{Voter a02787f6-97d3-11ae-ae8a-eff88e75b250 a02787f6-97d3-11ae-ae8a-eff88e75b250}" +testLoggerAdapter verbose: 820096a2-2c4d-d93e-d653-e1acec3ecd72: 2025-02-10T11:34:19.073Z [INFO] 820096a2-2c4d-d93e-d653-e1acec3ecd72: initial configuration: index=1 servers="[{Suffrage:Voter ID:820096a2-2c4d-d93e-d653-e1acec3ecd72 Address:820096a2-2c4d-d93e-d653-e1acec3ecd72} {Suffrage:Voter ID:879bd73a-7e9c-c4b9-83bf-69016e3a1b6e Address:879bd73a-7e9c-c4b9-83bf-69016e3a1b6e}]" + testing.go:149: 820096a2-2c4d-d93e-d653-e1acec3ecd72: 2025-02-10T11:34:19.073Z [INFO] 820096a2-2c4d-d93e-d653-e1acec3ecd72: initial configuration: index=1 servers="[{Suffrage:Voter ID:820096a2-2c4d-d93e-d653-e1acec3ecd72 Address:820096a2-2c4d-d93e-d653-e1acec3ecd72} {Suffrage:Voter ID:879bd73a-7e9c-c4b9-83bf-69016e3a1b6e Address:879bd73a-7e9c-c4b9-83bf-69016e3a1b6e}]" +testLoggerAdapter verbose: 820096a2-2c4d-d93e-d653-e1acec3ecd72: 2025-02-10T11:34:19.074Z [INFO] 820096a2-2c4d-d93e-d653-e1acec3ecd72: entering follower state: follower="Node at 820096a2-2c4d-d93e-d653-e1acec3ecd72 [Follower]" leader= + testing.go:149: 820096a2-2c4d-d93e-d653-e1acec3ecd72: 2025-02-10T11:34:19.074Z [INFO] 820096a2-2c4d-d93e-d653-e1acec3ecd72: entering follower state: follower="Node at 820096a2-2c4d-d93e-d653-e1acec3ecd72 [Follower]" leader= +testLoggerAdapter verbose: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.074Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:820096a2-2c4d-d93e-d653-e1acec3ecd72 Address:820096a2-2c4d-d93e-d653-e1acec3ecd72} {Suffrage:Voter ID:879bd73a-7e9c-c4b9-83bf-69016e3a1b6e Address:879bd73a-7e9c-c4b9-83bf-69016e3a1b6e}]" + testing.go:149: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.074Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:820096a2-2c4d-d93e-d653-e1acec3ecd72 Address:820096a2-2c4d-d93e-d653-e1acec3ecd72} {Suffrage:Voter ID:879bd73a-7e9c-c4b9-83bf-69016e3a1b6e Address:879bd73a-7e9c-c4b9-83bf-69016e3a1b6e}]" +testLoggerAdapter verbose: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.074Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: entering follower state: follower="Node at 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e [Follower]" leader= + testing.go:149: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.074Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: entering follower state: follower="Node at 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e [Follower]" leader= +testLoggerAdapter verbose: 57860ed7-e494-1908-3873-847abb3203bb: 2025-02-10T11:34:19.074Z [INFO] 57860ed7-e494-1908-3873-847abb3203bb: initial configuration: index=0 servers=[] + testing.go:149: 57860ed7-e494-1908-3873-847abb3203bb: 2025-02-10T11:34:19.074Z [INFO] 57860ed7-e494-1908-3873-847abb3203bb: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:19.074Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:19.074Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: 57860ed7-e494-1908-3873-847abb3203bb: 2025-02-10T11:34:19.074Z [INFO] 57860ed7-e494-1908-3873-847abb3203bb: entering follower state: follower="Node at 57860ed7-e494-1908-3873-847abb3203bb [Follower]" leader= + testing.go:149: 57860ed7-e494-1908-3873-847abb3203bb: 2025-02-10T11:34:19.074Z [INFO] 57860ed7-e494-1908-3873-847abb3203bb: entering follower state: follower="Node at 57860ed7-e494-1908-3873-847abb3203bb [Follower]" leader= +testLoggerAdapter verbose: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.159Z [WARN] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: heartbeat timeout reached, starting election: last-leader= + testing.go:149: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.159Z [WARN] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.159Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: entering candidate state: node="Node at 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e [Candidate]" term=2 + testing.go:149: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.159Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: entering candidate state: node="Node at 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e [Candidate]" term=2 +testLoggerAdapter verbose: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.159Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: election won: tally=2 + testing.go:149: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.159Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: election won: tally=2 +testLoggerAdapter verbose: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.159Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: entering leader state: leader="Node at 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e [Leader]" + testing.go:149: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.159Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: entering leader state: leader="Node at 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e [Leader]" +testLoggerAdapter verbose: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.159Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: added peer, starting replication: peer=820096a2-2c4d-d93e-d653-e1acec3ecd72 + testing.go:149: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.159Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: added peer, starting replication: peer=820096a2-2c4d-d93e-d653-e1acec3ecd72 +testLoggerAdapter verbose: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.159Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: pipelining replication: peer="{Voter 820096a2-2c4d-d93e-d653-e1acec3ecd72 820096a2-2c4d-d93e-d653-e1acec3ecd72}" + testing.go:149: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.159Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: pipelining replication: peer="{Voter 820096a2-2c4d-d93e-d653-e1acec3ecd72 820096a2-2c4d-d93e-d653-e1acec3ecd72}" +testLoggerAdapter verbose: 57860ed7-e494-1908-3873-847abb3203bb: 2025-02-10T11:34:19.169Z [WARN] 57860ed7-e494-1908-3873-847abb3203bb: no known peers, aborting election + testing.go:149: 57860ed7-e494-1908-3873-847abb3203bb: 2025-02-10T11:34:19.169Z [WARN] 57860ed7-e494-1908-3873-847abb3203bb: no known peers, aborting election +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:19.264Z [INFO] stable state for Leader reached at 2025-02-10 11:34:19.159122873 +0000 UTC m=+15.528027175 (1 nodes), 84.800972ms from start of poll, 85.273288ms from cluster start. Timeout at 2025-02-10 11:34:19.264220798 +0000 UTC m=+15.633125155, 105.09798ms after stability + testing.go:149: cluster: 2025-02-10T11:34:19.264Z [INFO] stable state for Leader reached at 2025-02-10 11:34:19.159122873 +0000 UTC m=+15.528027175 (1 nodes), 84.800972ms from start of poll, 85.273288ms from cluster start. Timeout at 2025-02-10 11:34:19.264220798 +0000 UTC m=+15.633125155, 105.09798ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:19.264Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:19.264Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:19.369Z [INFO] stable state for Leader reached at 2025-02-10 11:34:19.264376859 +0000 UTC m=+15.633281222 (1 nodes), 912ns from start of poll, 190.527335ms from cluster start. Timeout at 2025-02-10 11:34:19.369451587 +0000 UTC m=+15.738355885, 105.074663ms after stability + testing.go:149: cluster: 2025-02-10T11:34:19.369Z [INFO] stable state for Leader reached at 2025-02-10 11:34:19.264376859 +0000 UTC m=+15.633281222 (1 nodes), 912ns from start of poll, 190.527335ms from cluster start. Timeout at 2025-02-10 11:34:19.369451587 +0000 UTC m=+15.738355885, 105.074663ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:19.369Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:19.369Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:19.474Z [INFO] stable state for Leader reached at 2025-02-10 11:34:19.369525422 +0000 UTC m=+15.738429721 (1 nodes), 511ns from start of poll, 295.675834ms from cluster start. Timeout at 2025-02-10 11:34:19.474598367 +0000 UTC m=+15.843502661, 105.07294ms after stability + testing.go:149: cluster: 2025-02-10T11:34:19.474Z [INFO] stable state for Leader reached at 2025-02-10 11:34:19.369525422 +0000 UTC m=+15.738429721 (1 nodes), 511ns from start of poll, 295.675834ms from cluster start. Timeout at 2025-02-10 11:34:19.474598367 +0000 UTC m=+15.843502661, 105.07294ms after stability +testLoggerAdapter verbose: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.474Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: updating configuration: command=AddStaging server-id=57860ed7-e494-1908-3873-847abb3203bb server-addr=57860ed7-e494-1908-3873-847abb3203bb servers="[{Suffrage:Voter ID:820096a2-2c4d-d93e-d653-e1acec3ecd72 Address:820096a2-2c4d-d93e-d653-e1acec3ecd72} {Suffrage:Voter ID:879bd73a-7e9c-c4b9-83bf-69016e3a1b6e Address:879bd73a-7e9c-c4b9-83bf-69016e3a1b6e} {Suffrage:Voter ID:57860ed7-e494-1908-3873-847abb3203bb Address:57860ed7-e494-1908-3873-847abb3203bb}]" + testing.go:149: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.474Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: updating configuration: command=AddStaging server-id=57860ed7-e494-1908-3873-847abb3203bb server-addr=57860ed7-e494-1908-3873-847abb3203bb servers="[{Suffrage:Voter ID:820096a2-2c4d-d93e-d653-e1acec3ecd72 Address:820096a2-2c4d-d93e-d653-e1acec3ecd72} {Suffrage:Voter ID:879bd73a-7e9c-c4b9-83bf-69016e3a1b6e Address:879bd73a-7e9c-c4b9-83bf-69016e3a1b6e} {Suffrage:Voter ID:57860ed7-e494-1908-3873-847abb3203bb Address:57860ed7-e494-1908-3873-847abb3203bb}]" +testLoggerAdapter verbose: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.474Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: added peer, starting replication: peer=57860ed7-e494-1908-3873-847abb3203bb + testing.go:149: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.474Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: added peer, starting replication: peer=57860ed7-e494-1908-3873-847abb3203bb +testLoggerAdapter verbose: 57860ed7-e494-1908-3873-847abb3203bb: 2025-02-10T11:34:19.474Z [WARN] 57860ed7-e494-1908-3873-847abb3203bb: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: 57860ed7-e494-1908-3873-847abb3203bb: 2025-02-10T11:34:19.474Z [WARN] 57860ed7-e494-1908-3873-847abb3203bb: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.474Z [WARN] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: appendEntries rejected, sending older logs: peer="{Voter 57860ed7-e494-1908-3873-847abb3203bb 57860ed7-e494-1908-3873-847abb3203bb}" next=1 + testing.go:149: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.474Z [WARN] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: appendEntries rejected, sending older logs: peer="{Voter 57860ed7-e494-1908-3873-847abb3203bb 57860ed7-e494-1908-3873-847abb3203bb}" next=1 +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:19.474Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:19.474Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.474Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: pipelining replication: peer="{Voter 57860ed7-e494-1908-3873-847abb3203bb 57860ed7-e494-1908-3873-847abb3203bb}" + testing.go:149: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.474Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: pipelining replication: peer="{Voter 57860ed7-e494-1908-3873-847abb3203bb 57860ed7-e494-1908-3873-847abb3203bb}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:19.580Z [INFO] stable state for Leader reached at 2025-02-10 11:34:19.474974456 +0000 UTC m=+15.843878764 (1 nodes), 728ns from start of poll, 401.124877ms from cluster start. Timeout at 2025-02-10 11:34:19.580067964 +0000 UTC m=+15.948972317, 105.093553ms after stability + testing.go:149: cluster: 2025-02-10T11:34:19.580Z [INFO] stable state for Leader reached at 2025-02-10 11:34:19.474974456 +0000 UTC m=+15.843878764 (1 nodes), 728ns from start of poll, 401.124877ms from cluster start. Timeout at 2025-02-10 11:34:19.580067964 +0000 UTC m=+15.948972317, 105.093553ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:19.580Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:19.580Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:19.685Z [INFO] stable state for Leader reached at 2025-02-10 11:34:19.580208241 +0000 UTC m=+15.949112611 (1 nodes), 832ns from start of poll, 506.358724ms from cluster start. Timeout at 2025-02-10 11:34:19.685276684 +0000 UTC m=+16.054180981, 105.06837ms after stability + testing.go:149: cluster: 2025-02-10T11:34:19.685Z [INFO] stable state for Leader reached at 2025-02-10 11:34:19.580208241 +0000 UTC m=+15.949112611 (1 nodes), 832ns from start of poll, 506.358724ms from cluster start. Timeout at 2025-02-10 11:34:19.685276684 +0000 UTC m=+16.054180981, 105.06837ms after stability +testLoggerAdapter verbose: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.685Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: updating configuration: command=RemoveServer server-id=57860ed7-e494-1908-3873-847abb3203bb server-addr= servers="[{Suffrage:Voter ID:820096a2-2c4d-d93e-d653-e1acec3ecd72 Address:820096a2-2c4d-d93e-d653-e1acec3ecd72} {Suffrage:Voter ID:879bd73a-7e9c-c4b9-83bf-69016e3a1b6e Address:879bd73a-7e9c-c4b9-83bf-69016e3a1b6e}]" + testing.go:149: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.685Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: updating configuration: command=RemoveServer server-id=57860ed7-e494-1908-3873-847abb3203bb server-addr= servers="[{Suffrage:Voter ID:820096a2-2c4d-d93e-d653-e1acec3ecd72 Address:820096a2-2c4d-d93e-d653-e1acec3ecd72} {Suffrage:Voter ID:879bd73a-7e9c-c4b9-83bf-69016e3a1b6e Address:879bd73a-7e9c-c4b9-83bf-69016e3a1b6e}]" +testLoggerAdapter verbose: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.685Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: removed peer, stopping replication: peer=57860ed7-e494-1908-3873-847abb3203bb last-index=4 + testing.go:149: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.685Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: removed peer, stopping replication: peer=57860ed7-e494-1908-3873-847abb3203bb last-index=4 +testLoggerAdapter verbose: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.685Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: aborting pipeline replication: peer="{Voter 57860ed7-e494-1908-3873-847abb3203bb 57860ed7-e494-1908-3873-847abb3203bb}" + testing.go:149: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.685Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: aborting pipeline replication: peer="{Voter 57860ed7-e494-1908-3873-847abb3203bb 57860ed7-e494-1908-3873-847abb3203bb}" +testLoggerAdapter verbose: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.685Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: aborting pipeline replication: peer="{Voter 820096a2-2c4d-d93e-d653-e1acec3ecd72 820096a2-2c4d-d93e-d653-e1acec3ecd72}" + testing.go:149: 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: 2025-02-10T11:34:19.685Z [INFO] 879bd73a-7e9c-c4b9-83bf-69016e3a1b6e: aborting pipeline replication: peer="{Voter 820096a2-2c4d-d93e-d653-e1acec3ecd72 820096a2-2c4d-d93e-d653-e1acec3ecd72}" --- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.61s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 -testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.759Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc Address:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc} {Suffrage:Voter ID:a0acff85-887c-9857-f396-95021c279817 Address:a0acff85-887c-9857-f396-95021c279817}]" - testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.759Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc Address:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc} {Suffrage:Voter ID:a0acff85-887c-9857-f396-95021c279817 Address:a0acff85-887c-9857-f396-95021c279817}]" -testLoggerAdapter verbose: a0acff85-887c-9857-f396-95021c279817: 2024-01-09T05:10:05.759Z [INFO] a0acff85-887c-9857-f396-95021c279817: initial configuration: index=1 servers="[{Suffrage:Voter ID:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc Address:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc} {Suffrage:Voter ID:a0acff85-887c-9857-f396-95021c279817 Address:a0acff85-887c-9857-f396-95021c279817}]" - testing.go:149: a0acff85-887c-9857-f396-95021c279817: 2024-01-09T05:10:05.759Z [INFO] a0acff85-887c-9857-f396-95021c279817: initial configuration: index=1 servers="[{Suffrage:Voter ID:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc Address:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc} {Suffrage:Voter ID:a0acff85-887c-9857-f396-95021c279817 Address:a0acff85-887c-9857-f396-95021c279817}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.759Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:10:05.759Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.759Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: entering follower state: follower="Node at 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc [Follower]" leader= - testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.759Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: entering follower state: follower="Node at 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc [Follower]" leader= -testLoggerAdapter verbose: a0acff85-887c-9857-f396-95021c279817: 2024-01-09T05:10:05.759Z [INFO] a0acff85-887c-9857-f396-95021c279817: entering follower state: follower="Node at a0acff85-887c-9857-f396-95021c279817 [Follower]" leader= - testing.go:149: a0acff85-887c-9857-f396-95021c279817: 2024-01-09T05:10:05.759Z [INFO] a0acff85-887c-9857-f396-95021c279817: entering follower state: follower="Node at a0acff85-887c-9857-f396-95021c279817 [Follower]" leader= -testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [WARN] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: heartbeat timeout reached, starting election: last-leader= - testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [WARN] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: entering candidate state: node="Node at 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc [Candidate]" term=2 - testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: entering candidate state: node="Node at 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc [Candidate]" term=2 -testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: election won: tally=2 - testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: election won: tally=2 -testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: entering leader state: leader="Node at 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc [Leader]" - testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: entering leader state: leader="Node at 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc [Leader]" -testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: added peer, starting replication: peer=a0acff85-887c-9857-f396-95021c279817 - testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: added peer, starting replication: peer=a0acff85-887c-9857-f396-95021c279817 -testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: pipelining replication: peer="{Voter a0acff85-887c-9857-f396-95021c279817 a0acff85-887c-9857-f396-95021c279817}" - testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: pipelining replication: peer="{Voter a0acff85-887c-9857-f396-95021c279817 a0acff85-887c-9857-f396-95021c279817}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.919Z [INFO] stable state for Follower reached at 2024-01-09 05:10:05.814747193 +0000 UTC m=+17.067798297 (1 nodes), 55.273817ms from start of poll, 55.619466ms from cluster start. Timeout at 2024-01-09 05:10:05.919921604 +0000 UTC m=+17.172972731, 105.174434ms after stability - testing.go:149: cluster: 2024-01-09T05:10:05.919Z [INFO] stable state for Follower reached at 2024-01-09 05:10:05.814747193 +0000 UTC m=+17.067798297 (1 nodes), 55.273817ms from start of poll, 55.619466ms from cluster start. Timeout at 2024-01-09 05:10:05.919921604 +0000 UTC m=+17.172972731, 105.174434ms after stability -testLoggerAdapter verbose: server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: 2024-01-09T05:10:05.920Z [INFO] server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: initial configuration: index=0 servers=[] - testing.go:149: server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: 2024-01-09T05:10:05.920Z [INFO] server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.920Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:05.920Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: 2024-01-09T05:10:05.920Z [INFO] server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: entering follower state: follower="Node at 19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 [Follower]" leader= - testing.go:149: server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: 2024-01-09T05:10:05.920Z [INFO] server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: entering follower state: follower="Node at 19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 [Follower]" leader= -testLoggerAdapter verbose: server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: 2024-01-09T05:10:05.997Z [WARN] server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: no known peers, aborting election - testing.go:149: server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: 2024-01-09T05:10:05.997Z [WARN] server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: no known peers, aborting election -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.025Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.920501325 +0000 UTC m=+17.173552337 (1 nodes), 2.272µs from start of poll, 161.373506ms from cluster start. Timeout at 2024-01-09 05:10:06.025637916 +0000 UTC m=+17.278689066, 105.136729ms after stability - testing.go:149: cluster: 2024-01-09T05:10:06.025Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.920501325 +0000 UTC m=+17.173552337 (1 nodes), 2.272µs from start of poll, 161.373506ms from cluster start. Timeout at 2024-01-09 05:10:06.025637916 +0000 UTC m=+17.278689066, 105.136729ms after stability -testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.025Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: updating configuration: command=AddStaging server-id=server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 server-addr=19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 servers="[{Suffrage:Voter ID:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc Address:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc} {Suffrage:Voter ID:a0acff85-887c-9857-f396-95021c279817 Address:a0acff85-887c-9857-f396-95021c279817} {Suffrage:Voter ID:server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 Address:19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}]" - testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.025Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: updating configuration: command=AddStaging server-id=server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 server-addr=19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 servers="[{Suffrage:Voter ID:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc Address:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc} {Suffrage:Voter ID:a0acff85-887c-9857-f396-95021c279817 Address:a0acff85-887c-9857-f396-95021c279817} {Suffrage:Voter ID:server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 Address:19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}]" -testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.025Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: added peer, starting replication: peer=server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 - testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.025Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: added peer, starting replication: peer=server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 -testLoggerAdapter verbose: server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: 2024-01-09T05:10:06.026Z [WARN] server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: 2024-01-09T05:10:06.026Z [WARN] server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.026Z [WARN] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: appendEntries rejected, sending older logs: peer="{Voter server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}" next=1 - testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.026Z [WARN] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: appendEntries rejected, sending older logs: peer="{Voter server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}" next=1 -testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.026Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: pipelining replication: peer="{Voter server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}" - testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.026Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: pipelining replication: peer="{Voter server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.033Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:06.033Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.139Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.033852415 +0000 UTC m=+17.286903434 (1 nodes), 2.076µs from start of poll, 274.724603ms from cluster start. Timeout at 2024-01-09 05:10:06.138928335 +0000 UTC m=+17.391979595, 105.076161ms after stability - testing.go:149: cluster: 2024-01-09T05:10:06.139Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.033852415 +0000 UTC m=+17.286903434 (1 nodes), 2.076µs from start of poll, 274.724603ms from cluster start. Timeout at 2024-01-09 05:10:06.138928335 +0000 UTC m=+17.391979595, 105.076161ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.139Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:06.139Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.248Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.139128975 +0000 UTC m=+17.392180133 (1 nodes), 2.436µs from start of poll, 380.001302ms from cluster start. Timeout at 2024-01-09 05:10:06.248379404 +0000 UTC m=+17.501430492, 109.250359ms after stability - testing.go:149: cluster: 2024-01-09T05:10:06.248Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.139128975 +0000 UTC m=+17.392180133 (1 nodes), 2.436µs from start of poll, 380.001302ms from cluster start. Timeout at 2024-01-09 05:10:06.248379404 +0000 UTC m=+17.501430492, 109.250359ms after stability -testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.248Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: updating configuration: command=RemoveServer server-id=a0acff85-887c-9857-f396-95021c279817 server-addr= servers="[{Suffrage:Voter ID:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc Address:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc} {Suffrage:Voter ID:server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 Address:19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}]" - testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.248Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: updating configuration: command=RemoveServer server-id=a0acff85-887c-9857-f396-95021c279817 server-addr= servers="[{Suffrage:Voter ID:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc Address:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc} {Suffrage:Voter ID:server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 Address:19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}]" -testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.248Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: removed peer, stopping replication: peer=a0acff85-887c-9857-f396-95021c279817 last-index=4 - testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.248Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: removed peer, stopping replication: peer=a0acff85-887c-9857-f396-95021c279817 last-index=4 -testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.248Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: aborting pipeline replication: peer="{Voter a0acff85-887c-9857-f396-95021c279817 a0acff85-887c-9857-f396-95021c279817}" - testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.248Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: aborting pipeline replication: peer="{Voter a0acff85-887c-9857-f396-95021c279817 a0acff85-887c-9857-f396-95021c279817}" -testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.248Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: aborting pipeline replication: peer="{Voter server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}" - testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.248Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: aborting pipeline replication: peer="{Voter server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}" ---- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.49s) +testLoggerAdapter verbose: 2ff23827-10c6-84b8-e7a4-acd51c6d28fc: 2025-02-10T11:34:19.687Z [INFO] 2ff23827-10c6-84b8-e7a4-acd51c6d28fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:2ff23827-10c6-84b8-e7a4-acd51c6d28fc Address:2ff23827-10c6-84b8-e7a4-acd51c6d28fc} {Suffrage:Voter ID:e21603ea-911a-e69c-4c1b-44b4ff19324a Address:e21603ea-911a-e69c-4c1b-44b4ff19324a}]" + testing.go:149: 2ff23827-10c6-84b8-e7a4-acd51c6d28fc: 2025-02-10T11:34:19.687Z [INFO] 2ff23827-10c6-84b8-e7a4-acd51c6d28fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:2ff23827-10c6-84b8-e7a4-acd51c6d28fc Address:2ff23827-10c6-84b8-e7a4-acd51c6d28fc} {Suffrage:Voter ID:e21603ea-911a-e69c-4c1b-44b4ff19324a Address:e21603ea-911a-e69c-4c1b-44b4ff19324a}]" +testLoggerAdapter verbose: 2ff23827-10c6-84b8-e7a4-acd51c6d28fc: 2025-02-10T11:34:19.687Z [INFO] 2ff23827-10c6-84b8-e7a4-acd51c6d28fc: entering follower state: follower="Node at 2ff23827-10c6-84b8-e7a4-acd51c6d28fc [Follower]" leader= + testing.go:149: 2ff23827-10c6-84b8-e7a4-acd51c6d28fc: 2025-02-10T11:34:19.687Z [INFO] 2ff23827-10c6-84b8-e7a4-acd51c6d28fc: entering follower state: follower="Node at 2ff23827-10c6-84b8-e7a4-acd51c6d28fc [Follower]" leader= +testLoggerAdapter verbose: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.687Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: initial configuration: index=1 servers="[{Suffrage:Voter ID:2ff23827-10c6-84b8-e7a4-acd51c6d28fc Address:2ff23827-10c6-84b8-e7a4-acd51c6d28fc} {Suffrage:Voter ID:e21603ea-911a-e69c-4c1b-44b4ff19324a Address:e21603ea-911a-e69c-4c1b-44b4ff19324a}]" + testing.go:149: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.687Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: initial configuration: index=1 servers="[{Suffrage:Voter ID:2ff23827-10c6-84b8-e7a4-acd51c6d28fc Address:2ff23827-10c6-84b8-e7a4-acd51c6d28fc} {Suffrage:Voter ID:e21603ea-911a-e69c-4c1b-44b4ff19324a Address:e21603ea-911a-e69c-4c1b-44b4ff19324a}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:19.687Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.687Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: entering follower state: follower="Node at e21603ea-911a-e69c-4c1b-44b4ff19324a [Follower]" leader= + testing.go:149: cluster: 2025-02-10T11:34:19.687Z [INFO] starting stability test: raft-state=Follower + testing.go:149: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.687Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: entering follower state: follower="Node at e21603ea-911a-e69c-4c1b-44b4ff19324a [Follower]" leader= +testLoggerAdapter verbose: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.739Z [WARN] e21603ea-911a-e69c-4c1b-44b4ff19324a: heartbeat timeout reached, starting election: last-leader= + testing.go:149: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.739Z [WARN] e21603ea-911a-e69c-4c1b-44b4ff19324a: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.739Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: entering candidate state: node="Node at e21603ea-911a-e69c-4c1b-44b4ff19324a [Candidate]" term=2 + testing.go:149: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.739Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: entering candidate state: node="Node at e21603ea-911a-e69c-4c1b-44b4ff19324a [Candidate]" term=2 +testLoggerAdapter verbose: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.739Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: election won: tally=2 + testing.go:149: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.739Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: election won: tally=2 +testLoggerAdapter verbose: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.739Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: entering leader state: leader="Node at e21603ea-911a-e69c-4c1b-44b4ff19324a [Leader]" + testing.go:149: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.739Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: entering leader state: leader="Node at e21603ea-911a-e69c-4c1b-44b4ff19324a [Leader]" +testLoggerAdapter verbose: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.739Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: added peer, starting replication: peer=2ff23827-10c6-84b8-e7a4-acd51c6d28fc + testing.go:149: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.739Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: added peer, starting replication: peer=2ff23827-10c6-84b8-e7a4-acd51c6d28fc +testLoggerAdapter verbose: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.739Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: pipelining replication: peer="{Voter 2ff23827-10c6-84b8-e7a4-acd51c6d28fc 2ff23827-10c6-84b8-e7a4-acd51c6d28fc}" + testing.go:149: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.739Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: pipelining replication: peer="{Voter 2ff23827-10c6-84b8-e7a4-acd51c6d28fc 2ff23827-10c6-84b8-e7a4-acd51c6d28fc}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:19.845Z [INFO] stable state for Follower reached at 2025-02-10 11:34:19.739848152 +0000 UTC m=+16.108752493 (1 nodes), 52.340641ms from start of poll, 52.731566ms from cluster start. Timeout at 2025-02-10 11:34:19.844953176 +0000 UTC m=+16.213857532, 105.105039ms after stability + testing.go:149: cluster: 2025-02-10T11:34:19.845Z [INFO] stable state for Follower reached at 2025-02-10 11:34:19.739848152 +0000 UTC m=+16.108752493 (1 nodes), 52.340641ms from start of poll, 52.731566ms from cluster start. Timeout at 2025-02-10 11:34:19.844953176 +0000 UTC m=+16.213857532, 105.105039ms after stability +testLoggerAdapter verbose: server-dd42822a-5656-f993-ffac-3e11b416e675: 2025-02-10T11:34:19.845Z [INFO] server-dd42822a-5656-f993-ffac-3e11b416e675: initial configuration: index=0 servers=[] + testing.go:149: server-dd42822a-5656-f993-ffac-3e11b416e675: 2025-02-10T11:34:19.845Z [INFO] server-dd42822a-5656-f993-ffac-3e11b416e675: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:19.845Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:19.845Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-dd42822a-5656-f993-ffac-3e11b416e675: 2025-02-10T11:34:19.845Z [INFO] server-dd42822a-5656-f993-ffac-3e11b416e675: entering follower state: follower="Node at dd42822a-5656-f993-ffac-3e11b416e675 [Follower]" leader= + testing.go:149: server-dd42822a-5656-f993-ffac-3e11b416e675: 2025-02-10T11:34:19.845Z [INFO] server-dd42822a-5656-f993-ffac-3e11b416e675: entering follower state: follower="Node at dd42822a-5656-f993-ffac-3e11b416e675 [Follower]" leader= +testLoggerAdapter verbose: server-dd42822a-5656-f993-ffac-3e11b416e675: 2025-02-10T11:34:19.927Z [WARN] server-dd42822a-5656-f993-ffac-3e11b416e675: no known peers, aborting election + testing.go:149: server-dd42822a-5656-f993-ffac-3e11b416e675: 2025-02-10T11:34:19.927Z [WARN] server-dd42822a-5656-f993-ffac-3e11b416e675: no known peers, aborting election +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:19.950Z [INFO] stable state for Leader reached at 2025-02-10 11:34:19.845441527 +0000 UTC m=+16.214345828 (1 nodes), 482ns from start of poll, 158.324901ms from cluster start. Timeout at 2025-02-10 11:34:19.950514667 +0000 UTC m=+16.319418963, 105.073135ms after stability + testing.go:149: cluster: 2025-02-10T11:34:19.950Z [INFO] stable state for Leader reached at 2025-02-10 11:34:19.845441527 +0000 UTC m=+16.214345828 (1 nodes), 482ns from start of poll, 158.324901ms from cluster start. Timeout at 2025-02-10 11:34:19.950514667 +0000 UTC m=+16.319418963, 105.073135ms after stability +testLoggerAdapter verbose: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.950Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: updating configuration: command=AddStaging server-id=server-dd42822a-5656-f993-ffac-3e11b416e675 server-addr=dd42822a-5656-f993-ffac-3e11b416e675 servers="[{Suffrage:Voter ID:2ff23827-10c6-84b8-e7a4-acd51c6d28fc Address:2ff23827-10c6-84b8-e7a4-acd51c6d28fc} {Suffrage:Voter ID:e21603ea-911a-e69c-4c1b-44b4ff19324a Address:e21603ea-911a-e69c-4c1b-44b4ff19324a} {Suffrage:Voter ID:server-dd42822a-5656-f993-ffac-3e11b416e675 Address:dd42822a-5656-f993-ffac-3e11b416e675}]" + testing.go:149: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.950Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: updating configuration: command=AddStaging server-id=server-dd42822a-5656-f993-ffac-3e11b416e675 server-addr=dd42822a-5656-f993-ffac-3e11b416e675 servers="[{Suffrage:Voter ID:2ff23827-10c6-84b8-e7a4-acd51c6d28fc Address:2ff23827-10c6-84b8-e7a4-acd51c6d28fc} {Suffrage:Voter ID:e21603ea-911a-e69c-4c1b-44b4ff19324a Address:e21603ea-911a-e69c-4c1b-44b4ff19324a} {Suffrage:Voter ID:server-dd42822a-5656-f993-ffac-3e11b416e675 Address:dd42822a-5656-f993-ffac-3e11b416e675}]" +testLoggerAdapter verbose: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.950Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: added peer, starting replication: peer=server-dd42822a-5656-f993-ffac-3e11b416e675 + testing.go:149: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.950Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: added peer, starting replication: peer=server-dd42822a-5656-f993-ffac-3e11b416e675 +testLoggerAdapter verbose: server-dd42822a-5656-f993-ffac-3e11b416e675: 2025-02-10T11:34:19.950Z [WARN] server-dd42822a-5656-f993-ffac-3e11b416e675: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-dd42822a-5656-f993-ffac-3e11b416e675: 2025-02-10T11:34:19.950Z [WARN] server-dd42822a-5656-f993-ffac-3e11b416e675: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.950Z [WARN] e21603ea-911a-e69c-4c1b-44b4ff19324a: appendEntries rejected, sending older logs: peer="{Voter server-dd42822a-5656-f993-ffac-3e11b416e675 dd42822a-5656-f993-ffac-3e11b416e675}" next=1 + testing.go:149: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.950Z [WARN] e21603ea-911a-e69c-4c1b-44b4ff19324a: appendEntries rejected, sending older logs: peer="{Voter server-dd42822a-5656-f993-ffac-3e11b416e675 dd42822a-5656-f993-ffac-3e11b416e675}" next=1 +testLoggerAdapter verbose: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.950Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: pipelining replication: peer="{Voter server-dd42822a-5656-f993-ffac-3e11b416e675 dd42822a-5656-f993-ffac-3e11b416e675}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:19.950Z [INFO] starting stability test: raft-state=Leader + testing.go:149: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:19.950Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: pipelining replication: peer="{Voter server-dd42822a-5656-f993-ffac-3e11b416e675 dd42822a-5656-f993-ffac-3e11b416e675}" + testing.go:149: cluster: 2025-02-10T11:34:19.950Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:20.056Z [INFO] stable state for Leader reached at 2025-02-10 11:34:19.950942655 +0000 UTC m=+16.319846956 (1 nodes), 661ns from start of poll, 263.826029ms from cluster start. Timeout at 2025-02-10 11:34:20.05601515 +0000 UTC m=+16.424919446, 105.07249ms after stability + testing.go:149: cluster: 2025-02-10T11:34:20.056Z [INFO] stable state for Leader reached at 2025-02-10 11:34:19.950942655 +0000 UTC m=+16.319846956 (1 nodes), 661ns from start of poll, 263.826029ms from cluster start. Timeout at 2025-02-10 11:34:20.05601515 +0000 UTC m=+16.424919446, 105.07249ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:20.056Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:20.056Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:20.161Z [INFO] stable state for Leader reached at 2025-02-10 11:34:20.056095734 +0000 UTC m=+16.425000032 (1 nodes), 406ns from start of poll, 368.979105ms from cluster start. Timeout at 2025-02-10 11:34:20.161211244 +0000 UTC m=+16.530115565, 105.115533ms after stability + testing.go:149: cluster: 2025-02-10T11:34:20.161Z [INFO] stable state for Leader reached at 2025-02-10 11:34:20.056095734 +0000 UTC m=+16.425000032 (1 nodes), 406ns from start of poll, 368.979105ms from cluster start. Timeout at 2025-02-10 11:34:20.161211244 +0000 UTC m=+16.530115565, 105.115533ms after stability +testLoggerAdapter verbose: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:20.161Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: updating configuration: command=RemoveServer server-id=2ff23827-10c6-84b8-e7a4-acd51c6d28fc server-addr= servers="[{Suffrage:Voter ID:e21603ea-911a-e69c-4c1b-44b4ff19324a Address:e21603ea-911a-e69c-4c1b-44b4ff19324a} {Suffrage:Voter ID:server-dd42822a-5656-f993-ffac-3e11b416e675 Address:dd42822a-5656-f993-ffac-3e11b416e675}]" + testing.go:149: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:20.161Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: updating configuration: command=RemoveServer server-id=2ff23827-10c6-84b8-e7a4-acd51c6d28fc server-addr= servers="[{Suffrage:Voter ID:e21603ea-911a-e69c-4c1b-44b4ff19324a Address:e21603ea-911a-e69c-4c1b-44b4ff19324a} {Suffrage:Voter ID:server-dd42822a-5656-f993-ffac-3e11b416e675 Address:dd42822a-5656-f993-ffac-3e11b416e675}]" +testLoggerAdapter verbose: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:20.161Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: removed peer, stopping replication: peer=2ff23827-10c6-84b8-e7a4-acd51c6d28fc last-index=4 + testing.go:149: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:20.161Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: removed peer, stopping replication: peer=2ff23827-10c6-84b8-e7a4-acd51c6d28fc last-index=4 +testLoggerAdapter verbose: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:20.161Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: aborting pipeline replication: peer="{Voter 2ff23827-10c6-84b8-e7a4-acd51c6d28fc 2ff23827-10c6-84b8-e7a4-acd51c6d28fc}" + testing.go:149: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:20.161Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: aborting pipeline replication: peer="{Voter 2ff23827-10c6-84b8-e7a4-acd51c6d28fc 2ff23827-10c6-84b8-e7a4-acd51c6d28fc}" +testLoggerAdapter verbose: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:20.161Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: aborting pipeline replication: peer="{Voter server-dd42822a-5656-f993-ffac-3e11b416e675 dd42822a-5656-f993-ffac-3e11b416e675}" + testing.go:149: e21603ea-911a-e69c-4c1b-44b4ff19324a: 2025-02-10T11:34:20.161Z [INFO] e21603ea-911a-e69c-4c1b-44b4ff19324a: aborting pipeline replication: peer="{Voter server-dd42822a-5656-f993-ffac-3e11b416e675 dd42822a-5656-f993-ffac-3e11b416e675}" +--- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.48s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer -testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.250Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a19ff899-04fa-609d-505d-f02c746b4e5a Address:a19ff899-04fa-609d-505d-f02c746b4e5a} {Suffrage:Voter ID:server-5edbc0c0-7378-358b-2799-d963b1edf0fe Address:5edbc0c0-7378-358b-2799-d963b1edf0fe} {Suffrage:Voter ID:server-c41e38b3-6a17-7080-5877-70799996b8ea Address:c41e38b3-6a17-7080-5877-70799996b8ea}]" - testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.250Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a19ff899-04fa-609d-505d-f02c746b4e5a Address:a19ff899-04fa-609d-505d-f02c746b4e5a} {Suffrage:Voter ID:server-5edbc0c0-7378-358b-2799-d963b1edf0fe Address:5edbc0c0-7378-358b-2799-d963b1edf0fe} {Suffrage:Voter ID:server-c41e38b3-6a17-7080-5877-70799996b8ea Address:c41e38b3-6a17-7080-5877-70799996b8ea}]" -testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.250Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: entering follower state: follower="Node at a19ff899-04fa-609d-505d-f02c746b4e5a [Follower]" leader= - testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.250Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: entering follower state: follower="Node at a19ff899-04fa-609d-505d-f02c746b4e5a [Follower]" leader= -testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.250Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a19ff899-04fa-609d-505d-f02c746b4e5a Address:a19ff899-04fa-609d-505d-f02c746b4e5a} {Suffrage:Voter ID:server-5edbc0c0-7378-358b-2799-d963b1edf0fe Address:5edbc0c0-7378-358b-2799-d963b1edf0fe} {Suffrage:Voter ID:server-c41e38b3-6a17-7080-5877-70799996b8ea Address:c41e38b3-6a17-7080-5877-70799996b8ea}]" - testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.250Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a19ff899-04fa-609d-505d-f02c746b4e5a Address:a19ff899-04fa-609d-505d-f02c746b4e5a} {Suffrage:Voter ID:server-5edbc0c0-7378-358b-2799-d963b1edf0fe Address:5edbc0c0-7378-358b-2799-d963b1edf0fe} {Suffrage:Voter ID:server-c41e38b3-6a17-7080-5877-70799996b8ea Address:c41e38b3-6a17-7080-5877-70799996b8ea}]" -testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.251Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: entering follower state: follower="Node at 5edbc0c0-7378-358b-2799-d963b1edf0fe [Follower]" leader= - testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.251Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: entering follower state: follower="Node at 5edbc0c0-7378-358b-2799-d963b1edf0fe [Follower]" leader= -testLoggerAdapter verbose: server-c41e38b3-6a17-7080-5877-70799996b8ea: 2024-01-09T05:10:06.251Z [INFO] server-c41e38b3-6a17-7080-5877-70799996b8ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a19ff899-04fa-609d-505d-f02c746b4e5a Address:a19ff899-04fa-609d-505d-f02c746b4e5a} {Suffrage:Voter ID:server-5edbc0c0-7378-358b-2799-d963b1edf0fe Address:5edbc0c0-7378-358b-2799-d963b1edf0fe} {Suffrage:Voter ID:server-c41e38b3-6a17-7080-5877-70799996b8ea Address:c41e38b3-6a17-7080-5877-70799996b8ea}]" - testing.go:149: server-c41e38b3-6a17-7080-5877-70799996b8ea: 2024-01-09T05:10:06.251Z [INFO] server-c41e38b3-6a17-7080-5877-70799996b8ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a19ff899-04fa-609d-505d-f02c746b4e5a Address:a19ff899-04fa-609d-505d-f02c746b4e5a} {Suffrage:Voter ID:server-5edbc0c0-7378-358b-2799-d963b1edf0fe Address:5edbc0c0-7378-358b-2799-d963b1edf0fe} {Suffrage:Voter ID:server-c41e38b3-6a17-7080-5877-70799996b8ea Address:c41e38b3-6a17-7080-5877-70799996b8ea}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.251Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:06.251Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-c41e38b3-6a17-7080-5877-70799996b8ea: 2024-01-09T05:10:06.251Z [INFO] server-c41e38b3-6a17-7080-5877-70799996b8ea: entering follower state: follower="Node at c41e38b3-6a17-7080-5877-70799996b8ea [Follower]" leader= - testing.go:149: server-c41e38b3-6a17-7080-5877-70799996b8ea: 2024-01-09T05:10:06.251Z [INFO] server-c41e38b3-6a17-7080-5877-70799996b8ea: entering follower state: follower="Node at c41e38b3-6a17-7080-5877-70799996b8ea [Follower]" leader= -testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [WARN] server-a19ff899-04fa-609d-505d-f02c746b4e5a: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [WARN] server-a19ff899-04fa-609d-505d-f02c746b4e5a: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: entering candidate state: node="Node at a19ff899-04fa-609d-505d-f02c746b4e5a [Candidate]" term=2 - testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: entering candidate state: node="Node at a19ff899-04fa-609d-505d-f02c746b4e5a [Candidate]" term=2 -testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: election won: tally=2 - testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: election won: tally=2 -testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: entering leader state: leader="Node at a19ff899-04fa-609d-505d-f02c746b4e5a [Leader]" - testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: entering leader state: leader="Node at a19ff899-04fa-609d-505d-f02c746b4e5a [Leader]" -testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: added peer, starting replication: peer=server-5edbc0c0-7378-358b-2799-d963b1edf0fe - testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: added peer, starting replication: peer=server-5edbc0c0-7378-358b-2799-d963b1edf0fe -testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: added peer, starting replication: peer=server-c41e38b3-6a17-7080-5877-70799996b8ea - testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: added peer, starting replication: peer=server-c41e38b3-6a17-7080-5877-70799996b8ea -testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: pipelining replication: peer="{Voter server-c41e38b3-6a17-7080-5877-70799996b8ea c41e38b3-6a17-7080-5877-70799996b8ea}" - testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: pipelining replication: peer="{Voter server-c41e38b3-6a17-7080-5877-70799996b8ea c41e38b3-6a17-7080-5877-70799996b8ea}" -testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.309Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: pipelining replication: peer="{Voter server-5edbc0c0-7378-358b-2799-d963b1edf0fe 5edbc0c0-7378-358b-2799-d963b1edf0fe}" - testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.309Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: pipelining replication: peer="{Voter server-5edbc0c0-7378-358b-2799-d963b1edf0fe 5edbc0c0-7378-358b-2799-d963b1edf0fe}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.414Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.309102467 +0000 UTC m=+17.562153655 (1 nodes), 57.754861ms from start of poll, 58.562369ms from cluster start. Timeout at 2024-01-09 05:10:06.414600017 +0000 UTC m=+17.667651000, 105.497345ms after stability - testing.go:149: cluster: 2024-01-09T05:10:06.414Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.309102467 +0000 UTC m=+17.562153655 (1 nodes), 57.754861ms from start of poll, 58.562369ms from cluster start. Timeout at 2024-01-09 05:10:06.414600017 +0000 UTC m=+17.667651000, 105.497345ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.414Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:06.414Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.520Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.41478448 +0000 UTC m=+17.667835507 (1 nodes), 2.527µs from start of poll, 164.244221ms from cluster start. Timeout at 2024-01-09 05:10:06.520923969 +0000 UTC m=+17.773975067, 106.13956ms after stability - testing.go:149: cluster: 2024-01-09T05:10:06.520Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.41478448 +0000 UTC m=+17.667835507 (1 nodes), 2.527µs from start of poll, 164.244221ms from cluster start. Timeout at 2024-01-09 05:10:06.520923969 +0000 UTC m=+17.773975067, 106.13956ms after stability -testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: entering candidate state: node="Node at 5edbc0c0-7378-358b-2799-d963b1edf0fe [Candidate]" term=3 - testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: entering candidate state: node="Node at 5edbc0c0-7378-358b-2799-d963b1edf0fe [Candidate]" term=3 -testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: election won: tally=2 - testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: election won: tally=2 -testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: entering leader state: leader="Node at 5edbc0c0-7378-358b-2799-d963b1edf0fe [Leader]" - testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: entering leader state: leader="Node at 5edbc0c0-7378-358b-2799-d963b1edf0fe [Leader]" -testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: added peer, starting replication: peer=server-a19ff899-04fa-609d-505d-f02c746b4e5a - testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: added peer, starting replication: peer=server-a19ff899-04fa-609d-505d-f02c746b4e5a -testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: added peer, starting replication: peer=server-c41e38b3-6a17-7080-5877-70799996b8ea - testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: added peer, starting replication: peer=server-c41e38b3-6a17-7080-5877-70799996b8ea -testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: pipelining replication: peer="{Voter server-c41e38b3-6a17-7080-5877-70799996b8ea c41e38b3-6a17-7080-5877-70799996b8ea}" - testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: pipelining replication: peer="{Voter server-c41e38b3-6a17-7080-5877-70799996b8ea c41e38b3-6a17-7080-5877-70799996b8ea}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.521Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:06.521Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.521Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: entering follower state: follower="Node at a19ff899-04fa-609d-505d-f02c746b4e5a [Follower]" leader= - testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.521Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: entering follower state: follower="Node at a19ff899-04fa-609d-505d-f02c746b4e5a [Follower]" leader= -testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.521Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: aborting pipeline replication: peer="{Voter server-c41e38b3-6a17-7080-5877-70799996b8ea c41e38b3-6a17-7080-5877-70799996b8ea}" - testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.521Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: aborting pipeline replication: peer="{Voter server-c41e38b3-6a17-7080-5877-70799996b8ea c41e38b3-6a17-7080-5877-70799996b8ea}" -testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: pipelining replication: peer="{Voter server-a19ff899-04fa-609d-505d-f02c746b4e5a a19ff899-04fa-609d-505d-f02c746b4e5a}" - testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: pipelining replication: peer="{Voter server-a19ff899-04fa-609d-505d-f02c746b4e5a a19ff899-04fa-609d-505d-f02c746b4e5a}" -testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.521Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: aborting pipeline replication: peer="{Voter server-5edbc0c0-7378-358b-2799-d963b1edf0fe 5edbc0c0-7378-358b-2799-d963b1edf0fe}" - testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.521Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: aborting pipeline replication: peer="{Voter server-5edbc0c0-7378-358b-2799-d963b1edf0fe 5edbc0c0-7378-358b-2799-d963b1edf0fe}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.627Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.521861338 +0000 UTC m=+17.774912374 (1 nodes), 370.172µs from start of poll, 271.321088ms from cluster start. Timeout at 2024-01-09 05:10:06.627069519 +0000 UTC m=+17.880120631, 105.208257ms after stability - testing.go:149: cluster: 2024-01-09T05:10:06.627Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.521861338 +0000 UTC m=+17.774912374 (1 nodes), 370.172µs from start of poll, 271.321088ms from cluster start. Timeout at 2024-01-09 05:10:06.627069519 +0000 UTC m=+17.880120631, 105.208257ms after stability -testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.627Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: aborting pipeline replication: peer="{Voter server-c41e38b3-6a17-7080-5877-70799996b8ea c41e38b3-6a17-7080-5877-70799996b8ea}" - testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.627Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: aborting pipeline replication: peer="{Voter server-c41e38b3-6a17-7080-5877-70799996b8ea c41e38b3-6a17-7080-5877-70799996b8ea}" -testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.627Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: aborting pipeline replication: peer="{Voter server-a19ff899-04fa-609d-505d-f02c746b4e5a a19ff899-04fa-609d-505d-f02c746b4e5a}" - testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.627Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: aborting pipeline replication: peer="{Voter server-a19ff899-04fa-609d-505d-f02c746b4e5a a19ff899-04fa-609d-505d-f02c746b4e5a}" +testLoggerAdapter verbose: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.162Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 Address:a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44} {Suffrage:Voter ID:server-7572b4a9-ed33-44cd-89b5-751227669e8b Address:7572b4a9-ed33-44cd-89b5-751227669e8b} {Suffrage:Voter ID:server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca Address:e32b7316-c8b4-fd9c-7930-d8a3dbb037ca}]" + testing.go:149: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.162Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 Address:a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44} {Suffrage:Voter ID:server-7572b4a9-ed33-44cd-89b5-751227669e8b Address:7572b4a9-ed33-44cd-89b5-751227669e8b} {Suffrage:Voter ID:server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca Address:e32b7316-c8b4-fd9c-7930-d8a3dbb037ca}]" +testLoggerAdapter verbose: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.162Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: entering follower state: follower="Node at a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 [Follower]" leader= + testing.go:149: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.162Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: entering follower state: follower="Node at a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 [Follower]" leader= +testLoggerAdapter verbose: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.163Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 Address:a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44} {Suffrage:Voter ID:server-7572b4a9-ed33-44cd-89b5-751227669e8b Address:7572b4a9-ed33-44cd-89b5-751227669e8b} {Suffrage:Voter ID:server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca Address:e32b7316-c8b4-fd9c-7930-d8a3dbb037ca}]" + testing.go:149: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.163Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 Address:a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44} {Suffrage:Voter ID:server-7572b4a9-ed33-44cd-89b5-751227669e8b Address:7572b4a9-ed33-44cd-89b5-751227669e8b} {Suffrage:Voter ID:server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca Address:e32b7316-c8b4-fd9c-7930-d8a3dbb037ca}]" +testLoggerAdapter verbose: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.163Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: entering follower state: follower="Node at 7572b4a9-ed33-44cd-89b5-751227669e8b [Follower]" leader= + testing.go:149: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.163Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: entering follower state: follower="Node at 7572b4a9-ed33-44cd-89b5-751227669e8b [Follower]" leader= +testLoggerAdapter verbose: server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca: 2025-02-10T11:34:20.163Z [INFO] server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 Address:a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44} {Suffrage:Voter ID:server-7572b4a9-ed33-44cd-89b5-751227669e8b Address:7572b4a9-ed33-44cd-89b5-751227669e8b} {Suffrage:Voter ID:server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca Address:e32b7316-c8b4-fd9c-7930-d8a3dbb037ca}]" + testing.go:149: server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca: 2025-02-10T11:34:20.163Z [INFO] server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 Address:a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44} {Suffrage:Voter ID:server-7572b4a9-ed33-44cd-89b5-751227669e8b Address:7572b4a9-ed33-44cd-89b5-751227669e8b} {Suffrage:Voter ID:server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca Address:e32b7316-c8b4-fd9c-7930-d8a3dbb037ca}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:20.163Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca: 2025-02-10T11:34:20.163Z [INFO] server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca: entering follower state: follower="Node at e32b7316-c8b4-fd9c-7930-d8a3dbb037ca [Follower]" leader= + testing.go:149: cluster: 2025-02-10T11:34:20.163Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca: 2025-02-10T11:34:20.163Z [INFO] server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca: entering follower state: follower="Node at e32b7316-c8b4-fd9c-7930-d8a3dbb037ca [Follower]" leader= +testLoggerAdapter verbose: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.226Z [WARN] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.226Z [WARN] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.226Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: entering candidate state: node="Node at a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 [Candidate]" term=2 + testing.go:149: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.226Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: entering candidate state: node="Node at a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 [Candidate]" term=2 +testLoggerAdapter verbose: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.226Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: election won: tally=2 + testing.go:149: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.226Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: election won: tally=2 +testLoggerAdapter verbose: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.226Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: entering leader state: leader="Node at a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 [Leader]" + testing.go:149: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.226Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: entering leader state: leader="Node at a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 [Leader]" +testLoggerAdapter verbose: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.226Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: added peer, starting replication: peer=server-7572b4a9-ed33-44cd-89b5-751227669e8b + testing.go:149: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.226Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: added peer, starting replication: peer=server-7572b4a9-ed33-44cd-89b5-751227669e8b +testLoggerAdapter verbose: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.226Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: added peer, starting replication: peer=server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca + testing.go:149: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.226Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: added peer, starting replication: peer=server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca +testLoggerAdapter verbose: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.226Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: pipelining replication: peer="{Voter server-7572b4a9-ed33-44cd-89b5-751227669e8b 7572b4a9-ed33-44cd-89b5-751227669e8b}" + testing.go:149: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.226Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: pipelining replication: peer="{Voter server-7572b4a9-ed33-44cd-89b5-751227669e8b 7572b4a9-ed33-44cd-89b5-751227669e8b}" +testLoggerAdapter verbose: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.226Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: pipelining replication: peer="{Voter server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca e32b7316-c8b4-fd9c-7930-d8a3dbb037ca}" + testing.go:149: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.226Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: pipelining replication: peer="{Voter server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca e32b7316-c8b4-fd9c-7930-d8a3dbb037ca}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:20.331Z [INFO] stable state for Leader reached at 2025-02-10 11:34:20.226413377 +0000 UTC m=+16.595317684 (1 nodes), 63.104608ms from start of poll, 63.662272ms from cluster start. Timeout at 2025-02-10 11:34:20.331487019 +0000 UTC m=+16.700391315, 105.073631ms after stability + testing.go:149: cluster: 2025-02-10T11:34:20.331Z [INFO] stable state for Leader reached at 2025-02-10 11:34:20.226413377 +0000 UTC m=+16.595317684 (1 nodes), 63.104608ms from start of poll, 63.662272ms from cluster start. Timeout at 2025-02-10 11:34:20.331487019 +0000 UTC m=+16.700391315, 105.073631ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:20.331Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:20.331Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:20.436Z [INFO] stable state for Leader reached at 2025-02-10 11:34:20.33155468 +0000 UTC m=+16.700458980 (1 nodes), 471ns from start of poll, 168.803568ms from cluster start. Timeout at 2025-02-10 11:34:20.436603224 +0000 UTC m=+16.805507518, 105.048538ms after stability + testing.go:149: cluster: 2025-02-10T11:34:20.436Z [INFO] stable state for Leader reached at 2025-02-10 11:34:20.33155468 +0000 UTC m=+16.700458980 (1 nodes), 471ns from start of poll, 168.803568ms from cluster start. Timeout at 2025-02-10 11:34:20.436603224 +0000 UTC m=+16.805507518, 105.048538ms after stability +testLoggerAdapter verbose: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.436Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: entering candidate state: node="Node at 7572b4a9-ed33-44cd-89b5-751227669e8b [Candidate]" term=3 +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:20.436Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.436Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: entering candidate state: node="Node at 7572b4a9-ed33-44cd-89b5-751227669e8b [Candidate]" term=3 + testing.go:149: cluster: 2025-02-10T11:34:20.436Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.436Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: entering follower state: follower="Node at a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 [Follower]" leader= +testLoggerAdapter verbose: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.436Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: election won: tally=2 + testing.go:149: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.436Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: entering follower state: follower="Node at a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 [Follower]" leader= +testLoggerAdapter verbose: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.436Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: aborting pipeline replication: peer="{Voter server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca e32b7316-c8b4-fd9c-7930-d8a3dbb037ca}" + testing.go:149: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.436Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: election won: tally=2 +testLoggerAdapter verbose: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.436Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: entering leader state: leader="Node at 7572b4a9-ed33-44cd-89b5-751227669e8b [Leader]" + testing.go:149: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.436Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: aborting pipeline replication: peer="{Voter server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca e32b7316-c8b4-fd9c-7930-d8a3dbb037ca}" +testLoggerAdapter verbose: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.436Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: aborting pipeline replication: peer="{Voter server-7572b4a9-ed33-44cd-89b5-751227669e8b 7572b4a9-ed33-44cd-89b5-751227669e8b}" + testing.go:149: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.436Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: entering leader state: leader="Node at 7572b4a9-ed33-44cd-89b5-751227669e8b [Leader]" +testLoggerAdapter verbose: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.436Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: added peer, starting replication: peer=server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 + testing.go:149: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.436Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: added peer, starting replication: peer=server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 +testLoggerAdapter verbose: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.436Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: added peer, starting replication: peer=server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca + testing.go:149: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.436Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: added peer, starting replication: peer=server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca +testLoggerAdapter verbose: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.437Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: pipelining replication: peer="{Voter server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca e32b7316-c8b4-fd9c-7930-d8a3dbb037ca}" + testing.go:149: server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: 2025-02-10T11:34:20.436Z [INFO] server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44: aborting pipeline replication: peer="{Voter server-7572b4a9-ed33-44cd-89b5-751227669e8b 7572b4a9-ed33-44cd-89b5-751227669e8b}" + testing.go:149: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.437Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: pipelining replication: peer="{Voter server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca e32b7316-c8b4-fd9c-7930-d8a3dbb037ca}" +testLoggerAdapter verbose: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.437Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: pipelining replication: peer="{Voter server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44}" + testing.go:149: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.437Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: pipelining replication: peer="{Voter server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:20.542Z [INFO] stable state for Leader reached at 2025-02-10 11:34:20.436910549 +0000 UTC m=+16.805814867 (1 nodes), 73.749µs from start of poll, 274.159455ms from cluster start. Timeout at 2025-02-10 11:34:20.541982878 +0000 UTC m=+16.910887173, 105.072306ms after stability + testing.go:149: cluster: 2025-02-10T11:34:20.542Z [INFO] stable state for Leader reached at 2025-02-10 11:34:20.436910549 +0000 UTC m=+16.805814867 (1 nodes), 73.749µs from start of poll, 274.159455ms from cluster start. Timeout at 2025-02-10 11:34:20.541982878 +0000 UTC m=+16.910887173, 105.072306ms after stability +testLoggerAdapter verbose: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.542Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: aborting pipeline replication: peer="{Voter server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca e32b7316-c8b4-fd9c-7930-d8a3dbb037ca}" + testing.go:149: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.542Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: aborting pipeline replication: peer="{Voter server-e32b7316-c8b4-fd9c-7930-d8a3dbb037ca e32b7316-c8b4-fd9c-7930-d8a3dbb037ca}" +testLoggerAdapter verbose: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.542Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: aborting pipeline replication: peer="{Voter server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44}" + testing.go:149: server-7572b4a9-ed33-44cd-89b5-751227669e8b: 2025-02-10T11:34:20.542Z [INFO] server-7572b4a9-ed33-44cd-89b5-751227669e8b: aborting pipeline replication: peer="{Voter server-a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44 a5df6a28-fb4a-d9d3-d1f3-619ff0b2dc44}" --- PASS: TestRaft_LeadershipTransfer (0.38s) === RUN TestRaft_LeadershipTransferWithOneNode -testLoggerAdapter verbose: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.628Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-365a107c-90e7-6fc4-be0e-524397881df2 Address:365a107c-90e7-6fc4-be0e-524397881df2}]" - testing.go:149: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.628Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-365a107c-90e7-6fc4-be0e-524397881df2 Address:365a107c-90e7-6fc4-be0e-524397881df2}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.628Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:06.628Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.628Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: entering follower state: follower="Node at 365a107c-90e7-6fc4-be0e-524397881df2 [Follower]" leader= - testing.go:149: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.628Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: entering follower state: follower="Node at 365a107c-90e7-6fc4-be0e-524397881df2 [Follower]" leader= -testLoggerAdapter verbose: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.694Z [WARN] server-365a107c-90e7-6fc4-be0e-524397881df2: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.694Z [WARN] server-365a107c-90e7-6fc4-be0e-524397881df2: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.694Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: entering candidate state: node="Node at 365a107c-90e7-6fc4-be0e-524397881df2 [Candidate]" term=2 - testing.go:149: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.694Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: entering candidate state: node="Node at 365a107c-90e7-6fc4-be0e-524397881df2 [Candidate]" term=2 -testLoggerAdapter verbose: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.695Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: election won: tally=1 - testing.go:149: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.695Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: election won: tally=1 -testLoggerAdapter verbose: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.695Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: entering leader state: leader="Node at 365a107c-90e7-6fc4-be0e-524397881df2 [Leader]" - testing.go:149: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.695Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: entering leader state: leader="Node at 365a107c-90e7-6fc4-be0e-524397881df2 [Leader]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.806Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.695143229 +0000 UTC m=+17.948194276 (1 nodes), 66.599257ms from start of poll, 66.804662ms from cluster start. Timeout at 2024-01-09 05:10:06.8069404 +0000 UTC m=+18.059991508, 111.797232ms after stability - testing.go:149: cluster: 2024-01-09T05:10:06.806Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.695143229 +0000 UTC m=+17.948194276 (1 nodes), 66.599257ms from start of poll, 66.804662ms from cluster start. Timeout at 2024-01-09 05:10:06.8069404 +0000 UTC m=+18.059991508, 111.797232ms after stability ---- PASS: TestRaft_LeadershipTransferWithOneNode (0.18s) +testLoggerAdapter verbose: server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: 2025-02-10T11:34:20.542Z [INFO] server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2 Address:1ef26271-7495-7c2b-3ae7-f8b5ed3012a2}]" + testing.go:149: server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: 2025-02-10T11:34:20.542Z [INFO] server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2 Address:1ef26271-7495-7c2b-3ae7-f8b5ed3012a2}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:20.543Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:20.543Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: 2025-02-10T11:34:20.543Z [INFO] server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: entering follower state: follower="Node at 1ef26271-7495-7c2b-3ae7-f8b5ed3012a2 [Follower]" leader= + testing.go:149: server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: 2025-02-10T11:34:20.543Z [INFO] server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: entering follower state: follower="Node at 1ef26271-7495-7c2b-3ae7-f8b5ed3012a2 [Follower]" leader= +testLoggerAdapter verbose: server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: 2025-02-10T11:34:20.642Z [WARN] server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: 2025-02-10T11:34:20.642Z [WARN] server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: 2025-02-10T11:34:20.642Z [INFO] server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: entering candidate state: node="Node at 1ef26271-7495-7c2b-3ae7-f8b5ed3012a2 [Candidate]" term=2 + testing.go:149: server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: 2025-02-10T11:34:20.642Z [INFO] server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: entering candidate state: node="Node at 1ef26271-7495-7c2b-3ae7-f8b5ed3012a2 [Candidate]" term=2 +testLoggerAdapter verbose: server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: 2025-02-10T11:34:20.642Z [INFO] server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: election won: tally=1 + testing.go:149: server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: 2025-02-10T11:34:20.642Z [INFO] server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: election won: tally=1 +testLoggerAdapter verbose: server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: 2025-02-10T11:34:20.642Z [INFO] server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: entering leader state: leader="Node at 1ef26271-7495-7c2b-3ae7-f8b5ed3012a2 [Leader]" + testing.go:149: server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: 2025-02-10T11:34:20.642Z [INFO] server-1ef26271-7495-7c2b-3ae7-f8b5ed3012a2: entering leader state: leader="Node at 1ef26271-7495-7c2b-3ae7-f8b5ed3012a2 [Leader]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:20.747Z [INFO] stable state for Leader reached at 2025-02-10 11:34:20.642236096 +0000 UTC m=+17.011140396 (1 nodes), 99.21222ms from start of poll, 99.372209ms from cluster start. Timeout at 2025-02-10 11:34:20.747313529 +0000 UTC m=+17.116217835, 105.077439ms after stability + testing.go:149: cluster: 2025-02-10T11:34:20.747Z [INFO] stable state for Leader reached at 2025-02-10 11:34:20.642236096 +0000 UTC m=+17.011140396 (1 nodes), 99.21222ms from start of poll, 99.372209ms from cluster start. Timeout at 2025-02-10 11:34:20.747313529 +0000 UTC m=+17.116217835, 105.077439ms after stability +--- PASS: TestRaft_LeadershipTransferWithOneNode (0.20s) === RUN TestRaft_LeadershipTransferWithSevenNodes -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:06.808Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:06.808Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" -testLoggerAdapter verbose: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.808Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" - testing.go:149: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.808Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" -testLoggerAdapter verbose: server-7690a8da-ca4b-74d6-905a-84887782f70a: 2024-01-09T05:10:06.809Z [INFO] server-7690a8da-ca4b-74d6-905a-84887782f70a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" - testing.go:149: server-7690a8da-ca4b-74d6-905a-84887782f70a: 2024-01-09T05:10:06.809Z [INFO] server-7690a8da-ca4b-74d6-905a-84887782f70a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" -testLoggerAdapter verbose: server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: 2024-01-09T05:10:06.809Z [INFO] server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" - testing.go:149: server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: 2024-01-09T05:10:06.809Z [INFO] server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.809Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.809Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" -testLoggerAdapter verbose: server-a96c7b98-21ec-611e-8239-dea35c98ee1e: 2024-01-09T05:10:06.809Z [INFO] server-a96c7b98-21ec-611e-8239-dea35c98ee1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" - testing.go:149: server-a96c7b98-21ec-611e-8239-dea35c98ee1e: 2024-01-09T05:10:06.809Z [INFO] server-a96c7b98-21ec-611e-8239-dea35c98ee1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" -testLoggerAdapter verbose: server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: 2024-01-09T05:10:06.809Z [INFO] server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" - testing.go:149: server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: 2024-01-09T05:10:06.809Z [INFO] server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.809Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:06.809Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:06.809Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: entering follower state: follower="Node at c9b91bba-dc30-e04b-fb6e-30317aaae7f0 [Follower]" leader= - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:06.809Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: entering follower state: follower="Node at c9b91bba-dc30-e04b-fb6e-30317aaae7f0 [Follower]" leader= -testLoggerAdapter verbose: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.809Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: entering follower state: follower="Node at 78458f2b-7852-b1e6-0e47-696b8d2b327a [Follower]" leader= - testing.go:149: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.809Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: entering follower state: follower="Node at 78458f2b-7852-b1e6-0e47-696b8d2b327a [Follower]" leader= -testLoggerAdapter verbose: server-7690a8da-ca4b-74d6-905a-84887782f70a: 2024-01-09T05:10:06.809Z [INFO] server-7690a8da-ca4b-74d6-905a-84887782f70a: entering follower state: follower="Node at 7690a8da-ca4b-74d6-905a-84887782f70a [Follower]" leader= - testing.go:149: server-7690a8da-ca4b-74d6-905a-84887782f70a: 2024-01-09T05:10:06.809Z [INFO] server-7690a8da-ca4b-74d6-905a-84887782f70a: entering follower state: follower="Node at 7690a8da-ca4b-74d6-905a-84887782f70a [Follower]" leader= -testLoggerAdapter verbose: server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: 2024-01-09T05:10:06.810Z [INFO] server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: entering follower state: follower="Node at b1cf2ae1-f267-abda-3212-ac286ecd9fdb [Follower]" leader= - testing.go:149: server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: 2024-01-09T05:10:06.810Z [INFO] server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: entering follower state: follower="Node at b1cf2ae1-f267-abda-3212-ac286ecd9fdb [Follower]" leader= -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.810Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: entering follower state: follower="Node at e5774e4f-7d02-8923-9fa1-7d61c66aeee0 [Follower]" leader= - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.810Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: entering follower state: follower="Node at e5774e4f-7d02-8923-9fa1-7d61c66aeee0 [Follower]" leader= -testLoggerAdapter verbose: server-a96c7b98-21ec-611e-8239-dea35c98ee1e: 2024-01-09T05:10:06.810Z [INFO] server-a96c7b98-21ec-611e-8239-dea35c98ee1e: entering follower state: follower="Node at a96c7b98-21ec-611e-8239-dea35c98ee1e [Follower]" leader= - testing.go:149: server-a96c7b98-21ec-611e-8239-dea35c98ee1e: 2024-01-09T05:10:06.810Z [INFO] server-a96c7b98-21ec-611e-8239-dea35c98ee1e: entering follower state: follower="Node at a96c7b98-21ec-611e-8239-dea35c98ee1e [Follower]" leader= -testLoggerAdapter verbose: server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: 2024-01-09T05:10:06.810Z [INFO] server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: entering follower state: follower="Node at 7e15329d-e3f8-73ab-e49c-21e6c404a4ff [Follower]" leader= - testing.go:149: server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: 2024-01-09T05:10:06.810Z [INFO] server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: entering follower state: follower="Node at 7e15329d-e3f8-73ab-e49c-21e6c404a4ff [Follower]" leader= -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.862Z [WARN] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.862Z [WARN] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: entering candidate state: node="Node at e5774e4f-7d02-8923-9fa1-7d61c66aeee0 [Candidate]" term=2 - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: entering candidate state: node="Node at e5774e4f-7d02-8923-9fa1-7d61c66aeee0 [Candidate]" term=2 -testLoggerAdapter verbose: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.863Z [WARN] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.863Z [WARN] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.863Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: entering candidate state: node="Node at 78458f2b-7852-b1e6-0e47-696b8d2b327a [Candidate]" term=2 - testing.go:149: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.863Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: entering candidate state: node="Node at 78458f2b-7852-b1e6-0e47-696b8d2b327a [Candidate]" term=2 -testLoggerAdapter verbose: server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: 2024-01-09T05:10:06.863Z [INFO] server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: duplicate requestVote for same term: term=2 - testing.go:149: server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: 2024-01-09T05:10:06.863Z [INFO] server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.863Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: duplicate requestVote for same term: term=2 - testing.go:149: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.863Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: election won: tally=4 - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: election won: tally=4 -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: entering leader state: leader="Node at e5774e4f-7d02-8923-9fa1-7d61c66aeee0 [Leader]" - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: entering leader state: leader="Node at e5774e4f-7d02-8923-9fa1-7d61c66aeee0 [Leader]" -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-78458f2b-7852-b1e6-0e47-696b8d2b327a - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-78458f2b-7852-b1e6-0e47-696b8d2b327a -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-7690a8da-ca4b-74d6-905a-84887782f70a - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-7690a8da-ca4b-74d6-905a-84887782f70a -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-a96c7b98-21ec-611e-8239-dea35c98ee1e - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-a96c7b98-21ec-611e-8239-dea35c98ee1e -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff 7e15329d-e3f8-73ab-e49c-21e6c404a4ff}" - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff 7e15329d-e3f8-73ab-e49c-21e6c404a4ff}" -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:06.863Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: duplicate requestVote for same term: term=2 - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:06.863Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: server-7690a8da-ca4b-74d6-905a-84887782f70a: 2024-01-09T05:10:06.863Z [INFO] server-7690a8da-ca4b-74d6-905a-84887782f70a: duplicate requestVote for same term: term=2 - testing.go:149: server-7690a8da-ca4b-74d6-905a-84887782f70a: 2024-01-09T05:10:06.863Z [INFO] server-7690a8da-ca4b-74d6-905a-84887782f70a: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: 2024-01-09T05:10:06.863Z [INFO] server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: duplicate requestVote for same term: term=2 - testing.go:149: server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: 2024-01-09T05:10:06.863Z [INFO] server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [WARN] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: rejecting vote request since we have a leader: from=78458f2b-7852-b1e6-0e47-696b8d2b327a leader=e5774e4f-7d02-8923-9fa1-7d61c66aeee0 - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [WARN] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: rejecting vote request since we have a leader: from=78458f2b-7852-b1e6-0e47-696b8d2b327a leader=e5774e4f-7d02-8923-9fa1-7d61c66aeee0 -testLoggerAdapter verbose: server-a96c7b98-21ec-611e-8239-dea35c98ee1e: 2024-01-09T05:10:06.864Z [INFO] server-a96c7b98-21ec-611e-8239-dea35c98ee1e: duplicate requestVote for same term: term=2 - testing.go:149: server-a96c7b98-21ec-611e-8239-dea35c98ee1e: 2024-01-09T05:10:06.864Z [INFO] server-a96c7b98-21ec-611e-8239-dea35c98ee1e: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 c9b91bba-dc30-e04b-fb6e-30317aaae7f0}" - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 c9b91bba-dc30-e04b-fb6e-30317aaae7f0}" -testLoggerAdapter verbose: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.864Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: entering follower state: follower="Node at 78458f2b-7852-b1e6-0e47-696b8d2b327a [Follower]" leader=e5774e4f-7d02-8923-9fa1-7d61c66aeee0 - testing.go:149: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.864Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: entering follower state: follower="Node at 78458f2b-7852-b1e6-0e47-696b8d2b327a [Follower]" leader=e5774e4f-7d02-8923-9fa1-7d61c66aeee0 -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-78458f2b-7852-b1e6-0e47-696b8d2b327a 78458f2b-7852-b1e6-0e47-696b8d2b327a}" - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-78458f2b-7852-b1e6-0e47-696b8d2b327a 78458f2b-7852-b1e6-0e47-696b8d2b327a}" -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-7690a8da-ca4b-74d6-905a-84887782f70a 7690a8da-ca4b-74d6-905a-84887782f70a}" - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-7690a8da-ca4b-74d6-905a-84887782f70a 7690a8da-ca4b-74d6-905a-84887782f70a}" -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb b1cf2ae1-f267-abda-3212-ac286ecd9fdb}" - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb b1cf2ae1-f267-abda-3212-ac286ecd9fdb}" -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-a96c7b98-21ec-611e-8239-dea35c98ee1e a96c7b98-21ec-611e-8239-dea35c98ee1e}" - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-a96c7b98-21ec-611e-8239-dea35c98ee1e a96c7b98-21ec-611e-8239-dea35c98ee1e}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.971Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.864652799 +0000 UTC m=+18.117703871 (1 nodes), 54.803395ms from start of poll, 56.060659ms from cluster start. Timeout at 2024-01-09 05:10:06.971145069 +0000 UTC m=+18.224196037, 106.492166ms after stability - testing.go:149: cluster: 2024-01-09T05:10:06.971Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.864652799 +0000 UTC m=+18.117703871 (1 nodes), 54.803395ms from start of poll, 56.060659ms from cluster start. Timeout at 2024-01-09 05:10:06.971145069 +0000 UTC m=+18.224196037, 106.492166ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.971Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:10:06.971Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.078Z [INFO] stable state for Follower reached at 2024-01-09 05:10:06.971274314 +0000 UTC m=+18.224325289 (6 nodes), 3.036µs from start of poll, 162.682077ms from cluster start. Timeout at 2024-01-09 05:10:07.078938838 +0000 UTC m=+18.331989960, 107.664671ms after stability - testing.go:149: cluster: 2024-01-09T05:10:07.078Z [INFO] stable state for Follower reached at 2024-01-09 05:10:06.971274314 +0000 UTC m=+18.224325289 (6 nodes), 3.036µs from start of poll, 162.682077ms from cluster start. Timeout at 2024-01-09 05:10:07.078938838 +0000 UTC m=+18.331989960, 107.664671ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.079Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:07.079Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.184Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.079085435 +0000 UTC m=+18.332136436 (1 nodes), 2.79µs from start of poll, 270.493224ms from cluster start. Timeout at 2024-01-09 05:10:07.184875429 +0000 UTC m=+18.437926459, 105.790023ms after stability - testing.go:149: cluster: 2024-01-09T05:10:07.184Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.079085435 +0000 UTC m=+18.332136436 (1 nodes), 2.79µs from start of poll, 270.493224ms from cluster start. Timeout at 2024-01-09 05:10:07.184875429 +0000 UTC m=+18.437926459, 105.790023ms after stability -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: entering candidate state: node="Node at c9b91bba-dc30-e04b-fb6e-30317aaae7f0 [Candidate]" term=3 - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: entering candidate state: node="Node at c9b91bba-dc30-e04b-fb6e-30317aaae7f0 [Candidate]" term=3 -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.185Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:07.185Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: election won: tally=4 - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: election won: tally=4 -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: entering leader state: leader="Node at c9b91bba-dc30-e04b-fb6e-30317aaae7f0 [Leader]" - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: entering leader state: leader="Node at c9b91bba-dc30-e04b-fb6e-30317aaae7f0 [Leader]" -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-78458f2b-7852-b1e6-0e47-696b8d2b327a - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-78458f2b-7852-b1e6-0e47-696b8d2b327a -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-7690a8da-ca4b-74d6-905a-84887782f70a - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-7690a8da-ca4b-74d6-905a-84887782f70a -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-a96c7b98-21ec-611e-8239-dea35c98ee1e - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-a96c7b98-21ec-611e-8239-dea35c98ee1e -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: entering follower state: follower="Node at e5774e4f-7d02-8923-9fa1-7d61c66aeee0 [Follower]" leader= - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: entering follower state: follower="Node at e5774e4f-7d02-8923-9fa1-7d61c66aeee0 [Follower]" leader= -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.191Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 e5774e4f-7d02-8923-9fa1-7d61c66aeee0}" - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.191Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 e5774e4f-7d02-8923-9fa1-7d61c66aeee0}" -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.186Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-7690a8da-ca4b-74d6-905a-84887782f70a 7690a8da-ca4b-74d6-905a-84887782f70a}" - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.186Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-7690a8da-ca4b-74d6-905a-84887782f70a 7690a8da-ca4b-74d6-905a-84887782f70a}" -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.191Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff 7e15329d-e3f8-73ab-e49c-21e6c404a4ff}" - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.191Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff 7e15329d-e3f8-73ab-e49c-21e6c404a4ff}" -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-78458f2b-7852-b1e6-0e47-696b8d2b327a 78458f2b-7852-b1e6-0e47-696b8d2b327a}" - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-78458f2b-7852-b1e6-0e47-696b8d2b327a 78458f2b-7852-b1e6-0e47-696b8d2b327a}" -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.186Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-78458f2b-7852-b1e6-0e47-696b8d2b327a 78458f2b-7852-b1e6-0e47-696b8d2b327a}" - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.186Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-78458f2b-7852-b1e6-0e47-696b8d2b327a 78458f2b-7852-b1e6-0e47-696b8d2b327a}" -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-7690a8da-ca4b-74d6-905a-84887782f70a 7690a8da-ca4b-74d6-905a-84887782f70a}" - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-7690a8da-ca4b-74d6-905a-84887782f70a 7690a8da-ca4b-74d6-905a-84887782f70a}" -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.186Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb b1cf2ae1-f267-abda-3212-ac286ecd9fdb}" - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.186Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb b1cf2ae1-f267-abda-3212-ac286ecd9fdb}" -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb b1cf2ae1-f267-abda-3212-ac286ecd9fdb}" - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb b1cf2ae1-f267-abda-3212-ac286ecd9fdb}" -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.191Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-a96c7b98-21ec-611e-8239-dea35c98ee1e a96c7b98-21ec-611e-8239-dea35c98ee1e}" - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.191Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-a96c7b98-21ec-611e-8239-dea35c98ee1e a96c7b98-21ec-611e-8239-dea35c98ee1e}" -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-a96c7b98-21ec-611e-8239-dea35c98ee1e a96c7b98-21ec-611e-8239-dea35c98ee1e}" - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-a96c7b98-21ec-611e-8239-dea35c98ee1e a96c7b98-21ec-611e-8239-dea35c98ee1e}" -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff 7e15329d-e3f8-73ab-e49c-21e6c404a4ff}" - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff 7e15329d-e3f8-73ab-e49c-21e6c404a4ff}" -testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 c9b91bba-dc30-e04b-fb6e-30317aaae7f0}" - testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 c9b91bba-dc30-e04b-fb6e-30317aaae7f0}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.295Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.186429062 +0000 UTC m=+18.439480108 (1 nodes), 1.040734ms from start of poll, 377.836896ms from cluster start. Timeout at 2024-01-09 05:10:07.295900801 +0000 UTC m=+18.548951868, 109.47176ms after stability - testing.go:149: cluster: 2024-01-09T05:10:07.295Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.186429062 +0000 UTC m=+18.439480108 (1 nodes), 1.040734ms from start of poll, 377.836896ms from cluster start. Timeout at 2024-01-09 05:10:07.295900801 +0000 UTC m=+18.548951868, 109.47176ms after stability -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-7690a8da-ca4b-74d6-905a-84887782f70a 7690a8da-ca4b-74d6-905a-84887782f70a}" - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-7690a8da-ca4b-74d6-905a-84887782f70a 7690a8da-ca4b-74d6-905a-84887782f70a}" -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb b1cf2ae1-f267-abda-3212-ac286ecd9fdb}" - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb b1cf2ae1-f267-abda-3212-ac286ecd9fdb}" -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 e5774e4f-7d02-8923-9fa1-7d61c66aeee0}" - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 e5774e4f-7d02-8923-9fa1-7d61c66aeee0}" -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-a96c7b98-21ec-611e-8239-dea35c98ee1e a96c7b98-21ec-611e-8239-dea35c98ee1e}" - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-a96c7b98-21ec-611e-8239-dea35c98ee1e a96c7b98-21ec-611e-8239-dea35c98ee1e}" -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff 7e15329d-e3f8-73ab-e49c-21e6c404a4ff}" - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff 7e15329d-e3f8-73ab-e49c-21e6c404a4ff}" -testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-78458f2b-7852-b1e6-0e47-696b8d2b327a 78458f2b-7852-b1e6-0e47-696b8d2b327a}" - testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-78458f2b-7852-b1e6-0e47-696b8d2b327a 78458f2b-7852-b1e6-0e47-696b8d2b327a}" +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.748Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b11175b4-c9eb-f41e-51a0-11096ad0cfea Address:b11175b4-c9eb-f41e-51a0-11096ad0cfea} {Suffrage:Voter ID:server-25a854f9-f140-ed8d-2d3a-161f77156e27 Address:25a854f9-f140-ed8d-2d3a-161f77156e27} {Suffrage:Voter ID:server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b Address:dc67c5a0-ce21-9e8b-2780-aa037bb68e8b} {Suffrage:Voter ID:server-12b32eb4-d484-8f08-39d8-de6dca092171 Address:12b32eb4-d484-8f08-39d8-de6dca092171} {Suffrage:Voter ID:server-625613d0-9b59-45be-eb90-2d86fe02a2ef Address:625613d0-9b59-45be-eb90-2d86fe02a2ef} {Suffrage:Voter ID:server-440806d4-301c-3044-e2a4-a9fd8537f71e Address:440806d4-301c-3044-e2a4-a9fd8537f71e} {Suffrage:Voter ID:server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 Address:f4651c05-e7f2-0754-efb2-4b34bbcd6345}]" + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.748Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b11175b4-c9eb-f41e-51a0-11096ad0cfea Address:b11175b4-c9eb-f41e-51a0-11096ad0cfea} {Suffrage:Voter ID:server-25a854f9-f140-ed8d-2d3a-161f77156e27 Address:25a854f9-f140-ed8d-2d3a-161f77156e27} {Suffrage:Voter ID:server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b Address:dc67c5a0-ce21-9e8b-2780-aa037bb68e8b} {Suffrage:Voter ID:server-12b32eb4-d484-8f08-39d8-de6dca092171 Address:12b32eb4-d484-8f08-39d8-de6dca092171} {Suffrage:Voter ID:server-625613d0-9b59-45be-eb90-2d86fe02a2ef Address:625613d0-9b59-45be-eb90-2d86fe02a2ef} {Suffrage:Voter ID:server-440806d4-301c-3044-e2a4-a9fd8537f71e Address:440806d4-301c-3044-e2a4-a9fd8537f71e} {Suffrage:Voter ID:server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 Address:f4651c05-e7f2-0754-efb2-4b34bbcd6345}]" +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.748Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: entering follower state: follower="Node at b11175b4-c9eb-f41e-51a0-11096ad0cfea [Follower]" leader= + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.748Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: entering follower state: follower="Node at b11175b4-c9eb-f41e-51a0-11096ad0cfea [Follower]" leader= +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:20.748Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b11175b4-c9eb-f41e-51a0-11096ad0cfea Address:b11175b4-c9eb-f41e-51a0-11096ad0cfea} {Suffrage:Voter ID:server-25a854f9-f140-ed8d-2d3a-161f77156e27 Address:25a854f9-f140-ed8d-2d3a-161f77156e27} {Suffrage:Voter ID:server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b Address:dc67c5a0-ce21-9e8b-2780-aa037bb68e8b} {Suffrage:Voter ID:server-12b32eb4-d484-8f08-39d8-de6dca092171 Address:12b32eb4-d484-8f08-39d8-de6dca092171} {Suffrage:Voter ID:server-625613d0-9b59-45be-eb90-2d86fe02a2ef Address:625613d0-9b59-45be-eb90-2d86fe02a2ef} {Suffrage:Voter ID:server-440806d4-301c-3044-e2a4-a9fd8537f71e Address:440806d4-301c-3044-e2a4-a9fd8537f71e} {Suffrage:Voter ID:server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 Address:f4651c05-e7f2-0754-efb2-4b34bbcd6345}]" + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:20.748Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b11175b4-c9eb-f41e-51a0-11096ad0cfea Address:b11175b4-c9eb-f41e-51a0-11096ad0cfea} {Suffrage:Voter ID:server-25a854f9-f140-ed8d-2d3a-161f77156e27 Address:25a854f9-f140-ed8d-2d3a-161f77156e27} {Suffrage:Voter ID:server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b Address:dc67c5a0-ce21-9e8b-2780-aa037bb68e8b} {Suffrage:Voter ID:server-12b32eb4-d484-8f08-39d8-de6dca092171 Address:12b32eb4-d484-8f08-39d8-de6dca092171} {Suffrage:Voter ID:server-625613d0-9b59-45be-eb90-2d86fe02a2ef Address:625613d0-9b59-45be-eb90-2d86fe02a2ef} {Suffrage:Voter ID:server-440806d4-301c-3044-e2a4-a9fd8537f71e Address:440806d4-301c-3044-e2a4-a9fd8537f71e} {Suffrage:Voter ID:server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 Address:f4651c05-e7f2-0754-efb2-4b34bbcd6345}]" +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:20.748Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: entering follower state: follower="Node at 25a854f9-f140-ed8d-2d3a-161f77156e27 [Follower]" leader= + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:20.748Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: entering follower state: follower="Node at 25a854f9-f140-ed8d-2d3a-161f77156e27 [Follower]" leader= +testLoggerAdapter verbose: server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b: 2025-02-10T11:34:20.748Z [INFO] server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b11175b4-c9eb-f41e-51a0-11096ad0cfea Address:b11175b4-c9eb-f41e-51a0-11096ad0cfea} {Suffrage:Voter ID:server-25a854f9-f140-ed8d-2d3a-161f77156e27 Address:25a854f9-f140-ed8d-2d3a-161f77156e27} {Suffrage:Voter ID:server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b Address:dc67c5a0-ce21-9e8b-2780-aa037bb68e8b} {Suffrage:Voter ID:server-12b32eb4-d484-8f08-39d8-de6dca092171 Address:12b32eb4-d484-8f08-39d8-de6dca092171} {Suffrage:Voter ID:server-625613d0-9b59-45be-eb90-2d86fe02a2ef Address:625613d0-9b59-45be-eb90-2d86fe02a2ef} {Suffrage:Voter ID:server-440806d4-301c-3044-e2a4-a9fd8537f71e Address:440806d4-301c-3044-e2a4-a9fd8537f71e} {Suffrage:Voter ID:server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 Address:f4651c05-e7f2-0754-efb2-4b34bbcd6345}]" + testing.go:149: server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b: 2025-02-10T11:34:20.748Z [INFO] server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b11175b4-c9eb-f41e-51a0-11096ad0cfea Address:b11175b4-c9eb-f41e-51a0-11096ad0cfea} {Suffrage:Voter ID:server-25a854f9-f140-ed8d-2d3a-161f77156e27 Address:25a854f9-f140-ed8d-2d3a-161f77156e27} {Suffrage:Voter ID:server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b Address:dc67c5a0-ce21-9e8b-2780-aa037bb68e8b} {Suffrage:Voter ID:server-12b32eb4-d484-8f08-39d8-de6dca092171 Address:12b32eb4-d484-8f08-39d8-de6dca092171} {Suffrage:Voter ID:server-625613d0-9b59-45be-eb90-2d86fe02a2ef Address:625613d0-9b59-45be-eb90-2d86fe02a2ef} {Suffrage:Voter ID:server-440806d4-301c-3044-e2a4-a9fd8537f71e Address:440806d4-301c-3044-e2a4-a9fd8537f71e} {Suffrage:Voter ID:server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 Address:f4651c05-e7f2-0754-efb2-4b34bbcd6345}]" +testLoggerAdapter verbose: server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b: 2025-02-10T11:34:20.749Z [INFO] server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b: entering follower state: follower="Node at dc67c5a0-ce21-9e8b-2780-aa037bb68e8b [Follower]" leader= + testing.go:149: server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b: 2025-02-10T11:34:20.749Z [INFO] server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b: entering follower state: follower="Node at dc67c5a0-ce21-9e8b-2780-aa037bb68e8b [Follower]" leader= +testLoggerAdapter verbose: server-12b32eb4-d484-8f08-39d8-de6dca092171: 2025-02-10T11:34:20.749Z [INFO] server-12b32eb4-d484-8f08-39d8-de6dca092171: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b11175b4-c9eb-f41e-51a0-11096ad0cfea Address:b11175b4-c9eb-f41e-51a0-11096ad0cfea} {Suffrage:Voter ID:server-25a854f9-f140-ed8d-2d3a-161f77156e27 Address:25a854f9-f140-ed8d-2d3a-161f77156e27} {Suffrage:Voter ID:server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b Address:dc67c5a0-ce21-9e8b-2780-aa037bb68e8b} {Suffrage:Voter ID:server-12b32eb4-d484-8f08-39d8-de6dca092171 Address:12b32eb4-d484-8f08-39d8-de6dca092171} {Suffrage:Voter ID:server-625613d0-9b59-45be-eb90-2d86fe02a2ef Address:625613d0-9b59-45be-eb90-2d86fe02a2ef} {Suffrage:Voter ID:server-440806d4-301c-3044-e2a4-a9fd8537f71e Address:440806d4-301c-3044-e2a4-a9fd8537f71e} {Suffrage:Voter ID:server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 Address:f4651c05-e7f2-0754-efb2-4b34bbcd6345}]" + testing.go:149: server-12b32eb4-d484-8f08-39d8-de6dca092171: 2025-02-10T11:34:20.749Z [INFO] server-12b32eb4-d484-8f08-39d8-de6dca092171: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b11175b4-c9eb-f41e-51a0-11096ad0cfea Address:b11175b4-c9eb-f41e-51a0-11096ad0cfea} {Suffrage:Voter ID:server-25a854f9-f140-ed8d-2d3a-161f77156e27 Address:25a854f9-f140-ed8d-2d3a-161f77156e27} {Suffrage:Voter ID:server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b Address:dc67c5a0-ce21-9e8b-2780-aa037bb68e8b} {Suffrage:Voter ID:server-12b32eb4-d484-8f08-39d8-de6dca092171 Address:12b32eb4-d484-8f08-39d8-de6dca092171} {Suffrage:Voter ID:server-625613d0-9b59-45be-eb90-2d86fe02a2ef Address:625613d0-9b59-45be-eb90-2d86fe02a2ef} {Suffrage:Voter ID:server-440806d4-301c-3044-e2a4-a9fd8537f71e Address:440806d4-301c-3044-e2a4-a9fd8537f71e} {Suffrage:Voter ID:server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 Address:f4651c05-e7f2-0754-efb2-4b34bbcd6345}]" +testLoggerAdapter verbose: server-12b32eb4-d484-8f08-39d8-de6dca092171: 2025-02-10T11:34:20.749Z [INFO] server-12b32eb4-d484-8f08-39d8-de6dca092171: entering follower state: follower="Node at 12b32eb4-d484-8f08-39d8-de6dca092171 [Follower]" leader= + testing.go:149: server-12b32eb4-d484-8f08-39d8-de6dca092171: 2025-02-10T11:34:20.749Z [INFO] server-12b32eb4-d484-8f08-39d8-de6dca092171: entering follower state: follower="Node at 12b32eb4-d484-8f08-39d8-de6dca092171 [Follower]" leader= +testLoggerAdapter verbose: server-625613d0-9b59-45be-eb90-2d86fe02a2ef: 2025-02-10T11:34:20.749Z [INFO] server-625613d0-9b59-45be-eb90-2d86fe02a2ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b11175b4-c9eb-f41e-51a0-11096ad0cfea Address:b11175b4-c9eb-f41e-51a0-11096ad0cfea} {Suffrage:Voter ID:server-25a854f9-f140-ed8d-2d3a-161f77156e27 Address:25a854f9-f140-ed8d-2d3a-161f77156e27} {Suffrage:Voter ID:server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b Address:dc67c5a0-ce21-9e8b-2780-aa037bb68e8b} {Suffrage:Voter ID:server-12b32eb4-d484-8f08-39d8-de6dca092171 Address:12b32eb4-d484-8f08-39d8-de6dca092171} {Suffrage:Voter ID:server-625613d0-9b59-45be-eb90-2d86fe02a2ef Address:625613d0-9b59-45be-eb90-2d86fe02a2ef} {Suffrage:Voter ID:server-440806d4-301c-3044-e2a4-a9fd8537f71e Address:440806d4-301c-3044-e2a4-a9fd8537f71e} {Suffrage:Voter ID:server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 Address:f4651c05-e7f2-0754-efb2-4b34bbcd6345}]" + testing.go:149: server-625613d0-9b59-45be-eb90-2d86fe02a2ef: 2025-02-10T11:34:20.749Z [INFO] server-625613d0-9b59-45be-eb90-2d86fe02a2ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b11175b4-c9eb-f41e-51a0-11096ad0cfea Address:b11175b4-c9eb-f41e-51a0-11096ad0cfea} {Suffrage:Voter ID:server-25a854f9-f140-ed8d-2d3a-161f77156e27 Address:25a854f9-f140-ed8d-2d3a-161f77156e27} {Suffrage:Voter ID:server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b Address:dc67c5a0-ce21-9e8b-2780-aa037bb68e8b} {Suffrage:Voter ID:server-12b32eb4-d484-8f08-39d8-de6dca092171 Address:12b32eb4-d484-8f08-39d8-de6dca092171} {Suffrage:Voter ID:server-625613d0-9b59-45be-eb90-2d86fe02a2ef Address:625613d0-9b59-45be-eb90-2d86fe02a2ef} {Suffrage:Voter ID:server-440806d4-301c-3044-e2a4-a9fd8537f71e Address:440806d4-301c-3044-e2a4-a9fd8537f71e} {Suffrage:Voter ID:server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 Address:f4651c05-e7f2-0754-efb2-4b34bbcd6345}]" +testLoggerAdapter verbose: server-625613d0-9b59-45be-eb90-2d86fe02a2ef: 2025-02-10T11:34:20.749Z [INFO] server-625613d0-9b59-45be-eb90-2d86fe02a2ef: entering follower state: follower="Node at 625613d0-9b59-45be-eb90-2d86fe02a2ef [Follower]" leader= + testing.go:149: server-625613d0-9b59-45be-eb90-2d86fe02a2ef: 2025-02-10T11:34:20.749Z [INFO] server-625613d0-9b59-45be-eb90-2d86fe02a2ef: entering follower state: follower="Node at 625613d0-9b59-45be-eb90-2d86fe02a2ef [Follower]" leader= +testLoggerAdapter verbose: server-440806d4-301c-3044-e2a4-a9fd8537f71e: 2025-02-10T11:34:20.749Z [INFO] server-440806d4-301c-3044-e2a4-a9fd8537f71e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b11175b4-c9eb-f41e-51a0-11096ad0cfea Address:b11175b4-c9eb-f41e-51a0-11096ad0cfea} {Suffrage:Voter ID:server-25a854f9-f140-ed8d-2d3a-161f77156e27 Address:25a854f9-f140-ed8d-2d3a-161f77156e27} {Suffrage:Voter ID:server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b Address:dc67c5a0-ce21-9e8b-2780-aa037bb68e8b} {Suffrage:Voter ID:server-12b32eb4-d484-8f08-39d8-de6dca092171 Address:12b32eb4-d484-8f08-39d8-de6dca092171} {Suffrage:Voter ID:server-625613d0-9b59-45be-eb90-2d86fe02a2ef Address:625613d0-9b59-45be-eb90-2d86fe02a2ef} {Suffrage:Voter ID:server-440806d4-301c-3044-e2a4-a9fd8537f71e Address:440806d4-301c-3044-e2a4-a9fd8537f71e} {Suffrage:Voter ID:server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 Address:f4651c05-e7f2-0754-efb2-4b34bbcd6345}]" + testing.go:149: server-440806d4-301c-3044-e2a4-a9fd8537f71e: 2025-02-10T11:34:20.749Z [INFO] server-440806d4-301c-3044-e2a4-a9fd8537f71e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b11175b4-c9eb-f41e-51a0-11096ad0cfea Address:b11175b4-c9eb-f41e-51a0-11096ad0cfea} {Suffrage:Voter ID:server-25a854f9-f140-ed8d-2d3a-161f77156e27 Address:25a854f9-f140-ed8d-2d3a-161f77156e27} {Suffrage:Voter ID:server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b Address:dc67c5a0-ce21-9e8b-2780-aa037bb68e8b} {Suffrage:Voter ID:server-12b32eb4-d484-8f08-39d8-de6dca092171 Address:12b32eb4-d484-8f08-39d8-de6dca092171} {Suffrage:Voter ID:server-625613d0-9b59-45be-eb90-2d86fe02a2ef Address:625613d0-9b59-45be-eb90-2d86fe02a2ef} {Suffrage:Voter ID:server-440806d4-301c-3044-e2a4-a9fd8537f71e Address:440806d4-301c-3044-e2a4-a9fd8537f71e} {Suffrage:Voter ID:server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 Address:f4651c05-e7f2-0754-efb2-4b34bbcd6345}]" +testLoggerAdapter verbose: server-440806d4-301c-3044-e2a4-a9fd8537f71e: 2025-02-10T11:34:20.749Z [INFO] server-440806d4-301c-3044-e2a4-a9fd8537f71e: entering follower state: follower="Node at 440806d4-301c-3044-e2a4-a9fd8537f71e [Follower]" leader= + testing.go:149: server-440806d4-301c-3044-e2a4-a9fd8537f71e: 2025-02-10T11:34:20.749Z [INFO] server-440806d4-301c-3044-e2a4-a9fd8537f71e: entering follower state: follower="Node at 440806d4-301c-3044-e2a4-a9fd8537f71e [Follower]" leader= +testLoggerAdapter verbose: server-f4651c05-e7f2-0754-efb2-4b34bbcd6345: 2025-02-10T11:34:20.749Z [INFO] server-f4651c05-e7f2-0754-efb2-4b34bbcd6345: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b11175b4-c9eb-f41e-51a0-11096ad0cfea Address:b11175b4-c9eb-f41e-51a0-11096ad0cfea} {Suffrage:Voter ID:server-25a854f9-f140-ed8d-2d3a-161f77156e27 Address:25a854f9-f140-ed8d-2d3a-161f77156e27} {Suffrage:Voter ID:server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b Address:dc67c5a0-ce21-9e8b-2780-aa037bb68e8b} {Suffrage:Voter ID:server-12b32eb4-d484-8f08-39d8-de6dca092171 Address:12b32eb4-d484-8f08-39d8-de6dca092171} {Suffrage:Voter ID:server-625613d0-9b59-45be-eb90-2d86fe02a2ef Address:625613d0-9b59-45be-eb90-2d86fe02a2ef} {Suffrage:Voter ID:server-440806d4-301c-3044-e2a4-a9fd8537f71e Address:440806d4-301c-3044-e2a4-a9fd8537f71e} {Suffrage:Voter ID:server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 Address:f4651c05-e7f2-0754-efb2-4b34bbcd6345}]" + testing.go:149: server-f4651c05-e7f2-0754-efb2-4b34bbcd6345: 2025-02-10T11:34:20.749Z [INFO] server-f4651c05-e7f2-0754-efb2-4b34bbcd6345: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b11175b4-c9eb-f41e-51a0-11096ad0cfea Address:b11175b4-c9eb-f41e-51a0-11096ad0cfea} {Suffrage:Voter ID:server-25a854f9-f140-ed8d-2d3a-161f77156e27 Address:25a854f9-f140-ed8d-2d3a-161f77156e27} {Suffrage:Voter ID:server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b Address:dc67c5a0-ce21-9e8b-2780-aa037bb68e8b} {Suffrage:Voter ID:server-12b32eb4-d484-8f08-39d8-de6dca092171 Address:12b32eb4-d484-8f08-39d8-de6dca092171} {Suffrage:Voter ID:server-625613d0-9b59-45be-eb90-2d86fe02a2ef Address:625613d0-9b59-45be-eb90-2d86fe02a2ef} {Suffrage:Voter ID:server-440806d4-301c-3044-e2a4-a9fd8537f71e Address:440806d4-301c-3044-e2a4-a9fd8537f71e} {Suffrage:Voter ID:server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 Address:f4651c05-e7f2-0754-efb2-4b34bbcd6345}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:20.749Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:20.749Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-f4651c05-e7f2-0754-efb2-4b34bbcd6345: 2025-02-10T11:34:20.749Z [INFO] server-f4651c05-e7f2-0754-efb2-4b34bbcd6345: entering follower state: follower="Node at f4651c05-e7f2-0754-efb2-4b34bbcd6345 [Follower]" leader= + testing.go:149: server-f4651c05-e7f2-0754-efb2-4b34bbcd6345: 2025-02-10T11:34:20.749Z [INFO] server-f4651c05-e7f2-0754-efb2-4b34bbcd6345: entering follower state: follower="Node at f4651c05-e7f2-0754-efb2-4b34bbcd6345 [Follower]" leader= +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [WARN] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [WARN] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: entering candidate state: node="Node at b11175b4-c9eb-f41e-51a0-11096ad0cfea [Candidate]" term=2 + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: entering candidate state: node="Node at b11175b4-c9eb-f41e-51a0-11096ad0cfea [Candidate]" term=2 +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: election won: tally=4 + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: election won: tally=4 +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: entering leader state: leader="Node at b11175b4-c9eb-f41e-51a0-11096ad0cfea [Leader]" + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: entering leader state: leader="Node at b11175b4-c9eb-f41e-51a0-11096ad0cfea [Leader]" +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: added peer, starting replication: peer=server-25a854f9-f140-ed8d-2d3a-161f77156e27 + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: added peer, starting replication: peer=server-25a854f9-f140-ed8d-2d3a-161f77156e27 +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: added peer, starting replication: peer=server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: added peer, starting replication: peer=server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: added peer, starting replication: peer=server-12b32eb4-d484-8f08-39d8-de6dca092171 + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: added peer, starting replication: peer=server-12b32eb4-d484-8f08-39d8-de6dca092171 +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: added peer, starting replication: peer=server-625613d0-9b59-45be-eb90-2d86fe02a2ef + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: added peer, starting replication: peer=server-625613d0-9b59-45be-eb90-2d86fe02a2ef +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: pipelining replication: peer="{Voter server-25a854f9-f140-ed8d-2d3a-161f77156e27 25a854f9-f140-ed8d-2d3a-161f77156e27}" + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: pipelining replication: peer="{Voter server-25a854f9-f140-ed8d-2d3a-161f77156e27 25a854f9-f140-ed8d-2d3a-161f77156e27}" +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: added peer, starting replication: peer=server-440806d4-301c-3044-e2a4-a9fd8537f71e + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: added peer, starting replication: peer=server-440806d4-301c-3044-e2a4-a9fd8537f71e +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.811Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: added peer, starting replication: peer=server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.811Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: added peer, starting replication: peer=server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.811Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: pipelining replication: peer="{Voter server-440806d4-301c-3044-e2a4-a9fd8537f71e 440806d4-301c-3044-e2a4-a9fd8537f71e}" + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.811Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: pipelining replication: peer="{Voter server-440806d4-301c-3044-e2a4-a9fd8537f71e 440806d4-301c-3044-e2a4-a9fd8537f71e}" +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: pipelining replication: peer="{Voter server-625613d0-9b59-45be-eb90-2d86fe02a2ef 625613d0-9b59-45be-eb90-2d86fe02a2ef}" + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: pipelining replication: peer="{Voter server-625613d0-9b59-45be-eb90-2d86fe02a2ef 625613d0-9b59-45be-eb90-2d86fe02a2ef}" +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: pipelining replication: peer="{Voter server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b dc67c5a0-ce21-9e8b-2780-aa037bb68e8b}" + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: pipelining replication: peer="{Voter server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b dc67c5a0-ce21-9e8b-2780-aa037bb68e8b}" +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: pipelining replication: peer="{Voter server-12b32eb4-d484-8f08-39d8-de6dca092171 12b32eb4-d484-8f08-39d8-de6dca092171}" + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.810Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: pipelining replication: peer="{Voter server-12b32eb4-d484-8f08-39d8-de6dca092171 12b32eb4-d484-8f08-39d8-de6dca092171}" +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.811Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: pipelining replication: peer="{Voter server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 f4651c05-e7f2-0754-efb2-4b34bbcd6345}" + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:20.811Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: pipelining replication: peer="{Voter server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 f4651c05-e7f2-0754-efb2-4b34bbcd6345}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:20.915Z [INFO] stable state for Leader reached at 2025-02-10 11:34:20.810827432 +0000 UTC m=+17.179731733 (1 nodes), 61.050514ms from start of poll, 62.320167ms from cluster start. Timeout at 2025-02-10 11:34:20.915926001 +0000 UTC m=+17.284830297, 105.098564ms after stability + testing.go:149: cluster: 2025-02-10T11:34:20.915Z [INFO] stable state for Leader reached at 2025-02-10 11:34:20.810827432 +0000 UTC m=+17.179731733 (1 nodes), 61.050514ms from start of poll, 62.320167ms from cluster start. Timeout at 2025-02-10 11:34:20.915926001 +0000 UTC m=+17.284830297, 105.098564ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:20.915Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:20.915Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:21.021Z [INFO] stable state for Follower reached at 2025-02-10 11:34:20.91598905 +0000 UTC m=+17.284893351 (6 nodes), 1.13µs from start of poll, 167.481785ms from cluster start. Timeout at 2025-02-10 11:34:21.021050781 +0000 UTC m=+17.389955113, 105.061762ms after stability + testing.go:149: cluster: 2025-02-10T11:34:21.021Z [INFO] stable state for Follower reached at 2025-02-10 11:34:20.91598905 +0000 UTC m=+17.284893351 (6 nodes), 1.13µs from start of poll, 167.481785ms from cluster start. Timeout at 2025-02-10 11:34:21.021050781 +0000 UTC m=+17.389955113, 105.061762ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:21.021Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:21.021Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:21.126Z [INFO] stable state for Leader reached at 2025-02-10 11:34:21.021108931 +0000 UTC m=+17.390013232 (1 nodes), 504ns from start of poll, 272.601666ms from cluster start. Timeout at 2025-02-10 11:34:21.126184058 +0000 UTC m=+17.495088354, 105.075122ms after stability + testing.go:149: cluster: 2025-02-10T11:34:21.126Z [INFO] stable state for Leader reached at 2025-02-10 11:34:21.021108931 +0000 UTC m=+17.390013232 (1 nodes), 504ns from start of poll, 272.601666ms from cluster start. Timeout at 2025-02-10 11:34:21.126184058 +0000 UTC m=+17.495088354, 105.075122ms after stability +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: entering candidate state: node="Node at 25a854f9-f140-ed8d-2d3a-161f77156e27 [Candidate]" term=3 + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: entering candidate state: node="Node at 25a854f9-f140-ed8d-2d3a-161f77156e27 [Candidate]" term=3 +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:21.126Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: entering follower state: follower="Node at b11175b4-c9eb-f41e-51a0-11096ad0cfea [Follower]" leader= +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: election won: tally=4 + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:21.126Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: entering follower state: follower="Node at b11175b4-c9eb-f41e-51a0-11096ad0cfea [Follower]" leader= + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: election won: tally=4 +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:21.126Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: entering leader state: leader="Node at 25a854f9-f140-ed8d-2d3a-161f77156e27 [Leader]" +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:21.126Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: aborting pipeline replication: peer="{Voter server-12b32eb4-d484-8f08-39d8-de6dca092171 12b32eb4-d484-8f08-39d8-de6dca092171}" + testing.go:149: cluster: 2025-02-10T11:34:21.126Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: entering leader state: leader="Node at 25a854f9-f140-ed8d-2d3a-161f77156e27 [Leader]" +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: added peer, starting replication: peer=server-b11175b4-c9eb-f41e-51a0-11096ad0cfea + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: added peer, starting replication: peer=server-b11175b4-c9eb-f41e-51a0-11096ad0cfea +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: added peer, starting replication: peer=server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: added peer, starting replication: peer=server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: added peer, starting replication: peer=server-12b32eb4-d484-8f08-39d8-de6dca092171 + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: added peer, starting replication: peer=server-12b32eb4-d484-8f08-39d8-de6dca092171 +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: added peer, starting replication: peer=server-625613d0-9b59-45be-eb90-2d86fe02a2ef + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:21.126Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: aborting pipeline replication: peer="{Voter server-12b32eb4-d484-8f08-39d8-de6dca092171 12b32eb4-d484-8f08-39d8-de6dca092171}" +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:21.126Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: aborting pipeline replication: peer="{Voter server-625613d0-9b59-45be-eb90-2d86fe02a2ef 625613d0-9b59-45be-eb90-2d86fe02a2ef}" + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: added peer, starting replication: peer=server-625613d0-9b59-45be-eb90-2d86fe02a2ef +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: added peer, starting replication: peer=server-440806d4-301c-3044-e2a4-a9fd8537f71e + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: added peer, starting replication: peer=server-440806d4-301c-3044-e2a4-a9fd8537f71e +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: added peer, starting replication: peer=server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: added peer, starting replication: peer=server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: pipelining replication: peer="{Voter server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 f4651c05-e7f2-0754-efb2-4b34bbcd6345}" + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: pipelining replication: peer="{Voter server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 f4651c05-e7f2-0754-efb2-4b34bbcd6345}" + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:21.126Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: aborting pipeline replication: peer="{Voter server-625613d0-9b59-45be-eb90-2d86fe02a2ef 625613d0-9b59-45be-eb90-2d86fe02a2ef}" +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: pipelining replication: peer="{Voter server-b11175b4-c9eb-f41e-51a0-11096ad0cfea b11175b4-c9eb-f41e-51a0-11096ad0cfea}" + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: pipelining replication: peer="{Voter server-b11175b4-c9eb-f41e-51a0-11096ad0cfea b11175b4-c9eb-f41e-51a0-11096ad0cfea}" +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: pipelining replication: peer="{Voter server-625613d0-9b59-45be-eb90-2d86fe02a2ef 625613d0-9b59-45be-eb90-2d86fe02a2ef}" + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: pipelining replication: peer="{Voter server-625613d0-9b59-45be-eb90-2d86fe02a2ef 625613d0-9b59-45be-eb90-2d86fe02a2ef}" +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:21.126Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: aborting pipeline replication: peer="{Voter server-440806d4-301c-3044-e2a4-a9fd8537f71e 440806d4-301c-3044-e2a4-a9fd8537f71e}" + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:21.126Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: aborting pipeline replication: peer="{Voter server-440806d4-301c-3044-e2a4-a9fd8537f71e 440806d4-301c-3044-e2a4-a9fd8537f71e}" +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: pipelining replication: peer="{Voter server-12b32eb4-d484-8f08-39d8-de6dca092171 12b32eb4-d484-8f08-39d8-de6dca092171}" + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: pipelining replication: peer="{Voter server-12b32eb4-d484-8f08-39d8-de6dca092171 12b32eb4-d484-8f08-39d8-de6dca092171}" +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:21.126Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: aborting pipeline replication: peer="{Voter server-25a854f9-f140-ed8d-2d3a-161f77156e27 25a854f9-f140-ed8d-2d3a-161f77156e27}" + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:21.126Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: aborting pipeline replication: peer="{Voter server-25a854f9-f140-ed8d-2d3a-161f77156e27 25a854f9-f140-ed8d-2d3a-161f77156e27}" +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: pipelining replication: peer="{Voter server-440806d4-301c-3044-e2a4-a9fd8537f71e 440806d4-301c-3044-e2a4-a9fd8537f71e}" +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:21.126Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: aborting pipeline replication: peer="{Voter server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 f4651c05-e7f2-0754-efb2-4b34bbcd6345}" + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: pipelining replication: peer="{Voter server-440806d4-301c-3044-e2a4-a9fd8537f71e 440806d4-301c-3044-e2a4-a9fd8537f71e}" + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:21.126Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: aborting pipeline replication: peer="{Voter server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 f4651c05-e7f2-0754-efb2-4b34bbcd6345}" +testLoggerAdapter verbose: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:21.126Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: aborting pipeline replication: peer="{Voter server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b dc67c5a0-ce21-9e8b-2780-aa037bb68e8b}" +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: pipelining replication: peer="{Voter server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b dc67c5a0-ce21-9e8b-2780-aa037bb68e8b}" + testing.go:149: server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: 2025-02-10T11:34:21.126Z [INFO] server-b11175b4-c9eb-f41e-51a0-11096ad0cfea: aborting pipeline replication: peer="{Voter server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b dc67c5a0-ce21-9e8b-2780-aa037bb68e8b}" + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.126Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: pipelining replication: peer="{Voter server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b dc67c5a0-ce21-9e8b-2780-aa037bb68e8b}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:21.231Z [INFO] stable state for Leader reached at 2025-02-10 11:34:21.126416149 +0000 UTC m=+17.495320448 (1 nodes), 26.702µs from start of poll, 377.908882ms from cluster start. Timeout at 2025-02-10 11:34:21.231453139 +0000 UTC m=+17.600357435, 105.036987ms after stability + testing.go:149: cluster: 2025-02-10T11:34:21.231Z [INFO] stable state for Leader reached at 2025-02-10 11:34:21.126416149 +0000 UTC m=+17.495320448 (1 nodes), 26.702µs from start of poll, 377.908882ms from cluster start. Timeout at 2025-02-10 11:34:21.231453139 +0000 UTC m=+17.600357435, 105.036987ms after stability +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.231Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: aborting pipeline replication: peer="{Voter server-12b32eb4-d484-8f08-39d8-de6dca092171 12b32eb4-d484-8f08-39d8-de6dca092171}" + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.231Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: aborting pipeline replication: peer="{Voter server-12b32eb4-d484-8f08-39d8-de6dca092171 12b32eb4-d484-8f08-39d8-de6dca092171}" +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.231Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: aborting pipeline replication: peer="{Voter server-440806d4-301c-3044-e2a4-a9fd8537f71e 440806d4-301c-3044-e2a4-a9fd8537f71e}" + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.231Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: aborting pipeline replication: peer="{Voter server-440806d4-301c-3044-e2a4-a9fd8537f71e 440806d4-301c-3044-e2a4-a9fd8537f71e}" +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.231Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: aborting pipeline replication: peer="{Voter server-625613d0-9b59-45be-eb90-2d86fe02a2ef 625613d0-9b59-45be-eb90-2d86fe02a2ef}" + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.231Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: aborting pipeline replication: peer="{Voter server-625613d0-9b59-45be-eb90-2d86fe02a2ef 625613d0-9b59-45be-eb90-2d86fe02a2ef}" +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.231Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: aborting pipeline replication: peer="{Voter server-b11175b4-c9eb-f41e-51a0-11096ad0cfea b11175b4-c9eb-f41e-51a0-11096ad0cfea}" + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.231Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: aborting pipeline replication: peer="{Voter server-b11175b4-c9eb-f41e-51a0-11096ad0cfea b11175b4-c9eb-f41e-51a0-11096ad0cfea}" +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.231Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: aborting pipeline replication: peer="{Voter server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 f4651c05-e7f2-0754-efb2-4b34bbcd6345}" + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.231Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: aborting pipeline replication: peer="{Voter server-f4651c05-e7f2-0754-efb2-4b34bbcd6345 f4651c05-e7f2-0754-efb2-4b34bbcd6345}" +testLoggerAdapter verbose: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.231Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: aborting pipeline replication: peer="{Voter server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b dc67c5a0-ce21-9e8b-2780-aa037bb68e8b}" + testing.go:149: server-25a854f9-f140-ed8d-2d3a-161f77156e27: 2025-02-10T11:34:21.231Z [INFO] server-25a854f9-f140-ed8d-2d3a-161f77156e27: aborting pipeline replication: peer="{Voter server-dc67c5a0-ce21-9e8b-2780-aa037bb68e8b dc67c5a0-ce21-9e8b-2780-aa037bb68e8b}" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.49s) === RUN TestRaft_LeadershipTransferToInvalidID -testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.298Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5534b285-4693-2480-b4c7-7d20e1baeaea Address:5534b285-4693-2480-b4c7-7d20e1baeaea} {Suffrage:Voter ID:server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab Address:c7e6cb42-46e1-712e-ad2f-90b187bc6aab} {Suffrage:Voter ID:server-e54fb9e6-99fd-c048-711f-571f96b294bc Address:e54fb9e6-99fd-c048-711f-571f96b294bc}]" - testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.298Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5534b285-4693-2480-b4c7-7d20e1baeaea Address:5534b285-4693-2480-b4c7-7d20e1baeaea} {Suffrage:Voter ID:server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab Address:c7e6cb42-46e1-712e-ad2f-90b187bc6aab} {Suffrage:Voter ID:server-e54fb9e6-99fd-c048-711f-571f96b294bc Address:e54fb9e6-99fd-c048-711f-571f96b294bc}]" -testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.299Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: entering follower state: follower="Node at 5534b285-4693-2480-b4c7-7d20e1baeaea [Follower]" leader= - testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.299Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: entering follower state: follower="Node at 5534b285-4693-2480-b4c7-7d20e1baeaea [Follower]" leader= -testLoggerAdapter verbose: server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab: 2024-01-09T05:10:07.298Z [INFO] server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5534b285-4693-2480-b4c7-7d20e1baeaea Address:5534b285-4693-2480-b4c7-7d20e1baeaea} {Suffrage:Voter ID:server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab Address:c7e6cb42-46e1-712e-ad2f-90b187bc6aab} {Suffrage:Voter ID:server-e54fb9e6-99fd-c048-711f-571f96b294bc Address:e54fb9e6-99fd-c048-711f-571f96b294bc}]" - testing.go:149: server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab: 2024-01-09T05:10:07.298Z [INFO] server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5534b285-4693-2480-b4c7-7d20e1baeaea Address:5534b285-4693-2480-b4c7-7d20e1baeaea} {Suffrage:Voter ID:server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab Address:c7e6cb42-46e1-712e-ad2f-90b187bc6aab} {Suffrage:Voter ID:server-e54fb9e6-99fd-c048-711f-571f96b294bc Address:e54fb9e6-99fd-c048-711f-571f96b294bc}]" -testLoggerAdapter verbose: server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab: 2024-01-09T05:10:07.299Z [INFO] server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab: entering follower state: follower="Node at c7e6cb42-46e1-712e-ad2f-90b187bc6aab [Follower]" leader= - testing.go:149: server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab: 2024-01-09T05:10:07.299Z [INFO] server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab: entering follower state: follower="Node at c7e6cb42-46e1-712e-ad2f-90b187bc6aab [Follower]" leader= -testLoggerAdapter verbose: server-e54fb9e6-99fd-c048-711f-571f96b294bc: 2024-01-09T05:10:07.299Z [INFO] server-e54fb9e6-99fd-c048-711f-571f96b294bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5534b285-4693-2480-b4c7-7d20e1baeaea Address:5534b285-4693-2480-b4c7-7d20e1baeaea} {Suffrage:Voter ID:server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab Address:c7e6cb42-46e1-712e-ad2f-90b187bc6aab} {Suffrage:Voter ID:server-e54fb9e6-99fd-c048-711f-571f96b294bc Address:e54fb9e6-99fd-c048-711f-571f96b294bc}]" - testing.go:149: server-e54fb9e6-99fd-c048-711f-571f96b294bc: 2024-01-09T05:10:07.299Z [INFO] server-e54fb9e6-99fd-c048-711f-571f96b294bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5534b285-4693-2480-b4c7-7d20e1baeaea Address:5534b285-4693-2480-b4c7-7d20e1baeaea} {Suffrage:Voter ID:server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab Address:c7e6cb42-46e1-712e-ad2f-90b187bc6aab} {Suffrage:Voter ID:server-e54fb9e6-99fd-c048-711f-571f96b294bc Address:e54fb9e6-99fd-c048-711f-571f96b294bc}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.299Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:07.299Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-e54fb9e6-99fd-c048-711f-571f96b294bc: 2024-01-09T05:10:07.299Z [INFO] server-e54fb9e6-99fd-c048-711f-571f96b294bc: entering follower state: follower="Node at e54fb9e6-99fd-c048-711f-571f96b294bc [Follower]" leader= - testing.go:149: server-e54fb9e6-99fd-c048-711f-571f96b294bc: 2024-01-09T05:10:07.299Z [INFO] server-e54fb9e6-99fd-c048-711f-571f96b294bc: entering follower state: follower="Node at e54fb9e6-99fd-c048-711f-571f96b294bc [Follower]" leader= -testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.351Z [WARN] server-5534b285-4693-2480-b4c7-7d20e1baeaea: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.351Z [WARN] server-5534b285-4693-2480-b4c7-7d20e1baeaea: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.351Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: entering candidate state: node="Node at 5534b285-4693-2480-b4c7-7d20e1baeaea [Candidate]" term=2 - testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.351Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: entering candidate state: node="Node at 5534b285-4693-2480-b4c7-7d20e1baeaea [Candidate]" term=2 -testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: election won: tally=2 - testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: election won: tally=2 -testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: entering leader state: leader="Node at 5534b285-4693-2480-b4c7-7d20e1baeaea [Leader]" - testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: entering leader state: leader="Node at 5534b285-4693-2480-b4c7-7d20e1baeaea [Leader]" -testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: added peer, starting replication: peer=server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab - testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: added peer, starting replication: peer=server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab -testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: added peer, starting replication: peer=server-e54fb9e6-99fd-c048-711f-571f96b294bc - testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: added peer, starting replication: peer=server-e54fb9e6-99fd-c048-711f-571f96b294bc -testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: pipelining replication: peer="{Voter server-e54fb9e6-99fd-c048-711f-571f96b294bc e54fb9e6-99fd-c048-711f-571f96b294bc}" - testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: pipelining replication: peer="{Voter server-e54fb9e6-99fd-c048-711f-571f96b294bc e54fb9e6-99fd-c048-711f-571f96b294bc}" -testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: pipelining replication: peer="{Voter server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab c7e6cb42-46e1-712e-ad2f-90b187bc6aab}" - testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: pipelining replication: peer="{Voter server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab c7e6cb42-46e1-712e-ad2f-90b187bc6aab}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.457Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.352539423 +0000 UTC m=+18.605590523 (1 nodes), 52.947522ms from start of poll, 54.005282ms from cluster start. Timeout at 2024-01-09 05:10:07.457655023 +0000 UTC m=+18.710706104, 105.115581ms after stability - testing.go:149: cluster: 2024-01-09T05:10:07.457Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.352539423 +0000 UTC m=+18.605590523 (1 nodes), 52.947522ms from start of poll, 54.005282ms from cluster start. Timeout at 2024-01-09 05:10:07.457655023 +0000 UTC m=+18.710706104, 105.115581ms after stability -testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.457Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: aborting pipeline replication: peer="{Voter server-e54fb9e6-99fd-c048-711f-571f96b294bc e54fb9e6-99fd-c048-711f-571f96b294bc}" - testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.457Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: aborting pipeline replication: peer="{Voter server-e54fb9e6-99fd-c048-711f-571f96b294bc e54fb9e6-99fd-c048-711f-571f96b294bc}" -testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.458Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: aborting pipeline replication: peer="{Voter server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab c7e6cb42-46e1-712e-ad2f-90b187bc6aab}" - testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.458Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: aborting pipeline replication: peer="{Voter server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab c7e6cb42-46e1-712e-ad2f-90b187bc6aab}" +testLoggerAdapter verbose: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.234Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6 Address:75a215e4-4b2e-9bbf-83b7-221cc9fa82c6} {Suffrage:Voter ID:server-075183af-98dd-7927-699f-53ba4d3eeb7a Address:075183af-98dd-7927-699f-53ba4d3eeb7a} {Suffrage:Voter ID:server-73ce0480-bae7-fa62-3215-7c9d150b1cdf Address:73ce0480-bae7-fa62-3215-7c9d150b1cdf}]" + testing.go:149: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.234Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6 Address:75a215e4-4b2e-9bbf-83b7-221cc9fa82c6} {Suffrage:Voter ID:server-075183af-98dd-7927-699f-53ba4d3eeb7a Address:075183af-98dd-7927-699f-53ba4d3eeb7a} {Suffrage:Voter ID:server-73ce0480-bae7-fa62-3215-7c9d150b1cdf Address:73ce0480-bae7-fa62-3215-7c9d150b1cdf}]" +testLoggerAdapter verbose: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.234Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: entering follower state: follower="Node at 75a215e4-4b2e-9bbf-83b7-221cc9fa82c6 [Follower]" leader= + testing.go:149: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.234Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: entering follower state: follower="Node at 75a215e4-4b2e-9bbf-83b7-221cc9fa82c6 [Follower]" leader= +testLoggerAdapter verbose: server-075183af-98dd-7927-699f-53ba4d3eeb7a: 2025-02-10T11:34:21.234Z [INFO] server-075183af-98dd-7927-699f-53ba4d3eeb7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6 Address:75a215e4-4b2e-9bbf-83b7-221cc9fa82c6} {Suffrage:Voter ID:server-075183af-98dd-7927-699f-53ba4d3eeb7a Address:075183af-98dd-7927-699f-53ba4d3eeb7a} {Suffrage:Voter ID:server-73ce0480-bae7-fa62-3215-7c9d150b1cdf Address:73ce0480-bae7-fa62-3215-7c9d150b1cdf}]" + testing.go:149: server-075183af-98dd-7927-699f-53ba4d3eeb7a: 2025-02-10T11:34:21.234Z [INFO] server-075183af-98dd-7927-699f-53ba4d3eeb7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6 Address:75a215e4-4b2e-9bbf-83b7-221cc9fa82c6} {Suffrage:Voter ID:server-075183af-98dd-7927-699f-53ba4d3eeb7a Address:075183af-98dd-7927-699f-53ba4d3eeb7a} {Suffrage:Voter ID:server-73ce0480-bae7-fa62-3215-7c9d150b1cdf Address:73ce0480-bae7-fa62-3215-7c9d150b1cdf}]" +testLoggerAdapter verbose: server-075183af-98dd-7927-699f-53ba4d3eeb7a: 2025-02-10T11:34:21.234Z [INFO] server-075183af-98dd-7927-699f-53ba4d3eeb7a: entering follower state: follower="Node at 075183af-98dd-7927-699f-53ba4d3eeb7a [Follower]" leader= + testing.go:149: server-075183af-98dd-7927-699f-53ba4d3eeb7a: 2025-02-10T11:34:21.234Z [INFO] server-075183af-98dd-7927-699f-53ba4d3eeb7a: entering follower state: follower="Node at 075183af-98dd-7927-699f-53ba4d3eeb7a [Follower]" leader= +testLoggerAdapter verbose: server-73ce0480-bae7-fa62-3215-7c9d150b1cdf: 2025-02-10T11:34:21.234Z [INFO] server-73ce0480-bae7-fa62-3215-7c9d150b1cdf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6 Address:75a215e4-4b2e-9bbf-83b7-221cc9fa82c6} {Suffrage:Voter ID:server-075183af-98dd-7927-699f-53ba4d3eeb7a Address:075183af-98dd-7927-699f-53ba4d3eeb7a} {Suffrage:Voter ID:server-73ce0480-bae7-fa62-3215-7c9d150b1cdf Address:73ce0480-bae7-fa62-3215-7c9d150b1cdf}]" + testing.go:149: server-73ce0480-bae7-fa62-3215-7c9d150b1cdf: 2025-02-10T11:34:21.234Z [INFO] server-73ce0480-bae7-fa62-3215-7c9d150b1cdf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6 Address:75a215e4-4b2e-9bbf-83b7-221cc9fa82c6} {Suffrage:Voter ID:server-075183af-98dd-7927-699f-53ba4d3eeb7a Address:075183af-98dd-7927-699f-53ba4d3eeb7a} {Suffrage:Voter ID:server-73ce0480-bae7-fa62-3215-7c9d150b1cdf Address:73ce0480-bae7-fa62-3215-7c9d150b1cdf}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:21.235Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:21.235Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-73ce0480-bae7-fa62-3215-7c9d150b1cdf: 2025-02-10T11:34:21.235Z [INFO] server-73ce0480-bae7-fa62-3215-7c9d150b1cdf: entering follower state: follower="Node at 73ce0480-bae7-fa62-3215-7c9d150b1cdf [Follower]" leader= + testing.go:149: server-73ce0480-bae7-fa62-3215-7c9d150b1cdf: 2025-02-10T11:34:21.235Z [INFO] server-73ce0480-bae7-fa62-3215-7c9d150b1cdf: entering follower state: follower="Node at 73ce0480-bae7-fa62-3215-7c9d150b1cdf [Follower]" leader= +testLoggerAdapter verbose: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.290Z [WARN] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.290Z [WARN] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.290Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: entering candidate state: node="Node at 75a215e4-4b2e-9bbf-83b7-221cc9fa82c6 [Candidate]" term=2 + testing.go:149: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.290Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: entering candidate state: node="Node at 75a215e4-4b2e-9bbf-83b7-221cc9fa82c6 [Candidate]" term=2 +testLoggerAdapter verbose: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.291Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: election won: tally=2 + testing.go:149: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.291Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: election won: tally=2 +testLoggerAdapter verbose: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.291Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: entering leader state: leader="Node at 75a215e4-4b2e-9bbf-83b7-221cc9fa82c6 [Leader]" + testing.go:149: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.291Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: entering leader state: leader="Node at 75a215e4-4b2e-9bbf-83b7-221cc9fa82c6 [Leader]" +testLoggerAdapter verbose: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.291Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: added peer, starting replication: peer=server-075183af-98dd-7927-699f-53ba4d3eeb7a + testing.go:149: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.291Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: added peer, starting replication: peer=server-075183af-98dd-7927-699f-53ba4d3eeb7a +testLoggerAdapter verbose: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.291Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: added peer, starting replication: peer=server-73ce0480-bae7-fa62-3215-7c9d150b1cdf + testing.go:149: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.291Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: added peer, starting replication: peer=server-73ce0480-bae7-fa62-3215-7c9d150b1cdf +testLoggerAdapter verbose: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.291Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: pipelining replication: peer="{Voter server-73ce0480-bae7-fa62-3215-7c9d150b1cdf 73ce0480-bae7-fa62-3215-7c9d150b1cdf}" + testing.go:149: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.291Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: pipelining replication: peer="{Voter server-73ce0480-bae7-fa62-3215-7c9d150b1cdf 73ce0480-bae7-fa62-3215-7c9d150b1cdf}" +testLoggerAdapter verbose: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.291Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: pipelining replication: peer="{Voter server-075183af-98dd-7927-699f-53ba4d3eeb7a 075183af-98dd-7927-699f-53ba4d3eeb7a}" + testing.go:149: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.291Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: pipelining replication: peer="{Voter server-075183af-98dd-7927-699f-53ba4d3eeb7a 075183af-98dd-7927-699f-53ba4d3eeb7a}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:21.396Z [INFO] stable state for Leader reached at 2025-02-10 11:34:21.291156812 +0000 UTC m=+17.660061113 (1 nodes), 56.105269ms from start of poll, 56.482589ms from cluster start. Timeout at 2025-02-10 11:34:21.396275433 +0000 UTC m=+17.765179782, 105.118669ms after stability + testing.go:149: cluster: 2025-02-10T11:34:21.396Z [INFO] stable state for Leader reached at 2025-02-10 11:34:21.291156812 +0000 UTC m=+17.660061113 (1 nodes), 56.105269ms from start of poll, 56.482589ms from cluster start. Timeout at 2025-02-10 11:34:21.396275433 +0000 UTC m=+17.765179782, 105.118669ms after stability +testLoggerAdapter verbose: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.396Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: aborting pipeline replication: peer="{Voter server-73ce0480-bae7-fa62-3215-7c9d150b1cdf 73ce0480-bae7-fa62-3215-7c9d150b1cdf}" + testing.go:149: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.396Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: aborting pipeline replication: peer="{Voter server-73ce0480-bae7-fa62-3215-7c9d150b1cdf 73ce0480-bae7-fa62-3215-7c9d150b1cdf}" +testLoggerAdapter verbose: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.396Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: aborting pipeline replication: peer="{Voter server-075183af-98dd-7927-699f-53ba4d3eeb7a 075183af-98dd-7927-699f-53ba4d3eeb7a}" + testing.go:149: server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: 2025-02-10T11:34:21.396Z [INFO] server-75a215e4-4b2e-9bbf-83b7-221cc9fa82c6: aborting pipeline replication: peer="{Voter server-075183af-98dd-7927-699f-53ba4d3eeb7a 075183af-98dd-7927-699f-53ba4d3eeb7a}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.16s) === RUN TestRaft_LeadershipTransferToInvalidAddress -testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.459Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 Address:1f32a78b-3df2-ac0b-fcdd-5eebf03488a9} {Suffrage:Voter ID:server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e Address:05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e} {Suffrage:Voter ID:server-8e44a857-c59b-1833-4366-2673c370165a Address:8e44a857-c59b-1833-4366-2673c370165a}]" - testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.459Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 Address:1f32a78b-3df2-ac0b-fcdd-5eebf03488a9} {Suffrage:Voter ID:server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e Address:05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e} {Suffrage:Voter ID:server-8e44a857-c59b-1833-4366-2673c370165a Address:8e44a857-c59b-1833-4366-2673c370165a}]" -testLoggerAdapter verbose: server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e: 2024-01-09T05:10:07.459Z [INFO] server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 Address:1f32a78b-3df2-ac0b-fcdd-5eebf03488a9} {Suffrage:Voter ID:server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e Address:05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e} {Suffrage:Voter ID:server-8e44a857-c59b-1833-4366-2673c370165a Address:8e44a857-c59b-1833-4366-2673c370165a}]" - testing.go:149: server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e: 2024-01-09T05:10:07.459Z [INFO] server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 Address:1f32a78b-3df2-ac0b-fcdd-5eebf03488a9} {Suffrage:Voter ID:server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e Address:05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e} {Suffrage:Voter ID:server-8e44a857-c59b-1833-4366-2673c370165a Address:8e44a857-c59b-1833-4366-2673c370165a}]" -testLoggerAdapter verbose: server-8e44a857-c59b-1833-4366-2673c370165a: 2024-01-09T05:10:07.460Z [INFO] server-8e44a857-c59b-1833-4366-2673c370165a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 Address:1f32a78b-3df2-ac0b-fcdd-5eebf03488a9} {Suffrage:Voter ID:server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e Address:05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e} {Suffrage:Voter ID:server-8e44a857-c59b-1833-4366-2673c370165a Address:8e44a857-c59b-1833-4366-2673c370165a}]" - testing.go:149: server-8e44a857-c59b-1833-4366-2673c370165a: 2024-01-09T05:10:07.460Z [INFO] server-8e44a857-c59b-1833-4366-2673c370165a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 Address:1f32a78b-3df2-ac0b-fcdd-5eebf03488a9} {Suffrage:Voter ID:server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e Address:05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e} {Suffrage:Voter ID:server-8e44a857-c59b-1833-4366-2673c370165a Address:8e44a857-c59b-1833-4366-2673c370165a}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.460Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:10:07.460Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.460Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: entering follower state: follower="Node at 1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 [Follower]" leader= - testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.460Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: entering follower state: follower="Node at 1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 [Follower]" leader= -testLoggerAdapter verbose: server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e: 2024-01-09T05:10:07.460Z [INFO] server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e: entering follower state: follower="Node at 05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e [Follower]" leader= - testing.go:149: server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e: 2024-01-09T05:10:07.460Z [INFO] server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e: entering follower state: follower="Node at 05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e [Follower]" leader= -testLoggerAdapter verbose: server-8e44a857-c59b-1833-4366-2673c370165a: 2024-01-09T05:10:07.460Z [INFO] server-8e44a857-c59b-1833-4366-2673c370165a: entering follower state: follower="Node at 8e44a857-c59b-1833-4366-2673c370165a [Follower]" leader= - testing.go:149: server-8e44a857-c59b-1833-4366-2673c370165a: 2024-01-09T05:10:07.460Z [INFO] server-8e44a857-c59b-1833-4366-2673c370165a: entering follower state: follower="Node at 8e44a857-c59b-1833-4366-2673c370165a [Follower]" leader= -testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [WARN] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [WARN] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: entering candidate state: node="Node at 1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 [Candidate]" term=2 - testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: entering candidate state: node="Node at 1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 [Candidate]" term=2 -testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: election won: tally=2 - testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: election won: tally=2 -testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: entering leader state: leader="Node at 1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 [Leader]" - testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: entering leader state: leader="Node at 1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 [Leader]" -testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: added peer, starting replication: peer=server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e - testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: added peer, starting replication: peer=server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e -testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: added peer, starting replication: peer=server-8e44a857-c59b-1833-4366-2673c370165a - testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: added peer, starting replication: peer=server-8e44a857-c59b-1833-4366-2673c370165a -testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: pipelining replication: peer="{Voter server-8e44a857-c59b-1833-4366-2673c370165a 8e44a857-c59b-1833-4366-2673c370165a}" - testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: pipelining replication: peer="{Voter server-8e44a857-c59b-1833-4366-2673c370165a 8e44a857-c59b-1833-4366-2673c370165a}" -testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: pipelining replication: peer="{Voter server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e 05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e}" - testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: pipelining replication: peer="{Voter server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e 05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.629Z [INFO] stable state for Follower reached at 2024-01-09 05:10:07.52482094 +0000 UTC m=+18.777871995 (2 nodes), 64.711139ms from start of poll, 65.27158ms from cluster start. Timeout at 2024-01-09 05:10:07.629920362 +0000 UTC m=+18.882971392, 105.099397ms after stability - testing.go:149: cluster: 2024-01-09T05:10:07.629Z [INFO] stable state for Follower reached at 2024-01-09 05:10:07.52482094 +0000 UTC m=+18.777871995 (2 nodes), 64.711139ms from start of poll, 65.27158ms from cluster start. Timeout at 2024-01-09 05:10:07.629920362 +0000 UTC m=+18.882971392, 105.099397ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.630Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:07.630Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.735Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.630078372 +0000 UTC m=+18.883129399 (1 nodes), 2.443µs from start of poll, 170.528984ms from cluster start. Timeout at 2024-01-09 05:10:07.735262215 +0000 UTC m=+18.988313307, 105.183908ms after stability - testing.go:149: cluster: 2024-01-09T05:10:07.735Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.630078372 +0000 UTC m=+18.883129399 (1 nodes), 2.443µs from start of poll, 170.528984ms from cluster start. Timeout at 2024-01-09 05:10:07.735262215 +0000 UTC m=+18.988313307, 105.183908ms after stability -testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.735Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: aborting pipeline replication: peer="{Voter server-8e44a857-c59b-1833-4366-2673c370165a 8e44a857-c59b-1833-4366-2673c370165a}" - testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.735Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: aborting pipeline replication: peer="{Voter server-8e44a857-c59b-1833-4366-2673c370165a 8e44a857-c59b-1833-4366-2673c370165a}" -testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.735Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: aborting pipeline replication: peer="{Voter server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e 05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e}" - testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.735Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: aborting pipeline replication: peer="{Voter server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e 05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e}" ---- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.28s) +testLoggerAdapter verbose: server-8115d111-090f-b0b3-7491-04e0e0bd226a: 2025-02-10T11:34:21.398Z [INFO] server-8115d111-090f-b0b3-7491-04e0e0bd226a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8115d111-090f-b0b3-7491-04e0e0bd226a Address:8115d111-090f-b0b3-7491-04e0e0bd226a} {Suffrage:Voter ID:server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25 Address:29c44097-7ef6-4ac1-a9b4-2aa1521faa25} {Suffrage:Voter ID:server-c7f35895-4b95-1222-0a53-f702213edfa1 Address:c7f35895-4b95-1222-0a53-f702213edfa1}]" + testing.go:149: server-8115d111-090f-b0b3-7491-04e0e0bd226a: 2025-02-10T11:34:21.398Z [INFO] server-8115d111-090f-b0b3-7491-04e0e0bd226a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8115d111-090f-b0b3-7491-04e0e0bd226a Address:8115d111-090f-b0b3-7491-04e0e0bd226a} {Suffrage:Voter ID:server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25 Address:29c44097-7ef6-4ac1-a9b4-2aa1521faa25} {Suffrage:Voter ID:server-c7f35895-4b95-1222-0a53-f702213edfa1 Address:c7f35895-4b95-1222-0a53-f702213edfa1}]" +testLoggerAdapter verbose: server-8115d111-090f-b0b3-7491-04e0e0bd226a: 2025-02-10T11:34:21.398Z [INFO] server-8115d111-090f-b0b3-7491-04e0e0bd226a: entering follower state: follower="Node at 8115d111-090f-b0b3-7491-04e0e0bd226a [Follower]" leader= + testing.go:149: server-8115d111-090f-b0b3-7491-04e0e0bd226a: 2025-02-10T11:34:21.398Z [INFO] server-8115d111-090f-b0b3-7491-04e0e0bd226a: entering follower state: follower="Node at 8115d111-090f-b0b3-7491-04e0e0bd226a [Follower]" leader= +testLoggerAdapter verbose: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.398Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8115d111-090f-b0b3-7491-04e0e0bd226a Address:8115d111-090f-b0b3-7491-04e0e0bd226a} {Suffrage:Voter ID:server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25 Address:29c44097-7ef6-4ac1-a9b4-2aa1521faa25} {Suffrage:Voter ID:server-c7f35895-4b95-1222-0a53-f702213edfa1 Address:c7f35895-4b95-1222-0a53-f702213edfa1}]" + testing.go:149: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.398Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8115d111-090f-b0b3-7491-04e0e0bd226a Address:8115d111-090f-b0b3-7491-04e0e0bd226a} {Suffrage:Voter ID:server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25 Address:29c44097-7ef6-4ac1-a9b4-2aa1521faa25} {Suffrage:Voter ID:server-c7f35895-4b95-1222-0a53-f702213edfa1 Address:c7f35895-4b95-1222-0a53-f702213edfa1}]" +testLoggerAdapter verbose: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.398Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: entering follower state: follower="Node at 29c44097-7ef6-4ac1-a9b4-2aa1521faa25 [Follower]" leader= + testing.go:149: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.398Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: entering follower state: follower="Node at 29c44097-7ef6-4ac1-a9b4-2aa1521faa25 [Follower]" leader= +testLoggerAdapter verbose: server-c7f35895-4b95-1222-0a53-f702213edfa1: 2025-02-10T11:34:21.398Z [INFO] server-c7f35895-4b95-1222-0a53-f702213edfa1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8115d111-090f-b0b3-7491-04e0e0bd226a Address:8115d111-090f-b0b3-7491-04e0e0bd226a} {Suffrage:Voter ID:server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25 Address:29c44097-7ef6-4ac1-a9b4-2aa1521faa25} {Suffrage:Voter ID:server-c7f35895-4b95-1222-0a53-f702213edfa1 Address:c7f35895-4b95-1222-0a53-f702213edfa1}]" + testing.go:149: server-c7f35895-4b95-1222-0a53-f702213edfa1: 2025-02-10T11:34:21.398Z [INFO] server-c7f35895-4b95-1222-0a53-f702213edfa1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8115d111-090f-b0b3-7491-04e0e0bd226a Address:8115d111-090f-b0b3-7491-04e0e0bd226a} {Suffrage:Voter ID:server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25 Address:29c44097-7ef6-4ac1-a9b4-2aa1521faa25} {Suffrage:Voter ID:server-c7f35895-4b95-1222-0a53-f702213edfa1 Address:c7f35895-4b95-1222-0a53-f702213edfa1}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:21.398Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-c7f35895-4b95-1222-0a53-f702213edfa1: 2025-02-10T11:34:21.398Z [INFO] server-c7f35895-4b95-1222-0a53-f702213edfa1: entering follower state: follower="Node at c7f35895-4b95-1222-0a53-f702213edfa1 [Follower]" leader= + testing.go:149: cluster: 2025-02-10T11:34:21.398Z [INFO] starting stability test: raft-state=Follower + testing.go:149: server-c7f35895-4b95-1222-0a53-f702213edfa1: 2025-02-10T11:34:21.398Z [INFO] server-c7f35895-4b95-1222-0a53-f702213edfa1: entering follower state: follower="Node at c7f35895-4b95-1222-0a53-f702213edfa1 [Follower]" leader= +testLoggerAdapter verbose: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.456Z [WARN] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.456Z [WARN] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.456Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: entering candidate state: node="Node at 29c44097-7ef6-4ac1-a9b4-2aa1521faa25 [Candidate]" term=2 + testing.go:149: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.456Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: entering candidate state: node="Node at 29c44097-7ef6-4ac1-a9b4-2aa1521faa25 [Candidate]" term=2 +testLoggerAdapter verbose: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.456Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: election won: tally=2 + testing.go:149: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.456Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: election won: tally=2 +testLoggerAdapter verbose: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.456Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: entering leader state: leader="Node at 29c44097-7ef6-4ac1-a9b4-2aa1521faa25 [Leader]" + testing.go:149: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.456Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: entering leader state: leader="Node at 29c44097-7ef6-4ac1-a9b4-2aa1521faa25 [Leader]" +testLoggerAdapter verbose: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.456Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: added peer, starting replication: peer=server-8115d111-090f-b0b3-7491-04e0e0bd226a + testing.go:149: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.456Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: added peer, starting replication: peer=server-8115d111-090f-b0b3-7491-04e0e0bd226a +testLoggerAdapter verbose: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.456Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: added peer, starting replication: peer=server-c7f35895-4b95-1222-0a53-f702213edfa1 + testing.go:149: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.456Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: added peer, starting replication: peer=server-c7f35895-4b95-1222-0a53-f702213edfa1 +testLoggerAdapter verbose: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.456Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: pipelining replication: peer="{Voter server-8115d111-090f-b0b3-7491-04e0e0bd226a 8115d111-090f-b0b3-7491-04e0e0bd226a}" + testing.go:149: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.456Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: pipelining replication: peer="{Voter server-8115d111-090f-b0b3-7491-04e0e0bd226a 8115d111-090f-b0b3-7491-04e0e0bd226a}" +testLoggerAdapter verbose: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.456Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: pipelining replication: peer="{Voter server-c7f35895-4b95-1222-0a53-f702213edfa1 c7f35895-4b95-1222-0a53-f702213edfa1}" + testing.go:149: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.456Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: pipelining replication: peer="{Voter server-c7f35895-4b95-1222-0a53-f702213edfa1 c7f35895-4b95-1222-0a53-f702213edfa1}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:21.561Z [INFO] stable state for Follower reached at 2025-02-10 11:34:21.456877254 +0000 UTC m=+17.825781553 (2 nodes), 58.36095ms from start of poll, 58.741377ms from cluster start. Timeout at 2025-02-10 11:34:21.561950114 +0000 UTC m=+17.930854413, 105.07286ms after stability + testing.go:149: cluster: 2025-02-10T11:34:21.561Z [INFO] stable state for Follower reached at 2025-02-10 11:34:21.456877254 +0000 UTC m=+17.825781553 (2 nodes), 58.36095ms from start of poll, 58.741377ms from cluster start. Timeout at 2025-02-10 11:34:21.561950114 +0000 UTC m=+17.930854413, 105.07286ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:21.562Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:21.562Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:21.667Z [INFO] stable state for Leader reached at 2025-02-10 11:34:21.562037597 +0000 UTC m=+17.930941897 (1 nodes), 445ns from start of poll, 163.901721ms from cluster start. Timeout at 2025-02-10 11:34:21.667137931 +0000 UTC m=+18.036042284, 105.100387ms after stability + testing.go:149: cluster: 2025-02-10T11:34:21.667Z [INFO] stable state for Leader reached at 2025-02-10 11:34:21.562037597 +0000 UTC m=+17.930941897 (1 nodes), 445ns from start of poll, 163.901721ms from cluster start. Timeout at 2025-02-10 11:34:21.667137931 +0000 UTC m=+18.036042284, 105.100387ms after stability +testLoggerAdapter verbose: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.667Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: aborting pipeline replication: peer="{Voter server-c7f35895-4b95-1222-0a53-f702213edfa1 c7f35895-4b95-1222-0a53-f702213edfa1}" + testing.go:149: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.667Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: aborting pipeline replication: peer="{Voter server-c7f35895-4b95-1222-0a53-f702213edfa1 c7f35895-4b95-1222-0a53-f702213edfa1}" +testLoggerAdapter verbose: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.667Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: aborting pipeline replication: peer="{Voter server-8115d111-090f-b0b3-7491-04e0e0bd226a 8115d111-090f-b0b3-7491-04e0e0bd226a}" + testing.go:149: server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: 2025-02-10T11:34:21.667Z [INFO] server-29c44097-7ef6-4ac1-a9b4-2aa1521faa25: aborting pipeline replication: peer="{Voter server-8115d111-090f-b0b3-7491-04e0e0bd226a 8115d111-090f-b0b3-7491-04e0e0bd226a}" +--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.27s) === RUN TestRaft_LeadershipTransferToBehindServer -testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:07.737Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0189b842-19da-c595-de45-ddabfae2712d Address:0189b842-19da-c595-de45-ddabfae2712d} {Suffrage:Voter ID:server-3006c307-f604-7a72-ba1c-eeb8283298b2 Address:3006c307-f604-7a72-ba1c-eeb8283298b2} {Suffrage:Voter ID:server-688e118c-253e-992c-61f7-a58ca9e008bd Address:688e118c-253e-992c-61f7-a58ca9e008bd}]" - testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:07.737Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0189b842-19da-c595-de45-ddabfae2712d Address:0189b842-19da-c595-de45-ddabfae2712d} {Suffrage:Voter ID:server-3006c307-f604-7a72-ba1c-eeb8283298b2 Address:3006c307-f604-7a72-ba1c-eeb8283298b2} {Suffrage:Voter ID:server-688e118c-253e-992c-61f7-a58ca9e008bd Address:688e118c-253e-992c-61f7-a58ca9e008bd}]" -testLoggerAdapter verbose: server-3006c307-f604-7a72-ba1c-eeb8283298b2: 2024-01-09T05:10:07.737Z [INFO] server-3006c307-f604-7a72-ba1c-eeb8283298b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0189b842-19da-c595-de45-ddabfae2712d Address:0189b842-19da-c595-de45-ddabfae2712d} {Suffrage:Voter ID:server-3006c307-f604-7a72-ba1c-eeb8283298b2 Address:3006c307-f604-7a72-ba1c-eeb8283298b2} {Suffrage:Voter ID:server-688e118c-253e-992c-61f7-a58ca9e008bd Address:688e118c-253e-992c-61f7-a58ca9e008bd}]" - testing.go:149: server-3006c307-f604-7a72-ba1c-eeb8283298b2: 2024-01-09T05:10:07.737Z [INFO] server-3006c307-f604-7a72-ba1c-eeb8283298b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0189b842-19da-c595-de45-ddabfae2712d Address:0189b842-19da-c595-de45-ddabfae2712d} {Suffrage:Voter ID:server-3006c307-f604-7a72-ba1c-eeb8283298b2 Address:3006c307-f604-7a72-ba1c-eeb8283298b2} {Suffrage:Voter ID:server-688e118c-253e-992c-61f7-a58ca9e008bd Address:688e118c-253e-992c-61f7-a58ca9e008bd}]" -testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.737Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0189b842-19da-c595-de45-ddabfae2712d Address:0189b842-19da-c595-de45-ddabfae2712d} {Suffrage:Voter ID:server-3006c307-f604-7a72-ba1c-eeb8283298b2 Address:3006c307-f604-7a72-ba1c-eeb8283298b2} {Suffrage:Voter ID:server-688e118c-253e-992c-61f7-a58ca9e008bd Address:688e118c-253e-992c-61f7-a58ca9e008bd}]" - testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.737Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0189b842-19da-c595-de45-ddabfae2712d Address:0189b842-19da-c595-de45-ddabfae2712d} {Suffrage:Voter ID:server-3006c307-f604-7a72-ba1c-eeb8283298b2 Address:3006c307-f604-7a72-ba1c-eeb8283298b2} {Suffrage:Voter ID:server-688e118c-253e-992c-61f7-a58ca9e008bd Address:688e118c-253e-992c-61f7-a58ca9e008bd}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.737Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:07.737Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:07.737Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: entering follower state: follower="Node at 0189b842-19da-c595-de45-ddabfae2712d [Follower]" leader= - testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:07.737Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: entering follower state: follower="Node at 0189b842-19da-c595-de45-ddabfae2712d [Follower]" leader= -testLoggerAdapter verbose: server-3006c307-f604-7a72-ba1c-eeb8283298b2: 2024-01-09T05:10:07.737Z [INFO] server-3006c307-f604-7a72-ba1c-eeb8283298b2: entering follower state: follower="Node at 3006c307-f604-7a72-ba1c-eeb8283298b2 [Follower]" leader= - testing.go:149: server-3006c307-f604-7a72-ba1c-eeb8283298b2: 2024-01-09T05:10:07.737Z [INFO] server-3006c307-f604-7a72-ba1c-eeb8283298b2: entering follower state: follower="Node at 3006c307-f604-7a72-ba1c-eeb8283298b2 [Follower]" leader= -testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.737Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: entering follower state: follower="Node at 688e118c-253e-992c-61f7-a58ca9e008bd [Follower]" leader= - testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.737Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: entering follower state: follower="Node at 688e118c-253e-992c-61f7-a58ca9e008bd [Follower]" leader= -testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [WARN] server-688e118c-253e-992c-61f7-a58ca9e008bd: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [WARN] server-688e118c-253e-992c-61f7-a58ca9e008bd: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: entering candidate state: node="Node at 688e118c-253e-992c-61f7-a58ca9e008bd [Candidate]" term=2 - testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: entering candidate state: node="Node at 688e118c-253e-992c-61f7-a58ca9e008bd [Candidate]" term=2 -testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: election won: tally=2 - testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: election won: tally=2 -testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: entering leader state: leader="Node at 688e118c-253e-992c-61f7-a58ca9e008bd [Leader]" - testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: entering leader state: leader="Node at 688e118c-253e-992c-61f7-a58ca9e008bd [Leader]" -testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: added peer, starting replication: peer=server-0189b842-19da-c595-de45-ddabfae2712d - testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: added peer, starting replication: peer=server-0189b842-19da-c595-de45-ddabfae2712d -testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: added peer, starting replication: peer=server-3006c307-f604-7a72-ba1c-eeb8283298b2 - testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: added peer, starting replication: peer=server-3006c307-f604-7a72-ba1c-eeb8283298b2 -testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: pipelining replication: peer="{Voter server-3006c307-f604-7a72-ba1c-eeb8283298b2 3006c307-f604-7a72-ba1c-eeb8283298b2}" - testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: pipelining replication: peer="{Voter server-3006c307-f604-7a72-ba1c-eeb8283298b2 3006c307-f604-7a72-ba1c-eeb8283298b2}" -testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.789Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: pipelining replication: peer="{Voter server-0189b842-19da-c595-de45-ddabfae2712d 0189b842-19da-c595-de45-ddabfae2712d}" - testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.789Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: pipelining replication: peer="{Voter server-0189b842-19da-c595-de45-ddabfae2712d 0189b842-19da-c595-de45-ddabfae2712d}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.894Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.789103217 +0000 UTC m=+19.042154346 (1 nodes), 51.670716ms from start of poll, 52.175154ms from cluster start. Timeout at 2024-01-09 05:10:07.894230459 +0000 UTC m=+19.147281589, 105.127243ms after stability - testing.go:149: cluster: 2024-01-09T05:10:07.894Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.789103217 +0000 UTC m=+19.042154346 (1 nodes), 51.670716ms from start of poll, 52.175154ms from cluster start. Timeout at 2024-01-09 05:10:07.894230459 +0000 UTC m=+19.147281589, 105.127243ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.894Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:10:07.894Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.000Z [INFO] stable state for Follower reached at 2024-01-09 05:10:07.895793067 +0000 UTC m=+19.148844060 (2 nodes), 2.976µs from start of poll, 158.864868ms from cluster start. Timeout at 2024-01-09 05:10:08.000816822 +0000 UTC m=+19.253867842, 105.023782ms after stability - testing.go:149: cluster: 2024-01-09T05:10:08.000Z [INFO] stable state for Follower reached at 2024-01-09 05:10:07.895793067 +0000 UTC m=+19.148844060 (2 nodes), 2.976µs from start of poll, 158.864868ms from cluster start. Timeout at 2024-01-09 05:10:08.000816822 +0000 UTC m=+19.253867842, 105.023782ms after stability -testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.051Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: entering candidate state: node="Node at 0189b842-19da-c595-de45-ddabfae2712d [Candidate]" term=3 - testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.051Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: entering candidate state: node="Node at 0189b842-19da-c595-de45-ddabfae2712d [Candidate]" term=3 -testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:08.052Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: entering follower state: follower="Node at 688e118c-253e-992c-61f7-a58ca9e008bd [Follower]" leader= - testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:08.052Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: entering follower state: follower="Node at 688e118c-253e-992c-61f7-a58ca9e008bd [Follower]" leader= -testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:08.052Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: aborting pipeline replication: peer="{Voter server-3006c307-f604-7a72-ba1c-eeb8283298b2 3006c307-f604-7a72-ba1c-eeb8283298b2}" -testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: election won: tally=2 - testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:08.052Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: aborting pipeline replication: peer="{Voter server-3006c307-f604-7a72-ba1c-eeb8283298b2 3006c307-f604-7a72-ba1c-eeb8283298b2}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.052Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:08.052Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: aborting pipeline replication: peer="{Voter server-0189b842-19da-c595-de45-ddabfae2712d 0189b842-19da-c595-de45-ddabfae2712d}" - testing.go:149: cluster: 2024-01-09T05:10:08.052Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:08.052Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: aborting pipeline replication: peer="{Voter server-0189b842-19da-c595-de45-ddabfae2712d 0189b842-19da-c595-de45-ddabfae2712d}" - testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: election won: tally=2 -testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: entering leader state: leader="Node at 0189b842-19da-c595-de45-ddabfae2712d [Leader]" - testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: entering leader state: leader="Node at 0189b842-19da-c595-de45-ddabfae2712d [Leader]" -testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: added peer, starting replication: peer=server-3006c307-f604-7a72-ba1c-eeb8283298b2 - testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: added peer, starting replication: peer=server-3006c307-f604-7a72-ba1c-eeb8283298b2 -testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: added peer, starting replication: peer=server-688e118c-253e-992c-61f7-a58ca9e008bd - testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: added peer, starting replication: peer=server-688e118c-253e-992c-61f7-a58ca9e008bd -testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: pipelining replication: peer="{Voter server-688e118c-253e-992c-61f7-a58ca9e008bd 688e118c-253e-992c-61f7-a58ca9e008bd}" - testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: pipelining replication: peer="{Voter server-688e118c-253e-992c-61f7-a58ca9e008bd 688e118c-253e-992c-61f7-a58ca9e008bd}" -testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: pipelining replication: peer="{Voter server-3006c307-f604-7a72-ba1c-eeb8283298b2 3006c307-f604-7a72-ba1c-eeb8283298b2}" - testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: pipelining replication: peer="{Voter server-3006c307-f604-7a72-ba1c-eeb8283298b2 3006c307-f604-7a72-ba1c-eeb8283298b2}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.157Z [INFO] stable state for Leader reached at 2024-01-09 05:10:08.052448031 +0000 UTC m=+19.305499248 (1 nodes), 325.177µs from start of poll, 315.520056ms from cluster start. Timeout at 2024-01-09 05:10:08.157579275 +0000 UTC m=+19.410630342, 105.131094ms after stability - testing.go:149: cluster: 2024-01-09T05:10:08.157Z [INFO] stable state for Leader reached at 2024-01-09 05:10:08.052448031 +0000 UTC m=+19.305499248 (1 nodes), 325.177µs from start of poll, 315.520056ms from cluster start. Timeout at 2024-01-09 05:10:08.157579275 +0000 UTC m=+19.410630342, 105.131094ms after stability -testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.157Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: aborting pipeline replication: peer="{Voter server-688e118c-253e-992c-61f7-a58ca9e008bd 688e118c-253e-992c-61f7-a58ca9e008bd}" - testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.157Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: aborting pipeline replication: peer="{Voter server-688e118c-253e-992c-61f7-a58ca9e008bd 688e118c-253e-992c-61f7-a58ca9e008bd}" -testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.157Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: aborting pipeline replication: peer="{Voter server-3006c307-f604-7a72-ba1c-eeb8283298b2 3006c307-f604-7a72-ba1c-eeb8283298b2}" - testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.157Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: aborting pipeline replication: peer="{Voter server-3006c307-f604-7a72-ba1c-eeb8283298b2 3006c307-f604-7a72-ba1c-eeb8283298b2}" ---- PASS: TestRaft_LeadershipTransferToBehindServer (0.42s) +testLoggerAdapter verbose: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.668Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff256bae-ee32-9bed-349f-6e0a606ba47b Address:ff256bae-ee32-9bed-349f-6e0a606ba47b} {Suffrage:Voter ID:server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 Address:c098f7e6-002b-ac67-5e71-e0df37edf1c0} {Suffrage:Voter ID:server-0ae438cc-47a6-00c4-a233-e8547944a17f Address:0ae438cc-47a6-00c4-a233-e8547944a17f}]" + testing.go:149: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.668Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff256bae-ee32-9bed-349f-6e0a606ba47b Address:ff256bae-ee32-9bed-349f-6e0a606ba47b} {Suffrage:Voter ID:server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 Address:c098f7e6-002b-ac67-5e71-e0df37edf1c0} {Suffrage:Voter ID:server-0ae438cc-47a6-00c4-a233-e8547944a17f Address:0ae438cc-47a6-00c4-a233-e8547944a17f}]" +testLoggerAdapter verbose: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.668Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: entering follower state: follower="Node at ff256bae-ee32-9bed-349f-6e0a606ba47b [Follower]" leader= + testing.go:149: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.668Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: entering follower state: follower="Node at ff256bae-ee32-9bed-349f-6e0a606ba47b [Follower]" leader= +testLoggerAdapter verbose: server-c098f7e6-002b-ac67-5e71-e0df37edf1c0: 2025-02-10T11:34:21.668Z [INFO] server-c098f7e6-002b-ac67-5e71-e0df37edf1c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff256bae-ee32-9bed-349f-6e0a606ba47b Address:ff256bae-ee32-9bed-349f-6e0a606ba47b} {Suffrage:Voter ID:server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 Address:c098f7e6-002b-ac67-5e71-e0df37edf1c0} {Suffrage:Voter ID:server-0ae438cc-47a6-00c4-a233-e8547944a17f Address:0ae438cc-47a6-00c4-a233-e8547944a17f}]" + testing.go:149: server-c098f7e6-002b-ac67-5e71-e0df37edf1c0: 2025-02-10T11:34:21.668Z [INFO] server-c098f7e6-002b-ac67-5e71-e0df37edf1c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff256bae-ee32-9bed-349f-6e0a606ba47b Address:ff256bae-ee32-9bed-349f-6e0a606ba47b} {Suffrage:Voter ID:server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 Address:c098f7e6-002b-ac67-5e71-e0df37edf1c0} {Suffrage:Voter ID:server-0ae438cc-47a6-00c4-a233-e8547944a17f Address:0ae438cc-47a6-00c4-a233-e8547944a17f}]" +testLoggerAdapter verbose: server-c098f7e6-002b-ac67-5e71-e0df37edf1c0: 2025-02-10T11:34:21.668Z [INFO] server-c098f7e6-002b-ac67-5e71-e0df37edf1c0: entering follower state: follower="Node at c098f7e6-002b-ac67-5e71-e0df37edf1c0 [Follower]" leader= + testing.go:149: server-c098f7e6-002b-ac67-5e71-e0df37edf1c0: 2025-02-10T11:34:21.668Z [INFO] server-c098f7e6-002b-ac67-5e71-e0df37edf1c0: entering follower state: follower="Node at c098f7e6-002b-ac67-5e71-e0df37edf1c0 [Follower]" leader= +testLoggerAdapter verbose: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.668Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff256bae-ee32-9bed-349f-6e0a606ba47b Address:ff256bae-ee32-9bed-349f-6e0a606ba47b} {Suffrage:Voter ID:server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 Address:c098f7e6-002b-ac67-5e71-e0df37edf1c0} {Suffrage:Voter ID:server-0ae438cc-47a6-00c4-a233-e8547944a17f Address:0ae438cc-47a6-00c4-a233-e8547944a17f}]" + testing.go:149: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.668Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff256bae-ee32-9bed-349f-6e0a606ba47b Address:ff256bae-ee32-9bed-349f-6e0a606ba47b} {Suffrage:Voter ID:server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 Address:c098f7e6-002b-ac67-5e71-e0df37edf1c0} {Suffrage:Voter ID:server-0ae438cc-47a6-00c4-a233-e8547944a17f Address:0ae438cc-47a6-00c4-a233-e8547944a17f}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:21.668Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:21.668Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.668Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: entering follower state: follower="Node at 0ae438cc-47a6-00c4-a233-e8547944a17f [Follower]" leader= + testing.go:149: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.668Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: entering follower state: follower="Node at 0ae438cc-47a6-00c4-a233-e8547944a17f [Follower]" leader= +testLoggerAdapter verbose: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.728Z [WARN] server-0ae438cc-47a6-00c4-a233-e8547944a17f: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.728Z [WARN] server-0ae438cc-47a6-00c4-a233-e8547944a17f: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.728Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: entering candidate state: node="Node at 0ae438cc-47a6-00c4-a233-e8547944a17f [Candidate]" term=2 + testing.go:149: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.728Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: entering candidate state: node="Node at 0ae438cc-47a6-00c4-a233-e8547944a17f [Candidate]" term=2 +testLoggerAdapter verbose: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.728Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: election won: tally=2 + testing.go:149: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.728Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: election won: tally=2 +testLoggerAdapter verbose: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.728Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: entering leader state: leader="Node at 0ae438cc-47a6-00c4-a233-e8547944a17f [Leader]" + testing.go:149: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.728Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: entering leader state: leader="Node at 0ae438cc-47a6-00c4-a233-e8547944a17f [Leader]" +testLoggerAdapter verbose: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.728Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: added peer, starting replication: peer=server-ff256bae-ee32-9bed-349f-6e0a606ba47b + testing.go:149: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.728Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: added peer, starting replication: peer=server-ff256bae-ee32-9bed-349f-6e0a606ba47b +testLoggerAdapter verbose: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.728Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: added peer, starting replication: peer=server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 + testing.go:149: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.728Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: added peer, starting replication: peer=server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 +testLoggerAdapter verbose: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.728Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: pipelining replication: peer="{Voter server-ff256bae-ee32-9bed-349f-6e0a606ba47b ff256bae-ee32-9bed-349f-6e0a606ba47b}" + testing.go:149: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.728Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: pipelining replication: peer="{Voter server-ff256bae-ee32-9bed-349f-6e0a606ba47b ff256bae-ee32-9bed-349f-6e0a606ba47b}" +testLoggerAdapter verbose: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.728Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: pipelining replication: peer="{Voter server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 c098f7e6-002b-ac67-5e71-e0df37edf1c0}" + testing.go:149: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.728Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: pipelining replication: peer="{Voter server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 c098f7e6-002b-ac67-5e71-e0df37edf1c0}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:21.833Z [INFO] stable state for Leader reached at 2025-02-10 11:34:21.728608896 +0000 UTC m=+18.097513197 (1 nodes), 59.848473ms from start of poll, 60.193435ms from cluster start. Timeout at 2025-02-10 11:34:21.833682647 +0000 UTC m=+18.202586946, 105.073749ms after stability + testing.go:149: cluster: 2025-02-10T11:34:21.833Z [INFO] stable state for Leader reached at 2025-02-10 11:34:21.728608896 +0000 UTC m=+18.097513197 (1 nodes), 59.848473ms from start of poll, 60.193435ms from cluster start. Timeout at 2025-02-10 11:34:21.833682647 +0000 UTC m=+18.202586946, 105.073749ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:21.833Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:21.833Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:21.938Z [INFO] stable state for Follower reached at 2025-02-10 11:34:21.833744607 +0000 UTC m=+18.202648907 (2 nodes), 638ns from start of poll, 165.329145ms from cluster start. Timeout at 2025-02-10 11:34:21.938846097 +0000 UTC m=+18.307750450, 105.101543ms after stability + testing.go:149: cluster: 2025-02-10T11:34:21.938Z [INFO] stable state for Follower reached at 2025-02-10 11:34:21.833744607 +0000 UTC m=+18.202648907 (2 nodes), 638ns from start of poll, 165.329145ms from cluster start. Timeout at 2025-02-10 11:34:21.938846097 +0000 UTC m=+18.307750450, 105.101543ms after stability +testLoggerAdapter verbose: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.950Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: entering candidate state: node="Node at ff256bae-ee32-9bed-349f-6e0a606ba47b [Candidate]" term=3 + testing.go:149: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.950Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: entering candidate state: node="Node at ff256bae-ee32-9bed-349f-6e0a606ba47b [Candidate]" term=3 +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:21.950Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.950Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: entering follower state: follower="Node at 0ae438cc-47a6-00c4-a233-e8547944a17f [Follower]" leader= +testLoggerAdapter verbose: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.950Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: election won: tally=2 + testing.go:149: cluster: 2025-02-10T11:34:21.950Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.950Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: entering follower state: follower="Node at 0ae438cc-47a6-00c4-a233-e8547944a17f [Follower]" leader= +testLoggerAdapter verbose: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.950Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: aborting pipeline replication: peer="{Voter server-ff256bae-ee32-9bed-349f-6e0a606ba47b ff256bae-ee32-9bed-349f-6e0a606ba47b}" + testing.go:149: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.950Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: election won: tally=2 +testLoggerAdapter verbose: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.950Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: entering leader state: leader="Node at ff256bae-ee32-9bed-349f-6e0a606ba47b [Leader]" + testing.go:149: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.950Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: aborting pipeline replication: peer="{Voter server-ff256bae-ee32-9bed-349f-6e0a606ba47b ff256bae-ee32-9bed-349f-6e0a606ba47b}" +testLoggerAdapter verbose: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.950Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: aborting pipeline replication: peer="{Voter server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 c098f7e6-002b-ac67-5e71-e0df37edf1c0}" + testing.go:149: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.950Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: entering leader state: leader="Node at ff256bae-ee32-9bed-349f-6e0a606ba47b [Leader]" +testLoggerAdapter verbose: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.950Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: added peer, starting replication: peer=server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 + testing.go:149: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.950Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: added peer, starting replication: peer=server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 +testLoggerAdapter verbose: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.950Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: added peer, starting replication: peer=server-0ae438cc-47a6-00c4-a233-e8547944a17f + testing.go:149: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.950Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: added peer, starting replication: peer=server-0ae438cc-47a6-00c4-a233-e8547944a17f + testing.go:149: server-0ae438cc-47a6-00c4-a233-e8547944a17f: 2025-02-10T11:34:21.950Z [INFO] server-0ae438cc-47a6-00c4-a233-e8547944a17f: aborting pipeline replication: peer="{Voter server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 c098f7e6-002b-ac67-5e71-e0df37edf1c0}" +testLoggerAdapter verbose: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.951Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: pipelining replication: peer="{Voter server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 c098f7e6-002b-ac67-5e71-e0df37edf1c0}" + testing.go:149: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.951Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: pipelining replication: peer="{Voter server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 c098f7e6-002b-ac67-5e71-e0df37edf1c0}" +testLoggerAdapter verbose: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.951Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: pipelining replication: peer="{Voter server-0ae438cc-47a6-00c4-a233-e8547944a17f 0ae438cc-47a6-00c4-a233-e8547944a17f}" + testing.go:149: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:21.951Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: pipelining replication: peer="{Voter server-0ae438cc-47a6-00c4-a233-e8547944a17f 0ae438cc-47a6-00c4-a233-e8547944a17f}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:22.055Z [INFO] stable state for Leader reached at 2025-02-10 11:34:21.950897352 +0000 UTC m=+18.319801652 (1 nodes), 78.172µs from start of poll, 282.48189ms from cluster start. Timeout at 2025-02-10 11:34:22.05596965 +0000 UTC m=+18.424873944, 105.072292ms after stability + testing.go:149: cluster: 2025-02-10T11:34:22.055Z [INFO] stable state for Leader reached at 2025-02-10 11:34:21.950897352 +0000 UTC m=+18.319801652 (1 nodes), 78.172µs from start of poll, 282.48189ms from cluster start. Timeout at 2025-02-10 11:34:22.05596965 +0000 UTC m=+18.424873944, 105.072292ms after stability +testLoggerAdapter verbose: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:22.056Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: aborting pipeline replication: peer="{Voter server-0ae438cc-47a6-00c4-a233-e8547944a17f 0ae438cc-47a6-00c4-a233-e8547944a17f}" + testing.go:149: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:22.056Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: aborting pipeline replication: peer="{Voter server-0ae438cc-47a6-00c4-a233-e8547944a17f 0ae438cc-47a6-00c4-a233-e8547944a17f}" +testLoggerAdapter verbose: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:22.056Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: aborting pipeline replication: peer="{Voter server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 c098f7e6-002b-ac67-5e71-e0df37edf1c0}" + testing.go:149: server-ff256bae-ee32-9bed-349f-6e0a606ba47b: 2025-02-10T11:34:22.056Z [INFO] server-ff256bae-ee32-9bed-349f-6e0a606ba47b: aborting pipeline replication: peer="{Voter server-c098f7e6-002b-ac67-5e71-e0df37edf1c0 c098f7e6-002b-ac67-5e71-e0df37edf1c0}" +--- PASS: TestRaft_LeadershipTransferToBehindServer (0.39s) === RUN TestRaft_LeadershipTransferToItself -testLoggerAdapter verbose: server-30b2f93d-5285-84e7-551b-a48d711d6846: 2024-01-09T05:10:08.159Z [INFO] server-30b2f93d-5285-84e7-551b-a48d711d6846: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30b2f93d-5285-84e7-551b-a48d711d6846 Address:30b2f93d-5285-84e7-551b-a48d711d6846} {Suffrage:Voter ID:server-039a2132-e84d-30df-b4b5-6980889216e6 Address:039a2132-e84d-30df-b4b5-6980889216e6} {Suffrage:Voter ID:server-ce43a170-4e3e-1419-64a7-6b5df072e9b0 Address:ce43a170-4e3e-1419-64a7-6b5df072e9b0}]" - testing.go:149: server-30b2f93d-5285-84e7-551b-a48d711d6846: 2024-01-09T05:10:08.159Z [INFO] server-30b2f93d-5285-84e7-551b-a48d711d6846: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30b2f93d-5285-84e7-551b-a48d711d6846 Address:30b2f93d-5285-84e7-551b-a48d711d6846} {Suffrage:Voter ID:server-039a2132-e84d-30df-b4b5-6980889216e6 Address:039a2132-e84d-30df-b4b5-6980889216e6} {Suffrage:Voter ID:server-ce43a170-4e3e-1419-64a7-6b5df072e9b0 Address:ce43a170-4e3e-1419-64a7-6b5df072e9b0}]" -testLoggerAdapter verbose: server-039a2132-e84d-30df-b4b5-6980889216e6: 2024-01-09T05:10:08.159Z [INFO] server-039a2132-e84d-30df-b4b5-6980889216e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30b2f93d-5285-84e7-551b-a48d711d6846 Address:30b2f93d-5285-84e7-551b-a48d711d6846} {Suffrage:Voter ID:server-039a2132-e84d-30df-b4b5-6980889216e6 Address:039a2132-e84d-30df-b4b5-6980889216e6} {Suffrage:Voter ID:server-ce43a170-4e3e-1419-64a7-6b5df072e9b0 Address:ce43a170-4e3e-1419-64a7-6b5df072e9b0}]" - testing.go:149: server-039a2132-e84d-30df-b4b5-6980889216e6: 2024-01-09T05:10:08.159Z [INFO] server-039a2132-e84d-30df-b4b5-6980889216e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30b2f93d-5285-84e7-551b-a48d711d6846 Address:30b2f93d-5285-84e7-551b-a48d711d6846} {Suffrage:Voter ID:server-039a2132-e84d-30df-b4b5-6980889216e6 Address:039a2132-e84d-30df-b4b5-6980889216e6} {Suffrage:Voter ID:server-ce43a170-4e3e-1419-64a7-6b5df072e9b0 Address:ce43a170-4e3e-1419-64a7-6b5df072e9b0}]" -testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.160Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30b2f93d-5285-84e7-551b-a48d711d6846 Address:30b2f93d-5285-84e7-551b-a48d711d6846} {Suffrage:Voter ID:server-039a2132-e84d-30df-b4b5-6980889216e6 Address:039a2132-e84d-30df-b4b5-6980889216e6} {Suffrage:Voter ID:server-ce43a170-4e3e-1419-64a7-6b5df072e9b0 Address:ce43a170-4e3e-1419-64a7-6b5df072e9b0}]" - testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.160Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30b2f93d-5285-84e7-551b-a48d711d6846 Address:30b2f93d-5285-84e7-551b-a48d711d6846} {Suffrage:Voter ID:server-039a2132-e84d-30df-b4b5-6980889216e6 Address:039a2132-e84d-30df-b4b5-6980889216e6} {Suffrage:Voter ID:server-ce43a170-4e3e-1419-64a7-6b5df072e9b0 Address:ce43a170-4e3e-1419-64a7-6b5df072e9b0}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.160Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:08.160Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-30b2f93d-5285-84e7-551b-a48d711d6846: 2024-01-09T05:10:08.160Z [INFO] server-30b2f93d-5285-84e7-551b-a48d711d6846: entering follower state: follower="Node at 30b2f93d-5285-84e7-551b-a48d711d6846 [Follower]" leader= - testing.go:149: server-30b2f93d-5285-84e7-551b-a48d711d6846: 2024-01-09T05:10:08.160Z [INFO] server-30b2f93d-5285-84e7-551b-a48d711d6846: entering follower state: follower="Node at 30b2f93d-5285-84e7-551b-a48d711d6846 [Follower]" leader= -testLoggerAdapter verbose: server-039a2132-e84d-30df-b4b5-6980889216e6: 2024-01-09T05:10:08.160Z [INFO] server-039a2132-e84d-30df-b4b5-6980889216e6: entering follower state: follower="Node at 039a2132-e84d-30df-b4b5-6980889216e6 [Follower]" leader= - testing.go:149: server-039a2132-e84d-30df-b4b5-6980889216e6: 2024-01-09T05:10:08.160Z [INFO] server-039a2132-e84d-30df-b4b5-6980889216e6: entering follower state: follower="Node at 039a2132-e84d-30df-b4b5-6980889216e6 [Follower]" leader= -testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.160Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: entering follower state: follower="Node at ce43a170-4e3e-1419-64a7-6b5df072e9b0 [Follower]" leader= - testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.160Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: entering follower state: follower="Node at ce43a170-4e3e-1419-64a7-6b5df072e9b0 [Follower]" leader= -testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [WARN] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [WARN] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: entering candidate state: node="Node at ce43a170-4e3e-1419-64a7-6b5df072e9b0 [Candidate]" term=2 - testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: entering candidate state: node="Node at ce43a170-4e3e-1419-64a7-6b5df072e9b0 [Candidate]" term=2 -testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: election won: tally=2 - testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: election won: tally=2 -testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: entering leader state: leader="Node at ce43a170-4e3e-1419-64a7-6b5df072e9b0 [Leader]" - testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: entering leader state: leader="Node at ce43a170-4e3e-1419-64a7-6b5df072e9b0 [Leader]" -testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: added peer, starting replication: peer=server-30b2f93d-5285-84e7-551b-a48d711d6846 - testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: added peer, starting replication: peer=server-30b2f93d-5285-84e7-551b-a48d711d6846 -testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.232Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: added peer, starting replication: peer=server-039a2132-e84d-30df-b4b5-6980889216e6 - testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.232Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: added peer, starting replication: peer=server-039a2132-e84d-30df-b4b5-6980889216e6 -testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.232Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: pipelining replication: peer="{Voter server-039a2132-e84d-30df-b4b5-6980889216e6 039a2132-e84d-30df-b4b5-6980889216e6}" - testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.232Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: pipelining replication: peer="{Voter server-039a2132-e84d-30df-b4b5-6980889216e6 039a2132-e84d-30df-b4b5-6980889216e6}" -testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.232Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: pipelining replication: peer="{Voter server-30b2f93d-5285-84e7-551b-a48d711d6846 30b2f93d-5285-84e7-551b-a48d711d6846}" - testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.232Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: pipelining replication: peer="{Voter server-30b2f93d-5285-84e7-551b-a48d711d6846 30b2f93d-5285-84e7-551b-a48d711d6846}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.337Z [INFO] stable state for Leader reached at 2024-01-09 05:10:08.232348768 +0000 UTC m=+19.485399956 (1 nodes), 72.229917ms from start of poll, 72.7115ms from cluster start. Timeout at 2024-01-09 05:10:08.337443471 +0000 UTC m=+19.590494499, 105.094543ms after stability - testing.go:149: cluster: 2024-01-09T05:10:08.337Z [INFO] stable state for Leader reached at 2024-01-09 05:10:08.232348768 +0000 UTC m=+19.485399956 (1 nodes), 72.229917ms from start of poll, 72.7115ms from cluster start. Timeout at 2024-01-09 05:10:08.337443471 +0000 UTC m=+19.590494499, 105.094543ms after stability -testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.337Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: cannot transfer leadership to itself - testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.337Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: cannot transfer leadership to itself -testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.337Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: aborting pipeline replication: peer="{Voter server-039a2132-e84d-30df-b4b5-6980889216e6 039a2132-e84d-30df-b4b5-6980889216e6}" - testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.337Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: aborting pipeline replication: peer="{Voter server-039a2132-e84d-30df-b4b5-6980889216e6 039a2132-e84d-30df-b4b5-6980889216e6}" -testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.337Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: aborting pipeline replication: peer="{Voter server-30b2f93d-5285-84e7-551b-a48d711d6846 30b2f93d-5285-84e7-551b-a48d711d6846}" - testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.337Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: aborting pipeline replication: peer="{Voter server-30b2f93d-5285-84e7-551b-a48d711d6846 30b2f93d-5285-84e7-551b-a48d711d6846}" ---- PASS: TestRaft_LeadershipTransferToItself (0.18s) +testLoggerAdapter verbose: server-42198df6-ee40-1b10-5914-b94129e72bc0: 2025-02-10T11:34:22.057Z [INFO] server-42198df6-ee40-1b10-5914-b94129e72bc0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42198df6-ee40-1b10-5914-b94129e72bc0 Address:42198df6-ee40-1b10-5914-b94129e72bc0} {Suffrage:Voter ID:server-f286f4b5-df17-1e6c-cd26-acd57f83e12c Address:f286f4b5-df17-1e6c-cd26-acd57f83e12c} {Suffrage:Voter ID:server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a Address:0d4b481b-b09b-77da-b4ca-452aa42ccf2a}]" + testing.go:149: server-42198df6-ee40-1b10-5914-b94129e72bc0: 2025-02-10T11:34:22.057Z [INFO] server-42198df6-ee40-1b10-5914-b94129e72bc0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42198df6-ee40-1b10-5914-b94129e72bc0 Address:42198df6-ee40-1b10-5914-b94129e72bc0} {Suffrage:Voter ID:server-f286f4b5-df17-1e6c-cd26-acd57f83e12c Address:f286f4b5-df17-1e6c-cd26-acd57f83e12c} {Suffrage:Voter ID:server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a Address:0d4b481b-b09b-77da-b4ca-452aa42ccf2a}]" +testLoggerAdapter verbose: server-42198df6-ee40-1b10-5914-b94129e72bc0: 2025-02-10T11:34:22.057Z [INFO] server-42198df6-ee40-1b10-5914-b94129e72bc0: entering follower state: follower="Node at 42198df6-ee40-1b10-5914-b94129e72bc0 [Follower]" leader= + testing.go:149: server-42198df6-ee40-1b10-5914-b94129e72bc0: 2025-02-10T11:34:22.057Z [INFO] server-42198df6-ee40-1b10-5914-b94129e72bc0: entering follower state: follower="Node at 42198df6-ee40-1b10-5914-b94129e72bc0 [Follower]" leader= +testLoggerAdapter verbose: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.057Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42198df6-ee40-1b10-5914-b94129e72bc0 Address:42198df6-ee40-1b10-5914-b94129e72bc0} {Suffrage:Voter ID:server-f286f4b5-df17-1e6c-cd26-acd57f83e12c Address:f286f4b5-df17-1e6c-cd26-acd57f83e12c} {Suffrage:Voter ID:server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a Address:0d4b481b-b09b-77da-b4ca-452aa42ccf2a}]" + testing.go:149: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.057Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42198df6-ee40-1b10-5914-b94129e72bc0 Address:42198df6-ee40-1b10-5914-b94129e72bc0} {Suffrage:Voter ID:server-f286f4b5-df17-1e6c-cd26-acd57f83e12c Address:f286f4b5-df17-1e6c-cd26-acd57f83e12c} {Suffrage:Voter ID:server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a Address:0d4b481b-b09b-77da-b4ca-452aa42ccf2a}]" +testLoggerAdapter verbose: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.057Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: entering follower state: follower="Node at f286f4b5-df17-1e6c-cd26-acd57f83e12c [Follower]" leader= + testing.go:149: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.057Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: entering follower state: follower="Node at f286f4b5-df17-1e6c-cd26-acd57f83e12c [Follower]" leader= +testLoggerAdapter verbose: server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a: 2025-02-10T11:34:22.057Z [INFO] server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42198df6-ee40-1b10-5914-b94129e72bc0 Address:42198df6-ee40-1b10-5914-b94129e72bc0} {Suffrage:Voter ID:server-f286f4b5-df17-1e6c-cd26-acd57f83e12c Address:f286f4b5-df17-1e6c-cd26-acd57f83e12c} {Suffrage:Voter ID:server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a Address:0d4b481b-b09b-77da-b4ca-452aa42ccf2a}]" + testing.go:149: server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a: 2025-02-10T11:34:22.057Z [INFO] server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42198df6-ee40-1b10-5914-b94129e72bc0 Address:42198df6-ee40-1b10-5914-b94129e72bc0} {Suffrage:Voter ID:server-f286f4b5-df17-1e6c-cd26-acd57f83e12c Address:f286f4b5-df17-1e6c-cd26-acd57f83e12c} {Suffrage:Voter ID:server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a Address:0d4b481b-b09b-77da-b4ca-452aa42ccf2a}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:22.057Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a: 2025-02-10T11:34:22.057Z [INFO] server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a: entering follower state: follower="Node at 0d4b481b-b09b-77da-b4ca-452aa42ccf2a [Follower]" leader= + testing.go:149: cluster: 2025-02-10T11:34:22.057Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a: 2025-02-10T11:34:22.057Z [INFO] server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a: entering follower state: follower="Node at 0d4b481b-b09b-77da-b4ca-452aa42ccf2a [Follower]" leader= +testLoggerAdapter verbose: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.109Z [WARN] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.109Z [WARN] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.109Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: entering candidate state: node="Node at f286f4b5-df17-1e6c-cd26-acd57f83e12c [Candidate]" term=2 + testing.go:149: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.109Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: entering candidate state: node="Node at f286f4b5-df17-1e6c-cd26-acd57f83e12c [Candidate]" term=2 +testLoggerAdapter verbose: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.110Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: election won: tally=2 + testing.go:149: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.110Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: election won: tally=2 +testLoggerAdapter verbose: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.110Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: entering leader state: leader="Node at f286f4b5-df17-1e6c-cd26-acd57f83e12c [Leader]" + testing.go:149: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.110Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: entering leader state: leader="Node at f286f4b5-df17-1e6c-cd26-acd57f83e12c [Leader]" +testLoggerAdapter verbose: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.110Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: added peer, starting replication: peer=server-42198df6-ee40-1b10-5914-b94129e72bc0 + testing.go:149: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.110Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: added peer, starting replication: peer=server-42198df6-ee40-1b10-5914-b94129e72bc0 +testLoggerAdapter verbose: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.110Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: added peer, starting replication: peer=server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a + testing.go:149: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.110Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: added peer, starting replication: peer=server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a +testLoggerAdapter verbose: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.110Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: pipelining replication: peer="{Voter server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a 0d4b481b-b09b-77da-b4ca-452aa42ccf2a}" + testing.go:149: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.110Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: pipelining replication: peer="{Voter server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a 0d4b481b-b09b-77da-b4ca-452aa42ccf2a}" +testLoggerAdapter verbose: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.110Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: pipelining replication: peer="{Voter server-42198df6-ee40-1b10-5914-b94129e72bc0 42198df6-ee40-1b10-5914-b94129e72bc0}" + testing.go:149: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.110Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: pipelining replication: peer="{Voter server-42198df6-ee40-1b10-5914-b94129e72bc0 42198df6-ee40-1b10-5914-b94129e72bc0}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:22.215Z [INFO] stable state for Leader reached at 2025-02-10 11:34:22.110100435 +0000 UTC m=+18.479004736 (1 nodes), 52.771579ms from start of poll, 53.146643ms from cluster start. Timeout at 2025-02-10 11:34:22.215172189 +0000 UTC m=+18.584076483, 105.071747ms after stability + testing.go:149: cluster: 2025-02-10T11:34:22.215Z [INFO] stable state for Leader reached at 2025-02-10 11:34:22.110100435 +0000 UTC m=+18.479004736 (1 nodes), 52.771579ms from start of poll, 53.146643ms from cluster start. Timeout at 2025-02-10 11:34:22.215172189 +0000 UTC m=+18.584076483, 105.071747ms after stability +testLoggerAdapter verbose: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.215Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: cannot transfer leadership to itself + testing.go:149: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.215Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: cannot transfer leadership to itself +testLoggerAdapter verbose: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.215Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: aborting pipeline replication: peer="{Voter server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a 0d4b481b-b09b-77da-b4ca-452aa42ccf2a}" + testing.go:149: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.215Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: aborting pipeline replication: peer="{Voter server-0d4b481b-b09b-77da-b4ca-452aa42ccf2a 0d4b481b-b09b-77da-b4ca-452aa42ccf2a}" +testLoggerAdapter verbose: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.215Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: aborting pipeline replication: peer="{Voter server-42198df6-ee40-1b10-5914-b94129e72bc0 42198df6-ee40-1b10-5914-b94129e72bc0}" + testing.go:149: server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: 2025-02-10T11:34:22.215Z [INFO] server-f286f4b5-df17-1e6c-cd26-acd57f83e12c: aborting pipeline replication: peer="{Voter server-42198df6-ee40-1b10-5914-b94129e72bc0 42198df6-ee40-1b10-5914-b94129e72bc0}" +--- PASS: TestRaft_LeadershipTransferToItself (0.16s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests raft_test.go:2064: Disabled unreliable test --- SKIP: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.00s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout -testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.339Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 Address:b30cf42e-8ebf-2b2c-c15a-b8848dd9b080} {Suffrage:Voter ID:server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 Address:3c3001ba-c8f3-6e64-1597-3e71fc8e3810} {Suffrage:Voter ID:server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf Address:fa31d492-ba2b-6ed1-5267-4d4af9584eaf}]" - testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.339Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 Address:b30cf42e-8ebf-2b2c-c15a-b8848dd9b080} {Suffrage:Voter ID:server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 Address:3c3001ba-c8f3-6e64-1597-3e71fc8e3810} {Suffrage:Voter ID:server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf Address:fa31d492-ba2b-6ed1-5267-4d4af9584eaf}]" -testLoggerAdapter verbose: server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810: 2024-01-09T05:10:08.339Z [INFO] server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 Address:b30cf42e-8ebf-2b2c-c15a-b8848dd9b080} {Suffrage:Voter ID:server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 Address:3c3001ba-c8f3-6e64-1597-3e71fc8e3810} {Suffrage:Voter ID:server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf Address:fa31d492-ba2b-6ed1-5267-4d4af9584eaf}]" - testing.go:149: server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810: 2024-01-09T05:10:08.339Z [INFO] server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 Address:b30cf42e-8ebf-2b2c-c15a-b8848dd9b080} {Suffrage:Voter ID:server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 Address:3c3001ba-c8f3-6e64-1597-3e71fc8e3810} {Suffrage:Voter ID:server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf Address:fa31d492-ba2b-6ed1-5267-4d4af9584eaf}]" -testLoggerAdapter verbose: server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf: 2024-01-09T05:10:08.339Z [INFO] server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 Address:b30cf42e-8ebf-2b2c-c15a-b8848dd9b080} {Suffrage:Voter ID:server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 Address:3c3001ba-c8f3-6e64-1597-3e71fc8e3810} {Suffrage:Voter ID:server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf Address:fa31d492-ba2b-6ed1-5267-4d4af9584eaf}]" - testing.go:149: server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf: 2024-01-09T05:10:08.339Z [INFO] server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 Address:b30cf42e-8ebf-2b2c-c15a-b8848dd9b080} {Suffrage:Voter ID:server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 Address:3c3001ba-c8f3-6e64-1597-3e71fc8e3810} {Suffrage:Voter ID:server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf Address:fa31d492-ba2b-6ed1-5267-4d4af9584eaf}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.339Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:08.339Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.339Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: entering follower state: follower="Node at b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 [Follower]" leader= - testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.339Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: entering follower state: follower="Node at b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 [Follower]" leader= -testLoggerAdapter verbose: server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810: 2024-01-09T05:10:08.339Z [INFO] server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810: entering follower state: follower="Node at 3c3001ba-c8f3-6e64-1597-3e71fc8e3810 [Follower]" leader= - testing.go:149: server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810: 2024-01-09T05:10:08.339Z [INFO] server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810: entering follower state: follower="Node at 3c3001ba-c8f3-6e64-1597-3e71fc8e3810 [Follower]" leader= -testLoggerAdapter verbose: server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf: 2024-01-09T05:10:08.339Z [INFO] server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf: entering follower state: follower="Node at fa31d492-ba2b-6ed1-5267-4d4af9584eaf [Follower]" leader= - testing.go:149: server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf: 2024-01-09T05:10:08.339Z [INFO] server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf: entering follower state: follower="Node at fa31d492-ba2b-6ed1-5267-4d4af9584eaf [Follower]" leader= -testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [WARN] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [WARN] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: entering candidate state: node="Node at b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 [Candidate]" term=2 - testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: entering candidate state: node="Node at b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 [Candidate]" term=2 -testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: election won: tally=2 - testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: election won: tally=2 -testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: entering leader state: leader="Node at b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 [Leader]" - testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: entering leader state: leader="Node at b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 [Leader]" -testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: added peer, starting replication: peer=server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 - testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: added peer, starting replication: peer=server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 -testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.405Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: added peer, starting replication: peer=server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf - testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.405Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: added peer, starting replication: peer=server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf -testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.405Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: pipelining replication: peer="{Voter server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf fa31d492-ba2b-6ed1-5267-4d4af9584eaf}" - testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.405Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: pipelining replication: peer="{Voter server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf fa31d492-ba2b-6ed1-5267-4d4af9584eaf}" -testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.405Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: pipelining replication: peer="{Voter server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 3c3001ba-c8f3-6e64-1597-3e71fc8e3810}" - testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.405Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: pipelining replication: peer="{Voter server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 3c3001ba-c8f3-6e64-1597-3e71fc8e3810}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.514Z [INFO] stable state for Leader reached at 2024-01-09 05:10:08.405333527 +0000 UTC m=+19.658384553 (1 nodes), 65.765646ms from start of poll, 66.238579ms from cluster start. Timeout at 2024-01-09 05:10:08.513943353 +0000 UTC m=+19.766994375, 108.609822ms after stability - testing.go:149: cluster: 2024-01-09T05:10:08.514Z [INFO] stable state for Leader reached at 2024-01-09 05:10:08.405333527 +0000 UTC m=+19.658384553 (1 nodes), 65.765646ms from start of poll, 66.238579ms from cluster start. Timeout at 2024-01-09 05:10:08.513943353 +0000 UTC m=+19.766994375, 108.609822ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.514Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:10:08.514Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.619Z [INFO] stable state for Follower reached at 2024-01-09 05:10:08.514110956 +0000 UTC m=+19.767161959 (2 nodes), 2.505µs from start of poll, 175.015985ms from cluster start. Timeout at 2024-01-09 05:10:08.619219897 +0000 UTC m=+19.872270951, 105.108992ms after stability - testing.go:149: cluster: 2024-01-09T05:10:08.619Z [INFO] stable state for Follower reached at 2024-01-09 05:10:08.514110956 +0000 UTC m=+19.767161959 (2 nodes), 2.505µs from start of poll, 175.015985ms from cluster start. Timeout at 2024-01-09 05:10:08.619219897 +0000 UTC m=+19.872270951, 105.108992ms after stability -testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:09.125Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: aborting pipeline replication: peer="{Voter server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf fa31d492-ba2b-6ed1-5267-4d4af9584eaf}" - testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:09.125Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: aborting pipeline replication: peer="{Voter server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf fa31d492-ba2b-6ed1-5267-4d4af9584eaf}" -testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:09.125Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: aborting pipeline replication: peer="{Voter server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 3c3001ba-c8f3-6e64-1597-3e71fc8e3810}" - testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:09.125Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: aborting pipeline replication: peer="{Voter server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 3c3001ba-c8f3-6e64-1597-3e71fc8e3810}" ---- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.79s) +testLoggerAdapter verbose: server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e: 2025-02-10T11:34:22.216Z [INFO] server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e Address:3a12c92e-fc2d-1f28-050f-adf44c1a0f2e} {Suffrage:Voter ID:server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a Address:3a70638d-9f2a-c382-2ad7-b0e872d06a3a} {Suffrage:Voter ID:server-9050a661-80a4-d229-72a8-da77335bbc36 Address:9050a661-80a4-d229-72a8-da77335bbc36}]" + testing.go:149: server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e: 2025-02-10T11:34:22.216Z [INFO] server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e Address:3a12c92e-fc2d-1f28-050f-adf44c1a0f2e} {Suffrage:Voter ID:server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a Address:3a70638d-9f2a-c382-2ad7-b0e872d06a3a} {Suffrage:Voter ID:server-9050a661-80a4-d229-72a8-da77335bbc36 Address:9050a661-80a4-d229-72a8-da77335bbc36}]" +testLoggerAdapter verbose: server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e: 2025-02-10T11:34:22.216Z [INFO] server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e: entering follower state: follower="Node at 3a12c92e-fc2d-1f28-050f-adf44c1a0f2e [Follower]" leader= + testing.go:149: server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e: 2025-02-10T11:34:22.216Z [INFO] server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e: entering follower state: follower="Node at 3a12c92e-fc2d-1f28-050f-adf44c1a0f2e [Follower]" leader= +testLoggerAdapter verbose: server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a: 2025-02-10T11:34:22.216Z [INFO] server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e Address:3a12c92e-fc2d-1f28-050f-adf44c1a0f2e} {Suffrage:Voter ID:server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a Address:3a70638d-9f2a-c382-2ad7-b0e872d06a3a} {Suffrage:Voter ID:server-9050a661-80a4-d229-72a8-da77335bbc36 Address:9050a661-80a4-d229-72a8-da77335bbc36}]" + testing.go:149: server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a: 2025-02-10T11:34:22.216Z [INFO] server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e Address:3a12c92e-fc2d-1f28-050f-adf44c1a0f2e} {Suffrage:Voter ID:server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a Address:3a70638d-9f2a-c382-2ad7-b0e872d06a3a} {Suffrage:Voter ID:server-9050a661-80a4-d229-72a8-da77335bbc36 Address:9050a661-80a4-d229-72a8-da77335bbc36}]" +testLoggerAdapter verbose: server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a: 2025-02-10T11:34:22.216Z [INFO] server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a: entering follower state: follower="Node at 3a70638d-9f2a-c382-2ad7-b0e872d06a3a [Follower]" leader= + testing.go:149: server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a: 2025-02-10T11:34:22.216Z [INFO] server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a: entering follower state: follower="Node at 3a70638d-9f2a-c382-2ad7-b0e872d06a3a [Follower]" leader= +testLoggerAdapter verbose: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.216Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e Address:3a12c92e-fc2d-1f28-050f-adf44c1a0f2e} {Suffrage:Voter ID:server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a Address:3a70638d-9f2a-c382-2ad7-b0e872d06a3a} {Suffrage:Voter ID:server-9050a661-80a4-d229-72a8-da77335bbc36 Address:9050a661-80a4-d229-72a8-da77335bbc36}]" + testing.go:149: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.216Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e Address:3a12c92e-fc2d-1f28-050f-adf44c1a0f2e} {Suffrage:Voter ID:server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a Address:3a70638d-9f2a-c382-2ad7-b0e872d06a3a} {Suffrage:Voter ID:server-9050a661-80a4-d229-72a8-da77335bbc36 Address:9050a661-80a4-d229-72a8-da77335bbc36}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:22.216Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:22.216Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.216Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: entering follower state: follower="Node at 9050a661-80a4-d229-72a8-da77335bbc36 [Follower]" leader= + testing.go:149: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.216Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: entering follower state: follower="Node at 9050a661-80a4-d229-72a8-da77335bbc36 [Follower]" leader= +testLoggerAdapter verbose: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.289Z [WARN] server-9050a661-80a4-d229-72a8-da77335bbc36: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.289Z [WARN] server-9050a661-80a4-d229-72a8-da77335bbc36: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.289Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: entering candidate state: node="Node at 9050a661-80a4-d229-72a8-da77335bbc36 [Candidate]" term=2 + testing.go:149: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.289Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: entering candidate state: node="Node at 9050a661-80a4-d229-72a8-da77335bbc36 [Candidate]" term=2 +testLoggerAdapter verbose: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.289Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: election won: tally=2 + testing.go:149: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.289Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: election won: tally=2 +testLoggerAdapter verbose: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.289Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: entering leader state: leader="Node at 9050a661-80a4-d229-72a8-da77335bbc36 [Leader]" + testing.go:149: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.289Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: entering leader state: leader="Node at 9050a661-80a4-d229-72a8-da77335bbc36 [Leader]" +testLoggerAdapter verbose: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.289Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: added peer, starting replication: peer=server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e + testing.go:149: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.289Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: added peer, starting replication: peer=server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e +testLoggerAdapter verbose: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.289Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: added peer, starting replication: peer=server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a + testing.go:149: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.289Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: added peer, starting replication: peer=server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a +testLoggerAdapter verbose: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.289Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: pipelining replication: peer="{Voter server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e 3a12c92e-fc2d-1f28-050f-adf44c1a0f2e}" + testing.go:149: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.289Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: pipelining replication: peer="{Voter server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e 3a12c92e-fc2d-1f28-050f-adf44c1a0f2e}" +testLoggerAdapter verbose: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.289Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: pipelining replication: peer="{Voter server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a 3a70638d-9f2a-c382-2ad7-b0e872d06a3a}" + testing.go:149: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.289Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: pipelining replication: peer="{Voter server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a 3a70638d-9f2a-c382-2ad7-b0e872d06a3a}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:22.394Z [INFO] stable state for Leader reached at 2025-02-10 11:34:22.289543263 +0000 UTC m=+18.658447574 (1 nodes), 72.929579ms from start of poll, 73.324185ms from cluster start. Timeout at 2025-02-10 11:34:22.394615366 +0000 UTC m=+18.763519662, 105.072088ms after stability + testing.go:149: cluster: 2025-02-10T11:34:22.394Z [INFO] stable state for Leader reached at 2025-02-10 11:34:22.289543263 +0000 UTC m=+18.658447574 (1 nodes), 72.929579ms from start of poll, 73.324185ms from cluster start. Timeout at 2025-02-10 11:34:22.394615366 +0000 UTC m=+18.763519662, 105.072088ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:22.394Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:22.394Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:22.499Z [INFO] stable state for Follower reached at 2025-02-10 11:34:22.394693105 +0000 UTC m=+18.763597404 (2 nodes), 630ns from start of poll, 178.474015ms from cluster start. Timeout at 2025-02-10 11:34:22.49976504 +0000 UTC m=+18.868669336, 105.071932ms after stability + testing.go:149: cluster: 2025-02-10T11:34:22.499Z [INFO] stable state for Follower reached at 2025-02-10 11:34:22.394693105 +0000 UTC m=+18.763597404 (2 nodes), 630ns from start of poll, 178.474015ms from cluster start. Timeout at 2025-02-10 11:34:22.49976504 +0000 UTC m=+18.868669336, 105.071932ms after stability +testLoggerAdapter verbose: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.621Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: aborting pipeline replication: peer="{Voter server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a 3a70638d-9f2a-c382-2ad7-b0e872d06a3a}" + testing.go:149: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.621Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: aborting pipeline replication: peer="{Voter server-3a70638d-9f2a-c382-2ad7-b0e872d06a3a 3a70638d-9f2a-c382-2ad7-b0e872d06a3a}" +testLoggerAdapter verbose: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.621Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: aborting pipeline replication: peer="{Voter server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e 3a12c92e-fc2d-1f28-050f-adf44c1a0f2e}" + testing.go:149: server-9050a661-80a4-d229-72a8-da77335bbc36: 2025-02-10T11:34:22.621Z [INFO] server-9050a661-80a4-d229-72a8-da77335bbc36: aborting pipeline replication: peer="{Voter server-3a12c92e-fc2d-1f28-050f-adf44c1a0f2e 3a12c92e-fc2d-1f28-050f-adf44c1a0f2e}" +--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.41s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap -testLoggerAdapter verbose: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.127Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b39c6a3-adad-a25e-1c81-808a4efb2831 Address:3b39c6a3-adad-a25e-1c81-808a4efb2831} {Suffrage:Voter ID:server-6be33f16-9a5a-4a59-346e-9f2d05853da4 Address:6be33f16-9a5a-4a59-346e-9f2d05853da4}]" - testing.go:149: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.127Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b39c6a3-adad-a25e-1c81-808a4efb2831 Address:3b39c6a3-adad-a25e-1c81-808a4efb2831} {Suffrage:Voter ID:server-6be33f16-9a5a-4a59-346e-9f2d05853da4 Address:6be33f16-9a5a-4a59-346e-9f2d05853da4}]" -testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.127Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b39c6a3-adad-a25e-1c81-808a4efb2831 Address:3b39c6a3-adad-a25e-1c81-808a4efb2831} {Suffrage:Voter ID:server-6be33f16-9a5a-4a59-346e-9f2d05853da4 Address:6be33f16-9a5a-4a59-346e-9f2d05853da4}]" - testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.127Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b39c6a3-adad-a25e-1c81-808a4efb2831 Address:3b39c6a3-adad-a25e-1c81-808a4efb2831} {Suffrage:Voter ID:server-6be33f16-9a5a-4a59-346e-9f2d05853da4 Address:6be33f16-9a5a-4a59-346e-9f2d05853da4}]" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:09.127Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-01-09T05:10:09.127Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.127Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: entering follower state: follower="Node at 3b39c6a3-adad-a25e-1c81-808a4efb2831 [Follower]" leader= - testing.go:149: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.127Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: entering follower state: follower="Node at 3b39c6a3-adad-a25e-1c81-808a4efb2831 [Follower]" leader= -testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.127Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: entering follower state: follower="Node at 6be33f16-9a5a-4a59-346e-9f2d05853da4 [Follower]" leader= - testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.127Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: entering follower state: follower="Node at 6be33f16-9a5a-4a59-346e-9f2d05853da4 [Follower]" leader= -testLoggerAdapter verbose: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.210Z [WARN] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.210Z [WARN] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.211Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: entering candidate state: node="Node at 3b39c6a3-adad-a25e-1c81-808a4efb2831 [Candidate]" term=2 - testing.go:149: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.211Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: entering candidate state: node="Node at 3b39c6a3-adad-a25e-1c81-808a4efb2831 [Candidate]" term=2 -testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.211Z [WARN] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.211Z [WARN] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.211Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: entering candidate state: node="Node at 6be33f16-9a5a-4a59-346e-9f2d05853da4 [Candidate]" term=2 - testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.211Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: entering candidate state: node="Node at 6be33f16-9a5a-4a59-346e-9f2d05853da4 [Candidate]" term=2 -testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.211Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: duplicate requestVote for same term: term=2 - testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.211Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.211Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: duplicate requestVote for same term: term=2 - testing.go:149: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.211Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [WARN] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: Election timeout reached, restarting election - testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [WARN] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: Election timeout reached, restarting election -testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: entering candidate state: node="Node at 6be33f16-9a5a-4a59-346e-9f2d05853da4 [Candidate]" term=3 - testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: entering candidate state: node="Node at 6be33f16-9a5a-4a59-346e-9f2d05853da4 [Candidate]" term=3 -testLoggerAdapter verbose: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.295Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: entering follower state: follower="Node at 3b39c6a3-adad-a25e-1c81-808a4efb2831 [Follower]" leader= - testing.go:149: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.295Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: entering follower state: follower="Node at 3b39c6a3-adad-a25e-1c81-808a4efb2831 [Follower]" leader= -testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: election won: tally=2 - testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: election won: tally=2 -testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: entering leader state: leader="Node at 6be33f16-9a5a-4a59-346e-9f2d05853da4 [Leader]" - testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: entering leader state: leader="Node at 6be33f16-9a5a-4a59-346e-9f2d05853da4 [Leader]" -testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: added peer, starting replication: peer=server-3b39c6a3-adad-a25e-1c81-808a4efb2831 - testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: added peer, starting replication: peer=server-3b39c6a3-adad-a25e-1c81-808a4efb2831 -testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: pipelining replication: peer="{Voter server-3b39c6a3-adad-a25e-1c81-808a4efb2831 3b39c6a3-adad-a25e-1c81-808a4efb2831}" - testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: pipelining replication: peer="{Voter server-3b39c6a3-adad-a25e-1c81-808a4efb2831 3b39c6a3-adad-a25e-1c81-808a4efb2831}" -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:09.409Z [INFO] stable state for Follower reached at 2024-01-09 05:10:09.295662499 +0000 UTC m=+20.548713546 (1 nodes), 167.777211ms from start of poll, 168.244616ms from cluster start. Timeout at 2024-01-09 05:10:09.409853883 +0000 UTC m=+20.662904995, 114.191449ms after stability - testing.go:149: cluster: 2024-01-09T05:10:09.409Z [INFO] stable state for Follower reached at 2024-01-09 05:10:09.295662499 +0000 UTC m=+20.548713546 (1 nodes), 167.777211ms from start of poll, 168.244616ms from cluster start. Timeout at 2024-01-09 05:10:09.409853883 +0000 UTC m=+20.662904995, 114.191449ms after stability -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:09.409Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-01-09T05:10:09.409Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-01-09T05:10:09.527Z [INFO] stable state for Leader reached at 2024-01-09 05:10:09.410041706 +0000 UTC m=+20.663092804 (1 nodes), 2.58µs from start of poll, 282.623874ms from cluster start. Timeout at 2024-01-09 05:10:09.526937397 +0000 UTC m=+20.779988429, 116.895625ms after stability - testing.go:149: cluster: 2024-01-09T05:10:09.527Z [INFO] stable state for Leader reached at 2024-01-09 05:10:09.410041706 +0000 UTC m=+20.663092804 (1 nodes), 2.58µs from start of poll, 282.623874ms from cluster start. Timeout at 2024-01-09 05:10:09.526937397 +0000 UTC m=+20.779988429, 116.895625ms after stability -testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.542Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b39c6a3-adad-a25e-1c81-808a4efb2831 Address:3b39c6a3-adad-a25e-1c81-808a4efb2831} {Suffrage:Voter ID:server-6be33f16-9a5a-4a59-346e-9f2d05853da4 Address:6be33f16-9a5a-4a59-346e-9f2d05853da4}]" - testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.542Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b39c6a3-adad-a25e-1c81-808a4efb2831 Address:3b39c6a3-adad-a25e-1c81-808a4efb2831} {Suffrage:Voter ID:server-6be33f16-9a5a-4a59-346e-9f2d05853da4 Address:6be33f16-9a5a-4a59-346e-9f2d05853da4}]" -testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.542Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: aborting pipeline replication: peer="{Voter server-3b39c6a3-adad-a25e-1c81-808a4efb2831 3b39c6a3-adad-a25e-1c81-808a4efb2831}" - testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.542Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: aborting pipeline replication: peer="{Voter server-3b39c6a3-adad-a25e-1c81-808a4efb2831 3b39c6a3-adad-a25e-1c81-808a4efb2831}" ---- PASS: TestRaft_GetConfigurationNoBootstrap (0.42s) +testLoggerAdapter verbose: server-5d4aa415-ddd0-1158-abc6-271831494b0a: 2025-02-10T11:34:22.622Z [INFO] server-5d4aa415-ddd0-1158-abc6-271831494b0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5d4aa415-ddd0-1158-abc6-271831494b0a Address:5d4aa415-ddd0-1158-abc6-271831494b0a} {Suffrage:Voter ID:server-ccd79914-4e22-1b53-2c72-90211931b3e5 Address:ccd79914-4e22-1b53-2c72-90211931b3e5}]" + testing.go:149: server-5d4aa415-ddd0-1158-abc6-271831494b0a: 2025-02-10T11:34:22.622Z [INFO] server-5d4aa415-ddd0-1158-abc6-271831494b0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5d4aa415-ddd0-1158-abc6-271831494b0a Address:5d4aa415-ddd0-1158-abc6-271831494b0a} {Suffrage:Voter ID:server-ccd79914-4e22-1b53-2c72-90211931b3e5 Address:ccd79914-4e22-1b53-2c72-90211931b3e5}]" +testLoggerAdapter verbose: server-5d4aa415-ddd0-1158-abc6-271831494b0a: 2025-02-10T11:34:22.622Z [INFO] server-5d4aa415-ddd0-1158-abc6-271831494b0a: entering follower state: follower="Node at 5d4aa415-ddd0-1158-abc6-271831494b0a [Follower]" leader= +testLoggerAdapter verbose: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.622Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5d4aa415-ddd0-1158-abc6-271831494b0a Address:5d4aa415-ddd0-1158-abc6-271831494b0a} {Suffrage:Voter ID:server-ccd79914-4e22-1b53-2c72-90211931b3e5 Address:ccd79914-4e22-1b53-2c72-90211931b3e5}]" + testing.go:149: server-5d4aa415-ddd0-1158-abc6-271831494b0a: 2025-02-10T11:34:22.622Z [INFO] server-5d4aa415-ddd0-1158-abc6-271831494b0a: entering follower state: follower="Node at 5d4aa415-ddd0-1158-abc6-271831494b0a [Follower]" leader= + testing.go:149: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.622Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5d4aa415-ddd0-1158-abc6-271831494b0a Address:5d4aa415-ddd0-1158-abc6-271831494b0a} {Suffrage:Voter ID:server-ccd79914-4e22-1b53-2c72-90211931b3e5 Address:ccd79914-4e22-1b53-2c72-90211931b3e5}]" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:22.622Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2025-02-10T11:34:22.622Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.622Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: entering follower state: follower="Node at ccd79914-4e22-1b53-2c72-90211931b3e5 [Follower]" leader= + testing.go:149: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.622Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: entering follower state: follower="Node at ccd79914-4e22-1b53-2c72-90211931b3e5 [Follower]" leader= +testLoggerAdapter verbose: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.705Z [WARN] server-ccd79914-4e22-1b53-2c72-90211931b3e5: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.705Z [WARN] server-ccd79914-4e22-1b53-2c72-90211931b3e5: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.705Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: entering candidate state: node="Node at ccd79914-4e22-1b53-2c72-90211931b3e5 [Candidate]" term=2 + testing.go:149: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.705Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: entering candidate state: node="Node at ccd79914-4e22-1b53-2c72-90211931b3e5 [Candidate]" term=2 +testLoggerAdapter verbose: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.705Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: election won: tally=2 + testing.go:149: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.705Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: election won: tally=2 +testLoggerAdapter verbose: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.705Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: entering leader state: leader="Node at ccd79914-4e22-1b53-2c72-90211931b3e5 [Leader]" + testing.go:149: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.705Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: entering leader state: leader="Node at ccd79914-4e22-1b53-2c72-90211931b3e5 [Leader]" +testLoggerAdapter verbose: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.705Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: added peer, starting replication: peer=server-5d4aa415-ddd0-1158-abc6-271831494b0a + testing.go:149: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.705Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: added peer, starting replication: peer=server-5d4aa415-ddd0-1158-abc6-271831494b0a +testLoggerAdapter verbose: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.705Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: pipelining replication: peer="{Voter server-5d4aa415-ddd0-1158-abc6-271831494b0a 5d4aa415-ddd0-1158-abc6-271831494b0a}" + testing.go:149: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.705Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: pipelining replication: peer="{Voter server-5d4aa415-ddd0-1158-abc6-271831494b0a 5d4aa415-ddd0-1158-abc6-271831494b0a}" +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:22.810Z [INFO] stable state for Follower reached at 2025-02-10 11:34:22.705849563 +0000 UTC m=+19.074753862 (1 nodes), 83.618531ms from start of poll, 83.895021ms from cluster start. Timeout at 2025-02-10 11:34:22.810947068 +0000 UTC m=+19.179851365, 105.097503ms after stability + testing.go:149: cluster: 2025-02-10T11:34:22.810Z [INFO] stable state for Follower reached at 2025-02-10 11:34:22.705849563 +0000 UTC m=+19.074753862 (1 nodes), 83.618531ms from start of poll, 83.895021ms from cluster start. Timeout at 2025-02-10 11:34:22.810947068 +0000 UTC m=+19.179851365, 105.097503ms after stability +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:22.811Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2025-02-10T11:34:22.811Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2025-02-10T11:34:22.916Z [INFO] stable state for Leader reached at 2025-02-10 11:34:22.811028212 +0000 UTC m=+19.179932513 (1 nodes), 758ns from start of poll, 189.073672ms from cluster start. Timeout at 2025-02-10 11:34:22.916046832 +0000 UTC m=+19.284951142, 105.018629ms after stability + testing.go:149: cluster: 2025-02-10T11:34:22.916Z [INFO] stable state for Leader reached at 2025-02-10 11:34:22.811028212 +0000 UTC m=+19.179932513 (1 nodes), 758ns from start of poll, 189.073672ms from cluster start. Timeout at 2025-02-10 11:34:22.916046832 +0000 UTC m=+19.284951142, 105.018629ms after stability +testLoggerAdapter verbose: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.926Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5d4aa415-ddd0-1158-abc6-271831494b0a Address:5d4aa415-ddd0-1158-abc6-271831494b0a} {Suffrage:Voter ID:server-ccd79914-4e22-1b53-2c72-90211931b3e5 Address:ccd79914-4e22-1b53-2c72-90211931b3e5}]" + testing.go:149: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.926Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5d4aa415-ddd0-1158-abc6-271831494b0a Address:5d4aa415-ddd0-1158-abc6-271831494b0a} {Suffrage:Voter ID:server-ccd79914-4e22-1b53-2c72-90211931b3e5 Address:ccd79914-4e22-1b53-2c72-90211931b3e5}]" +testLoggerAdapter verbose: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.926Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: aborting pipeline replication: peer="{Voter server-5d4aa415-ddd0-1158-abc6-271831494b0a 5d4aa415-ddd0-1158-abc6-271831494b0a}" + testing.go:149: server-ccd79914-4e22-1b53-2c72-90211931b3e5: 2025-02-10T11:34:22.926Z [INFO] server-ccd79914-4e22-1b53-2c72-90211931b3e5: aborting pipeline replication: peer="{Voter server-5d4aa415-ddd0-1158-abc6-271831494b0a 5d4aa415-ddd0-1158-abc6-271831494b0a}" +--- PASS: TestRaft_GetConfigurationNoBootstrap (0.31s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_WithAdvertise @@ -3993,7 +4022,7 @@ === RUN TestBackoff --- PASS: TestBackoff (0.00s) PASS -ok github.com/hashicorp/raft 20.832s +ok github.com/hashicorp/raft 19.306s ? github.com/hashicorp/raft/bench [no test files] create-stamp debian/debhelper-build-stamp fakeroot debian/rules binary @@ -4025,12 +4054,14 @@ dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration +I: user script /srv/workspace/pbuilder/63060/tmp/hooks/B01_cleanup starting +I: user script /srv/workspace/pbuilder/63060/tmp/hooks/B01_cleanup finished I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env -I: removing directory /srv/workspace/pbuilder/904 and its subdirectories -I: Current time: Mon Jan 8 17:10:20 -12 2024 -I: pbuilder-time-stamp: 1704777020 +I: removing directory /srv/workspace/pbuilder/63060 and its subdirectories +I: Current time: Tue Feb 11 01:34:26 +14 2025 +I: pbuilder-time-stamp: 1739187266