Diff of the two buildlogs: -- --- b1/build.log 2023-05-15 14:03:55.060256591 +0000 +++ b2/build.log 2023-05-15 14:06:37.997090750 +0000 @@ -1,6 +1,6 @@ I: pbuilder: network access will be disabled during build -I: Current time: Sun Jun 16 08:24:52 -12 2024 -I: pbuilder-time-stamp: 1718569492 +I: Current time: Tue May 16 04:03:58 +14 2023 +I: pbuilder-time-stamp: 1684159438 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/bookworm-reproducible-base.tgz] I: copying local configuration @@ -16,7 +16,7 @@ I: copying [./golang-github-hashicorp-raft_1.1.2.orig.tar.xz] I: copying [./golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz] I: Extracting source -gpgv: Signature made Sat Jan 18 19:24:28 2020 -12 +gpgv: Signature made Sun Jan 19 21:24:28 2020 +14 gpgv: using RSA key 50BC7CF939D20C272A6B065652B6BBD953968D1B gpgv: Can't check signature: No public key dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.1.2-1.dsc: no acceptable signature found @@ -31,52 +31,84 @@ dpkg-source: info: applying t-skip--TestRaft_SnapshotRestore_PeerChange.patch I: using fakeroot in build. I: Installing the build-deps -I: user script /srv/workspace/pbuilder/1861/tmp/hooks/D02_print_environment starting +I: user script /srv/workspace/pbuilder/12871/tmp/hooks/D01_modify_environment starting +debug: Running on codethink10-arm64. +I: Changing host+domainname to test build reproducibility +I: Adding a custom variable just for the fun of it... +I: Changing /bin/sh to bash +'/bin/sh' -> '/bin/bash' +lrwxrwxrwx 1 root root 9 May 16 04:04 /bin/sh -> /bin/bash +I: Setting pbuilder2's login shell to /bin/bash +I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other +I: user script /srv/workspace/pbuilder/12871/tmp/hooks/D01_modify_environment finished +I: user script /srv/workspace/pbuilder/12871/tmp/hooks/D02_print_environment starting I: set - BUILDDIR='/build' - BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' - BUILDUSERNAME='pbuilder1' - BUILD_ARCH='arm64' - DEBIAN_FRONTEND='noninteractive' + BASH=/bin/sh + BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:globskipdots:hostcomplete:interactive_comments:patsub_replacement:progcomp:promptvars:sourcepath + BASH_ALIASES=() + BASH_ARGC=() + BASH_ARGV=() + BASH_CMDS=() + BASH_LINENO=([0]="12" [1]="0") + BASH_LOADABLES_PATH=/usr/local/lib/bash:/usr/lib/bash:/opt/local/lib/bash:/usr/pkg/lib/bash:/opt/pkg/lib/bash:. + BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment") + BASH_VERSINFO=([0]="5" [1]="2" [2]="15" [3]="1" [4]="release" [5]="aarch64-unknown-linux-gnu") + BASH_VERSION='5.2.15(1)-release' + BUILDDIR=/build + BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other' + BUILDUSERNAME=pbuilder2 + BUILD_ARCH=arm64 + DEBIAN_FRONTEND=noninteractive DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=8' - DISTRIBUTION='bookworm' - HOME='/var/lib/jenkins' - HOST_ARCH='arm64' + DIRSTACK=() + DISTRIBUTION=bookworm + EUID=0 + FUNCNAME=([0]="Echo" [1]="main") + GROUPS=() + HOME=/var/lib/jenkins + HOSTNAME=i-capture-the-hostname + HOSTTYPE=aarch64 + HOST_ARCH=arm64 IFS=' ' - LANG='C' - LANGUAGE='en_US:en' - LC_ALL='C' - MAIL='/var/mail/root' - OPTIND='1' - PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' - PBCURRENTCOMMANDLINEOPERATION='build' - PBUILDER_OPERATION='build' - PBUILDER_PKGDATADIR='/usr/share/pbuilder' - PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' - PBUILDER_SYSCONFDIR='/etc' - PPID='1861' - PS1='# ' - PS2='> ' + LANG=C + LANGUAGE=nl_BE:nl + LC_ALL=C + MACHTYPE=aarch64-unknown-linux-gnu + MAIL=/var/mail/root + OPTERR=1 + OPTIND=1 + OSTYPE=linux-gnu + PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path + PBCURRENTCOMMANDLINEOPERATION=build + PBUILDER_OPERATION=build + PBUILDER_PKGDATADIR=/usr/share/pbuilder + PBUILDER_PKGLIBDIR=/usr/lib/pbuilder + PBUILDER_SYSCONFDIR=/etc + PIPESTATUS=([0]="0") + POSIXLY_CORRECT=y + PPID=12871 PS4='+ ' - PWD='/' - SHELL='/bin/bash' - SHLVL='2' - SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.GKUE5Ueh/pbuilderrc_VmCd --distribution bookworm --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bookworm-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.GKUE5Ueh/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.1.2-1.dsc' - SUDO_GID='117' - SUDO_UID='110' - SUDO_USER='jenkins' - TERM='unknown' - TZ='/usr/share/zoneinfo/Etc/GMT+12' - USER='root' - USERNAME='root' - _='/usr/bin/systemd-run' - http_proxy='http://192.168.101.16:3128' + PWD=/ + SHELL=/bin/bash + SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix + SHLVL=3 + SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.GKUE5Ueh/pbuilderrc_o6zw --distribution bookworm --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bookworm-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.GKUE5Ueh/b2 --logfile b2/build.log --extrapackages usrmerge golang-github-hashicorp-raft_1.1.2-1.dsc' + SUDO_GID=117 + SUDO_UID=110 + SUDO_USER=jenkins + TERM=unknown + TZ=/usr/share/zoneinfo/Etc/GMT-14 + UID=0 + USER=root + USERNAME=root + _='I: set' + http_proxy=http://192.168.101.16:3128 I: uname -a - Linux codethink9-arm64 4.15.0-210-generic #221-Ubuntu SMP Tue Apr 18 08:32:48 UTC 2023 aarch64 GNU/Linux + Linux i-capture-the-hostname 4.15.0-210-generic #221-Ubuntu SMP Tue Apr 18 08:32:48 UTC 2023 aarch64 GNU/Linux I: ls -l /bin - lrwxrwxrwx 1 root root 7 Jun 16 04:48 /bin -> usr/bin -I: user script /srv/workspace/pbuilder/1861/tmp/hooks/D02_print_environment finished + lrwxrwxrwx 1 root root 7 May 15 00:25 /bin -> usr/bin +I: user script /srv/workspace/pbuilder/12871/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy @@ -220,7 +252,7 @@ Get: 96 http://deb.debian.org/debian bookworm/main arm64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 97 http://deb.debian.org/debian bookworm/main arm64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 98 http://deb.debian.org/debian bookworm/main arm64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] -Fetched 112 MB in 4s (26.0 MB/s) +Fetched 112 MB in 2s (49.8 MB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package sensible-utils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19616 files and directories currently installed.) @@ -629,10 +661,15 @@ Reading package lists... Building dependency tree... Reading state information... +usrmerge is already the newest version (35). fakeroot is already the newest version (1.31-1.2). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. I: Building the package -I: Running cd /build/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes +I: user script /srv/workspace/pbuilder/12871/tmp/hooks/A99_set_merged_usr starting +Re-configuring usrmerge... +I: user script /srv/workspace/pbuilder/12871/tmp/hooks/A99_set_merged_usr finished +hostname: Temporary failure in name resolution +I: Running cd /build/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.1.2-1 dpkg-buildpackage: info: source distribution unstable @@ -651,29 +688,29 @@ dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-aarch64-linux-gnu && go install -trimpath -v -p 8 github.com/hashicorp/raft github.com/hashicorp/raft/bench -internal/unsafeheader -internal/goexperiment -internal/goos internal/goarch internal/race -runtime/internal/syscall -sync/atomic -internal/cpu -unicode/utf8 +internal/goos +internal/goexperiment +internal/unsafeheader unicode -internal/abi +internal/cpu +sync/atomic +runtime/internal/syscall runtime/internal/math +internal/abi runtime/internal/sys +unicode/utf8 container/list internal/itoa math/bits crypto/internal/subtle crypto/subtle -crypto/internal/boring/sig internal/bytealg runtime/internal/atomic -encoding +crypto/internal/boring/sig unicode/utf16 +encoding internal/nettrace vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/subtle @@ -681,82 +718,82 @@ runtime internal/reflectlite sync -internal/sysinfo internal/singleflight internal/testlog math/rand +internal/sysinfo runtime/cgo errors sort internal/oserror +io internal/safefilepath path github.com/hashicorp/golang-lru/simplelru -io strconv syscall vendor/golang.org/x/net/dns/dnsmessage crypto/internal/nistec/fiat +bytes strings hash -bytes crypto/internal/randutil hash/crc32 +hash/crc64 +vendor/golang.org/x/text/transform crypto reflect crypto/rc4 -vendor/golang.org/x/text/transform -hash/crc64 -bufio github.com/hashicorp/go-immutable-radix net/http/internal/ascii regexp/syntax +bufio html time internal/syscall/unix internal/syscall/execenv regexp context -internal/poll io/fs +internal/poll embed internal/fmtsort encoding/binary os encoding/base64 +crypto/internal/edwards25519/field +crypto/cipher crypto/md5 vendor/golang.org/x/crypto/curve25519/internal/field -crypto/cipher vendor/golang.org/x/crypto/internal/poly1305 -crypto/internal/edwards25519/field crypto/internal/nistec encoding/pem golang.org/x/sys/unix crypto/des -crypto/internal/boring vendor/golang.org/x/crypto/chacha20 +crypto/internal/boring crypto/internal/edwards25519 -internal/godebug -fmt crypto/aes -internal/intern crypto/sha512 -net/netip crypto/hmac +fmt +internal/godebug crypto/sha1 +internal/intern crypto/sha256 +net/netip path/filepath vendor/golang.org/x/crypto/chacha20poly1305 vendor/golang.org/x/crypto/hkdf os/signal io/ioutil -net -log encoding/json -encoding/hex -compress/flate +log net/url math/big +encoding/hex +compress/flate +net vendor/golang.org/x/crypto/curve25519 vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/text/unicode/norm @@ -766,15 +803,17 @@ mime/quotedprintable net/http/internal vendor/golang.org/x/text/secure/bidirule -github.com/mattn/go-isatty encoding/gob +github.com/mattn/go-isatty go/token github.com/mattn/go-colorable text/template/parse -github.com/fatih/color flag +github.com/fatih/color runtime/debug +vendor/golang.org/x/net/idna github.com/hashicorp/go-hclog +runtime/trace crypto/rand crypto/elliptic crypto/internal/boring/bbig @@ -782,10 +821,8 @@ crypto/ed25519 crypto/rsa crypto/dsa -vendor/golang.org/x/net/idna -runtime/trace -testing text/template +testing crypto/x509/pkix vendor/golang.org/x/crypto/cryptobyte crypto/ecdsa @@ -846,32 +883,32 @@ === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir - testing.go:153: 2024-06-16T08:26:22.998-1200 [INFO] creating new snapshot: path=/tmp/raft2427954899/raft193094651/snapshots/3-10-1718569582998.tmp + testing.go:153: 2023-05-16T04:06:03.312+1400 [INFO] creating new snapshot: path=/tmp/raft3407069546/raft3811143837/snapshots/3-10-1684159563312.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s) === RUN TestFileSS_CreateSnapshot - testing.go:153: 2024-06-16T08:26:22.999-1200 [INFO] creating new snapshot: path=/tmp/raft3117812912/snapshots/3-10-1718569582999.tmp - testing.go:153: 2024-06-16T08:26:23.000-1200 [WARN] found temporary snapshot: name=3-10-1718569582999.tmp + testing.go:153: 2023-05-16T04:06:03.313+1400 [INFO] creating new snapshot: path=/tmp/raft244181675/snapshots/3-10-1684159563313.tmp + testing.go:153: 2023-05-16T04:06:03.314+1400 [WARN] found temporary snapshot: name=3-10-1684159563313.tmp --- PASS: TestFileSS_CreateSnapshot (0.00s) === RUN TestFileSS_CancelSnapshot - testing.go:153: 2024-06-16T08:26:23.001-1200 [INFO] creating new snapshot: path=/tmp/raft3896822289/snapshots/3-10-1718569583001.tmp + testing.go:153: 2023-05-16T04:06:03.315+1400 [INFO] creating new snapshot: path=/tmp/raft1789053147/snapshots/3-10-1684159563315.tmp --- PASS: TestFileSS_CancelSnapshot (0.00s) === RUN TestFileSS_Retention - testing.go:153: 2024-06-16T08:26:23.001-1200 [INFO] creating new snapshot: path=/tmp/raft3216926845/snapshots/3-10-1718569583001.tmp - testing.go:153: 2024-06-16T08:26:23.002-1200 [INFO] creating new snapshot: path=/tmp/raft3216926845/snapshots/3-11-1718569583002.tmp - testing.go:153: 2024-06-16T08:26:23.002-1200 [INFO] creating new snapshot: path=/tmp/raft3216926845/snapshots/3-12-1718569583002.tmp - testing.go:153: 2024-06-16T08:26:23.003-1200 [INFO] reaping snapshot: path=/tmp/raft3216926845/snapshots/3-10-1718569583001 - testing.go:153: 2024-06-16T08:26:23.003-1200 [INFO] creating new snapshot: path=/tmp/raft3216926845/snapshots/3-13-1718569583003.tmp - testing.go:153: 2024-06-16T08:26:23.003-1200 [INFO] reaping snapshot: path=/tmp/raft3216926845/snapshots/3-11-1718569583002 - testing.go:153: 2024-06-16T08:26:23.003-1200 [INFO] creating new snapshot: path=/tmp/raft3216926845/snapshots/3-14-1718569583003.tmp - testing.go:153: 2024-06-16T08:26:23.004-1200 [INFO] reaping snapshot: path=/tmp/raft3216926845/snapshots/3-12-1718569583002 + testing.go:153: 2023-05-16T04:06:03.315+1400 [INFO] creating new snapshot: path=/tmp/raft3323901890/snapshots/3-10-1684159563315.tmp + testing.go:153: 2023-05-16T04:06:03.316+1400 [INFO] creating new snapshot: path=/tmp/raft3323901890/snapshots/3-11-1684159563316.tmp + testing.go:153: 2023-05-16T04:06:03.316+1400 [INFO] creating new snapshot: path=/tmp/raft3323901890/snapshots/3-12-1684159563316.tmp + testing.go:153: 2023-05-16T04:06:03.317+1400 [INFO] reaping snapshot: path=/tmp/raft3323901890/snapshots/3-10-1684159563315 + testing.go:153: 2023-05-16T04:06:03.317+1400 [INFO] creating new snapshot: path=/tmp/raft3323901890/snapshots/3-13-1684159563317.tmp + testing.go:153: 2023-05-16T04:06:03.318+1400 [INFO] reaping snapshot: path=/tmp/raft3323901890/snapshots/3-11-1684159563316 + testing.go:153: 2023-05-16T04:06:03.318+1400 [INFO] creating new snapshot: path=/tmp/raft3323901890/snapshots/3-14-1684159563318.tmp + testing.go:153: 2023-05-16T04:06:03.318+1400 [INFO] reaping snapshot: path=/tmp/raft3323901890/snapshots/3-12-1684159563316 --- PASS: TestFileSS_Retention (0.00s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering - testing.go:153: 2024-06-16T08:26:23.005-1200 [INFO] creating new snapshot: path=/tmp/raft179444173/snapshots/5-130350-1718569583005.tmp - testing.go:153: 2024-06-16T08:26:23.006-1200 [INFO] creating new snapshot: path=/tmp/raft179444173/snapshots/36-204917-1718569583006.tmp + testing.go:153: 2023-05-16T04:06:03.320+1400 [INFO] creating new snapshot: path=/tmp/raft3270018814/snapshots/5-130350-1684159563320.tmp + testing.go:153: 2023-05-16T04:06:03.320+1400 [INFO] creating new snapshot: path=/tmp/raft3270018814/snapshots/36-204917-1684159563320.tmp --- PASS: TestFileSS_Ordering (0.00s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) @@ -896,7 +933,7 @@ === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestNetworkTransport_CloseStreams ---- PASS: TestNetworkTransport_CloseStreams (0.00s) +--- PASS: TestNetworkTransport_CloseStreams (0.08s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath @@ -904,7 +941,7 @@ === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline ---- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s) +--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.01s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams net_transport_test.go:339: Disabled unreliable test --- SKIP: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.00s) @@ -917,29 +954,29 @@ === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn ---- PASS: TestNetworkTransport_PooledConn (0.00s) +--- PASS: TestNetworkTransport_PooledConn (0.01s) === RUN TestNetworkTransport_ListenBackoff - net_transport_test.go:718: countingWriter: 2024-06-16T08:26:23.028-1200 [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-05-16T04:06:03.444+1400 [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-06-16T08:26:23.033-1200 [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-05-16T04:06:03.450+1400 [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-06-16T08:26:23.044-1200 [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-05-16T04:06:03.464+1400 [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-06-16T08:26:23.064-1200 [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-05-16T04:06:03.488+1400 [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-06-16T08:26:23.104-1200 [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-05-16T04:06:03.532+1400 [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-06-16T08:26:23.185-1200 [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-05-16T04:06:03.616+1400 [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-06-16T08:26:23.345-1200 [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-05-16T04:06:03.777+1400 [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-06-16T08:26:23.666-1200 [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-05-16T04:06:04.098+1400 [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-06-16T08:26:24.307-1200 [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-05-16T04:06:04.739+1400 [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-06-16T08:26:25.311-1200 [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-05-16T04:06:05.740+1400 [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-06-16T08:26:26.315-1200 [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-05-16T04:06:06.744+1400 [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration @@ -949,2833 +986,2873 @@ === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop -testLoggerAdapter verbose: server-4cf87f40-e7a2-7d13-6257-4047987d2979: 2024-06-16T08:26:27.032-1200 [INFO] server-4cf87f40-e7a2-7d13-6257-4047987d2979: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cf87f40-e7a2-7d13-6257-4047987d2979 Address:4cf87f40-e7a2-7d13-6257-4047987d2979}]" - testing.go:149: server-4cf87f40-e7a2-7d13-6257-4047987d2979: 2024-06-16T08:26:27.032-1200 [INFO] server-4cf87f40-e7a2-7d13-6257-4047987d2979: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cf87f40-e7a2-7d13-6257-4047987d2979 Address:4cf87f40-e7a2-7d13-6257-4047987d2979}]" +testLoggerAdapter verbose: server-84686a94-bc38-8170-28f0-f77a57ac3307: 2023-05-16T04:06:07.450+1400 [INFO] server-84686a94-bc38-8170-28f0-f77a57ac3307: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84686a94-bc38-8170-28f0-f77a57ac3307 Address:84686a94-bc38-8170-28f0-f77a57ac3307}]" + testing.go:149: server-84686a94-bc38-8170-28f0-f77a57ac3307: 2023-05-16T04:06:07.450+1400 [INFO] server-84686a94-bc38-8170-28f0-f77a57ac3307: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84686a94-bc38-8170-28f0-f77a57ac3307 Address:84686a94-bc38-8170-28f0-f77a57ac3307}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown -testLoggerAdapter verbose: server-4ecc849f-4a26-e961-aa02-7169fdf8546c: 2024-06-16T08:26:27.033-1200 [INFO] server-4ecc849f-4a26-e961-aa02-7169fdf8546c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ecc849f-4a26-e961-aa02-7169fdf8546c Address:4ecc849f-4a26-e961-aa02-7169fdf8546c}]" - testing.go:149: server-4ecc849f-4a26-e961-aa02-7169fdf8546c: 2024-06-16T08:26:27.033-1200 [INFO] server-4ecc849f-4a26-e961-aa02-7169fdf8546c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ecc849f-4a26-e961-aa02-7169fdf8546c Address:4ecc849f-4a26-e961-aa02-7169fdf8546c}]" +testLoggerAdapter verbose: server-f510d0eb-4ab4-b26f-78eb-06f157c1a8b1: 2023-05-16T04:06:07.451+1400 [INFO] server-f510d0eb-4ab4-b26f-78eb-06f157c1a8b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f510d0eb-4ab4-b26f-78eb-06f157c1a8b1 Address:f510d0eb-4ab4-b26f-78eb-06f157c1a8b1}]" + testing.go:149: server-f510d0eb-4ab4-b26f-78eb-06f157c1a8b1: 2023-05-16T04:06:07.451+1400 [INFO] server-f510d0eb-4ab4-b26f-78eb-06f157c1a8b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f510d0eb-4ab4-b26f-78eb-06f157c1a8b1 Address:f510d0eb-4ab4-b26f-78eb-06f157c1a8b1}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap -testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.034-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: initial configuration: index=0 servers=[] - testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.034-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: 2024-06-16T08:26:27.034-1200 [INFO] server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: initial configuration: index=0 servers=[] - testing.go:149: server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: 2024-06-16T08:26:27.034-1200 [INFO] server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.034-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: entering follower state: follower="Node at 9abc4764-99bc-f34e-08f7-0d4ca615e0ce [Follower]" leader= -testLoggerAdapter verbose: server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: 2024-06-16T08:26:27.034-1200 [INFO] server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: entering follower state: follower="Node at 225d14b1-8665-f8dd-15e9-0cf4e9946d71 [Follower]" leader= -testLoggerAdapter verbose: server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: 2024-06-16T08:26:27.034-1200 [INFO] server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: initial configuration: index=0 servers=[] - testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.034-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: entering follower state: follower="Node at 9abc4764-99bc-f34e-08f7-0d4ca615e0ce [Follower]" leader= - testing.go:149: server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: 2024-06-16T08:26:27.034-1200 [INFO] server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: 2024-06-16T08:26:27.034-1200 [INFO] server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: entering follower state: follower="Node at b38698c3-4377-2a9b-1d2d-68dfdf988e0a [Follower]" leader= - testing.go:149: server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: 2024-06-16T08:26:27.034-1200 [INFO] server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: entering follower state: follower="Node at 225d14b1-8665-f8dd-15e9-0cf4e9946d71 [Follower]" leader= - testing.go:149: server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: 2024-06-16T08:26:27.034-1200 [INFO] server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: entering follower state: follower="Node at b38698c3-4377-2a9b-1d2d-68dfdf988e0a [Follower]" leader= -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.034-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:27.034-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: 2024-06-16T08:26:27.090-1200 [WARN] server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: no known peers, aborting election - testing.go:149: server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: 2024-06-16T08:26:27.090-1200 [WARN] server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: no known peers, aborting election -testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [WARN] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [WARN] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: entering candidate state: node="Node at 9abc4764-99bc-f34e-08f7-0d4ca615e0ce [Candidate]" term=2 - testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: entering candidate state: node="Node at 9abc4764-99bc-f34e-08f7-0d4ca615e0ce [Candidate]" term=2 -testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: election won: tally=2 - testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: election won: tally=2 -testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: entering leader state: leader="Node at 9abc4764-99bc-f34e-08f7-0d4ca615e0ce [Leader]" - testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: entering leader state: leader="Node at 9abc4764-99bc-f34e-08f7-0d4ca615e0ce [Leader]" -testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: added peer, starting replication: peer=server-225d14b1-8665-f8dd-15e9-0cf4e9946d71 - testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: added peer, starting replication: peer=server-225d14b1-8665-f8dd-15e9-0cf4e9946d71 -testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: added peer, starting replication: peer=server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a - testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: added peer, starting replication: peer=server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a -testLoggerAdapter verbose: server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: 2024-06-16T08:26:27.093-1200 [WARN] server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: failed to get previous log: previous-index=1 last-index=0 error="log not found" -testLoggerAdapter verbose: server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: 2024-06-16T08:26:27.093-1200 [WARN] server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: failed to get previous log: previous-index=1 last-index=0 error="log not found" - testing.go:149: server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: 2024-06-16T08:26:27.093-1200 [WARN] server-225d14b1-8665-f8dd-15e9-0cf4e9946d71: failed to get previous log: previous-index=1 last-index=0 error="log not found" - testing.go:149: server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: 2024-06-16T08:26:27.093-1200 [WARN] server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a: failed to get previous log: previous-index=1 last-index=0 error="log not found" -testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [WARN] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: appendEntries rejected, sending older logs: peer="{Voter server-225d14b1-8665-f8dd-15e9-0cf4e9946d71 225d14b1-8665-f8dd-15e9-0cf4e9946d71}" next=1 - testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [WARN] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: appendEntries rejected, sending older logs: peer="{Voter server-225d14b1-8665-f8dd-15e9-0cf4e9946d71 225d14b1-8665-f8dd-15e9-0cf4e9946d71}" next=1 -testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [WARN] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: appendEntries rejected, sending older logs: peer="{Voter server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a b38698c3-4377-2a9b-1d2d-68dfdf988e0a}" next=1 - testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.093-1200 [WARN] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: appendEntries rejected, sending older logs: peer="{Voter server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a b38698c3-4377-2a9b-1d2d-68dfdf988e0a}" next=1 -testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.094-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: pipelining replication: peer="{Voter server-225d14b1-8665-f8dd-15e9-0cf4e9946d71 225d14b1-8665-f8dd-15e9-0cf4e9946d71}" - testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.094-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: pipelining replication: peer="{Voter server-225d14b1-8665-f8dd-15e9-0cf4e9946d71 225d14b1-8665-f8dd-15e9-0cf4e9946d71}" -testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.094-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: pipelining replication: peer="{Voter server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a b38698c3-4377-2a9b-1d2d-68dfdf988e0a}" - testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.094-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: pipelining replication: peer="{Voter server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a b38698c3-4377-2a9b-1d2d-68dfdf988e0a}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.199-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:27.09363038 -1200 -12 m=+4.100946861 (2 nodes), 58.8971ms from start of poll, 59.5852ms from cluster start. Timeout at 2024-06-16 08:26:27.19915354 -1200 -12 m=+4.206470021, 105.52316ms after stability - testing.go:149: cluster: 2024-06-16T08:26:27.199-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:27.09363038 -1200 -12 m=+4.100946861 (2 nodes), 58.8971ms from start of poll, 59.5852ms from cluster start. Timeout at 2024-06-16 08:26:27.19915354 -1200 -12 m=+4.206470021, 105.52316ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.199-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:27.199-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.305-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:27.19941434 -1200 -12 m=+4.206730801 (1 nodes), 1.56µs from start of poll, 165.36914ms from cluster start. Timeout at 2024-06-16 08:26:27.30525452 -1200 -12 m=+4.312571001, 105.8402ms after stability - testing.go:149: cluster: 2024-06-16T08:26:27.305-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:27.19941434 -1200 -12 m=+4.206730801 (1 nodes), 1.56µs from start of poll, 165.36914ms from cluster start. Timeout at 2024-06-16 08:26:27.30525452 -1200 -12 m=+4.312571001, 105.8402ms after stability -testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.316-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: aborting pipeline replication: peer="{Voter server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a b38698c3-4377-2a9b-1d2d-68dfdf988e0a}" - testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.316-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: aborting pipeline replication: peer="{Voter server-b38698c3-4377-2a9b-1d2d-68dfdf988e0a b38698c3-4377-2a9b-1d2d-68dfdf988e0a}" -testLoggerAdapter verbose: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.316-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: aborting pipeline replication: peer="{Voter server-225d14b1-8665-f8dd-15e9-0cf4e9946d71 225d14b1-8665-f8dd-15e9-0cf4e9946d71}" - testing.go:149: server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: 2024-06-16T08:26:27.316-1200 [INFO] server-9abc4764-99bc-f34e-08f7-0d4ca615e0ce: aborting pipeline replication: peer="{Voter server-225d14b1-8665-f8dd-15e9-0cf4e9946d71 225d14b1-8665-f8dd-15e9-0cf4e9946d71}" ---- PASS: TestRaft_LiveBootstrap (0.28s) +testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.452+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: initial configuration: index=0 servers=[] + testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.452+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-f9281b59-ecbf-f154-4840-0818345c6687: 2023-05-16T04:06:07.452+1400 [INFO] server-f9281b59-ecbf-f154-4840-0818345c6687: initial configuration: index=0 servers=[] + testing.go:149: server-f9281b59-ecbf-f154-4840-0818345c6687: 2023-05-16T04:06:07.452+1400 [INFO] server-f9281b59-ecbf-f154-4840-0818345c6687: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.452+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: entering follower state: follower="Node at 08e2fec0-69f3-a1c8-e2ae-6910759f4c8d [Follower]" leader= + testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.452+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: entering follower state: follower="Node at 08e2fec0-69f3-a1c8-e2ae-6910759f4c8d [Follower]" leader= +testLoggerAdapter verbose: server-f9281b59-ecbf-f154-4840-0818345c6687: 2023-05-16T04:06:07.452+1400 [INFO] server-f9281b59-ecbf-f154-4840-0818345c6687: entering follower state: follower="Node at f9281b59-ecbf-f154-4840-0818345c6687 [Follower]" leader= + testing.go:149: server-f9281b59-ecbf-f154-4840-0818345c6687: 2023-05-16T04:06:07.452+1400 [INFO] server-f9281b59-ecbf-f154-4840-0818345c6687: entering follower state: follower="Node at f9281b59-ecbf-f154-4840-0818345c6687 [Follower]" leader= +testLoggerAdapter verbose: server-c1b136b0-48f0-5622-80f5-752005b26c00: 2023-05-16T04:06:07.452+1400 [INFO] server-c1b136b0-48f0-5622-80f5-752005b26c00: initial configuration: index=0 servers=[] + testing.go:149: server-c1b136b0-48f0-5622-80f5-752005b26c00: 2023-05-16T04:06:07.452+1400 [INFO] server-c1b136b0-48f0-5622-80f5-752005b26c00: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:07.452+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:07.452+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-c1b136b0-48f0-5622-80f5-752005b26c00: 2023-05-16T04:06:07.452+1400 [INFO] server-c1b136b0-48f0-5622-80f5-752005b26c00: entering follower state: follower="Node at c1b136b0-48f0-5622-80f5-752005b26c00 [Follower]" leader= + testing.go:149: server-c1b136b0-48f0-5622-80f5-752005b26c00: 2023-05-16T04:06:07.452+1400 [INFO] server-c1b136b0-48f0-5622-80f5-752005b26c00: entering follower state: follower="Node at c1b136b0-48f0-5622-80f5-752005b26c00 [Follower]" leader= +testLoggerAdapter verbose: server-c1b136b0-48f0-5622-80f5-752005b26c00: 2023-05-16T04:06:07.511+1400 [WARN] server-c1b136b0-48f0-5622-80f5-752005b26c00: no known peers, aborting election + testing.go:149: server-c1b136b0-48f0-5622-80f5-752005b26c00: 2023-05-16T04:06:07.511+1400 [WARN] server-c1b136b0-48f0-5622-80f5-752005b26c00: no known peers, aborting election +testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.530+1400 [WARN] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.530+1400 [WARN] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.530+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: entering candidate state: node="Node at 08e2fec0-69f3-a1c8-e2ae-6910759f4c8d [Candidate]" term=2 + testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.530+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: entering candidate state: node="Node at 08e2fec0-69f3-a1c8-e2ae-6910759f4c8d [Candidate]" term=2 +testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: election won: tally=2 + testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: election won: tally=2 +testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: entering leader state: leader="Node at 08e2fec0-69f3-a1c8-e2ae-6910759f4c8d [Leader]" + testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: entering leader state: leader="Node at 08e2fec0-69f3-a1c8-e2ae-6910759f4c8d [Leader]" +testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: added peer, starting replication: peer=server-f9281b59-ecbf-f154-4840-0818345c6687 + testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: added peer, starting replication: peer=server-f9281b59-ecbf-f154-4840-0818345c6687 +testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: added peer, starting replication: peer=server-c1b136b0-48f0-5622-80f5-752005b26c00 + testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: added peer, starting replication: peer=server-c1b136b0-48f0-5622-80f5-752005b26c00 +testLoggerAdapter verbose: server-c1b136b0-48f0-5622-80f5-752005b26c00: 2023-05-16T04:06:07.531+1400 [WARN] server-c1b136b0-48f0-5622-80f5-752005b26c00: failed to get previous log: previous-index=1 last-index=0 error="log not found" + testing.go:149: server-c1b136b0-48f0-5622-80f5-752005b26c00: 2023-05-16T04:06:07.531+1400 [WARN] server-c1b136b0-48f0-5622-80f5-752005b26c00: failed to get previous log: previous-index=1 last-index=0 error="log not found" +testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [WARN] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: appendEntries rejected, sending older logs: peer="{Voter server-c1b136b0-48f0-5622-80f5-752005b26c00 c1b136b0-48f0-5622-80f5-752005b26c00}" next=1 + testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [WARN] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: appendEntries rejected, sending older logs: peer="{Voter server-c1b136b0-48f0-5622-80f5-752005b26c00 c1b136b0-48f0-5622-80f5-752005b26c00}" next=1 +testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: pipelining replication: peer="{Voter server-c1b136b0-48f0-5622-80f5-752005b26c00 c1b136b0-48f0-5622-80f5-752005b26c00}" + testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.531+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: pipelining replication: peer="{Voter server-c1b136b0-48f0-5622-80f5-752005b26c00 c1b136b0-48f0-5622-80f5-752005b26c00}" +testLoggerAdapter verbose: server-f9281b59-ecbf-f154-4840-0818345c6687: 2023-05-16T04:06:07.532+1400 [WARN] server-f9281b59-ecbf-f154-4840-0818345c6687: failed to get previous log: previous-index=1 last-index=0 error="log not found" + testing.go:149: server-f9281b59-ecbf-f154-4840-0818345c6687: 2023-05-16T04:06:07.532+1400 [WARN] server-f9281b59-ecbf-f154-4840-0818345c6687: failed to get previous log: previous-index=1 last-index=0 error="log not found" +testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.532+1400 [WARN] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: appendEntries rejected, sending older logs: peer="{Voter server-f9281b59-ecbf-f154-4840-0818345c6687 f9281b59-ecbf-f154-4840-0818345c6687}" next=1 + testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.532+1400 [WARN] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: appendEntries rejected, sending older logs: peer="{Voter server-f9281b59-ecbf-f154-4840-0818345c6687 f9281b59-ecbf-f154-4840-0818345c6687}" next=1 +testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.532+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: pipelining replication: peer="{Voter server-f9281b59-ecbf-f154-4840-0818345c6687 f9281b59-ecbf-f154-4840-0818345c6687}" + testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.532+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: pipelining replication: peer="{Voter server-f9281b59-ecbf-f154-4840-0818345c6687 f9281b59-ecbf-f154-4840-0818345c6687}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:07.637+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:07.532024362 +1400 +14 m=+4.334605997 (2 nodes), 79.147234ms from start of poll, 79.944063ms from cluster start. Timeout at 2023-05-16 04:06:07.637868271 +1400 +14 m=+4.440449906, 105.843909ms after stability + testing.go:149: cluster: 2023-05-16T04:06:07.637+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:07.532024362 +1400 +14 m=+4.334605997 (2 nodes), 79.147234ms from start of poll, 79.944063ms from cluster start. Timeout at 2023-05-16 04:06:07.637868271 +1400 +14 m=+4.440449906, 105.843909ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:07.638+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:07.638+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:07.744+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:07.638175755 +1400 +14 m=+4.440757390 (1 nodes), 1.9µs from start of poll, 186.095456ms from cluster start. Timeout at 2023-05-16 04:06:07.744056605 +1400 +14 m=+4.546638240, 105.88085ms after stability + testing.go:149: cluster: 2023-05-16T04:06:07.744+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:07.638175755 +1400 +14 m=+4.440757390 (1 nodes), 1.9µs from start of poll, 186.095456ms from cluster start. Timeout at 2023-05-16 04:06:07.744056605 +1400 +14 m=+4.546638240, 105.88085ms after stability +testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.755+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: aborting pipeline replication: peer="{Voter server-c1b136b0-48f0-5622-80f5-752005b26c00 c1b136b0-48f0-5622-80f5-752005b26c00}" + testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.755+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: aborting pipeline replication: peer="{Voter server-c1b136b0-48f0-5622-80f5-752005b26c00 c1b136b0-48f0-5622-80f5-752005b26c00}" +testLoggerAdapter verbose: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.755+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: aborting pipeline replication: peer="{Voter server-f9281b59-ecbf-f154-4840-0818345c6687 f9281b59-ecbf-f154-4840-0818345c6687}" + testing.go:149: server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: 2023-05-16T04:06:07.755+1400 [INFO] server-08e2fec0-69f3-a1c8-e2ae-6910759f4c8d: aborting pipeline replication: peer="{Voter server-f9281b59-ecbf-f154-4840-0818345c6687 f9281b59-ecbf-f154-4840-0818345c6687}" +--- PASS: TestRaft_LiveBootstrap (0.30s) === RUN TestRaft_RecoverCluster_NoState -testLoggerAdapter verbose: server-49d40753-db4d-ac6a-8b5a-e434493f3691: 2024-06-16T08:26:27.317-1200 [INFO] server-49d40753-db4d-ac6a-8b5a-e434493f3691: initial configuration: index=0 servers=[] - testing.go:149: server-49d40753-db4d-ac6a-8b5a-e434493f3691: 2024-06-16T08:26:27.317-1200 [INFO] server-49d40753-db4d-ac6a-8b5a-e434493f3691: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-49d40753-db4d-ac6a-8b5a-e434493f3691: 2024-06-16T08:26:27.317-1200 [INFO] server-49d40753-db4d-ac6a-8b5a-e434493f3691: entering follower state: follower="Node at 49d40753-db4d-ac6a-8b5a-e434493f3691 [Follower]" leader= - testing.go:149: server-49d40753-db4d-ac6a-8b5a-e434493f3691: 2024-06-16T08:26:27.317-1200 [INFO] server-49d40753-db4d-ac6a-8b5a-e434493f3691: entering follower state: follower="Node at 49d40753-db4d-ac6a-8b5a-e434493f3691 [Follower]" leader= +testLoggerAdapter verbose: server-3717b61b-db44-903d-3823-d8fa15255f74: 2023-05-16T04:06:07.756+1400 [INFO] server-3717b61b-db44-903d-3823-d8fa15255f74: initial configuration: index=0 servers=[] + testing.go:149: server-3717b61b-db44-903d-3823-d8fa15255f74: 2023-05-16T04:06:07.756+1400 [INFO] server-3717b61b-db44-903d-3823-d8fa15255f74: initial configuration: index=0 servers=[] --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster raft_test.go:119: Disabled unreliable test --- SKIP: TestRaft_RecoverCluster (0.00s) === RUN TestRaft_HasExistingState -testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.318-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-053d0ff9-3367-ed8b-7005-639a89cfda6a Address:053d0ff9-3367-ed8b-7005-639a89cfda6a} {Suffrage:Voter ID:server-39203453-104e-4506-2072-0b11fde5e926 Address:39203453-104e-4506-2072-0b11fde5e926}]" - testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.318-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-053d0ff9-3367-ed8b-7005-639a89cfda6a Address:053d0ff9-3367-ed8b-7005-639a89cfda6a} {Suffrage:Voter ID:server-39203453-104e-4506-2072-0b11fde5e926 Address:39203453-104e-4506-2072-0b11fde5e926}]" -testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.318-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: entering follower state: follower="Node at 053d0ff9-3367-ed8b-7005-639a89cfda6a [Follower]" leader= - testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.318-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: entering follower state: follower="Node at 053d0ff9-3367-ed8b-7005-639a89cfda6a [Follower]" leader= -testLoggerAdapter verbose: server-39203453-104e-4506-2072-0b11fde5e926: 2024-06-16T08:26:27.319-1200 [INFO] server-39203453-104e-4506-2072-0b11fde5e926: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-053d0ff9-3367-ed8b-7005-639a89cfda6a Address:053d0ff9-3367-ed8b-7005-639a89cfda6a} {Suffrage:Voter ID:server-39203453-104e-4506-2072-0b11fde5e926 Address:39203453-104e-4506-2072-0b11fde5e926}]" - testing.go:149: server-39203453-104e-4506-2072-0b11fde5e926: 2024-06-16T08:26:27.319-1200 [INFO] server-39203453-104e-4506-2072-0b11fde5e926: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-053d0ff9-3367-ed8b-7005-639a89cfda6a Address:053d0ff9-3367-ed8b-7005-639a89cfda6a} {Suffrage:Voter ID:server-39203453-104e-4506-2072-0b11fde5e926 Address:39203453-104e-4506-2072-0b11fde5e926}]" -testLoggerAdapter verbose: server-39203453-104e-4506-2072-0b11fde5e926: 2024-06-16T08:26:27.319-1200 [INFO] server-39203453-104e-4506-2072-0b11fde5e926: entering follower state: follower="Node at 39203453-104e-4506-2072-0b11fde5e926 [Follower]" leader= - testing.go:149: server-39203453-104e-4506-2072-0b11fde5e926: 2024-06-16T08:26:27.319-1200 [INFO] server-39203453-104e-4506-2072-0b11fde5e926: entering follower state: follower="Node at 39203453-104e-4506-2072-0b11fde5e926 [Follower]" leader= -testLoggerAdapter verbose: server-3681658f-1671-150f-ad54-42c5b997b434: 2024-06-16T08:26:27.319-1200 [INFO] server-3681658f-1671-150f-ad54-42c5b997b434: initial configuration: index=0 servers=[] - testing.go:149: server-3681658f-1671-150f-ad54-42c5b997b434: 2024-06-16T08:26:27.319-1200 [INFO] server-3681658f-1671-150f-ad54-42c5b997b434: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-3681658f-1671-150f-ad54-42c5b997b434: 2024-06-16T08:26:27.320-1200 [INFO] server-3681658f-1671-150f-ad54-42c5b997b434: entering follower state: follower="Node at 3681658f-1671-150f-ad54-42c5b997b434 [Follower]" leader= - testing.go:149: server-3681658f-1671-150f-ad54-42c5b997b434: 2024-06-16T08:26:27.320-1200 [INFO] server-3681658f-1671-150f-ad54-42c5b997b434: entering follower state: follower="Node at 3681658f-1671-150f-ad54-42c5b997b434 [Follower]" leader= -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.320-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:27.320-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-3681658f-1671-150f-ad54-42c5b997b434: 2024-06-16T08:26:27.378-1200 [WARN] server-3681658f-1671-150f-ad54-42c5b997b434: no known peers, aborting election - testing.go:149: server-3681658f-1671-150f-ad54-42c5b997b434: 2024-06-16T08:26:27.378-1200 [WARN] server-3681658f-1671-150f-ad54-42c5b997b434: no known peers, aborting election -testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.385-1200 [WARN] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.385-1200 [WARN] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.385-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: entering candidate state: node="Node at 053d0ff9-3367-ed8b-7005-639a89cfda6a [Candidate]" term=2 - testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.385-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: entering candidate state: node="Node at 053d0ff9-3367-ed8b-7005-639a89cfda6a [Candidate]" term=2 -testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.386-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: election won: tally=2 - testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.386-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: election won: tally=2 -testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.386-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: entering leader state: leader="Node at 053d0ff9-3367-ed8b-7005-639a89cfda6a [Leader]" - testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.386-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: entering leader state: leader="Node at 053d0ff9-3367-ed8b-7005-639a89cfda6a [Leader]" -testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.386-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: added peer, starting replication: peer=server-39203453-104e-4506-2072-0b11fde5e926 - testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.386-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: added peer, starting replication: peer=server-39203453-104e-4506-2072-0b11fde5e926 -testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.386-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: pipelining replication: peer="{Voter server-39203453-104e-4506-2072-0b11fde5e926 39203453-104e-4506-2072-0b11fde5e926}" - testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.386-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: pipelining replication: peer="{Voter server-39203453-104e-4506-2072-0b11fde5e926 39203453-104e-4506-2072-0b11fde5e926}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.492-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:27.38632488 -1200 -12 m=+4.393641361 (1 nodes), 65.99142ms from start of poll, 68.17602ms from cluster start. Timeout at 2024-06-16 08:26:27.49223536 -1200 -12 m=+4.499551841, 105.91048ms after stability - testing.go:149: cluster: 2024-06-16T08:26:27.492-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:27.38632488 -1200 -12 m=+4.393641361 (1 nodes), 65.99142ms from start of poll, 68.17602ms from cluster start. Timeout at 2024-06-16 08:26:27.49223536 -1200 -12 m=+4.499551841, 105.91048ms after stability -testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.492-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: updating configuration: command=AddStaging server-id=server-3681658f-1671-150f-ad54-42c5b997b434 server-addr=3681658f-1671-150f-ad54-42c5b997b434 servers="[{Suffrage:Voter ID:server-053d0ff9-3367-ed8b-7005-639a89cfda6a Address:053d0ff9-3367-ed8b-7005-639a89cfda6a} {Suffrage:Voter ID:server-39203453-104e-4506-2072-0b11fde5e926 Address:39203453-104e-4506-2072-0b11fde5e926} {Suffrage:Voter ID:server-3681658f-1671-150f-ad54-42c5b997b434 Address:3681658f-1671-150f-ad54-42c5b997b434}]" - testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.492-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: updating configuration: command=AddStaging server-id=server-3681658f-1671-150f-ad54-42c5b997b434 server-addr=3681658f-1671-150f-ad54-42c5b997b434 servers="[{Suffrage:Voter ID:server-053d0ff9-3367-ed8b-7005-639a89cfda6a Address:053d0ff9-3367-ed8b-7005-639a89cfda6a} {Suffrage:Voter ID:server-39203453-104e-4506-2072-0b11fde5e926 Address:39203453-104e-4506-2072-0b11fde5e926} {Suffrage:Voter ID:server-3681658f-1671-150f-ad54-42c5b997b434 Address:3681658f-1671-150f-ad54-42c5b997b434}]" -testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.492-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: added peer, starting replication: peer=server-3681658f-1671-150f-ad54-42c5b997b434 - testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.492-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: added peer, starting replication: peer=server-3681658f-1671-150f-ad54-42c5b997b434 -testLoggerAdapter verbose: server-3681658f-1671-150f-ad54-42c5b997b434: 2024-06-16T08:26:27.492-1200 [WARN] server-3681658f-1671-150f-ad54-42c5b997b434: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-3681658f-1671-150f-ad54-42c5b997b434: 2024-06-16T08:26:27.492-1200 [WARN] server-3681658f-1671-150f-ad54-42c5b997b434: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.492-1200 [WARN] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: appendEntries rejected, sending older logs: peer="{Voter server-3681658f-1671-150f-ad54-42c5b997b434 3681658f-1671-150f-ad54-42c5b997b434}" next=1 - testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.492-1200 [WARN] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: appendEntries rejected, sending older logs: peer="{Voter server-3681658f-1671-150f-ad54-42c5b997b434 3681658f-1671-150f-ad54-42c5b997b434}" next=1 -testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.493-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: pipelining replication: peer="{Voter server-3681658f-1671-150f-ad54-42c5b997b434 3681658f-1671-150f-ad54-42c5b997b434}" - testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.493-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: pipelining replication: peer="{Voter server-3681658f-1671-150f-ad54-42c5b997b434 3681658f-1671-150f-ad54-42c5b997b434}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.498-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:27.498-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.604-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:27.49873366 -1200 -12 m=+4.506050141 (1 nodes), 1.36µs from start of poll, 180.5848ms from cluster start. Timeout at 2024-06-16 08:26:27.6047072 -1200 -12 m=+4.612023681, 105.97354ms after stability - testing.go:149: cluster: 2024-06-16T08:26:27.604-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:27.49873366 -1200 -12 m=+4.506050141 (1 nodes), 1.36µs from start of poll, 180.5848ms from cluster start. Timeout at 2024-06-16 08:26:27.6047072 -1200 -12 m=+4.612023681, 105.97354ms after stability -testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.604-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: aborting pipeline replication: peer="{Voter server-39203453-104e-4506-2072-0b11fde5e926 39203453-104e-4506-2072-0b11fde5e926}" - testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.604-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: aborting pipeline replication: peer="{Voter server-39203453-104e-4506-2072-0b11fde5e926 39203453-104e-4506-2072-0b11fde5e926}" -testLoggerAdapter verbose: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.604-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: aborting pipeline replication: peer="{Voter server-3681658f-1671-150f-ad54-42c5b997b434 3681658f-1671-150f-ad54-42c5b997b434}" - testing.go:149: server-053d0ff9-3367-ed8b-7005-639a89cfda6a: 2024-06-16T08:26:27.604-1200 [INFO] server-053d0ff9-3367-ed8b-7005-639a89cfda6a: aborting pipeline replication: peer="{Voter server-3681658f-1671-150f-ad54-42c5b997b434 3681658f-1671-150f-ad54-42c5b997b434}" ---- PASS: TestRaft_HasExistingState (0.29s) +testLoggerAdapter verbose: server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70: 2023-05-16T04:06:07.758+1400 [INFO] server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 Address:30a8f5ff-33cb-0492-35ae-e9bad2aa8a70} {Suffrage:Voter ID:server-0dc5a37e-8893-7781-d777-c558123ef2dd Address:0dc5a37e-8893-7781-d777-c558123ef2dd}]" + testing.go:149: server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70: 2023-05-16T04:06:07.758+1400 [INFO] server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 Address:30a8f5ff-33cb-0492-35ae-e9bad2aa8a70} {Suffrage:Voter ID:server-0dc5a37e-8893-7781-d777-c558123ef2dd Address:0dc5a37e-8893-7781-d777-c558123ef2dd}]" +testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.758+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 Address:30a8f5ff-33cb-0492-35ae-e9bad2aa8a70} {Suffrage:Voter ID:server-0dc5a37e-8893-7781-d777-c558123ef2dd Address:0dc5a37e-8893-7781-d777-c558123ef2dd}]" + testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.758+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 Address:30a8f5ff-33cb-0492-35ae-e9bad2aa8a70} {Suffrage:Voter ID:server-0dc5a37e-8893-7781-d777-c558123ef2dd Address:0dc5a37e-8893-7781-d777-c558123ef2dd}]" +testLoggerAdapter verbose: server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: 2023-05-16T04:06:07.758+1400 [INFO] server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: initial configuration: index=0 servers=[] + testing.go:149: server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: 2023-05-16T04:06:07.758+1400 [INFO] server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:07.758+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:07.758+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70: 2023-05-16T04:06:07.758+1400 [INFO] server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70: entering follower state: follower="Node at 30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 [Follower]" leader= + testing.go:149: server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70: 2023-05-16T04:06:07.758+1400 [INFO] server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70: entering follower state: follower="Node at 30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 [Follower]" leader= +testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.759+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: entering follower state: follower="Node at 0dc5a37e-8893-7781-d777-c558123ef2dd [Follower]" leader= + testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.759+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: entering follower state: follower="Node at 0dc5a37e-8893-7781-d777-c558123ef2dd [Follower]" leader= +testLoggerAdapter verbose: server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: 2023-05-16T04:06:07.759+1400 [INFO] server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: entering follower state: follower="Node at 5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 [Follower]" leader= + testing.go:149: server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: 2023-05-16T04:06:07.759+1400 [INFO] server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: entering follower state: follower="Node at 5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 [Follower]" leader= +testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [WARN] server-0dc5a37e-8893-7781-d777-c558123ef2dd: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [WARN] server-0dc5a37e-8893-7781-d777-c558123ef2dd: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: entering candidate state: node="Node at 0dc5a37e-8893-7781-d777-c558123ef2dd [Candidate]" term=2 + testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: entering candidate state: node="Node at 0dc5a37e-8893-7781-d777-c558123ef2dd [Candidate]" term=2 +testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: election won: tally=2 + testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: election won: tally=2 +testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: entering leader state: leader="Node at 0dc5a37e-8893-7781-d777-c558123ef2dd [Leader]" + testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: entering leader state: leader="Node at 0dc5a37e-8893-7781-d777-c558123ef2dd [Leader]" +testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: added peer, starting replication: peer=server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 + testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.823+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: added peer, starting replication: peer=server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 +testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.824+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: pipelining replication: peer="{Voter server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 30a8f5ff-33cb-0492-35ae-e9bad2aa8a70}" + testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.824+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: pipelining replication: peer="{Voter server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 30a8f5ff-33cb-0492-35ae-e9bad2aa8a70}" +testLoggerAdapter verbose: server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: 2023-05-16T04:06:07.827+1400 [WARN] server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: no known peers, aborting election + testing.go:149: server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: 2023-05-16T04:06:07.827+1400 [WARN] server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: no known peers, aborting election +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:07.929+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:07.823959428 +1400 +14 m=+4.626541063 (1 nodes), 65.054088ms from start of poll, 66.128741ms from cluster start. Timeout at 2023-05-16 04:06:07.929297351 +1400 +14 m=+4.731878986, 105.337923ms after stability + testing.go:149: cluster: 2023-05-16T04:06:07.929+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:07.823959428 +1400 +14 m=+4.626541063 (1 nodes), 65.054088ms from start of poll, 66.128741ms from cluster start. Timeout at 2023-05-16 04:06:07.929297351 +1400 +14 m=+4.731878986, 105.337923ms after stability +testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.929+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: updating configuration: command=AddStaging server-id=server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 server-addr=5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 servers="[{Suffrage:Voter ID:server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 Address:30a8f5ff-33cb-0492-35ae-e9bad2aa8a70} {Suffrage:Voter ID:server-0dc5a37e-8893-7781-d777-c558123ef2dd Address:0dc5a37e-8893-7781-d777-c558123ef2dd} {Suffrage:Voter ID:server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 Address:5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69}]" + testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.929+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: updating configuration: command=AddStaging server-id=server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 server-addr=5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 servers="[{Suffrage:Voter ID:server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 Address:30a8f5ff-33cb-0492-35ae-e9bad2aa8a70} {Suffrage:Voter ID:server-0dc5a37e-8893-7781-d777-c558123ef2dd Address:0dc5a37e-8893-7781-d777-c558123ef2dd} {Suffrage:Voter ID:server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 Address:5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69}]" +testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.929+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: added peer, starting replication: peer=server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 + testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.929+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: added peer, starting replication: peer=server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 +testLoggerAdapter verbose: server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: 2023-05-16T04:06:07.930+1400 [WARN] server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: 2023-05-16T04:06:07.930+1400 [WARN] server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.930+1400 [WARN] server-0dc5a37e-8893-7781-d777-c558123ef2dd: appendEntries rejected, sending older logs: peer="{Voter server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69}" next=1 + testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.930+1400 [WARN] server-0dc5a37e-8893-7781-d777-c558123ef2dd: appendEntries rejected, sending older logs: peer="{Voter server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69}" next=1 +testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.930+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: pipelining replication: peer="{Voter server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69}" + testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:07.930+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: pipelining replication: peer="{Voter server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:07.930+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:07.930+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.036+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:07.930718028 +1400 +14 m=+4.733299663 (1 nodes), 1.4µs from start of poll, 172.887341ms from cluster start. Timeout at 2023-05-16 04:06:08.035940709 +1400 +14 m=+4.838522364, 105.222701ms after stability + testing.go:149: cluster: 2023-05-16T04:06:08.036+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:07.930718028 +1400 +14 m=+4.733299663 (1 nodes), 1.4µs from start of poll, 172.887341ms from cluster start. Timeout at 2023-05-16 04:06:08.035940709 +1400 +14 m=+4.838522364, 105.222701ms after stability +testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:08.036+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: aborting pipeline replication: peer="{Voter server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69}" + testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:08.036+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: aborting pipeline replication: peer="{Voter server-5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69 5dd1ed0a-e2df-56e0-cc9c-9e49c3519e69}" +testLoggerAdapter verbose: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:08.036+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: aborting pipeline replication: peer="{Voter server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 30a8f5ff-33cb-0492-35ae-e9bad2aa8a70}" + testing.go:149: server-0dc5a37e-8893-7781-d777-c558123ef2dd: 2023-05-16T04:06:08.036+1400 [INFO] server-0dc5a37e-8893-7781-d777-c558123ef2dd: aborting pipeline replication: peer="{Voter server-30a8f5ff-33cb-0492-35ae-e9bad2aa8a70 30a8f5ff-33cb-0492-35ae-e9bad2aa8a70}" +--- PASS: TestRaft_HasExistingState (0.28s) === RUN TestRaft_SingleNode -testLoggerAdapter verbose: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.606-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3420e22-7732-1e08-338b-a49595ae2e5f Address:e3420e22-7732-1e08-338b-a49595ae2e5f}]" - testing.go:149: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.606-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3420e22-7732-1e08-338b-a49595ae2e5f Address:e3420e22-7732-1e08-338b-a49595ae2e5f}]" -testLoggerAdapter verbose: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.606-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: entering follower state: follower="Node at e3420e22-7732-1e08-338b-a49595ae2e5f [Follower]" leader= - testing.go:149: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.606-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: entering follower state: follower="Node at e3420e22-7732-1e08-338b-a49595ae2e5f [Follower]" leader= -testLoggerAdapter verbose: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.673-1200 [WARN] server-e3420e22-7732-1e08-338b-a49595ae2e5f: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.673-1200 [WARN] server-e3420e22-7732-1e08-338b-a49595ae2e5f: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.673-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: entering candidate state: node="Node at e3420e22-7732-1e08-338b-a49595ae2e5f [Candidate]" term=2 - testing.go:149: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.673-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: entering candidate state: node="Node at e3420e22-7732-1e08-338b-a49595ae2e5f [Candidate]" term=2 -testLoggerAdapter verbose: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.673-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: election won: tally=1 - testing.go:149: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.673-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: election won: tally=1 -testLoggerAdapter verbose: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.673-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: entering leader state: leader="Node at e3420e22-7732-1e08-338b-a49595ae2e5f [Leader]" - testing.go:149: server-e3420e22-7732-1e08-338b-a49595ae2e5f: 2024-06-16T08:26:27.673-1200 [INFO] server-e3420e22-7732-1e08-338b-a49595ae2e5f: entering leader state: leader="Node at e3420e22-7732-1e08-338b-a49595ae2e5f [Leader]" +testLoggerAdapter verbose: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.037+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35 Address:ca2d9894-d5b0-e52c-3865-c5408b8cdc35}]" + testing.go:149: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.037+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35 Address:ca2d9894-d5b0-e52c-3865-c5408b8cdc35}]" +testLoggerAdapter verbose: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.038+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: entering follower state: follower="Node at ca2d9894-d5b0-e52c-3865-c5408b8cdc35 [Follower]" leader= + testing.go:149: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.038+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: entering follower state: follower="Node at ca2d9894-d5b0-e52c-3865-c5408b8cdc35 [Follower]" leader= +testLoggerAdapter verbose: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.109+1400 [WARN] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.109+1400 [WARN] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.109+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: entering candidate state: node="Node at ca2d9894-d5b0-e52c-3865-c5408b8cdc35 [Candidate]" term=2 + testing.go:149: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.109+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: entering candidate state: node="Node at ca2d9894-d5b0-e52c-3865-c5408b8cdc35 [Candidate]" term=2 +testLoggerAdapter verbose: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.109+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: election won: tally=1 + testing.go:149: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.109+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: election won: tally=1 +testLoggerAdapter verbose: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.110+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: entering leader state: leader="Node at ca2d9894-d5b0-e52c-3865-c5408b8cdc35 [Leader]" + testing.go:149: server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: 2023-05-16T04:06:08.110+1400 [INFO] server-ca2d9894-d5b0-e52c-3865-c5408b8cdc35: entering leader state: leader="Node at ca2d9894-d5b0-e52c-3865-c5408b8cdc35 [Leader]" --- PASS: TestRaft_SingleNode (0.07s) === RUN TestRaft_TripleNode -testLoggerAdapter verbose: server-3a9ba939-cc84-618a-6a50-b9f80f122c65: 2024-06-16T08:26:27.675-1200 [INFO] server-3a9ba939-cc84-618a-6a50-b9f80f122c65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9ba939-cc84-618a-6a50-b9f80f122c65 Address:3a9ba939-cc84-618a-6a50-b9f80f122c65} {Suffrage:Voter ID:server-7d5daf0f-9c93-a2b1-7850-866aee4da412 Address:7d5daf0f-9c93-a2b1-7850-866aee4da412} {Suffrage:Voter ID:server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d Address:7b68fe81-8f04-e1e0-66fd-6430782d3d4d}]" - testing.go:149: server-3a9ba939-cc84-618a-6a50-b9f80f122c65: 2024-06-16T08:26:27.675-1200 [INFO] server-3a9ba939-cc84-618a-6a50-b9f80f122c65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9ba939-cc84-618a-6a50-b9f80f122c65 Address:3a9ba939-cc84-618a-6a50-b9f80f122c65} {Suffrage:Voter ID:server-7d5daf0f-9c93-a2b1-7850-866aee4da412 Address:7d5daf0f-9c93-a2b1-7850-866aee4da412} {Suffrage:Voter ID:server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d Address:7b68fe81-8f04-e1e0-66fd-6430782d3d4d}]" -testLoggerAdapter verbose: server-3a9ba939-cc84-618a-6a50-b9f80f122c65: 2024-06-16T08:26:27.675-1200 [INFO] server-3a9ba939-cc84-618a-6a50-b9f80f122c65: entering follower state: follower="Node at 3a9ba939-cc84-618a-6a50-b9f80f122c65 [Follower]" leader= - testing.go:149: server-3a9ba939-cc84-618a-6a50-b9f80f122c65: 2024-06-16T08:26:27.675-1200 [INFO] server-3a9ba939-cc84-618a-6a50-b9f80f122c65: entering follower state: follower="Node at 3a9ba939-cc84-618a-6a50-b9f80f122c65 [Follower]" leader= -testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.675-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9ba939-cc84-618a-6a50-b9f80f122c65 Address:3a9ba939-cc84-618a-6a50-b9f80f122c65} {Suffrage:Voter ID:server-7d5daf0f-9c93-a2b1-7850-866aee4da412 Address:7d5daf0f-9c93-a2b1-7850-866aee4da412} {Suffrage:Voter ID:server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d Address:7b68fe81-8f04-e1e0-66fd-6430782d3d4d}]" - testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.675-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9ba939-cc84-618a-6a50-b9f80f122c65 Address:3a9ba939-cc84-618a-6a50-b9f80f122c65} {Suffrage:Voter ID:server-7d5daf0f-9c93-a2b1-7850-866aee4da412 Address:7d5daf0f-9c93-a2b1-7850-866aee4da412} {Suffrage:Voter ID:server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d Address:7b68fe81-8f04-e1e0-66fd-6430782d3d4d}]" -testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.675-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: entering follower state: follower="Node at 7d5daf0f-9c93-a2b1-7850-866aee4da412 [Follower]" leader= - testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.675-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: entering follower state: follower="Node at 7d5daf0f-9c93-a2b1-7850-866aee4da412 [Follower]" leader= -testLoggerAdapter verbose: server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d: 2024-06-16T08:26:27.675-1200 [INFO] server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9ba939-cc84-618a-6a50-b9f80f122c65 Address:3a9ba939-cc84-618a-6a50-b9f80f122c65} {Suffrage:Voter ID:server-7d5daf0f-9c93-a2b1-7850-866aee4da412 Address:7d5daf0f-9c93-a2b1-7850-866aee4da412} {Suffrage:Voter ID:server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d Address:7b68fe81-8f04-e1e0-66fd-6430782d3d4d}]" - testing.go:149: server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d: 2024-06-16T08:26:27.675-1200 [INFO] server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9ba939-cc84-618a-6a50-b9f80f122c65 Address:3a9ba939-cc84-618a-6a50-b9f80f122c65} {Suffrage:Voter ID:server-7d5daf0f-9c93-a2b1-7850-866aee4da412 Address:7d5daf0f-9c93-a2b1-7850-866aee4da412} {Suffrage:Voter ID:server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d Address:7b68fe81-8f04-e1e0-66fd-6430782d3d4d}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.675-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:27.675-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d: 2024-06-16T08:26:27.675-1200 [INFO] server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d: entering follower state: follower="Node at 7b68fe81-8f04-e1e0-66fd-6430782d3d4d [Follower]" leader= - testing.go:149: server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d: 2024-06-16T08:26:27.675-1200 [INFO] server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d: entering follower state: follower="Node at 7b68fe81-8f04-e1e0-66fd-6430782d3d4d [Follower]" leader= -testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.745-1200 [WARN] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.745-1200 [WARN] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: entering candidate state: node="Node at 7d5daf0f-9c93-a2b1-7850-866aee4da412 [Candidate]" term=2 - testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: entering candidate state: node="Node at 7d5daf0f-9c93-a2b1-7850-866aee4da412 [Candidate]" term=2 -testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: election won: tally=2 - testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: election won: tally=2 -testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: entering leader state: leader="Node at 7d5daf0f-9c93-a2b1-7850-866aee4da412 [Leader]" - testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: entering leader state: leader="Node at 7d5daf0f-9c93-a2b1-7850-866aee4da412 [Leader]" -testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: added peer, starting replication: peer=server-3a9ba939-cc84-618a-6a50-b9f80f122c65 - testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: added peer, starting replication: peer=server-3a9ba939-cc84-618a-6a50-b9f80f122c65 -testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: added peer, starting replication: peer=server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d - testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: added peer, starting replication: peer=server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d -testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: pipelining replication: peer="{Voter server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d 7b68fe81-8f04-e1e0-66fd-6430782d3d4d}" - testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: pipelining replication: peer="{Voter server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d 7b68fe81-8f04-e1e0-66fd-6430782d3d4d}" -testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: pipelining replication: peer="{Voter server-3a9ba939-cc84-618a-6a50-b9f80f122c65 3a9ba939-cc84-618a-6a50-b9f80f122c65}" - testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.746-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: pipelining replication: peer="{Voter server-3a9ba939-cc84-618a-6a50-b9f80f122c65 3a9ba939-cc84-618a-6a50-b9f80f122c65}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.851-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:27.74642138 -1200 -12 m=+4.753737981 (2 nodes), 70.5818ms from start of poll, 71.43196ms from cluster start. Timeout at 2024-06-16 08:26:27.85147534 -1200 -12 m=+4.858791841, 105.05386ms after stability - testing.go:149: cluster: 2024-06-16T08:26:27.851-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:27.74642138 -1200 -12 m=+4.753737981 (2 nodes), 70.5818ms from start of poll, 71.43196ms from cluster start. Timeout at 2024-06-16 08:26:27.85147534 -1200 -12 m=+4.858791841, 105.05386ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.851-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:27.851-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.957-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:27.85172458 -1200 -12 m=+4.859041061 (1 nodes), 2µs from start of poll, 176.73504ms from cluster start. Timeout at 2024-06-16 08:26:27.95747648 -1200 -12 m=+4.964792941, 105.75188ms after stability - testing.go:149: cluster: 2024-06-16T08:26:27.957-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:27.85172458 -1200 -12 m=+4.859041061 (1 nodes), 2µs from start of poll, 176.73504ms from cluster start. Timeout at 2024-06-16 08:26:27.95747648 -1200 -12 m=+4.964792941, 105.75188ms after stability -testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.973-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: aborting pipeline replication: peer="{Voter server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d 7b68fe81-8f04-e1e0-66fd-6430782d3d4d}" - testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.973-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: aborting pipeline replication: peer="{Voter server-7b68fe81-8f04-e1e0-66fd-6430782d3d4d 7b68fe81-8f04-e1e0-66fd-6430782d3d4d}" -testLoggerAdapter verbose: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.973-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: aborting pipeline replication: peer="{Voter server-3a9ba939-cc84-618a-6a50-b9f80f122c65 3a9ba939-cc84-618a-6a50-b9f80f122c65}" - testing.go:149: server-7d5daf0f-9c93-a2b1-7850-866aee4da412: 2024-06-16T08:26:27.973-1200 [INFO] server-7d5daf0f-9c93-a2b1-7850-866aee4da412: aborting pipeline replication: peer="{Voter server-3a9ba939-cc84-618a-6a50-b9f80f122c65 3a9ba939-cc84-618a-6a50-b9f80f122c65}" ---- PASS: TestRaft_TripleNode (0.30s) +testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.111+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e8c35a3-9043-5c93-e6a1-352462a09cc2 Address:9e8c35a3-9043-5c93-e6a1-352462a09cc2} {Suffrage:Voter ID:server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 Address:ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480} {Suffrage:Voter ID:server-d512c739-8ebb-7e56-8d0d-5440bf285c29 Address:d512c739-8ebb-7e56-8d0d-5440bf285c29}]" + testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.111+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e8c35a3-9043-5c93-e6a1-352462a09cc2 Address:9e8c35a3-9043-5c93-e6a1-352462a09cc2} {Suffrage:Voter ID:server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 Address:ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480} {Suffrage:Voter ID:server-d512c739-8ebb-7e56-8d0d-5440bf285c29 Address:d512c739-8ebb-7e56-8d0d-5440bf285c29}]" +testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.111+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: entering follower state: follower="Node at 9e8c35a3-9043-5c93-e6a1-352462a09cc2 [Follower]" leader= + testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.111+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: entering follower state: follower="Node at 9e8c35a3-9043-5c93-e6a1-352462a09cc2 [Follower]" leader= +testLoggerAdapter verbose: server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480: 2023-05-16T04:06:08.111+1400 [INFO] server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e8c35a3-9043-5c93-e6a1-352462a09cc2 Address:9e8c35a3-9043-5c93-e6a1-352462a09cc2} {Suffrage:Voter ID:server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 Address:ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480} {Suffrage:Voter ID:server-d512c739-8ebb-7e56-8d0d-5440bf285c29 Address:d512c739-8ebb-7e56-8d0d-5440bf285c29}]" + testing.go:149: server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480: 2023-05-16T04:06:08.111+1400 [INFO] server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e8c35a3-9043-5c93-e6a1-352462a09cc2 Address:9e8c35a3-9043-5c93-e6a1-352462a09cc2} {Suffrage:Voter ID:server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 Address:ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480} {Suffrage:Voter ID:server-d512c739-8ebb-7e56-8d0d-5440bf285c29 Address:d512c739-8ebb-7e56-8d0d-5440bf285c29}]" +testLoggerAdapter verbose: server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480: 2023-05-16T04:06:08.111+1400 [INFO] server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480: entering follower state: follower="Node at ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 [Follower]" leader= + testing.go:149: server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480: 2023-05-16T04:06:08.111+1400 [INFO] server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480: entering follower state: follower="Node at ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 [Follower]" leader= +testLoggerAdapter verbose: server-d512c739-8ebb-7e56-8d0d-5440bf285c29: 2023-05-16T04:06:08.112+1400 [INFO] server-d512c739-8ebb-7e56-8d0d-5440bf285c29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e8c35a3-9043-5c93-e6a1-352462a09cc2 Address:9e8c35a3-9043-5c93-e6a1-352462a09cc2} {Suffrage:Voter ID:server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 Address:ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480} {Suffrage:Voter ID:server-d512c739-8ebb-7e56-8d0d-5440bf285c29 Address:d512c739-8ebb-7e56-8d0d-5440bf285c29}]" + testing.go:149: server-d512c739-8ebb-7e56-8d0d-5440bf285c29: 2023-05-16T04:06:08.112+1400 [INFO] server-d512c739-8ebb-7e56-8d0d-5440bf285c29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e8c35a3-9043-5c93-e6a1-352462a09cc2 Address:9e8c35a3-9043-5c93-e6a1-352462a09cc2} {Suffrage:Voter ID:server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 Address:ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480} {Suffrage:Voter ID:server-d512c739-8ebb-7e56-8d0d-5440bf285c29 Address:d512c739-8ebb-7e56-8d0d-5440bf285c29}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.112+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:08.112+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-d512c739-8ebb-7e56-8d0d-5440bf285c29: 2023-05-16T04:06:08.112+1400 [INFO] server-d512c739-8ebb-7e56-8d0d-5440bf285c29: entering follower state: follower="Node at d512c739-8ebb-7e56-8d0d-5440bf285c29 [Follower]" leader= + testing.go:149: server-d512c739-8ebb-7e56-8d0d-5440bf285c29: 2023-05-16T04:06:08.112+1400 [INFO] server-d512c739-8ebb-7e56-8d0d-5440bf285c29: entering follower state: follower="Node at d512c739-8ebb-7e56-8d0d-5440bf285c29 [Follower]" leader= +testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [WARN] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [WARN] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: entering candidate state: node="Node at 9e8c35a3-9043-5c93-e6a1-352462a09cc2 [Candidate]" term=2 + testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: entering candidate state: node="Node at 9e8c35a3-9043-5c93-e6a1-352462a09cc2 [Candidate]" term=2 +testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: election won: tally=2 + testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: election won: tally=2 +testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: entering leader state: leader="Node at 9e8c35a3-9043-5c93-e6a1-352462a09cc2 [Leader]" + testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: entering leader state: leader="Node at 9e8c35a3-9043-5c93-e6a1-352462a09cc2 [Leader]" +testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: added peer, starting replication: peer=server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 + testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: added peer, starting replication: peer=server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 +testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: added peer, starting replication: peer=server-d512c739-8ebb-7e56-8d0d-5440bf285c29 + testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: added peer, starting replication: peer=server-d512c739-8ebb-7e56-8d0d-5440bf285c29 +testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: pipelining replication: peer="{Voter server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480}" + testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: pipelining replication: peer="{Voter server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480}" +testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: pipelining replication: peer="{Voter server-d512c739-8ebb-7e56-8d0d-5440bf285c29 d512c739-8ebb-7e56-8d0d-5440bf285c29}" + testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.179+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: pipelining replication: peer="{Voter server-d512c739-8ebb-7e56-8d0d-5440bf285c29 d512c739-8ebb-7e56-8d0d-5440bf285c29}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.285+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:08.179678926 +1400 +14 m=+4.982260561 (2 nodes), 67.405116ms from start of poll, 68.440488ms from cluster start. Timeout at 2023-05-16 04:06:08.284991869 +1400 +14 m=+5.087573484, 105.312923ms after stability + testing.go:149: cluster: 2023-05-16T04:06:08.285+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:08.179678926 +1400 +14 m=+4.982260561 (2 nodes), 67.405116ms from start of poll, 68.440488ms from cluster start. Timeout at 2023-05-16 04:06:08.284991869 +1400 +14 m=+5.087573484, 105.312923ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.285+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:08.285+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.390+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:08.285246352 +1400 +14 m=+5.087827987 (1 nodes), 2.24µs from start of poll, 174.007914ms from cluster start. Timeout at 2023-05-16 04:06:08.390862878 +1400 +14 m=+5.193444493, 105.616506ms after stability + testing.go:149: cluster: 2023-05-16T04:06:08.390+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:08.285246352 +1400 +14 m=+5.087827987 (1 nodes), 2.24µs from start of poll, 174.007914ms from cluster start. Timeout at 2023-05-16 04:06:08.390862878 +1400 +14 m=+5.193444493, 105.616506ms after stability +testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.403+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: aborting pipeline replication: peer="{Voter server-d512c739-8ebb-7e56-8d0d-5440bf285c29 d512c739-8ebb-7e56-8d0d-5440bf285c29}" + testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.403+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: aborting pipeline replication: peer="{Voter server-d512c739-8ebb-7e56-8d0d-5440bf285c29 d512c739-8ebb-7e56-8d0d-5440bf285c29}" +testLoggerAdapter verbose: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.403+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: aborting pipeline replication: peer="{Voter server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480}" + testing.go:149: server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: 2023-05-16T04:06:08.403+1400 [INFO] server-9e8c35a3-9043-5c93-e6a1-352462a09cc2: aborting pipeline replication: peer="{Voter server-ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480 ce2a5a3c-ec0d-f590-ba92-1b6cbdf57480}" +--- PASS: TestRaft_TripleNode (0.29s) === RUN TestRaft_LeaderFail -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:27.975-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6085a72a-2d6f-0e11-9b40-c701837dc407 Address:6085a72a-2d6f-0e11-9b40-c701837dc407} {Suffrage:Voter ID:server-d7fde817-1e05-8df2-4277-e2c3b1ce543c Address:d7fde817-1e05-8df2-4277-e2c3b1ce543c} {Suffrage:Voter ID:server-ee94f1af-0538-dd75-0868-3267cffc729b Address:ee94f1af-0538-dd75-0868-3267cffc729b}]" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:27.975-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6085a72a-2d6f-0e11-9b40-c701837dc407 Address:6085a72a-2d6f-0e11-9b40-c701837dc407} {Suffrage:Voter ID:server-d7fde817-1e05-8df2-4277-e2c3b1ce543c Address:d7fde817-1e05-8df2-4277-e2c3b1ce543c} {Suffrage:Voter ID:server-ee94f1af-0538-dd75-0868-3267cffc729b Address:ee94f1af-0538-dd75-0868-3267cffc729b}]" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:27.975-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering follower state: follower="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Follower]" leader= - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:27.975-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering follower state: follower="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Follower]" leader= -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:27.975-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6085a72a-2d6f-0e11-9b40-c701837dc407 Address:6085a72a-2d6f-0e11-9b40-c701837dc407} {Suffrage:Voter ID:server-d7fde817-1e05-8df2-4277-e2c3b1ce543c Address:d7fde817-1e05-8df2-4277-e2c3b1ce543c} {Suffrage:Voter ID:server-ee94f1af-0538-dd75-0868-3267cffc729b Address:ee94f1af-0538-dd75-0868-3267cffc729b}]" - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:27.975-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6085a72a-2d6f-0e11-9b40-c701837dc407 Address:6085a72a-2d6f-0e11-9b40-c701837dc407} {Suffrage:Voter ID:server-d7fde817-1e05-8df2-4277-e2c3b1ce543c Address:d7fde817-1e05-8df2-4277-e2c3b1ce543c} {Suffrage:Voter ID:server-ee94f1af-0538-dd75-0868-3267cffc729b Address:ee94f1af-0538-dd75-0868-3267cffc729b}]" -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:27.975-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: entering follower state: follower="Node at d7fde817-1e05-8df2-4277-e2c3b1ce543c [Follower]" leader= - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:27.975-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: entering follower state: follower="Node at d7fde817-1e05-8df2-4277-e2c3b1ce543c [Follower]" leader= -testLoggerAdapter verbose: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:27.975-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6085a72a-2d6f-0e11-9b40-c701837dc407 Address:6085a72a-2d6f-0e11-9b40-c701837dc407} {Suffrage:Voter ID:server-d7fde817-1e05-8df2-4277-e2c3b1ce543c Address:d7fde817-1e05-8df2-4277-e2c3b1ce543c} {Suffrage:Voter ID:server-ee94f1af-0538-dd75-0868-3267cffc729b Address:ee94f1af-0538-dd75-0868-3267cffc729b}]" - testing.go:149: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:27.975-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6085a72a-2d6f-0e11-9b40-c701837dc407 Address:6085a72a-2d6f-0e11-9b40-c701837dc407} {Suffrage:Voter ID:server-d7fde817-1e05-8df2-4277-e2c3b1ce543c Address:d7fde817-1e05-8df2-4277-e2c3b1ce543c} {Suffrage:Voter ID:server-ee94f1af-0538-dd75-0868-3267cffc729b Address:ee94f1af-0538-dd75-0868-3267cffc729b}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:27.975-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:27.975-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:27.975-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: entering follower state: follower="Node at ee94f1af-0538-dd75-0868-3267cffc729b [Follower]" leader= - testing.go:149: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:27.975-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: entering follower state: follower="Node at ee94f1af-0538-dd75-0868-3267cffc729b [Follower]" leader= -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering candidate state: node="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Candidate]" term=2 - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering candidate state: node="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Candidate]" term=2 -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: election won: tally=2 - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: election won: tally=2 -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering leader state: leader="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Leader]" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering leader state: leader="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Leader]" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: added peer, starting replication: peer=server-d7fde817-1e05-8df2-4277-e2c3b1ce543c - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: added peer, starting replication: peer=server-d7fde817-1e05-8df2-4277-e2c3b1ce543c -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: added peer, starting replication: peer=server-ee94f1af-0538-dd75-0868-3267cffc729b - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.036-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: added peer, starting replication: peer=server-ee94f1af-0538-dd75-0868-3267cffc729b -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.037-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: pipelining replication: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.037-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: pipelining replication: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.037-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: pipelining replication: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.037-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: pipelining replication: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.142-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:28.03693486 -1200 -12 m=+5.044251361 (2 nodes), 61.01946ms from start of poll, 61.93708ms from cluster start. Timeout at 2024-06-16 08:26:28.14278572 -1200 -12 m=+5.150102201, 105.85084ms after stability - testing.go:149: cluster: 2024-06-16T08:26:28.142-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:28.03693486 -1200 -12 m=+5.044251361 (2 nodes), 61.01946ms from start of poll, 61.93708ms from cluster start. Timeout at 2024-06-16 08:26:28.14278572 -1200 -12 m=+5.150102201, 105.85084ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.142-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:28.142-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.248-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.14299162 -1200 -12 m=+5.150308081 (1 nodes), 2.24µs from start of poll, 167.9938ms from cluster start. Timeout at 2024-06-16 08:26:28.24832532 -1200 -12 m=+5.255641821, 105.33374ms after stability - testing.go:149: cluster: 2024-06-16T08:26:28.248-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.14299162 -1200 -12 m=+5.150308081 (1 nodes), 2.24µs from start of poll, 167.9938ms from cluster start. Timeout at 2024-06-16 08:26:28.24832532 -1200 -12 m=+5.255641821, 105.33374ms after stability - raft_test.go:336: [INFO] Disconnecting Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Leader] -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.264-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=d7fde817-1e05-8df2-4277-e2c3b1ce543c error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.264-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=d7fde817-1e05-8df2-4277-e2c3b1ce543c error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.265-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:28.265-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to pipeline appendEntries: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="append pipeline closed" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to pipeline appendEntries: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="append pipeline closed" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to pipeline appendEntries: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="append pipeline closed" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to pipeline appendEntries: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="append pipeline closed" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=ee94f1af-0538-dd75-0868-3267cffc729b error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=ee94f1af-0538-dd75-0868-3267cffc729b error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: aborting pipeline replication: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: aborting pipeline replication: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: aborting pipeline replication: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.267-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: aborting pipeline replication: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.278-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.278-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.278-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.278-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.281-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=d7fde817-1e05-8df2-4277-e2c3b1ce543c error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.281-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=d7fde817-1e05-8df2-4277-e2c3b1ce543c error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.286-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=ee94f1af-0538-dd75-0868-3267cffc729b error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.286-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=ee94f1af-0538-dd75-0868-3267cffc729b error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.296-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.296-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.297-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.297-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.301-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=d7fde817-1e05-8df2-4277-e2c3b1ce543c error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.301-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=d7fde817-1e05-8df2-4277-e2c3b1ce543c error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.307-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=ee94f1af-0538-dd75-0868-3267cffc729b error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.307-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to heartbeat to: peer=ee94f1af-0538-dd75-0868-3267cffc729b error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.308-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to contact: server-id=server-d7fde817-1e05-8df2-4277-e2c3b1ce543c time=50.12326ms - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.308-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to contact: server-id=server-d7fde817-1e05-8df2-4277-e2c3b1ce543c time=50.12326ms -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.308-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to contact: server-id=server-ee94f1af-0538-dd75-0868-3267cffc729b time=50.14058ms - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.308-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to contact: server-id=server-ee94f1af-0538-dd75-0868-3267cffc729b time=50.14058ms -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.308-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to contact quorum of nodes, stepping down - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.308-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to contact quorum of nodes, stepping down -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.308-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering follower state: follower="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Follower]" leader= - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.308-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering follower state: follower="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Follower]" leader= -testLoggerAdapter verbose: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.309-1200 [WARN] server-ee94f1af-0538-dd75-0868-3267cffc729b: heartbeat timeout reached, starting election: last-leader=6085a72a-2d6f-0e11-9b40-c701837dc407 - testing.go:149: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.309-1200 [WARN] server-ee94f1af-0538-dd75-0868-3267cffc729b: heartbeat timeout reached, starting election: last-leader=6085a72a-2d6f-0e11-9b40-c701837dc407 -testLoggerAdapter verbose: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.309-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: entering candidate state: node="Node at ee94f1af-0538-dd75-0868-3267cffc729b [Candidate]" term=3 - testing.go:149: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.309-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: entering candidate state: node="Node at ee94f1af-0538-dd75-0868-3267cffc729b [Candidate]" term=3 -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.309-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: rejecting vote request since we have a leader: from=ee94f1af-0538-dd75-0868-3267cffc729b leader=6085a72a-2d6f-0e11-9b40-c701837dc407 - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.309-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: rejecting vote request since we have a leader: from=ee94f1af-0538-dd75-0868-3267cffc729b leader=6085a72a-2d6f-0e11-9b40-c701837dc407 -testLoggerAdapter verbose: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.309-1200 [ERROR] server-ee94f1af-0538-dd75-0868-3267cffc729b: failed to make requestVote RPC: target="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" - testing.go:149: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.309-1200 [ERROR] server-ee94f1af-0538-dd75-0868-3267cffc729b: failed to make requestVote RPC: target="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.316-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.316-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.316-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.316-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to appendEntries to: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.342-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: heartbeat timeout reached, starting election: last-leader=6085a72a-2d6f-0e11-9b40-c701837dc407 - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.342-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: heartbeat timeout reached, starting election: last-leader=6085a72a-2d6f-0e11-9b40-c701837dc407 -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.342-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: entering candidate state: node="Node at d7fde817-1e05-8df2-4277-e2c3b1ce543c [Candidate]" term=3 - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.342-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: entering candidate state: node="Node at d7fde817-1e05-8df2-4277-e2c3b1ce543c [Candidate]" term=3 -testLoggerAdapter verbose: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.342-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: duplicate requestVote for same term: term=3 - testing.go:149: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.342-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: duplicate requestVote for same term: term=3 -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.342-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to make requestVote RPC: target="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.342-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to make requestVote RPC: target="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.395-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: Election timeout reached, restarting election - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.395-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: Election timeout reached, restarting election -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.395-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: entering candidate state: node="Node at d7fde817-1e05-8df2-4277-e2c3b1ce543c [Candidate]" term=4 - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.395-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: entering candidate state: node="Node at d7fde817-1e05-8df2-4277-e2c3b1ce543c [Candidate]" term=4 -testLoggerAdapter verbose: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.396-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: entering follower state: follower="Node at ee94f1af-0538-dd75-0868-3267cffc729b [Follower]" leader= - testing.go:149: server-ee94f1af-0538-dd75-0868-3267cffc729b: 2024-06-16T08:26:28.396-1200 [INFO] server-ee94f1af-0538-dd75-0868-3267cffc729b: entering follower state: follower="Node at ee94f1af-0538-dd75-0868-3267cffc729b [Follower]" leader= -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: election won: tally=2 - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: election won: tally=2 -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: entering leader state: leader="Node at d7fde817-1e05-8df2-4277-e2c3b1ce543c [Leader]" - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: entering leader state: leader="Node at d7fde817-1e05-8df2-4277-e2c3b1ce543c [Leader]" -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: added peer, starting replication: peer=server-6085a72a-2d6f-0e11-9b40-c701837dc407 - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: added peer, starting replication: peer=server-6085a72a-2d6f-0e11-9b40-c701837dc407 -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: added peer, starting replication: peer=server-ee94f1af-0538-dd75-0868-3267cffc729b - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: added peer, starting replication: peer=server-ee94f1af-0538-dd75-0868-3267cffc729b -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: pipelining replication: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: pipelining replication: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to make requestVote RPC: target="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to make requestVote RPC: target="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to appendEntries to: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.396-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to appendEntries to: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.402-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.402-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.402-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering candidate state: node="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Candidate]" term=3 - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.402-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering candidate state: node="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Candidate]" term=3 -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.402-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to make requestVote RPC: target="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.402-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to make requestVote RPC: target="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.402-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to make requestVote RPC: target="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.402-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to make requestVote RPC: target="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.406-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to heartbeat to: peer=6085a72a-2d6f-0e11-9b40-c701837dc407 error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.406-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to heartbeat to: peer=6085a72a-2d6f-0e11-9b40-c701837dc407 error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.417-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to appendEntries to: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.417-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to appendEntries to: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.423-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to heartbeat to: peer=6085a72a-2d6f-0e11-9b40-c701837dc407 error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.423-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to heartbeat to: peer=6085a72a-2d6f-0e11-9b40-c701837dc407 error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.436-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to appendEntries to: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.436-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to appendEntries to: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.443-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to heartbeat to: peer=6085a72a-2d6f-0e11-9b40-c701837dc407 error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.443-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to heartbeat to: peer=6085a72a-2d6f-0e11-9b40-c701837dc407 error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.447-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to contact: server-id=server-6085a72a-2d6f-0e11-9b40-c701837dc407 time=50.84708ms - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.447-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to contact: server-id=server-6085a72a-2d6f-0e11-9b40-c701837dc407 time=50.84708ms -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.466-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to appendEntries to: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.466-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to appendEntries to: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.473-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to heartbeat to: peer=6085a72a-2d6f-0e11-9b40-c701837dc407 error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.473-1200 [ERROR] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to heartbeat to: peer=6085a72a-2d6f-0e11-9b40-c701837dc407 error="failed to connect to peer: 6085a72a-2d6f-0e11-9b40-c701837dc407" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.477-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: Election timeout reached, restarting election - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.477-1200 [WARN] server-6085a72a-2d6f-0e11-9b40-c701837dc407: Election timeout reached, restarting election -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.478-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering candidate state: node="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Candidate]" term=4 - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.478-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering candidate state: node="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Candidate]" term=4 -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.478-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to make requestVote RPC: target="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.478-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to make requestVote RPC: target="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" error="failed to connect to peer: ee94f1af-0538-dd75-0868-3267cffc729b" -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.478-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to make requestVote RPC: target="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.478-1200 [ERROR] server-6085a72a-2d6f-0e11-9b40-c701837dc407: failed to make requestVote RPC: target="{Voter server-d7fde817-1e05-8df2-4277-e2c3b1ce543c d7fde817-1e05-8df2-4277-e2c3b1ce543c}" error="failed to connect to peer: d7fde817-1e05-8df2-4277-e2c3b1ce543c" -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.497-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to contact: server-id=server-6085a72a-2d6f-0e11-9b40-c701837dc407 time=100.7616ms - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.497-1200 [WARN] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: failed to contact: server-id=server-6085a72a-2d6f-0e11-9b40-c701837dc407 time=100.7616ms -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.508-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.40253032 -1200 -12 m=+5.409846781 (1 nodes), 137.0046ms from start of poll, 427.5325ms from cluster start. Timeout at 2024-06-16 08:26:28.50796646 -1200 -12 m=+5.515282981, 105.4362ms after stability - testing.go:149: cluster: 2024-06-16T08:26:28.508-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.40253032 -1200 -12 m=+5.409846781 (1 nodes), 137.0046ms from start of poll, 427.5325ms from cluster start. Timeout at 2024-06-16 08:26:28.50796646 -1200 -12 m=+5.515282981, 105.4362ms after stability - raft_test.go:371: [INFO] Reconnecting Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Candidate] -testLoggerAdapter verbose: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.515-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering follower state: follower="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Follower]" leader=d7fde817-1e05-8df2-4277-e2c3b1ce543c - testing.go:149: server-6085a72a-2d6f-0e11-9b40-c701837dc407: 2024-06-16T08:26:28.515-1200 [INFO] server-6085a72a-2d6f-0e11-9b40-c701837dc407: entering follower state: follower="Node at 6085a72a-2d6f-0e11-9b40-c701837dc407 [Follower]" leader=d7fde817-1e05-8df2-4277-e2c3b1ce543c -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.515-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: pipelining replication: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.515-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: pipelining replication: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.520-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: aborting pipeline replication: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.520-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: aborting pipeline replication: peer="{Voter server-ee94f1af-0538-dd75-0868-3267cffc729b ee94f1af-0538-dd75-0868-3267cffc729b}" -testLoggerAdapter verbose: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.521-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: aborting pipeline replication: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" - testing.go:149: server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: 2024-06-16T08:26:28.521-1200 [INFO] server-d7fde817-1e05-8df2-4277-e2c3b1ce543c: aborting pipeline replication: peer="{Voter server-6085a72a-2d6f-0e11-9b40-c701837dc407 6085a72a-2d6f-0e11-9b40-c701837dc407}" ---- PASS: TestRaft_LeaderFail (0.55s) +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.404+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38d37955-a474-b68e-e73a-2da64fa8722b Address:38d37955-a474-b68e-e73a-2da64fa8722b} {Suffrage:Voter ID:server-b5d8336e-7956-5b47-63b1-d4661764bc2f Address:b5d8336e-7956-5b47-63b1-d4661764bc2f} {Suffrage:Voter ID:server-ba1d3188-287f-debb-8816-e3da0c85f3b7 Address:ba1d3188-287f-debb-8816-e3da0c85f3b7}]" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.404+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38d37955-a474-b68e-e73a-2da64fa8722b Address:38d37955-a474-b68e-e73a-2da64fa8722b} {Suffrage:Voter ID:server-b5d8336e-7956-5b47-63b1-d4661764bc2f Address:b5d8336e-7956-5b47-63b1-d4661764bc2f} {Suffrage:Voter ID:server-ba1d3188-287f-debb-8816-e3da0c85f3b7 Address:ba1d3188-287f-debb-8816-e3da0c85f3b7}]" +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.405+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38d37955-a474-b68e-e73a-2da64fa8722b Address:38d37955-a474-b68e-e73a-2da64fa8722b} {Suffrage:Voter ID:server-b5d8336e-7956-5b47-63b1-d4661764bc2f Address:b5d8336e-7956-5b47-63b1-d4661764bc2f} {Suffrage:Voter ID:server-ba1d3188-287f-debb-8816-e3da0c85f3b7 Address:ba1d3188-287f-debb-8816-e3da0c85f3b7}]" + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.405+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38d37955-a474-b68e-e73a-2da64fa8722b Address:38d37955-a474-b68e-e73a-2da64fa8722b} {Suffrage:Voter ID:server-b5d8336e-7956-5b47-63b1-d4661764bc2f Address:b5d8336e-7956-5b47-63b1-d4661764bc2f} {Suffrage:Voter ID:server-ba1d3188-287f-debb-8816-e3da0c85f3b7 Address:ba1d3188-287f-debb-8816-e3da0c85f3b7}]" +testLoggerAdapter verbose: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.405+1400 [INFO] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38d37955-a474-b68e-e73a-2da64fa8722b Address:38d37955-a474-b68e-e73a-2da64fa8722b} {Suffrage:Voter ID:server-b5d8336e-7956-5b47-63b1-d4661764bc2f Address:b5d8336e-7956-5b47-63b1-d4661764bc2f} {Suffrage:Voter ID:server-ba1d3188-287f-debb-8816-e3da0c85f3b7 Address:ba1d3188-287f-debb-8816-e3da0c85f3b7}]" + testing.go:149: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.405+1400 [INFO] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38d37955-a474-b68e-e73a-2da64fa8722b Address:38d37955-a474-b68e-e73a-2da64fa8722b} {Suffrage:Voter ID:server-b5d8336e-7956-5b47-63b1-d4661764bc2f Address:b5d8336e-7956-5b47-63b1-d4661764bc2f} {Suffrage:Voter ID:server-ba1d3188-287f-debb-8816-e3da0c85f3b7 Address:ba1d3188-287f-debb-8816-e3da0c85f3b7}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.405+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:08.405+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.405+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering follower state: follower="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Follower]" leader= + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.405+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering follower state: follower="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Follower]" leader= +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.405+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: entering follower state: follower="Node at b5d8336e-7956-5b47-63b1-d4661764bc2f [Follower]" leader= + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.405+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: entering follower state: follower="Node at b5d8336e-7956-5b47-63b1-d4661764bc2f [Follower]" leader= +testLoggerAdapter verbose: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.405+1400 [INFO] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: entering follower state: follower="Node at ba1d3188-287f-debb-8816-e3da0c85f3b7 [Follower]" leader= + testing.go:149: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.405+1400 [INFO] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: entering follower state: follower="Node at ba1d3188-287f-debb-8816-e3da0c85f3b7 [Follower]" leader= +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.461+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.461+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.461+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering candidate state: node="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Candidate]" term=2 + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.461+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering candidate state: node="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Candidate]" term=2 +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: election won: tally=2 + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: election won: tally=2 +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering leader state: leader="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Leader]" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering leader state: leader="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Leader]" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: added peer, starting replication: peer=server-b5d8336e-7956-5b47-63b1-d4661764bc2f + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: added peer, starting replication: peer=server-b5d8336e-7956-5b47-63b1-d4661764bc2f +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: added peer, starting replication: peer=server-ba1d3188-287f-debb-8816-e3da0c85f3b7 + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: added peer, starting replication: peer=server-ba1d3188-287f-debb-8816-e3da0c85f3b7 +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: pipelining replication: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: pipelining replication: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: pipelining replication: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.462+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: pipelining replication: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.568+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:08.462838907 +1400 +14 m=+5.265420523 (2 nodes), 57.280197ms from start of poll, 58.257288ms from cluster start. Timeout at 2023-05-16 04:06:08.568296772 +1400 +14 m=+5.370878427, 105.457904ms after stability + testing.go:149: cluster: 2023-05-16T04:06:08.568+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:08.462838907 +1400 +14 m=+5.265420523 (2 nodes), 57.280197ms from start of poll, 58.257288ms from cluster start. Timeout at 2023-05-16 04:06:08.568296772 +1400 +14 m=+5.370878427, 105.457904ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.568+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:08.568+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.674+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:08.568603476 +1400 +14 m=+5.371185111 (1 nodes), 1.9µs from start of poll, 164.021876ms from cluster start. Timeout at 2023-05-16 04:06:08.674153461 +1400 +14 m=+5.476735116, 105.550005ms after stability + testing.go:149: cluster: 2023-05-16T04:06:08.674+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:08.568603476 +1400 +14 m=+5.371185111 (1 nodes), 1.9µs from start of poll, 164.021876ms from cluster start. Timeout at 2023-05-16 04:06:08.674153461 +1400 +14 m=+5.476735116, 105.550005ms after stability + raft_test.go:336: [INFO] Disconnecting Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Leader] +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.687+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=ba1d3188-287f-debb-8816-e3da0c85f3b7 error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.687+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=ba1d3188-287f-debb-8816-e3da0c85f3b7 error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.687+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to pipeline appendEntries: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="append pipeline closed" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.687+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to pipeline appendEntries: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="append pipeline closed" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.687+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: aborting pipeline replication: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.687+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: aborting pipeline replication: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.689+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=b5d8336e-7956-5b47-63b1-d4661764bc2f error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.689+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=b5d8336e-7956-5b47-63b1-d4661764bc2f error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.691+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:08.691+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.692+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to pipeline appendEntries: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="append pipeline closed" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.692+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to pipeline appendEntries: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="append pipeline closed" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.692+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: aborting pipeline replication: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.692+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: aborting pipeline replication: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.698+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.698+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.700+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.700+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.707+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=ba1d3188-287f-debb-8816-e3da0c85f3b7 error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.707+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=ba1d3188-287f-debb-8816-e3da0c85f3b7 error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.707+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=b5d8336e-7956-5b47-63b1-d4661764bc2f error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.707+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=b5d8336e-7956-5b47-63b1-d4661764bc2f error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.715+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.715+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.721+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.721+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.726+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=b5d8336e-7956-5b47-63b1-d4661764bc2f error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.726+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=b5d8336e-7956-5b47-63b1-d4661764bc2f error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.727+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=ba1d3188-287f-debb-8816-e3da0c85f3b7 error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.727+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=ba1d3188-287f-debb-8816-e3da0c85f3b7 error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.731+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to contact: server-id=server-ba1d3188-287f-debb-8816-e3da0c85f3b7 time=50.758979ms + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.731+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to contact: server-id=server-ba1d3188-287f-debb-8816-e3da0c85f3b7 time=50.758979ms +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.732+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.732+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.739+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.739+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.741+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to contact: server-id=server-b5d8336e-7956-5b47-63b1-d4661764bc2f time=57.909424ms + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.741+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to contact: server-id=server-b5d8336e-7956-5b47-63b1-d4661764bc2f time=57.909424ms +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.741+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to contact: server-id=server-ba1d3188-287f-debb-8816-e3da0c85f3b7 time=61.191202ms + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.741+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to contact: server-id=server-ba1d3188-287f-debb-8816-e3da0c85f3b7 time=61.191202ms +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.741+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to contact quorum of nodes, stepping down + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.741+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to contact quorum of nodes, stepping down +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.741+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering follower state: follower="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Follower]" leader= + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.741+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering follower state: follower="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Follower]" leader= +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.754+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=ba1d3188-287f-debb-8816-e3da0c85f3b7 error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.754+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to heartbeat to: peer=ba1d3188-287f-debb-8816-e3da0c85f3b7 error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.758+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.758+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to appendEntries to: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.781+1400 [WARN] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: heartbeat timeout reached, starting election: last-leader=38d37955-a474-b68e-e73a-2da64fa8722b + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.781+1400 [WARN] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: heartbeat timeout reached, starting election: last-leader=38d37955-a474-b68e-e73a-2da64fa8722b +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.781+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: entering candidate state: node="Node at b5d8336e-7956-5b47-63b1-d4661764bc2f [Candidate]" term=3 + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.781+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: entering candidate state: node="Node at b5d8336e-7956-5b47-63b1-d4661764bc2f [Candidate]" term=3 +testLoggerAdapter verbose: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.781+1400 [WARN] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: rejecting vote request since we have a leader: from=b5d8336e-7956-5b47-63b1-d4661764bc2f leader=38d37955-a474-b68e-e73a-2da64fa8722b + testing.go:149: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.781+1400 [WARN] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: rejecting vote request since we have a leader: from=b5d8336e-7956-5b47-63b1-d4661764bc2f leader=38d37955-a474-b68e-e73a-2da64fa8722b +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.781+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to make requestVote RPC: target="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.781+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to make requestVote RPC: target="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.803+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.803+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.803+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering candidate state: node="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Candidate]" term=3 + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.803+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering candidate state: node="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Candidate]" term=3 +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.803+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to make requestVote RPC: target="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.803+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to make requestVote RPC: target="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.803+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to make requestVote RPC: target="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.803+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to make requestVote RPC: target="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" +testLoggerAdapter verbose: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.806+1400 [WARN] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: heartbeat timeout reached, starting election: last-leader=38d37955-a474-b68e-e73a-2da64fa8722b + testing.go:149: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.806+1400 [WARN] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: heartbeat timeout reached, starting election: last-leader=38d37955-a474-b68e-e73a-2da64fa8722b +testLoggerAdapter verbose: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.806+1400 [INFO] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: entering candidate state: node="Node at ba1d3188-287f-debb-8816-e3da0c85f3b7 [Candidate]" term=3 + testing.go:149: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.806+1400 [INFO] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: entering candidate state: node="Node at ba1d3188-287f-debb-8816-e3da0c85f3b7 [Candidate]" term=3 +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.806+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: duplicate requestVote for same term: term=3 + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.806+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: duplicate requestVote for same term: term=3 +testLoggerAdapter verbose: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.806+1400 [ERROR] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: failed to make requestVote RPC: target="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" + testing.go:149: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.806+1400 [ERROR] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: failed to make requestVote RPC: target="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [WARN] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: Election timeout reached, restarting election + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [WARN] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: Election timeout reached, restarting election +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: entering candidate state: node="Node at b5d8336e-7956-5b47-63b1-d4661764bc2f [Candidate]" term=4 + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: entering candidate state: node="Node at b5d8336e-7956-5b47-63b1-d4661764bc2f [Candidate]" term=4 +testLoggerAdapter verbose: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.852+1400 [INFO] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: entering follower state: follower="Node at ba1d3188-287f-debb-8816-e3da0c85f3b7 [Follower]" leader= + testing.go:149: server-ba1d3188-287f-debb-8816-e3da0c85f3b7: 2023-05-16T04:06:08.852+1400 [INFO] server-ba1d3188-287f-debb-8816-e3da0c85f3b7: entering follower state: follower="Node at ba1d3188-287f-debb-8816-e3da0c85f3b7 [Follower]" leader= +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: election won: tally=2 + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: election won: tally=2 +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: entering leader state: leader="Node at b5d8336e-7956-5b47-63b1-d4661764bc2f [Leader]" + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: entering leader state: leader="Node at b5d8336e-7956-5b47-63b1-d4661764bc2f [Leader]" +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: added peer, starting replication: peer=server-38d37955-a474-b68e-e73a-2da64fa8722b + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: added peer, starting replication: peer=server-38d37955-a474-b68e-e73a-2da64fa8722b +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: added peer, starting replication: peer=server-ba1d3188-287f-debb-8816-e3da0c85f3b7 + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.852+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: added peer, starting replication: peer=server-ba1d3188-287f-debb-8816-e3da0c85f3b7 +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.853+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: pipelining replication: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.853+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: pipelining replication: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.853+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to make requestVote RPC: target="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.853+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to make requestVote RPC: target="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.853+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to appendEntries to: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.853+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to appendEntries to: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.861+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to heartbeat to: peer=38d37955-a474-b68e-e73a-2da64fa8722b error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.861+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to heartbeat to: peer=38d37955-a474-b68e-e73a-2da64fa8722b error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.872+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to appendEntries to: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.872+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to appendEntries to: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.880+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to heartbeat to: peer=38d37955-a474-b68e-e73a-2da64fa8722b error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.880+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to heartbeat to: peer=38d37955-a474-b68e-e73a-2da64fa8722b error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.881+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: Election timeout reached, restarting election + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.881+1400 [WARN] server-38d37955-a474-b68e-e73a-2da64fa8722b: Election timeout reached, restarting election +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.882+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering candidate state: node="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Candidate]" term=4 + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.882+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering candidate state: node="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Candidate]" term=4 +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.882+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to make requestVote RPC: target="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.882+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to make requestVote RPC: target="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" error="failed to connect to peer: ba1d3188-287f-debb-8816-e3da0c85f3b7" +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.882+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to make requestVote RPC: target="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.882+1400 [ERROR] server-38d37955-a474-b68e-e73a-2da64fa8722b: failed to make requestVote RPC: target="{Voter server-b5d8336e-7956-5b47-63b1-d4661764bc2f b5d8336e-7956-5b47-63b1-d4661764bc2f}" error="failed to connect to peer: b5d8336e-7956-5b47-63b1-d4661764bc2f" +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.891+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to appendEntries to: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.891+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to appendEntries to: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.900+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to heartbeat to: peer=38d37955-a474-b68e-e73a-2da64fa8722b error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.900+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to heartbeat to: peer=38d37955-a474-b68e-e73a-2da64fa8722b error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.904+1400 [WARN] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to contact: server-id=server-38d37955-a474-b68e-e73a-2da64fa8722b time=51.141963ms + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.904+1400 [WARN] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to contact: server-id=server-38d37955-a474-b68e-e73a-2da64fa8722b time=51.141963ms +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.919+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to appendEntries to: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.919+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to appendEntries to: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.930+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to heartbeat to: peer=38d37955-a474-b68e-e73a-2da64fa8722b error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.930+1400 [ERROR] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to heartbeat to: peer=38d37955-a474-b68e-e73a-2da64fa8722b error="failed to connect to peer: 38d37955-a474-b68e-e73a-2da64fa8722b" +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.950+1400 [WARN] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to contact: server-id=server-38d37955-a474-b68e-e73a-2da64fa8722b time=97.116286ms + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.950+1400 [WARN] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: failed to contact: server-id=server-38d37955-a474-b68e-e73a-2da64fa8722b time=97.116286ms +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.959+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:08.853334416 +1400 +14 m=+5.655916051 (1 nodes), 161.83455ms from start of poll, 448.752816ms from cluster start. Timeout at 2023-05-16 04:06:08.959014683 +1400 +14 m=+5.761596318, 105.680267ms after stability + testing.go:149: cluster: 2023-05-16T04:06:08.959+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:08.853334416 +1400 +14 m=+5.655916051 (1 nodes), 161.83455ms from start of poll, 448.752816ms from cluster start. Timeout at 2023-05-16 04:06:08.959014683 +1400 +14 m=+5.761596318, 105.680267ms after stability + raft_test.go:371: [INFO] Reconnecting Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Candidate] +testLoggerAdapter verbose: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.967+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering follower state: follower="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Follower]" leader=b5d8336e-7956-5b47-63b1-d4661764bc2f + testing.go:149: server-38d37955-a474-b68e-e73a-2da64fa8722b: 2023-05-16T04:06:08.967+1400 [INFO] server-38d37955-a474-b68e-e73a-2da64fa8722b: entering follower state: follower="Node at 38d37955-a474-b68e-e73a-2da64fa8722b [Follower]" leader=b5d8336e-7956-5b47-63b1-d4661764bc2f +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.967+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: pipelining replication: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.967+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: pipelining replication: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.968+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: aborting pipeline replication: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.968+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: aborting pipeline replication: peer="{Voter server-ba1d3188-287f-debb-8816-e3da0c85f3b7 ba1d3188-287f-debb-8816-e3da0c85f3b7}" +testLoggerAdapter verbose: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.968+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: aborting pipeline replication: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" + testing.go:149: server-b5d8336e-7956-5b47-63b1-d4661764bc2f: 2023-05-16T04:06:08.968+1400 [INFO] server-b5d8336e-7956-5b47-63b1-d4661764bc2f: aborting pipeline replication: peer="{Voter server-38d37955-a474-b68e-e73a-2da64fa8722b 38d37955-a474-b68e-e73a-2da64fa8722b}" +--- PASS: TestRaft_LeaderFail (0.56s) === RUN TestRaft_BehindFollower -testLoggerAdapter verbose: server-d934f546-fac8-ac51-ea1e-ac58446887da: 2024-06-16T08:26:28.522-1200 [INFO] server-d934f546-fac8-ac51-ea1e-ac58446887da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d934f546-fac8-ac51-ea1e-ac58446887da Address:d934f546-fac8-ac51-ea1e-ac58446887da} {Suffrage:Voter ID:server-03150ad5-4bb9-cb55-53ca-a6546c897d87 Address:03150ad5-4bb9-cb55-53ca-a6546c897d87} {Suffrage:Voter ID:server-64bb9548-637e-d65a-be35-d88cd71db743 Address:64bb9548-637e-d65a-be35-d88cd71db743}]" - testing.go:149: server-d934f546-fac8-ac51-ea1e-ac58446887da: 2024-06-16T08:26:28.522-1200 [INFO] server-d934f546-fac8-ac51-ea1e-ac58446887da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d934f546-fac8-ac51-ea1e-ac58446887da Address:d934f546-fac8-ac51-ea1e-ac58446887da} {Suffrage:Voter ID:server-03150ad5-4bb9-cb55-53ca-a6546c897d87 Address:03150ad5-4bb9-cb55-53ca-a6546c897d87} {Suffrage:Voter ID:server-64bb9548-637e-d65a-be35-d88cd71db743 Address:64bb9548-637e-d65a-be35-d88cd71db743}]" -testLoggerAdapter verbose: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.522-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d934f546-fac8-ac51-ea1e-ac58446887da Address:d934f546-fac8-ac51-ea1e-ac58446887da} {Suffrage:Voter ID:server-03150ad5-4bb9-cb55-53ca-a6546c897d87 Address:03150ad5-4bb9-cb55-53ca-a6546c897d87} {Suffrage:Voter ID:server-64bb9548-637e-d65a-be35-d88cd71db743 Address:64bb9548-637e-d65a-be35-d88cd71db743}]" - testing.go:149: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.522-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d934f546-fac8-ac51-ea1e-ac58446887da Address:d934f546-fac8-ac51-ea1e-ac58446887da} {Suffrage:Voter ID:server-03150ad5-4bb9-cb55-53ca-a6546c897d87 Address:03150ad5-4bb9-cb55-53ca-a6546c897d87} {Suffrage:Voter ID:server-64bb9548-637e-d65a-be35-d88cd71db743 Address:64bb9548-637e-d65a-be35-d88cd71db743}]" -testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.523-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d934f546-fac8-ac51-ea1e-ac58446887da Address:d934f546-fac8-ac51-ea1e-ac58446887da} {Suffrage:Voter ID:server-03150ad5-4bb9-cb55-53ca-a6546c897d87 Address:03150ad5-4bb9-cb55-53ca-a6546c897d87} {Suffrage:Voter ID:server-64bb9548-637e-d65a-be35-d88cd71db743 Address:64bb9548-637e-d65a-be35-d88cd71db743}]" - testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.523-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d934f546-fac8-ac51-ea1e-ac58446887da Address:d934f546-fac8-ac51-ea1e-ac58446887da} {Suffrage:Voter ID:server-03150ad5-4bb9-cb55-53ca-a6546c897d87 Address:03150ad5-4bb9-cb55-53ca-a6546c897d87} {Suffrage:Voter ID:server-64bb9548-637e-d65a-be35-d88cd71db743 Address:64bb9548-637e-d65a-be35-d88cd71db743}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.523-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:28.523-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-d934f546-fac8-ac51-ea1e-ac58446887da: 2024-06-16T08:26:28.523-1200 [INFO] server-d934f546-fac8-ac51-ea1e-ac58446887da: entering follower state: follower="Node at d934f546-fac8-ac51-ea1e-ac58446887da [Follower]" leader= - testing.go:149: server-d934f546-fac8-ac51-ea1e-ac58446887da: 2024-06-16T08:26:28.523-1200 [INFO] server-d934f546-fac8-ac51-ea1e-ac58446887da: entering follower state: follower="Node at d934f546-fac8-ac51-ea1e-ac58446887da [Follower]" leader= -testLoggerAdapter verbose: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.523-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: entering follower state: follower="Node at 03150ad5-4bb9-cb55-53ca-a6546c897d87 [Follower]" leader= - testing.go:149: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.523-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: entering follower state: follower="Node at 03150ad5-4bb9-cb55-53ca-a6546c897d87 [Follower]" leader= -testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.523-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: entering follower state: follower="Node at 64bb9548-637e-d65a-be35-d88cd71db743 [Follower]" leader= - testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.523-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: entering follower state: follower="Node at 64bb9548-637e-d65a-be35-d88cd71db743 [Follower]" leader= -testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [WARN] server-64bb9548-637e-d65a-be35-d88cd71db743: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [WARN] server-64bb9548-637e-d65a-be35-d88cd71db743: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: entering candidate state: node="Node at 64bb9548-637e-d65a-be35-d88cd71db743 [Candidate]" term=2 - testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: entering candidate state: node="Node at 64bb9548-637e-d65a-be35-d88cd71db743 [Candidate]" term=2 -testLoggerAdapter verbose: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.576-1200 [WARN] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.576-1200 [WARN] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.576-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: entering candidate state: node="Node at 03150ad5-4bb9-cb55-53ca-a6546c897d87 [Candidate]" term=2 - testing.go:149: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.576-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: entering candidate state: node="Node at 03150ad5-4bb9-cb55-53ca-a6546c897d87 [Candidate]" term=2 -testLoggerAdapter verbose: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.576-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: duplicate requestVote for same term: term=2 - testing.go:149: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.576-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: election won: tally=2 - testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: election won: tally=2 -testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: entering leader state: leader="Node at 64bb9548-637e-d65a-be35-d88cd71db743 [Leader]" - testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: entering leader state: leader="Node at 64bb9548-637e-d65a-be35-d88cd71db743 [Leader]" -testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: added peer, starting replication: peer=server-d934f546-fac8-ac51-ea1e-ac58446887da - testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: added peer, starting replication: peer=server-d934f546-fac8-ac51-ea1e-ac58446887da -testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: added peer, starting replication: peer=server-03150ad5-4bb9-cb55-53ca-a6546c897d87 - testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: added peer, starting replication: peer=server-03150ad5-4bb9-cb55-53ca-a6546c897d87 -testLoggerAdapter verbose: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.576-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: entering follower state: follower="Node at 03150ad5-4bb9-cb55-53ca-a6546c897d87 [Follower]" leader=64bb9548-637e-d65a-be35-d88cd71db743 - testing.go:149: server-03150ad5-4bb9-cb55-53ca-a6546c897d87: 2024-06-16T08:26:28.576-1200 [INFO] server-03150ad5-4bb9-cb55-53ca-a6546c897d87: entering follower state: follower="Node at 03150ad5-4bb9-cb55-53ca-a6546c897d87 [Follower]" leader=64bb9548-637e-d65a-be35-d88cd71db743 -testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: pipelining replication: peer="{Voter server-03150ad5-4bb9-cb55-53ca-a6546c897d87 03150ad5-4bb9-cb55-53ca-a6546c897d87}" - testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.576-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: pipelining replication: peer="{Voter server-03150ad5-4bb9-cb55-53ca-a6546c897d87 03150ad5-4bb9-cb55-53ca-a6546c897d87}" -testLoggerAdapter verbose: server-d934f546-fac8-ac51-ea1e-ac58446887da: 2024-06-16T08:26:28.577-1200 [INFO] server-d934f546-fac8-ac51-ea1e-ac58446887da: duplicate requestVote for same term: term=2 - testing.go:149: server-d934f546-fac8-ac51-ea1e-ac58446887da: 2024-06-16T08:26:28.577-1200 [INFO] server-d934f546-fac8-ac51-ea1e-ac58446887da: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.577-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: pipelining replication: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" - testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.577-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: pipelining replication: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" -testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.577-1200 [WARN] server-64bb9548-637e-d65a-be35-d88cd71db743: rejecting vote request since we have a leader: from=03150ad5-4bb9-cb55-53ca-a6546c897d87 leader=64bb9548-637e-d65a-be35-d88cd71db743 - testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.577-1200 [WARN] server-64bb9548-637e-d65a-be35-d88cd71db743: rejecting vote request since we have a leader: from=03150ad5-4bb9-cb55-53ca-a6546c897d87 leader=64bb9548-637e-d65a-be35-d88cd71db743 -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.682-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.57750126 -1200 -12 m=+5.584817721 (1 nodes), 54.2845ms from start of poll, 55.22396ms from cluster start. Timeout at 2024-06-16 08:26:28.68289086 -1200 -12 m=+5.690207321, 105.3896ms after stability - testing.go:149: cluster: 2024-06-16T08:26:28.682-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.57750126 -1200 -12 m=+5.584817721 (1 nodes), 54.2845ms from start of poll, 55.22396ms from cluster start. Timeout at 2024-06-16 08:26:28.68289086 -1200 -12 m=+5.690207321, 105.3896ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.683-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:28.683-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.788-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:28.68313366 -1200 -12 m=+5.690450121 (2 nodes), 2.94µs from start of poll, 160.85636ms from cluster start. Timeout at 2024-06-16 08:26:28.78845418 -1200 -12 m=+5.795770661, 105.32054ms after stability - testing.go:149: cluster: 2024-06-16T08:26:28.788-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:28.68313366 -1200 -12 m=+5.690450121 (2 nodes), 2.94µs from start of poll, 160.85636ms from cluster start. Timeout at 2024-06-16 08:26:28.78845418 -1200 -12 m=+5.795770661, 105.32054ms after stability -testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.788-1200 [ERROR] server-64bb9548-637e-d65a-be35-d88cd71db743: failed to pipeline appendEntries: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" error="append pipeline closed" - testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.788-1200 [ERROR] server-64bb9548-637e-d65a-be35-d88cd71db743: failed to pipeline appendEntries: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" error="append pipeline closed" -testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.788-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: aborting pipeline replication: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" - testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.788-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: aborting pipeline replication: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" -testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.788-1200 [ERROR] server-64bb9548-637e-d65a-be35-d88cd71db743: failed to appendEntries to: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" error="failed to connect to peer: d934f546-fac8-ac51-ea1e-ac58446887da" - testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.788-1200 [ERROR] server-64bb9548-637e-d65a-be35-d88cd71db743: failed to appendEntries to: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" error="failed to connect to peer: d934f546-fac8-ac51-ea1e-ac58446887da" +testLoggerAdapter verbose: server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a: 2023-05-16T04:06:08.969+1400 [INFO] server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a Address:0b440392-2aab-a6c3-452a-6c9b5a4f2b6a} {Suffrage:Voter ID:server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab Address:e610e25c-c779-5e75-a6ea-29f9a10cc8ab} {Suffrage:Voter ID:server-7d849bf5-59ad-984d-0abd-fe66b861e8ef Address:7d849bf5-59ad-984d-0abd-fe66b861e8ef}]" + testing.go:149: server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a: 2023-05-16T04:06:08.969+1400 [INFO] server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a Address:0b440392-2aab-a6c3-452a-6c9b5a4f2b6a} {Suffrage:Voter ID:server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab Address:e610e25c-c779-5e75-a6ea-29f9a10cc8ab} {Suffrage:Voter ID:server-7d849bf5-59ad-984d-0abd-fe66b861e8ef Address:7d849bf5-59ad-984d-0abd-fe66b861e8ef}]" +testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:08.970+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a Address:0b440392-2aab-a6c3-452a-6c9b5a4f2b6a} {Suffrage:Voter ID:server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab Address:e610e25c-c779-5e75-a6ea-29f9a10cc8ab} {Suffrage:Voter ID:server-7d849bf5-59ad-984d-0abd-fe66b861e8ef Address:7d849bf5-59ad-984d-0abd-fe66b861e8ef}]" + testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:08.970+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a Address:0b440392-2aab-a6c3-452a-6c9b5a4f2b6a} {Suffrage:Voter ID:server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab Address:e610e25c-c779-5e75-a6ea-29f9a10cc8ab} {Suffrage:Voter ID:server-7d849bf5-59ad-984d-0abd-fe66b861e8ef Address:7d849bf5-59ad-984d-0abd-fe66b861e8ef}]" +testLoggerAdapter verbose: server-7d849bf5-59ad-984d-0abd-fe66b861e8ef: 2023-05-16T04:06:08.970+1400 [INFO] server-7d849bf5-59ad-984d-0abd-fe66b861e8ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a Address:0b440392-2aab-a6c3-452a-6c9b5a4f2b6a} {Suffrage:Voter ID:server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab Address:e610e25c-c779-5e75-a6ea-29f9a10cc8ab} {Suffrage:Voter ID:server-7d849bf5-59ad-984d-0abd-fe66b861e8ef Address:7d849bf5-59ad-984d-0abd-fe66b861e8ef}]" + testing.go:149: server-7d849bf5-59ad-984d-0abd-fe66b861e8ef: 2023-05-16T04:06:08.970+1400 [INFO] server-7d849bf5-59ad-984d-0abd-fe66b861e8ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a Address:0b440392-2aab-a6c3-452a-6c9b5a4f2b6a} {Suffrage:Voter ID:server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab Address:e610e25c-c779-5e75-a6ea-29f9a10cc8ab} {Suffrage:Voter ID:server-7d849bf5-59ad-984d-0abd-fe66b861e8ef Address:7d849bf5-59ad-984d-0abd-fe66b861e8ef}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:08.970+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:08.970+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a: 2023-05-16T04:06:08.970+1400 [INFO] server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a: entering follower state: follower="Node at 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a [Follower]" leader= + testing.go:149: server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a: 2023-05-16T04:06:08.970+1400 [INFO] server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a: entering follower state: follower="Node at 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a [Follower]" leader= +testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:08.970+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: entering follower state: follower="Node at e610e25c-c779-5e75-a6ea-29f9a10cc8ab [Follower]" leader= + testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:08.970+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: entering follower state: follower="Node at e610e25c-c779-5e75-a6ea-29f9a10cc8ab [Follower]" leader= +testLoggerAdapter verbose: server-7d849bf5-59ad-984d-0abd-fe66b861e8ef: 2023-05-16T04:06:08.970+1400 [INFO] server-7d849bf5-59ad-984d-0abd-fe66b861e8ef: entering follower state: follower="Node at 7d849bf5-59ad-984d-0abd-fe66b861e8ef [Follower]" leader= + testing.go:149: server-7d849bf5-59ad-984d-0abd-fe66b861e8ef: 2023-05-16T04:06:08.970+1400 [INFO] server-7d849bf5-59ad-984d-0abd-fe66b861e8ef: entering follower state: follower="Node at 7d849bf5-59ad-984d-0abd-fe66b861e8ef [Follower]" leader= +testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.021+1400 [WARN] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.021+1400 [WARN] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.021+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: entering candidate state: node="Node at e610e25c-c779-5e75-a6ea-29f9a10cc8ab [Candidate]" term=2 + testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.021+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: entering candidate state: node="Node at e610e25c-c779-5e75-a6ea-29f9a10cc8ab [Candidate]" term=2 +testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: election won: tally=2 + testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: election won: tally=2 +testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: entering leader state: leader="Node at e610e25c-c779-5e75-a6ea-29f9a10cc8ab [Leader]" + testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: entering leader state: leader="Node at e610e25c-c779-5e75-a6ea-29f9a10cc8ab [Leader]" +testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: added peer, starting replication: peer=server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a + testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: added peer, starting replication: peer=server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a +testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: added peer, starting replication: peer=server-7d849bf5-59ad-984d-0abd-fe66b861e8ef + testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: added peer, starting replication: peer=server-7d849bf5-59ad-984d-0abd-fe66b861e8ef +testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: pipelining replication: peer="{Voter server-7d849bf5-59ad-984d-0abd-fe66b861e8ef 7d849bf5-59ad-984d-0abd-fe66b861e8ef}" + testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: pipelining replication: peer="{Voter server-7d849bf5-59ad-984d-0abd-fe66b861e8ef 7d849bf5-59ad-984d-0abd-fe66b861e8ef}" +testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: pipelining replication: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" + testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.022+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: pipelining replication: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.128+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:09.022953398 +1400 +14 m=+5.825535033 (1 nodes), 52.437479ms from start of poll, 53.432211ms from cluster start. Timeout at 2023-05-16 04:06:09.128848627 +1400 +14 m=+5.931430282, 105.895249ms after stability + testing.go:149: cluster: 2023-05-16T04:06:09.128+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:09.022953398 +1400 +14 m=+5.825535033 (1 nodes), 52.437479ms from start of poll, 53.432211ms from cluster start. Timeout at 2023-05-16 04:06:09.128848627 +1400 +14 m=+5.931430282, 105.895249ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.129+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:09.129+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.234+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:09.129128291 +1400 +14 m=+5.931709926 (2 nodes), 2.98µs from start of poll, 159.607104ms from cluster start. Timeout at 2023-05-16 04:06:09.234232611 +1400 +14 m=+6.036814226, 105.1043ms after stability + testing.go:149: cluster: 2023-05-16T04:06:09.234+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:09.129128291 +1400 +14 m=+5.931709926 (2 nodes), 2.98µs from start of poll, 159.607104ms from cluster start. Timeout at 2023-05-16 04:06:09.234232611 +1400 +14 m=+6.036814226, 105.1043ms after stability +testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.234+1400 [ERROR] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: failed to heartbeat to: peer=0b440392-2aab-a6c3-452a-6c9b5a4f2b6a error="failed to connect to peer: 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a" + testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.234+1400 [ERROR] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: failed to heartbeat to: peer=0b440392-2aab-a6c3-452a-6c9b5a4f2b6a error="failed to connect to peer: 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a" +testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.234+1400 [ERROR] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: failed to pipeline appendEntries: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" error="append pipeline closed" + testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.234+1400 [ERROR] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: failed to pipeline appendEntries: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" error="append pipeline closed" +testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.234+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: aborting pipeline replication: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" + testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.234+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: aborting pipeline replication: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" +testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.234+1400 [ERROR] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: failed to appendEntries to: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" error="failed to connect to peer: 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a" + testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.234+1400 [ERROR] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: failed to appendEntries to: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" error="failed to connect to peer: 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a" raft_test.go:420: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.799-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: pipelining replication: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" - testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.799-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: pipelining replication: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.813-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:28.813-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.919-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.81329822 -1200 -12 m=+5.820614681 (1 nodes), 1.58µs from start of poll, 291.02092ms from cluster start. Timeout at 2024-06-16 08:26:28.91920956 -1200 -12 m=+5.926526061, 105.91138ms after stability - testing.go:149: cluster: 2024-06-16T08:26:28.919-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.81329822 -1200 -12 m=+5.820614681 (1 nodes), 1.58µs from start of poll, 291.02092ms from cluster start. Timeout at 2024-06-16 08:26:28.91920956 -1200 -12 m=+5.926526061, 105.91138ms after stability -testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.919-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: aborting pipeline replication: peer="{Voter server-03150ad5-4bb9-cb55-53ca-a6546c897d87 03150ad5-4bb9-cb55-53ca-a6546c897d87}" - testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.919-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: aborting pipeline replication: peer="{Voter server-03150ad5-4bb9-cb55-53ca-a6546c897d87 03150ad5-4bb9-cb55-53ca-a6546c897d87}" -testLoggerAdapter verbose: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.919-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: aborting pipeline replication: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" - testing.go:149: server-64bb9548-637e-d65a-be35-d88cd71db743: 2024-06-16T08:26:28.919-1200 [INFO] server-64bb9548-637e-d65a-be35-d88cd71db743: aborting pipeline replication: peer="{Voter server-d934f546-fac8-ac51-ea1e-ac58446887da d934f546-fac8-ac51-ea1e-ac58446887da}" ---- PASS: TestRaft_BehindFollower (0.40s) +testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.246+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: pipelining replication: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" + testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.246+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: pipelining replication: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.256+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:09.256+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.361+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:09.256438813 +1400 +14 m=+6.059020428 (1 nodes), 2.38µs from start of poll, 286.917606ms from cluster start. Timeout at 2023-05-16 04:06:09.361539094 +1400 +14 m=+6.164120709, 105.100281ms after stability + testing.go:149: cluster: 2023-05-16T04:06:09.361+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:09.256438813 +1400 +14 m=+6.059020428 (1 nodes), 2.38µs from start of poll, 286.917606ms from cluster start. Timeout at 2023-05-16 04:06:09.361539094 +1400 +14 m=+6.164120709, 105.100281ms after stability +testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.361+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: aborting pipeline replication: peer="{Voter server-7d849bf5-59ad-984d-0abd-fe66b861e8ef 7d849bf5-59ad-984d-0abd-fe66b861e8ef}" + testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.361+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: aborting pipeline replication: peer="{Voter server-7d849bf5-59ad-984d-0abd-fe66b861e8ef 7d849bf5-59ad-984d-0abd-fe66b861e8ef}" +testLoggerAdapter verbose: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.361+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: aborting pipeline replication: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" + testing.go:149: server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: 2023-05-16T04:06:09.361+1400 [INFO] server-e610e25c-c779-5e75-a6ea-29f9a10cc8ab: aborting pipeline replication: peer="{Voter server-0b440392-2aab-a6c3-452a-6c9b5a4f2b6a 0b440392-2aab-a6c3-452a-6c9b5a4f2b6a}" +--- PASS: TestRaft_BehindFollower (0.39s) === RUN TestRaft_ApplyNonLeader -testLoggerAdapter verbose: server-11d515dd-acc8-771d-addc-4dfd5981f34a: 2024-06-16T08:26:28.920-1200 [INFO] server-11d515dd-acc8-771d-addc-4dfd5981f34a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d515dd-acc8-771d-addc-4dfd5981f34a Address:11d515dd-acc8-771d-addc-4dfd5981f34a} {Suffrage:Voter ID:server-f0710021-f920-40c9-ab9c-55569a00223c Address:f0710021-f920-40c9-ab9c-55569a00223c} {Suffrage:Voter ID:server-6fdbcaa1-dffd-eddf-6dad-578847450335 Address:6fdbcaa1-dffd-eddf-6dad-578847450335}]" - testing.go:149: server-11d515dd-acc8-771d-addc-4dfd5981f34a: 2024-06-16T08:26:28.920-1200 [INFO] server-11d515dd-acc8-771d-addc-4dfd5981f34a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d515dd-acc8-771d-addc-4dfd5981f34a Address:11d515dd-acc8-771d-addc-4dfd5981f34a} {Suffrage:Voter ID:server-f0710021-f920-40c9-ab9c-55569a00223c Address:f0710021-f920-40c9-ab9c-55569a00223c} {Suffrage:Voter ID:server-6fdbcaa1-dffd-eddf-6dad-578847450335 Address:6fdbcaa1-dffd-eddf-6dad-578847450335}]" -testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.921-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d515dd-acc8-771d-addc-4dfd5981f34a Address:11d515dd-acc8-771d-addc-4dfd5981f34a} {Suffrage:Voter ID:server-f0710021-f920-40c9-ab9c-55569a00223c Address:f0710021-f920-40c9-ab9c-55569a00223c} {Suffrage:Voter ID:server-6fdbcaa1-dffd-eddf-6dad-578847450335 Address:6fdbcaa1-dffd-eddf-6dad-578847450335}]" - testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.921-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d515dd-acc8-771d-addc-4dfd5981f34a Address:11d515dd-acc8-771d-addc-4dfd5981f34a} {Suffrage:Voter ID:server-f0710021-f920-40c9-ab9c-55569a00223c Address:f0710021-f920-40c9-ab9c-55569a00223c} {Suffrage:Voter ID:server-6fdbcaa1-dffd-eddf-6dad-578847450335 Address:6fdbcaa1-dffd-eddf-6dad-578847450335}]" -testLoggerAdapter verbose: server-6fdbcaa1-dffd-eddf-6dad-578847450335: 2024-06-16T08:26:28.921-1200 [INFO] server-6fdbcaa1-dffd-eddf-6dad-578847450335: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d515dd-acc8-771d-addc-4dfd5981f34a Address:11d515dd-acc8-771d-addc-4dfd5981f34a} {Suffrage:Voter ID:server-f0710021-f920-40c9-ab9c-55569a00223c Address:f0710021-f920-40c9-ab9c-55569a00223c} {Suffrage:Voter ID:server-6fdbcaa1-dffd-eddf-6dad-578847450335 Address:6fdbcaa1-dffd-eddf-6dad-578847450335}]" -testLoggerAdapter verbose: server-11d515dd-acc8-771d-addc-4dfd5981f34a: 2024-06-16T08:26:28.921-1200 [INFO] server-11d515dd-acc8-771d-addc-4dfd5981f34a: entering follower state: follower="Node at 11d515dd-acc8-771d-addc-4dfd5981f34a [Follower]" leader= - testing.go:149: server-6fdbcaa1-dffd-eddf-6dad-578847450335: 2024-06-16T08:26:28.921-1200 [INFO] server-6fdbcaa1-dffd-eddf-6dad-578847450335: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d515dd-acc8-771d-addc-4dfd5981f34a Address:11d515dd-acc8-771d-addc-4dfd5981f34a} {Suffrage:Voter ID:server-f0710021-f920-40c9-ab9c-55569a00223c Address:f0710021-f920-40c9-ab9c-55569a00223c} {Suffrage:Voter ID:server-6fdbcaa1-dffd-eddf-6dad-578847450335 Address:6fdbcaa1-dffd-eddf-6dad-578847450335}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:28.921-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.921-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: entering follower state: follower="Node at f0710021-f920-40c9-ab9c-55569a00223c [Follower]" leader= - testing.go:149: cluster: 2024-06-16T08:26:28.921-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: server-11d515dd-acc8-771d-addc-4dfd5981f34a: 2024-06-16T08:26:28.921-1200 [INFO] server-11d515dd-acc8-771d-addc-4dfd5981f34a: entering follower state: follower="Node at 11d515dd-acc8-771d-addc-4dfd5981f34a [Follower]" leader= -testLoggerAdapter verbose: server-6fdbcaa1-dffd-eddf-6dad-578847450335: 2024-06-16T08:26:28.921-1200 [INFO] server-6fdbcaa1-dffd-eddf-6dad-578847450335: entering follower state: follower="Node at 6fdbcaa1-dffd-eddf-6dad-578847450335 [Follower]" leader= - testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.921-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: entering follower state: follower="Node at f0710021-f920-40c9-ab9c-55569a00223c [Follower]" leader= - testing.go:149: server-6fdbcaa1-dffd-eddf-6dad-578847450335: 2024-06-16T08:26:28.921-1200 [INFO] server-6fdbcaa1-dffd-eddf-6dad-578847450335: entering follower state: follower="Node at 6fdbcaa1-dffd-eddf-6dad-578847450335 [Follower]" leader= -testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [WARN] server-f0710021-f920-40c9-ab9c-55569a00223c: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [WARN] server-f0710021-f920-40c9-ab9c-55569a00223c: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: entering candidate state: node="Node at f0710021-f920-40c9-ab9c-55569a00223c [Candidate]" term=2 - testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: entering candidate state: node="Node at f0710021-f920-40c9-ab9c-55569a00223c [Candidate]" term=2 -testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: election won: tally=2 - testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: election won: tally=2 -testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: entering leader state: leader="Node at f0710021-f920-40c9-ab9c-55569a00223c [Leader]" - testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: entering leader state: leader="Node at f0710021-f920-40c9-ab9c-55569a00223c [Leader]" -testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: added peer, starting replication: peer=server-11d515dd-acc8-771d-addc-4dfd5981f34a - testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: added peer, starting replication: peer=server-11d515dd-acc8-771d-addc-4dfd5981f34a -testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: added peer, starting replication: peer=server-6fdbcaa1-dffd-eddf-6dad-578847450335 - testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: added peer, starting replication: peer=server-6fdbcaa1-dffd-eddf-6dad-578847450335 -testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: pipelining replication: peer="{Voter server-11d515dd-acc8-771d-addc-4dfd5981f34a 11d515dd-acc8-771d-addc-4dfd5981f34a}" - testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: pipelining replication: peer="{Voter server-11d515dd-acc8-771d-addc-4dfd5981f34a 11d515dd-acc8-771d-addc-4dfd5981f34a}" -testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: pipelining replication: peer="{Voter server-6fdbcaa1-dffd-eddf-6dad-578847450335 6fdbcaa1-dffd-eddf-6dad-578847450335}" - testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:28.980-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: pipelining replication: peer="{Voter server-6fdbcaa1-dffd-eddf-6dad-578847450335 6fdbcaa1-dffd-eddf-6dad-578847450335}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:29.085-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.98045696 -1200 -12 m=+5.987773441 (1 nodes), 58.93466ms from start of poll, 59.83936ms from cluster start. Timeout at 2024-06-16 08:26:29.08585896 -1200 -12 m=+6.093175461, 105.40202ms after stability - testing.go:149: cluster: 2024-06-16T08:26:29.085-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:28.98045696 -1200 -12 m=+5.987773441 (1 nodes), 58.93466ms from start of poll, 59.83936ms from cluster start. Timeout at 2024-06-16 08:26:29.08585896 -1200 -12 m=+6.093175461, 105.40202ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:29.086-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:29.086-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:29.192-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:29.0860566 -1200 -12 m=+6.093373081 (2 nodes), 2.5µs from start of poll, 165.439ms from cluster start. Timeout at 2024-06-16 08:26:29.19207548 -1200 -12 m=+6.199391961, 106.01888ms after stability - testing.go:149: cluster: 2024-06-16T08:26:29.192-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:29.0860566 -1200 -12 m=+6.093373081 (2 nodes), 2.5µs from start of poll, 165.439ms from cluster start. Timeout at 2024-06-16 08:26:29.19207548 -1200 -12 m=+6.199391961, 106.01888ms after stability -testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:29.192-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: aborting pipeline replication: peer="{Voter server-6fdbcaa1-dffd-eddf-6dad-578847450335 6fdbcaa1-dffd-eddf-6dad-578847450335}" - testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:29.192-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: aborting pipeline replication: peer="{Voter server-6fdbcaa1-dffd-eddf-6dad-578847450335 6fdbcaa1-dffd-eddf-6dad-578847450335}" -testLoggerAdapter verbose: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:29.192-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: aborting pipeline replication: peer="{Voter server-11d515dd-acc8-771d-addc-4dfd5981f34a 11d515dd-acc8-771d-addc-4dfd5981f34a}" - testing.go:149: server-f0710021-f920-40c9-ab9c-55569a00223c: 2024-06-16T08:26:29.192-1200 [INFO] server-f0710021-f920-40c9-ab9c-55569a00223c: aborting pipeline replication: peer="{Voter server-11d515dd-acc8-771d-addc-4dfd5981f34a 11d515dd-acc8-771d-addc-4dfd5981f34a}" +testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.363+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d44644-a293-320e-c5c8-0e1d2ca208a4 Address:11d44644-a293-320e-c5c8-0e1d2ca208a4} {Suffrage:Voter ID:server-7985cc23-7611-73cf-a113-89a9858f0e0c Address:7985cc23-7611-73cf-a113-89a9858f0e0c} {Suffrage:Voter ID:server-9703effd-27b2-0749-7493-c09e6e6f7dcf Address:9703effd-27b2-0749-7493-c09e6e6f7dcf}]" + testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.363+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d44644-a293-320e-c5c8-0e1d2ca208a4 Address:11d44644-a293-320e-c5c8-0e1d2ca208a4} {Suffrage:Voter ID:server-7985cc23-7611-73cf-a113-89a9858f0e0c Address:7985cc23-7611-73cf-a113-89a9858f0e0c} {Suffrage:Voter ID:server-9703effd-27b2-0749-7493-c09e6e6f7dcf Address:9703effd-27b2-0749-7493-c09e6e6f7dcf}]" +testLoggerAdapter verbose: server-7985cc23-7611-73cf-a113-89a9858f0e0c: 2023-05-16T04:06:09.363+1400 [INFO] server-7985cc23-7611-73cf-a113-89a9858f0e0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d44644-a293-320e-c5c8-0e1d2ca208a4 Address:11d44644-a293-320e-c5c8-0e1d2ca208a4} {Suffrage:Voter ID:server-7985cc23-7611-73cf-a113-89a9858f0e0c Address:7985cc23-7611-73cf-a113-89a9858f0e0c} {Suffrage:Voter ID:server-9703effd-27b2-0749-7493-c09e6e6f7dcf Address:9703effd-27b2-0749-7493-c09e6e6f7dcf}]" + testing.go:149: server-7985cc23-7611-73cf-a113-89a9858f0e0c: 2023-05-16T04:06:09.363+1400 [INFO] server-7985cc23-7611-73cf-a113-89a9858f0e0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d44644-a293-320e-c5c8-0e1d2ca208a4 Address:11d44644-a293-320e-c5c8-0e1d2ca208a4} {Suffrage:Voter ID:server-7985cc23-7611-73cf-a113-89a9858f0e0c Address:7985cc23-7611-73cf-a113-89a9858f0e0c} {Suffrage:Voter ID:server-9703effd-27b2-0749-7493-c09e6e6f7dcf Address:9703effd-27b2-0749-7493-c09e6e6f7dcf}]" +testLoggerAdapter verbose: server-9703effd-27b2-0749-7493-c09e6e6f7dcf: 2023-05-16T04:06:09.364+1400 [INFO] server-9703effd-27b2-0749-7493-c09e6e6f7dcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d44644-a293-320e-c5c8-0e1d2ca208a4 Address:11d44644-a293-320e-c5c8-0e1d2ca208a4} {Suffrage:Voter ID:server-7985cc23-7611-73cf-a113-89a9858f0e0c Address:7985cc23-7611-73cf-a113-89a9858f0e0c} {Suffrage:Voter ID:server-9703effd-27b2-0749-7493-c09e6e6f7dcf Address:9703effd-27b2-0749-7493-c09e6e6f7dcf}]" + testing.go:149: server-9703effd-27b2-0749-7493-c09e6e6f7dcf: 2023-05-16T04:06:09.364+1400 [INFO] server-9703effd-27b2-0749-7493-c09e6e6f7dcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11d44644-a293-320e-c5c8-0e1d2ca208a4 Address:11d44644-a293-320e-c5c8-0e1d2ca208a4} {Suffrage:Voter ID:server-7985cc23-7611-73cf-a113-89a9858f0e0c Address:7985cc23-7611-73cf-a113-89a9858f0e0c} {Suffrage:Voter ID:server-9703effd-27b2-0749-7493-c09e6e6f7dcf Address:9703effd-27b2-0749-7493-c09e6e6f7dcf}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.364+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:09.364+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.364+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: entering follower state: follower="Node at 11d44644-a293-320e-c5c8-0e1d2ca208a4 [Follower]" leader= + testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.364+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: entering follower state: follower="Node at 11d44644-a293-320e-c5c8-0e1d2ca208a4 [Follower]" leader= +testLoggerAdapter verbose: server-7985cc23-7611-73cf-a113-89a9858f0e0c: 2023-05-16T04:06:09.364+1400 [INFO] server-7985cc23-7611-73cf-a113-89a9858f0e0c: entering follower state: follower="Node at 7985cc23-7611-73cf-a113-89a9858f0e0c [Follower]" leader= + testing.go:149: server-7985cc23-7611-73cf-a113-89a9858f0e0c: 2023-05-16T04:06:09.364+1400 [INFO] server-7985cc23-7611-73cf-a113-89a9858f0e0c: entering follower state: follower="Node at 7985cc23-7611-73cf-a113-89a9858f0e0c [Follower]" leader= +testLoggerAdapter verbose: server-9703effd-27b2-0749-7493-c09e6e6f7dcf: 2023-05-16T04:06:09.364+1400 [INFO] server-9703effd-27b2-0749-7493-c09e6e6f7dcf: entering follower state: follower="Node at 9703effd-27b2-0749-7493-c09e6e6f7dcf [Follower]" leader= + testing.go:149: server-9703effd-27b2-0749-7493-c09e6e6f7dcf: 2023-05-16T04:06:09.364+1400 [INFO] server-9703effd-27b2-0749-7493-c09e6e6f7dcf: entering follower state: follower="Node at 9703effd-27b2-0749-7493-c09e6e6f7dcf [Follower]" leader= +testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.419+1400 [WARN] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.419+1400 [WARN] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: entering candidate state: node="Node at 11d44644-a293-320e-c5c8-0e1d2ca208a4 [Candidate]" term=2 + testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: entering candidate state: node="Node at 11d44644-a293-320e-c5c8-0e1d2ca208a4 [Candidate]" term=2 +testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: election won: tally=2 + testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: election won: tally=2 +testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: entering leader state: leader="Node at 11d44644-a293-320e-c5c8-0e1d2ca208a4 [Leader]" + testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: entering leader state: leader="Node at 11d44644-a293-320e-c5c8-0e1d2ca208a4 [Leader]" +testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: added peer, starting replication: peer=server-7985cc23-7611-73cf-a113-89a9858f0e0c + testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: added peer, starting replication: peer=server-7985cc23-7611-73cf-a113-89a9858f0e0c +testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: added peer, starting replication: peer=server-9703effd-27b2-0749-7493-c09e6e6f7dcf + testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: added peer, starting replication: peer=server-9703effd-27b2-0749-7493-c09e6e6f7dcf +testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: pipelining replication: peer="{Voter server-9703effd-27b2-0749-7493-c09e6e6f7dcf 9703effd-27b2-0749-7493-c09e6e6f7dcf}" + testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.420+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: pipelining replication: peer="{Voter server-9703effd-27b2-0749-7493-c09e6e6f7dcf 9703effd-27b2-0749-7493-c09e6e6f7dcf}" +testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.421+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: pipelining replication: peer="{Voter server-7985cc23-7611-73cf-a113-89a9858f0e0c 7985cc23-7611-73cf-a113-89a9858f0e0c}" + testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.421+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: pipelining replication: peer="{Voter server-7985cc23-7611-73cf-a113-89a9858f0e0c 7985cc23-7611-73cf-a113-89a9858f0e0c}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.526+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:09.421234878 +1400 +14 m=+6.223816513 (1 nodes), 57.078394ms from start of poll, 57.975664ms from cluster start. Timeout at 2023-05-16 04:06:09.526416699 +1400 +14 m=+6.328998314, 105.181801ms after stability + testing.go:149: cluster: 2023-05-16T04:06:09.526+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:09.421234878 +1400 +14 m=+6.223816513 (1 nodes), 57.078394ms from start of poll, 57.975664ms from cluster start. Timeout at 2023-05-16 04:06:09.526416699 +1400 +14 m=+6.328998314, 105.181801ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.526+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:09.526+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.632+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:09.526693663 +1400 +14 m=+6.329275278 (2 nodes), 2.64µs from start of poll, 163.434429ms from cluster start. Timeout at 2023-05-16 04:06:09.632340569 +1400 +14 m=+6.434922184, 105.646906ms after stability + testing.go:149: cluster: 2023-05-16T04:06:09.632+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:09.526693663 +1400 +14 m=+6.329275278 (2 nodes), 2.64µs from start of poll, 163.434429ms from cluster start. Timeout at 2023-05-16 04:06:09.632340569 +1400 +14 m=+6.434922184, 105.646906ms after stability +testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.632+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: aborting pipeline replication: peer="{Voter server-9703effd-27b2-0749-7493-c09e6e6f7dcf 9703effd-27b2-0749-7493-c09e6e6f7dcf}" + testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.632+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: aborting pipeline replication: peer="{Voter server-9703effd-27b2-0749-7493-c09e6e6f7dcf 9703effd-27b2-0749-7493-c09e6e6f7dcf}" +testLoggerAdapter verbose: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.632+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: aborting pipeline replication: peer="{Voter server-7985cc23-7611-73cf-a113-89a9858f0e0c 7985cc23-7611-73cf-a113-89a9858f0e0c}" + testing.go:149: server-11d44644-a293-320e-c5c8-0e1d2ca208a4: 2023-05-16T04:06:09.632+1400 [INFO] server-11d44644-a293-320e-c5c8-0e1d2ca208a4: aborting pipeline replication: peer="{Voter server-7985cc23-7611-73cf-a113-89a9858f0e0c 7985cc23-7611-73cf-a113-89a9858f0e0c}" --- PASS: TestRaft_ApplyNonLeader (0.27s) === RUN TestRaft_ApplyConcurrent -testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.193-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c08ad73e-2c44-c4b2-db7a-80198ed974cc Address:c08ad73e-2c44-c4b2-db7a-80198ed974cc} {Suffrage:Voter ID:server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e Address:c2bc3a84-64d4-1ea9-6937-06935dd1a27e} {Suffrage:Voter ID:server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 Address:8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}]" - testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.193-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c08ad73e-2c44-c4b2-db7a-80198ed974cc Address:c08ad73e-2c44-c4b2-db7a-80198ed974cc} {Suffrage:Voter ID:server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e Address:c2bc3a84-64d4-1ea9-6937-06935dd1a27e} {Suffrage:Voter ID:server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 Address:8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}]" -testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.193-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: entering follower state: follower="Node at c08ad73e-2c44-c4b2-db7a-80198ed974cc [Follower]" leader= - testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.193-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: entering follower state: follower="Node at c08ad73e-2c44-c4b2-db7a-80198ed974cc [Follower]" leader= -testLoggerAdapter verbose: server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e: 2024-06-16T08:26:29.193-1200 [INFO] server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c08ad73e-2c44-c4b2-db7a-80198ed974cc Address:c08ad73e-2c44-c4b2-db7a-80198ed974cc} {Suffrage:Voter ID:server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e Address:c2bc3a84-64d4-1ea9-6937-06935dd1a27e} {Suffrage:Voter ID:server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 Address:8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}]" - testing.go:149: server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e: 2024-06-16T08:26:29.193-1200 [INFO] server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c08ad73e-2c44-c4b2-db7a-80198ed974cc Address:c08ad73e-2c44-c4b2-db7a-80198ed974cc} {Suffrage:Voter ID:server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e Address:c2bc3a84-64d4-1ea9-6937-06935dd1a27e} {Suffrage:Voter ID:server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 Address:8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}]" -testLoggerAdapter verbose: server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e: 2024-06-16T08:26:29.194-1200 [INFO] server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e: entering follower state: follower="Node at c2bc3a84-64d4-1ea9-6937-06935dd1a27e [Follower]" leader= - testing.go:149: server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e: 2024-06-16T08:26:29.194-1200 [INFO] server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e: entering follower state: follower="Node at c2bc3a84-64d4-1ea9-6937-06935dd1a27e [Follower]" leader= -testLoggerAdapter verbose: server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8: 2024-06-16T08:26:29.194-1200 [INFO] server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c08ad73e-2c44-c4b2-db7a-80198ed974cc Address:c08ad73e-2c44-c4b2-db7a-80198ed974cc} {Suffrage:Voter ID:server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e Address:c2bc3a84-64d4-1ea9-6937-06935dd1a27e} {Suffrage:Voter ID:server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 Address:8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}]" - testing.go:149: server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8: 2024-06-16T08:26:29.194-1200 [INFO] server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c08ad73e-2c44-c4b2-db7a-80198ed974cc Address:c08ad73e-2c44-c4b2-db7a-80198ed974cc} {Suffrage:Voter ID:server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e Address:c2bc3a84-64d4-1ea9-6937-06935dd1a27e} {Suffrage:Voter ID:server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 Address:8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:29.194-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:29.194-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8: 2024-06-16T08:26:29.194-1200 [INFO] server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8: entering follower state: follower="Node at 8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 [Follower]" leader= - testing.go:149: server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8: 2024-06-16T08:26:29.194-1200 [INFO] server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8: entering follower state: follower="Node at 8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 [Follower]" leader= -testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.318-1200 [WARN] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.318-1200 [WARN] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: entering candidate state: node="Node at c08ad73e-2c44-c4b2-db7a-80198ed974cc [Candidate]" term=2 - testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: entering candidate state: node="Node at c08ad73e-2c44-c4b2-db7a-80198ed974cc [Candidate]" term=2 -testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: election won: tally=2 - testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: election won: tally=2 -testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: entering leader state: leader="Node at c08ad73e-2c44-c4b2-db7a-80198ed974cc [Leader]" - testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: entering leader state: leader="Node at c08ad73e-2c44-c4b2-db7a-80198ed974cc [Leader]" -testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: added peer, starting replication: peer=server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e - testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: added peer, starting replication: peer=server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e -testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: added peer, starting replication: peer=server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 - testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: added peer, starting replication: peer=server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 -testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: pipelining replication: peer="{Voter server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e c2bc3a84-64d4-1ea9-6937-06935dd1a27e}" - testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: pipelining replication: peer="{Voter server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e c2bc3a84-64d4-1ea9-6937-06935dd1a27e}" -testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: pipelining replication: peer="{Voter server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}" - testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.319-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: pipelining replication: peer="{Voter server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:29.525-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:29.31940152 -1200 -12 m=+6.326718001 (1 nodes), 125.04392ms from start of poll, 125.90122ms from cluster start. Timeout at 2024-06-16 08:26:29.52535004 -1200 -12 m=+6.532666521, 205.94852ms after stability - testing.go:149: cluster: 2024-06-16T08:26:29.525-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:29.31940152 -1200 -12 m=+6.326718001 (1 nodes), 125.04392ms from start of poll, 125.90122ms from cluster start. Timeout at 2024-06-16 08:26:29.52535004 -1200 -12 m=+6.532666521, 205.94852ms after stability -testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.537-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: aborting pipeline replication: peer="{Voter server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e c2bc3a84-64d4-1ea9-6937-06935dd1a27e}" - testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.537-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: aborting pipeline replication: peer="{Voter server-c2bc3a84-64d4-1ea9-6937-06935dd1a27e c2bc3a84-64d4-1ea9-6937-06935dd1a27e}" -testLoggerAdapter verbose: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.537-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: aborting pipeline replication: peer="{Voter server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}" - testing.go:149: server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: 2024-06-16T08:26:29.537-1200 [INFO] server-c08ad73e-2c44-c4b2-db7a-80198ed974cc: aborting pipeline replication: peer="{Voter server-8cf8acb2-8659-e649-8c44-71fe8f9a8ec8 8cf8acb2-8659-e649-8c44-71fe8f9a8ec8}" ---- PASS: TestRaft_ApplyConcurrent (0.35s) +testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.634+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54e4f296-2322-c79c-b366-f01788e2f8dd Address:54e4f296-2322-c79c-b366-f01788e2f8dd} {Suffrage:Voter ID:server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 Address:c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8} {Suffrage:Voter ID:server-6c1d3058-ab57-c09f-f73b-038557ebf908 Address:6c1d3058-ab57-c09f-f73b-038557ebf908}]" + testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.634+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54e4f296-2322-c79c-b366-f01788e2f8dd Address:54e4f296-2322-c79c-b366-f01788e2f8dd} {Suffrage:Voter ID:server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 Address:c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8} {Suffrage:Voter ID:server-6c1d3058-ab57-c09f-f73b-038557ebf908 Address:6c1d3058-ab57-c09f-f73b-038557ebf908}]" +testLoggerAdapter verbose: server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8: 2023-05-16T04:06:09.634+1400 [INFO] server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54e4f296-2322-c79c-b366-f01788e2f8dd Address:54e4f296-2322-c79c-b366-f01788e2f8dd} {Suffrage:Voter ID:server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 Address:c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8} {Suffrage:Voter ID:server-6c1d3058-ab57-c09f-f73b-038557ebf908 Address:6c1d3058-ab57-c09f-f73b-038557ebf908}]" + testing.go:149: server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8: 2023-05-16T04:06:09.634+1400 [INFO] server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54e4f296-2322-c79c-b366-f01788e2f8dd Address:54e4f296-2322-c79c-b366-f01788e2f8dd} {Suffrage:Voter ID:server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 Address:c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8} {Suffrage:Voter ID:server-6c1d3058-ab57-c09f-f73b-038557ebf908 Address:6c1d3058-ab57-c09f-f73b-038557ebf908}]" +testLoggerAdapter verbose: server-6c1d3058-ab57-c09f-f73b-038557ebf908: 2023-05-16T04:06:09.634+1400 [INFO] server-6c1d3058-ab57-c09f-f73b-038557ebf908: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54e4f296-2322-c79c-b366-f01788e2f8dd Address:54e4f296-2322-c79c-b366-f01788e2f8dd} {Suffrage:Voter ID:server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 Address:c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8} {Suffrage:Voter ID:server-6c1d3058-ab57-c09f-f73b-038557ebf908 Address:6c1d3058-ab57-c09f-f73b-038557ebf908}]" + testing.go:149: server-6c1d3058-ab57-c09f-f73b-038557ebf908: 2023-05-16T04:06:09.634+1400 [INFO] server-6c1d3058-ab57-c09f-f73b-038557ebf908: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54e4f296-2322-c79c-b366-f01788e2f8dd Address:54e4f296-2322-c79c-b366-f01788e2f8dd} {Suffrage:Voter ID:server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 Address:c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8} {Suffrage:Voter ID:server-6c1d3058-ab57-c09f-f73b-038557ebf908 Address:6c1d3058-ab57-c09f-f73b-038557ebf908}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:09.635+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:09.635+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.635+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: entering follower state: follower="Node at 54e4f296-2322-c79c-b366-f01788e2f8dd [Follower]" leader= + testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.635+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: entering follower state: follower="Node at 54e4f296-2322-c79c-b366-f01788e2f8dd [Follower]" leader= +testLoggerAdapter verbose: server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8: 2023-05-16T04:06:09.635+1400 [INFO] server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8: entering follower state: follower="Node at c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 [Follower]" leader= + testing.go:149: server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8: 2023-05-16T04:06:09.635+1400 [INFO] server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8: entering follower state: follower="Node at c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 [Follower]" leader= +testLoggerAdapter verbose: server-6c1d3058-ab57-c09f-f73b-038557ebf908: 2023-05-16T04:06:09.635+1400 [INFO] server-6c1d3058-ab57-c09f-f73b-038557ebf908: entering follower state: follower="Node at 6c1d3058-ab57-c09f-f73b-038557ebf908 [Follower]" leader= + testing.go:149: server-6c1d3058-ab57-c09f-f73b-038557ebf908: 2023-05-16T04:06:09.635+1400 [INFO] server-6c1d3058-ab57-c09f-f73b-038557ebf908: entering follower state: follower="Node at 6c1d3058-ab57-c09f-f73b-038557ebf908 [Follower]" leader= +testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [WARN] server-54e4f296-2322-c79c-b366-f01788e2f8dd: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [WARN] server-54e4f296-2322-c79c-b366-f01788e2f8dd: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: entering candidate state: node="Node at 54e4f296-2322-c79c-b366-f01788e2f8dd [Candidate]" term=2 + testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: entering candidate state: node="Node at 54e4f296-2322-c79c-b366-f01788e2f8dd [Candidate]" term=2 +testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: election won: tally=2 + testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: election won: tally=2 +testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: entering leader state: leader="Node at 54e4f296-2322-c79c-b366-f01788e2f8dd [Leader]" + testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: entering leader state: leader="Node at 54e4f296-2322-c79c-b366-f01788e2f8dd [Leader]" +testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: added peer, starting replication: peer=server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 + testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: added peer, starting replication: peer=server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 +testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: added peer, starting replication: peer=server-6c1d3058-ab57-c09f-f73b-038557ebf908 + testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.800+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: added peer, starting replication: peer=server-6c1d3058-ab57-c09f-f73b-038557ebf908 +testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.801+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: pipelining replication: peer="{Voter server-6c1d3058-ab57-c09f-f73b-038557ebf908 6c1d3058-ab57-c09f-f73b-038557ebf908}" + testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.801+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: pipelining replication: peer="{Voter server-6c1d3058-ab57-c09f-f73b-038557ebf908 6c1d3058-ab57-c09f-f73b-038557ebf908}" +testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.801+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: pipelining replication: peer="{Voter server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8}" + testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:09.801+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: pipelining replication: peer="{Voter server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.006+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:09.801721708 +1400 +14 m=+6.604303323 (1 nodes), 166.589706ms from start of poll, 167.561057ms from cluster start. Timeout at 2023-05-16 04:06:10.006815189 +1400 +14 m=+6.809396804, 205.093481ms after stability + testing.go:149: cluster: 2023-05-16T04:06:10.006+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:09.801721708 +1400 +14 m=+6.604303323 (1 nodes), 166.589706ms from start of poll, 167.561057ms from cluster start. Timeout at 2023-05-16 04:06:10.006815189 +1400 +14 m=+6.809396804, 205.093481ms after stability +testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:10.023+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: aborting pipeline replication: peer="{Voter server-6c1d3058-ab57-c09f-f73b-038557ebf908 6c1d3058-ab57-c09f-f73b-038557ebf908}" + testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:10.023+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: aborting pipeline replication: peer="{Voter server-6c1d3058-ab57-c09f-f73b-038557ebf908 6c1d3058-ab57-c09f-f73b-038557ebf908}" +testLoggerAdapter verbose: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:10.024+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: aborting pipeline replication: peer="{Voter server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8}" + testing.go:149: server-54e4f296-2322-c79c-b366-f01788e2f8dd: 2023-05-16T04:06:10.024+1400 [INFO] server-54e4f296-2322-c79c-b366-f01788e2f8dd: aborting pipeline replication: peer="{Voter server-c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8 c8e90a20-c1ff-82c5-e7e7-9f2fdb184dc8}" +--- PASS: TestRaft_ApplyConcurrent (0.39s) === RUN TestRaft_ApplyConcurrent_Timeout -testLoggerAdapter verbose: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.538-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eae5a569-8074-3e76-b6c8-c068bf13107a Address:eae5a569-8074-3e76-b6c8-c068bf13107a}]" - testing.go:149: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.538-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eae5a569-8074-3e76-b6c8-c068bf13107a Address:eae5a569-8074-3e76-b6c8-c068bf13107a}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:29.538-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:29.538-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.539-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: entering follower state: follower="Node at eae5a569-8074-3e76-b6c8-c068bf13107a [Follower]" leader= - testing.go:149: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.539-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: entering follower state: follower="Node at eae5a569-8074-3e76-b6c8-c068bf13107a [Follower]" leader= -testLoggerAdapter verbose: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.717-1200 [WARN] server-eae5a569-8074-3e76-b6c8-c068bf13107a: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.717-1200 [WARN] server-eae5a569-8074-3e76-b6c8-c068bf13107a: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.717-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: entering candidate state: node="Node at eae5a569-8074-3e76-b6c8-c068bf13107a [Candidate]" term=2 - testing.go:149: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.717-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: entering candidate state: node="Node at eae5a569-8074-3e76-b6c8-c068bf13107a [Candidate]" term=2 -testLoggerAdapter verbose: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.717-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: election won: tally=1 - testing.go:149: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.717-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: election won: tally=1 -testLoggerAdapter verbose: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.717-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: entering leader state: leader="Node at eae5a569-8074-3e76-b6c8-c068bf13107a [Leader]" - testing.go:149: server-eae5a569-8074-3e76-b6c8-c068bf13107a: 2024-06-16T08:26:29.717-1200 [INFO] server-eae5a569-8074-3e76-b6c8-c068bf13107a: entering leader state: leader="Node at eae5a569-8074-3e76-b6c8-c068bf13107a [Leader]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:29.918-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:29.7176431 -1200 -12 m=+6.724959581 (1 nodes), 178.60046ms from start of poll, 178.9662ms from cluster start. Timeout at 2024-06-16 08:26:29.91878398 -1200 -12 m=+6.926100461, 201.14088ms after stability - testing.go:149: cluster: 2024-06-16T08:26:29.918-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:29.7176431 -1200 -12 m=+6.724959581 (1 nodes), 178.60046ms from start of poll, 178.9662ms from cluster start. Timeout at 2024-06-16 08:26:29.91878398 -1200 -12 m=+6.926100461, 201.14088ms after stability ---- PASS: TestRaft_ApplyConcurrent_Timeout (0.38s) +testLoggerAdapter verbose: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.025+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24cb2fea-5c16-88ba-4e61-5bf572919754 Address:24cb2fea-5c16-88ba-4e61-5bf572919754}]" + testing.go:149: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.025+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24cb2fea-5c16-88ba-4e61-5bf572919754 Address:24cb2fea-5c16-88ba-4e61-5bf572919754}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.025+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:10.025+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.025+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: entering follower state: follower="Node at 24cb2fea-5c16-88ba-4e61-5bf572919754 [Follower]" leader= + testing.go:149: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.025+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: entering follower state: follower="Node at 24cb2fea-5c16-88ba-4e61-5bf572919754 [Follower]" leader= +testLoggerAdapter verbose: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.130+1400 [WARN] server-24cb2fea-5c16-88ba-4e61-5bf572919754: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.130+1400 [WARN] server-24cb2fea-5c16-88ba-4e61-5bf572919754: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.130+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: entering candidate state: node="Node at 24cb2fea-5c16-88ba-4e61-5bf572919754 [Candidate]" term=2 + testing.go:149: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.130+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: entering candidate state: node="Node at 24cb2fea-5c16-88ba-4e61-5bf572919754 [Candidate]" term=2 +testLoggerAdapter verbose: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.130+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: election won: tally=1 + testing.go:149: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.130+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: election won: tally=1 +testLoggerAdapter verbose: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.130+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: entering leader state: leader="Node at 24cb2fea-5c16-88ba-4e61-5bf572919754 [Leader]" + testing.go:149: server-24cb2fea-5c16-88ba-4e61-5bf572919754: 2023-05-16T04:06:10.130+1400 [INFO] server-24cb2fea-5c16-88ba-4e61-5bf572919754: entering leader state: leader="Node at 24cb2fea-5c16-88ba-4e61-5bf572919754 [Leader]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.332+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.131085075 +1400 +14 m=+6.933666730 (1 nodes), 105.344463ms from start of poll, 105.781268ms from cluster start. Timeout at 2023-05-16 04:06:10.332868837 +1400 +14 m=+7.135450472, 201.783742ms after stability + testing.go:149: cluster: 2023-05-16T04:06:10.332+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.131085075 +1400 +14 m=+6.933666730 (1 nodes), 105.344463ms from start of poll, 105.781268ms from cluster start. Timeout at 2023-05-16 04:06:10.332868837 +1400 +14 m=+7.135450472, 201.783742ms after stability +--- PASS: TestRaft_ApplyConcurrent_Timeout (0.31s) === RUN TestRaft_JoinNode -testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.920-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73c90638-2409-d3ea-eec6-372fbb2e896c Address:73c90638-2409-d3ea-eec6-372fbb2e896c} {Suffrage:Voter ID:server-6a24f9ca-593d-7129-aa83-48db88a724e4 Address:6a24f9ca-593d-7129-aa83-48db88a724e4}]" - testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.920-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73c90638-2409-d3ea-eec6-372fbb2e896c Address:73c90638-2409-d3ea-eec6-372fbb2e896c} {Suffrage:Voter ID:server-6a24f9ca-593d-7129-aa83-48db88a724e4 Address:6a24f9ca-593d-7129-aa83-48db88a724e4}]" -testLoggerAdapter verbose: server-6a24f9ca-593d-7129-aa83-48db88a724e4: 2024-06-16T08:26:29.920-1200 [INFO] server-6a24f9ca-593d-7129-aa83-48db88a724e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73c90638-2409-d3ea-eec6-372fbb2e896c Address:73c90638-2409-d3ea-eec6-372fbb2e896c} {Suffrage:Voter ID:server-6a24f9ca-593d-7129-aa83-48db88a724e4 Address:6a24f9ca-593d-7129-aa83-48db88a724e4}]" - testing.go:149: server-6a24f9ca-593d-7129-aa83-48db88a724e4: 2024-06-16T08:26:29.920-1200 [INFO] server-6a24f9ca-593d-7129-aa83-48db88a724e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73c90638-2409-d3ea-eec6-372fbb2e896c Address:73c90638-2409-d3ea-eec6-372fbb2e896c} {Suffrage:Voter ID:server-6a24f9ca-593d-7129-aa83-48db88a724e4 Address:6a24f9ca-593d-7129-aa83-48db88a724e4}]" -testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.920-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: entering follower state: follower="Node at 73c90638-2409-d3ea-eec6-372fbb2e896c [Follower]" leader= - testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.920-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: entering follower state: follower="Node at 73c90638-2409-d3ea-eec6-372fbb2e896c [Follower]" leader= -testLoggerAdapter verbose: server-6a24f9ca-593d-7129-aa83-48db88a724e4: 2024-06-16T08:26:29.920-1200 [INFO] server-6a24f9ca-593d-7129-aa83-48db88a724e4: entering follower state: follower="Node at 6a24f9ca-593d-7129-aa83-48db88a724e4 [Follower]" leader= - testing.go:149: server-6a24f9ca-593d-7129-aa83-48db88a724e4: 2024-06-16T08:26:29.920-1200 [INFO] server-6a24f9ca-593d-7129-aa83-48db88a724e4: entering follower state: follower="Node at 6a24f9ca-593d-7129-aa83-48db88a724e4 [Follower]" leader= -testLoggerAdapter verbose: server-e25e0d45-e527-3038-4ff3-da4f002f0c86: 2024-06-16T08:26:29.920-1200 [INFO] server-e25e0d45-e527-3038-4ff3-da4f002f0c86: initial configuration: index=0 servers=[] - testing.go:149: server-e25e0d45-e527-3038-4ff3-da4f002f0c86: 2024-06-16T08:26:29.920-1200 [INFO] server-e25e0d45-e527-3038-4ff3-da4f002f0c86: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:29.920-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:29.920-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-e25e0d45-e527-3038-4ff3-da4f002f0c86: 2024-06-16T08:26:29.920-1200 [INFO] server-e25e0d45-e527-3038-4ff3-da4f002f0c86: entering follower state: follower="Node at e25e0d45-e527-3038-4ff3-da4f002f0c86 [Follower]" leader= - testing.go:149: server-e25e0d45-e527-3038-4ff3-da4f002f0c86: 2024-06-16T08:26:29.920-1200 [INFO] server-e25e0d45-e527-3038-4ff3-da4f002f0c86: entering follower state: follower="Node at e25e0d45-e527-3038-4ff3-da4f002f0c86 [Follower]" leader= -testLoggerAdapter verbose: server-e25e0d45-e527-3038-4ff3-da4f002f0c86: 2024-06-16T08:26:29.983-1200 [WARN] server-e25e0d45-e527-3038-4ff3-da4f002f0c86: no known peers, aborting election - testing.go:149: server-e25e0d45-e527-3038-4ff3-da4f002f0c86: 2024-06-16T08:26:29.983-1200 [WARN] server-e25e0d45-e527-3038-4ff3-da4f002f0c86: no known peers, aborting election -testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [WARN] server-73c90638-2409-d3ea-eec6-372fbb2e896c: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [WARN] server-73c90638-2409-d3ea-eec6-372fbb2e896c: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: entering candidate state: node="Node at 73c90638-2409-d3ea-eec6-372fbb2e896c [Candidate]" term=2 - testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: entering candidate state: node="Node at 73c90638-2409-d3ea-eec6-372fbb2e896c [Candidate]" term=2 -testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: election won: tally=2 - testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: election won: tally=2 -testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: entering leader state: leader="Node at 73c90638-2409-d3ea-eec6-372fbb2e896c [Leader]" - testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: entering leader state: leader="Node at 73c90638-2409-d3ea-eec6-372fbb2e896c [Leader]" -testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: added peer, starting replication: peer=server-6a24f9ca-593d-7129-aa83-48db88a724e4 - testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: added peer, starting replication: peer=server-6a24f9ca-593d-7129-aa83-48db88a724e4 -testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: pipelining replication: peer="{Voter server-6a24f9ca-593d-7129-aa83-48db88a724e4 6a24f9ca-593d-7129-aa83-48db88a724e4}" - testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:29.984-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: pipelining replication: peer="{Voter server-6a24f9ca-593d-7129-aa83-48db88a724e4 6a24f9ca-593d-7129-aa83-48db88a724e4}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.089-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:29.98465606 -1200 -12 m=+6.991972541 (1 nodes), 63.98704ms from start of poll, 64.87644ms from cluster start. Timeout at 2024-06-16 08:26:30.08992704 -1200 -12 m=+7.097243501, 105.27096ms after stability - testing.go:149: cluster: 2024-06-16T08:26:30.089-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:29.98465606 -1200 -12 m=+6.991972541 (1 nodes), 63.98704ms from start of poll, 64.87644ms from cluster start. Timeout at 2024-06-16 08:26:30.08992704 -1200 -12 m=+7.097243501, 105.27096ms after stability -testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.090-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: updating configuration: command=AddStaging server-id=server-e25e0d45-e527-3038-4ff3-da4f002f0c86 server-addr=e25e0d45-e527-3038-4ff3-da4f002f0c86 servers="[{Suffrage:Voter ID:server-73c90638-2409-d3ea-eec6-372fbb2e896c Address:73c90638-2409-d3ea-eec6-372fbb2e896c} {Suffrage:Voter ID:server-6a24f9ca-593d-7129-aa83-48db88a724e4 Address:6a24f9ca-593d-7129-aa83-48db88a724e4} {Suffrage:Voter ID:server-e25e0d45-e527-3038-4ff3-da4f002f0c86 Address:e25e0d45-e527-3038-4ff3-da4f002f0c86}]" - testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.090-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: updating configuration: command=AddStaging server-id=server-e25e0d45-e527-3038-4ff3-da4f002f0c86 server-addr=e25e0d45-e527-3038-4ff3-da4f002f0c86 servers="[{Suffrage:Voter ID:server-73c90638-2409-d3ea-eec6-372fbb2e896c Address:73c90638-2409-d3ea-eec6-372fbb2e896c} {Suffrage:Voter ID:server-6a24f9ca-593d-7129-aa83-48db88a724e4 Address:6a24f9ca-593d-7129-aa83-48db88a724e4} {Suffrage:Voter ID:server-e25e0d45-e527-3038-4ff3-da4f002f0c86 Address:e25e0d45-e527-3038-4ff3-da4f002f0c86}]" -testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.090-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: added peer, starting replication: peer=server-e25e0d45-e527-3038-4ff3-da4f002f0c86 - testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.090-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: added peer, starting replication: peer=server-e25e0d45-e527-3038-4ff3-da4f002f0c86 -testLoggerAdapter verbose: server-e25e0d45-e527-3038-4ff3-da4f002f0c86: 2024-06-16T08:26:30.090-1200 [WARN] server-e25e0d45-e527-3038-4ff3-da4f002f0c86: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-e25e0d45-e527-3038-4ff3-da4f002f0c86: 2024-06-16T08:26:30.090-1200 [WARN] server-e25e0d45-e527-3038-4ff3-da4f002f0c86: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.090-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.090-1200 [WARN] server-73c90638-2409-d3ea-eec6-372fbb2e896c: appendEntries rejected, sending older logs: peer="{Voter server-e25e0d45-e527-3038-4ff3-da4f002f0c86 e25e0d45-e527-3038-4ff3-da4f002f0c86}" next=1 - testing.go:149: cluster: 2024-06-16T08:26:30.090-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.090-1200 [WARN] server-73c90638-2409-d3ea-eec6-372fbb2e896c: appendEntries rejected, sending older logs: peer="{Voter server-e25e0d45-e527-3038-4ff3-da4f002f0c86 e25e0d45-e527-3038-4ff3-da4f002f0c86}" next=1 -testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.090-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: pipelining replication: peer="{Voter server-e25e0d45-e527-3038-4ff3-da4f002f0c86 e25e0d45-e527-3038-4ff3-da4f002f0c86}" - testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.090-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: pipelining replication: peer="{Voter server-e25e0d45-e527-3038-4ff3-da4f002f0c86 e25e0d45-e527-3038-4ff3-da4f002f0c86}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.196-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.09066014 -1200 -12 m=+7.097976601 (1 nodes), 1.88µs from start of poll, 170.8805ms from cluster start. Timeout at 2024-06-16 08:26:30.196184 -1200 -12 m=+7.203500481, 105.52388ms after stability - testing.go:149: cluster: 2024-06-16T08:26:30.196-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.09066014 -1200 -12 m=+7.097976601 (1 nodes), 1.88µs from start of poll, 170.8805ms from cluster start. Timeout at 2024-06-16 08:26:30.196184 -1200 -12 m=+7.203500481, 105.52388ms after stability -testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.196-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: aborting pipeline replication: peer="{Voter server-6a24f9ca-593d-7129-aa83-48db88a724e4 6a24f9ca-593d-7129-aa83-48db88a724e4}" - testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.196-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: aborting pipeline replication: peer="{Voter server-6a24f9ca-593d-7129-aa83-48db88a724e4 6a24f9ca-593d-7129-aa83-48db88a724e4}" -testLoggerAdapter verbose: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.196-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: aborting pipeline replication: peer="{Voter server-e25e0d45-e527-3038-4ff3-da4f002f0c86 e25e0d45-e527-3038-4ff3-da4f002f0c86}" - testing.go:149: server-73c90638-2409-d3ea-eec6-372fbb2e896c: 2024-06-16T08:26:30.196-1200 [INFO] server-73c90638-2409-d3ea-eec6-372fbb2e896c: aborting pipeline replication: peer="{Voter server-e25e0d45-e527-3038-4ff3-da4f002f0c86 e25e0d45-e527-3038-4ff3-da4f002f0c86}" ---- PASS: TestRaft_JoinNode (0.28s) +testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.340+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 Address:c0eaf7c9-96a4-7f7e-144a-2adf280d2b38} {Suffrage:Voter ID:server-92c6272c-d873-ff1e-12a8-43fe516c3d6c Address:92c6272c-d873-ff1e-12a8-43fe516c3d6c}]" + testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.340+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 Address:c0eaf7c9-96a4-7f7e-144a-2adf280d2b38} {Suffrage:Voter ID:server-92c6272c-d873-ff1e-12a8-43fe516c3d6c Address:92c6272c-d873-ff1e-12a8-43fe516c3d6c}]" +testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.341+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: entering follower state: follower="Node at c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 [Follower]" leader= + testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.341+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: entering follower state: follower="Node at c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 [Follower]" leader= +testLoggerAdapter verbose: server-92c6272c-d873-ff1e-12a8-43fe516c3d6c: 2023-05-16T04:06:10.341+1400 [INFO] server-92c6272c-d873-ff1e-12a8-43fe516c3d6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 Address:c0eaf7c9-96a4-7f7e-144a-2adf280d2b38} {Suffrage:Voter ID:server-92c6272c-d873-ff1e-12a8-43fe516c3d6c Address:92c6272c-d873-ff1e-12a8-43fe516c3d6c}]" + testing.go:149: server-92c6272c-d873-ff1e-12a8-43fe516c3d6c: 2023-05-16T04:06:10.341+1400 [INFO] server-92c6272c-d873-ff1e-12a8-43fe516c3d6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 Address:c0eaf7c9-96a4-7f7e-144a-2adf280d2b38} {Suffrage:Voter ID:server-92c6272c-d873-ff1e-12a8-43fe516c3d6c Address:92c6272c-d873-ff1e-12a8-43fe516c3d6c}]" +testLoggerAdapter verbose: server-92c6272c-d873-ff1e-12a8-43fe516c3d6c: 2023-05-16T04:06:10.341+1400 [INFO] server-92c6272c-d873-ff1e-12a8-43fe516c3d6c: entering follower state: follower="Node at 92c6272c-d873-ff1e-12a8-43fe516c3d6c [Follower]" leader= + testing.go:149: server-92c6272c-d873-ff1e-12a8-43fe516c3d6c: 2023-05-16T04:06:10.341+1400 [INFO] server-92c6272c-d873-ff1e-12a8-43fe516c3d6c: entering follower state: follower="Node at 92c6272c-d873-ff1e-12a8-43fe516c3d6c [Follower]" leader= +testLoggerAdapter verbose: server-c406396c-e0d4-7306-17ae-18698aa688af: 2023-05-16T04:06:10.341+1400 [INFO] server-c406396c-e0d4-7306-17ae-18698aa688af: initial configuration: index=0 servers=[] + testing.go:149: server-c406396c-e0d4-7306-17ae-18698aa688af: 2023-05-16T04:06:10.341+1400 [INFO] server-c406396c-e0d4-7306-17ae-18698aa688af: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.341+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:10.341+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-c406396c-e0d4-7306-17ae-18698aa688af: 2023-05-16T04:06:10.341+1400 [INFO] server-c406396c-e0d4-7306-17ae-18698aa688af: entering follower state: follower="Node at c406396c-e0d4-7306-17ae-18698aa688af [Follower]" leader= + testing.go:149: server-c406396c-e0d4-7306-17ae-18698aa688af: 2023-05-16T04:06:10.341+1400 [INFO] server-c406396c-e0d4-7306-17ae-18698aa688af: entering follower state: follower="Node at c406396c-e0d4-7306-17ae-18698aa688af [Follower]" leader= +testLoggerAdapter verbose: server-c406396c-e0d4-7306-17ae-18698aa688af: 2023-05-16T04:06:10.410+1400 [WARN] server-c406396c-e0d4-7306-17ae-18698aa688af: no known peers, aborting election + testing.go:149: server-c406396c-e0d4-7306-17ae-18698aa688af: 2023-05-16T04:06:10.410+1400 [WARN] server-c406396c-e0d4-7306-17ae-18698aa688af: no known peers, aborting election +testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.411+1400 [WARN] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.411+1400 [WARN] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.411+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: entering candidate state: node="Node at c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 [Candidate]" term=2 + testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.411+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: entering candidate state: node="Node at c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 [Candidate]" term=2 +testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.412+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: election won: tally=2 + testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.412+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: election won: tally=2 +testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.412+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: entering leader state: leader="Node at c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 [Leader]" + testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.412+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: entering leader state: leader="Node at c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 [Leader]" +testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.412+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: added peer, starting replication: peer=server-92c6272c-d873-ff1e-12a8-43fe516c3d6c + testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.412+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: added peer, starting replication: peer=server-92c6272c-d873-ff1e-12a8-43fe516c3d6c +testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.412+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: pipelining replication: peer="{Voter server-92c6272c-d873-ff1e-12a8-43fe516c3d6c 92c6272c-d873-ff1e-12a8-43fe516c3d6c}" + testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.412+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: pipelining replication: peer="{Voter server-92c6272c-d873-ff1e-12a8-43fe516c3d6c 92c6272c-d873-ff1e-12a8-43fe516c3d6c}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.518+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.412629998 +1400 +14 m=+7.215211633 (1 nodes), 70.959617ms from start of poll, 72.123231ms from cluster start. Timeout at 2023-05-16 04:06:10.518556428 +1400 +14 m=+7.321138063, 105.92643ms after stability + testing.go:149: cluster: 2023-05-16T04:06:10.518+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.412629998 +1400 +14 m=+7.215211633 (1 nodes), 70.959617ms from start of poll, 72.123231ms from cluster start. Timeout at 2023-05-16 04:06:10.518556428 +1400 +14 m=+7.321138063, 105.92643ms after stability +testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.518+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: updating configuration: command=AddStaging server-id=server-c406396c-e0d4-7306-17ae-18698aa688af server-addr=c406396c-e0d4-7306-17ae-18698aa688af servers="[{Suffrage:Voter ID:server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 Address:c0eaf7c9-96a4-7f7e-144a-2adf280d2b38} {Suffrage:Voter ID:server-92c6272c-d873-ff1e-12a8-43fe516c3d6c Address:92c6272c-d873-ff1e-12a8-43fe516c3d6c} {Suffrage:Voter ID:server-c406396c-e0d4-7306-17ae-18698aa688af Address:c406396c-e0d4-7306-17ae-18698aa688af}]" + testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.518+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: updating configuration: command=AddStaging server-id=server-c406396c-e0d4-7306-17ae-18698aa688af server-addr=c406396c-e0d4-7306-17ae-18698aa688af servers="[{Suffrage:Voter ID:server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38 Address:c0eaf7c9-96a4-7f7e-144a-2adf280d2b38} {Suffrage:Voter ID:server-92c6272c-d873-ff1e-12a8-43fe516c3d6c Address:92c6272c-d873-ff1e-12a8-43fe516c3d6c} {Suffrage:Voter ID:server-c406396c-e0d4-7306-17ae-18698aa688af Address:c406396c-e0d4-7306-17ae-18698aa688af}]" +testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.519+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: added peer, starting replication: peer=server-c406396c-e0d4-7306-17ae-18698aa688af + testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.519+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: added peer, starting replication: peer=server-c406396c-e0d4-7306-17ae-18698aa688af +testLoggerAdapter verbose: server-c406396c-e0d4-7306-17ae-18698aa688af: 2023-05-16T04:06:10.519+1400 [WARN] server-c406396c-e0d4-7306-17ae-18698aa688af: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-c406396c-e0d4-7306-17ae-18698aa688af: 2023-05-16T04:06:10.519+1400 [WARN] server-c406396c-e0d4-7306-17ae-18698aa688af: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.519+1400 [WARN] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: appendEntries rejected, sending older logs: peer="{Voter server-c406396c-e0d4-7306-17ae-18698aa688af c406396c-e0d4-7306-17ae-18698aa688af}" next=1 + testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.519+1400 [WARN] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: appendEntries rejected, sending older logs: peer="{Voter server-c406396c-e0d4-7306-17ae-18698aa688af c406396c-e0d4-7306-17ae-18698aa688af}" next=1 +testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.519+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: pipelining replication: peer="{Voter server-c406396c-e0d4-7306-17ae-18698aa688af c406396c-e0d4-7306-17ae-18698aa688af}" + testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.519+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: pipelining replication: peer="{Voter server-c406396c-e0d4-7306-17ae-18698aa688af c406396c-e0d4-7306-17ae-18698aa688af}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.519+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:10.519+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.625+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.519864664 +1400 +14 m=+7.322446299 (1 nodes), 1.72µs from start of poll, 179.357897ms from cluster start. Timeout at 2023-05-16 04:06:10.625726273 +1400 +14 m=+7.428307968, 105.861669ms after stability + testing.go:149: cluster: 2023-05-16T04:06:10.625+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.519864664 +1400 +14 m=+7.322446299 (1 nodes), 1.72µs from start of poll, 179.357897ms from cluster start. Timeout at 2023-05-16 04:06:10.625726273 +1400 +14 m=+7.428307968, 105.861669ms after stability +testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.626+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: aborting pipeline replication: peer="{Voter server-c406396c-e0d4-7306-17ae-18698aa688af c406396c-e0d4-7306-17ae-18698aa688af}" + testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.626+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: aborting pipeline replication: peer="{Voter server-c406396c-e0d4-7306-17ae-18698aa688af c406396c-e0d4-7306-17ae-18698aa688af}" +testLoggerAdapter verbose: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.626+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: aborting pipeline replication: peer="{Voter server-92c6272c-d873-ff1e-12a8-43fe516c3d6c 92c6272c-d873-ff1e-12a8-43fe516c3d6c}" + testing.go:149: server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: 2023-05-16T04:06:10.626+1400 [INFO] server-c0eaf7c9-96a4-7f7e-144a-2adf280d2b38: aborting pipeline replication: peer="{Voter server-92c6272c-d873-ff1e-12a8-43fe516c3d6c 92c6272c-d873-ff1e-12a8-43fe516c3d6c}" +--- PASS: TestRaft_JoinNode (0.29s) === RUN TestRaft_JoinNode_ConfigStore -testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.197-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-598e58fc-cdf7-8d06-08d5-666ba4497f4a Address:598e58fc-cdf7-8d06-08d5-666ba4497f4a}]" - testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.197-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-598e58fc-cdf7-8d06-08d5-666ba4497f4a Address:598e58fc-cdf7-8d06-08d5-666ba4497f4a}]" -testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.197-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: entering follower state: follower="Node at 598e58fc-cdf7-8d06-08d5-666ba4497f4a [Follower]" leader= - testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.197-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: entering follower state: follower="Node at 598e58fc-cdf7-8d06-08d5-666ba4497f4a [Follower]" leader= -testLoggerAdapter verbose: server-07f472e8-ca3b-7e2a-414f-b15085e0d778: 2024-06-16T08:26:30.197-1200 [INFO] server-07f472e8-ca3b-7e2a-414f-b15085e0d778: initial configuration: index=0 servers=[] - testing.go:149: server-07f472e8-ca3b-7e2a-414f-b15085e0d778: 2024-06-16T08:26:30.197-1200 [INFO] server-07f472e8-ca3b-7e2a-414f-b15085e0d778: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-07f472e8-ca3b-7e2a-414f-b15085e0d778: 2024-06-16T08:26:30.197-1200 [INFO] server-07f472e8-ca3b-7e2a-414f-b15085e0d778: entering follower state: follower="Node at 07f472e8-ca3b-7e2a-414f-b15085e0d778 [Follower]" leader= - testing.go:149: server-07f472e8-ca3b-7e2a-414f-b15085e0d778: 2024-06-16T08:26:30.197-1200 [INFO] server-07f472e8-ca3b-7e2a-414f-b15085e0d778: entering follower state: follower="Node at 07f472e8-ca3b-7e2a-414f-b15085e0d778 [Follower]" leader= -testLoggerAdapter verbose: server-898769a0-7c22-9455-0fbd-cc7e431e89a4: 2024-06-16T08:26:30.198-1200 [INFO] server-898769a0-7c22-9455-0fbd-cc7e431e89a4: initial configuration: index=0 servers=[] - testing.go:149: server-898769a0-7c22-9455-0fbd-cc7e431e89a4: 2024-06-16T08:26:30.198-1200 [INFO] server-898769a0-7c22-9455-0fbd-cc7e431e89a4: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.198-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-898769a0-7c22-9455-0fbd-cc7e431e89a4: 2024-06-16T08:26:30.198-1200 [INFO] server-898769a0-7c22-9455-0fbd-cc7e431e89a4: entering follower state: follower="Node at 898769a0-7c22-9455-0fbd-cc7e431e89a4 [Follower]" leader= - testing.go:149: cluster: 2024-06-16T08:26:30.198-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: server-898769a0-7c22-9455-0fbd-cc7e431e89a4: 2024-06-16T08:26:30.198-1200 [INFO] server-898769a0-7c22-9455-0fbd-cc7e431e89a4: entering follower state: follower="Node at 898769a0-7c22-9455-0fbd-cc7e431e89a4 [Follower]" leader= -testLoggerAdapter verbose: server-07f472e8-ca3b-7e2a-414f-b15085e0d778: 2024-06-16T08:26:30.260-1200 [WARN] server-07f472e8-ca3b-7e2a-414f-b15085e0d778: no known peers, aborting election - testing.go:149: server-07f472e8-ca3b-7e2a-414f-b15085e0d778: 2024-06-16T08:26:30.260-1200 [WARN] server-07f472e8-ca3b-7e2a-414f-b15085e0d778: no known peers, aborting election -testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.274-1200 [WARN] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.274-1200 [WARN] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.274-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: entering candidate state: node="Node at 598e58fc-cdf7-8d06-08d5-666ba4497f4a [Candidate]" term=2 - testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.274-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: entering candidate state: node="Node at 598e58fc-cdf7-8d06-08d5-666ba4497f4a [Candidate]" term=2 -testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.274-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: election won: tally=1 - testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.274-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: election won: tally=1 -testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.275-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: entering leader state: leader="Node at 598e58fc-cdf7-8d06-08d5-666ba4497f4a [Leader]" - testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.275-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: entering leader state: leader="Node at 598e58fc-cdf7-8d06-08d5-666ba4497f4a [Leader]" -testLoggerAdapter verbose: server-898769a0-7c22-9455-0fbd-cc7e431e89a4: 2024-06-16T08:26:30.294-1200 [WARN] server-898769a0-7c22-9455-0fbd-cc7e431e89a4: no known peers, aborting election - testing.go:149: server-898769a0-7c22-9455-0fbd-cc7e431e89a4: 2024-06-16T08:26:30.294-1200 [WARN] server-898769a0-7c22-9455-0fbd-cc7e431e89a4: no known peers, aborting election -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.381-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.27511884 -1200 -12 m=+7.282435321 (1 nodes), 76.86966ms from start of poll, 77.80958ms from cluster start. Timeout at 2024-06-16 08:26:30.38116972 -1200 -12 m=+7.388486221, 106.0509ms after stability - testing.go:149: cluster: 2024-06-16T08:26:30.381-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.27511884 -1200 -12 m=+7.282435321 (1 nodes), 76.86966ms from start of poll, 77.80958ms from cluster start. Timeout at 2024-06-16 08:26:30.38116972 -1200 -12 m=+7.388486221, 106.0509ms after stability -testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.381-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: updating configuration: command=AddStaging server-id=server-07f472e8-ca3b-7e2a-414f-b15085e0d778 server-addr=07f472e8-ca3b-7e2a-414f-b15085e0d778 servers="[{Suffrage:Voter ID:server-598e58fc-cdf7-8d06-08d5-666ba4497f4a Address:598e58fc-cdf7-8d06-08d5-666ba4497f4a} {Suffrage:Voter ID:server-07f472e8-ca3b-7e2a-414f-b15085e0d778 Address:07f472e8-ca3b-7e2a-414f-b15085e0d778}]" - testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.381-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: updating configuration: command=AddStaging server-id=server-07f472e8-ca3b-7e2a-414f-b15085e0d778 server-addr=07f472e8-ca3b-7e2a-414f-b15085e0d778 servers="[{Suffrage:Voter ID:server-598e58fc-cdf7-8d06-08d5-666ba4497f4a Address:598e58fc-cdf7-8d06-08d5-666ba4497f4a} {Suffrage:Voter ID:server-07f472e8-ca3b-7e2a-414f-b15085e0d778 Address:07f472e8-ca3b-7e2a-414f-b15085e0d778}]" -testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.381-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: added peer, starting replication: peer=server-07f472e8-ca3b-7e2a-414f-b15085e0d778 - testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.381-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: added peer, starting replication: peer=server-07f472e8-ca3b-7e2a-414f-b15085e0d778 -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.381-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-07f472e8-ca3b-7e2a-414f-b15085e0d778: 2024-06-16T08:26:30.381-1200 [WARN] server-07f472e8-ca3b-7e2a-414f-b15085e0d778: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: cluster: 2024-06-16T08:26:30.381-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: server-07f472e8-ca3b-7e2a-414f-b15085e0d778: 2024-06-16T08:26:30.381-1200 [WARN] server-07f472e8-ca3b-7e2a-414f-b15085e0d778: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.381-1200 [WARN] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: appendEntries rejected, sending older logs: peer="{Voter server-07f472e8-ca3b-7e2a-414f-b15085e0d778 07f472e8-ca3b-7e2a-414f-b15085e0d778}" next=1 - testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.381-1200 [WARN] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: appendEntries rejected, sending older logs: peer="{Voter server-07f472e8-ca3b-7e2a-414f-b15085e0d778 07f472e8-ca3b-7e2a-414f-b15085e0d778}" next=1 -testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.381-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: pipelining replication: peer="{Voter server-07f472e8-ca3b-7e2a-414f-b15085e0d778 07f472e8-ca3b-7e2a-414f-b15085e0d778}" - testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.381-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: pipelining replication: peer="{Voter server-07f472e8-ca3b-7e2a-414f-b15085e0d778 07f472e8-ca3b-7e2a-414f-b15085e0d778}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.487-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.3817044 -1200 -12 m=+7.389020881 (1 nodes), 960ns from start of poll, 184.39514ms from cluster start. Timeout at 2024-06-16 08:26:30.48769232 -1200 -12 m=+7.495008781, 105.9879ms after stability - testing.go:149: cluster: 2024-06-16T08:26:30.487-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.3817044 -1200 -12 m=+7.389020881 (1 nodes), 960ns from start of poll, 184.39514ms from cluster start. Timeout at 2024-06-16 08:26:30.48769232 -1200 -12 m=+7.495008781, 105.9879ms after stability -testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.487-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: updating configuration: command=AddStaging server-id=server-898769a0-7c22-9455-0fbd-cc7e431e89a4 server-addr=898769a0-7c22-9455-0fbd-cc7e431e89a4 servers="[{Suffrage:Voter ID:server-598e58fc-cdf7-8d06-08d5-666ba4497f4a Address:598e58fc-cdf7-8d06-08d5-666ba4497f4a} {Suffrage:Voter ID:server-07f472e8-ca3b-7e2a-414f-b15085e0d778 Address:07f472e8-ca3b-7e2a-414f-b15085e0d778} {Suffrage:Voter ID:server-898769a0-7c22-9455-0fbd-cc7e431e89a4 Address:898769a0-7c22-9455-0fbd-cc7e431e89a4}]" - testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.487-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: updating configuration: command=AddStaging server-id=server-898769a0-7c22-9455-0fbd-cc7e431e89a4 server-addr=898769a0-7c22-9455-0fbd-cc7e431e89a4 servers="[{Suffrage:Voter ID:server-598e58fc-cdf7-8d06-08d5-666ba4497f4a Address:598e58fc-cdf7-8d06-08d5-666ba4497f4a} {Suffrage:Voter ID:server-07f472e8-ca3b-7e2a-414f-b15085e0d778 Address:07f472e8-ca3b-7e2a-414f-b15085e0d778} {Suffrage:Voter ID:server-898769a0-7c22-9455-0fbd-cc7e431e89a4 Address:898769a0-7c22-9455-0fbd-cc7e431e89a4}]" -testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.488-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: added peer, starting replication: peer=server-898769a0-7c22-9455-0fbd-cc7e431e89a4 - testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.488-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: added peer, starting replication: peer=server-898769a0-7c22-9455-0fbd-cc7e431e89a4 -testLoggerAdapter verbose: server-898769a0-7c22-9455-0fbd-cc7e431e89a4: 2024-06-16T08:26:30.488-1200 [WARN] server-898769a0-7c22-9455-0fbd-cc7e431e89a4: failed to get previous log: previous-index=4 last-index=0 error="log not found" - testing.go:149: server-898769a0-7c22-9455-0fbd-cc7e431e89a4: 2024-06-16T08:26:30.488-1200 [WARN] server-898769a0-7c22-9455-0fbd-cc7e431e89a4: failed to get previous log: previous-index=4 last-index=0 error="log not found" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.488-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.488-1200 [WARN] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: appendEntries rejected, sending older logs: peer="{Voter server-898769a0-7c22-9455-0fbd-cc7e431e89a4 898769a0-7c22-9455-0fbd-cc7e431e89a4}" next=1 - testing.go:149: cluster: 2024-06-16T08:26:30.488-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.488-1200 [WARN] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: appendEntries rejected, sending older logs: peer="{Voter server-898769a0-7c22-9455-0fbd-cc7e431e89a4 898769a0-7c22-9455-0fbd-cc7e431e89a4}" next=1 -testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.488-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: pipelining replication: peer="{Voter server-898769a0-7c22-9455-0fbd-cc7e431e89a4 898769a0-7c22-9455-0fbd-cc7e431e89a4}" - testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.488-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: pipelining replication: peer="{Voter server-898769a0-7c22-9455-0fbd-cc7e431e89a4 898769a0-7c22-9455-0fbd-cc7e431e89a4}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.593-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.48833266 -1200 -12 m=+7.495649141 (1 nodes), 1.1µs from start of poll, 291.0234ms from cluster start. Timeout at 2024-06-16 08:26:30.59335244 -1200 -12 m=+7.600668901, 105.01976ms after stability - testing.go:149: cluster: 2024-06-16T08:26:30.593-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.48833266 -1200 -12 m=+7.495649141 (1 nodes), 1.1µs from start of poll, 291.0234ms from cluster start. Timeout at 2024-06-16 08:26:30.59335244 -1200 -12 m=+7.600668901, 105.01976ms after stability -testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.593-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: aborting pipeline replication: peer="{Voter server-898769a0-7c22-9455-0fbd-cc7e431e89a4 898769a0-7c22-9455-0fbd-cc7e431e89a4}" - testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.593-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: aborting pipeline replication: peer="{Voter server-898769a0-7c22-9455-0fbd-cc7e431e89a4 898769a0-7c22-9455-0fbd-cc7e431e89a4}" -testLoggerAdapter verbose: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.593-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: aborting pipeline replication: peer="{Voter server-07f472e8-ca3b-7e2a-414f-b15085e0d778 07f472e8-ca3b-7e2a-414f-b15085e0d778}" - testing.go:149: server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: 2024-06-16T08:26:30.593-1200 [INFO] server-598e58fc-cdf7-8d06-08d5-666ba4497f4a: aborting pipeline replication: peer="{Voter server-07f472e8-ca3b-7e2a-414f-b15085e0d778 07f472e8-ca3b-7e2a-414f-b15085e0d778}" +testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.627+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-061d9543-072c-8615-a656-d8bdbe208c86 Address:061d9543-072c-8615-a656-d8bdbe208c86}]" + testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.627+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-061d9543-072c-8615-a656-d8bdbe208c86 Address:061d9543-072c-8615-a656-d8bdbe208c86}]" +testLoggerAdapter verbose: server-e6860ba8-8e12-83cd-f247-b2285f13e711: 2023-05-16T04:06:10.627+1400 [INFO] server-e6860ba8-8e12-83cd-f247-b2285f13e711: initial configuration: index=0 servers=[] + testing.go:149: server-e6860ba8-8e12-83cd-f247-b2285f13e711: 2023-05-16T04:06:10.627+1400 [INFO] server-e6860ba8-8e12-83cd-f247-b2285f13e711: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-57141e17-8ef4-1822-30de-a908d71f1065: 2023-05-16T04:06:10.628+1400 [INFO] server-57141e17-8ef4-1822-30de-a908d71f1065: initial configuration: index=0 servers=[] + testing.go:149: server-57141e17-8ef4-1822-30de-a908d71f1065: 2023-05-16T04:06:10.628+1400 [INFO] server-57141e17-8ef4-1822-30de-a908d71f1065: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.628+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:10.628+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.628+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: entering follower state: follower="Node at 061d9543-072c-8615-a656-d8bdbe208c86 [Follower]" leader= + testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.628+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: entering follower state: follower="Node at 061d9543-072c-8615-a656-d8bdbe208c86 [Follower]" leader= +testLoggerAdapter verbose: server-e6860ba8-8e12-83cd-f247-b2285f13e711: 2023-05-16T04:06:10.628+1400 [INFO] server-e6860ba8-8e12-83cd-f247-b2285f13e711: entering follower state: follower="Node at e6860ba8-8e12-83cd-f247-b2285f13e711 [Follower]" leader= + testing.go:149: server-e6860ba8-8e12-83cd-f247-b2285f13e711: 2023-05-16T04:06:10.628+1400 [INFO] server-e6860ba8-8e12-83cd-f247-b2285f13e711: entering follower state: follower="Node at e6860ba8-8e12-83cd-f247-b2285f13e711 [Follower]" leader= +testLoggerAdapter verbose: server-57141e17-8ef4-1822-30de-a908d71f1065: 2023-05-16T04:06:10.629+1400 [INFO] server-57141e17-8ef4-1822-30de-a908d71f1065: entering follower state: follower="Node at 57141e17-8ef4-1822-30de-a908d71f1065 [Follower]" leader= + testing.go:149: server-57141e17-8ef4-1822-30de-a908d71f1065: 2023-05-16T04:06:10.629+1400 [INFO] server-57141e17-8ef4-1822-30de-a908d71f1065: entering follower state: follower="Node at 57141e17-8ef4-1822-30de-a908d71f1065 [Follower]" leader= +testLoggerAdapter verbose: server-e6860ba8-8e12-83cd-f247-b2285f13e711: 2023-05-16T04:06:10.693+1400 [WARN] server-e6860ba8-8e12-83cd-f247-b2285f13e711: no known peers, aborting election + testing.go:149: server-e6860ba8-8e12-83cd-f247-b2285f13e711: 2023-05-16T04:06:10.693+1400 [WARN] server-e6860ba8-8e12-83cd-f247-b2285f13e711: no known peers, aborting election +testLoggerAdapter verbose: server-57141e17-8ef4-1822-30de-a908d71f1065: 2023-05-16T04:06:10.693+1400 [WARN] server-57141e17-8ef4-1822-30de-a908d71f1065: no known peers, aborting election + testing.go:149: server-57141e17-8ef4-1822-30de-a908d71f1065: 2023-05-16T04:06:10.693+1400 [WARN] server-57141e17-8ef4-1822-30de-a908d71f1065: no known peers, aborting election +testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.707+1400 [WARN] server-061d9543-072c-8615-a656-d8bdbe208c86: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.707+1400 [WARN] server-061d9543-072c-8615-a656-d8bdbe208c86: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.707+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: entering candidate state: node="Node at 061d9543-072c-8615-a656-d8bdbe208c86 [Candidate]" term=2 + testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.707+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: entering candidate state: node="Node at 061d9543-072c-8615-a656-d8bdbe208c86 [Candidate]" term=2 +testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.707+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: election won: tally=1 + testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.707+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: election won: tally=1 +testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.707+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: entering leader state: leader="Node at 061d9543-072c-8615-a656-d8bdbe208c86 [Leader]" + testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.707+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: entering leader state: leader="Node at 061d9543-072c-8615-a656-d8bdbe208c86 [Leader]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.813+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.708056825 +1400 +14 m=+7.510638460 (1 nodes), 79.417358ms from start of poll, 80.763114ms from cluster start. Timeout at 2023-05-16 04:06:10.813538289 +1400 +14 m=+7.616119944, 105.481484ms after stability + testing.go:149: cluster: 2023-05-16T04:06:10.813+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.708056825 +1400 +14 m=+7.510638460 (1 nodes), 79.417358ms from start of poll, 80.763114ms from cluster start. Timeout at 2023-05-16 04:06:10.813538289 +1400 +14 m=+7.616119944, 105.481484ms after stability +testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.813+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: updating configuration: command=AddStaging server-id=server-e6860ba8-8e12-83cd-f247-b2285f13e711 server-addr=e6860ba8-8e12-83cd-f247-b2285f13e711 servers="[{Suffrage:Voter ID:server-061d9543-072c-8615-a656-d8bdbe208c86 Address:061d9543-072c-8615-a656-d8bdbe208c86} {Suffrage:Voter ID:server-e6860ba8-8e12-83cd-f247-b2285f13e711 Address:e6860ba8-8e12-83cd-f247-b2285f13e711}]" + testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.813+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: updating configuration: command=AddStaging server-id=server-e6860ba8-8e12-83cd-f247-b2285f13e711 server-addr=e6860ba8-8e12-83cd-f247-b2285f13e711 servers="[{Suffrage:Voter ID:server-061d9543-072c-8615-a656-d8bdbe208c86 Address:061d9543-072c-8615-a656-d8bdbe208c86} {Suffrage:Voter ID:server-e6860ba8-8e12-83cd-f247-b2285f13e711 Address:e6860ba8-8e12-83cd-f247-b2285f13e711}]" +testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.814+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: added peer, starting replication: peer=server-e6860ba8-8e12-83cd-f247-b2285f13e711 + testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.814+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: added peer, starting replication: peer=server-e6860ba8-8e12-83cd-f247-b2285f13e711 +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.814+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:10.814+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-e6860ba8-8e12-83cd-f247-b2285f13e711: 2023-05-16T04:06:10.814+1400 [WARN] server-e6860ba8-8e12-83cd-f247-b2285f13e711: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-e6860ba8-8e12-83cd-f247-b2285f13e711: 2023-05-16T04:06:10.814+1400 [WARN] server-e6860ba8-8e12-83cd-f247-b2285f13e711: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.814+1400 [WARN] server-061d9543-072c-8615-a656-d8bdbe208c86: appendEntries rejected, sending older logs: peer="{Voter server-e6860ba8-8e12-83cd-f247-b2285f13e711 e6860ba8-8e12-83cd-f247-b2285f13e711}" next=1 + testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.814+1400 [WARN] server-061d9543-072c-8615-a656-d8bdbe208c86: appendEntries rejected, sending older logs: peer="{Voter server-e6860ba8-8e12-83cd-f247-b2285f13e711 e6860ba8-8e12-83cd-f247-b2285f13e711}" next=1 +testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.814+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: pipelining replication: peer="{Voter server-e6860ba8-8e12-83cd-f247-b2285f13e711 e6860ba8-8e12-83cd-f247-b2285f13e711}" + testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.814+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: pipelining replication: peer="{Voter server-e6860ba8-8e12-83cd-f247-b2285f13e711 e6860ba8-8e12-83cd-f247-b2285f13e711}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.919+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.814279458 +1400 +14 m=+7.616861093 (1 nodes), 2.46µs from start of poll, 186.985747ms from cluster start. Timeout at 2023-05-16 04:06:10.919554361 +1400 +14 m=+7.722136016, 105.274923ms after stability + testing.go:149: cluster: 2023-05-16T04:06:10.919+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.814279458 +1400 +14 m=+7.616861093 (1 nodes), 2.46µs from start of poll, 186.985747ms from cluster start. Timeout at 2023-05-16 04:06:10.919554361 +1400 +14 m=+7.722136016, 105.274923ms after stability +testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.919+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: updating configuration: command=AddStaging server-id=server-57141e17-8ef4-1822-30de-a908d71f1065 server-addr=57141e17-8ef4-1822-30de-a908d71f1065 servers="[{Suffrage:Voter ID:server-061d9543-072c-8615-a656-d8bdbe208c86 Address:061d9543-072c-8615-a656-d8bdbe208c86} {Suffrage:Voter ID:server-e6860ba8-8e12-83cd-f247-b2285f13e711 Address:e6860ba8-8e12-83cd-f247-b2285f13e711} {Suffrage:Voter ID:server-57141e17-8ef4-1822-30de-a908d71f1065 Address:57141e17-8ef4-1822-30de-a908d71f1065}]" + testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.919+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: updating configuration: command=AddStaging server-id=server-57141e17-8ef4-1822-30de-a908d71f1065 server-addr=57141e17-8ef4-1822-30de-a908d71f1065 servers="[{Suffrage:Voter ID:server-061d9543-072c-8615-a656-d8bdbe208c86 Address:061d9543-072c-8615-a656-d8bdbe208c86} {Suffrage:Voter ID:server-e6860ba8-8e12-83cd-f247-b2285f13e711 Address:e6860ba8-8e12-83cd-f247-b2285f13e711} {Suffrage:Voter ID:server-57141e17-8ef4-1822-30de-a908d71f1065 Address:57141e17-8ef4-1822-30de-a908d71f1065}]" +testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.920+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: added peer, starting replication: peer=server-57141e17-8ef4-1822-30de-a908d71f1065 + testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.920+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: added peer, starting replication: peer=server-57141e17-8ef4-1822-30de-a908d71f1065 +testLoggerAdapter verbose: server-57141e17-8ef4-1822-30de-a908d71f1065: 2023-05-16T04:06:10.920+1400 [WARN] server-57141e17-8ef4-1822-30de-a908d71f1065: failed to get previous log: previous-index=4 last-index=0 error="log not found" + testing.go:149: server-57141e17-8ef4-1822-30de-a908d71f1065: 2023-05-16T04:06:10.920+1400 [WARN] server-57141e17-8ef4-1822-30de-a908d71f1065: failed to get previous log: previous-index=4 last-index=0 error="log not found" +testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.920+1400 [WARN] server-061d9543-072c-8615-a656-d8bdbe208c86: appendEntries rejected, sending older logs: peer="{Voter server-57141e17-8ef4-1822-30de-a908d71f1065 57141e17-8ef4-1822-30de-a908d71f1065}" next=1 + testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.920+1400 [WARN] server-061d9543-072c-8615-a656-d8bdbe208c86: appendEntries rejected, sending older logs: peer="{Voter server-57141e17-8ef4-1822-30de-a908d71f1065 57141e17-8ef4-1822-30de-a908d71f1065}" next=1 +testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.920+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: pipelining replication: peer="{Voter server-57141e17-8ef4-1822-30de-a908d71f1065 57141e17-8ef4-1822-30de-a908d71f1065}" + testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:10.920+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: pipelining replication: peer="{Voter server-57141e17-8ef4-1822-30de-a908d71f1065 57141e17-8ef4-1822-30de-a908d71f1065}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:10.920+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:10.920+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.026+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.920995618 +1400 +14 m=+7.723577233 (1 nodes), 1.56µs from start of poll, 293.701887ms from cluster start. Timeout at 2023-05-16 04:06:11.026666165 +1400 +14 m=+7.829247800, 105.670567ms after stability + testing.go:149: cluster: 2023-05-16T04:06:11.026+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:10.920995618 +1400 +14 m=+7.723577233 (1 nodes), 1.56µs from start of poll, 293.701887ms from cluster start. Timeout at 2023-05-16 04:06:11.026666165 +1400 +14 m=+7.829247800, 105.670567ms after stability +testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:11.027+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: aborting pipeline replication: peer="{Voter server-57141e17-8ef4-1822-30de-a908d71f1065 57141e17-8ef4-1822-30de-a908d71f1065}" + testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:11.027+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: aborting pipeline replication: peer="{Voter server-57141e17-8ef4-1822-30de-a908d71f1065 57141e17-8ef4-1822-30de-a908d71f1065}" +testLoggerAdapter verbose: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:11.027+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: aborting pipeline replication: peer="{Voter server-e6860ba8-8e12-83cd-f247-b2285f13e711 e6860ba8-8e12-83cd-f247-b2285f13e711}" + testing.go:149: server-061d9543-072c-8615-a656-d8bdbe208c86: 2023-05-16T04:06:11.027+1400 [INFO] server-061d9543-072c-8615-a656-d8bdbe208c86: aborting pipeline replication: peer="{Voter server-e6860ba8-8e12-83cd-f247-b2285f13e711 e6860ba8-8e12-83cd-f247-b2285f13e711}" --- PASS: TestRaft_JoinNode_ConfigStore (0.40s) === RUN TestRaft_RemoveFollower -testLoggerAdapter verbose: server-af6a489e-95cc-80db-7e90-e7324b7fafa1: 2024-06-16T08:26:30.594-1200 [INFO] server-af6a489e-95cc-80db-7e90-e7324b7fafa1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af6a489e-95cc-80db-7e90-e7324b7fafa1 Address:af6a489e-95cc-80db-7e90-e7324b7fafa1} {Suffrage:Voter ID:server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254 Address:e8d6972d-b4c1-035a-a7b9-e8efc3a95254} {Suffrage:Voter ID:server-a9662707-ebcd-22e7-1816-adfd93dc412c Address:a9662707-ebcd-22e7-1816-adfd93dc412c}]" - testing.go:149: server-af6a489e-95cc-80db-7e90-e7324b7fafa1: 2024-06-16T08:26:30.594-1200 [INFO] server-af6a489e-95cc-80db-7e90-e7324b7fafa1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af6a489e-95cc-80db-7e90-e7324b7fafa1 Address:af6a489e-95cc-80db-7e90-e7324b7fafa1} {Suffrage:Voter ID:server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254 Address:e8d6972d-b4c1-035a-a7b9-e8efc3a95254} {Suffrage:Voter ID:server-a9662707-ebcd-22e7-1816-adfd93dc412c Address:a9662707-ebcd-22e7-1816-adfd93dc412c}]" -testLoggerAdapter verbose: server-af6a489e-95cc-80db-7e90-e7324b7fafa1: 2024-06-16T08:26:30.595-1200 [INFO] server-af6a489e-95cc-80db-7e90-e7324b7fafa1: entering follower state: follower="Node at af6a489e-95cc-80db-7e90-e7324b7fafa1 [Follower]" leader= - testing.go:149: server-af6a489e-95cc-80db-7e90-e7324b7fafa1: 2024-06-16T08:26:30.595-1200 [INFO] server-af6a489e-95cc-80db-7e90-e7324b7fafa1: entering follower state: follower="Node at af6a489e-95cc-80db-7e90-e7324b7fafa1 [Follower]" leader= -testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.595-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af6a489e-95cc-80db-7e90-e7324b7fafa1 Address:af6a489e-95cc-80db-7e90-e7324b7fafa1} {Suffrage:Voter ID:server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254 Address:e8d6972d-b4c1-035a-a7b9-e8efc3a95254} {Suffrage:Voter ID:server-a9662707-ebcd-22e7-1816-adfd93dc412c Address:a9662707-ebcd-22e7-1816-adfd93dc412c}]" - testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.595-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af6a489e-95cc-80db-7e90-e7324b7fafa1 Address:af6a489e-95cc-80db-7e90-e7324b7fafa1} {Suffrage:Voter ID:server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254 Address:e8d6972d-b4c1-035a-a7b9-e8efc3a95254} {Suffrage:Voter ID:server-a9662707-ebcd-22e7-1816-adfd93dc412c Address:a9662707-ebcd-22e7-1816-adfd93dc412c}]" -testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.595-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: entering follower state: follower="Node at e8d6972d-b4c1-035a-a7b9-e8efc3a95254 [Follower]" leader= - testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.595-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: entering follower state: follower="Node at e8d6972d-b4c1-035a-a7b9-e8efc3a95254 [Follower]" leader= -testLoggerAdapter verbose: server-a9662707-ebcd-22e7-1816-adfd93dc412c: 2024-06-16T08:26:30.595-1200 [INFO] server-a9662707-ebcd-22e7-1816-adfd93dc412c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af6a489e-95cc-80db-7e90-e7324b7fafa1 Address:af6a489e-95cc-80db-7e90-e7324b7fafa1} {Suffrage:Voter ID:server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254 Address:e8d6972d-b4c1-035a-a7b9-e8efc3a95254} {Suffrage:Voter ID:server-a9662707-ebcd-22e7-1816-adfd93dc412c Address:a9662707-ebcd-22e7-1816-adfd93dc412c}]" - testing.go:149: server-a9662707-ebcd-22e7-1816-adfd93dc412c: 2024-06-16T08:26:30.595-1200 [INFO] server-a9662707-ebcd-22e7-1816-adfd93dc412c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af6a489e-95cc-80db-7e90-e7324b7fafa1 Address:af6a489e-95cc-80db-7e90-e7324b7fafa1} {Suffrage:Voter ID:server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254 Address:e8d6972d-b4c1-035a-a7b9-e8efc3a95254} {Suffrage:Voter ID:server-a9662707-ebcd-22e7-1816-adfd93dc412c Address:a9662707-ebcd-22e7-1816-adfd93dc412c}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.595-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:30.595-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-a9662707-ebcd-22e7-1816-adfd93dc412c: 2024-06-16T08:26:30.595-1200 [INFO] server-a9662707-ebcd-22e7-1816-adfd93dc412c: entering follower state: follower="Node at a9662707-ebcd-22e7-1816-adfd93dc412c [Follower]" leader= - testing.go:149: server-a9662707-ebcd-22e7-1816-adfd93dc412c: 2024-06-16T08:26:30.595-1200 [INFO] server-a9662707-ebcd-22e7-1816-adfd93dc412c: entering follower state: follower="Node at a9662707-ebcd-22e7-1816-adfd93dc412c [Follower]" leader= -testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [WARN] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [WARN] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: entering candidate state: node="Node at e8d6972d-b4c1-035a-a7b9-e8efc3a95254 [Candidate]" term=2 - testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: entering candidate state: node="Node at e8d6972d-b4c1-035a-a7b9-e8efc3a95254 [Candidate]" term=2 -testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: election won: tally=2 - testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: election won: tally=2 -testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: entering leader state: leader="Node at e8d6972d-b4c1-035a-a7b9-e8efc3a95254 [Leader]" - testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: entering leader state: leader="Node at e8d6972d-b4c1-035a-a7b9-e8efc3a95254 [Leader]" -testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: added peer, starting replication: peer=server-af6a489e-95cc-80db-7e90-e7324b7fafa1 - testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.662-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: added peer, starting replication: peer=server-af6a489e-95cc-80db-7e90-e7324b7fafa1 -testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.663-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: added peer, starting replication: peer=server-a9662707-ebcd-22e7-1816-adfd93dc412c - testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.663-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: added peer, starting replication: peer=server-a9662707-ebcd-22e7-1816-adfd93dc412c -testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.663-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: pipelining replication: peer="{Voter server-af6a489e-95cc-80db-7e90-e7324b7fafa1 af6a489e-95cc-80db-7e90-e7324b7fafa1}" - testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.663-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: pipelining replication: peer="{Voter server-af6a489e-95cc-80db-7e90-e7324b7fafa1 af6a489e-95cc-80db-7e90-e7324b7fafa1}" -testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.663-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: pipelining replication: peer="{Voter server-a9662707-ebcd-22e7-1816-adfd93dc412c a9662707-ebcd-22e7-1816-adfd93dc412c}" - testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.663-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: pipelining replication: peer="{Voter server-a9662707-ebcd-22e7-1816-adfd93dc412c a9662707-ebcd-22e7-1816-adfd93dc412c}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.768-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.66295126 -1200 -12 m=+7.670267741 (1 nodes), 67.37054ms from start of poll, 68.21782ms from cluster start. Timeout at 2024-06-16 08:26:30.7684797 -1200 -12 m=+7.775796181, 105.52844ms after stability - testing.go:149: cluster: 2024-06-16T08:26:30.768-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:30.66295126 -1200 -12 m=+7.670267741 (1 nodes), 67.37054ms from start of poll, 68.21782ms from cluster start. Timeout at 2024-06-16 08:26:30.7684797 -1200 -12 m=+7.775796181, 105.52844ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.773-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:30.773-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.879-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:30.77393992 -1200 -12 m=+7.781256401 (2 nodes), 1.9µs from start of poll, 179.20648ms from cluster start. Timeout at 2024-06-16 08:26:30.87988976 -1200 -12 m=+7.887206221, 105.94982ms after stability - testing.go:149: cluster: 2024-06-16T08:26:30.879-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:30.77393992 -1200 -12 m=+7.781256401 (2 nodes), 1.9µs from start of poll, 179.20648ms from cluster start. Timeout at 2024-06-16 08:26:30.87988976 -1200 -12 m=+7.887206221, 105.94982ms after stability -testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.880-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: updating configuration: command=RemoveServer server-id=server-af6a489e-95cc-80db-7e90-e7324b7fafa1 server-addr= servers="[{Suffrage:Voter ID:server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254 Address:e8d6972d-b4c1-035a-a7b9-e8efc3a95254} {Suffrage:Voter ID:server-a9662707-ebcd-22e7-1816-adfd93dc412c Address:a9662707-ebcd-22e7-1816-adfd93dc412c}]" - testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.880-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: updating configuration: command=RemoveServer server-id=server-af6a489e-95cc-80db-7e90-e7324b7fafa1 server-addr= servers="[{Suffrage:Voter ID:server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254 Address:e8d6972d-b4c1-035a-a7b9-e8efc3a95254} {Suffrage:Voter ID:server-a9662707-ebcd-22e7-1816-adfd93dc412c Address:a9662707-ebcd-22e7-1816-adfd93dc412c}]" -testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.880-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: removed peer, stopping replication: peer=server-af6a489e-95cc-80db-7e90-e7324b7fafa1 last-index=3 - testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.880-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: removed peer, stopping replication: peer=server-af6a489e-95cc-80db-7e90-e7324b7fafa1 last-index=3 -testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.880-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: aborting pipeline replication: peer="{Voter server-af6a489e-95cc-80db-7e90-e7324b7fafa1 af6a489e-95cc-80db-7e90-e7324b7fafa1}" - testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.880-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: aborting pipeline replication: peer="{Voter server-af6a489e-95cc-80db-7e90-e7324b7fafa1 af6a489e-95cc-80db-7e90-e7324b7fafa1}" -testLoggerAdapter verbose: server-af6a489e-95cc-80db-7e90-e7324b7fafa1: 2024-06-16T08:26:30.961-1200 [WARN] server-af6a489e-95cc-80db-7e90-e7324b7fafa1: heartbeat timeout reached, starting election: last-leader=e8d6972d-b4c1-035a-a7b9-e8efc3a95254 - testing.go:149: server-af6a489e-95cc-80db-7e90-e7324b7fafa1: 2024-06-16T08:26:30.961-1200 [WARN] server-af6a489e-95cc-80db-7e90-e7324b7fafa1: heartbeat timeout reached, starting election: last-leader=e8d6972d-b4c1-035a-a7b9-e8efc3a95254 -testLoggerAdapter verbose: server-af6a489e-95cc-80db-7e90-e7324b7fafa1: 2024-06-16T08:26:30.961-1200 [INFO] server-af6a489e-95cc-80db-7e90-e7324b7fafa1: entering candidate state: node="Node at af6a489e-95cc-80db-7e90-e7324b7fafa1 [Candidate]" term=3 - testing.go:149: server-af6a489e-95cc-80db-7e90-e7324b7fafa1: 2024-06-16T08:26:30.961-1200 [INFO] server-af6a489e-95cc-80db-7e90-e7324b7fafa1: entering candidate state: node="Node at af6a489e-95cc-80db-7e90-e7324b7fafa1 [Candidate]" term=3 -testLoggerAdapter verbose: server-a9662707-ebcd-22e7-1816-adfd93dc412c: 2024-06-16T08:26:30.961-1200 [WARN] server-a9662707-ebcd-22e7-1816-adfd93dc412c: rejecting vote request since we have a leader: from=af6a489e-95cc-80db-7e90-e7324b7fafa1 leader=e8d6972d-b4c1-035a-a7b9-e8efc3a95254 - testing.go:149: server-a9662707-ebcd-22e7-1816-adfd93dc412c: 2024-06-16T08:26:30.961-1200 [WARN] server-a9662707-ebcd-22e7-1816-adfd93dc412c: rejecting vote request since we have a leader: from=af6a489e-95cc-80db-7e90-e7324b7fafa1 leader=e8d6972d-b4c1-035a-a7b9-e8efc3a95254 -testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.961-1200 [WARN] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: rejecting vote request since we have a leader: from=af6a489e-95cc-80db-7e90-e7324b7fafa1 leader=e8d6972d-b4c1-035a-a7b9-e8efc3a95254 - testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.961-1200 [WARN] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: rejecting vote request since we have a leader: from=af6a489e-95cc-80db-7e90-e7324b7fafa1 leader=e8d6972d-b4c1-035a-a7b9-e8efc3a95254 -testLoggerAdapter verbose: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.989-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: aborting pipeline replication: peer="{Voter server-a9662707-ebcd-22e7-1816-adfd93dc412c a9662707-ebcd-22e7-1816-adfd93dc412c}" - testing.go:149: server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: 2024-06-16T08:26:30.989-1200 [INFO] server-e8d6972d-b4c1-035a-a7b9-e8efc3a95254: aborting pipeline replication: peer="{Voter server-a9662707-ebcd-22e7-1816-adfd93dc412c a9662707-ebcd-22e7-1816-adfd93dc412c}" +testLoggerAdapter verbose: server-29ae47ac-1aec-8c65-d250-223c0072191e: 2023-05-16T04:06:11.028+1400 [INFO] server-29ae47ac-1aec-8c65-d250-223c0072191e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29ae47ac-1aec-8c65-d250-223c0072191e Address:29ae47ac-1aec-8c65-d250-223c0072191e} {Suffrage:Voter ID:server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad Address:e8c922c9-b53f-3531-0a15-6f87c8c7b0ad} {Suffrage:Voter ID:server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e Address:b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}]" + testing.go:149: server-29ae47ac-1aec-8c65-d250-223c0072191e: 2023-05-16T04:06:11.028+1400 [INFO] server-29ae47ac-1aec-8c65-d250-223c0072191e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29ae47ac-1aec-8c65-d250-223c0072191e Address:29ae47ac-1aec-8c65-d250-223c0072191e} {Suffrage:Voter ID:server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad Address:e8c922c9-b53f-3531-0a15-6f87c8c7b0ad} {Suffrage:Voter ID:server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e Address:b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}]" +testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.029+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29ae47ac-1aec-8c65-d250-223c0072191e Address:29ae47ac-1aec-8c65-d250-223c0072191e} {Suffrage:Voter ID:server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad Address:e8c922c9-b53f-3531-0a15-6f87c8c7b0ad} {Suffrage:Voter ID:server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e Address:b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}]" + testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.029+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29ae47ac-1aec-8c65-d250-223c0072191e Address:29ae47ac-1aec-8c65-d250-223c0072191e} {Suffrage:Voter ID:server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad Address:e8c922c9-b53f-3531-0a15-6f87c8c7b0ad} {Suffrage:Voter ID:server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e Address:b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}]" +testLoggerAdapter verbose: server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: 2023-05-16T04:06:11.029+1400 [INFO] server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29ae47ac-1aec-8c65-d250-223c0072191e Address:29ae47ac-1aec-8c65-d250-223c0072191e} {Suffrage:Voter ID:server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad Address:e8c922c9-b53f-3531-0a15-6f87c8c7b0ad} {Suffrage:Voter ID:server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e Address:b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}]" + testing.go:149: server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: 2023-05-16T04:06:11.029+1400 [INFO] server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29ae47ac-1aec-8c65-d250-223c0072191e Address:29ae47ac-1aec-8c65-d250-223c0072191e} {Suffrage:Voter ID:server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad Address:e8c922c9-b53f-3531-0a15-6f87c8c7b0ad} {Suffrage:Voter ID:server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e Address:b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.029+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:11.029+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-29ae47ac-1aec-8c65-d250-223c0072191e: 2023-05-16T04:06:11.029+1400 [INFO] server-29ae47ac-1aec-8c65-d250-223c0072191e: entering follower state: follower="Node at 29ae47ac-1aec-8c65-d250-223c0072191e [Follower]" leader= + testing.go:149: server-29ae47ac-1aec-8c65-d250-223c0072191e: 2023-05-16T04:06:11.029+1400 [INFO] server-29ae47ac-1aec-8c65-d250-223c0072191e: entering follower state: follower="Node at 29ae47ac-1aec-8c65-d250-223c0072191e [Follower]" leader= +testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.029+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: entering follower state: follower="Node at e8c922c9-b53f-3531-0a15-6f87c8c7b0ad [Follower]" leader= + testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.029+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: entering follower state: follower="Node at e8c922c9-b53f-3531-0a15-6f87c8c7b0ad [Follower]" leader= +testLoggerAdapter verbose: server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: 2023-05-16T04:06:11.029+1400 [INFO] server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: entering follower state: follower="Node at b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e [Follower]" leader= + testing.go:149: server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: 2023-05-16T04:06:11.029+1400 [INFO] server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: entering follower state: follower="Node at b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e [Follower]" leader= +testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [WARN] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [WARN] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: entering candidate state: node="Node at e8c922c9-b53f-3531-0a15-6f87c8c7b0ad [Candidate]" term=2 + testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: entering candidate state: node="Node at e8c922c9-b53f-3531-0a15-6f87c8c7b0ad [Candidate]" term=2 +testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: election won: tally=2 + testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: election won: tally=2 +testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: entering leader state: leader="Node at e8c922c9-b53f-3531-0a15-6f87c8c7b0ad [Leader]" + testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: entering leader state: leader="Node at e8c922c9-b53f-3531-0a15-6f87c8c7b0ad [Leader]" +testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: added peer, starting replication: peer=server-29ae47ac-1aec-8c65-d250-223c0072191e + testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.101+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: added peer, starting replication: peer=server-29ae47ac-1aec-8c65-d250-223c0072191e +testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.102+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: added peer, starting replication: peer=server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e + testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.102+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: added peer, starting replication: peer=server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e +testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.102+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: pipelining replication: peer="{Voter server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}" + testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.102+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: pipelining replication: peer="{Voter server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}" +testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.102+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: pipelining replication: peer="{Voter server-29ae47ac-1aec-8c65-d250-223c0072191e 29ae47ac-1aec-8c65-d250-223c0072191e}" + testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.102+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: pipelining replication: peer="{Voter server-29ae47ac-1aec-8c65-d250-223c0072191e 29ae47ac-1aec-8c65-d250-223c0072191e}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.208+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:11.102616161 +1400 +14 m=+7.905197796 (1 nodes), 73.000262ms from start of poll, 74.040334ms from cluster start. Timeout at 2023-05-16 04:06:11.208040805 +1400 +14 m=+8.010622480, 105.424684ms after stability + testing.go:149: cluster: 2023-05-16T04:06:11.208+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:11.102616161 +1400 +14 m=+7.905197796 (1 nodes), 73.000262ms from start of poll, 74.040334ms from cluster start. Timeout at 2023-05-16 04:06:11.208040805 +1400 +14 m=+8.010622480, 105.424684ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.214+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:11.214+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.319+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:11.214281459 +1400 +14 m=+8.016863094 (2 nodes), 2.98µs from start of poll, 185.705632ms from cluster start. Timeout at 2023-05-16 04:06:11.319536181 +1400 +14 m=+8.122117856, 105.254762ms after stability + testing.go:149: cluster: 2023-05-16T04:06:11.319+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:11.214281459 +1400 +14 m=+8.016863094 (2 nodes), 2.98µs from start of poll, 185.705632ms from cluster start. Timeout at 2023-05-16 04:06:11.319536181 +1400 +14 m=+8.122117856, 105.254762ms after stability +testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.319+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: updating configuration: command=RemoveServer server-id=server-29ae47ac-1aec-8c65-d250-223c0072191e server-addr= servers="[{Suffrage:Voter ID:server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad Address:e8c922c9-b53f-3531-0a15-6f87c8c7b0ad} {Suffrage:Voter ID:server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e Address:b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}]" + testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.319+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: updating configuration: command=RemoveServer server-id=server-29ae47ac-1aec-8c65-d250-223c0072191e server-addr= servers="[{Suffrage:Voter ID:server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad Address:e8c922c9-b53f-3531-0a15-6f87c8c7b0ad} {Suffrage:Voter ID:server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e Address:b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}]" +testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.319+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: removed peer, stopping replication: peer=server-29ae47ac-1aec-8c65-d250-223c0072191e last-index=3 + testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.319+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: removed peer, stopping replication: peer=server-29ae47ac-1aec-8c65-d250-223c0072191e last-index=3 +testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.320+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: aborting pipeline replication: peer="{Voter server-29ae47ac-1aec-8c65-d250-223c0072191e 29ae47ac-1aec-8c65-d250-223c0072191e}" + testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.320+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: aborting pipeline replication: peer="{Voter server-29ae47ac-1aec-8c65-d250-223c0072191e 29ae47ac-1aec-8c65-d250-223c0072191e}" +testLoggerAdapter verbose: server-29ae47ac-1aec-8c65-d250-223c0072191e: 2023-05-16T04:06:11.373+1400 [WARN] server-29ae47ac-1aec-8c65-d250-223c0072191e: heartbeat timeout reached, starting election: last-leader=e8c922c9-b53f-3531-0a15-6f87c8c7b0ad + testing.go:149: server-29ae47ac-1aec-8c65-d250-223c0072191e: 2023-05-16T04:06:11.373+1400 [WARN] server-29ae47ac-1aec-8c65-d250-223c0072191e: heartbeat timeout reached, starting election: last-leader=e8c922c9-b53f-3531-0a15-6f87c8c7b0ad +testLoggerAdapter verbose: server-29ae47ac-1aec-8c65-d250-223c0072191e: 2023-05-16T04:06:11.373+1400 [INFO] server-29ae47ac-1aec-8c65-d250-223c0072191e: entering candidate state: node="Node at 29ae47ac-1aec-8c65-d250-223c0072191e [Candidate]" term=3 + testing.go:149: server-29ae47ac-1aec-8c65-d250-223c0072191e: 2023-05-16T04:06:11.373+1400 [INFO] server-29ae47ac-1aec-8c65-d250-223c0072191e: entering candidate state: node="Node at 29ae47ac-1aec-8c65-d250-223c0072191e [Candidate]" term=3 +testLoggerAdapter verbose: server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: 2023-05-16T04:06:11.373+1400 [WARN] server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: rejecting vote request since we have a leader: from=29ae47ac-1aec-8c65-d250-223c0072191e leader=e8c922c9-b53f-3531-0a15-6f87c8c7b0ad + testing.go:149: server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: 2023-05-16T04:06:11.373+1400 [WARN] server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e: rejecting vote request since we have a leader: from=29ae47ac-1aec-8c65-d250-223c0072191e leader=e8c922c9-b53f-3531-0a15-6f87c8c7b0ad +testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.373+1400 [WARN] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: rejecting vote request since we have a leader: from=29ae47ac-1aec-8c65-d250-223c0072191e leader=e8c922c9-b53f-3531-0a15-6f87c8c7b0ad + testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.373+1400 [WARN] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: rejecting vote request since we have a leader: from=29ae47ac-1aec-8c65-d250-223c0072191e leader=e8c922c9-b53f-3531-0a15-6f87c8c7b0ad +testLoggerAdapter verbose: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.426+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: aborting pipeline replication: peer="{Voter server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}" + testing.go:149: server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: 2023-05-16T04:06:11.426+1400 [INFO] server-e8c922c9-b53f-3531-0a15-6f87c8c7b0ad: aborting pipeline replication: peer="{Voter server-b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e b17e1d42-7a7d-f9a1-f86a-71c27ec9f19e}" --- PASS: TestRaft_RemoveFollower (0.40s) === RUN TestRaft_RemoveLeader -testLoggerAdapter verbose: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:30.991-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e3a4095-7fbf-c2ef-587d-49551ac414fa Address:0e3a4095-7fbf-c2ef-587d-49551ac414fa} {Suffrage:Voter ID:server-cd4d8f62-3e79-73a2-1349-639a3e2f621b Address:cd4d8f62-3e79-73a2-1349-639a3e2f621b} {Suffrage:Voter ID:server-0062dc58-ef6b-0e34-7796-f85c50beed5d Address:0062dc58-ef6b-0e34-7796-f85c50beed5d}]" - testing.go:149: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:30.991-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e3a4095-7fbf-c2ef-587d-49551ac414fa Address:0e3a4095-7fbf-c2ef-587d-49551ac414fa} {Suffrage:Voter ID:server-cd4d8f62-3e79-73a2-1349-639a3e2f621b Address:cd4d8f62-3e79-73a2-1349-639a3e2f621b} {Suffrage:Voter ID:server-0062dc58-ef6b-0e34-7796-f85c50beed5d Address:0062dc58-ef6b-0e34-7796-f85c50beed5d}]" -testLoggerAdapter verbose: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:30.991-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: entering follower state: follower="Node at 0e3a4095-7fbf-c2ef-587d-49551ac414fa [Follower]" leader= - testing.go:149: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:30.991-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: entering follower state: follower="Node at 0e3a4095-7fbf-c2ef-587d-49551ac414fa [Follower]" leader= -testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:30.991-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e3a4095-7fbf-c2ef-587d-49551ac414fa Address:0e3a4095-7fbf-c2ef-587d-49551ac414fa} {Suffrage:Voter ID:server-cd4d8f62-3e79-73a2-1349-639a3e2f621b Address:cd4d8f62-3e79-73a2-1349-639a3e2f621b} {Suffrage:Voter ID:server-0062dc58-ef6b-0e34-7796-f85c50beed5d Address:0062dc58-ef6b-0e34-7796-f85c50beed5d}]" - testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:30.991-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e3a4095-7fbf-c2ef-587d-49551ac414fa Address:0e3a4095-7fbf-c2ef-587d-49551ac414fa} {Suffrage:Voter ID:server-cd4d8f62-3e79-73a2-1349-639a3e2f621b Address:cd4d8f62-3e79-73a2-1349-639a3e2f621b} {Suffrage:Voter ID:server-0062dc58-ef6b-0e34-7796-f85c50beed5d Address:0062dc58-ef6b-0e34-7796-f85c50beed5d}]" -testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:30.991-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: entering follower state: follower="Node at cd4d8f62-3e79-73a2-1349-639a3e2f621b [Follower]" leader= - testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:30.991-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: entering follower state: follower="Node at cd4d8f62-3e79-73a2-1349-639a3e2f621b [Follower]" leader= -testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:30.991-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e3a4095-7fbf-c2ef-587d-49551ac414fa Address:0e3a4095-7fbf-c2ef-587d-49551ac414fa} {Suffrage:Voter ID:server-cd4d8f62-3e79-73a2-1349-639a3e2f621b Address:cd4d8f62-3e79-73a2-1349-639a3e2f621b} {Suffrage:Voter ID:server-0062dc58-ef6b-0e34-7796-f85c50beed5d Address:0062dc58-ef6b-0e34-7796-f85c50beed5d}]" - testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:30.991-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e3a4095-7fbf-c2ef-587d-49551ac414fa Address:0e3a4095-7fbf-c2ef-587d-49551ac414fa} {Suffrage:Voter ID:server-cd4d8f62-3e79-73a2-1349-639a3e2f621b Address:cd4d8f62-3e79-73a2-1349-639a3e2f621b} {Suffrage:Voter ID:server-0062dc58-ef6b-0e34-7796-f85c50beed5d Address:0062dc58-ef6b-0e34-7796-f85c50beed5d}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:30.991-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:30.991-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:30.991-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: entering follower state: follower="Node at 0062dc58-ef6b-0e34-7796-f85c50beed5d [Follower]" leader= - testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:30.991-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: entering follower state: follower="Node at 0062dc58-ef6b-0e34-7796-f85c50beed5d [Follower]" leader= -testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.052-1200 [WARN] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.052-1200 [WARN] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.052-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: entering candidate state: node="Node at 0062dc58-ef6b-0e34-7796-f85c50beed5d [Candidate]" term=2 - testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.052-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: entering candidate state: node="Node at 0062dc58-ef6b-0e34-7796-f85c50beed5d [Candidate]" term=2 -testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.052-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: election won: tally=2 - testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.052-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: election won: tally=2 -testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.053-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: entering leader state: leader="Node at 0062dc58-ef6b-0e34-7796-f85c50beed5d [Leader]" - testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.053-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: entering leader state: leader="Node at 0062dc58-ef6b-0e34-7796-f85c50beed5d [Leader]" -testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.053-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: added peer, starting replication: peer=server-0e3a4095-7fbf-c2ef-587d-49551ac414fa - testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.053-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: added peer, starting replication: peer=server-0e3a4095-7fbf-c2ef-587d-49551ac414fa -testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.053-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: added peer, starting replication: peer=server-cd4d8f62-3e79-73a2-1349-639a3e2f621b - testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.053-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: added peer, starting replication: peer=server-cd4d8f62-3e79-73a2-1349-639a3e2f621b -testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.054-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: pipelining replication: peer="{Voter server-0e3a4095-7fbf-c2ef-587d-49551ac414fa 0e3a4095-7fbf-c2ef-587d-49551ac414fa}" - testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.054-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: pipelining replication: peer="{Voter server-0e3a4095-7fbf-c2ef-587d-49551ac414fa 0e3a4095-7fbf-c2ef-587d-49551ac414fa}" -testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.054-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: pipelining replication: peer="{Voter server-cd4d8f62-3e79-73a2-1349-639a3e2f621b cd4d8f62-3e79-73a2-1349-639a3e2f621b}" - testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.054-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: pipelining replication: peer="{Voter server-cd4d8f62-3e79-73a2-1349-639a3e2f621b cd4d8f62-3e79-73a2-1349-639a3e2f621b}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.159-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:31.05346856 -1200 -12 m=+8.060785041 (1 nodes), 61.80372ms from start of poll, 62.69052ms from cluster start. Timeout at 2024-06-16 08:26:31.15918994 -1200 -12 m=+8.166506421, 105.72138ms after stability - testing.go:149: cluster: 2024-06-16T08:26:31.159-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:31.05346856 -1200 -12 m=+8.060785041 (1 nodes), 61.80372ms from start of poll, 62.69052ms from cluster start. Timeout at 2024-06-16 08:26:31.15918994 -1200 -12 m=+8.166506421, 105.72138ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.164-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:31.164-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.270-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:31.16466338 -1200 -12 m=+8.171979861 (2 nodes), 3.38µs from start of poll, 173.88534ms from cluster start. Timeout at 2024-06-16 08:26:31.27035898 -1200 -12 m=+8.277675541, 105.69568ms after stability - testing.go:149: cluster: 2024-06-16T08:26:31.270-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:31.16466338 -1200 -12 m=+8.171979861 (2 nodes), 3.38µs from start of poll, 173.88534ms from cluster start. Timeout at 2024-06-16 08:26:31.27035898 -1200 -12 m=+8.277675541, 105.69568ms after stability -testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.270-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: updating configuration: command=RemoveServer server-id=server-0062dc58-ef6b-0e34-7796-f85c50beed5d server-addr= servers="[{Suffrage:Voter ID:server-0e3a4095-7fbf-c2ef-587d-49551ac414fa Address:0e3a4095-7fbf-c2ef-587d-49551ac414fa} {Suffrage:Voter ID:server-cd4d8f62-3e79-73a2-1349-639a3e2f621b Address:cd4d8f62-3e79-73a2-1349-639a3e2f621b}]" - testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.270-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: updating configuration: command=RemoveServer server-id=server-0062dc58-ef6b-0e34-7796-f85c50beed5d server-addr= servers="[{Suffrage:Voter ID:server-0e3a4095-7fbf-c2ef-587d-49551ac414fa Address:0e3a4095-7fbf-c2ef-587d-49551ac414fa} {Suffrage:Voter ID:server-cd4d8f62-3e79-73a2-1349-639a3e2f621b Address:cd4d8f62-3e79-73a2-1349-639a3e2f621b}]" -testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.271-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: removed ourself, shutting down - testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.271-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: removed ourself, shutting down -testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.271-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: aborting pipeline replication: peer="{Voter server-cd4d8f62-3e79-73a2-1349-639a3e2f621b cd4d8f62-3e79-73a2-1349-639a3e2f621b}" - testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.271-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: aborting pipeline replication: peer="{Voter server-cd4d8f62-3e79-73a2-1349-639a3e2f621b cd4d8f62-3e79-73a2-1349-639a3e2f621b}" -testLoggerAdapter verbose: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.271-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: aborting pipeline replication: peer="{Voter server-0e3a4095-7fbf-c2ef-587d-49551ac414fa 0e3a4095-7fbf-c2ef-587d-49551ac414fa}" - testing.go:149: server-0062dc58-ef6b-0e34-7796-f85c50beed5d: 2024-06-16T08:26:31.271-1200 [INFO] server-0062dc58-ef6b-0e34-7796-f85c50beed5d: aborting pipeline replication: peer="{Voter server-0e3a4095-7fbf-c2ef-587d-49551ac414fa 0e3a4095-7fbf-c2ef-587d-49551ac414fa}" -testLoggerAdapter verbose: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:31.375-1200 [WARN] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: heartbeat timeout reached, starting election: last-leader=0062dc58-ef6b-0e34-7796-f85c50beed5d - testing.go:149: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:31.375-1200 [WARN] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: heartbeat timeout reached, starting election: last-leader=0062dc58-ef6b-0e34-7796-f85c50beed5d -testLoggerAdapter verbose: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:31.375-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: entering candidate state: node="Node at 0e3a4095-7fbf-c2ef-587d-49551ac414fa [Candidate]" term=3 - testing.go:149: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:31.375-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: entering candidate state: node="Node at 0e3a4095-7fbf-c2ef-587d-49551ac414fa [Candidate]" term=3 -testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.375-1200 [WARN] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: rejecting vote request since we have a leader: from=0e3a4095-7fbf-c2ef-587d-49551ac414fa leader=0062dc58-ef6b-0e34-7796-f85c50beed5d - testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.375-1200 [WARN] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: rejecting vote request since we have a leader: from=0e3a4095-7fbf-c2ef-587d-49551ac414fa leader=0062dc58-ef6b-0e34-7796-f85c50beed5d -testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.375-1200 [WARN] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: heartbeat timeout reached, starting election: last-leader=0062dc58-ef6b-0e34-7796-f85c50beed5d - testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.375-1200 [WARN] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: heartbeat timeout reached, starting election: last-leader=0062dc58-ef6b-0e34-7796-f85c50beed5d -testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.375-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: entering candidate state: node="Node at cd4d8f62-3e79-73a2-1349-639a3e2f621b [Candidate]" term=3 - testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.375-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: entering candidate state: node="Node at cd4d8f62-3e79-73a2-1349-639a3e2f621b [Candidate]" term=3 -testLoggerAdapter verbose: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:31.375-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: duplicate requestVote for same term: term=3 - testing.go:149: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:31.375-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: duplicate requestVote for same term: term=3 -testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.442-1200 [WARN] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: Election timeout reached, restarting election - testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.442-1200 [WARN] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: Election timeout reached, restarting election -testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.442-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: entering candidate state: node="Node at cd4d8f62-3e79-73a2-1349-639a3e2f621b [Candidate]" term=4 - testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.442-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: entering candidate state: node="Node at cd4d8f62-3e79-73a2-1349-639a3e2f621b [Candidate]" term=4 -testLoggerAdapter verbose: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:31.442-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: entering follower state: follower="Node at 0e3a4095-7fbf-c2ef-587d-49551ac414fa [Follower]" leader= - testing.go:149: server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: 2024-06-16T08:26:31.442-1200 [INFO] server-0e3a4095-7fbf-c2ef-587d-49551ac414fa: entering follower state: follower="Node at 0e3a4095-7fbf-c2ef-587d-49551ac414fa [Follower]" leader= -testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.443-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: election won: tally=2 - testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.443-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: election won: tally=2 -testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.443-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: entering leader state: leader="Node at cd4d8f62-3e79-73a2-1349-639a3e2f621b [Leader]" - testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.443-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: entering leader state: leader="Node at cd4d8f62-3e79-73a2-1349-639a3e2f621b [Leader]" -testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.443-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: added peer, starting replication: peer=server-0e3a4095-7fbf-c2ef-587d-49551ac414fa - testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.443-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: added peer, starting replication: peer=server-0e3a4095-7fbf-c2ef-587d-49551ac414fa -testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.443-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: pipelining replication: peer="{Voter server-0e3a4095-7fbf-c2ef-587d-49551ac414fa 0e3a4095-7fbf-c2ef-587d-49551ac414fa}" - testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.443-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: pipelining replication: peer="{Voter server-0e3a4095-7fbf-c2ef-587d-49551ac414fa 0e3a4095-7fbf-c2ef-587d-49551ac414fa}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.482-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:31.482-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.588-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:31.48306386 -1200 -12 m=+8.490380321 (1 nodes), 54.82µs from start of poll, 492.2858ms from cluster start. Timeout at 2024-06-16 08:26:31.58823992 -1200 -12 m=+8.595556401, 105.17608ms after stability - testing.go:149: cluster: 2024-06-16T08:26:31.588-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:31.48306386 -1200 -12 m=+8.490380321 (1 nodes), 54.82µs from start of poll, 492.2858ms from cluster start. Timeout at 2024-06-16 08:26:31.58823992 -1200 -12 m=+8.595556401, 105.17608ms after stability -testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.588-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: aborting pipeline replication: peer="{Voter server-0e3a4095-7fbf-c2ef-587d-49551ac414fa 0e3a4095-7fbf-c2ef-587d-49551ac414fa}" - testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.588-1200 [INFO] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: aborting pipeline replication: peer="{Voter server-0e3a4095-7fbf-c2ef-587d-49551ac414fa 0e3a4095-7fbf-c2ef-587d-49551ac414fa}" -testLoggerAdapter verbose: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.638-1200 [ERROR] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: failed to heartbeat to: peer=0e3a4095-7fbf-c2ef-587d-49551ac414fa error="command timed out" - testing.go:149: server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: 2024-06-16T08:26:31.638-1200 [ERROR] server-cd4d8f62-3e79-73a2-1349-639a3e2f621b: failed to heartbeat to: peer=0e3a4095-7fbf-c2ef-587d-49551ac414fa error="command timed out" ---- PASS: TestRaft_RemoveLeader (0.65s) +testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.427+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a592c42f-3461-e569-9c07-b354084d8ff7 Address:a592c42f-3461-e569-9c07-b354084d8ff7} {Suffrage:Voter ID:server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 Address:8b196570-6b00-d4e2-54d5-d43ed3f56f82} {Suffrage:Voter ID:server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 Address:e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}]" + testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.427+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a592c42f-3461-e569-9c07-b354084d8ff7 Address:a592c42f-3461-e569-9c07-b354084d8ff7} {Suffrage:Voter ID:server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 Address:8b196570-6b00-d4e2-54d5-d43ed3f56f82} {Suffrage:Voter ID:server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 Address:e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}]" +testLoggerAdapter verbose: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.428+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a592c42f-3461-e569-9c07-b354084d8ff7 Address:a592c42f-3461-e569-9c07-b354084d8ff7} {Suffrage:Voter ID:server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 Address:8b196570-6b00-d4e2-54d5-d43ed3f56f82} {Suffrage:Voter ID:server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 Address:e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}]" + testing.go:149: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.428+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a592c42f-3461-e569-9c07-b354084d8ff7 Address:a592c42f-3461-e569-9c07-b354084d8ff7} {Suffrage:Voter ID:server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 Address:8b196570-6b00-d4e2-54d5-d43ed3f56f82} {Suffrage:Voter ID:server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 Address:e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}]" +testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.428+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a592c42f-3461-e569-9c07-b354084d8ff7 Address:a592c42f-3461-e569-9c07-b354084d8ff7} {Suffrage:Voter ID:server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 Address:8b196570-6b00-d4e2-54d5-d43ed3f56f82} {Suffrage:Voter ID:server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 Address:e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}]" + testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.428+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a592c42f-3461-e569-9c07-b354084d8ff7 Address:a592c42f-3461-e569-9c07-b354084d8ff7} {Suffrage:Voter ID:server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 Address:8b196570-6b00-d4e2-54d5-d43ed3f56f82} {Suffrage:Voter ID:server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 Address:e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.428+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:11.428+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.428+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: entering follower state: follower="Node at a592c42f-3461-e569-9c07-b354084d8ff7 [Follower]" leader= + testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.428+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: entering follower state: follower="Node at a592c42f-3461-e569-9c07-b354084d8ff7 [Follower]" leader= +testLoggerAdapter verbose: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.428+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: entering follower state: follower="Node at 8b196570-6b00-d4e2-54d5-d43ed3f56f82 [Follower]" leader= + testing.go:149: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.428+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: entering follower state: follower="Node at 8b196570-6b00-d4e2-54d5-d43ed3f56f82 [Follower]" leader= +testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.429+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: entering follower state: follower="Node at e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 [Follower]" leader= + testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.429+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: entering follower state: follower="Node at e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 [Follower]" leader= +testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.481+1400 [WARN] server-a592c42f-3461-e569-9c07-b354084d8ff7: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.481+1400 [WARN] server-a592c42f-3461-e569-9c07-b354084d8ff7: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: entering candidate state: node="Node at a592c42f-3461-e569-9c07-b354084d8ff7 [Candidate]" term=2 + testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: entering candidate state: node="Node at a592c42f-3461-e569-9c07-b354084d8ff7 [Candidate]" term=2 +testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: election won: tally=2 + testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: election won: tally=2 +testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: entering leader state: leader="Node at a592c42f-3461-e569-9c07-b354084d8ff7 [Leader]" + testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: entering leader state: leader="Node at a592c42f-3461-e569-9c07-b354084d8ff7 [Leader]" +testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: added peer, starting replication: peer=server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 + testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: added peer, starting replication: peer=server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 +testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: added peer, starting replication: peer=server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 + testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: added peer, starting replication: peer=server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 +testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: pipelining replication: peer="{Voter server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}" + testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.482+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: pipelining replication: peer="{Voter server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}" +testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.483+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: pipelining replication: peer="{Voter server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 8b196570-6b00-d4e2-54d5-d43ed3f56f82}" + testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.483+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: pipelining replication: peer="{Voter server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 8b196570-6b00-d4e2-54d5-d43ed3f56f82}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.589+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:11.483279713 +1400 +14 m=+8.285861328 (1 nodes), 54.484363ms from start of poll, 55.612616ms from cluster start. Timeout at 2023-05-16 04:06:11.589061442 +1400 +14 m=+8.391643077, 105.781749ms after stability + testing.go:149: cluster: 2023-05-16T04:06:11.589+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:11.483279713 +1400 +14 m=+8.285861328 (1 nodes), 54.484363ms from start of poll, 55.612616ms from cluster start. Timeout at 2023-05-16 04:06:11.589061442 +1400 +14 m=+8.391643077, 105.781749ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.595+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:11.595+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.700+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:11.595304355 +1400 +14 m=+8.397885970 (2 nodes), 3.44µs from start of poll, 167.637258ms from cluster start. Timeout at 2023-05-16 04:06:11.700311235 +1400 +14 m=+8.502892870, 105.0069ms after stability + testing.go:149: cluster: 2023-05-16T04:06:11.700+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:11.595304355 +1400 +14 m=+8.397885970 (2 nodes), 3.44µs from start of poll, 167.637258ms from cluster start. Timeout at 2023-05-16 04:06:11.700311235 +1400 +14 m=+8.502892870, 105.0069ms after stability +testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.700+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: updating configuration: command=RemoveServer server-id=server-a592c42f-3461-e569-9c07-b354084d8ff7 server-addr= servers="[{Suffrage:Voter ID:server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 Address:8b196570-6b00-d4e2-54d5-d43ed3f56f82} {Suffrage:Voter ID:server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 Address:e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}]" + testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.700+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: updating configuration: command=RemoveServer server-id=server-a592c42f-3461-e569-9c07-b354084d8ff7 server-addr= servers="[{Suffrage:Voter ID:server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 Address:8b196570-6b00-d4e2-54d5-d43ed3f56f82} {Suffrage:Voter ID:server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 Address:e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}]" +testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.700+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: removed ourself, shutting down + testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.700+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: removed ourself, shutting down +testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.701+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: aborting pipeline replication: peer="{Voter server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}" + testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.701+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: aborting pipeline replication: peer="{Voter server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65}" +testLoggerAdapter verbose: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.701+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: aborting pipeline replication: peer="{Voter server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 8b196570-6b00-d4e2-54d5-d43ed3f56f82}" + testing.go:149: server-a592c42f-3461-e569-9c07-b354084d8ff7: 2023-05-16T04:06:11.701+1400 [INFO] server-a592c42f-3461-e569-9c07-b354084d8ff7: aborting pipeline replication: peer="{Voter server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 8b196570-6b00-d4e2-54d5-d43ed3f56f82}" +testLoggerAdapter verbose: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.769+1400 [WARN] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: heartbeat timeout reached, starting election: last-leader=a592c42f-3461-e569-9c07-b354084d8ff7 + testing.go:149: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.769+1400 [WARN] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: heartbeat timeout reached, starting election: last-leader=a592c42f-3461-e569-9c07-b354084d8ff7 +testLoggerAdapter verbose: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.769+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: entering candidate state: node="Node at 8b196570-6b00-d4e2-54d5-d43ed3f56f82 [Candidate]" term=3 + testing.go:149: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.769+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: entering candidate state: node="Node at 8b196570-6b00-d4e2-54d5-d43ed3f56f82 [Candidate]" term=3 +testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.769+1400 [WARN] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: rejecting vote request since we have a leader: from=8b196570-6b00-d4e2-54d5-d43ed3f56f82 leader=a592c42f-3461-e569-9c07-b354084d8ff7 + testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.769+1400 [WARN] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: rejecting vote request since we have a leader: from=8b196570-6b00-d4e2-54d5-d43ed3f56f82 leader=a592c42f-3461-e569-9c07-b354084d8ff7 +testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.780+1400 [WARN] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: heartbeat timeout reached, starting election: last-leader=a592c42f-3461-e569-9c07-b354084d8ff7 + testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.780+1400 [WARN] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: heartbeat timeout reached, starting election: last-leader=a592c42f-3461-e569-9c07-b354084d8ff7 +testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.781+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: entering candidate state: node="Node at e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 [Candidate]" term=3 + testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.781+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: entering candidate state: node="Node at e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 [Candidate]" term=3 +testLoggerAdapter verbose: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.781+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: duplicate requestVote for same term: term=3 + testing.go:149: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.781+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: duplicate requestVote for same term: term=3 +testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.836+1400 [WARN] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: Election timeout reached, restarting election + testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.836+1400 [WARN] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: Election timeout reached, restarting election +testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.836+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: entering candidate state: node="Node at e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 [Candidate]" term=4 + testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.836+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: entering candidate state: node="Node at e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 [Candidate]" term=4 +testLoggerAdapter verbose: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.836+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: entering follower state: follower="Node at 8b196570-6b00-d4e2-54d5-d43ed3f56f82 [Follower]" leader= + testing.go:149: server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: 2023-05-16T04:06:11.836+1400 [INFO] server-8b196570-6b00-d4e2-54d5-d43ed3f56f82: entering follower state: follower="Node at 8b196570-6b00-d4e2-54d5-d43ed3f56f82 [Follower]" leader= +testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.837+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: election won: tally=2 + testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.837+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: election won: tally=2 +testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.837+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: entering leader state: leader="Node at e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 [Leader]" + testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.837+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: entering leader state: leader="Node at e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65 [Leader]" +testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.837+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: added peer, starting replication: peer=server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 + testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.837+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: added peer, starting replication: peer=server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 +testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.837+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: pipelining replication: peer="{Voter server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 8b196570-6b00-d4e2-54d5-d43ed3f56f82}" + testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:11.837+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: pipelining replication: peer="{Voter server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 8b196570-6b00-d4e2-54d5-d43ed3f56f82}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:11.912+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:11.912+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.018+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:11.912402318 +1400 +14 m=+8.714983933 (1 nodes), 116.642µs from start of poll, 484.735221ms from cluster start. Timeout at 2023-05-16 04:06:12.018406869 +1400 +14 m=+8.820988524, 106.004591ms after stability + testing.go:149: cluster: 2023-05-16T04:06:12.018+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:11.912402318 +1400 +14 m=+8.714983933 (1 nodes), 116.642µs from start of poll, 484.735221ms from cluster start. Timeout at 2023-05-16 04:06:12.018406869 +1400 +14 m=+8.820988524, 106.004591ms after stability +testLoggerAdapter verbose: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:12.018+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: aborting pipeline replication: peer="{Voter server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 8b196570-6b00-d4e2-54d5-d43ed3f56f82}" + testing.go:149: server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: 2023-05-16T04:06:12.018+1400 [INFO] server-e6bed5c7-ffc5-f402-3ea7-8d5c19ccbf65: aborting pipeline replication: peer="{Voter server-8b196570-6b00-d4e2-54d5-d43ed3f56f82 8b196570-6b00-d4e2-54d5-d43ed3f56f82}" +--- PASS: TestRaft_RemoveLeader (0.59s) === RUN TestRaft_RemoveLeader_NoShutdown -testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.640-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b89d8880-c899-eb95-6af3-fbceba7c6165 Address:b89d8880-c899-eb95-6af3-fbceba7c6165} {Suffrage:Voter ID:server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc Address:fbb6f22a-8dde-c5a7-a827-f24b878c8ebc} {Suffrage:Voter ID:server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 Address:2d93c88f-aa4f-2764-cd77-73ee9c125b42}]" - testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.640-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b89d8880-c899-eb95-6af3-fbceba7c6165 Address:b89d8880-c899-eb95-6af3-fbceba7c6165} {Suffrage:Voter ID:server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc Address:fbb6f22a-8dde-c5a7-a827-f24b878c8ebc} {Suffrage:Voter ID:server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 Address:2d93c88f-aa4f-2764-cd77-73ee9c125b42}]" -testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:31.640-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b89d8880-c899-eb95-6af3-fbceba7c6165 Address:b89d8880-c899-eb95-6af3-fbceba7c6165} {Suffrage:Voter ID:server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc Address:fbb6f22a-8dde-c5a7-a827-f24b878c8ebc} {Suffrage:Voter ID:server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 Address:2d93c88f-aa4f-2764-cd77-73ee9c125b42}]" - testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:31.640-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b89d8880-c899-eb95-6af3-fbceba7c6165 Address:b89d8880-c899-eb95-6af3-fbceba7c6165} {Suffrage:Voter ID:server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc Address:fbb6f22a-8dde-c5a7-a827-f24b878c8ebc} {Suffrage:Voter ID:server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 Address:2d93c88f-aa4f-2764-cd77-73ee9c125b42}]" -testLoggerAdapter verbose: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:31.640-1200 [INFO] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b89d8880-c899-eb95-6af3-fbceba7c6165 Address:b89d8880-c899-eb95-6af3-fbceba7c6165} {Suffrage:Voter ID:server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc Address:fbb6f22a-8dde-c5a7-a827-f24b878c8ebc} {Suffrage:Voter ID:server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 Address:2d93c88f-aa4f-2764-cd77-73ee9c125b42}]" - testing.go:149: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:31.640-1200 [INFO] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b89d8880-c899-eb95-6af3-fbceba7c6165 Address:b89d8880-c899-eb95-6af3-fbceba7c6165} {Suffrage:Voter ID:server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc Address:fbb6f22a-8dde-c5a7-a827-f24b878c8ebc} {Suffrage:Voter ID:server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 Address:2d93c88f-aa4f-2764-cd77-73ee9c125b42}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.640-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:31.640-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.641-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: entering follower state: follower="Node at b89d8880-c899-eb95-6af3-fbceba7c6165 [Follower]" leader= - testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.641-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: entering follower state: follower="Node at b89d8880-c899-eb95-6af3-fbceba7c6165 [Follower]" leader= -testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:31.641-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: entering follower state: follower="Node at fbb6f22a-8dde-c5a7-a827-f24b878c8ebc [Follower]" leader= - testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:31.641-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: entering follower state: follower="Node at fbb6f22a-8dde-c5a7-a827-f24b878c8ebc [Follower]" leader= -testLoggerAdapter verbose: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:31.641-1200 [INFO] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: entering follower state: follower="Node at 2d93c88f-aa4f-2764-cd77-73ee9c125b42 [Follower]" leader= - testing.go:149: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:31.641-1200 [INFO] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: entering follower state: follower="Node at 2d93c88f-aa4f-2764-cd77-73ee9c125b42 [Follower]" leader= -testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.709-1200 [WARN] server-b89d8880-c899-eb95-6af3-fbceba7c6165: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.709-1200 [WARN] server-b89d8880-c899-eb95-6af3-fbceba7c6165: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: entering candidate state: node="Node at b89d8880-c899-eb95-6af3-fbceba7c6165 [Candidate]" term=2 - testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: entering candidate state: node="Node at b89d8880-c899-eb95-6af3-fbceba7c6165 [Candidate]" term=2 -testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: election won: tally=2 - testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: election won: tally=2 -testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: entering leader state: leader="Node at b89d8880-c899-eb95-6af3-fbceba7c6165 [Leader]" - testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: entering leader state: leader="Node at b89d8880-c899-eb95-6af3-fbceba7c6165 [Leader]" -testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: added peer, starting replication: peer=server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc - testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: added peer, starting replication: peer=server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc -testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: added peer, starting replication: peer=server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 - testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: added peer, starting replication: peer=server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 -testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: pipelining replication: peer="{Voter server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 2d93c88f-aa4f-2764-cd77-73ee9c125b42}" - testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: pipelining replication: peer="{Voter server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 2d93c88f-aa4f-2764-cd77-73ee9c125b42}" -testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: pipelining replication: peer="{Voter server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc fbb6f22a-8dde-c5a7-a827-f24b878c8ebc}" - testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.710-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: pipelining replication: peer="{Voter server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc fbb6f22a-8dde-c5a7-a827-f24b878c8ebc}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.816-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:31.71111416 -1200 -12 m=+8.718430641 (2 nodes), 70.13074ms from start of poll, 70.98378ms from cluster start. Timeout at 2024-06-16 08:26:31.81663246 -1200 -12 m=+8.823948921, 105.51828ms after stability - testing.go:149: cluster: 2024-06-16T08:26:31.816-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:31.71111416 -1200 -12 m=+8.718430641 (2 nodes), 70.13074ms from start of poll, 70.98378ms from cluster start. Timeout at 2024-06-16 08:26:31.81663246 -1200 -12 m=+8.823948921, 105.51828ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.816-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:31.816-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:31.922-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:31.81686032 -1200 -12 m=+8.824176781 (1 nodes), 1.8µs from start of poll, 176.72992ms from cluster start. Timeout at 2024-06-16 08:26:31.92223826 -1200 -12 m=+8.929554741, 105.37796ms after stability - testing.go:149: cluster: 2024-06-16T08:26:31.922-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:31.81686032 -1200 -12 m=+8.824176781 (1 nodes), 1.8µs from start of poll, 176.72992ms from cluster start. Timeout at 2024-06-16 08:26:31.92223826 -1200 -12 m=+8.929554741, 105.37796ms after stability -testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.924-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: updating configuration: command=RemoveServer server-id=server-b89d8880-c899-eb95-6af3-fbceba7c6165 server-addr= servers="[{Suffrage:Voter ID:server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc Address:fbb6f22a-8dde-c5a7-a827-f24b878c8ebc} {Suffrage:Voter ID:server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 Address:2d93c88f-aa4f-2764-cd77-73ee9c125b42}]" - testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.924-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: updating configuration: command=RemoveServer server-id=server-b89d8880-c899-eb95-6af3-fbceba7c6165 server-addr= servers="[{Suffrage:Voter ID:server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc Address:fbb6f22a-8dde-c5a7-a827-f24b878c8ebc} {Suffrage:Voter ID:server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 Address:2d93c88f-aa4f-2764-cd77-73ee9c125b42}]" -testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.924-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: removed ourself, transitioning to follower - testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.924-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: removed ourself, transitioning to follower -testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.924-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: entering follower state: follower="Node at b89d8880-c899-eb95-6af3-fbceba7c6165 [Follower]" leader= - testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.924-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: entering follower state: follower="Node at b89d8880-c899-eb95-6af3-fbceba7c6165 [Follower]" leader= -testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.925-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: aborting pipeline replication: peer="{Voter server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 2d93c88f-aa4f-2764-cd77-73ee9c125b42}" - testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.925-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: aborting pipeline replication: peer="{Voter server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 2d93c88f-aa4f-2764-cd77-73ee9c125b42}" -testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.925-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: aborting pipeline replication: peer="{Voter server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc fbb6f22a-8dde-c5a7-a827-f24b878c8ebc}" - testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:31.925-1200 [INFO] server-b89d8880-c899-eb95-6af3-fbceba7c6165: aborting pipeline replication: peer="{Voter server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc fbb6f22a-8dde-c5a7-a827-f24b878c8ebc}" -testLoggerAdapter verbose: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:32.000-1200 [WARN] server-b89d8880-c899-eb95-6af3-fbceba7c6165: not part of stable configuration, aborting election - testing.go:149: server-b89d8880-c899-eb95-6af3-fbceba7c6165: 2024-06-16T08:26:32.000-1200 [WARN] server-b89d8880-c899-eb95-6af3-fbceba7c6165: not part of stable configuration, aborting election -testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.017-1200 [WARN] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: heartbeat timeout reached, starting election: last-leader=b89d8880-c899-eb95-6af3-fbceba7c6165 - testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.017-1200 [WARN] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: heartbeat timeout reached, starting election: last-leader=b89d8880-c899-eb95-6af3-fbceba7c6165 -testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.017-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: entering candidate state: node="Node at fbb6f22a-8dde-c5a7-a827-f24b878c8ebc [Candidate]" term=3 - testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.017-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: entering candidate state: node="Node at fbb6f22a-8dde-c5a7-a827-f24b878c8ebc [Candidate]" term=3 -testLoggerAdapter verbose: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:32.017-1200 [WARN] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: rejecting vote request since we have a leader: from=fbb6f22a-8dde-c5a7-a827-f24b878c8ebc leader=b89d8880-c899-eb95-6af3-fbceba7c6165 - testing.go:149: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:32.017-1200 [WARN] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: rejecting vote request since we have a leader: from=fbb6f22a-8dde-c5a7-a827-f24b878c8ebc leader=b89d8880-c899-eb95-6af3-fbceba7c6165 -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.031-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:32.031-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:32.033-1200 [WARN] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: heartbeat timeout reached, starting election: last-leader=b89d8880-c899-eb95-6af3-fbceba7c6165 - testing.go:149: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:32.033-1200 [WARN] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: heartbeat timeout reached, starting election: last-leader=b89d8880-c899-eb95-6af3-fbceba7c6165 -testLoggerAdapter verbose: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:32.033-1200 [INFO] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: entering candidate state: node="Node at 2d93c88f-aa4f-2764-cd77-73ee9c125b42 [Candidate]" term=3 - testing.go:149: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:32.033-1200 [INFO] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: entering candidate state: node="Node at 2d93c88f-aa4f-2764-cd77-73ee9c125b42 [Candidate]" term=3 -testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.033-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: duplicate requestVote for same term: term=3 - testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.033-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: duplicate requestVote for same term: term=3 -testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.071-1200 [WARN] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: Election timeout reached, restarting election - testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.071-1200 [WARN] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: Election timeout reached, restarting election -testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.071-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: entering candidate state: node="Node at fbb6f22a-8dde-c5a7-a827-f24b878c8ebc [Candidate]" term=4 - testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.071-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: entering candidate state: node="Node at fbb6f22a-8dde-c5a7-a827-f24b878c8ebc [Candidate]" term=4 -testLoggerAdapter verbose: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:32.072-1200 [INFO] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: entering follower state: follower="Node at 2d93c88f-aa4f-2764-cd77-73ee9c125b42 [Follower]" leader= - testing.go:149: server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: 2024-06-16T08:26:32.072-1200 [INFO] server-2d93c88f-aa4f-2764-cd77-73ee9c125b42: entering follower state: follower="Node at 2d93c88f-aa4f-2764-cd77-73ee9c125b42 [Follower]" leader= -testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.072-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: election won: tally=2 - testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.072-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: election won: tally=2 -testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.072-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: entering leader state: leader="Node at fbb6f22a-8dde-c5a7-a827-f24b878c8ebc [Leader]" - testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.072-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: entering leader state: leader="Node at fbb6f22a-8dde-c5a7-a827-f24b878c8ebc [Leader]" -testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.072-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: added peer, starting replication: peer=server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 - testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.072-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: added peer, starting replication: peer=server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 -testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.072-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: pipelining replication: peer="{Voter server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 2d93c88f-aa4f-2764-cd77-73ee9c125b42}" - testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.072-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: pipelining replication: peer="{Voter server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 2d93c88f-aa4f-2764-cd77-73ee9c125b42}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.177-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:32.07232734 -1200 -12 m=+9.079643861 (1 nodes), 40.98038ms from start of poll, 432.197ms from cluster start. Timeout at 2024-06-16 08:26:32.17767624 -1200 -12 m=+9.184992741, 105.34888ms after stability - testing.go:149: cluster: 2024-06-16T08:26:32.177-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:32.07232734 -1200 -12 m=+9.079643861 (1 nodes), 40.98038ms from start of poll, 432.197ms from cluster start. Timeout at 2024-06-16 08:26:32.17767624 -1200 -12 m=+9.184992741, 105.34888ms after stability -testLoggerAdapter verbose: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.283-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: aborting pipeline replication: peer="{Voter server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 2d93c88f-aa4f-2764-cd77-73ee9c125b42}" - testing.go:149: server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: 2024-06-16T08:26:32.283-1200 [INFO] server-fbb6f22a-8dde-c5a7-a827-f24b878c8ebc: aborting pipeline replication: peer="{Voter server-2d93c88f-aa4f-2764-cd77-73ee9c125b42 2d93c88f-aa4f-2764-cd77-73ee9c125b42}" ---- PASS: TestRaft_RemoveLeader_NoShutdown (0.64s) +testLoggerAdapter verbose: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.020+1400 [INFO] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31be2701-4a01-b534-e88d-bfd3c41c0d2a Address:31be2701-4a01-b534-e88d-bfd3c41c0d2a} {Suffrage:Voter ID:server-abb26b36-c69e-229d-3c5e-3e915fb40726 Address:abb26b36-c69e-229d-3c5e-3e915fb40726} {Suffrage:Voter ID:server-83088d1a-625d-a43f-87c7-2e81e437ca2e Address:83088d1a-625d-a43f-87c7-2e81e437ca2e}]" + testing.go:149: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.020+1400 [INFO] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31be2701-4a01-b534-e88d-bfd3c41c0d2a Address:31be2701-4a01-b534-e88d-bfd3c41c0d2a} {Suffrage:Voter ID:server-abb26b36-c69e-229d-3c5e-3e915fb40726 Address:abb26b36-c69e-229d-3c5e-3e915fb40726} {Suffrage:Voter ID:server-83088d1a-625d-a43f-87c7-2e81e437ca2e Address:83088d1a-625d-a43f-87c7-2e81e437ca2e}]" +testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.020+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31be2701-4a01-b534-e88d-bfd3c41c0d2a Address:31be2701-4a01-b534-e88d-bfd3c41c0d2a} {Suffrage:Voter ID:server-abb26b36-c69e-229d-3c5e-3e915fb40726 Address:abb26b36-c69e-229d-3c5e-3e915fb40726} {Suffrage:Voter ID:server-83088d1a-625d-a43f-87c7-2e81e437ca2e Address:83088d1a-625d-a43f-87c7-2e81e437ca2e}]" + testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.020+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31be2701-4a01-b534-e88d-bfd3c41c0d2a Address:31be2701-4a01-b534-e88d-bfd3c41c0d2a} {Suffrage:Voter ID:server-abb26b36-c69e-229d-3c5e-3e915fb40726 Address:abb26b36-c69e-229d-3c5e-3e915fb40726} {Suffrage:Voter ID:server-83088d1a-625d-a43f-87c7-2e81e437ca2e Address:83088d1a-625d-a43f-87c7-2e81e437ca2e}]" +testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.021+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31be2701-4a01-b534-e88d-bfd3c41c0d2a Address:31be2701-4a01-b534-e88d-bfd3c41c0d2a} {Suffrage:Voter ID:server-abb26b36-c69e-229d-3c5e-3e915fb40726 Address:abb26b36-c69e-229d-3c5e-3e915fb40726} {Suffrage:Voter ID:server-83088d1a-625d-a43f-87c7-2e81e437ca2e Address:83088d1a-625d-a43f-87c7-2e81e437ca2e}]" + testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.021+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31be2701-4a01-b534-e88d-bfd3c41c0d2a Address:31be2701-4a01-b534-e88d-bfd3c41c0d2a} {Suffrage:Voter ID:server-abb26b36-c69e-229d-3c5e-3e915fb40726 Address:abb26b36-c69e-229d-3c5e-3e915fb40726} {Suffrage:Voter ID:server-83088d1a-625d-a43f-87c7-2e81e437ca2e Address:83088d1a-625d-a43f-87c7-2e81e437ca2e}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.021+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:12.021+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.021+1400 [INFO] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: entering follower state: follower="Node at 31be2701-4a01-b534-e88d-bfd3c41c0d2a [Follower]" leader= + testing.go:149: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.021+1400 [INFO] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: entering follower state: follower="Node at 31be2701-4a01-b534-e88d-bfd3c41c0d2a [Follower]" leader= +testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.021+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: entering follower state: follower="Node at abb26b36-c69e-229d-3c5e-3e915fb40726 [Follower]" leader= + testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.021+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: entering follower state: follower="Node at abb26b36-c69e-229d-3c5e-3e915fb40726 [Follower]" leader= +testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.021+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: entering follower state: follower="Node at 83088d1a-625d-a43f-87c7-2e81e437ca2e [Follower]" leader= + testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.021+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: entering follower state: follower="Node at 83088d1a-625d-a43f-87c7-2e81e437ca2e [Follower]" leader= +testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [WARN] server-abb26b36-c69e-229d-3c5e-3e915fb40726: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [WARN] server-abb26b36-c69e-229d-3c5e-3e915fb40726: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: entering candidate state: node="Node at abb26b36-c69e-229d-3c5e-3e915fb40726 [Candidate]" term=2 + testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: entering candidate state: node="Node at abb26b36-c69e-229d-3c5e-3e915fb40726 [Candidate]" term=2 +testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: election won: tally=2 + testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: election won: tally=2 +testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: entering leader state: leader="Node at abb26b36-c69e-229d-3c5e-3e915fb40726 [Leader]" + testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: entering leader state: leader="Node at abb26b36-c69e-229d-3c5e-3e915fb40726 [Leader]" +testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: added peer, starting replication: peer=server-31be2701-4a01-b534-e88d-bfd3c41c0d2a + testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: added peer, starting replication: peer=server-31be2701-4a01-b534-e88d-bfd3c41c0d2a +testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: added peer, starting replication: peer=server-83088d1a-625d-a43f-87c7-2e81e437ca2e + testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: added peer, starting replication: peer=server-83088d1a-625d-a43f-87c7-2e81e437ca2e +testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: pipelining replication: peer="{Voter server-83088d1a-625d-a43f-87c7-2e81e437ca2e 83088d1a-625d-a43f-87c7-2e81e437ca2e}" + testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.082+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: pipelining replication: peer="{Voter server-83088d1a-625d-a43f-87c7-2e81e437ca2e 83088d1a-625d-a43f-87c7-2e81e437ca2e}" +testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.083+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: pipelining replication: peer="{Voter server-31be2701-4a01-b534-e88d-bfd3c41c0d2a 31be2701-4a01-b534-e88d-bfd3c41c0d2a}" + testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.083+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: pipelining replication: peer="{Voter server-31be2701-4a01-b534-e88d-bfd3c41c0d2a 31be2701-4a01-b534-e88d-bfd3c41c0d2a}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.188+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:12.083331775 +1400 +14 m=+8.885913390 (2 nodes), 62.175554ms from start of poll, 63.222106ms from cluster start. Timeout at 2023-05-16 04:06:12.188653078 +1400 +14 m=+8.991234713, 105.321323ms after stability + testing.go:149: cluster: 2023-05-16T04:06:12.188+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:12.083331775 +1400 +14 m=+8.885913390 (2 nodes), 62.175554ms from start of poll, 63.222106ms from cluster start. Timeout at 2023-05-16 04:06:12.188653078 +1400 +14 m=+8.991234713, 105.321323ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.188+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:12.188+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.294+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:12.188907501 +1400 +14 m=+8.991489116 (1 nodes), 2.52µs from start of poll, 168.797832ms from cluster start. Timeout at 2023-05-16 04:06:12.293987221 +1400 +14 m=+9.096568956, 105.07984ms after stability + testing.go:149: cluster: 2023-05-16T04:06:12.294+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:12.188907501 +1400 +14 m=+8.991489116 (1 nodes), 2.52µs from start of poll, 168.797832ms from cluster start. Timeout at 2023-05-16 04:06:12.293987221 +1400 +14 m=+9.096568956, 105.07984ms after stability +testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.299+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: updating configuration: command=RemoveServer server-id=server-abb26b36-c69e-229d-3c5e-3e915fb40726 server-addr= servers="[{Suffrage:Voter ID:server-31be2701-4a01-b534-e88d-bfd3c41c0d2a Address:31be2701-4a01-b534-e88d-bfd3c41c0d2a} {Suffrage:Voter ID:server-83088d1a-625d-a43f-87c7-2e81e437ca2e Address:83088d1a-625d-a43f-87c7-2e81e437ca2e}]" + testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.299+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: updating configuration: command=RemoveServer server-id=server-abb26b36-c69e-229d-3c5e-3e915fb40726 server-addr= servers="[{Suffrage:Voter ID:server-31be2701-4a01-b534-e88d-bfd3c41c0d2a Address:31be2701-4a01-b534-e88d-bfd3c41c0d2a} {Suffrage:Voter ID:server-83088d1a-625d-a43f-87c7-2e81e437ca2e Address:83088d1a-625d-a43f-87c7-2e81e437ca2e}]" +testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.299+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: removed ourself, transitioning to follower + testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.299+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: removed ourself, transitioning to follower +testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.299+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: entering follower state: follower="Node at abb26b36-c69e-229d-3c5e-3e915fb40726 [Follower]" leader= + testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.299+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: entering follower state: follower="Node at abb26b36-c69e-229d-3c5e-3e915fb40726 [Follower]" leader= +testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.299+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: aborting pipeline replication: peer="{Voter server-83088d1a-625d-a43f-87c7-2e81e437ca2e 83088d1a-625d-a43f-87c7-2e81e437ca2e}" + testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.299+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: aborting pipeline replication: peer="{Voter server-83088d1a-625d-a43f-87c7-2e81e437ca2e 83088d1a-625d-a43f-87c7-2e81e437ca2e}" +testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.300+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: aborting pipeline replication: peer="{Voter server-31be2701-4a01-b534-e88d-bfd3c41c0d2a 31be2701-4a01-b534-e88d-bfd3c41c0d2a}" + testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.300+1400 [INFO] server-abb26b36-c69e-229d-3c5e-3e915fb40726: aborting pipeline replication: peer="{Voter server-31be2701-4a01-b534-e88d-bfd3c41c0d2a 31be2701-4a01-b534-e88d-bfd3c41c0d2a}" +testLoggerAdapter verbose: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.385+1400 [WARN] server-abb26b36-c69e-229d-3c5e-3e915fb40726: not part of stable configuration, aborting election + testing.go:149: server-abb26b36-c69e-229d-3c5e-3e915fb40726: 2023-05-16T04:06:12.385+1400 [WARN] server-abb26b36-c69e-229d-3c5e-3e915fb40726: not part of stable configuration, aborting election +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.406+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:12.406+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.415+1400 [WARN] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: heartbeat timeout reached, starting election: last-leader=abb26b36-c69e-229d-3c5e-3e915fb40726 + testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.415+1400 [WARN] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: heartbeat timeout reached, starting election: last-leader=abb26b36-c69e-229d-3c5e-3e915fb40726 +testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.415+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: entering candidate state: node="Node at 83088d1a-625d-a43f-87c7-2e81e437ca2e [Candidate]" term=3 + testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.415+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: entering candidate state: node="Node at 83088d1a-625d-a43f-87c7-2e81e437ca2e [Candidate]" term=3 +testLoggerAdapter verbose: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.416+1400 [WARN] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: rejecting vote request since we have a leader: from=83088d1a-625d-a43f-87c7-2e81e437ca2e leader=abb26b36-c69e-229d-3c5e-3e915fb40726 + testing.go:149: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.416+1400 [WARN] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: rejecting vote request since we have a leader: from=83088d1a-625d-a43f-87c7-2e81e437ca2e leader=abb26b36-c69e-229d-3c5e-3e915fb40726 +testLoggerAdapter verbose: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.428+1400 [WARN] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: heartbeat timeout reached, starting election: last-leader=abb26b36-c69e-229d-3c5e-3e915fb40726 + testing.go:149: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.428+1400 [WARN] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: heartbeat timeout reached, starting election: last-leader=abb26b36-c69e-229d-3c5e-3e915fb40726 +testLoggerAdapter verbose: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.428+1400 [INFO] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: entering candidate state: node="Node at 31be2701-4a01-b534-e88d-bfd3c41c0d2a [Candidate]" term=3 + testing.go:149: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.428+1400 [INFO] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: entering candidate state: node="Node at 31be2701-4a01-b534-e88d-bfd3c41c0d2a [Candidate]" term=3 +testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.428+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: duplicate requestVote for same term: term=3 + testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.428+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: duplicate requestVote for same term: term=3 +testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.512+1400 [WARN] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: Election timeout reached, restarting election + testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.512+1400 [WARN] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: Election timeout reached, restarting election +testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.512+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: entering candidate state: node="Node at 83088d1a-625d-a43f-87c7-2e81e437ca2e [Candidate]" term=4 + testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.512+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: entering candidate state: node="Node at 83088d1a-625d-a43f-87c7-2e81e437ca2e [Candidate]" term=4 +testLoggerAdapter verbose: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.512+1400 [INFO] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: entering follower state: follower="Node at 31be2701-4a01-b534-e88d-bfd3c41c0d2a [Follower]" leader= + testing.go:149: server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: 2023-05-16T04:06:12.512+1400 [INFO] server-31be2701-4a01-b534-e88d-bfd3c41c0d2a: entering follower state: follower="Node at 31be2701-4a01-b534-e88d-bfd3c41c0d2a [Follower]" leader= +testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.512+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: election won: tally=2 + testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.512+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: election won: tally=2 +testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.512+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: entering leader state: leader="Node at 83088d1a-625d-a43f-87c7-2e81e437ca2e [Leader]" + testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.512+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: entering leader state: leader="Node at 83088d1a-625d-a43f-87c7-2e81e437ca2e [Leader]" +testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.513+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: added peer, starting replication: peer=server-31be2701-4a01-b534-e88d-bfd3c41c0d2a + testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.513+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: added peer, starting replication: peer=server-31be2701-4a01-b534-e88d-bfd3c41c0d2a +testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.516+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: pipelining replication: peer="{Voter server-31be2701-4a01-b534-e88d-bfd3c41c0d2a 31be2701-4a01-b534-e88d-bfd3c41c0d2a}" + testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.516+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: pipelining replication: peer="{Voter server-31be2701-4a01-b534-e88d-bfd3c41c0d2a 31be2701-4a01-b534-e88d-bfd3c41c0d2a}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.620+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:12.515814319 +1400 +14 m=+9.318395954 (1 nodes), 109.635854ms from start of poll, 495.70467ms from cluster start. Timeout at 2023-05-16 04:06:12.620902039 +1400 +14 m=+9.423483694, 105.08774ms after stability + testing.go:149: cluster: 2023-05-16T04:06:12.620+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:12.515814319 +1400 +14 m=+9.318395954 (1 nodes), 109.635854ms from start of poll, 495.70467ms from cluster start. Timeout at 2023-05-16 04:06:12.620902039 +1400 +14 m=+9.423483694, 105.08774ms after stability +testLoggerAdapter verbose: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.726+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: aborting pipeline replication: peer="{Voter server-31be2701-4a01-b534-e88d-bfd3c41c0d2a 31be2701-4a01-b534-e88d-bfd3c41c0d2a}" + testing.go:149: server-83088d1a-625d-a43f-87c7-2e81e437ca2e: 2023-05-16T04:06:12.726+1400 [INFO] server-83088d1a-625d-a43f-87c7-2e81e437ca2e: aborting pipeline replication: peer="{Voter server-31be2701-4a01-b534-e88d-bfd3c41c0d2a 31be2701-4a01-b534-e88d-bfd3c41c0d2a}" +--- PASS: TestRaft_RemoveLeader_NoShutdown (0.71s) === RUN TestRaft_RemoveFollower_SplitCluster -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.284-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d} {Suffrage:Voter ID:server-fc14acdc-48c9-ff27-6746-526d724f67ff Address:fc14acdc-48c9-ff27-6746-526d724f67ff}]" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.284-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d} {Suffrage:Voter ID:server-fc14acdc-48c9-ff27-6746-526d724f67ff Address:fc14acdc-48c9-ff27-6746-526d724f67ff}]" -testLoggerAdapter verbose: server-fddfb24d-a02d-e4da-534f-901802878db2: 2024-06-16T08:26:32.284-1200 [INFO] server-fddfb24d-a02d-e4da-534f-901802878db2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d} {Suffrage:Voter ID:server-fc14acdc-48c9-ff27-6746-526d724f67ff Address:fc14acdc-48c9-ff27-6746-526d724f67ff}]" - testing.go:149: server-fddfb24d-a02d-e4da-534f-901802878db2: 2024-06-16T08:26:32.284-1200 [INFO] server-fddfb24d-a02d-e4da-534f-901802878db2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d} {Suffrage:Voter ID:server-fc14acdc-48c9-ff27-6746-526d724f67ff Address:fc14acdc-48c9-ff27-6746-526d724f67ff}]" -testLoggerAdapter verbose: server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d: 2024-06-16T08:26:32.285-1200 [INFO] server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d} {Suffrage:Voter ID:server-fc14acdc-48c9-ff27-6746-526d724f67ff Address:fc14acdc-48c9-ff27-6746-526d724f67ff}]" - testing.go:149: server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d: 2024-06-16T08:26:32.285-1200 [INFO] server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d} {Suffrage:Voter ID:server-fc14acdc-48c9-ff27-6746-526d724f67ff Address:fc14acdc-48c9-ff27-6746-526d724f67ff}]" -testLoggerAdapter verbose: server-fc14acdc-48c9-ff27-6746-526d724f67ff: 2024-06-16T08:26:32.285-1200 [INFO] server-fc14acdc-48c9-ff27-6746-526d724f67ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d} {Suffrage:Voter ID:server-fc14acdc-48c9-ff27-6746-526d724f67ff Address:fc14acdc-48c9-ff27-6746-526d724f67ff}]" - testing.go:149: server-fc14acdc-48c9-ff27-6746-526d724f67ff: 2024-06-16T08:26:32.285-1200 [INFO] server-fc14acdc-48c9-ff27-6746-526d724f67ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d} {Suffrage:Voter ID:server-fc14acdc-48c9-ff27-6746-526d724f67ff Address:fc14acdc-48c9-ff27-6746-526d724f67ff}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.285-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:32.285-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.285-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: entering follower state: follower="Node at ace08458-2bc3-f370-6676-ce732b04d814 [Follower]" leader= - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.285-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: entering follower state: follower="Node at ace08458-2bc3-f370-6676-ce732b04d814 [Follower]" leader= -testLoggerAdapter verbose: server-fddfb24d-a02d-e4da-534f-901802878db2: 2024-06-16T08:26:32.285-1200 [INFO] server-fddfb24d-a02d-e4da-534f-901802878db2: entering follower state: follower="Node at fddfb24d-a02d-e4da-534f-901802878db2 [Follower]" leader= - testing.go:149: server-fddfb24d-a02d-e4da-534f-901802878db2: 2024-06-16T08:26:32.285-1200 [INFO] server-fddfb24d-a02d-e4da-534f-901802878db2: entering follower state: follower="Node at fddfb24d-a02d-e4da-534f-901802878db2 [Follower]" leader= -testLoggerAdapter verbose: server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d: 2024-06-16T08:26:32.285-1200 [INFO] server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d: entering follower state: follower="Node at bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d [Follower]" leader= - testing.go:149: server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d: 2024-06-16T08:26:32.285-1200 [INFO] server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d: entering follower state: follower="Node at bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d [Follower]" leader= -testLoggerAdapter verbose: server-fc14acdc-48c9-ff27-6746-526d724f67ff: 2024-06-16T08:26:32.285-1200 [INFO] server-fc14acdc-48c9-ff27-6746-526d724f67ff: entering follower state: follower="Node at fc14acdc-48c9-ff27-6746-526d724f67ff [Follower]" leader= - testing.go:149: server-fc14acdc-48c9-ff27-6746-526d724f67ff: 2024-06-16T08:26:32.285-1200 [INFO] server-fc14acdc-48c9-ff27-6746-526d724f67ff: entering follower state: follower="Node at fc14acdc-48c9-ff27-6746-526d724f67ff [Follower]" leader= -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: entering candidate state: node="Node at ace08458-2bc3-f370-6676-ce732b04d814 [Candidate]" term=2 - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: entering candidate state: node="Node at ace08458-2bc3-f370-6676-ce732b04d814 [Candidate]" term=2 -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: election won: tally=3 - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: election won: tally=3 -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: entering leader state: leader="Node at ace08458-2bc3-f370-6676-ce732b04d814 [Leader]" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: entering leader state: leader="Node at ace08458-2bc3-f370-6676-ce732b04d814 [Leader]" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: added peer, starting replication: peer=server-fddfb24d-a02d-e4da-534f-901802878db2 - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: added peer, starting replication: peer=server-fddfb24d-a02d-e4da-534f-901802878db2 -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: added peer, starting replication: peer=server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.338-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: added peer, starting replication: peer=server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.339-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: added peer, starting replication: peer=server-fc14acdc-48c9-ff27-6746-526d724f67ff - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.339-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: added peer, starting replication: peer=server-fc14acdc-48c9-ff27-6746-526d724f67ff -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.339-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: pipelining replication: peer="{Voter server-fc14acdc-48c9-ff27-6746-526d724f67ff fc14acdc-48c9-ff27-6746-526d724f67ff}" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.339-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: pipelining replication: peer="{Voter server-fc14acdc-48c9-ff27-6746-526d724f67ff fc14acdc-48c9-ff27-6746-526d724f67ff}" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.339-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: pipelining replication: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.339-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: pipelining replication: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.339-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: pipelining replication: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.339-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: pipelining replication: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.444-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:32.33934194 -1200 -12 m=+9.346658401 (1 nodes), 53.80772ms from start of poll, 54.95956ms from cluster start. Timeout at 2024-06-16 08:26:32.44483336 -1200 -12 m=+9.452149841, 105.49144ms after stability - testing.go:149: cluster: 2024-06-16T08:26:32.444-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:32.33934194 -1200 -12 m=+9.346658401 (1 nodes), 53.80772ms from start of poll, 54.95956ms from cluster start. Timeout at 2024-06-16 08:26:32.44483336 -1200 -12 m=+9.452149841, 105.49144ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.550-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:32.550-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.656-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:32.55073952 -1200 -12 m=+9.558056521 (3 nodes), 3.66µs from start of poll, 266.35768ms from cluster start. Timeout at 2024-06-16 08:26:32.65630494 -1200 -12 m=+9.663621421, 105.5649ms after stability - testing.go:149: cluster: 2024-06-16T08:26:32.656-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:32.55073952 -1200 -12 m=+9.558056521 (3 nodes), 3.66µs from start of poll, 266.35768ms from cluster start. Timeout at 2024-06-16 08:26:32.65630494 -1200 -12 m=+9.663621421, 105.5649ms after stability -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: updating configuration: command=RemoveServer server-id=server-fc14acdc-48c9-ff27-6746-526d724f67ff server-addr= servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}]" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: updating configuration: command=RemoveServer server-id=server-fc14acdc-48c9-ff27-6746-526d724f67ff server-addr= servers="[{Suffrage:Voter ID:server-ace08458-2bc3-f370-6676-ce732b04d814 Address:ace08458-2bc3-f370-6676-ce732b04d814} {Suffrage:Voter ID:server-fddfb24d-a02d-e4da-534f-901802878db2 Address:fddfb24d-a02d-e4da-534f-901802878db2} {Suffrage:Voter ID:server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d Address:bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}]" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to pipeline appendEntries: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="append pipeline closed" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to pipeline appendEntries: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="append pipeline closed" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: removed peer, stopping replication: peer=server-fc14acdc-48c9-ff27-6746-526d724f67ff last-index=3 - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: removed peer, stopping replication: peer=server-fc14acdc-48c9-ff27-6746-526d724f67ff last-index=3 -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to pipeline appendEntries: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="append pipeline closed" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to pipeline appendEntries: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="append pipeline closed" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.657-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: aborting pipeline replication: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.657-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: aborting pipeline replication: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: aborting pipeline replication: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.656-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: aborting pipeline replication: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.657-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.657-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.657-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: aborting pipeline replication: peer="{Voter server-fc14acdc-48c9-ff27-6746-526d724f67ff fc14acdc-48c9-ff27-6746-526d724f67ff}" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.657-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: aborting pipeline replication: peer="{Voter server-fc14acdc-48c9-ff27-6746-526d724f67ff fc14acdc-48c9-ff27-6746-526d724f67ff}" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.663-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=fddfb24d-a02d-e4da-534f-901802878db2 error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.663-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=fddfb24d-a02d-e4da-534f-901802878db2 error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.663-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.663-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.667-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.667-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.676-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.676-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.681-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.681-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.683-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=fddfb24d-a02d-e4da-534f-901802878db2 error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.683-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=fddfb24d-a02d-e4da-534f-901802878db2 error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.686-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.686-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.695-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.695-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.699-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.699-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.701-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=fddfb24d-a02d-e4da-534f-901802878db2 error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.701-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to heartbeat to: peer=fddfb24d-a02d-e4da-534f-901802878db2 error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.704-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.704-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.705-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to contact: server-id=server-fddfb24d-a02d-e4da-534f-901802878db2 time=50.5593ms - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.705-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to contact: server-id=server-fddfb24d-a02d-e4da-534f-901802878db2 time=50.5593ms -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to contact: server-id=server-fddfb24d-a02d-e4da-534f-901802878db2 time=61.05256ms - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to contact: server-id=server-fddfb24d-a02d-e4da-534f-901802878db2 time=61.05256ms -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to contact: server-id=server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d time=59.97694ms - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to contact: server-id=server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d time=59.97694ms -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to contact quorum of nodes, stepping down - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [WARN] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to contact quorum of nodes, stepping down -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: entering follower state: follower="Node at ace08458-2bc3-f370-6676-ce732b04d814 [Follower]" leader= - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [INFO] server-ace08458-2bc3-f370-6676-ce732b04d814: entering follower state: follower="Node at ace08458-2bc3-f370-6676-ce732b04d814 [Follower]" leader= -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d}" error="failed to connect to peer: bcccaf6d-2e0e-ba4a-70b9-1d64b5a7779d" -testLoggerAdapter verbose: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" - testing.go:149: server-ace08458-2bc3-f370-6676-ce732b04d814: 2024-06-16T08:26:32.716-1200 [ERROR] server-ace08458-2bc3-f370-6676-ce732b04d814: failed to appendEntries to: peer="{Voter server-fddfb24d-a02d-e4da-534f-901802878db2 fddfb24d-a02d-e4da-534f-901802878db2}" error="failed to connect to peer: fddfb24d-a02d-e4da-534f-901802878db2" ---- PASS: TestRaft_RemoveFollower_SplitCluster (0.43s) +testLoggerAdapter verbose: server-baa415f7-9446-b5be-a195-a2bf823f3d5d: 2023-05-16T04:06:12.728+1400 [INFO] server-baa415f7-9446-b5be-a195-a2bf823f3d5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379} {Suffrage:Voter ID:server-7048faf4-9562-b685-9784-5d59b6d871f9 Address:7048faf4-9562-b685-9784-5d59b6d871f9}]" + testing.go:149: server-baa415f7-9446-b5be-a195-a2bf823f3d5d: 2023-05-16T04:06:12.728+1400 [INFO] server-baa415f7-9446-b5be-a195-a2bf823f3d5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379} {Suffrage:Voter ID:server-7048faf4-9562-b685-9784-5d59b6d871f9 Address:7048faf4-9562-b685-9784-5d59b6d871f9}]" +testLoggerAdapter verbose: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.728+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379} {Suffrage:Voter ID:server-7048faf4-9562-b685-9784-5d59b6d871f9 Address:7048faf4-9562-b685-9784-5d59b6d871f9}]" + testing.go:149: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.728+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379} {Suffrage:Voter ID:server-7048faf4-9562-b685-9784-5d59b6d871f9 Address:7048faf4-9562-b685-9784-5d59b6d871f9}]" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.728+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379} {Suffrage:Voter ID:server-7048faf4-9562-b685-9784-5d59b6d871f9 Address:7048faf4-9562-b685-9784-5d59b6d871f9}]" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.728+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379} {Suffrage:Voter ID:server-7048faf4-9562-b685-9784-5d59b6d871f9 Address:7048faf4-9562-b685-9784-5d59b6d871f9}]" +testLoggerAdapter verbose: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:12.728+1400 [INFO] server-7048faf4-9562-b685-9784-5d59b6d871f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379} {Suffrage:Voter ID:server-7048faf4-9562-b685-9784-5d59b6d871f9 Address:7048faf4-9562-b685-9784-5d59b6d871f9}]" + testing.go:149: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:12.728+1400 [INFO] server-7048faf4-9562-b685-9784-5d59b6d871f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379} {Suffrage:Voter ID:server-7048faf4-9562-b685-9784-5d59b6d871f9 Address:7048faf4-9562-b685-9784-5d59b6d871f9}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.729+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:12.729+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-baa415f7-9446-b5be-a195-a2bf823f3d5d: 2023-05-16T04:06:12.729+1400 [INFO] server-baa415f7-9446-b5be-a195-a2bf823f3d5d: entering follower state: follower="Node at baa415f7-9446-b5be-a195-a2bf823f3d5d [Follower]" leader= + testing.go:149: server-baa415f7-9446-b5be-a195-a2bf823f3d5d: 2023-05-16T04:06:12.729+1400 [INFO] server-baa415f7-9446-b5be-a195-a2bf823f3d5d: entering follower state: follower="Node at baa415f7-9446-b5be-a195-a2bf823f3d5d [Follower]" leader= +testLoggerAdapter verbose: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.729+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: entering follower state: follower="Node at f523c432-5146-60d1-40a1-c11499200c01 [Follower]" leader= + testing.go:149: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.729+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: entering follower state: follower="Node at f523c432-5146-60d1-40a1-c11499200c01 [Follower]" leader= +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.729+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: entering follower state: follower="Node at fee8c395-7056-a61e-b56b-dbd61b51f379 [Follower]" leader= + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.729+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: entering follower state: follower="Node at fee8c395-7056-a61e-b56b-dbd61b51f379 [Follower]" leader= +testLoggerAdapter verbose: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:12.729+1400 [INFO] server-7048faf4-9562-b685-9784-5d59b6d871f9: entering follower state: follower="Node at 7048faf4-9562-b685-9784-5d59b6d871f9 [Follower]" leader= + testing.go:149: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:12.729+1400 [INFO] server-7048faf4-9562-b685-9784-5d59b6d871f9: entering follower state: follower="Node at 7048faf4-9562-b685-9784-5d59b6d871f9 [Follower]" leader= +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.792+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.792+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.792+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: entering candidate state: node="Node at fee8c395-7056-a61e-b56b-dbd61b51f379 [Candidate]" term=2 + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.792+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: entering candidate state: node="Node at fee8c395-7056-a61e-b56b-dbd61b51f379 [Candidate]" term=2 +testLoggerAdapter verbose: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.793+1400 [WARN] server-f523c432-5146-60d1-40a1-c11499200c01: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.793+1400 [WARN] server-f523c432-5146-60d1-40a1-c11499200c01: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.793+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: entering candidate state: node="Node at f523c432-5146-60d1-40a1-c11499200c01 [Candidate]" term=2 + testing.go:149: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.793+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: entering candidate state: node="Node at f523c432-5146-60d1-40a1-c11499200c01 [Candidate]" term=2 +testLoggerAdapter verbose: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:12.793+1400 [INFO] server-7048faf4-9562-b685-9784-5d59b6d871f9: duplicate requestVote for same term: term=2 + testing.go:149: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:12.793+1400 [INFO] server-7048faf4-9562-b685-9784-5d59b6d871f9: duplicate requestVote for same term: term=2 +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: election won: tally=3 + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: election won: tally=3 +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: entering leader state: leader="Node at fee8c395-7056-a61e-b56b-dbd61b51f379 [Leader]" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: entering leader state: leader="Node at fee8c395-7056-a61e-b56b-dbd61b51f379 [Leader]" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: added peer, starting replication: peer=server-baa415f7-9446-b5be-a195-a2bf823f3d5d + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: added peer, starting replication: peer=server-baa415f7-9446-b5be-a195-a2bf823f3d5d +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: added peer, starting replication: peer=server-f523c432-5146-60d1-40a1-c11499200c01 + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: added peer, starting replication: peer=server-f523c432-5146-60d1-40a1-c11499200c01 +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: added peer, starting replication: peer=server-7048faf4-9562-b685-9784-5d59b6d871f9 + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: added peer, starting replication: peer=server-7048faf4-9562-b685-9784-5d59b6d871f9 +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: pipelining replication: peer="{Voter server-7048faf4-9562-b685-9784-5d59b6d871f9 7048faf4-9562-b685-9784-5d59b6d871f9}" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.793+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: pipelining replication: peer="{Voter server-7048faf4-9562-b685-9784-5d59b6d871f9 7048faf4-9562-b685-9784-5d59b6d871f9}" +testLoggerAdapter verbose: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.794+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: duplicate requestVote for same term: term=2 + testing.go:149: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.794+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: duplicate requestVote for same term: term=2 +testLoggerAdapter verbose: server-baa415f7-9446-b5be-a195-a2bf823f3d5d: 2023-05-16T04:06:12.794+1400 [INFO] server-baa415f7-9446-b5be-a195-a2bf823f3d5d: duplicate requestVote for same term: term=2 + testing.go:149: server-baa415f7-9446-b5be-a195-a2bf823f3d5d: 2023-05-16T04:06:12.794+1400 [INFO] server-baa415f7-9446-b5be-a195-a2bf823f3d5d: duplicate requestVote for same term: term=2 +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.794+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: rejecting vote request since we have a leader: from=f523c432-5146-60d1-40a1-c11499200c01 leader=fee8c395-7056-a61e-b56b-dbd61b51f379 + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.794+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: rejecting vote request since we have a leader: from=f523c432-5146-60d1-40a1-c11499200c01 leader=fee8c395-7056-a61e-b56b-dbd61b51f379 +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.794+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: pipelining replication: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.794+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: pipelining replication: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" +testLoggerAdapter verbose: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.794+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: entering follower state: follower="Node at f523c432-5146-60d1-40a1-c11499200c01 [Follower]" leader=fee8c395-7056-a61e-b56b-dbd61b51f379 + testing.go:149: server-f523c432-5146-60d1-40a1-c11499200c01: 2023-05-16T04:06:12.794+1400 [INFO] server-f523c432-5146-60d1-40a1-c11499200c01: entering follower state: follower="Node at f523c432-5146-60d1-40a1-c11499200c01 [Follower]" leader=fee8c395-7056-a61e-b56b-dbd61b51f379 +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.794+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: pipelining replication: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:12.794+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: pipelining replication: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:12.900+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:12.79474087 +1400 +14 m=+9.597322505 (1 nodes), 65.648874ms from start of poll, 66.904069ms from cluster start. Timeout at 2023-05-16 04:06:12.900398857 +1400 +14 m=+9.702980492, 105.657987ms after stability + testing.go:149: cluster: 2023-05-16T04:06:12.900+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:12.79474087 +1400 +14 m=+9.597322505 (1 nodes), 65.648874ms from start of poll, 66.904069ms from cluster start. Timeout at 2023-05-16 04:06:12.900398857 +1400 +14 m=+9.702980492, 105.657987ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.006+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:13.006+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.112+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:13.006665751 +1400 +14 m=+9.809247386 (3 nodes), 3.62µs from start of poll, 278.82895ms from cluster start. Timeout at 2023-05-16 04:06:13.112160996 +1400 +14 m=+9.914742611, 105.495225ms after stability + testing.go:149: cluster: 2023-05-16T04:06:13.112+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:13.006665751 +1400 +14 m=+9.809247386 (3 nodes), 3.62µs from start of poll, 278.82895ms from cluster start. Timeout at 2023-05-16 04:06:13.112160996 +1400 +14 m=+9.914742611, 105.495225ms after stability +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: updating configuration: command=RemoveServer server-id=server-7048faf4-9562-b685-9784-5d59b6d871f9 server-addr= servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379}]" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: updating configuration: command=RemoveServer server-id=server-7048faf4-9562-b685-9784-5d59b6d871f9 server-addr= servers="[{Suffrage:Voter ID:server-baa415f7-9446-b5be-a195-a2bf823f3d5d Address:baa415f7-9446-b5be-a195-a2bf823f3d5d} {Suffrage:Voter ID:server-f523c432-5146-60d1-40a1-c11499200c01 Address:f523c432-5146-60d1-40a1-c11499200c01} {Suffrage:Voter ID:server-fee8c395-7056-a61e-b56b-dbd61b51f379 Address:fee8c395-7056-a61e-b56b-dbd61b51f379}]" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: removed peer, stopping replication: peer=server-7048faf4-9562-b685-9784-5d59b6d871f9 last-index=3 + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: removed peer, stopping replication: peer=server-7048faf4-9562-b685-9784-5d59b6d871f9 last-index=3 +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: aborting pipeline replication: peer="{Voter server-7048faf4-9562-b685-9784-5d59b6d871f9 7048faf4-9562-b685-9784-5d59b6d871f9}" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: aborting pipeline replication: peer="{Voter server-7048faf4-9562-b685-9784-5d59b6d871f9 7048faf4-9562-b685-9784-5d59b6d871f9}" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to pipeline appendEntries: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="append pipeline closed" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to pipeline appendEntries: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="append pipeline closed" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: aborting pipeline replication: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: aborting pipeline replication: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to pipeline appendEntries: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="append pipeline closed" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.112+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to pipeline appendEntries: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="append pipeline closed" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.113+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: aborting pipeline replication: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.113+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: aborting pipeline replication: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.115+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=f523c432-5146-60d1-40a1-c11499200c01 error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.115+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=f523c432-5146-60d1-40a1-c11499200c01 error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.115+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=baa415f7-9446-b5be-a195-a2bf823f3d5d error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.115+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=baa415f7-9446-b5be-a195-a2bf823f3d5d error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.121+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.121+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.123+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.123+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.133+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=baa415f7-9446-b5be-a195-a2bf823f3d5d error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.133+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=baa415f7-9446-b5be-a195-a2bf823f3d5d error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.135+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=f523c432-5146-60d1-40a1-c11499200c01 error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.135+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=f523c432-5146-60d1-40a1-c11499200c01 error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.139+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.139+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.142+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.142+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.150+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=baa415f7-9446-b5be-a195-a2bf823f3d5d error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.150+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=baa415f7-9446-b5be-a195-a2bf823f3d5d error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.152+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=f523c432-5146-60d1-40a1-c11499200c01 error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.152+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to heartbeat to: peer=f523c432-5146-60d1-40a1-c11499200c01 error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.159+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.159+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.160+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to contact: server-id=server-baa415f7-9446-b5be-a195-a2bf823f3d5d time=50.771519ms + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.160+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to contact: server-id=server-baa415f7-9446-b5be-a195-a2bf823f3d5d time=50.771519ms +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.160+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.160+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" +testLoggerAdapter verbose: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:13.166+1400 [WARN] server-7048faf4-9562-b685-9784-5d59b6d871f9: heartbeat timeout reached, starting election: last-leader=fee8c395-7056-a61e-b56b-dbd61b51f379 + testing.go:149: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:13.166+1400 [WARN] server-7048faf4-9562-b685-9784-5d59b6d871f9: heartbeat timeout reached, starting election: last-leader=fee8c395-7056-a61e-b56b-dbd61b51f379 +testLoggerAdapter verbose: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:13.166+1400 [INFO] server-7048faf4-9562-b685-9784-5d59b6d871f9: entering candidate state: node="Node at 7048faf4-9562-b685-9784-5d59b6d871f9 [Candidate]" term=3 + testing.go:149: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:13.166+1400 [INFO] server-7048faf4-9562-b685-9784-5d59b6d871f9: entering candidate state: node="Node at 7048faf4-9562-b685-9784-5d59b6d871f9 [Candidate]" term=3 +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.166+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: rejecting vote request since we have a leader: from=7048faf4-9562-b685-9784-5d59b6d871f9 leader=fee8c395-7056-a61e-b56b-dbd61b51f379 + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.166+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: rejecting vote request since we have a leader: from=7048faf4-9562-b685-9784-5d59b6d871f9 leader=fee8c395-7056-a61e-b56b-dbd61b51f379 +testLoggerAdapter verbose: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:13.166+1400 [ERROR] server-7048faf4-9562-b685-9784-5d59b6d871f9: failed to make requestVote RPC: target="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" + testing.go:149: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:13.166+1400 [ERROR] server-7048faf4-9562-b685-9784-5d59b6d871f9: failed to make requestVote RPC: target="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" +testLoggerAdapter verbose: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:13.166+1400 [ERROR] server-7048faf4-9562-b685-9784-5d59b6d871f9: failed to make requestVote RPC: target="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" + testing.go:149: server-7048faf4-9562-b685-9784-5d59b6d871f9: 2023-05-16T04:06:13.166+1400 [ERROR] server-7048faf4-9562-b685-9784-5d59b6d871f9: failed to make requestVote RPC: target="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to contact: server-id=server-baa415f7-9446-b5be-a195-a2bf823f3d5d time=61.777109ms + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to contact: server-id=server-baa415f7-9446-b5be-a195-a2bf823f3d5d time=61.777109ms +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to contact: server-id=server-f523c432-5146-60d1-40a1-c11499200c01 time=59.391621ms + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to contact: server-id=server-f523c432-5146-60d1-40a1-c11499200c01 time=59.391621ms +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to contact quorum of nodes, stepping down + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [WARN] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to contact quorum of nodes, stepping down +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: entering follower state: follower="Node at fee8c395-7056-a61e-b56b-dbd61b51f379 [Follower]" leader= + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [INFO] server-fee8c395-7056-a61e-b56b-dbd61b51f379: entering follower state: follower="Node at fee8c395-7056-a61e-b56b-dbd61b51f379 [Follower]" leader= +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.171+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-baa415f7-9446-b5be-a195-a2bf823f3d5d baa415f7-9446-b5be-a195-a2bf823f3d5d}" error="failed to connect to peer: baa415f7-9446-b5be-a195-a2bf823f3d5d" +testLoggerAdapter verbose: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.172+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" + testing.go:149: server-fee8c395-7056-a61e-b56b-dbd61b51f379: 2023-05-16T04:06:13.172+1400 [ERROR] server-fee8c395-7056-a61e-b56b-dbd61b51f379: failed to appendEntries to: peer="{Voter server-f523c432-5146-60d1-40a1-c11499200c01 f523c432-5146-60d1-40a1-c11499200c01}" error="failed to connect to peer: f523c432-5146-60d1-40a1-c11499200c01" +--- PASS: TestRaft_RemoveFollower_SplitCluster (0.45s) === RUN TestRaft_AddKnownPeer -testLoggerAdapter verbose: server-af538914-1821-42ed-618e-9b1ec3c36966: 2024-06-16T08:26:32.718-1200 [INFO] server-af538914-1821-42ed-618e-9b1ec3c36966: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af538914-1821-42ed-618e-9b1ec3c36966 Address:af538914-1821-42ed-618e-9b1ec3c36966} {Suffrage:Voter ID:server-6dd18996-5082-c331-dddb-6c455a171679 Address:6dd18996-5082-c331-dddb-6c455a171679} {Suffrage:Voter ID:server-4270c846-57e6-1c86-bab8-bd630057d2be Address:4270c846-57e6-1c86-bab8-bd630057d2be}]" - testing.go:149: server-af538914-1821-42ed-618e-9b1ec3c36966: 2024-06-16T08:26:32.718-1200 [INFO] server-af538914-1821-42ed-618e-9b1ec3c36966: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af538914-1821-42ed-618e-9b1ec3c36966 Address:af538914-1821-42ed-618e-9b1ec3c36966} {Suffrage:Voter ID:server-6dd18996-5082-c331-dddb-6c455a171679 Address:6dd18996-5082-c331-dddb-6c455a171679} {Suffrage:Voter ID:server-4270c846-57e6-1c86-bab8-bd630057d2be Address:4270c846-57e6-1c86-bab8-bd630057d2be}]" -testLoggerAdapter verbose: server-af538914-1821-42ed-618e-9b1ec3c36966: 2024-06-16T08:26:32.718-1200 [INFO] server-af538914-1821-42ed-618e-9b1ec3c36966: entering follower state: follower="Node at af538914-1821-42ed-618e-9b1ec3c36966 [Follower]" leader= - testing.go:149: server-af538914-1821-42ed-618e-9b1ec3c36966: 2024-06-16T08:26:32.718-1200 [INFO] server-af538914-1821-42ed-618e-9b1ec3c36966: entering follower state: follower="Node at af538914-1821-42ed-618e-9b1ec3c36966 [Follower]" leader= -testLoggerAdapter verbose: server-6dd18996-5082-c331-dddb-6c455a171679: 2024-06-16T08:26:32.718-1200 [INFO] server-6dd18996-5082-c331-dddb-6c455a171679: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af538914-1821-42ed-618e-9b1ec3c36966 Address:af538914-1821-42ed-618e-9b1ec3c36966} {Suffrage:Voter ID:server-6dd18996-5082-c331-dddb-6c455a171679 Address:6dd18996-5082-c331-dddb-6c455a171679} {Suffrage:Voter ID:server-4270c846-57e6-1c86-bab8-bd630057d2be Address:4270c846-57e6-1c86-bab8-bd630057d2be}]" - testing.go:149: server-6dd18996-5082-c331-dddb-6c455a171679: 2024-06-16T08:26:32.718-1200 [INFO] server-6dd18996-5082-c331-dddb-6c455a171679: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af538914-1821-42ed-618e-9b1ec3c36966 Address:af538914-1821-42ed-618e-9b1ec3c36966} {Suffrage:Voter ID:server-6dd18996-5082-c331-dddb-6c455a171679 Address:6dd18996-5082-c331-dddb-6c455a171679} {Suffrage:Voter ID:server-4270c846-57e6-1c86-bab8-bd630057d2be Address:4270c846-57e6-1c86-bab8-bd630057d2be}]" -testLoggerAdapter verbose: server-6dd18996-5082-c331-dddb-6c455a171679: 2024-06-16T08:26:32.718-1200 [INFO] server-6dd18996-5082-c331-dddb-6c455a171679: entering follower state: follower="Node at 6dd18996-5082-c331-dddb-6c455a171679 [Follower]" leader= - testing.go:149: server-6dd18996-5082-c331-dddb-6c455a171679: 2024-06-16T08:26:32.718-1200 [INFO] server-6dd18996-5082-c331-dddb-6c455a171679: entering follower state: follower="Node at 6dd18996-5082-c331-dddb-6c455a171679 [Follower]" leader= -testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.718-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af538914-1821-42ed-618e-9b1ec3c36966 Address:af538914-1821-42ed-618e-9b1ec3c36966} {Suffrage:Voter ID:server-6dd18996-5082-c331-dddb-6c455a171679 Address:6dd18996-5082-c331-dddb-6c455a171679} {Suffrage:Voter ID:server-4270c846-57e6-1c86-bab8-bd630057d2be Address:4270c846-57e6-1c86-bab8-bd630057d2be}]" - testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.718-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af538914-1821-42ed-618e-9b1ec3c36966 Address:af538914-1821-42ed-618e-9b1ec3c36966} {Suffrage:Voter ID:server-6dd18996-5082-c331-dddb-6c455a171679 Address:6dd18996-5082-c331-dddb-6c455a171679} {Suffrage:Voter ID:server-4270c846-57e6-1c86-bab8-bd630057d2be Address:4270c846-57e6-1c86-bab8-bd630057d2be}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.718-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:32.718-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.718-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: entering follower state: follower="Node at 4270c846-57e6-1c86-bab8-bd630057d2be [Follower]" leader= - testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.718-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: entering follower state: follower="Node at 4270c846-57e6-1c86-bab8-bd630057d2be [Follower]" leader= -testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.775-1200 [WARN] server-4270c846-57e6-1c86-bab8-bd630057d2be: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.775-1200 [WARN] server-4270c846-57e6-1c86-bab8-bd630057d2be: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.775-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: entering candidate state: node="Node at 4270c846-57e6-1c86-bab8-bd630057d2be [Candidate]" term=2 - testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.775-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: entering candidate state: node="Node at 4270c846-57e6-1c86-bab8-bd630057d2be [Candidate]" term=2 -testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: election won: tally=2 - testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: election won: tally=2 -testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: entering leader state: leader="Node at 4270c846-57e6-1c86-bab8-bd630057d2be [Leader]" - testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: entering leader state: leader="Node at 4270c846-57e6-1c86-bab8-bd630057d2be [Leader]" -testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: added peer, starting replication: peer=server-af538914-1821-42ed-618e-9b1ec3c36966 - testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: added peer, starting replication: peer=server-af538914-1821-42ed-618e-9b1ec3c36966 -testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: added peer, starting replication: peer=server-6dd18996-5082-c331-dddb-6c455a171679 - testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: added peer, starting replication: peer=server-6dd18996-5082-c331-dddb-6c455a171679 -testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: pipelining replication: peer="{Voter server-af538914-1821-42ed-618e-9b1ec3c36966 af538914-1821-42ed-618e-9b1ec3c36966}" - testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: pipelining replication: peer="{Voter server-af538914-1821-42ed-618e-9b1ec3c36966 af538914-1821-42ed-618e-9b1ec3c36966}" -testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: pipelining replication: peer="{Voter server-6dd18996-5082-c331-dddb-6c455a171679 6dd18996-5082-c331-dddb-6c455a171679}" - testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.776-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: pipelining replication: peer="{Voter server-6dd18996-5082-c331-dddb-6c455a171679 6dd18996-5082-c331-dddb-6c455a171679}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.882-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:32.77633578 -1200 -12 m=+9.783652261 (1 nodes), 57.44554ms from start of poll, 58.36796ms from cluster start. Timeout at 2024-06-16 08:26:32.8820105 -1200 -12 m=+9.889327121, 105.67486ms after stability - testing.go:149: cluster: 2024-06-16T08:26:32.882-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:32.77633578 -1200 -12 m=+9.783652261 (1 nodes), 57.44554ms from start of poll, 58.36796ms from cluster start. Timeout at 2024-06-16 08:26:32.8820105 -1200 -12 m=+9.889327121, 105.67486ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.882-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:32.882-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.987-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:32.88223498 -1200 -12 m=+9.889551461 (2 nodes), 4.34µs from start of poll, 164.26716ms from cluster start. Timeout at 2024-06-16 08:26:32.98762006 -1200 -12 m=+9.994936541, 105.38508ms after stability - testing.go:149: cluster: 2024-06-16T08:26:32.987-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:32.88223498 -1200 -12 m=+9.889551461 (2 nodes), 4.34µs from start of poll, 164.26716ms from cluster start. Timeout at 2024-06-16 08:26:32.98762006 -1200 -12 m=+9.994936541, 105.38508ms after stability -testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.987-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: updating configuration: command=AddStaging server-id=server-af538914-1821-42ed-618e-9b1ec3c36966 server-addr=af538914-1821-42ed-618e-9b1ec3c36966 servers="[{Suffrage:Voter ID:server-af538914-1821-42ed-618e-9b1ec3c36966 Address:af538914-1821-42ed-618e-9b1ec3c36966} {Suffrage:Voter ID:server-6dd18996-5082-c331-dddb-6c455a171679 Address:6dd18996-5082-c331-dddb-6c455a171679} {Suffrage:Voter ID:server-4270c846-57e6-1c86-bab8-bd630057d2be Address:4270c846-57e6-1c86-bab8-bd630057d2be}]" - testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.987-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: updating configuration: command=AddStaging server-id=server-af538914-1821-42ed-618e-9b1ec3c36966 server-addr=af538914-1821-42ed-618e-9b1ec3c36966 servers="[{Suffrage:Voter ID:server-af538914-1821-42ed-618e-9b1ec3c36966 Address:af538914-1821-42ed-618e-9b1ec3c36966} {Suffrage:Voter ID:server-6dd18996-5082-c331-dddb-6c455a171679 Address:6dd18996-5082-c331-dddb-6c455a171679} {Suffrage:Voter ID:server-4270c846-57e6-1c86-bab8-bd630057d2be Address:4270c846-57e6-1c86-bab8-bd630057d2be}]" -testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.988-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: aborting pipeline replication: peer="{Voter server-af538914-1821-42ed-618e-9b1ec3c36966 af538914-1821-42ed-618e-9b1ec3c36966}" - testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.988-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: aborting pipeline replication: peer="{Voter server-af538914-1821-42ed-618e-9b1ec3c36966 af538914-1821-42ed-618e-9b1ec3c36966}" -testLoggerAdapter verbose: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.988-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: aborting pipeline replication: peer="{Voter server-6dd18996-5082-c331-dddb-6c455a171679 6dd18996-5082-c331-dddb-6c455a171679}" - testing.go:149: server-4270c846-57e6-1c86-bab8-bd630057d2be: 2024-06-16T08:26:32.988-1200 [INFO] server-4270c846-57e6-1c86-bab8-bd630057d2be: aborting pipeline replication: peer="{Voter server-6dd18996-5082-c331-dddb-6c455a171679 6dd18996-5082-c331-dddb-6c455a171679}" ---- PASS: TestRaft_AddKnownPeer (0.27s) +testLoggerAdapter verbose: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.173+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-369040ce-7c94-1215-9462-9dc9ac78065f Address:369040ce-7c94-1215-9462-9dc9ac78065f} {Suffrage:Voter ID:server-4a6c9f8d-2eae-7808-557e-996d16420ebc Address:4a6c9f8d-2eae-7808-557e-996d16420ebc} {Suffrage:Voter ID:server-fb311ca9-cff9-5558-1506-c706834f6a6b Address:fb311ca9-cff9-5558-1506-c706834f6a6b}]" + testing.go:149: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.173+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-369040ce-7c94-1215-9462-9dc9ac78065f Address:369040ce-7c94-1215-9462-9dc9ac78065f} {Suffrage:Voter ID:server-4a6c9f8d-2eae-7808-557e-996d16420ebc Address:4a6c9f8d-2eae-7808-557e-996d16420ebc} {Suffrage:Voter ID:server-fb311ca9-cff9-5558-1506-c706834f6a6b Address:fb311ca9-cff9-5558-1506-c706834f6a6b}]" +testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.173+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-369040ce-7c94-1215-9462-9dc9ac78065f Address:369040ce-7c94-1215-9462-9dc9ac78065f} {Suffrage:Voter ID:server-4a6c9f8d-2eae-7808-557e-996d16420ebc Address:4a6c9f8d-2eae-7808-557e-996d16420ebc} {Suffrage:Voter ID:server-fb311ca9-cff9-5558-1506-c706834f6a6b Address:fb311ca9-cff9-5558-1506-c706834f6a6b}]" + testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.173+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-369040ce-7c94-1215-9462-9dc9ac78065f Address:369040ce-7c94-1215-9462-9dc9ac78065f} {Suffrage:Voter ID:server-4a6c9f8d-2eae-7808-557e-996d16420ebc Address:4a6c9f8d-2eae-7808-557e-996d16420ebc} {Suffrage:Voter ID:server-fb311ca9-cff9-5558-1506-c706834f6a6b Address:fb311ca9-cff9-5558-1506-c706834f6a6b}]" +testLoggerAdapter verbose: server-fb311ca9-cff9-5558-1506-c706834f6a6b: 2023-05-16T04:06:13.174+1400 [INFO] server-fb311ca9-cff9-5558-1506-c706834f6a6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-369040ce-7c94-1215-9462-9dc9ac78065f Address:369040ce-7c94-1215-9462-9dc9ac78065f} {Suffrage:Voter ID:server-4a6c9f8d-2eae-7808-557e-996d16420ebc Address:4a6c9f8d-2eae-7808-557e-996d16420ebc} {Suffrage:Voter ID:server-fb311ca9-cff9-5558-1506-c706834f6a6b Address:fb311ca9-cff9-5558-1506-c706834f6a6b}]" + testing.go:149: server-fb311ca9-cff9-5558-1506-c706834f6a6b: 2023-05-16T04:06:13.174+1400 [INFO] server-fb311ca9-cff9-5558-1506-c706834f6a6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-369040ce-7c94-1215-9462-9dc9ac78065f Address:369040ce-7c94-1215-9462-9dc9ac78065f} {Suffrage:Voter ID:server-4a6c9f8d-2eae-7808-557e-996d16420ebc Address:4a6c9f8d-2eae-7808-557e-996d16420ebc} {Suffrage:Voter ID:server-fb311ca9-cff9-5558-1506-c706834f6a6b Address:fb311ca9-cff9-5558-1506-c706834f6a6b}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.174+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:13.174+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.174+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: entering follower state: follower="Node at 369040ce-7c94-1215-9462-9dc9ac78065f [Follower]" leader= + testing.go:149: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.174+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: entering follower state: follower="Node at 369040ce-7c94-1215-9462-9dc9ac78065f [Follower]" leader= +testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.174+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: entering follower state: follower="Node at 4a6c9f8d-2eae-7808-557e-996d16420ebc [Follower]" leader= + testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.174+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: entering follower state: follower="Node at 4a6c9f8d-2eae-7808-557e-996d16420ebc [Follower]" leader= +testLoggerAdapter verbose: server-fb311ca9-cff9-5558-1506-c706834f6a6b: 2023-05-16T04:06:13.174+1400 [INFO] server-fb311ca9-cff9-5558-1506-c706834f6a6b: entering follower state: follower="Node at fb311ca9-cff9-5558-1506-c706834f6a6b [Follower]" leader= + testing.go:149: server-fb311ca9-cff9-5558-1506-c706834f6a6b: 2023-05-16T04:06:13.174+1400 [INFO] server-fb311ca9-cff9-5558-1506-c706834f6a6b: entering follower state: follower="Node at fb311ca9-cff9-5558-1506-c706834f6a6b [Follower]" leader= +testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.255+1400 [WARN] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.255+1400 [WARN] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: entering candidate state: node="Node at 4a6c9f8d-2eae-7808-557e-996d16420ebc [Candidate]" term=2 + testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: entering candidate state: node="Node at 4a6c9f8d-2eae-7808-557e-996d16420ebc [Candidate]" term=2 +testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: election won: tally=2 + testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: election won: tally=2 +testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: entering leader state: leader="Node at 4a6c9f8d-2eae-7808-557e-996d16420ebc [Leader]" + testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: entering leader state: leader="Node at 4a6c9f8d-2eae-7808-557e-996d16420ebc [Leader]" +testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: added peer, starting replication: peer=server-369040ce-7c94-1215-9462-9dc9ac78065f + testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: added peer, starting replication: peer=server-369040ce-7c94-1215-9462-9dc9ac78065f +testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: added peer, starting replication: peer=server-fb311ca9-cff9-5558-1506-c706834f6a6b + testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: added peer, starting replication: peer=server-fb311ca9-cff9-5558-1506-c706834f6a6b +testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: pipelining replication: peer="{Voter server-fb311ca9-cff9-5558-1506-c706834f6a6b fb311ca9-cff9-5558-1506-c706834f6a6b}" + testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.256+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: pipelining replication: peer="{Voter server-fb311ca9-cff9-5558-1506-c706834f6a6b fb311ca9-cff9-5558-1506-c706834f6a6b}" +testLoggerAdapter verbose: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.256+1400 [WARN] server-369040ce-7c94-1215-9462-9dc9ac78065f: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.256+1400 [WARN] server-369040ce-7c94-1215-9462-9dc9ac78065f: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.256+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: entering candidate state: node="Node at 369040ce-7c94-1215-9462-9dc9ac78065f [Candidate]" term=2 + testing.go:149: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.256+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: entering candidate state: node="Node at 369040ce-7c94-1215-9462-9dc9ac78065f [Candidate]" term=2 +testLoggerAdapter verbose: server-fb311ca9-cff9-5558-1506-c706834f6a6b: 2023-05-16T04:06:13.257+1400 [WARN] server-fb311ca9-cff9-5558-1506-c706834f6a6b: rejecting vote request since we have a leader: from=369040ce-7c94-1215-9462-9dc9ac78065f leader=4a6c9f8d-2eae-7808-557e-996d16420ebc + testing.go:149: server-fb311ca9-cff9-5558-1506-c706834f6a6b: 2023-05-16T04:06:13.257+1400 [WARN] server-fb311ca9-cff9-5558-1506-c706834f6a6b: rejecting vote request since we have a leader: from=369040ce-7c94-1215-9462-9dc9ac78065f leader=4a6c9f8d-2eae-7808-557e-996d16420ebc +testLoggerAdapter verbose: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.257+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: duplicate requestVote for same term: term=2 + testing.go:149: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.257+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: duplicate requestVote for same term: term=2 +testLoggerAdapter verbose: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.257+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: entering follower state: follower="Node at 369040ce-7c94-1215-9462-9dc9ac78065f [Follower]" leader=4a6c9f8d-2eae-7808-557e-996d16420ebc + testing.go:149: server-369040ce-7c94-1215-9462-9dc9ac78065f: 2023-05-16T04:06:13.257+1400 [INFO] server-369040ce-7c94-1215-9462-9dc9ac78065f: entering follower state: follower="Node at 369040ce-7c94-1215-9462-9dc9ac78065f [Follower]" leader=4a6c9f8d-2eae-7808-557e-996d16420ebc +testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.257+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: pipelining replication: peer="{Voter server-369040ce-7c94-1215-9462-9dc9ac78065f 369040ce-7c94-1215-9462-9dc9ac78065f}" + testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.257+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: pipelining replication: peer="{Voter server-369040ce-7c94-1215-9462-9dc9ac78065f 369040ce-7c94-1215-9462-9dc9ac78065f}" +testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.257+1400 [WARN] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: rejecting vote request since we have a leader: from=369040ce-7c94-1215-9462-9dc9ac78065f leader=4a6c9f8d-2eae-7808-557e-996d16420ebc + testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.257+1400 [WARN] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: rejecting vote request since we have a leader: from=369040ce-7c94-1215-9462-9dc9ac78065f leader=4a6c9f8d-2eae-7808-557e-996d16420ebc +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.363+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:13.257613513 +1400 +14 m=+10.060195128 (1 nodes), 83.324623ms from start of poll, 84.191414ms from cluster start. Timeout at 2023-05-16 04:06:13.363647564 +1400 +14 m=+10.166229199, 106.034071ms after stability + testing.go:149: cluster: 2023-05-16T04:06:13.363+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:13.257613513 +1400 +14 m=+10.060195128 (1 nodes), 83.324623ms from start of poll, 84.191414ms from cluster start. Timeout at 2023-05-16 04:06:13.363647564 +1400 +14 m=+10.166229199, 106.034071ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.363+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:13.363+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.469+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:13.363977088 +1400 +14 m=+10.166558723 (2 nodes), 3.06µs from start of poll, 190.555009ms from cluster start. Timeout at 2023-05-16 04:06:13.469851978 +1400 +14 m=+10.272433713, 105.87499ms after stability + testing.go:149: cluster: 2023-05-16T04:06:13.469+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:13.363977088 +1400 +14 m=+10.166558723 (2 nodes), 3.06µs from start of poll, 190.555009ms from cluster start. Timeout at 2023-05-16 04:06:13.469851978 +1400 +14 m=+10.272433713, 105.87499ms after stability +testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.470+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: updating configuration: command=AddStaging server-id=server-369040ce-7c94-1215-9462-9dc9ac78065f server-addr=369040ce-7c94-1215-9462-9dc9ac78065f servers="[{Suffrage:Voter ID:server-369040ce-7c94-1215-9462-9dc9ac78065f Address:369040ce-7c94-1215-9462-9dc9ac78065f} {Suffrage:Voter ID:server-4a6c9f8d-2eae-7808-557e-996d16420ebc Address:4a6c9f8d-2eae-7808-557e-996d16420ebc} {Suffrage:Voter ID:server-fb311ca9-cff9-5558-1506-c706834f6a6b Address:fb311ca9-cff9-5558-1506-c706834f6a6b}]" + testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.470+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: updating configuration: command=AddStaging server-id=server-369040ce-7c94-1215-9462-9dc9ac78065f server-addr=369040ce-7c94-1215-9462-9dc9ac78065f servers="[{Suffrage:Voter ID:server-369040ce-7c94-1215-9462-9dc9ac78065f Address:369040ce-7c94-1215-9462-9dc9ac78065f} {Suffrage:Voter ID:server-4a6c9f8d-2eae-7808-557e-996d16420ebc Address:4a6c9f8d-2eae-7808-557e-996d16420ebc} {Suffrage:Voter ID:server-fb311ca9-cff9-5558-1506-c706834f6a6b Address:fb311ca9-cff9-5558-1506-c706834f6a6b}]" +testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.470+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: aborting pipeline replication: peer="{Voter server-fb311ca9-cff9-5558-1506-c706834f6a6b fb311ca9-cff9-5558-1506-c706834f6a6b}" + testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.470+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: aborting pipeline replication: peer="{Voter server-fb311ca9-cff9-5558-1506-c706834f6a6b fb311ca9-cff9-5558-1506-c706834f6a6b}" +testLoggerAdapter verbose: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.470+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: aborting pipeline replication: peer="{Voter server-369040ce-7c94-1215-9462-9dc9ac78065f 369040ce-7c94-1215-9462-9dc9ac78065f}" + testing.go:149: server-4a6c9f8d-2eae-7808-557e-996d16420ebc: 2023-05-16T04:06:13.470+1400 [INFO] server-4a6c9f8d-2eae-7808-557e-996d16420ebc: aborting pipeline replication: peer="{Voter server-369040ce-7c94-1215-9462-9dc9ac78065f 369040ce-7c94-1215-9462-9dc9ac78065f}" +--- PASS: TestRaft_AddKnownPeer (0.30s) === RUN TestRaft_RemoveUnknownPeer -testLoggerAdapter verbose: server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2: 2024-06-16T08:26:32.989-1200 [INFO] server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 Address:b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2} {Suffrage:Voter ID:server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d Address:6faaeb84-e589-5cf1-7604-6cb47ad8e92d} {Suffrage:Voter ID:server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede Address:adf5b989-ee08-bcd5-98a2-26f1e7ca3ede}]" - testing.go:149: server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2: 2024-06-16T08:26:32.989-1200 [INFO] server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 Address:b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2} {Suffrage:Voter ID:server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d Address:6faaeb84-e589-5cf1-7604-6cb47ad8e92d} {Suffrage:Voter ID:server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede Address:adf5b989-ee08-bcd5-98a2-26f1e7ca3ede}]" -testLoggerAdapter verbose: server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2: 2024-06-16T08:26:32.989-1200 [INFO] server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2: entering follower state: follower="Node at b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 [Follower]" leader= - testing.go:149: server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2: 2024-06-16T08:26:32.989-1200 [INFO] server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2: entering follower state: follower="Node at b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 [Follower]" leader= -testLoggerAdapter verbose: server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d: 2024-06-16T08:26:32.990-1200 [INFO] server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 Address:b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2} {Suffrage:Voter ID:server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d Address:6faaeb84-e589-5cf1-7604-6cb47ad8e92d} {Suffrage:Voter ID:server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede Address:adf5b989-ee08-bcd5-98a2-26f1e7ca3ede}]" - testing.go:149: server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d: 2024-06-16T08:26:32.990-1200 [INFO] server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 Address:b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2} {Suffrage:Voter ID:server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d Address:6faaeb84-e589-5cf1-7604-6cb47ad8e92d} {Suffrage:Voter ID:server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede Address:adf5b989-ee08-bcd5-98a2-26f1e7ca3ede}]" -testLoggerAdapter verbose: server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d: 2024-06-16T08:26:32.990-1200 [INFO] server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d: entering follower state: follower="Node at 6faaeb84-e589-5cf1-7604-6cb47ad8e92d [Follower]" leader= - testing.go:149: server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d: 2024-06-16T08:26:32.990-1200 [INFO] server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d: entering follower state: follower="Node at 6faaeb84-e589-5cf1-7604-6cb47ad8e92d [Follower]" leader= -testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:32.990-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 Address:b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2} {Suffrage:Voter ID:server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d Address:6faaeb84-e589-5cf1-7604-6cb47ad8e92d} {Suffrage:Voter ID:server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede Address:adf5b989-ee08-bcd5-98a2-26f1e7ca3ede}]" - testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:32.990-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 Address:b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2} {Suffrage:Voter ID:server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d Address:6faaeb84-e589-5cf1-7604-6cb47ad8e92d} {Suffrage:Voter ID:server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede Address:adf5b989-ee08-bcd5-98a2-26f1e7ca3ede}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:32.990-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:32.990-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:32.990-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: entering follower state: follower="Node at adf5b989-ee08-bcd5-98a2-26f1e7ca3ede [Follower]" leader= - testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:32.990-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: entering follower state: follower="Node at adf5b989-ee08-bcd5-98a2-26f1e7ca3ede [Follower]" leader= -testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [WARN] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [WARN] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: entering candidate state: node="Node at adf5b989-ee08-bcd5-98a2-26f1e7ca3ede [Candidate]" term=2 - testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: entering candidate state: node="Node at adf5b989-ee08-bcd5-98a2-26f1e7ca3ede [Candidate]" term=2 -testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: election won: tally=2 - testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: election won: tally=2 -testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: entering leader state: leader="Node at adf5b989-ee08-bcd5-98a2-26f1e7ca3ede [Leader]" - testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: entering leader state: leader="Node at adf5b989-ee08-bcd5-98a2-26f1e7ca3ede [Leader]" -testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: added peer, starting replication: peer=server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 - testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: added peer, starting replication: peer=server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 -testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: added peer, starting replication: peer=server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d - testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: added peer, starting replication: peer=server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d -testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: pipelining replication: peer="{Voter server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2}" - testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: pipelining replication: peer="{Voter server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2}" -testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: pipelining replication: peer="{Voter server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d 6faaeb84-e589-5cf1-7604-6cb47ad8e92d}" - testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.043-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: pipelining replication: peer="{Voter server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d 6faaeb84-e589-5cf1-7604-6cb47ad8e92d}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.149-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.0437789 -1200 -12 m=+10.051095361 (1 nodes), 53.30942ms from start of poll, 54.24352ms from cluster start. Timeout at 2024-06-16 08:26:33.14894488 -1200 -12 m=+10.156261381, 105.16602ms after stability - testing.go:149: cluster: 2024-06-16T08:26:33.149-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.0437789 -1200 -12 m=+10.051095361 (1 nodes), 53.30942ms from start of poll, 54.24352ms from cluster start. Timeout at 2024-06-16 08:26:33.14894488 -1200 -12 m=+10.156261381, 105.16602ms after stability -testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.149-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: updating configuration: command=RemoveServer server-id=5705c359-0d59-4ebd-4fe8-01c185a91d9c server-addr= servers="[{Suffrage:Voter ID:server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 Address:b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2} {Suffrage:Voter ID:server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d Address:6faaeb84-e589-5cf1-7604-6cb47ad8e92d} {Suffrage:Voter ID:server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede Address:adf5b989-ee08-bcd5-98a2-26f1e7ca3ede}]" - testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.149-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: updating configuration: command=RemoveServer server-id=5705c359-0d59-4ebd-4fe8-01c185a91d9c server-addr= servers="[{Suffrage:Voter ID:server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 Address:b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2} {Suffrage:Voter ID:server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d Address:6faaeb84-e589-5cf1-7604-6cb47ad8e92d} {Suffrage:Voter ID:server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede Address:adf5b989-ee08-bcd5-98a2-26f1e7ca3ede}]" -testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.149-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: aborting pipeline replication: peer="{Voter server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d 6faaeb84-e589-5cf1-7604-6cb47ad8e92d}" - testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.149-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: aborting pipeline replication: peer="{Voter server-6faaeb84-e589-5cf1-7604-6cb47ad8e92d 6faaeb84-e589-5cf1-7604-6cb47ad8e92d}" -testLoggerAdapter verbose: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.149-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: aborting pipeline replication: peer="{Voter server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2}" - testing.go:149: server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: 2024-06-16T08:26:33.149-1200 [INFO] server-adf5b989-ee08-bcd5-98a2-26f1e7ca3ede: aborting pipeline replication: peer="{Voter server-b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2 b838e724-2b33-6ea5-0dd3-bcea8b1e5eb2}" ---- PASS: TestRaft_RemoveUnknownPeer (0.16s) +testLoggerAdapter verbose: server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171: 2023-05-16T04:06:13.472+1400 [INFO] server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 Address:ff128af1-a6a0-9f9b-6060-d13a4bd0a171} {Suffrage:Voter ID:server-69f24695-c539-8820-3e6a-4aa30dd1a7be Address:69f24695-c539-8820-3e6a-4aa30dd1a7be} {Suffrage:Voter ID:server-f91e2e08-5cb3-532d-1da6-1712cfd5038c Address:f91e2e08-5cb3-532d-1da6-1712cfd5038c}]" + testing.go:149: server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171: 2023-05-16T04:06:13.472+1400 [INFO] server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 Address:ff128af1-a6a0-9f9b-6060-d13a4bd0a171} {Suffrage:Voter ID:server-69f24695-c539-8820-3e6a-4aa30dd1a7be Address:69f24695-c539-8820-3e6a-4aa30dd1a7be} {Suffrage:Voter ID:server-f91e2e08-5cb3-532d-1da6-1712cfd5038c Address:f91e2e08-5cb3-532d-1da6-1712cfd5038c}]" +testLoggerAdapter verbose: server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171: 2023-05-16T04:06:13.472+1400 [INFO] server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171: entering follower state: follower="Node at ff128af1-a6a0-9f9b-6060-d13a4bd0a171 [Follower]" leader= + testing.go:149: server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171: 2023-05-16T04:06:13.472+1400 [INFO] server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171: entering follower state: follower="Node at ff128af1-a6a0-9f9b-6060-d13a4bd0a171 [Follower]" leader= +testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.472+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 Address:ff128af1-a6a0-9f9b-6060-d13a4bd0a171} {Suffrage:Voter ID:server-69f24695-c539-8820-3e6a-4aa30dd1a7be Address:69f24695-c539-8820-3e6a-4aa30dd1a7be} {Suffrage:Voter ID:server-f91e2e08-5cb3-532d-1da6-1712cfd5038c Address:f91e2e08-5cb3-532d-1da6-1712cfd5038c}]" + testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.472+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 Address:ff128af1-a6a0-9f9b-6060-d13a4bd0a171} {Suffrage:Voter ID:server-69f24695-c539-8820-3e6a-4aa30dd1a7be Address:69f24695-c539-8820-3e6a-4aa30dd1a7be} {Suffrage:Voter ID:server-f91e2e08-5cb3-532d-1da6-1712cfd5038c Address:f91e2e08-5cb3-532d-1da6-1712cfd5038c}]" +testLoggerAdapter verbose: server-f91e2e08-5cb3-532d-1da6-1712cfd5038c: 2023-05-16T04:06:13.473+1400 [INFO] server-f91e2e08-5cb3-532d-1da6-1712cfd5038c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 Address:ff128af1-a6a0-9f9b-6060-d13a4bd0a171} {Suffrage:Voter ID:server-69f24695-c539-8820-3e6a-4aa30dd1a7be Address:69f24695-c539-8820-3e6a-4aa30dd1a7be} {Suffrage:Voter ID:server-f91e2e08-5cb3-532d-1da6-1712cfd5038c Address:f91e2e08-5cb3-532d-1da6-1712cfd5038c}]" + testing.go:149: server-f91e2e08-5cb3-532d-1da6-1712cfd5038c: 2023-05-16T04:06:13.473+1400 [INFO] server-f91e2e08-5cb3-532d-1da6-1712cfd5038c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 Address:ff128af1-a6a0-9f9b-6060-d13a4bd0a171} {Suffrage:Voter ID:server-69f24695-c539-8820-3e6a-4aa30dd1a7be Address:69f24695-c539-8820-3e6a-4aa30dd1a7be} {Suffrage:Voter ID:server-f91e2e08-5cb3-532d-1da6-1712cfd5038c Address:f91e2e08-5cb3-532d-1da6-1712cfd5038c}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.473+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:13.473+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.473+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: entering follower state: follower="Node at 69f24695-c539-8820-3e6a-4aa30dd1a7be [Follower]" leader= + testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.473+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: entering follower state: follower="Node at 69f24695-c539-8820-3e6a-4aa30dd1a7be [Follower]" leader= +testLoggerAdapter verbose: server-f91e2e08-5cb3-532d-1da6-1712cfd5038c: 2023-05-16T04:06:13.473+1400 [INFO] server-f91e2e08-5cb3-532d-1da6-1712cfd5038c: entering follower state: follower="Node at f91e2e08-5cb3-532d-1da6-1712cfd5038c [Follower]" leader= + testing.go:149: server-f91e2e08-5cb3-532d-1da6-1712cfd5038c: 2023-05-16T04:06:13.473+1400 [INFO] server-f91e2e08-5cb3-532d-1da6-1712cfd5038c: entering follower state: follower="Node at f91e2e08-5cb3-532d-1da6-1712cfd5038c [Follower]" leader= +testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.534+1400 [WARN] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.534+1400 [WARN] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.534+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: entering candidate state: node="Node at 69f24695-c539-8820-3e6a-4aa30dd1a7be [Candidate]" term=2 + testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.534+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: entering candidate state: node="Node at 69f24695-c539-8820-3e6a-4aa30dd1a7be [Candidate]" term=2 +testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: election won: tally=2 + testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: election won: tally=2 +testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: entering leader state: leader="Node at 69f24695-c539-8820-3e6a-4aa30dd1a7be [Leader]" + testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: entering leader state: leader="Node at 69f24695-c539-8820-3e6a-4aa30dd1a7be [Leader]" +testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: added peer, starting replication: peer=server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 + testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: added peer, starting replication: peer=server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 +testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: added peer, starting replication: peer=server-f91e2e08-5cb3-532d-1da6-1712cfd5038c + testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: added peer, starting replication: peer=server-f91e2e08-5cb3-532d-1da6-1712cfd5038c +testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: pipelining replication: peer="{Voter server-f91e2e08-5cb3-532d-1da6-1712cfd5038c f91e2e08-5cb3-532d-1da6-1712cfd5038c}" + testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: pipelining replication: peer="{Voter server-f91e2e08-5cb3-532d-1da6-1712cfd5038c f91e2e08-5cb3-532d-1da6-1712cfd5038c}" +testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: pipelining replication: peer="{Voter server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 ff128af1-a6a0-9f9b-6060-d13a4bd0a171}" + testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.535+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: pipelining replication: peer="{Voter server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 ff128af1-a6a0-9f9b-6060-d13a4bd0a171}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.641+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:13.535896917 +1400 +14 m=+10.338478552 (1 nodes), 62.075272ms from start of poll, 63.59989ms from cluster start. Timeout at 2023-05-16 04:06:13.640939277 +1400 +14 m=+10.443520912, 105.04236ms after stability + testing.go:149: cluster: 2023-05-16T04:06:13.641+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:13.535896917 +1400 +14 m=+10.338478552 (1 nodes), 62.075272ms from start of poll, 63.59989ms from cluster start. Timeout at 2023-05-16 04:06:13.640939277 +1400 +14 m=+10.443520912, 105.04236ms after stability +testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.641+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: updating configuration: command=RemoveServer server-id=3cb46c98-ecf6-9a7c-cda8-18a095bb4872 server-addr= servers="[{Suffrage:Voter ID:server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 Address:ff128af1-a6a0-9f9b-6060-d13a4bd0a171} {Suffrage:Voter ID:server-69f24695-c539-8820-3e6a-4aa30dd1a7be Address:69f24695-c539-8820-3e6a-4aa30dd1a7be} {Suffrage:Voter ID:server-f91e2e08-5cb3-532d-1da6-1712cfd5038c Address:f91e2e08-5cb3-532d-1da6-1712cfd5038c}]" + testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.641+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: updating configuration: command=RemoveServer server-id=3cb46c98-ecf6-9a7c-cda8-18a095bb4872 server-addr= servers="[{Suffrage:Voter ID:server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 Address:ff128af1-a6a0-9f9b-6060-d13a4bd0a171} {Suffrage:Voter ID:server-69f24695-c539-8820-3e6a-4aa30dd1a7be Address:69f24695-c539-8820-3e6a-4aa30dd1a7be} {Suffrage:Voter ID:server-f91e2e08-5cb3-532d-1da6-1712cfd5038c Address:f91e2e08-5cb3-532d-1da6-1712cfd5038c}]" +testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.641+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: aborting pipeline replication: peer="{Voter server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 ff128af1-a6a0-9f9b-6060-d13a4bd0a171}" + testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.641+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: aborting pipeline replication: peer="{Voter server-ff128af1-a6a0-9f9b-6060-d13a4bd0a171 ff128af1-a6a0-9f9b-6060-d13a4bd0a171}" +testLoggerAdapter verbose: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.641+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: aborting pipeline replication: peer="{Voter server-f91e2e08-5cb3-532d-1da6-1712cfd5038c f91e2e08-5cb3-532d-1da6-1712cfd5038c}" + testing.go:149: server-69f24695-c539-8820-3e6a-4aa30dd1a7be: 2023-05-16T04:06:13.641+1400 [INFO] server-69f24695-c539-8820-3e6a-4aa30dd1a7be: aborting pipeline replication: peer="{Voter server-f91e2e08-5cb3-532d-1da6-1712cfd5038c f91e2e08-5cb3-532d-1da6-1712cfd5038c}" +--- PASS: TestRaft_RemoveUnknownPeer (0.17s) === RUN TestRaft_SnapshotRestore -testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.150-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36ab0502-1ee0-9019-44be-303864e2599d Address:36ab0502-1ee0-9019-44be-303864e2599d}]" - testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.150-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36ab0502-1ee0-9019-44be-303864e2599d Address:36ab0502-1ee0-9019-44be-303864e2599d}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.150-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:33.150-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.150-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: entering follower state: follower="Node at 36ab0502-1ee0-9019-44be-303864e2599d [Follower]" leader= - testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.150-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: entering follower state: follower="Node at 36ab0502-1ee0-9019-44be-303864e2599d [Follower]" leader= -testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.223-1200 [WARN] server-36ab0502-1ee0-9019-44be-303864e2599d: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.223-1200 [WARN] server-36ab0502-1ee0-9019-44be-303864e2599d: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.223-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: entering candidate state: node="Node at 36ab0502-1ee0-9019-44be-303864e2599d [Candidate]" term=2 - testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.223-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: entering candidate state: node="Node at 36ab0502-1ee0-9019-44be-303864e2599d [Candidate]" term=2 -testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.223-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: election won: tally=1 - testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.223-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: election won: tally=1 -testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.223-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: entering leader state: leader="Node at 36ab0502-1ee0-9019-44be-303864e2599d [Leader]" - testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.223-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: entering leader state: leader="Node at 36ab0502-1ee0-9019-44be-303864e2599d [Leader]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.329-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.22349062 -1200 -12 m=+10.230807161 (1 nodes), 72.59678ms from start of poll, 72.94072ms from cluster start. Timeout at 2024-06-16 08:26:33.3292815 -1200 -12 m=+10.336597981, 105.79082ms after stability - testing.go:149: cluster: 2024-06-16T08:26:33.329-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.22349062 -1200 -12 m=+10.230807161 (1 nodes), 72.59678ms from start of poll, 72.94072ms from cluster start. Timeout at 2024-06-16 08:26:33.3292815 -1200 -12 m=+10.336597981, 105.79082ms after stability -testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.333-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: starting snapshot up to: index=102 - testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.333-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: starting snapshot up to: index=102 - testing.go:153: 2024-06-16T08:26:33.333-1200 [INFO] creating new snapshot: path=/tmp/raft696262971/snapshots/2-102-1718569593333.tmp -testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.335-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: compacting logs: from=1 to=92 - testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.335-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.335-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: snapshot complete up to: index=102 - testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.335-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.336-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: restored from snapshot: id=2-102-1718569593333 - testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.336-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: restored from snapshot: id=2-102-1718569593333 -testLoggerAdapter verbose: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.336-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36ab0502-1ee0-9019-44be-303864e2599d Address:36ab0502-1ee0-9019-44be-303864e2599d}]" - testing.go:149: server-36ab0502-1ee0-9019-44be-303864e2599d: 2024-06-16T08:26:33.336-1200 [INFO] server-36ab0502-1ee0-9019-44be-303864e2599d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-36ab0502-1ee0-9019-44be-303864e2599d Address:36ab0502-1ee0-9019-44be-303864e2599d}]" ---- PASS: TestRaft_SnapshotRestore (0.19s) +testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.643+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc Address:6b6dcef1-2623-d4a6-b4f1-86c151141dbc}]" + testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.643+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc Address:6b6dcef1-2623-d4a6-b4f1-86c151141dbc}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.643+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:13.643+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.643+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: entering follower state: follower="Node at 6b6dcef1-2623-d4a6-b4f1-86c151141dbc [Follower]" leader= + testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.643+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: entering follower state: follower="Node at 6b6dcef1-2623-d4a6-b4f1-86c151141dbc [Follower]" leader= +testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.705+1400 [WARN] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.705+1400 [WARN] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.705+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: entering candidate state: node="Node at 6b6dcef1-2623-d4a6-b4f1-86c151141dbc [Candidate]" term=2 + testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.705+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: entering candidate state: node="Node at 6b6dcef1-2623-d4a6-b4f1-86c151141dbc [Candidate]" term=2 +testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.705+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: election won: tally=1 + testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.705+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: election won: tally=1 +testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.705+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: entering leader state: leader="Node at 6b6dcef1-2623-d4a6-b4f1-86c151141dbc [Leader]" + testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.705+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: entering leader state: leader="Node at 6b6dcef1-2623-d4a6-b4f1-86c151141dbc [Leader]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.811+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:13.705729721 +1400 +14 m=+10.508311356 (1 nodes), 62.422996ms from start of poll, 62.808341ms from cluster start. Timeout at 2023-05-16 04:06:13.811703352 +1400 +14 m=+10.614285007, 105.973651ms after stability + testing.go:149: cluster: 2023-05-16T04:06:13.811+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:13.705729721 +1400 +14 m=+10.508311356 (1 nodes), 62.422996ms from start of poll, 62.808341ms from cluster start. Timeout at 2023-05-16 04:06:13.811703352 +1400 +14 m=+10.614285007, 105.973651ms after stability +testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.813+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: starting snapshot up to: index=102 + testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.813+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: starting snapshot up to: index=102 + testing.go:153: 2023-05-16T04:06:13.813+1400 [INFO] creating new snapshot: path=/tmp/raft3765765489/snapshots/2-102-1684159573813.tmp +testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.814+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: compacting logs: from=1 to=92 + testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.814+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.814+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: snapshot complete up to: index=102 + testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.814+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.815+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: restored from snapshot: id=2-102-1684159573813 + testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.815+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: restored from snapshot: id=2-102-1684159573813 +testLoggerAdapter verbose: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.815+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc Address:6b6dcef1-2623-d4a6-b4f1-86c151141dbc}]" + testing.go:149: server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: 2023-05-16T04:06:13.815+1400 [INFO] server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b6dcef1-2623-d4a6-b4f1-86c151141dbc Address:6b6dcef1-2623-d4a6-b4f1-86c151141dbc}]" +--- PASS: TestRaft_SnapshotRestore (0.17s) === RUN TestRaft_NoRestoreOnStart -testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.337-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e Address:1e5e2d89-1e1c-2da4-23b2-027a3523a11e}]" - testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.337-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e Address:1e5e2d89-1e1c-2da4-23b2-027a3523a11e}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.337-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:33.337-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.337-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering follower state: follower="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Follower]" leader= - testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.337-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering follower state: follower="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Follower]" leader= -testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.393-1200 [WARN] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.393-1200 [WARN] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.394-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering candidate state: node="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Candidate]" term=2 - testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.394-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering candidate state: node="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Candidate]" term=2 -testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.394-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: election won: tally=1 - testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.394-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: election won: tally=1 -testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.394-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering leader state: leader="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Leader]" - testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.394-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering leader state: leader="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Leader]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.500-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.39432972 -1200 -12 m=+10.401646201 (1 nodes), 57.06134ms from start of poll, 57.4243ms from cluster start. Timeout at 2024-06-16 08:26:33.5000756 -1200 -12 m=+10.507392101, 105.7459ms after stability - testing.go:149: cluster: 2024-06-16T08:26:33.500-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.39432972 -1200 -12 m=+10.401646201 (1 nodes), 57.06134ms from start of poll, 57.4243ms from cluster start. Timeout at 2024-06-16 08:26:33.5000756 -1200 -12 m=+10.507392101, 105.7459ms after stability -testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.502-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: starting snapshot up to: index=102 - testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.502-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: starting snapshot up to: index=102 - testing.go:153: 2024-06-16T08:26:33.502-1200 [INFO] creating new snapshot: path=/tmp/raft704908476/snapshots/2-102-1718569593502.tmp -testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.503-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: compacting logs: from=1 to=92 - testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.503-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.503-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: snapshot complete up to: index=102 - testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.503-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.503-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e Address:1e5e2d89-1e1c-2da4-23b2-027a3523a11e}]" - testing.go:149: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.503-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e Address:1e5e2d89-1e1c-2da4-23b2-027a3523a11e}]" +testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.816+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed Address:ad232f03-1d75-0e61-8bf5-74f8f9af77ed}]" + testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.816+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed Address:ad232f03-1d75-0e61-8bf5-74f8f9af77ed}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.817+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:13.817+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.817+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering follower state: follower="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Follower]" leader= + testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.817+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering follower state: follower="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Follower]" leader= +testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.875+1400 [WARN] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.875+1400 [WARN] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.876+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering candidate state: node="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Candidate]" term=2 + testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.876+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering candidate state: node="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Candidate]" term=2 +testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.876+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: election won: tally=1 + testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.876+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: election won: tally=1 +testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.876+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering leader state: leader="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Leader]" + testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.876+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering leader state: leader="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Leader]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.981+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:13.876395635 +1400 +14 m=+10.678977270 (1 nodes), 59.288419ms from start of poll, 59.710364ms from cluster start. Timeout at 2023-05-16 04:06:13.981718878 +1400 +14 m=+10.784300513, 105.323243ms after stability + testing.go:149: cluster: 2023-05-16T04:06:13.981+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:13.876395635 +1400 +14 m=+10.678977270 (1 nodes), 59.288419ms from start of poll, 59.710364ms from cluster start. Timeout at 2023-05-16 04:06:13.981718878 +1400 +14 m=+10.784300513, 105.323243ms after stability +testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.983+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: starting snapshot up to: index=102 + testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.983+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: starting snapshot up to: index=102 + testing.go:153: 2023-05-16T04:06:13.983+1400 [INFO] creating new snapshot: path=/tmp/raft2876506391/snapshots/2-102-1684159573983.tmp +testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.984+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: compacting logs: from=1 to=92 + testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.984+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.984+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: snapshot complete up to: index=102 + testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.984+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.985+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed Address:ad232f03-1d75-0e61-8bf5-74f8f9af77ed}]" + testing.go:149: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.985+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed Address:ad232f03-1d75-0e61-8bf5-74f8f9af77ed}]" --- PASS: TestRaft_NoRestoreOnStart (0.17s) === RUN TestRaft_SnapshotRestore_PeerChange -testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.503-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering follower state: follower="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Follower]" leader= raft_test.go:1026: Disabled unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot raft_test.go:1132: Disabled unreliable test --- SKIP: TestRaft_AutoSnapshot (0.00s) === RUN TestRaft_UserSnapshot -testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.504-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0fc5db7f-287a-e808-6b09-c0c758b1d12b Address:0fc5db7f-287a-e808-6b09-c0c758b1d12b}]" - testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.504-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0fc5db7f-287a-e808-6b09-c0c758b1d12b Address:0fc5db7f-287a-e808-6b09-c0c758b1d12b}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.504-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:33.504-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.504-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: entering follower state: follower="Node at 0fc5db7f-287a-e808-6b09-c0c758b1d12b [Follower]" leader= - testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.504-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: entering follower state: follower="Node at 0fc5db7f-287a-e808-6b09-c0c758b1d12b [Follower]" leader= -testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.560-1200 [WARN] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.560-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering candidate state: node="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Candidate]" term=3 -testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.560-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: election won: tally=1 -testLoggerAdapter verbose: server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: 2024-06-16T08:26:33.560-1200 [INFO] server-1e5e2d89-1e1c-2da4-23b2-027a3523a11e: entering leader state: leader="Node at 1e5e2d89-1e1c-2da4-23b2-027a3523a11e [Leader]" -testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.595-1200 [WARN] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.595-1200 [WARN] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.596-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: entering candidate state: node="Node at 0fc5db7f-287a-e808-6b09-c0c758b1d12b [Candidate]" term=2 - testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.596-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: entering candidate state: node="Node at 0fc5db7f-287a-e808-6b09-c0c758b1d12b [Candidate]" term=2 -testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.596-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: election won: tally=1 - testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.596-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: election won: tally=1 -testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.596-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: entering leader state: leader="Node at 0fc5db7f-287a-e808-6b09-c0c758b1d12b [Leader]" - testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.596-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: entering leader state: leader="Node at 0fc5db7f-287a-e808-6b09-c0c758b1d12b [Leader]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.701-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.59629328 -1200 -12 m=+10.603609761 (1 nodes), 91.64214ms from start of poll, 91.9887ms from cluster start. Timeout at 2024-06-16 08:26:33.70157222 -1200 -12 m=+10.708888721, 105.27896ms after stability - testing.go:149: cluster: 2024-06-16T08:26:33.701-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.59629328 -1200 -12 m=+10.603609761 (1 nodes), 91.64214ms from start of poll, 91.9887ms from cluster start. Timeout at 2024-06-16 08:26:33.70157222 -1200 -12 m=+10.708888721, 105.27896ms after stability -testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.701-1200 [ERROR] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: failed to take snapshot: error="nothing new to snapshot" - testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.701-1200 [ERROR] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: failed to take snapshot: error="nothing new to snapshot" -testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.702-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: starting snapshot up to: index=12 - testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.702-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: starting snapshot up to: index=12 - testing.go:153: 2024-06-16T08:26:33.702-1200 [INFO] creating new snapshot: path=/tmp/raft617634784/snapshots/2-12-1718569593702.tmp -testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.702-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: compacting logs: from=1 to=2 - testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.702-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: compacting logs: from=1 to=2 -testLoggerAdapter verbose: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.702-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: snapshot complete up to: index=12 - testing.go:149: server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: 2024-06-16T08:26:33.702-1200 [INFO] server-0fc5db7f-287a-e808-6b09-c0c758b1d12b: snapshot complete up to: index=12 ---- PASS: TestRaft_UserSnapshot (0.20s) +testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:13.985+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc1c8995-2994-6afa-6868-b168abdfc0aa Address:dc1c8995-2994-6afa-6868-b168abdfc0aa}]" + testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:13.985+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc1c8995-2994-6afa-6868-b168abdfc0aa Address:dc1c8995-2994-6afa-6868-b168abdfc0aa}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:13.986+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:13.986+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:13.986+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering follower state: follower="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Follower]" leader= +testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:13.986+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: entering follower state: follower="Node at dc1c8995-2994-6afa-6868-b168abdfc0aa [Follower]" leader= + testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:13.986+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: entering follower state: follower="Node at dc1c8995-2994-6afa-6868-b168abdfc0aa [Follower]" leader= +testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.051+1400 [WARN] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.051+1400 [WARN] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.052+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: entering candidate state: node="Node at dc1c8995-2994-6afa-6868-b168abdfc0aa [Candidate]" term=2 + testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.052+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: entering candidate state: node="Node at dc1c8995-2994-6afa-6868-b168abdfc0aa [Candidate]" term=2 +testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.052+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: election won: tally=1 + testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.052+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: election won: tally=1 +testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.052+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: entering leader state: leader="Node at dc1c8995-2994-6afa-6868-b168abdfc0aa [Leader]" + testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.052+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: entering leader state: leader="Node at dc1c8995-2994-6afa-6868-b168abdfc0aa [Leader]" +testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:14.081+1400 [WARN] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:14.082+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering candidate state: node="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Candidate]" term=3 +testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:14.082+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: election won: tally=1 +testLoggerAdapter verbose: server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: 2023-05-16T04:06:14.082+1400 [INFO] server-ad232f03-1d75-0e61-8bf5-74f8f9af77ed: entering leader state: leader="Node at ad232f03-1d75-0e61-8bf5-74f8f9af77ed [Leader]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.158+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.052504874 +1400 +14 m=+10.855086509 (1 nodes), 66.409704ms from start of poll, 66.755148ms from cluster start. Timeout at 2023-05-16 04:06:14.15814252 +1400 +14 m=+10.960724175, 105.637666ms after stability + testing.go:149: cluster: 2023-05-16T04:06:14.158+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.052504874 +1400 +14 m=+10.855086509 (1 nodes), 66.409704ms from start of poll, 66.755148ms from cluster start. Timeout at 2023-05-16 04:06:14.15814252 +1400 +14 m=+10.960724175, 105.637666ms after stability +testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.158+1400 [ERROR] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: failed to take snapshot: error="nothing new to snapshot" + testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.158+1400 [ERROR] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: failed to take snapshot: error="nothing new to snapshot" +testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.158+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: starting snapshot up to: index=12 + testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.158+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: starting snapshot up to: index=12 + testing.go:153: 2023-05-16T04:06:14.158+1400 [INFO] creating new snapshot: path=/tmp/raft3468561679/snapshots/2-12-1684159574158.tmp +testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.159+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: compacting logs: from=1 to=2 + testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.159+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: compacting logs: from=1 to=2 +testLoggerAdapter verbose: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.159+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: snapshot complete up to: index=12 + testing.go:149: server-dc1c8995-2994-6afa-6868-b168abdfc0aa: 2023-05-16T04:06:14.159+1400 [INFO] server-dc1c8995-2994-6afa-6868-b168abdfc0aa: snapshot complete up to: index=12 +--- PASS: TestRaft_UserSnapshot (0.17s) === RUN TestRaft_UserRestore -testLoggerAdapter verbose: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.704-1200 [INFO] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5071370-c7f5-11c6-8f28-fa99b8f7618e Address:b5071370-c7f5-11c6-8f28-fa99b8f7618e} {Suffrage:Voter ID:server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd Address:4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd} {Suffrage:Voter ID:server-12bfc68f-2775-5023-31e0-b166f8e418b0 Address:12bfc68f-2775-5023-31e0-b166f8e418b0}]" - testing.go:149: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.704-1200 [INFO] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5071370-c7f5-11c6-8f28-fa99b8f7618e Address:b5071370-c7f5-11c6-8f28-fa99b8f7618e} {Suffrage:Voter ID:server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd Address:4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd} {Suffrage:Voter ID:server-12bfc68f-2775-5023-31e0-b166f8e418b0 Address:12bfc68f-2775-5023-31e0-b166f8e418b0}]" -testLoggerAdapter verbose: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.704-1200 [INFO] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: entering follower state: follower="Node at b5071370-c7f5-11c6-8f28-fa99b8f7618e [Follower]" leader= - testing.go:149: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.704-1200 [INFO] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: entering follower state: follower="Node at b5071370-c7f5-11c6-8f28-fa99b8f7618e [Follower]" leader= -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.704-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5071370-c7f5-11c6-8f28-fa99b8f7618e Address:b5071370-c7f5-11c6-8f28-fa99b8f7618e} {Suffrage:Voter ID:server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd Address:4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd} {Suffrage:Voter ID:server-12bfc68f-2775-5023-31e0-b166f8e418b0 Address:12bfc68f-2775-5023-31e0-b166f8e418b0}]" - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.704-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5071370-c7f5-11c6-8f28-fa99b8f7618e Address:b5071370-c7f5-11c6-8f28-fa99b8f7618e} {Suffrage:Voter ID:server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd Address:4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd} {Suffrage:Voter ID:server-12bfc68f-2775-5023-31e0-b166f8e418b0 Address:12bfc68f-2775-5023-31e0-b166f8e418b0}]" -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.704-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: entering follower state: follower="Node at 4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd [Follower]" leader= - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.704-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: entering follower state: follower="Node at 4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd [Follower]" leader= -testLoggerAdapter verbose: server-12bfc68f-2775-5023-31e0-b166f8e418b0: 2024-06-16T08:26:33.704-1200 [INFO] server-12bfc68f-2775-5023-31e0-b166f8e418b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5071370-c7f5-11c6-8f28-fa99b8f7618e Address:b5071370-c7f5-11c6-8f28-fa99b8f7618e} {Suffrage:Voter ID:server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd Address:4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd} {Suffrage:Voter ID:server-12bfc68f-2775-5023-31e0-b166f8e418b0 Address:12bfc68f-2775-5023-31e0-b166f8e418b0}]" - testing.go:149: server-12bfc68f-2775-5023-31e0-b166f8e418b0: 2024-06-16T08:26:33.704-1200 [INFO] server-12bfc68f-2775-5023-31e0-b166f8e418b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5071370-c7f5-11c6-8f28-fa99b8f7618e Address:b5071370-c7f5-11c6-8f28-fa99b8f7618e} {Suffrage:Voter ID:server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd Address:4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd} {Suffrage:Voter ID:server-12bfc68f-2775-5023-31e0-b166f8e418b0 Address:12bfc68f-2775-5023-31e0-b166f8e418b0}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.704-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:33.704-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-12bfc68f-2775-5023-31e0-b166f8e418b0: 2024-06-16T08:26:33.704-1200 [INFO] server-12bfc68f-2775-5023-31e0-b166f8e418b0: entering follower state: follower="Node at 12bfc68f-2775-5023-31e0-b166f8e418b0 [Follower]" leader= - testing.go:149: server-12bfc68f-2775-5023-31e0-b166f8e418b0: 2024-06-16T08:26:33.704-1200 [INFO] server-12bfc68f-2775-5023-31e0-b166f8e418b0: entering follower state: follower="Node at 12bfc68f-2775-5023-31e0-b166f8e418b0 [Follower]" leader= -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [WARN] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [WARN] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: entering candidate state: node="Node at 4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd [Candidate]" term=2 - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: entering candidate state: node="Node at 4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd [Candidate]" term=2 -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: election won: tally=2 - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: election won: tally=2 -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: entering leader state: leader="Node at 4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd [Leader]" - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: entering leader state: leader="Node at 4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd [Leader]" -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: added peer, starting replication: peer=server-b5071370-c7f5-11c6-8f28-fa99b8f7618e - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: added peer, starting replication: peer=server-b5071370-c7f5-11c6-8f28-fa99b8f7618e -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: added peer, starting replication: peer=server-12bfc68f-2775-5023-31e0-b166f8e418b0 - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: added peer, starting replication: peer=server-12bfc68f-2775-5023-31e0-b166f8e418b0 -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: pipelining replication: peer="{Voter server-b5071370-c7f5-11c6-8f28-fa99b8f7618e b5071370-c7f5-11c6-8f28-fa99b8f7618e}" - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.765-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: pipelining replication: peer="{Voter server-b5071370-c7f5-11c6-8f28-fa99b8f7618e b5071370-c7f5-11c6-8f28-fa99b8f7618e}" -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.766-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: pipelining replication: peer="{Voter server-12bfc68f-2775-5023-31e0-b166f8e418b0 12bfc68f-2775-5023-31e0-b166f8e418b0}" - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.766-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: pipelining replication: peer="{Voter server-12bfc68f-2775-5023-31e0-b166f8e418b0 12bfc68f-2775-5023-31e0-b166f8e418b0}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.871-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.76584174 -1200 -12 m=+10.773158201 (1 nodes), 61.02984ms from start of poll, 61.8581ms from cluster start. Timeout at 2024-06-16 08:26:33.8714222 -1200 -12 m=+10.878738681, 105.58048ms after stability - testing.go:149: cluster: 2024-06-16T08:26:33.871-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.76584174 -1200 -12 m=+10.773158201 (1 nodes), 61.02984ms from start of poll, 61.8581ms from cluster start. Timeout at 2024-06-16 08:26:33.8714222 -1200 -12 m=+10.878738681, 105.58048ms after stability -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.872-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: starting snapshot up to: index=12 - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.872-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: starting snapshot up to: index=12 - testing.go:153: 2024-06-16T08:26:33.872-1200 [INFO] creating new snapshot: path=/tmp/raft4118912225/snapshots/2-12-1718569593872.tmp -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.872-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: snapshot complete up to: index=12 - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.872-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: snapshot complete up to: index=12 - testing.go:153: 2024-06-16T08:26:33.873-1200 [INFO] creating new snapshot: path=/tmp/raft4118912225/snapshots/2-23-1718569593873.tmp -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: copied to local snapshot: bytes=71 - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: restored user snapshot: index=1 - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: restored user snapshot: index=1 -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [ERROR] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: failed to get log: index=23 error="log not found" - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [ERROR] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: aborting pipeline replication: peer="{Voter server-12bfc68f-2775-5023-31e0-b166f8e418b0 12bfc68f-2775-5023-31e0-b166f8e418b0}" - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: aborting pipeline replication: peer="{Voter server-12bfc68f-2775-5023-31e0-b166f8e418b0 12bfc68f-2775-5023-31e0-b166f8e418b0}" -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [ERROR] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: failed to get log: index=23 error="log not found" - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [ERROR] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: aborting pipeline replication: peer="{Voter server-b5071370-c7f5-11c6-8f28-fa99b8f7618e b5071370-c7f5-11c6-8f28-fa99b8f7618e}" - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.874-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: aborting pipeline replication: peer="{Voter server-b5071370-c7f5-11c6-8f28-fa99b8f7618e b5071370-c7f5-11c6-8f28-fa99b8f7618e}" -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.880-1200 [ERROR] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: failed to get log: index=23 error="log not found" - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.880-1200 [ERROR] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: failed to get log: index=23 error="log not found" - testing.go:153: 2024-06-16T08:26:33.880-1200 [INFO] creating new snapshot: path=/tmp/raft389080924/snapshots/2-23-1718569593880.tmp -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.881-1200 [ERROR] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.881-1200 [INFO] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: copied to local snapshot: bytes=71 - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.881-1200 [ERROR] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: failed to get log: index=23 error="log not found" - testing.go:149: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.881-1200 [INFO] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.881-1200 [INFO] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: Installed remote snapshot - testing.go:153: 2024-06-16T08:26:33.881-1200 [INFO] creating new snapshot: path=/tmp/raft584946946/snapshots/2-23-1718569593881.tmp - testing.go:149: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.881-1200 [INFO] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: Installed remote snapshot -testLoggerAdapter verbose: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.882-1200 [WARN] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: skipping application of old log: index=22 - testing.go:149: server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: 2024-06-16T08:26:33.882-1200 [WARN] server-b5071370-c7f5-11c6-8f28-fa99b8f7618e: skipping application of old log: index=22 -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.882-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: pipelining replication: peer="{Voter server-b5071370-c7f5-11c6-8f28-fa99b8f7618e b5071370-c7f5-11c6-8f28-fa99b8f7618e}" - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.882-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: pipelining replication: peer="{Voter server-b5071370-c7f5-11c6-8f28-fa99b8f7618e b5071370-c7f5-11c6-8f28-fa99b8f7618e}" -testLoggerAdapter verbose: server-12bfc68f-2775-5023-31e0-b166f8e418b0: 2024-06-16T08:26:33.882-1200 [INFO] server-12bfc68f-2775-5023-31e0-b166f8e418b0: copied to local snapshot: bytes=71 - testing.go:149: server-12bfc68f-2775-5023-31e0-b166f8e418b0: 2024-06-16T08:26:33.882-1200 [INFO] server-12bfc68f-2775-5023-31e0-b166f8e418b0: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-12bfc68f-2775-5023-31e0-b166f8e418b0: 2024-06-16T08:26:33.882-1200 [INFO] server-12bfc68f-2775-5023-31e0-b166f8e418b0: Installed remote snapshot - testing.go:149: server-12bfc68f-2775-5023-31e0-b166f8e418b0: 2024-06-16T08:26:33.882-1200 [INFO] server-12bfc68f-2775-5023-31e0-b166f8e418b0: Installed remote snapshot -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.883-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: pipelining replication: peer="{Voter server-12bfc68f-2775-5023-31e0-b166f8e418b0 12bfc68f-2775-5023-31e0-b166f8e418b0}" - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.883-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: pipelining replication: peer="{Voter server-12bfc68f-2775-5023-31e0-b166f8e418b0 12bfc68f-2775-5023-31e0-b166f8e418b0}" -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.899-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: aborting pipeline replication: peer="{Voter server-12bfc68f-2775-5023-31e0-b166f8e418b0 12bfc68f-2775-5023-31e0-b166f8e418b0}" - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.899-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: aborting pipeline replication: peer="{Voter server-12bfc68f-2775-5023-31e0-b166f8e418b0 12bfc68f-2775-5023-31e0-b166f8e418b0}" -testLoggerAdapter verbose: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.899-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: aborting pipeline replication: peer="{Voter server-b5071370-c7f5-11c6-8f28-fa99b8f7618e b5071370-c7f5-11c6-8f28-fa99b8f7618e}" - testing.go:149: server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: 2024-06-16T08:26:33.899-1200 [INFO] server-4eb8f8df-ef66-5fc7-5c7d-cc11e6ba6ecd: aborting pipeline replication: peer="{Voter server-b5071370-c7f5-11c6-8f28-fa99b8f7618e b5071370-c7f5-11c6-8f28-fa99b8f7618e}" -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.901-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c921841-4197-8da2-36d8-c32b894b1a29 Address:9c921841-4197-8da2-36d8-c32b894b1a29} {Suffrage:Voter ID:server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 Address:fac2cacf-c72b-be2e-7e42-eeea6ad8a487} {Suffrage:Voter ID:server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 Address:5d0d7372-098f-5f1d-2c02-e4efe3e63b76}]" - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.901-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c921841-4197-8da2-36d8-c32b894b1a29 Address:9c921841-4197-8da2-36d8-c32b894b1a29} {Suffrage:Voter ID:server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 Address:fac2cacf-c72b-be2e-7e42-eeea6ad8a487} {Suffrage:Voter ID:server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 Address:5d0d7372-098f-5f1d-2c02-e4efe3e63b76}]" -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.901-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: entering follower state: follower="Node at 9c921841-4197-8da2-36d8-c32b894b1a29 [Follower]" leader= - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.901-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: entering follower state: follower="Node at 9c921841-4197-8da2-36d8-c32b894b1a29 [Follower]" leader= -testLoggerAdapter verbose: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:33.901-1200 [INFO] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c921841-4197-8da2-36d8-c32b894b1a29 Address:9c921841-4197-8da2-36d8-c32b894b1a29} {Suffrage:Voter ID:server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 Address:fac2cacf-c72b-be2e-7e42-eeea6ad8a487} {Suffrage:Voter ID:server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 Address:5d0d7372-098f-5f1d-2c02-e4efe3e63b76}]" - testing.go:149: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:33.901-1200 [INFO] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c921841-4197-8da2-36d8-c32b894b1a29 Address:9c921841-4197-8da2-36d8-c32b894b1a29} {Suffrage:Voter ID:server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 Address:fac2cacf-c72b-be2e-7e42-eeea6ad8a487} {Suffrage:Voter ID:server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 Address:5d0d7372-098f-5f1d-2c02-e4efe3e63b76}]" -testLoggerAdapter verbose: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:33.901-1200 [INFO] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: entering follower state: follower="Node at fac2cacf-c72b-be2e-7e42-eeea6ad8a487 [Follower]" leader= - testing.go:149: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:33.901-1200 [INFO] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: entering follower state: follower="Node at fac2cacf-c72b-be2e-7e42-eeea6ad8a487 [Follower]" leader= -testLoggerAdapter verbose: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:33.902-1200 [INFO] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c921841-4197-8da2-36d8-c32b894b1a29 Address:9c921841-4197-8da2-36d8-c32b894b1a29} {Suffrage:Voter ID:server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 Address:fac2cacf-c72b-be2e-7e42-eeea6ad8a487} {Suffrage:Voter ID:server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 Address:5d0d7372-098f-5f1d-2c02-e4efe3e63b76}]" - testing.go:149: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:33.902-1200 [INFO] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c921841-4197-8da2-36d8-c32b894b1a29 Address:9c921841-4197-8da2-36d8-c32b894b1a29} {Suffrage:Voter ID:server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 Address:fac2cacf-c72b-be2e-7e42-eeea6ad8a487} {Suffrage:Voter ID:server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 Address:5d0d7372-098f-5f1d-2c02-e4efe3e63b76}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:33.902-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:33.902-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:33.902-1200 [INFO] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: entering follower state: follower="Node at 5d0d7372-098f-5f1d-2c02-e4efe3e63b76 [Follower]" leader= - testing.go:149: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:33.902-1200 [INFO] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: entering follower state: follower="Node at 5d0d7372-098f-5f1d-2c02-e4efe3e63b76 [Follower]" leader= -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [WARN] server-9c921841-4197-8da2-36d8-c32b894b1a29: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [WARN] server-9c921841-4197-8da2-36d8-c32b894b1a29: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: entering candidate state: node="Node at 9c921841-4197-8da2-36d8-c32b894b1a29 [Candidate]" term=2 - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: entering candidate state: node="Node at 9c921841-4197-8da2-36d8-c32b894b1a29 [Candidate]" term=2 -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: election won: tally=2 - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: election won: tally=2 -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: entering leader state: leader="Node at 9c921841-4197-8da2-36d8-c32b894b1a29 [Leader]" - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: entering leader state: leader="Node at 9c921841-4197-8da2-36d8-c32b894b1a29 [Leader]" -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: added peer, starting replication: peer=server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.954-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: added peer, starting replication: peer=server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.955-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: added peer, starting replication: peer=server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.955-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: added peer, starting replication: peer=server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.955-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: pipelining replication: peer="{Voter server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 fac2cacf-c72b-be2e-7e42-eeea6ad8a487}" - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.955-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: pipelining replication: peer="{Voter server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 fac2cacf-c72b-be2e-7e42-eeea6ad8a487}" -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.955-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: pipelining replication: peer="{Voter server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 5d0d7372-098f-5f1d-2c02-e4efe3e63b76}" - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:33.955-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: pipelining replication: peer="{Voter server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 5d0d7372-098f-5f1d-2c02-e4efe3e63b76}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.060-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.9548837 -1200 -12 m=+10.962200181 (1 nodes), 52.66668ms from start of poll, 53.58104ms from cluster start. Timeout at 2024-06-16 08:26:34.06018984 -1200 -12 m=+11.067506321, 105.30614ms after stability - testing.go:149: cluster: 2024-06-16T08:26:34.060-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:33.9548837 -1200 -12 m=+10.962200181 (1 nodes), 52.66668ms from start of poll, 53.58104ms from cluster start. Timeout at 2024-06-16 08:26:34.06018984 -1200 -12 m=+11.067506321, 105.30614ms after stability -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.060-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: starting snapshot up to: index=12 - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.060-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: starting snapshot up to: index=12 - testing.go:153: 2024-06-16T08:26:34.060-1200 [INFO] creating new snapshot: path=/tmp/raft4232237734/snapshots/2-12-1718569594060.tmp -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.061-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: snapshot complete up to: index=12 - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.061-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: snapshot complete up to: index=12 - testing.go:153: 2024-06-16T08:26:34.062-1200 [INFO] creating new snapshot: path=/tmp/raft4232237734/snapshots/2-23-1718569594062.tmp -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.062-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: copied to local snapshot: bytes=71 - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.062-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: restored user snapshot: index=1 - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: restored user snapshot: index=1 -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [ERROR] server-9c921841-4197-8da2-36d8-c32b894b1a29: failed to get log: index=23 error="log not found" - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [ERROR] server-9c921841-4197-8da2-36d8-c32b894b1a29: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [ERROR] server-9c921841-4197-8da2-36d8-c32b894b1a29: failed to get log: index=23 error="log not found" - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [ERROR] server-9c921841-4197-8da2-36d8-c32b894b1a29: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: aborting pipeline replication: peer="{Voter server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 5d0d7372-098f-5f1d-2c02-e4efe3e63b76}" - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: aborting pipeline replication: peer="{Voter server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 5d0d7372-098f-5f1d-2c02-e4efe3e63b76}" -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: aborting pipeline replication: peer="{Voter server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 fac2cacf-c72b-be2e-7e42-eeea6ad8a487}" - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.063-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: aborting pipeline replication: peer="{Voter server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 fac2cacf-c72b-be2e-7e42-eeea6ad8a487}" -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.072-1200 [ERROR] server-9c921841-4197-8da2-36d8-c32b894b1a29: failed to get log: index=23 error="log not found" - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.072-1200 [ERROR] server-9c921841-4197-8da2-36d8-c32b894b1a29: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.072-1200 [ERROR] server-9c921841-4197-8da2-36d8-c32b894b1a29: failed to get log: index=23 error="log not found" - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.072-1200 [ERROR] server-9c921841-4197-8da2-36d8-c32b894b1a29: failed to get log: index=23 error="log not found" - testing.go:153: 2024-06-16T08:26:34.072-1200 [INFO] creating new snapshot: path=/tmp/raft3318448802/snapshots/2-23-1718569594072.tmp - testing.go:153: 2024-06-16T08:26:34.072-1200 [INFO] creating new snapshot: path=/tmp/raft3024632065/snapshots/2-23-1718569594072.tmp -testLoggerAdapter verbose: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:34.073-1200 [INFO] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:34.073-1200 [INFO] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: copied to local snapshot: bytes=71 - testing.go:149: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:34.073-1200 [INFO] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: copied to local snapshot: bytes=71 - testing.go:149: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:34.073-1200 [INFO] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:34.073-1200 [INFO] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: Installed remote snapshot -testLoggerAdapter verbose: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:34.073-1200 [INFO] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: Installed remote snapshot - testing.go:149: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:34.073-1200 [INFO] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: Installed remote snapshot - testing.go:149: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:34.073-1200 [INFO] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: Installed remote snapshot -testLoggerAdapter verbose: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:34.073-1200 [WARN] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: skipping application of old log: index=23 - testing.go:149: server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: 2024-06-16T08:26:34.073-1200 [WARN] server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76: skipping application of old log: index=23 -testLoggerAdapter verbose: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:34.073-1200 [WARN] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: skipping application of old log: index=22 -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.073-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: pipelining replication: peer="{Voter server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 5d0d7372-098f-5f1d-2c02-e4efe3e63b76}" - testing.go:149: server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: 2024-06-16T08:26:34.073-1200 [WARN] server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487: skipping application of old log: index=22 - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.073-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: pipelining replication: peer="{Voter server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 5d0d7372-098f-5f1d-2c02-e4efe3e63b76}" -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.074-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: pipelining replication: peer="{Voter server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 fac2cacf-c72b-be2e-7e42-eeea6ad8a487}" - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.074-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: pipelining replication: peer="{Voter server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 fac2cacf-c72b-be2e-7e42-eeea6ad8a487}" -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.085-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: aborting pipeline replication: peer="{Voter server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 fac2cacf-c72b-be2e-7e42-eeea6ad8a487}" - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.085-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: aborting pipeline replication: peer="{Voter server-fac2cacf-c72b-be2e-7e42-eeea6ad8a487 fac2cacf-c72b-be2e-7e42-eeea6ad8a487}" -testLoggerAdapter verbose: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.085-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: aborting pipeline replication: peer="{Voter server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 5d0d7372-098f-5f1d-2c02-e4efe3e63b76}" - testing.go:149: server-9c921841-4197-8da2-36d8-c32b894b1a29: 2024-06-16T08:26:34.085-1200 [INFO] server-9c921841-4197-8da2-36d8-c32b894b1a29: aborting pipeline replication: peer="{Voter server-5d0d7372-098f-5f1d-2c02-e4efe3e63b76 5d0d7372-098f-5f1d-2c02-e4efe3e63b76}" -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.086-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75f47187-606a-287e-3ada-fd9fdf7652db Address:75f47187-606a-287e-3ada-fd9fdf7652db} {Suffrage:Voter ID:server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 Address:748b1c07-aff2-cf9c-d4bd-4ada2dbe6116} {Suffrage:Voter ID:server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 Address:a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}]" - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.086-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75f47187-606a-287e-3ada-fd9fdf7652db Address:75f47187-606a-287e-3ada-fd9fdf7652db} {Suffrage:Voter ID:server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 Address:748b1c07-aff2-cf9c-d4bd-4ada2dbe6116} {Suffrage:Voter ID:server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 Address:a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}]" -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.086-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: entering follower state: follower="Node at 75f47187-606a-287e-3ada-fd9fdf7652db [Follower]" leader= - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.086-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: entering follower state: follower="Node at 75f47187-606a-287e-3ada-fd9fdf7652db [Follower]" leader= -testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.086-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75f47187-606a-287e-3ada-fd9fdf7652db Address:75f47187-606a-287e-3ada-fd9fdf7652db} {Suffrage:Voter ID:server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 Address:748b1c07-aff2-cf9c-d4bd-4ada2dbe6116} {Suffrage:Voter ID:server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 Address:a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}]" - testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.086-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75f47187-606a-287e-3ada-fd9fdf7652db Address:75f47187-606a-287e-3ada-fd9fdf7652db} {Suffrage:Voter ID:server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 Address:748b1c07-aff2-cf9c-d4bd-4ada2dbe6116} {Suffrage:Voter ID:server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 Address:a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}]" -testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.087-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: entering follower state: follower="Node at 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 [Follower]" leader= - testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.087-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: entering follower state: follower="Node at 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 [Follower]" leader= -testLoggerAdapter verbose: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.087-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75f47187-606a-287e-3ada-fd9fdf7652db Address:75f47187-606a-287e-3ada-fd9fdf7652db} {Suffrage:Voter ID:server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 Address:748b1c07-aff2-cf9c-d4bd-4ada2dbe6116} {Suffrage:Voter ID:server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 Address:a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}]" - testing.go:149: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.087-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75f47187-606a-287e-3ada-fd9fdf7652db Address:75f47187-606a-287e-3ada-fd9fdf7652db} {Suffrage:Voter ID:server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 Address:748b1c07-aff2-cf9c-d4bd-4ada2dbe6116} {Suffrage:Voter ID:server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 Address:a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.087-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:34.087-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.087-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: entering follower state: follower="Node at a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 [Follower]" leader= - testing.go:149: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.087-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: entering follower state: follower="Node at a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 [Follower]" leader= -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [WARN] server-75f47187-606a-287e-3ada-fd9fdf7652db: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [WARN] server-75f47187-606a-287e-3ada-fd9fdf7652db: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: entering candidate state: node="Node at 75f47187-606a-287e-3ada-fd9fdf7652db [Candidate]" term=2 -testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.161-1200 [WARN] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: entering candidate state: node="Node at 75f47187-606a-287e-3ada-fd9fdf7652db [Candidate]" term=2 - testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.161-1200 [WARN] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.161-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: entering candidate state: node="Node at 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 [Candidate]" term=2 -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: election won: tally=2 - testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.161-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: entering candidate state: node="Node at 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 [Candidate]" term=2 - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: election won: tally=2 -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: entering leader state: leader="Node at 75f47187-606a-287e-3ada-fd9fdf7652db [Leader]" -testLoggerAdapter verbose: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.161-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: duplicate requestVote for same term: term=2 - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: entering leader state: leader="Node at 75f47187-606a-287e-3ada-fd9fdf7652db [Leader]" -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: added peer, starting replication: peer=server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 -testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.161-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: duplicate requestVote for same term: term=2 - testing.go:149: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.161-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: duplicate requestVote for same term: term=2 - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: added peer, starting replication: peer=server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: added peer, starting replication: peer=server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: added peer, starting replication: peer=server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 - testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.161-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [WARN] server-75f47187-606a-287e-3ada-fd9fdf7652db: rejecting vote request since we have a leader: from=748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 leader=75f47187-606a-287e-3ada-fd9fdf7652db - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [WARN] server-75f47187-606a-287e-3ada-fd9fdf7652db: rejecting vote request since we have a leader: from=748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 leader=75f47187-606a-287e-3ada-fd9fdf7652db -testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.161-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: entering follower state: follower="Node at 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 [Follower]" leader=75f47187-606a-287e-3ada-fd9fdf7652db - testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.161-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: entering follower state: follower="Node at 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 [Follower]" leader=75f47187-606a-287e-3ada-fd9fdf7652db -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: pipelining replication: peer="{Voter server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}" - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: pipelining replication: peer="{Voter server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}" -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: pipelining replication: peer="{Voter server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116}" - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.161-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: pipelining replication: peer="{Voter server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.267-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:34.16194074 -1200 -12 m=+11.169257221 (1 nodes), 74.55118ms from start of poll, 75.44748ms from cluster start. Timeout at 2024-06-16 08:26:34.26708708 -1200 -12 m=+11.274403581, 105.14636ms after stability - testing.go:149: cluster: 2024-06-16T08:26:34.267-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:34.16194074 -1200 -12 m=+11.169257221 (1 nodes), 74.55118ms from start of poll, 75.44748ms from cluster start. Timeout at 2024-06-16 08:26:34.26708708 -1200 -12 m=+11.274403581, 105.14636ms after stability -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.267-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: starting snapshot up to: index=12 - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.267-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: starting snapshot up to: index=12 - testing.go:153: 2024-06-16T08:26:34.267-1200 [INFO] creating new snapshot: path=/tmp/raft3582674493/snapshots/2-12-1718569594267.tmp -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.268-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: snapshot complete up to: index=12 - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.268-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: snapshot complete up to: index=12 - testing.go:153: 2024-06-16T08:26:34.269-1200 [INFO] creating new snapshot: path=/tmp/raft3582674493/snapshots/2-23-1718569594269.tmp -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.269-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: copied to local snapshot: bytes=71 - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.269-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: restored user snapshot: index=1 - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: restored user snapshot: index=1 -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [ERROR] server-75f47187-606a-287e-3ada-fd9fdf7652db: failed to get log: index=23 error="log not found" - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [ERROR] server-75f47187-606a-287e-3ada-fd9fdf7652db: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [ERROR] server-75f47187-606a-287e-3ada-fd9fdf7652db: failed to get log: index=23 error="log not found" - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [ERROR] server-75f47187-606a-287e-3ada-fd9fdf7652db: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: aborting pipeline replication: peer="{Voter server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116}" - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: aborting pipeline replication: peer="{Voter server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116}" -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: aborting pipeline replication: peer="{Voter server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}" - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.270-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: aborting pipeline replication: peer="{Voter server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}" -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.276-1200 [ERROR] server-75f47187-606a-287e-3ada-fd9fdf7652db: failed to get log: index=23 error="log not found" - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.276-1200 [ERROR] server-75f47187-606a-287e-3ada-fd9fdf7652db: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.277-1200 [ERROR] server-75f47187-606a-287e-3ada-fd9fdf7652db: failed to get log: index=23 error="log not found" - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.277-1200 [ERROR] server-75f47187-606a-287e-3ada-fd9fdf7652db: failed to get log: index=23 error="log not found" - testing.go:153: 2024-06-16T08:26:34.277-1200 [INFO] creating new snapshot: path=/tmp/raft53509326/snapshots/2-23-1718569594277.tmp - testing.go:153: 2024-06-16T08:26:34.277-1200 [INFO] creating new snapshot: path=/tmp/raft1487043319/snapshots/2-23-1718569594277.tmp -testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.277-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: copied to local snapshot: bytes=71 - testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.277-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.278-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.278-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: Installed remote snapshot - testing.go:149: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.278-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: copied to local snapshot: bytes=71 - testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.278-1200 [INFO] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: Installed remote snapshot -testLoggerAdapter verbose: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.278-1200 [WARN] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: skipping application of old log: index=22 - testing.go:149: server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: 2024-06-16T08:26:34.278-1200 [WARN] server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116: skipping application of old log: index=22 -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.278-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: pipelining replication: peer="{Voter server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116}" -testLoggerAdapter verbose: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.278-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: Installed remote snapshot - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.278-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: pipelining replication: peer="{Voter server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116}" - testing.go:149: server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: 2024-06-16T08:26:34.278-1200 [INFO] server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792: Installed remote snapshot -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.278-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: pipelining replication: peer="{Voter server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}" - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.278-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: pipelining replication: peer="{Voter server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}" -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.289-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: aborting pipeline replication: peer="{Voter server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}" - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.289-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: aborting pipeline replication: peer="{Voter server-a4a24ac4-334f-8a3a-61a9-3e20cbbf3792 a4a24ac4-334f-8a3a-61a9-3e20cbbf3792}" -testLoggerAdapter verbose: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.289-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: aborting pipeline replication: peer="{Voter server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116}" - testing.go:149: server-75f47187-606a-287e-3ada-fd9fdf7652db: 2024-06-16T08:26:34.289-1200 [INFO] server-75f47187-606a-287e-3ada-fd9fdf7652db: aborting pipeline replication: peer="{Voter server-748b1c07-aff2-cf9c-d4bd-4ada2dbe6116 748b1c07-aff2-cf9c-d4bd-4ada2dbe6116}" -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.291-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d368e76-e271-64e5-4534-ce7ef1c3b813 Address:0d368e76-e271-64e5-4534-ce7ef1c3b813} {Suffrage:Voter ID:server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 Address:8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72} {Suffrage:Voter ID:server-f8a9cce6-167c-32ae-626d-909270ad700e Address:f8a9cce6-167c-32ae-626d-909270ad700e}]" - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.291-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d368e76-e271-64e5-4534-ce7ef1c3b813 Address:0d368e76-e271-64e5-4534-ce7ef1c3b813} {Suffrage:Voter ID:server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 Address:8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72} {Suffrage:Voter ID:server-f8a9cce6-167c-32ae-626d-909270ad700e Address:f8a9cce6-167c-32ae-626d-909270ad700e}]" -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.291-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: entering follower state: follower="Node at 0d368e76-e271-64e5-4534-ce7ef1c3b813 [Follower]" leader= - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.291-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: entering follower state: follower="Node at 0d368e76-e271-64e5-4534-ce7ef1c3b813 [Follower]" leader= -testLoggerAdapter verbose: server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: 2024-06-16T08:26:34.291-1200 [INFO] server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d368e76-e271-64e5-4534-ce7ef1c3b813 Address:0d368e76-e271-64e5-4534-ce7ef1c3b813} {Suffrage:Voter ID:server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 Address:8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72} {Suffrage:Voter ID:server-f8a9cce6-167c-32ae-626d-909270ad700e Address:f8a9cce6-167c-32ae-626d-909270ad700e}]" - testing.go:149: server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: 2024-06-16T08:26:34.291-1200 [INFO] server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d368e76-e271-64e5-4534-ce7ef1c3b813 Address:0d368e76-e271-64e5-4534-ce7ef1c3b813} {Suffrage:Voter ID:server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 Address:8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72} {Suffrage:Voter ID:server-f8a9cce6-167c-32ae-626d-909270ad700e Address:f8a9cce6-167c-32ae-626d-909270ad700e}]" -testLoggerAdapter verbose: server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: 2024-06-16T08:26:34.291-1200 [INFO] server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: entering follower state: follower="Node at 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 [Follower]" leader= - testing.go:149: server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: 2024-06-16T08:26:34.291-1200 [INFO] server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: entering follower state: follower="Node at 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 [Follower]" leader= -testLoggerAdapter verbose: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.291-1200 [INFO] server-f8a9cce6-167c-32ae-626d-909270ad700e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d368e76-e271-64e5-4534-ce7ef1c3b813 Address:0d368e76-e271-64e5-4534-ce7ef1c3b813} {Suffrage:Voter ID:server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 Address:8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72} {Suffrage:Voter ID:server-f8a9cce6-167c-32ae-626d-909270ad700e Address:f8a9cce6-167c-32ae-626d-909270ad700e}]" - testing.go:149: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.291-1200 [INFO] server-f8a9cce6-167c-32ae-626d-909270ad700e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d368e76-e271-64e5-4534-ce7ef1c3b813 Address:0d368e76-e271-64e5-4534-ce7ef1c3b813} {Suffrage:Voter ID:server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 Address:8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72} {Suffrage:Voter ID:server-f8a9cce6-167c-32ae-626d-909270ad700e Address:f8a9cce6-167c-32ae-626d-909270ad700e}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.292-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:34.292-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.292-1200 [INFO] server-f8a9cce6-167c-32ae-626d-909270ad700e: entering follower state: follower="Node at f8a9cce6-167c-32ae-626d-909270ad700e [Follower]" leader= - testing.go:149: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.292-1200 [INFO] server-f8a9cce6-167c-32ae-626d-909270ad700e: entering follower state: follower="Node at f8a9cce6-167c-32ae-626d-909270ad700e [Follower]" leader= -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [WARN] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [WARN] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: entering candidate state: node="Node at 0d368e76-e271-64e5-4534-ce7ef1c3b813 [Candidate]" term=2 - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: entering candidate state: node="Node at 0d368e76-e271-64e5-4534-ce7ef1c3b813 [Candidate]" term=2 -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: election won: tally=2 - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: election won: tally=2 -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: entering leader state: leader="Node at 0d368e76-e271-64e5-4534-ce7ef1c3b813 [Leader]" - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: entering leader state: leader="Node at 0d368e76-e271-64e5-4534-ce7ef1c3b813 [Leader]" -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: added peer, starting replication: peer=server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: added peer, starting replication: peer=server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: added peer, starting replication: peer=server-f8a9cce6-167c-32ae-626d-909270ad700e - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: added peer, starting replication: peer=server-f8a9cce6-167c-32ae-626d-909270ad700e -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: pipelining replication: peer="{Voter server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72}" - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: pipelining replication: peer="{Voter server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72}" -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: pipelining replication: peer="{Voter server-f8a9cce6-167c-32ae-626d-909270ad700e f8a9cce6-167c-32ae-626d-909270ad700e}" - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.343-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: pipelining replication: peer="{Voter server-f8a9cce6-167c-32ae-626d-909270ad700e f8a9cce6-167c-32ae-626d-909270ad700e}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.449-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:34.34359024 -1200 -12 m=+11.350906721 (1 nodes), 51.51246ms from start of poll, 52.41596ms from cluster start. Timeout at 2024-06-16 08:26:34.44952492 -1200 -12 m=+11.456841401, 105.93468ms after stability - testing.go:149: cluster: 2024-06-16T08:26:34.449-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:34.34359024 -1200 -12 m=+11.350906721 (1 nodes), 51.51246ms from start of poll, 52.41596ms from cluster start. Timeout at 2024-06-16 08:26:34.44952492 -1200 -12 m=+11.456841401, 105.93468ms after stability -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.450-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: starting snapshot up to: index=12 - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.450-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: starting snapshot up to: index=12 - testing.go:153: 2024-06-16T08:26:34.450-1200 [INFO] creating new snapshot: path=/tmp/raft2487656942/snapshots/2-12-1718569594450.tmp -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.450-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: snapshot complete up to: index=12 - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.450-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: snapshot complete up to: index=12 - testing.go:153: 2024-06-16T08:26:34.451-1200 [INFO] creating new snapshot: path=/tmp/raft2487656942/snapshots/2-113-1718569594451.tmp -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: copied to local snapshot: bytes=71 - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: restored user snapshot: index=1 - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: restored user snapshot: index=1 -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [ERROR] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: failed to get log: index=23 error="log not found" - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [ERROR] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [ERROR] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: failed to get log: index=23 error="log not found" - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [ERROR] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: aborting pipeline replication: peer="{Voter server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72}" - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: aborting pipeline replication: peer="{Voter server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72}" -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: aborting pipeline replication: peer="{Voter server-f8a9cce6-167c-32ae-626d-909270ad700e f8a9cce6-167c-32ae-626d-909270ad700e}" - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.452-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: aborting pipeline replication: peer="{Voter server-f8a9cce6-167c-32ae-626d-909270ad700e f8a9cce6-167c-32ae-626d-909270ad700e}" -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.459-1200 [ERROR] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: failed to get log: index=23 error="log not found" - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.459-1200 [ERROR] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: failed to get log: index=23 error="log not found" - testing.go:153: 2024-06-16T08:26:34.459-1200 [INFO] creating new snapshot: path=/tmp/raft2616903032/snapshots/2-113-1718569594459.tmp -testLoggerAdapter verbose: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.460-1200 [INFO] server-f8a9cce6-167c-32ae-626d-909270ad700e: copied to local snapshot: bytes=71 - testing.go:149: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.460-1200 [INFO] server-f8a9cce6-167c-32ae-626d-909270ad700e: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.460-1200 [INFO] server-f8a9cce6-167c-32ae-626d-909270ad700e: Installed remote snapshot - testing.go:149: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.460-1200 [INFO] server-f8a9cce6-167c-32ae-626d-909270ad700e: Installed remote snapshot -testLoggerAdapter verbose: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.460-1200 [WARN] server-f8a9cce6-167c-32ae-626d-909270ad700e: skipping application of old log: index=22 - testing.go:149: server-f8a9cce6-167c-32ae-626d-909270ad700e: 2024-06-16T08:26:34.460-1200 [WARN] server-f8a9cce6-167c-32ae-626d-909270ad700e: skipping application of old log: index=22 -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.460-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: pipelining replication: peer="{Voter server-f8a9cce6-167c-32ae-626d-909270ad700e f8a9cce6-167c-32ae-626d-909270ad700e}" - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.460-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: pipelining replication: peer="{Voter server-f8a9cce6-167c-32ae-626d-909270ad700e f8a9cce6-167c-32ae-626d-909270ad700e}" -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.462-1200 [ERROR] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: failed to get log: index=23 error="log not found" - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.462-1200 [ERROR] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: failed to get log: index=23 error="log not found" - testing.go:153: 2024-06-16T08:26:34.462-1200 [INFO] creating new snapshot: path=/tmp/raft783977926/snapshots/2-113-1718569594462.tmp -testLoggerAdapter verbose: server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: 2024-06-16T08:26:34.463-1200 [INFO] server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: copied to local snapshot: bytes=71 - testing.go:149: server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: 2024-06-16T08:26:34.463-1200 [INFO] server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: 2024-06-16T08:26:34.463-1200 [INFO] server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: Installed remote snapshot - testing.go:149: server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: 2024-06-16T08:26:34.463-1200 [INFO] server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72: Installed remote snapshot -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.463-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: pipelining replication: peer="{Voter server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72}" - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.463-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: pipelining replication: peer="{Voter server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72}" -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.477-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: aborting pipeline replication: peer="{Voter server-f8a9cce6-167c-32ae-626d-909270ad700e f8a9cce6-167c-32ae-626d-909270ad700e}" - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.477-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: aborting pipeline replication: peer="{Voter server-f8a9cce6-167c-32ae-626d-909270ad700e f8a9cce6-167c-32ae-626d-909270ad700e}" -testLoggerAdapter verbose: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.477-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: aborting pipeline replication: peer="{Voter server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72}" - testing.go:149: server-0d368e76-e271-64e5-4534-ce7ef1c3b813: 2024-06-16T08:26:34.477-1200 [INFO] server-0d368e76-e271-64e5-4534-ce7ef1c3b813: aborting pipeline replication: peer="{Voter server-8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72 8d01f1e3-0d9e-5b2f-8c65-beeb518cbd72}" -testLoggerAdapter verbose: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.479-1200 [INFO] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 Address:ccb1cd15-a369-56a9-bd89-ee3eca03a529} {Suffrage:Voter ID:server-f921f1b9-52d8-92ec-d031-2021083efbc1 Address:f921f1b9-52d8-92ec-d031-2021083efbc1} {Suffrage:Voter ID:server-20f55512-047a-c61f-7524-298ee940e3f5 Address:20f55512-047a-c61f-7524-298ee940e3f5}]" - testing.go:149: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.479-1200 [INFO] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 Address:ccb1cd15-a369-56a9-bd89-ee3eca03a529} {Suffrage:Voter ID:server-f921f1b9-52d8-92ec-d031-2021083efbc1 Address:f921f1b9-52d8-92ec-d031-2021083efbc1} {Suffrage:Voter ID:server-20f55512-047a-c61f-7524-298ee940e3f5 Address:20f55512-047a-c61f-7524-298ee940e3f5}]" -testLoggerAdapter verbose: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.479-1200 [INFO] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: entering follower state: follower="Node at ccb1cd15-a369-56a9-bd89-ee3eca03a529 [Follower]" leader= - testing.go:149: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.479-1200 [INFO] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: entering follower state: follower="Node at ccb1cd15-a369-56a9-bd89-ee3eca03a529 [Follower]" leader= -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.479-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 Address:ccb1cd15-a369-56a9-bd89-ee3eca03a529} {Suffrage:Voter ID:server-f921f1b9-52d8-92ec-d031-2021083efbc1 Address:f921f1b9-52d8-92ec-d031-2021083efbc1} {Suffrage:Voter ID:server-20f55512-047a-c61f-7524-298ee940e3f5 Address:20f55512-047a-c61f-7524-298ee940e3f5}]" - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.479-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 Address:ccb1cd15-a369-56a9-bd89-ee3eca03a529} {Suffrage:Voter ID:server-f921f1b9-52d8-92ec-d031-2021083efbc1 Address:f921f1b9-52d8-92ec-d031-2021083efbc1} {Suffrage:Voter ID:server-20f55512-047a-c61f-7524-298ee940e3f5 Address:20f55512-047a-c61f-7524-298ee940e3f5}]" -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.479-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: entering follower state: follower="Node at f921f1b9-52d8-92ec-d031-2021083efbc1 [Follower]" leader= - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.479-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: entering follower state: follower="Node at f921f1b9-52d8-92ec-d031-2021083efbc1 [Follower]" leader= -testLoggerAdapter verbose: server-20f55512-047a-c61f-7524-298ee940e3f5: 2024-06-16T08:26:34.479-1200 [INFO] server-20f55512-047a-c61f-7524-298ee940e3f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 Address:ccb1cd15-a369-56a9-bd89-ee3eca03a529} {Suffrage:Voter ID:server-f921f1b9-52d8-92ec-d031-2021083efbc1 Address:f921f1b9-52d8-92ec-d031-2021083efbc1} {Suffrage:Voter ID:server-20f55512-047a-c61f-7524-298ee940e3f5 Address:20f55512-047a-c61f-7524-298ee940e3f5}]" - testing.go:149: server-20f55512-047a-c61f-7524-298ee940e3f5: 2024-06-16T08:26:34.479-1200 [INFO] server-20f55512-047a-c61f-7524-298ee940e3f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 Address:ccb1cd15-a369-56a9-bd89-ee3eca03a529} {Suffrage:Voter ID:server-f921f1b9-52d8-92ec-d031-2021083efbc1 Address:f921f1b9-52d8-92ec-d031-2021083efbc1} {Suffrage:Voter ID:server-20f55512-047a-c61f-7524-298ee940e3f5 Address:20f55512-047a-c61f-7524-298ee940e3f5}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.479-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:34.479-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-20f55512-047a-c61f-7524-298ee940e3f5: 2024-06-16T08:26:34.479-1200 [INFO] server-20f55512-047a-c61f-7524-298ee940e3f5: entering follower state: follower="Node at 20f55512-047a-c61f-7524-298ee940e3f5 [Follower]" leader= - testing.go:149: server-20f55512-047a-c61f-7524-298ee940e3f5: 2024-06-16T08:26:34.479-1200 [INFO] server-20f55512-047a-c61f-7524-298ee940e3f5: entering follower state: follower="Node at 20f55512-047a-c61f-7524-298ee940e3f5 [Follower]" leader= -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [WARN] server-f921f1b9-52d8-92ec-d031-2021083efbc1: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [WARN] server-f921f1b9-52d8-92ec-d031-2021083efbc1: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: entering candidate state: node="Node at f921f1b9-52d8-92ec-d031-2021083efbc1 [Candidate]" term=2 - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: entering candidate state: node="Node at f921f1b9-52d8-92ec-d031-2021083efbc1 [Candidate]" term=2 -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: election won: tally=2 - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: election won: tally=2 -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: entering leader state: leader="Node at f921f1b9-52d8-92ec-d031-2021083efbc1 [Leader]" - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: entering leader state: leader="Node at f921f1b9-52d8-92ec-d031-2021083efbc1 [Leader]" -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: added peer, starting replication: peer=server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: added peer, starting replication: peer=server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: added peer, starting replication: peer=server-20f55512-047a-c61f-7524-298ee940e3f5 - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: added peer, starting replication: peer=server-20f55512-047a-c61f-7524-298ee940e3f5 -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: pipelining replication: peer="{Voter server-20f55512-047a-c61f-7524-298ee940e3f5 20f55512-047a-c61f-7524-298ee940e3f5}" - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: pipelining replication: peer="{Voter server-20f55512-047a-c61f-7524-298ee940e3f5 20f55512-047a-c61f-7524-298ee940e3f5}" -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: pipelining replication: peer="{Voter server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 ccb1cd15-a369-56a9-bd89-ee3eca03a529}" - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.551-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: pipelining replication: peer="{Voter server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 ccb1cd15-a369-56a9-bd89-ee3eca03a529}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.656-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:34.55144138 -1200 -12 m=+11.558757861 (1 nodes), 71.7895ms from start of poll, 72.5549ms from cluster start. Timeout at 2024-06-16 08:26:34.65662068 -1200 -12 m=+11.663937161, 105.1793ms after stability - testing.go:149: cluster: 2024-06-16T08:26:34.656-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:34.55144138 -1200 -12 m=+11.558757861 (1 nodes), 71.7895ms from start of poll, 72.5549ms from cluster start. Timeout at 2024-06-16 08:26:34.65662068 -1200 -12 m=+11.663937161, 105.1793ms after stability -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.657-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: starting snapshot up to: index=12 - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.657-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: starting snapshot up to: index=12 - testing.go:153: 2024-06-16T08:26:34.657-1200 [INFO] creating new snapshot: path=/tmp/raft3519587632/snapshots/2-12-1718569594657.tmp -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.658-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: snapshot complete up to: index=12 - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.658-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: snapshot complete up to: index=12 - testing.go:153: 2024-06-16T08:26:34.658-1200 [INFO] creating new snapshot: path=/tmp/raft3519587632/snapshots/2-1013-1718569594658.tmp -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: copied to local snapshot: bytes=71 - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: restored user snapshot: index=1 - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: restored user snapshot: index=1 -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [ERROR] server-f921f1b9-52d8-92ec-d031-2021083efbc1: failed to get log: index=23 error="log not found" - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [ERROR] server-f921f1b9-52d8-92ec-d031-2021083efbc1: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [ERROR] server-f921f1b9-52d8-92ec-d031-2021083efbc1: failed to get log: index=23 error="log not found" - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [ERROR] server-f921f1b9-52d8-92ec-d031-2021083efbc1: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: aborting pipeline replication: peer="{Voter server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 ccb1cd15-a369-56a9-bd89-ee3eca03a529}" - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: aborting pipeline replication: peer="{Voter server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 ccb1cd15-a369-56a9-bd89-ee3eca03a529}" -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: aborting pipeline replication: peer="{Voter server-20f55512-047a-c61f-7524-298ee940e3f5 20f55512-047a-c61f-7524-298ee940e3f5}" - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.659-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: aborting pipeline replication: peer="{Voter server-20f55512-047a-c61f-7524-298ee940e3f5 20f55512-047a-c61f-7524-298ee940e3f5}" -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.665-1200 [ERROR] server-f921f1b9-52d8-92ec-d031-2021083efbc1: failed to get log: index=23 error="log not found" - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.665-1200 [ERROR] server-f921f1b9-52d8-92ec-d031-2021083efbc1: failed to get log: index=23 error="log not found" - testing.go:153: 2024-06-16T08:26:34.665-1200 [INFO] creating new snapshot: path=/tmp/raft3103839251/snapshots/2-1013-1718569594665.tmp -testLoggerAdapter verbose: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.666-1200 [INFO] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: copied to local snapshot: bytes=71 - testing.go:149: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.666-1200 [INFO] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.666-1200 [INFO] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: Installed remote snapshot - testing.go:149: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.666-1200 [INFO] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: Installed remote snapshot -testLoggerAdapter verbose: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.666-1200 [WARN] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: skipping application of old log: index=22 - testing.go:149: server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: 2024-06-16T08:26:34.666-1200 [WARN] server-ccb1cd15-a369-56a9-bd89-ee3eca03a529: skipping application of old log: index=22 -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.666-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: pipelining replication: peer="{Voter server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 ccb1cd15-a369-56a9-bd89-ee3eca03a529}" - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.666-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: pipelining replication: peer="{Voter server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 ccb1cd15-a369-56a9-bd89-ee3eca03a529}" -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.667-1200 [ERROR] server-f921f1b9-52d8-92ec-d031-2021083efbc1: failed to get log: index=23 error="log not found" - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.667-1200 [ERROR] server-f921f1b9-52d8-92ec-d031-2021083efbc1: failed to get log: index=23 error="log not found" - testing.go:153: 2024-06-16T08:26:34.668-1200 [INFO] creating new snapshot: path=/tmp/raft3458475960/snapshots/2-1013-1718569594668.tmp -testLoggerAdapter verbose: server-20f55512-047a-c61f-7524-298ee940e3f5: 2024-06-16T08:26:34.668-1200 [INFO] server-20f55512-047a-c61f-7524-298ee940e3f5: copied to local snapshot: bytes=71 - testing.go:149: server-20f55512-047a-c61f-7524-298ee940e3f5: 2024-06-16T08:26:34.668-1200 [INFO] server-20f55512-047a-c61f-7524-298ee940e3f5: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-20f55512-047a-c61f-7524-298ee940e3f5: 2024-06-16T08:26:34.669-1200 [INFO] server-20f55512-047a-c61f-7524-298ee940e3f5: Installed remote snapshot - testing.go:149: server-20f55512-047a-c61f-7524-298ee940e3f5: 2024-06-16T08:26:34.669-1200 [INFO] server-20f55512-047a-c61f-7524-298ee940e3f5: Installed remote snapshot -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.669-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: pipelining replication: peer="{Voter server-20f55512-047a-c61f-7524-298ee940e3f5 20f55512-047a-c61f-7524-298ee940e3f5}" - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.669-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: pipelining replication: peer="{Voter server-20f55512-047a-c61f-7524-298ee940e3f5 20f55512-047a-c61f-7524-298ee940e3f5}" -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.683-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: aborting pipeline replication: peer="{Voter server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 ccb1cd15-a369-56a9-bd89-ee3eca03a529}" - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.683-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: aborting pipeline replication: peer="{Voter server-ccb1cd15-a369-56a9-bd89-ee3eca03a529 ccb1cd15-a369-56a9-bd89-ee3eca03a529}" -testLoggerAdapter verbose: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.683-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: aborting pipeline replication: peer="{Voter server-20f55512-047a-c61f-7524-298ee940e3f5 20f55512-047a-c61f-7524-298ee940e3f5}" - testing.go:149: server-f921f1b9-52d8-92ec-d031-2021083efbc1: 2024-06-16T08:26:34.683-1200 [INFO] server-f921f1b9-52d8-92ec-d031-2021083efbc1: aborting pipeline replication: peer="{Voter server-20f55512-047a-c61f-7524-298ee940e3f5 20f55512-047a-c61f-7524-298ee940e3f5}" -testLoggerAdapter verbose: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.685-1200 [INFO] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a58ac0e2-2965-4558-48c1-70c75bd2e518 Address:a58ac0e2-2965-4558-48c1-70c75bd2e518} {Suffrage:Voter ID:server-d5c12e9f-fc69-9895-05c2-b6003485ec6f Address:d5c12e9f-fc69-9895-05c2-b6003485ec6f} {Suffrage:Voter ID:server-080b15d9-535a-359f-f790-4c3c8ad340ac Address:080b15d9-535a-359f-f790-4c3c8ad340ac}]" - testing.go:149: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.685-1200 [INFO] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a58ac0e2-2965-4558-48c1-70c75bd2e518 Address:a58ac0e2-2965-4558-48c1-70c75bd2e518} {Suffrage:Voter ID:server-d5c12e9f-fc69-9895-05c2-b6003485ec6f Address:d5c12e9f-fc69-9895-05c2-b6003485ec6f} {Suffrage:Voter ID:server-080b15d9-535a-359f-f790-4c3c8ad340ac Address:080b15d9-535a-359f-f790-4c3c8ad340ac}]" -testLoggerAdapter verbose: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.685-1200 [INFO] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: entering follower state: follower="Node at a58ac0e2-2965-4558-48c1-70c75bd2e518 [Follower]" leader= - testing.go:149: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.685-1200 [INFO] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: entering follower state: follower="Node at a58ac0e2-2965-4558-48c1-70c75bd2e518 [Follower]" leader= -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.685-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a58ac0e2-2965-4558-48c1-70c75bd2e518 Address:a58ac0e2-2965-4558-48c1-70c75bd2e518} {Suffrage:Voter ID:server-d5c12e9f-fc69-9895-05c2-b6003485ec6f Address:d5c12e9f-fc69-9895-05c2-b6003485ec6f} {Suffrage:Voter ID:server-080b15d9-535a-359f-f790-4c3c8ad340ac Address:080b15d9-535a-359f-f790-4c3c8ad340ac}]" - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.685-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a58ac0e2-2965-4558-48c1-70c75bd2e518 Address:a58ac0e2-2965-4558-48c1-70c75bd2e518} {Suffrage:Voter ID:server-d5c12e9f-fc69-9895-05c2-b6003485ec6f Address:d5c12e9f-fc69-9895-05c2-b6003485ec6f} {Suffrage:Voter ID:server-080b15d9-535a-359f-f790-4c3c8ad340ac Address:080b15d9-535a-359f-f790-4c3c8ad340ac}]" -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.685-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: entering follower state: follower="Node at d5c12e9f-fc69-9895-05c2-b6003485ec6f [Follower]" leader= - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.685-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: entering follower state: follower="Node at d5c12e9f-fc69-9895-05c2-b6003485ec6f [Follower]" leader= -testLoggerAdapter verbose: server-080b15d9-535a-359f-f790-4c3c8ad340ac: 2024-06-16T08:26:34.685-1200 [INFO] server-080b15d9-535a-359f-f790-4c3c8ad340ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a58ac0e2-2965-4558-48c1-70c75bd2e518 Address:a58ac0e2-2965-4558-48c1-70c75bd2e518} {Suffrage:Voter ID:server-d5c12e9f-fc69-9895-05c2-b6003485ec6f Address:d5c12e9f-fc69-9895-05c2-b6003485ec6f} {Suffrage:Voter ID:server-080b15d9-535a-359f-f790-4c3c8ad340ac Address:080b15d9-535a-359f-f790-4c3c8ad340ac}]" - testing.go:149: server-080b15d9-535a-359f-f790-4c3c8ad340ac: 2024-06-16T08:26:34.685-1200 [INFO] server-080b15d9-535a-359f-f790-4c3c8ad340ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a58ac0e2-2965-4558-48c1-70c75bd2e518 Address:a58ac0e2-2965-4558-48c1-70c75bd2e518} {Suffrage:Voter ID:server-d5c12e9f-fc69-9895-05c2-b6003485ec6f Address:d5c12e9f-fc69-9895-05c2-b6003485ec6f} {Suffrage:Voter ID:server-080b15d9-535a-359f-f790-4c3c8ad340ac Address:080b15d9-535a-359f-f790-4c3c8ad340ac}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.685-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:34.685-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-080b15d9-535a-359f-f790-4c3c8ad340ac: 2024-06-16T08:26:34.685-1200 [INFO] server-080b15d9-535a-359f-f790-4c3c8ad340ac: entering follower state: follower="Node at 080b15d9-535a-359f-f790-4c3c8ad340ac [Follower]" leader= - testing.go:149: server-080b15d9-535a-359f-f790-4c3c8ad340ac: 2024-06-16T08:26:34.685-1200 [INFO] server-080b15d9-535a-359f-f790-4c3c8ad340ac: entering follower state: follower="Node at 080b15d9-535a-359f-f790-4c3c8ad340ac [Follower]" leader= -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.756-1200 [WARN] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.756-1200 [WARN] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.756-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: entering candidate state: node="Node at d5c12e9f-fc69-9895-05c2-b6003485ec6f [Candidate]" term=2 - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.756-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: entering candidate state: node="Node at d5c12e9f-fc69-9895-05c2-b6003485ec6f [Candidate]" term=2 -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: election won: tally=2 - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: election won: tally=2 -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: entering leader state: leader="Node at d5c12e9f-fc69-9895-05c2-b6003485ec6f [Leader]" - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: entering leader state: leader="Node at d5c12e9f-fc69-9895-05c2-b6003485ec6f [Leader]" -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: added peer, starting replication: peer=server-a58ac0e2-2965-4558-48c1-70c75bd2e518 - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: added peer, starting replication: peer=server-a58ac0e2-2965-4558-48c1-70c75bd2e518 -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: added peer, starting replication: peer=server-080b15d9-535a-359f-f790-4c3c8ad340ac - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: added peer, starting replication: peer=server-080b15d9-535a-359f-f790-4c3c8ad340ac -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: pipelining replication: peer="{Voter server-080b15d9-535a-359f-f790-4c3c8ad340ac 080b15d9-535a-359f-f790-4c3c8ad340ac}" - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.757-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: pipelining replication: peer="{Voter server-080b15d9-535a-359f-f790-4c3c8ad340ac 080b15d9-535a-359f-f790-4c3c8ad340ac}" -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.758-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: pipelining replication: peer="{Voter server-a58ac0e2-2965-4558-48c1-70c75bd2e518 a58ac0e2-2965-4558-48c1-70c75bd2e518}" - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.758-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: pipelining replication: peer="{Voter server-a58ac0e2-2965-4558-48c1-70c75bd2e518 a58ac0e2-2965-4558-48c1-70c75bd2e518}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.863-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:34.75729106 -1200 -12 m=+11.764607521 (1 nodes), 71.5385ms from start of poll, 72.31258ms from cluster start. Timeout at 2024-06-16 08:26:34.8632906 -1200 -12 m=+11.870607081, 105.99956ms after stability - testing.go:149: cluster: 2024-06-16T08:26:34.863-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:34.75729106 -1200 -12 m=+11.764607521 (1 nodes), 71.5385ms from start of poll, 72.31258ms from cluster start. Timeout at 2024-06-16 08:26:34.8632906 -1200 -12 m=+11.870607081, 105.99956ms after stability -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.864-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: starting snapshot up to: index=12 - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.864-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: starting snapshot up to: index=12 - testing.go:153: 2024-06-16T08:26:34.864-1200 [INFO] creating new snapshot: path=/tmp/raft3326532870/snapshots/2-12-1718569594864.tmp -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.864-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: snapshot complete up to: index=12 - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.864-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: snapshot complete up to: index=12 - testing.go:153: 2024-06-16T08:26:34.865-1200 [INFO] creating new snapshot: path=/tmp/raft3326532870/snapshots/2-10013-1718569594865.tmp -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: copied to local snapshot: bytes=71 - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: restored user snapshot: index=1 - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: restored user snapshot: index=1 -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [ERROR] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: failed to get log: index=23 error="log not found" - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [ERROR] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [ERROR] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: failed to get log: index=23 error="log not found" - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [ERROR] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: aborting pipeline replication: peer="{Voter server-080b15d9-535a-359f-f790-4c3c8ad340ac 080b15d9-535a-359f-f790-4c3c8ad340ac}" - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: aborting pipeline replication: peer="{Voter server-080b15d9-535a-359f-f790-4c3c8ad340ac 080b15d9-535a-359f-f790-4c3c8ad340ac}" -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: aborting pipeline replication: peer="{Voter server-a58ac0e2-2965-4558-48c1-70c75bd2e518 a58ac0e2-2965-4558-48c1-70c75bd2e518}" - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.866-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: aborting pipeline replication: peer="{Voter server-a58ac0e2-2965-4558-48c1-70c75bd2e518 a58ac0e2-2965-4558-48c1-70c75bd2e518}" -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.873-1200 [ERROR] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: failed to get log: index=23 error="log not found" - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.873-1200 [ERROR] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: failed to get log: index=23 error="log not found" - testing.go:153: 2024-06-16T08:26:34.873-1200 [INFO] creating new snapshot: path=/tmp/raft788190039/snapshots/2-10013-1718569594873.tmp -testLoggerAdapter verbose: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.874-1200 [INFO] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: copied to local snapshot: bytes=71 - testing.go:149: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.874-1200 [INFO] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.874-1200 [INFO] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: Installed remote snapshot - testing.go:149: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.874-1200 [INFO] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: Installed remote snapshot -testLoggerAdapter verbose: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.874-1200 [WARN] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: skipping application of old log: index=22 - testing.go:149: server-a58ac0e2-2965-4558-48c1-70c75bd2e518: 2024-06-16T08:26:34.874-1200 [WARN] server-a58ac0e2-2965-4558-48c1-70c75bd2e518: skipping application of old log: index=22 -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.874-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: pipelining replication: peer="{Voter server-a58ac0e2-2965-4558-48c1-70c75bd2e518 a58ac0e2-2965-4558-48c1-70c75bd2e518}" - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.874-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: pipelining replication: peer="{Voter server-a58ac0e2-2965-4558-48c1-70c75bd2e518 a58ac0e2-2965-4558-48c1-70c75bd2e518}" -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.876-1200 [ERROR] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: failed to get log: index=23 error="log not found" - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.876-1200 [ERROR] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: failed to get log: index=23 error="log not found" - testing.go:153: 2024-06-16T08:26:34.876-1200 [INFO] creating new snapshot: path=/tmp/raft4216072965/snapshots/2-10013-1718569594876.tmp -testLoggerAdapter verbose: server-080b15d9-535a-359f-f790-4c3c8ad340ac: 2024-06-16T08:26:34.877-1200 [INFO] server-080b15d9-535a-359f-f790-4c3c8ad340ac: copied to local snapshot: bytes=71 - testing.go:149: server-080b15d9-535a-359f-f790-4c3c8ad340ac: 2024-06-16T08:26:34.877-1200 [INFO] server-080b15d9-535a-359f-f790-4c3c8ad340ac: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-080b15d9-535a-359f-f790-4c3c8ad340ac: 2024-06-16T08:26:34.877-1200 [INFO] server-080b15d9-535a-359f-f790-4c3c8ad340ac: Installed remote snapshot - testing.go:149: server-080b15d9-535a-359f-f790-4c3c8ad340ac: 2024-06-16T08:26:34.877-1200 [INFO] server-080b15d9-535a-359f-f790-4c3c8ad340ac: Installed remote snapshot -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.877-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: pipelining replication: peer="{Voter server-080b15d9-535a-359f-f790-4c3c8ad340ac 080b15d9-535a-359f-f790-4c3c8ad340ac}" - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.877-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: pipelining replication: peer="{Voter server-080b15d9-535a-359f-f790-4c3c8ad340ac 080b15d9-535a-359f-f790-4c3c8ad340ac}" -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.891-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: aborting pipeline replication: peer="{Voter server-080b15d9-535a-359f-f790-4c3c8ad340ac 080b15d9-535a-359f-f790-4c3c8ad340ac}" - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.891-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: aborting pipeline replication: peer="{Voter server-080b15d9-535a-359f-f790-4c3c8ad340ac 080b15d9-535a-359f-f790-4c3c8ad340ac}" -testLoggerAdapter verbose: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.891-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: aborting pipeline replication: peer="{Voter server-a58ac0e2-2965-4558-48c1-70c75bd2e518 a58ac0e2-2965-4558-48c1-70c75bd2e518}" - testing.go:149: server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: 2024-06-16T08:26:34.891-1200 [INFO] server-d5c12e9f-fc69-9895-05c2-b6003485ec6f: aborting pipeline replication: peer="{Voter server-a58ac0e2-2965-4558-48c1-70c75bd2e518 a58ac0e2-2965-4558-48c1-70c75bd2e518}" +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.161+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 Address:7c992213-0f6b-dd6d-c5ac-0a2096b1bf06} {Suffrage:Voter ID:server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 Address:f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72} {Suffrage:Voter ID:server-594c8d6b-ac40-b588-ec84-4542ea29e002 Address:594c8d6b-ac40-b588-ec84-4542ea29e002}]" + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.161+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 Address:7c992213-0f6b-dd6d-c5ac-0a2096b1bf06} {Suffrage:Voter ID:server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 Address:f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72} {Suffrage:Voter ID:server-594c8d6b-ac40-b588-ec84-4542ea29e002 Address:594c8d6b-ac40-b588-ec84-4542ea29e002}]" +testLoggerAdapter verbose: server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: 2023-05-16T04:06:14.161+1400 [INFO] server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 Address:7c992213-0f6b-dd6d-c5ac-0a2096b1bf06} {Suffrage:Voter ID:server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 Address:f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72} {Suffrage:Voter ID:server-594c8d6b-ac40-b588-ec84-4542ea29e002 Address:594c8d6b-ac40-b588-ec84-4542ea29e002}]" + testing.go:149: server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: 2023-05-16T04:06:14.161+1400 [INFO] server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 Address:7c992213-0f6b-dd6d-c5ac-0a2096b1bf06} {Suffrage:Voter ID:server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 Address:f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72} {Suffrage:Voter ID:server-594c8d6b-ac40-b588-ec84-4542ea29e002 Address:594c8d6b-ac40-b588-ec84-4542ea29e002}]" +testLoggerAdapter verbose: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.161+1400 [INFO] server-594c8d6b-ac40-b588-ec84-4542ea29e002: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 Address:7c992213-0f6b-dd6d-c5ac-0a2096b1bf06} {Suffrage:Voter ID:server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 Address:f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72} {Suffrage:Voter ID:server-594c8d6b-ac40-b588-ec84-4542ea29e002 Address:594c8d6b-ac40-b588-ec84-4542ea29e002}]" + testing.go:149: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.161+1400 [INFO] server-594c8d6b-ac40-b588-ec84-4542ea29e002: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 Address:7c992213-0f6b-dd6d-c5ac-0a2096b1bf06} {Suffrage:Voter ID:server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 Address:f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72} {Suffrage:Voter ID:server-594c8d6b-ac40-b588-ec84-4542ea29e002 Address:594c8d6b-ac40-b588-ec84-4542ea29e002}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.161+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:14.161+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.161+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: entering follower state: follower="Node at 7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 [Follower]" leader= + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.161+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: entering follower state: follower="Node at 7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 [Follower]" leader= +testLoggerAdapter verbose: server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: 2023-05-16T04:06:14.162+1400 [INFO] server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: entering follower state: follower="Node at f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 [Follower]" leader= + testing.go:149: server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: 2023-05-16T04:06:14.162+1400 [INFO] server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: entering follower state: follower="Node at f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 [Follower]" leader= +testLoggerAdapter verbose: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.162+1400 [INFO] server-594c8d6b-ac40-b588-ec84-4542ea29e002: entering follower state: follower="Node at 594c8d6b-ac40-b588-ec84-4542ea29e002 [Follower]" leader= + testing.go:149: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.162+1400 [INFO] server-594c8d6b-ac40-b588-ec84-4542ea29e002: entering follower state: follower="Node at 594c8d6b-ac40-b588-ec84-4542ea29e002 [Follower]" leader= +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.216+1400 [WARN] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.216+1400 [WARN] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.216+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: entering candidate state: node="Node at 7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 [Candidate]" term=2 + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.216+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: entering candidate state: node="Node at 7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 [Candidate]" term=2 +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.216+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: election won: tally=2 + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.216+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: election won: tally=2 +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.216+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: entering leader state: leader="Node at 7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 [Leader]" + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.216+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: entering leader state: leader="Node at 7c992213-0f6b-dd6d-c5ac-0a2096b1bf06 [Leader]" +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.217+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: added peer, starting replication: peer=server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.217+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: added peer, starting replication: peer=server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.217+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: added peer, starting replication: peer=server-594c8d6b-ac40-b588-ec84-4542ea29e002 + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.217+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: added peer, starting replication: peer=server-594c8d6b-ac40-b588-ec84-4542ea29e002 +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.217+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: pipelining replication: peer="{Voter server-594c8d6b-ac40-b588-ec84-4542ea29e002 594c8d6b-ac40-b588-ec84-4542ea29e002}" + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.217+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: pipelining replication: peer="{Voter server-594c8d6b-ac40-b588-ec84-4542ea29e002 594c8d6b-ac40-b588-ec84-4542ea29e002}" +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.217+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: pipelining replication: peer="{Voter server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72}" + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.217+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: pipelining replication: peer="{Voter server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.323+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.217656603 +1400 +14 m=+11.020238218 (1 nodes), 55.752998ms from start of poll, 56.649369ms from cluster start. Timeout at 2023-05-16 04:06:14.322969866 +1400 +14 m=+11.125551501, 105.313283ms after stability + testing.go:149: cluster: 2023-05-16T04:06:14.323+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.217656603 +1400 +14 m=+11.020238218 (1 nodes), 55.752998ms from start of poll, 56.649369ms from cluster start. Timeout at 2023-05-16 04:06:14.322969866 +1400 +14 m=+11.125551501, 105.313283ms after stability +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.324+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: starting snapshot up to: index=12 + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.324+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: starting snapshot up to: index=12 + testing.go:153: 2023-05-16T04:06:14.324+1400 [INFO] creating new snapshot: path=/tmp/raft1344906792/snapshots/2-12-1684159574324.tmp +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.325+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: snapshot complete up to: index=12 + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.325+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: snapshot complete up to: index=12 + testing.go:153: 2023-05-16T04:06:14.326+1400 [INFO] creating new snapshot: path=/tmp/raft1344906792/snapshots/2-23-1684159574326.tmp +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.326+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: copied to local snapshot: bytes=71 + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.326+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: restored user snapshot: index=1 + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: restored user snapshot: index=1 +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [ERROR] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: failed to get log: index=23 error="log not found" + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [ERROR] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: aborting pipeline replication: peer="{Voter server-594c8d6b-ac40-b588-ec84-4542ea29e002 594c8d6b-ac40-b588-ec84-4542ea29e002}" + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: aborting pipeline replication: peer="{Voter server-594c8d6b-ac40-b588-ec84-4542ea29e002 594c8d6b-ac40-b588-ec84-4542ea29e002}" +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [ERROR] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: failed to get log: index=23 error="log not found" + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [ERROR] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: aborting pipeline replication: peer="{Voter server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72}" + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.327+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: aborting pipeline replication: peer="{Voter server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72}" +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.333+1400 [ERROR] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: failed to get log: index=23 error="log not found" + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.333+1400 [ERROR] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: failed to get log: index=23 error="log not found" + testing.go:153: 2023-05-16T04:06:14.334+1400 [INFO] creating new snapshot: path=/tmp/raft3505364732/snapshots/2-23-1684159574334.tmp +testLoggerAdapter verbose: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.335+1400 [INFO] server-594c8d6b-ac40-b588-ec84-4542ea29e002: copied to local snapshot: bytes=71 + testing.go:149: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.335+1400 [INFO] server-594c8d6b-ac40-b588-ec84-4542ea29e002: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.335+1400 [INFO] server-594c8d6b-ac40-b588-ec84-4542ea29e002: Installed remote snapshot + testing.go:149: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.335+1400 [INFO] server-594c8d6b-ac40-b588-ec84-4542ea29e002: Installed remote snapshot +testLoggerAdapter verbose: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.335+1400 [WARN] server-594c8d6b-ac40-b588-ec84-4542ea29e002: skipping application of old log: index=22 + testing.go:149: server-594c8d6b-ac40-b588-ec84-4542ea29e002: 2023-05-16T04:06:14.335+1400 [WARN] server-594c8d6b-ac40-b588-ec84-4542ea29e002: skipping application of old log: index=22 +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.336+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: pipelining replication: peer="{Voter server-594c8d6b-ac40-b588-ec84-4542ea29e002 594c8d6b-ac40-b588-ec84-4542ea29e002}" + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.336+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: pipelining replication: peer="{Voter server-594c8d6b-ac40-b588-ec84-4542ea29e002 594c8d6b-ac40-b588-ec84-4542ea29e002}" +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.336+1400 [ERROR] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: failed to get log: index=23 error="log not found" + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.336+1400 [ERROR] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: failed to get log: index=23 error="log not found" + testing.go:153: 2023-05-16T04:06:14.336+1400 [INFO] creating new snapshot: path=/tmp/raft3368977623/snapshots/2-23-1684159574336.tmp +testLoggerAdapter verbose: server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: 2023-05-16T04:06:14.338+1400 [INFO] server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: copied to local snapshot: bytes=71 + testing.go:149: server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: 2023-05-16T04:06:14.338+1400 [INFO] server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: 2023-05-16T04:06:14.341+1400 [INFO] server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: Installed remote snapshot + testing.go:149: server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: 2023-05-16T04:06:14.341+1400 [INFO] server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72: Installed remote snapshot +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.342+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: pipelining replication: peer="{Voter server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72}" + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.342+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: pipelining replication: peer="{Voter server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72}" +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.359+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: aborting pipeline replication: peer="{Voter server-594c8d6b-ac40-b588-ec84-4542ea29e002 594c8d6b-ac40-b588-ec84-4542ea29e002}" + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.359+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: aborting pipeline replication: peer="{Voter server-594c8d6b-ac40-b588-ec84-4542ea29e002 594c8d6b-ac40-b588-ec84-4542ea29e002}" +testLoggerAdapter verbose: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.359+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: aborting pipeline replication: peer="{Voter server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72}" + testing.go:149: server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: 2023-05-16T04:06:14.359+1400 [INFO] server-7c992213-0f6b-dd6d-c5ac-0a2096b1bf06: aborting pipeline replication: peer="{Voter server-f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72 f5be5bbb-b221-ecaa-3a7e-f8f50cf19d72}" +testLoggerAdapter verbose: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.361+1400 [INFO] server-472e3a07-d252-55b6-b13c-14eee42fc623: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-472e3a07-d252-55b6-b13c-14eee42fc623 Address:472e3a07-d252-55b6-b13c-14eee42fc623} {Suffrage:Voter ID:server-35a1d88f-5dc0-ec1f-ea16-314026fe3627 Address:35a1d88f-5dc0-ec1f-ea16-314026fe3627} {Suffrage:Voter ID:server-8b64f623-830c-0393-f155-d64c94c92e98 Address:8b64f623-830c-0393-f155-d64c94c92e98}]" + testing.go:149: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.361+1400 [INFO] server-472e3a07-d252-55b6-b13c-14eee42fc623: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-472e3a07-d252-55b6-b13c-14eee42fc623 Address:472e3a07-d252-55b6-b13c-14eee42fc623} {Suffrage:Voter ID:server-35a1d88f-5dc0-ec1f-ea16-314026fe3627 Address:35a1d88f-5dc0-ec1f-ea16-314026fe3627} {Suffrage:Voter ID:server-8b64f623-830c-0393-f155-d64c94c92e98 Address:8b64f623-830c-0393-f155-d64c94c92e98}]" +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.361+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-472e3a07-d252-55b6-b13c-14eee42fc623 Address:472e3a07-d252-55b6-b13c-14eee42fc623} {Suffrage:Voter ID:server-35a1d88f-5dc0-ec1f-ea16-314026fe3627 Address:35a1d88f-5dc0-ec1f-ea16-314026fe3627} {Suffrage:Voter ID:server-8b64f623-830c-0393-f155-d64c94c92e98 Address:8b64f623-830c-0393-f155-d64c94c92e98}]" + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.361+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-472e3a07-d252-55b6-b13c-14eee42fc623 Address:472e3a07-d252-55b6-b13c-14eee42fc623} {Suffrage:Voter ID:server-35a1d88f-5dc0-ec1f-ea16-314026fe3627 Address:35a1d88f-5dc0-ec1f-ea16-314026fe3627} {Suffrage:Voter ID:server-8b64f623-830c-0393-f155-d64c94c92e98 Address:8b64f623-830c-0393-f155-d64c94c92e98}]" +testLoggerAdapter verbose: server-8b64f623-830c-0393-f155-d64c94c92e98: 2023-05-16T04:06:14.361+1400 [INFO] server-8b64f623-830c-0393-f155-d64c94c92e98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-472e3a07-d252-55b6-b13c-14eee42fc623 Address:472e3a07-d252-55b6-b13c-14eee42fc623} {Suffrage:Voter ID:server-35a1d88f-5dc0-ec1f-ea16-314026fe3627 Address:35a1d88f-5dc0-ec1f-ea16-314026fe3627} {Suffrage:Voter ID:server-8b64f623-830c-0393-f155-d64c94c92e98 Address:8b64f623-830c-0393-f155-d64c94c92e98}]" + testing.go:149: server-8b64f623-830c-0393-f155-d64c94c92e98: 2023-05-16T04:06:14.361+1400 [INFO] server-8b64f623-830c-0393-f155-d64c94c92e98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-472e3a07-d252-55b6-b13c-14eee42fc623 Address:472e3a07-d252-55b6-b13c-14eee42fc623} {Suffrage:Voter ID:server-35a1d88f-5dc0-ec1f-ea16-314026fe3627 Address:35a1d88f-5dc0-ec1f-ea16-314026fe3627} {Suffrage:Voter ID:server-8b64f623-830c-0393-f155-d64c94c92e98 Address:8b64f623-830c-0393-f155-d64c94c92e98}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.361+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:14.361+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.362+1400 [INFO] server-472e3a07-d252-55b6-b13c-14eee42fc623: entering follower state: follower="Node at 472e3a07-d252-55b6-b13c-14eee42fc623 [Follower]" leader= + testing.go:149: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.362+1400 [INFO] server-472e3a07-d252-55b6-b13c-14eee42fc623: entering follower state: follower="Node at 472e3a07-d252-55b6-b13c-14eee42fc623 [Follower]" leader= +testLoggerAdapter verbose: server-8b64f623-830c-0393-f155-d64c94c92e98: 2023-05-16T04:06:14.362+1400 [INFO] server-8b64f623-830c-0393-f155-d64c94c92e98: entering follower state: follower="Node at 8b64f623-830c-0393-f155-d64c94c92e98 [Follower]" leader= +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.362+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: entering follower state: follower="Node at 35a1d88f-5dc0-ec1f-ea16-314026fe3627 [Follower]" leader= + testing.go:149: server-8b64f623-830c-0393-f155-d64c94c92e98: 2023-05-16T04:06:14.362+1400 [INFO] server-8b64f623-830c-0393-f155-d64c94c92e98: entering follower state: follower="Node at 8b64f623-830c-0393-f155-d64c94c92e98 [Follower]" leader= + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.362+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: entering follower state: follower="Node at 35a1d88f-5dc0-ec1f-ea16-314026fe3627 [Follower]" leader= +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [WARN] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [WARN] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: entering candidate state: node="Node at 35a1d88f-5dc0-ec1f-ea16-314026fe3627 [Candidate]" term=2 + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: entering candidate state: node="Node at 35a1d88f-5dc0-ec1f-ea16-314026fe3627 [Candidate]" term=2 +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: election won: tally=2 + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: election won: tally=2 +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: entering leader state: leader="Node at 35a1d88f-5dc0-ec1f-ea16-314026fe3627 [Leader]" + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: entering leader state: leader="Node at 35a1d88f-5dc0-ec1f-ea16-314026fe3627 [Leader]" +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: added peer, starting replication: peer=server-472e3a07-d252-55b6-b13c-14eee42fc623 + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: added peer, starting replication: peer=server-472e3a07-d252-55b6-b13c-14eee42fc623 +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: added peer, starting replication: peer=server-8b64f623-830c-0393-f155-d64c94c92e98 + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: added peer, starting replication: peer=server-8b64f623-830c-0393-f155-d64c94c92e98 +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: pipelining replication: peer="{Voter server-8b64f623-830c-0393-f155-d64c94c92e98 8b64f623-830c-0393-f155-d64c94c92e98}" + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.419+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: pipelining replication: peer="{Voter server-8b64f623-830c-0393-f155-d64c94c92e98 8b64f623-830c-0393-f155-d64c94c92e98}" +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.420+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: pipelining replication: peer="{Voter server-472e3a07-d252-55b6-b13c-14eee42fc623 472e3a07-d252-55b6-b13c-14eee42fc623}" + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.420+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: pipelining replication: peer="{Voter server-472e3a07-d252-55b6-b13c-14eee42fc623 472e3a07-d252-55b6-b13c-14eee42fc623}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.526+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.420445716 +1400 +14 m=+11.223027351 (1 nodes), 58.391689ms from start of poll, 59.498602ms from cluster start. Timeout at 2023-05-16 04:06:14.525960801 +1400 +14 m=+11.328542776, 105.515425ms after stability + testing.go:149: cluster: 2023-05-16T04:06:14.526+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.420445716 +1400 +14 m=+11.223027351 (1 nodes), 58.391689ms from start of poll, 59.498602ms from cluster start. Timeout at 2023-05-16 04:06:14.525960801 +1400 +14 m=+11.328542776, 105.515425ms after stability +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.526+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: starting snapshot up to: index=12 + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.526+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: starting snapshot up to: index=12 + testing.go:153: 2023-05-16T04:06:14.527+1400 [INFO] creating new snapshot: path=/tmp/raft1633987355/snapshots/2-12-1684159574527.tmp +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.528+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: snapshot complete up to: index=12 + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.528+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: snapshot complete up to: index=12 + testing.go:153: 2023-05-16T04:06:14.529+1400 [INFO] creating new snapshot: path=/tmp/raft1633987355/snapshots/2-23-1684159574529.tmp +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.529+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: copied to local snapshot: bytes=71 + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.529+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: restored user snapshot: index=1 + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: restored user snapshot: index=1 +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [ERROR] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: failed to get log: index=23 error="log not found" + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [ERROR] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: aborting pipeline replication: peer="{Voter server-472e3a07-d252-55b6-b13c-14eee42fc623 472e3a07-d252-55b6-b13c-14eee42fc623}" + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: aborting pipeline replication: peer="{Voter server-472e3a07-d252-55b6-b13c-14eee42fc623 472e3a07-d252-55b6-b13c-14eee42fc623}" +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [ERROR] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: failed to get log: index=23 error="log not found" + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [ERROR] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: aborting pipeline replication: peer="{Voter server-8b64f623-830c-0393-f155-d64c94c92e98 8b64f623-830c-0393-f155-d64c94c92e98}" + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.530+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: aborting pipeline replication: peer="{Voter server-8b64f623-830c-0393-f155-d64c94c92e98 8b64f623-830c-0393-f155-d64c94c92e98}" +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.537+1400 [ERROR] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: failed to get log: index=23 error="log not found" + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.537+1400 [ERROR] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.538+1400 [ERROR] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: failed to get log: index=23 error="log not found" + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.538+1400 [ERROR] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: failed to get log: index=23 error="log not found" + testing.go:153: 2023-05-16T04:06:14.538+1400 [INFO] creating new snapshot: path=/tmp/raft685787767/snapshots/2-23-1684159574538.tmp + testing.go:153: 2023-05-16T04:06:14.538+1400 [INFO] creating new snapshot: path=/tmp/raft1580458597/snapshots/2-23-1684159574538.tmp +testLoggerAdapter verbose: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.539+1400 [INFO] server-472e3a07-d252-55b6-b13c-14eee42fc623: copied to local snapshot: bytes=71 + testing.go:149: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.539+1400 [INFO] server-472e3a07-d252-55b6-b13c-14eee42fc623: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.539+1400 [INFO] server-472e3a07-d252-55b6-b13c-14eee42fc623: Installed remote snapshot + testing.go:149: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.539+1400 [INFO] server-472e3a07-d252-55b6-b13c-14eee42fc623: Installed remote snapshot +testLoggerAdapter verbose: server-8b64f623-830c-0393-f155-d64c94c92e98: 2023-05-16T04:06:14.539+1400 [INFO] server-8b64f623-830c-0393-f155-d64c94c92e98: copied to local snapshot: bytes=71 + testing.go:149: server-8b64f623-830c-0393-f155-d64c94c92e98: 2023-05-16T04:06:14.539+1400 [INFO] server-8b64f623-830c-0393-f155-d64c94c92e98: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.539+1400 [WARN] server-472e3a07-d252-55b6-b13c-14eee42fc623: skipping application of old log: index=22 + testing.go:149: server-472e3a07-d252-55b6-b13c-14eee42fc623: 2023-05-16T04:06:14.539+1400 [WARN] server-472e3a07-d252-55b6-b13c-14eee42fc623: skipping application of old log: index=22 +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.539+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: pipelining replication: peer="{Voter server-472e3a07-d252-55b6-b13c-14eee42fc623 472e3a07-d252-55b6-b13c-14eee42fc623}" + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.539+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: pipelining replication: peer="{Voter server-472e3a07-d252-55b6-b13c-14eee42fc623 472e3a07-d252-55b6-b13c-14eee42fc623}" +testLoggerAdapter verbose: server-8b64f623-830c-0393-f155-d64c94c92e98: 2023-05-16T04:06:14.539+1400 [INFO] server-8b64f623-830c-0393-f155-d64c94c92e98: Installed remote snapshot + testing.go:149: server-8b64f623-830c-0393-f155-d64c94c92e98: 2023-05-16T04:06:14.539+1400 [INFO] server-8b64f623-830c-0393-f155-d64c94c92e98: Installed remote snapshot +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.540+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: pipelining replication: peer="{Voter server-8b64f623-830c-0393-f155-d64c94c92e98 8b64f623-830c-0393-f155-d64c94c92e98}" + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.540+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: pipelining replication: peer="{Voter server-8b64f623-830c-0393-f155-d64c94c92e98 8b64f623-830c-0393-f155-d64c94c92e98}" +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.557+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: aborting pipeline replication: peer="{Voter server-8b64f623-830c-0393-f155-d64c94c92e98 8b64f623-830c-0393-f155-d64c94c92e98}" + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.557+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: aborting pipeline replication: peer="{Voter server-8b64f623-830c-0393-f155-d64c94c92e98 8b64f623-830c-0393-f155-d64c94c92e98}" +testLoggerAdapter verbose: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.558+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: aborting pipeline replication: peer="{Voter server-472e3a07-d252-55b6-b13c-14eee42fc623 472e3a07-d252-55b6-b13c-14eee42fc623}" + testing.go:149: server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: 2023-05-16T04:06:14.558+1400 [INFO] server-35a1d88f-5dc0-ec1f-ea16-314026fe3627: aborting pipeline replication: peer="{Voter server-472e3a07-d252-55b6-b13c-14eee42fc623 472e3a07-d252-55b6-b13c-14eee42fc623}" +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.559+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 Address:ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56} {Suffrage:Voter ID:server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 Address:f477a1fe-85b0-ac5c-0300-95e12f7640e0} {Suffrage:Voter ID:server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 Address:3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}]" + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.559+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 Address:ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56} {Suffrage:Voter ID:server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 Address:f477a1fe-85b0-ac5c-0300-95e12f7640e0} {Suffrage:Voter ID:server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 Address:3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}]" +testLoggerAdapter verbose: server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: 2023-05-16T04:06:14.560+1400 [INFO] server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 Address:ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56} {Suffrage:Voter ID:server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 Address:f477a1fe-85b0-ac5c-0300-95e12f7640e0} {Suffrage:Voter ID:server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 Address:3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}]" + testing.go:149: server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: 2023-05-16T04:06:14.560+1400 [INFO] server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 Address:ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56} {Suffrage:Voter ID:server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 Address:f477a1fe-85b0-ac5c-0300-95e12f7640e0} {Suffrage:Voter ID:server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 Address:3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}]" +testLoggerAdapter verbose: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.560+1400 [INFO] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 Address:ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56} {Suffrage:Voter ID:server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 Address:f477a1fe-85b0-ac5c-0300-95e12f7640e0} {Suffrage:Voter ID:server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 Address:3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}]" + testing.go:149: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.560+1400 [INFO] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 Address:ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56} {Suffrage:Voter ID:server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 Address:f477a1fe-85b0-ac5c-0300-95e12f7640e0} {Suffrage:Voter ID:server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 Address:3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.560+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:14.560+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.560+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: entering follower state: follower="Node at ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 [Follower]" leader= + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.560+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: entering follower state: follower="Node at ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 [Follower]" leader= +testLoggerAdapter verbose: server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: 2023-05-16T04:06:14.560+1400 [INFO] server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: entering follower state: follower="Node at f477a1fe-85b0-ac5c-0300-95e12f7640e0 [Follower]" leader= + testing.go:149: server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: 2023-05-16T04:06:14.560+1400 [INFO] server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: entering follower state: follower="Node at f477a1fe-85b0-ac5c-0300-95e12f7640e0 [Follower]" leader= +testLoggerAdapter verbose: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.560+1400 [INFO] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: entering follower state: follower="Node at 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 [Follower]" leader= + testing.go:149: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.560+1400 [INFO] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: entering follower state: follower="Node at 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 [Follower]" leader= +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [WARN] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [WARN] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: entering candidate state: node="Node at ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 [Candidate]" term=2 + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: entering candidate state: node="Node at ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 [Candidate]" term=2 +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: election won: tally=2 + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: election won: tally=2 +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: entering leader state: leader="Node at ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 [Leader]" + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: entering leader state: leader="Node at ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56 [Leader]" +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: added peer, starting replication: peer=server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: added peer, starting replication: peer=server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: added peer, starting replication: peer=server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: added peer, starting replication: peer=server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: pipelining replication: peer="{Voter server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}" + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.613+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: pipelining replication: peer="{Voter server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}" +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.614+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: pipelining replication: peer="{Voter server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 f477a1fe-85b0-ac5c-0300-95e12f7640e0}" + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.614+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: pipelining replication: peer="{Voter server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 f477a1fe-85b0-ac5c-0300-95e12f7640e0}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.719+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.614204503 +1400 +14 m=+11.416786138 (1 nodes), 53.635153ms from start of poll, 54.593725ms from cluster start. Timeout at 2023-05-16 04:06:14.719279883 +1400 +14 m=+11.521861518, 105.07538ms after stability + testing.go:149: cluster: 2023-05-16T04:06:14.719+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.614204503 +1400 +14 m=+11.416786138 (1 nodes), 53.635153ms from start of poll, 54.593725ms from cluster start. Timeout at 2023-05-16 04:06:14.719279883 +1400 +14 m=+11.521861518, 105.07538ms after stability +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.720+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: starting snapshot up to: index=12 + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.720+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: starting snapshot up to: index=12 + testing.go:153: 2023-05-16T04:06:14.720+1400 [INFO] creating new snapshot: path=/tmp/raft2598925801/snapshots/2-12-1684159574720.tmp +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.721+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: snapshot complete up to: index=12 + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.721+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: snapshot complete up to: index=12 + testing.go:153: 2023-05-16T04:06:14.722+1400 [INFO] creating new snapshot: path=/tmp/raft2598925801/snapshots/2-23-1684159574722.tmp +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.723+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: copied to local snapshot: bytes=71 + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.723+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.723+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: restored user snapshot: index=1 + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.723+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: restored user snapshot: index=1 +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.723+1400 [ERROR] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: failed to get log: index=23 error="log not found" + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.723+1400 [ERROR] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.723+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: aborting pipeline replication: peer="{Voter server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}" + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.723+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: aborting pipeline replication: peer="{Voter server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}" +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.724+1400 [ERROR] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: failed to get log: index=23 error="log not found" + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.724+1400 [ERROR] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.724+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: aborting pipeline replication: peer="{Voter server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 f477a1fe-85b0-ac5c-0300-95e12f7640e0}" + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.724+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: aborting pipeline replication: peer="{Voter server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 f477a1fe-85b0-ac5c-0300-95e12f7640e0}" +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.733+1400 [ERROR] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: failed to get log: index=23 error="log not found" + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.733+1400 [ERROR] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: failed to get log: index=23 error="log not found" + testing.go:153: 2023-05-16T04:06:14.734+1400 [INFO] creating new snapshot: path=/tmp/raft1270729232/snapshots/2-23-1684159574734.tmp +testLoggerAdapter verbose: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.735+1400 [INFO] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: copied to local snapshot: bytes=71 + testing.go:149: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.735+1400 [INFO] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.735+1400 [ERROR] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: failed to get log: index=23 error="log not found" + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.735+1400 [ERROR] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: failed to get log: index=23 error="log not found" + testing.go:153: 2023-05-16T04:06:14.735+1400 [INFO] creating new snapshot: path=/tmp/raft4082407490/snapshots/2-23-1684159574735.tmp +testLoggerAdapter verbose: server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: 2023-05-16T04:06:14.736+1400 [INFO] server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: copied to local snapshot: bytes=71 + testing.go:149: server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: 2023-05-16T04:06:14.736+1400 [INFO] server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.740+1400 [INFO] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: Installed remote snapshot + testing.go:149: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.740+1400 [INFO] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: Installed remote snapshot +testLoggerAdapter verbose: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.740+1400 [WARN] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: skipping application of old log: index=22 + testing.go:149: server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: 2023-05-16T04:06:14.740+1400 [WARN] server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6: skipping application of old log: index=22 +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.740+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: pipelining replication: peer="{Voter server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}" + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.740+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: pipelining replication: peer="{Voter server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}" +testLoggerAdapter verbose: server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: 2023-05-16T04:06:14.742+1400 [INFO] server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: Installed remote snapshot + testing.go:149: server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: 2023-05-16T04:06:14.742+1400 [INFO] server-f477a1fe-85b0-ac5c-0300-95e12f7640e0: Installed remote snapshot +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.742+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: pipelining replication: peer="{Voter server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 f477a1fe-85b0-ac5c-0300-95e12f7640e0}" + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.742+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: pipelining replication: peer="{Voter server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 f477a1fe-85b0-ac5c-0300-95e12f7640e0}" +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.753+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: aborting pipeline replication: peer="{Voter server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}" + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.753+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: aborting pipeline replication: peer="{Voter server-3b28c983-e72e-7e46-2a16-ba4ead4ff2f6 3b28c983-e72e-7e46-2a16-ba4ead4ff2f6}" +testLoggerAdapter verbose: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.753+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: aborting pipeline replication: peer="{Voter server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 f477a1fe-85b0-ac5c-0300-95e12f7640e0}" + testing.go:149: server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: 2023-05-16T04:06:14.753+1400 [INFO] server-ecc4f8b3-05f6-9bfa-7ebb-bb078cc3ba56: aborting pipeline replication: peer="{Voter server-f477a1fe-85b0-ac5c-0300-95e12f7640e0 f477a1fe-85b0-ac5c-0300-95e12f7640e0}" +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.755+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4 Address:4b99900c-4420-ba47-b2e2-3e74c6bef2e4} {Suffrage:Voter ID:server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c Address:3dea05ed-8854-a0de-4b55-250f9c5e0d6c} {Suffrage:Voter ID:server-0cb1df11-83a6-892b-b1d6-679df7901410 Address:0cb1df11-83a6-892b-b1d6-679df7901410}]" + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.755+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4 Address:4b99900c-4420-ba47-b2e2-3e74c6bef2e4} {Suffrage:Voter ID:server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c Address:3dea05ed-8854-a0de-4b55-250f9c5e0d6c} {Suffrage:Voter ID:server-0cb1df11-83a6-892b-b1d6-679df7901410 Address:0cb1df11-83a6-892b-b1d6-679df7901410}]" +testLoggerAdapter verbose: server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: 2023-05-16T04:06:14.755+1400 [INFO] server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4 Address:4b99900c-4420-ba47-b2e2-3e74c6bef2e4} {Suffrage:Voter ID:server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c Address:3dea05ed-8854-a0de-4b55-250f9c5e0d6c} {Suffrage:Voter ID:server-0cb1df11-83a6-892b-b1d6-679df7901410 Address:0cb1df11-83a6-892b-b1d6-679df7901410}]" + testing.go:149: server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: 2023-05-16T04:06:14.755+1400 [INFO] server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4 Address:4b99900c-4420-ba47-b2e2-3e74c6bef2e4} {Suffrage:Voter ID:server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c Address:3dea05ed-8854-a0de-4b55-250f9c5e0d6c} {Suffrage:Voter ID:server-0cb1df11-83a6-892b-b1d6-679df7901410 Address:0cb1df11-83a6-892b-b1d6-679df7901410}]" +testLoggerAdapter verbose: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.755+1400 [INFO] server-0cb1df11-83a6-892b-b1d6-679df7901410: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4 Address:4b99900c-4420-ba47-b2e2-3e74c6bef2e4} {Suffrage:Voter ID:server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c Address:3dea05ed-8854-a0de-4b55-250f9c5e0d6c} {Suffrage:Voter ID:server-0cb1df11-83a6-892b-b1d6-679df7901410 Address:0cb1df11-83a6-892b-b1d6-679df7901410}]" + testing.go:149: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.755+1400 [INFO] server-0cb1df11-83a6-892b-b1d6-679df7901410: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4 Address:4b99900c-4420-ba47-b2e2-3e74c6bef2e4} {Suffrage:Voter ID:server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c Address:3dea05ed-8854-a0de-4b55-250f9c5e0d6c} {Suffrage:Voter ID:server-0cb1df11-83a6-892b-b1d6-679df7901410 Address:0cb1df11-83a6-892b-b1d6-679df7901410}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.755+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:14.755+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.755+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: entering follower state: follower="Node at 4b99900c-4420-ba47-b2e2-3e74c6bef2e4 [Follower]" leader= + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.755+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: entering follower state: follower="Node at 4b99900c-4420-ba47-b2e2-3e74c6bef2e4 [Follower]" leader= +testLoggerAdapter verbose: server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: 2023-05-16T04:06:14.755+1400 [INFO] server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: entering follower state: follower="Node at 3dea05ed-8854-a0de-4b55-250f9c5e0d6c [Follower]" leader= + testing.go:149: server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: 2023-05-16T04:06:14.755+1400 [INFO] server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: entering follower state: follower="Node at 3dea05ed-8854-a0de-4b55-250f9c5e0d6c [Follower]" leader= +testLoggerAdapter verbose: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.755+1400 [INFO] server-0cb1df11-83a6-892b-b1d6-679df7901410: entering follower state: follower="Node at 0cb1df11-83a6-892b-b1d6-679df7901410 [Follower]" leader= + testing.go:149: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.755+1400 [INFO] server-0cb1df11-83a6-892b-b1d6-679df7901410: entering follower state: follower="Node at 0cb1df11-83a6-892b-b1d6-679df7901410 [Follower]" leader= +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.808+1400 [WARN] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.808+1400 [WARN] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: entering candidate state: node="Node at 4b99900c-4420-ba47-b2e2-3e74c6bef2e4 [Candidate]" term=2 + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: entering candidate state: node="Node at 4b99900c-4420-ba47-b2e2-3e74c6bef2e4 [Candidate]" term=2 +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: election won: tally=2 + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: election won: tally=2 +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: entering leader state: leader="Node at 4b99900c-4420-ba47-b2e2-3e74c6bef2e4 [Leader]" + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: entering leader state: leader="Node at 4b99900c-4420-ba47-b2e2-3e74c6bef2e4 [Leader]" +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: added peer, starting replication: peer=server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: added peer, starting replication: peer=server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: added peer, starting replication: peer=server-0cb1df11-83a6-892b-b1d6-679df7901410 + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: added peer, starting replication: peer=server-0cb1df11-83a6-892b-b1d6-679df7901410 +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: pipelining replication: peer="{Voter server-0cb1df11-83a6-892b-b1d6-679df7901410 0cb1df11-83a6-892b-b1d6-679df7901410}" + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: pipelining replication: peer="{Voter server-0cb1df11-83a6-892b-b1d6-679df7901410 0cb1df11-83a6-892b-b1d6-679df7901410}" +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: pipelining replication: peer="{Voter server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c 3dea05ed-8854-a0de-4b55-250f9c5e0d6c}" + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.809+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: pipelining replication: peer="{Voter server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c 3dea05ed-8854-a0de-4b55-250f9c5e0d6c}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.915+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.810114095 +1400 +14 m=+11.612695730 (1 nodes), 54.456203ms from start of poll, 55.295393ms from cluster start. Timeout at 2023-05-16 04:06:14.915266715 +1400 +14 m=+11.717848350, 105.15262ms after stability + testing.go:149: cluster: 2023-05-16T04:06:14.915+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:14.810114095 +1400 +14 m=+11.612695730 (1 nodes), 54.456203ms from start of poll, 55.295393ms from cluster start. Timeout at 2023-05-16 04:06:14.915266715 +1400 +14 m=+11.717848350, 105.15262ms after stability +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.916+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: starting snapshot up to: index=12 + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.916+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: starting snapshot up to: index=12 + testing.go:153: 2023-05-16T04:06:14.916+1400 [INFO] creating new snapshot: path=/tmp/raft3176389591/snapshots/2-12-1684159574916.tmp +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.917+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: snapshot complete up to: index=12 + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.917+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: snapshot complete up to: index=12 + testing.go:153: 2023-05-16T04:06:14.918+1400 [INFO] creating new snapshot: path=/tmp/raft3176389591/snapshots/2-113-1684159574918.tmp +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.918+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: copied to local snapshot: bytes=71 + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.918+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: restored user snapshot: index=1 + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: restored user snapshot: index=1 +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [ERROR] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: failed to get log: index=23 error="log not found" + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [ERROR] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: aborting pipeline replication: peer="{Voter server-0cb1df11-83a6-892b-b1d6-679df7901410 0cb1df11-83a6-892b-b1d6-679df7901410}" + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: aborting pipeline replication: peer="{Voter server-0cb1df11-83a6-892b-b1d6-679df7901410 0cb1df11-83a6-892b-b1d6-679df7901410}" +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [ERROR] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: failed to get log: index=23 error="log not found" + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [ERROR] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: aborting pipeline replication: peer="{Voter server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c 3dea05ed-8854-a0de-4b55-250f9c5e0d6c}" + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.919+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: aborting pipeline replication: peer="{Voter server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c 3dea05ed-8854-a0de-4b55-250f9c5e0d6c}" +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.926+1400 [ERROR] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: failed to get log: index=23 error="log not found" + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.926+1400 [ERROR] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: failed to get log: index=23 error="log not found" + testing.go:153: 2023-05-16T04:06:14.927+1400 [INFO] creating new snapshot: path=/tmp/raft670186945/snapshots/2-113-1684159574927.tmp +testLoggerAdapter verbose: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.927+1400 [INFO] server-0cb1df11-83a6-892b-b1d6-679df7901410: copied to local snapshot: bytes=71 + testing.go:149: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.927+1400 [INFO] server-0cb1df11-83a6-892b-b1d6-679df7901410: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.928+1400 [INFO] server-0cb1df11-83a6-892b-b1d6-679df7901410: Installed remote snapshot + testing.go:149: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.928+1400 [INFO] server-0cb1df11-83a6-892b-b1d6-679df7901410: Installed remote snapshot +testLoggerAdapter verbose: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.928+1400 [WARN] server-0cb1df11-83a6-892b-b1d6-679df7901410: skipping application of old log: index=22 + testing.go:149: server-0cb1df11-83a6-892b-b1d6-679df7901410: 2023-05-16T04:06:14.928+1400 [WARN] server-0cb1df11-83a6-892b-b1d6-679df7901410: skipping application of old log: index=22 +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.928+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: pipelining replication: peer="{Voter server-0cb1df11-83a6-892b-b1d6-679df7901410 0cb1df11-83a6-892b-b1d6-679df7901410}" + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.928+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: pipelining replication: peer="{Voter server-0cb1df11-83a6-892b-b1d6-679df7901410 0cb1df11-83a6-892b-b1d6-679df7901410}" +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.929+1400 [ERROR] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: failed to get log: index=23 error="log not found" + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.929+1400 [ERROR] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: failed to get log: index=23 error="log not found" + testing.go:153: 2023-05-16T04:06:14.930+1400 [INFO] creating new snapshot: path=/tmp/raft2096542586/snapshots/2-113-1684159574930.tmp +testLoggerAdapter verbose: server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: 2023-05-16T04:06:14.931+1400 [INFO] server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: copied to local snapshot: bytes=71 + testing.go:149: server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: 2023-05-16T04:06:14.931+1400 [INFO] server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: 2023-05-16T04:06:14.931+1400 [INFO] server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: Installed remote snapshot + testing.go:149: server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: 2023-05-16T04:06:14.931+1400 [INFO] server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c: Installed remote snapshot +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.931+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: pipelining replication: peer="{Voter server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c 3dea05ed-8854-a0de-4b55-250f9c5e0d6c}" + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.931+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: pipelining replication: peer="{Voter server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c 3dea05ed-8854-a0de-4b55-250f9c5e0d6c}" +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.946+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: aborting pipeline replication: peer="{Voter server-0cb1df11-83a6-892b-b1d6-679df7901410 0cb1df11-83a6-892b-b1d6-679df7901410}" + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.946+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: aborting pipeline replication: peer="{Voter server-0cb1df11-83a6-892b-b1d6-679df7901410 0cb1df11-83a6-892b-b1d6-679df7901410}" +testLoggerAdapter verbose: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.946+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: aborting pipeline replication: peer="{Voter server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c 3dea05ed-8854-a0de-4b55-250f9c5e0d6c}" + testing.go:149: server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: 2023-05-16T04:06:14.946+1400 [INFO] server-4b99900c-4420-ba47-b2e2-3e74c6bef2e4: aborting pipeline replication: peer="{Voter server-3dea05ed-8854-a0de-4b55-250f9c5e0d6c 3dea05ed-8854-a0de-4b55-250f9c5e0d6c}" +testLoggerAdapter verbose: server-a6f3b0b9-5799-c956-00ee-89e057d28452: 2023-05-16T04:06:14.948+1400 [INFO] server-a6f3b0b9-5799-c956-00ee-89e057d28452: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f3b0b9-5799-c956-00ee-89e057d28452 Address:a6f3b0b9-5799-c956-00ee-89e057d28452} {Suffrage:Voter ID:server-db6545b6-ca28-eb82-c19c-a993af35b3ba Address:db6545b6-ca28-eb82-c19c-a993af35b3ba} {Suffrage:Voter ID:server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f Address:e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}]" + testing.go:149: server-a6f3b0b9-5799-c956-00ee-89e057d28452: 2023-05-16T04:06:14.948+1400 [INFO] server-a6f3b0b9-5799-c956-00ee-89e057d28452: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f3b0b9-5799-c956-00ee-89e057d28452 Address:a6f3b0b9-5799-c956-00ee-89e057d28452} {Suffrage:Voter ID:server-db6545b6-ca28-eb82-c19c-a993af35b3ba Address:db6545b6-ca28-eb82-c19c-a993af35b3ba} {Suffrage:Voter ID:server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f Address:e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}]" +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:14.949+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f3b0b9-5799-c956-00ee-89e057d28452 Address:a6f3b0b9-5799-c956-00ee-89e057d28452} {Suffrage:Voter ID:server-db6545b6-ca28-eb82-c19c-a993af35b3ba Address:db6545b6-ca28-eb82-c19c-a993af35b3ba} {Suffrage:Voter ID:server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f Address:e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}]" + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:14.949+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f3b0b9-5799-c956-00ee-89e057d28452 Address:a6f3b0b9-5799-c956-00ee-89e057d28452} {Suffrage:Voter ID:server-db6545b6-ca28-eb82-c19c-a993af35b3ba Address:db6545b6-ca28-eb82-c19c-a993af35b3ba} {Suffrage:Voter ID:server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f Address:e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}]" +testLoggerAdapter verbose: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:14.949+1400 [INFO] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f3b0b9-5799-c956-00ee-89e057d28452 Address:a6f3b0b9-5799-c956-00ee-89e057d28452} {Suffrage:Voter ID:server-db6545b6-ca28-eb82-c19c-a993af35b3ba Address:db6545b6-ca28-eb82-c19c-a993af35b3ba} {Suffrage:Voter ID:server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f Address:e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}]" + testing.go:149: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:14.949+1400 [INFO] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6f3b0b9-5799-c956-00ee-89e057d28452 Address:a6f3b0b9-5799-c956-00ee-89e057d28452} {Suffrage:Voter ID:server-db6545b6-ca28-eb82-c19c-a993af35b3ba Address:db6545b6-ca28-eb82-c19c-a993af35b3ba} {Suffrage:Voter ID:server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f Address:e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:14.949+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:14.949+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-a6f3b0b9-5799-c956-00ee-89e057d28452: 2023-05-16T04:06:14.949+1400 [INFO] server-a6f3b0b9-5799-c956-00ee-89e057d28452: entering follower state: follower="Node at a6f3b0b9-5799-c956-00ee-89e057d28452 [Follower]" leader= + testing.go:149: server-a6f3b0b9-5799-c956-00ee-89e057d28452: 2023-05-16T04:06:14.949+1400 [INFO] server-a6f3b0b9-5799-c956-00ee-89e057d28452: entering follower state: follower="Node at a6f3b0b9-5799-c956-00ee-89e057d28452 [Follower]" leader= +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:14.949+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: entering follower state: follower="Node at db6545b6-ca28-eb82-c19c-a993af35b3ba [Follower]" leader= + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:14.949+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: entering follower state: follower="Node at db6545b6-ca28-eb82-c19c-a993af35b3ba [Follower]" leader= +testLoggerAdapter verbose: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:14.949+1400 [INFO] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: entering follower state: follower="Node at e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f [Follower]" leader= + testing.go:149: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:14.949+1400 [INFO] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: entering follower state: follower="Node at e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f [Follower]" leader= +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [WARN] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [WARN] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: entering candidate state: node="Node at db6545b6-ca28-eb82-c19c-a993af35b3ba [Candidate]" term=2 + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: entering candidate state: node="Node at db6545b6-ca28-eb82-c19c-a993af35b3ba [Candidate]" term=2 +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: election won: tally=2 + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: election won: tally=2 +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: entering leader state: leader="Node at db6545b6-ca28-eb82-c19c-a993af35b3ba [Leader]" + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: entering leader state: leader="Node at db6545b6-ca28-eb82-c19c-a993af35b3ba [Leader]" +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: added peer, starting replication: peer=server-a6f3b0b9-5799-c956-00ee-89e057d28452 + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: added peer, starting replication: peer=server-a6f3b0b9-5799-c956-00ee-89e057d28452 +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: added peer, starting replication: peer=server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: added peer, starting replication: peer=server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: pipelining replication: peer="{Voter server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}" + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.006+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: pipelining replication: peer="{Voter server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}" +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.007+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: pipelining replication: peer="{Voter server-a6f3b0b9-5799-c956-00ee-89e057d28452 a6f3b0b9-5799-c956-00ee-89e057d28452}" + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.007+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: pipelining replication: peer="{Voter server-a6f3b0b9-5799-c956-00ee-89e057d28452 a6f3b0b9-5799-c956-00ee-89e057d28452}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.112+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:15.007343222 +1400 +14 m=+11.809924837 (1 nodes), 57.850923ms from start of poll, 58.828354ms from cluster start. Timeout at 2023-05-16 04:06:15.112641365 +1400 +14 m=+11.915223020, 105.298183ms after stability + testing.go:149: cluster: 2023-05-16T04:06:15.112+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:15.007343222 +1400 +14 m=+11.809924837 (1 nodes), 57.850923ms from start of poll, 58.828354ms from cluster start. Timeout at 2023-05-16 04:06:15.112641365 +1400 +14 m=+11.915223020, 105.298183ms after stability +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.113+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: starting snapshot up to: index=12 + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.113+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: starting snapshot up to: index=12 + testing.go:153: 2023-05-16T04:06:15.113+1400 [INFO] creating new snapshot: path=/tmp/raft4245320530/snapshots/2-12-1684159575113.tmp +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.114+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: snapshot complete up to: index=12 + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.114+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: snapshot complete up to: index=12 + testing.go:153: 2023-05-16T04:06:15.115+1400 [INFO] creating new snapshot: path=/tmp/raft4245320530/snapshots/2-1013-1684159575115.tmp +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.116+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: copied to local snapshot: bytes=71 + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.116+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: restored user snapshot: index=1 + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: restored user snapshot: index=1 +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [ERROR] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: failed to get log: index=23 error="log not found" + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [ERROR] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: aborting pipeline replication: peer="{Voter server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}" + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: aborting pipeline replication: peer="{Voter server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}" +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [ERROR] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: failed to get log: index=23 error="log not found" + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [ERROR] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: aborting pipeline replication: peer="{Voter server-a6f3b0b9-5799-c956-00ee-89e057d28452 a6f3b0b9-5799-c956-00ee-89e057d28452}" + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.117+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: aborting pipeline replication: peer="{Voter server-a6f3b0b9-5799-c956-00ee-89e057d28452 a6f3b0b9-5799-c956-00ee-89e057d28452}" +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.124+1400 [ERROR] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: failed to get log: index=23 error="log not found" + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.124+1400 [ERROR] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: failed to get log: index=23 error="log not found" + testing.go:153: 2023-05-16T04:06:15.124+1400 [INFO] creating new snapshot: path=/tmp/raft4044288030/snapshots/2-1013-1684159575124.tmp +testLoggerAdapter verbose: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:15.125+1400 [INFO] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: copied to local snapshot: bytes=71 + testing.go:149: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:15.125+1400 [INFO] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:15.126+1400 [INFO] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: Installed remote snapshot + testing.go:149: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:15.126+1400 [INFO] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: Installed remote snapshot +testLoggerAdapter verbose: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:15.126+1400 [WARN] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: skipping application of old log: index=22 + testing.go:149: server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: 2023-05-16T04:06:15.126+1400 [WARN] server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f: skipping application of old log: index=22 +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.126+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: pipelining replication: peer="{Voter server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}" + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.126+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: pipelining replication: peer="{Voter server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}" +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.126+1400 [ERROR] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: failed to get log: index=23 error="log not found" + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.126+1400 [ERROR] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: failed to get log: index=23 error="log not found" + testing.go:153: 2023-05-16T04:06:15.127+1400 [INFO] creating new snapshot: path=/tmp/raft549836052/snapshots/2-1013-1684159575127.tmp +testLoggerAdapter verbose: server-a6f3b0b9-5799-c956-00ee-89e057d28452: 2023-05-16T04:06:15.127+1400 [INFO] server-a6f3b0b9-5799-c956-00ee-89e057d28452: copied to local snapshot: bytes=71 + testing.go:149: server-a6f3b0b9-5799-c956-00ee-89e057d28452: 2023-05-16T04:06:15.127+1400 [INFO] server-a6f3b0b9-5799-c956-00ee-89e057d28452: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-a6f3b0b9-5799-c956-00ee-89e057d28452: 2023-05-16T04:06:15.128+1400 [INFO] server-a6f3b0b9-5799-c956-00ee-89e057d28452: Installed remote snapshot + testing.go:149: server-a6f3b0b9-5799-c956-00ee-89e057d28452: 2023-05-16T04:06:15.128+1400 [INFO] server-a6f3b0b9-5799-c956-00ee-89e057d28452: Installed remote snapshot +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.128+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: pipelining replication: peer="{Voter server-a6f3b0b9-5799-c956-00ee-89e057d28452 a6f3b0b9-5799-c956-00ee-89e057d28452}" + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.128+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: pipelining replication: peer="{Voter server-a6f3b0b9-5799-c956-00ee-89e057d28452 a6f3b0b9-5799-c956-00ee-89e057d28452}" +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.144+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: aborting pipeline replication: peer="{Voter server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}" + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.144+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: aborting pipeline replication: peer="{Voter server-e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f e393a1b3-d58f-aa0e-0831-e3a5e3c46d7f}" +testLoggerAdapter verbose: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.144+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: aborting pipeline replication: peer="{Voter server-a6f3b0b9-5799-c956-00ee-89e057d28452 a6f3b0b9-5799-c956-00ee-89e057d28452}" + testing.go:149: server-db6545b6-ca28-eb82-c19c-a993af35b3ba: 2023-05-16T04:06:15.144+1400 [INFO] server-db6545b6-ca28-eb82-c19c-a993af35b3ba: aborting pipeline replication: peer="{Voter server-a6f3b0b9-5799-c956-00ee-89e057d28452 a6f3b0b9-5799-c956-00ee-89e057d28452}" +testLoggerAdapter verbose: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.146+1400 [INFO] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 Address:e53c871b-99d0-0c11-33a7-7f1952c12ec0} {Suffrage:Voter ID:server-f920bc19-a021-8683-63d4-5c7dd32a7ad3 Address:f920bc19-a021-8683-63d4-5c7dd32a7ad3} {Suffrage:Voter ID:server-c539e53a-1c75-cfe2-70eb-898edcf8d51c Address:c539e53a-1c75-cfe2-70eb-898edcf8d51c}]" + testing.go:149: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.146+1400 [INFO] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 Address:e53c871b-99d0-0c11-33a7-7f1952c12ec0} {Suffrage:Voter ID:server-f920bc19-a021-8683-63d4-5c7dd32a7ad3 Address:f920bc19-a021-8683-63d4-5c7dd32a7ad3} {Suffrage:Voter ID:server-c539e53a-1c75-cfe2-70eb-898edcf8d51c Address:c539e53a-1c75-cfe2-70eb-898edcf8d51c}]" +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.146+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 Address:e53c871b-99d0-0c11-33a7-7f1952c12ec0} {Suffrage:Voter ID:server-f920bc19-a021-8683-63d4-5c7dd32a7ad3 Address:f920bc19-a021-8683-63d4-5c7dd32a7ad3} {Suffrage:Voter ID:server-c539e53a-1c75-cfe2-70eb-898edcf8d51c Address:c539e53a-1c75-cfe2-70eb-898edcf8d51c}]" + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.146+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 Address:e53c871b-99d0-0c11-33a7-7f1952c12ec0} {Suffrage:Voter ID:server-f920bc19-a021-8683-63d4-5c7dd32a7ad3 Address:f920bc19-a021-8683-63d4-5c7dd32a7ad3} {Suffrage:Voter ID:server-c539e53a-1c75-cfe2-70eb-898edcf8d51c Address:c539e53a-1c75-cfe2-70eb-898edcf8d51c}]" +testLoggerAdapter verbose: server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: 2023-05-16T04:06:15.146+1400 [INFO] server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 Address:e53c871b-99d0-0c11-33a7-7f1952c12ec0} {Suffrage:Voter ID:server-f920bc19-a021-8683-63d4-5c7dd32a7ad3 Address:f920bc19-a021-8683-63d4-5c7dd32a7ad3} {Suffrage:Voter ID:server-c539e53a-1c75-cfe2-70eb-898edcf8d51c Address:c539e53a-1c75-cfe2-70eb-898edcf8d51c}]" + testing.go:149: server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: 2023-05-16T04:06:15.146+1400 [INFO] server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 Address:e53c871b-99d0-0c11-33a7-7f1952c12ec0} {Suffrage:Voter ID:server-f920bc19-a021-8683-63d4-5c7dd32a7ad3 Address:f920bc19-a021-8683-63d4-5c7dd32a7ad3} {Suffrage:Voter ID:server-c539e53a-1c75-cfe2-70eb-898edcf8d51c Address:c539e53a-1c75-cfe2-70eb-898edcf8d51c}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.147+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:15.147+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.147+1400 [INFO] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: entering follower state: follower="Node at e53c871b-99d0-0c11-33a7-7f1952c12ec0 [Follower]" leader= + testing.go:149: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.147+1400 [INFO] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: entering follower state: follower="Node at e53c871b-99d0-0c11-33a7-7f1952c12ec0 [Follower]" leader= +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.147+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: entering follower state: follower="Node at f920bc19-a021-8683-63d4-5c7dd32a7ad3 [Follower]" leader= + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.147+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: entering follower state: follower="Node at f920bc19-a021-8683-63d4-5c7dd32a7ad3 [Follower]" leader= +testLoggerAdapter verbose: server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: 2023-05-16T04:06:15.147+1400 [INFO] server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: entering follower state: follower="Node at c539e53a-1c75-cfe2-70eb-898edcf8d51c [Follower]" leader= + testing.go:149: server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: 2023-05-16T04:06:15.147+1400 [INFO] server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: entering follower state: follower="Node at c539e53a-1c75-cfe2-70eb-898edcf8d51c [Follower]" leader= +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [WARN] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [WARN] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: entering candidate state: node="Node at f920bc19-a021-8683-63d4-5c7dd32a7ad3 [Candidate]" term=2 + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: entering candidate state: node="Node at f920bc19-a021-8683-63d4-5c7dd32a7ad3 [Candidate]" term=2 +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: election won: tally=2 + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: election won: tally=2 +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: entering leader state: leader="Node at f920bc19-a021-8683-63d4-5c7dd32a7ad3 [Leader]" + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: entering leader state: leader="Node at f920bc19-a021-8683-63d4-5c7dd32a7ad3 [Leader]" +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: added peer, starting replication: peer=server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: added peer, starting replication: peer=server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: added peer, starting replication: peer=server-c539e53a-1c75-cfe2-70eb-898edcf8d51c + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: added peer, starting replication: peer=server-c539e53a-1c75-cfe2-70eb-898edcf8d51c +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: pipelining replication: peer="{Voter server-c539e53a-1c75-cfe2-70eb-898edcf8d51c c539e53a-1c75-cfe2-70eb-898edcf8d51c}" + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.209+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: pipelining replication: peer="{Voter server-c539e53a-1c75-cfe2-70eb-898edcf8d51c c539e53a-1c75-cfe2-70eb-898edcf8d51c}" +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.210+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: pipelining replication: peer="{Voter server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 e53c871b-99d0-0c11-33a7-7f1952c12ec0}" + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.210+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: pipelining replication: peer="{Voter server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 e53c871b-99d0-0c11-33a7-7f1952c12ec0}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.316+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:15.210383358 +1400 +14 m=+12.012964993 (1 nodes), 63.304507ms from start of poll, 64.284438ms from cluster start. Timeout at 2023-05-16 04:06:15.315950844 +1400 +14 m=+12.118532479, 105.567486ms after stability + testing.go:149: cluster: 2023-05-16T04:06:15.316+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:15.210383358 +1400 +14 m=+12.012964993 (1 nodes), 63.304507ms from start of poll, 64.284438ms from cluster start. Timeout at 2023-05-16 04:06:15.315950844 +1400 +14 m=+12.118532479, 105.567486ms after stability +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.316+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: starting snapshot up to: index=12 + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.316+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: starting snapshot up to: index=12 + testing.go:153: 2023-05-16T04:06:15.317+1400 [INFO] creating new snapshot: path=/tmp/raft198941943/snapshots/2-12-1684159575317.tmp +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.317+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: snapshot complete up to: index=12 + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.317+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: snapshot complete up to: index=12 + testing.go:153: 2023-05-16T04:06:15.318+1400 [INFO] creating new snapshot: path=/tmp/raft198941943/snapshots/2-10013-1684159575318.tmp +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.320+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: copied to local snapshot: bytes=71 + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.320+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: restored user snapshot: index=1 + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: restored user snapshot: index=1 +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [ERROR] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: failed to get log: index=23 error="log not found" + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [ERROR] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: aborting pipeline replication: peer="{Voter server-c539e53a-1c75-cfe2-70eb-898edcf8d51c c539e53a-1c75-cfe2-70eb-898edcf8d51c}" + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: aborting pipeline replication: peer="{Voter server-c539e53a-1c75-cfe2-70eb-898edcf8d51c c539e53a-1c75-cfe2-70eb-898edcf8d51c}" +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [ERROR] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: failed to get log: index=23 error="log not found" + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [ERROR] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: aborting pipeline replication: peer="{Voter server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 e53c871b-99d0-0c11-33a7-7f1952c12ec0}" + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.324+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: aborting pipeline replication: peer="{Voter server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 e53c871b-99d0-0c11-33a7-7f1952c12ec0}" +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.331+1400 [ERROR] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: failed to get log: index=23 error="log not found" + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.331+1400 [ERROR] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: failed to get log: index=23 error="log not found" + testing.go:153: 2023-05-16T04:06:15.332+1400 [INFO] creating new snapshot: path=/tmp/raft2486864240/snapshots/2-10013-1684159575332.tmp +testLoggerAdapter verbose: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.333+1400 [INFO] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: copied to local snapshot: bytes=71 + testing.go:149: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.333+1400 [INFO] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.333+1400 [ERROR] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: failed to get log: index=23 error="log not found" + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.333+1400 [ERROR] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.333+1400 [INFO] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: Installed remote snapshot + testing.go:149: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.333+1400 [INFO] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: Installed remote snapshot +testLoggerAdapter verbose: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.333+1400 [WARN] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: skipping application of old log: index=22 + testing.go:149: server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: 2023-05-16T04:06:15.333+1400 [WARN] server-e53c871b-99d0-0c11-33a7-7f1952c12ec0: skipping application of old log: index=22 +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.333+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: pipelining replication: peer="{Voter server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 e53c871b-99d0-0c11-33a7-7f1952c12ec0}" + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.333+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: pipelining replication: peer="{Voter server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 e53c871b-99d0-0c11-33a7-7f1952c12ec0}" + testing.go:153: 2023-05-16T04:06:15.334+1400 [INFO] creating new snapshot: path=/tmp/raft688072939/snapshots/2-10013-1684159575334.tmp +testLoggerAdapter verbose: server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: 2023-05-16T04:06:15.334+1400 [INFO] server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: copied to local snapshot: bytes=71 + testing.go:149: server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: 2023-05-16T04:06:15.334+1400 [INFO] server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: 2023-05-16T04:06:15.335+1400 [INFO] server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: Installed remote snapshot + testing.go:149: server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: 2023-05-16T04:06:15.335+1400 [INFO] server-c539e53a-1c75-cfe2-70eb-898edcf8d51c: Installed remote snapshot +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.335+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: pipelining replication: peer="{Voter server-c539e53a-1c75-cfe2-70eb-898edcf8d51c c539e53a-1c75-cfe2-70eb-898edcf8d51c}" + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.335+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: pipelining replication: peer="{Voter server-c539e53a-1c75-cfe2-70eb-898edcf8d51c c539e53a-1c75-cfe2-70eb-898edcf8d51c}" +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.352+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: aborting pipeline replication: peer="{Voter server-c539e53a-1c75-cfe2-70eb-898edcf8d51c c539e53a-1c75-cfe2-70eb-898edcf8d51c}" + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.352+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: aborting pipeline replication: peer="{Voter server-c539e53a-1c75-cfe2-70eb-898edcf8d51c c539e53a-1c75-cfe2-70eb-898edcf8d51c}" +testLoggerAdapter verbose: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.352+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: aborting pipeline replication: peer="{Voter server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 e53c871b-99d0-0c11-33a7-7f1952c12ec0}" + testing.go:149: server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: 2023-05-16T04:06:15.352+1400 [INFO] server-f920bc19-a021-8683-63d4-5c7dd32a7ad3: aborting pipeline replication: peer="{Voter server-e53c871b-99d0-0c11-33a7-7f1952c12ec0 e53c871b-99d0-0c11-33a7-7f1952c12ec0}" --- PASS: TestRaft_UserRestore (1.19s) === RUN TestRaft_SendSnapshotFollower -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.893-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 Address:629dc9d9-a9d2-e9be-7e56-8c6a792f52e7} {Suffrage:Voter ID:server-30b015ac-9bf9-5785-9244-abf831fd55ad Address:30b015ac-9bf9-5785-9244-abf831fd55ad} {Suffrage:Voter ID:server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee Address:b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}]" - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.893-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 Address:629dc9d9-a9d2-e9be-7e56-8c6a792f52e7} {Suffrage:Voter ID:server-30b015ac-9bf9-5785-9244-abf831fd55ad Address:30b015ac-9bf9-5785-9244-abf831fd55ad} {Suffrage:Voter ID:server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee Address:b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}]" -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.893-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: entering follower state: follower="Node at 629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 [Follower]" leader= - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.893-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: entering follower state: follower="Node at 629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 [Follower]" leader= -testLoggerAdapter verbose: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:34.893-1200 [INFO] server-30b015ac-9bf9-5785-9244-abf831fd55ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 Address:629dc9d9-a9d2-e9be-7e56-8c6a792f52e7} {Suffrage:Voter ID:server-30b015ac-9bf9-5785-9244-abf831fd55ad Address:30b015ac-9bf9-5785-9244-abf831fd55ad} {Suffrage:Voter ID:server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee Address:b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}]" - testing.go:149: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:34.893-1200 [INFO] server-30b015ac-9bf9-5785-9244-abf831fd55ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 Address:629dc9d9-a9d2-e9be-7e56-8c6a792f52e7} {Suffrage:Voter ID:server-30b015ac-9bf9-5785-9244-abf831fd55ad Address:30b015ac-9bf9-5785-9244-abf831fd55ad} {Suffrage:Voter ID:server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee Address:b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}]" -testLoggerAdapter verbose: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:34.893-1200 [INFO] server-30b015ac-9bf9-5785-9244-abf831fd55ad: entering follower state: follower="Node at 30b015ac-9bf9-5785-9244-abf831fd55ad [Follower]" leader= - testing.go:149: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:34.893-1200 [INFO] server-30b015ac-9bf9-5785-9244-abf831fd55ad: entering follower state: follower="Node at 30b015ac-9bf9-5785-9244-abf831fd55ad [Follower]" leader= -testLoggerAdapter verbose: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:34.893-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 Address:629dc9d9-a9d2-e9be-7e56-8c6a792f52e7} {Suffrage:Voter ID:server-30b015ac-9bf9-5785-9244-abf831fd55ad Address:30b015ac-9bf9-5785-9244-abf831fd55ad} {Suffrage:Voter ID:server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee Address:b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}]" - testing.go:149: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:34.893-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 Address:629dc9d9-a9d2-e9be-7e56-8c6a792f52e7} {Suffrage:Voter ID:server-30b015ac-9bf9-5785-9244-abf831fd55ad Address:30b015ac-9bf9-5785-9244-abf831fd55ad} {Suffrage:Voter ID:server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee Address:b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:34.893-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:34.893-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:34.893-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: entering follower state: follower="Node at b357c0c5-4d5a-09d9-0c03-78e414f2b6ee [Follower]" leader= - testing.go:149: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:34.893-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: entering follower state: follower="Node at b357c0c5-4d5a-09d9-0c03-78e414f2b6ee [Follower]" leader= -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.947-1200 [WARN] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.947-1200 [WARN] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.947-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: entering candidate state: node="Node at 629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 [Candidate]" term=2 - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.947-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: entering candidate state: node="Node at 629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 [Candidate]" term=2 -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.947-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: election won: tally=2 - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.947-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: election won: tally=2 -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.947-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: entering leader state: leader="Node at 629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 [Leader]" - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.947-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: entering leader state: leader="Node at 629dc9d9-a9d2-e9be-7e56-8c6a792f52e7 [Leader]" -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.948-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: added peer, starting replication: peer=server-30b015ac-9bf9-5785-9244-abf831fd55ad - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.948-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: added peer, starting replication: peer=server-30b015ac-9bf9-5785-9244-abf831fd55ad -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.948-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: added peer, starting replication: peer=server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.948-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: added peer, starting replication: peer=server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.948-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: pipelining replication: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.948-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: pipelining replication: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.948-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: pipelining replication: peer="{Voter server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}" - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:34.948-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: pipelining replication: peer="{Voter server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.054-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:34.94804662 -1200 -12 m=+11.955363081 (2 nodes), 54.0292ms from start of poll, 54.87626ms from cluster start. Timeout at 2024-06-16 08:26:35.05395696 -1200 -12 m=+12.061273421, 105.91034ms after stability - testing.go:149: cluster: 2024-06-16T08:26:35.054-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:34.94804662 -1200 -12 m=+11.955363081 (2 nodes), 54.0292ms from start of poll, 54.87626ms from cluster start. Timeout at 2024-06-16 08:26:35.05395696 -1200 -12 m=+12.061273421, 105.91034ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.054-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:35.054-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.159-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:35.05419498 -1200 -12 m=+12.061511441 (1 nodes), 1.68µs from start of poll, 161.02462ms from cluster start. Timeout at 2024-06-16 08:26:35.1597508 -1200 -12 m=+12.167067261, 105.55582ms after stability - testing.go:149: cluster: 2024-06-16T08:26:35.159-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:35.05419498 -1200 -12 m=+12.061511441 (1 nodes), 1.68µs from start of poll, 161.02462ms from cluster start. Timeout at 2024-06-16 08:26:35.1597508 -1200 -12 m=+12.167067261, 105.55582ms after stability -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.160-1200 [ERROR] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: failed to pipeline appendEntries: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" error="append pipeline closed" - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.160-1200 [ERROR] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: failed to pipeline appendEntries: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" error="append pipeline closed" -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.160-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: aborting pipeline replication: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.160-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: aborting pipeline replication: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.160-1200 [ERROR] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: failed to appendEntries to: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" error="failed to connect to peer: 30b015ac-9bf9-5785-9244-abf831fd55ad" - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.160-1200 [ERROR] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: failed to appendEntries to: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" error="failed to connect to peer: 30b015ac-9bf9-5785-9244-abf831fd55ad" -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.160-1200 [ERROR] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: failed to heartbeat to: peer=30b015ac-9bf9-5785-9244-abf831fd55ad error="failed to connect to peer: 30b015ac-9bf9-5785-9244-abf831fd55ad" - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.160-1200 [ERROR] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: failed to heartbeat to: peer=30b015ac-9bf9-5785-9244-abf831fd55ad error="failed to connect to peer: 30b015ac-9bf9-5785-9244-abf831fd55ad" +testLoggerAdapter verbose: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.354+1400 [INFO] server-03622733-cd5d-ccfb-64e0-3a8c894869be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03622733-cd5d-ccfb-64e0-3a8c894869be Address:03622733-cd5d-ccfb-64e0-3a8c894869be} {Suffrage:Voter ID:server-1ed42e07-fa6f-872e-16cd-268f263532c0 Address:1ed42e07-fa6f-872e-16cd-268f263532c0} {Suffrage:Voter ID:server-e7cba1d4-896d-0553-c5ee-41509e938548 Address:e7cba1d4-896d-0553-c5ee-41509e938548}]" + testing.go:149: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.354+1400 [INFO] server-03622733-cd5d-ccfb-64e0-3a8c894869be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03622733-cd5d-ccfb-64e0-3a8c894869be Address:03622733-cd5d-ccfb-64e0-3a8c894869be} {Suffrage:Voter ID:server-1ed42e07-fa6f-872e-16cd-268f263532c0 Address:1ed42e07-fa6f-872e-16cd-268f263532c0} {Suffrage:Voter ID:server-e7cba1d4-896d-0553-c5ee-41509e938548 Address:e7cba1d4-896d-0553-c5ee-41509e938548}]" +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.354+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03622733-cd5d-ccfb-64e0-3a8c894869be Address:03622733-cd5d-ccfb-64e0-3a8c894869be} {Suffrage:Voter ID:server-1ed42e07-fa6f-872e-16cd-268f263532c0 Address:1ed42e07-fa6f-872e-16cd-268f263532c0} {Suffrage:Voter ID:server-e7cba1d4-896d-0553-c5ee-41509e938548 Address:e7cba1d4-896d-0553-c5ee-41509e938548}]" + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.354+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03622733-cd5d-ccfb-64e0-3a8c894869be Address:03622733-cd5d-ccfb-64e0-3a8c894869be} {Suffrage:Voter ID:server-1ed42e07-fa6f-872e-16cd-268f263532c0 Address:1ed42e07-fa6f-872e-16cd-268f263532c0} {Suffrage:Voter ID:server-e7cba1d4-896d-0553-c5ee-41509e938548 Address:e7cba1d4-896d-0553-c5ee-41509e938548}]" +testLoggerAdapter verbose: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.354+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03622733-cd5d-ccfb-64e0-3a8c894869be Address:03622733-cd5d-ccfb-64e0-3a8c894869be} {Suffrage:Voter ID:server-1ed42e07-fa6f-872e-16cd-268f263532c0 Address:1ed42e07-fa6f-872e-16cd-268f263532c0} {Suffrage:Voter ID:server-e7cba1d4-896d-0553-c5ee-41509e938548 Address:e7cba1d4-896d-0553-c5ee-41509e938548}]" + testing.go:149: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.354+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03622733-cd5d-ccfb-64e0-3a8c894869be Address:03622733-cd5d-ccfb-64e0-3a8c894869be} {Suffrage:Voter ID:server-1ed42e07-fa6f-872e-16cd-268f263532c0 Address:1ed42e07-fa6f-872e-16cd-268f263532c0} {Suffrage:Voter ID:server-e7cba1d4-896d-0553-c5ee-41509e938548 Address:e7cba1d4-896d-0553-c5ee-41509e938548}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.355+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:15.355+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.355+1400 [INFO] server-03622733-cd5d-ccfb-64e0-3a8c894869be: entering follower state: follower="Node at 03622733-cd5d-ccfb-64e0-3a8c894869be [Follower]" leader= + testing.go:149: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.355+1400 [INFO] server-03622733-cd5d-ccfb-64e0-3a8c894869be: entering follower state: follower="Node at 03622733-cd5d-ccfb-64e0-3a8c894869be [Follower]" leader= +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.355+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: entering follower state: follower="Node at 1ed42e07-fa6f-872e-16cd-268f263532c0 [Follower]" leader= + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.355+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: entering follower state: follower="Node at 1ed42e07-fa6f-872e-16cd-268f263532c0 [Follower]" leader= +testLoggerAdapter verbose: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.355+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: entering follower state: follower="Node at e7cba1d4-896d-0553-c5ee-41509e938548 [Follower]" leader= + testing.go:149: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.355+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: entering follower state: follower="Node at e7cba1d4-896d-0553-c5ee-41509e938548 [Follower]" leader= +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.415+1400 [WARN] server-1ed42e07-fa6f-872e-16cd-268f263532c0: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.415+1400 [WARN] server-1ed42e07-fa6f-872e-16cd-268f263532c0: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.415+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: entering candidate state: node="Node at 1ed42e07-fa6f-872e-16cd-268f263532c0 [Candidate]" term=2 + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.415+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: entering candidate state: node="Node at 1ed42e07-fa6f-872e-16cd-268f263532c0 [Candidate]" term=2 +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.415+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: election won: tally=2 + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.415+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: election won: tally=2 +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: entering leader state: leader="Node at 1ed42e07-fa6f-872e-16cd-268f263532c0 [Leader]" + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: entering leader state: leader="Node at 1ed42e07-fa6f-872e-16cd-268f263532c0 [Leader]" +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: added peer, starting replication: peer=server-03622733-cd5d-ccfb-64e0-3a8c894869be + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: added peer, starting replication: peer=server-03622733-cd5d-ccfb-64e0-3a8c894869be +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: added peer, starting replication: peer=server-e7cba1d4-896d-0553-c5ee-41509e938548 + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: added peer, starting replication: peer=server-e7cba1d4-896d-0553-c5ee-41509e938548 +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: pipelining replication: peer="{Voter server-e7cba1d4-896d-0553-c5ee-41509e938548 e7cba1d4-896d-0553-c5ee-41509e938548}" + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: pipelining replication: peer="{Voter server-e7cba1d4-896d-0553-c5ee-41509e938548 e7cba1d4-896d-0553-c5ee-41509e938548}" +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: pipelining replication: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.416+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: pipelining replication: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.522+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:15.416699793 +1400 +14 m=+12.219281408 (2 nodes), 61.577767ms from start of poll, 62.463157ms from cluster start. Timeout at 2023-05-16 04:06:15.5224056 +1400 +14 m=+12.324987235, 105.705827ms after stability + testing.go:149: cluster: 2023-05-16T04:06:15.522+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:15.416699793 +1400 +14 m=+12.219281408 (2 nodes), 61.577767ms from start of poll, 62.463157ms from cluster start. Timeout at 2023-05-16 04:06:15.5224056 +1400 +14 m=+12.324987235, 105.705827ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.522+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:15.522+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.628+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:15.522731144 +1400 +14 m=+12.325312779 (1 nodes), 2.82µs from start of poll, 168.494528ms from cluster start. Timeout at 2023-05-16 04:06:15.628689015 +1400 +14 m=+12.431270670, 105.957891ms after stability + testing.go:149: cluster: 2023-05-16T04:06:15.628+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:15.522731144 +1400 +14 m=+12.325312779 (1 nodes), 2.82µs from start of poll, 168.494528ms from cluster start. Timeout at 2023-05-16 04:06:15.628689015 +1400 +14 m=+12.431270670, 105.957891ms after stability +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.629+1400 [ERROR] server-1ed42e07-fa6f-872e-16cd-268f263532c0: failed to heartbeat to: peer=03622733-cd5d-ccfb-64e0-3a8c894869be error="failed to connect to peer: 03622733-cd5d-ccfb-64e0-3a8c894869be" + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.629+1400 [ERROR] server-1ed42e07-fa6f-872e-16cd-268f263532c0: failed to heartbeat to: peer=03622733-cd5d-ccfb-64e0-3a8c894869be error="failed to connect to peer: 03622733-cd5d-ccfb-64e0-3a8c894869be" +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.629+1400 [ERROR] server-1ed42e07-fa6f-872e-16cd-268f263532c0: failed to pipeline appendEntries: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" error="append pipeline closed" + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.629+1400 [ERROR] server-1ed42e07-fa6f-872e-16cd-268f263532c0: failed to pipeline appendEntries: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" error="append pipeline closed" +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.629+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: aborting pipeline replication: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.629+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: aborting pipeline replication: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.629+1400 [ERROR] server-1ed42e07-fa6f-872e-16cd-268f263532c0: failed to appendEntries to: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" error="failed to connect to peer: 03622733-cd5d-ccfb-64e0-3a8c894869be" + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.629+1400 [ERROR] server-1ed42e07-fa6f-872e-16cd-268f263532c0: failed to appendEntries to: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" error="failed to connect to peer: 03622733-cd5d-ccfb-64e0-3a8c894869be" raft_test.go:1320: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.162-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: starting snapshot up to: index=102 - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.162-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: starting snapshot up to: index=102 - testing.go:153: 2024-06-16T08:26:35.162-1200 [INFO] creating new snapshot: path=/tmp/raft2304783895/snapshots/2-102-1718569595162.tmp -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.162-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: compacting logs: from=1 to=92 - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.162-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.163-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: snapshot complete up to: index=102 - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.163-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:35.163-1200 [ERROR] server-30b015ac-9bf9-5785-9244-abf831fd55ad: failed to take snapshot: error="nothing new to snapshot" - testing.go:149: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:35.163-1200 [ERROR] server-30b015ac-9bf9-5785-9244-abf831fd55ad: failed to take snapshot: error="nothing new to snapshot" -testLoggerAdapter verbose: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:35.163-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: starting snapshot up to: index=99 - testing.go:149: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:35.163-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: starting snapshot up to: index=99 - testing.go:153: 2024-06-16T08:26:35.163-1200 [INFO] creating new snapshot: path=/tmp/raft2637793511/snapshots/2-99-1718569595163.tmp -testLoggerAdapter verbose: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:35.163-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: compacting logs: from=1 to=92 - testing.go:149: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:35.163-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:35.163-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: snapshot complete up to: index=99 - testing.go:149: server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: 2024-06-16T08:26:35.163-1200 [INFO] server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee: snapshot complete up to: index=99 -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.171-1200 [ERROR] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: failed to get log: index=2 error="log not found" - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.171-1200 [ERROR] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: failed to get log: index=2 error="log not found" - testing.go:153: 2024-06-16T08:26:35.171-1200 [INFO] creating new snapshot: path=/tmp/raft1561014670/snapshots/2-102-1718569595171.tmp -testLoggerAdapter verbose: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:35.172-1200 [INFO] server-30b015ac-9bf9-5785-9244-abf831fd55ad: copied to local snapshot: bytes=693 - testing.go:149: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:35.172-1200 [INFO] server-30b015ac-9bf9-5785-9244-abf831fd55ad: copied to local snapshot: bytes=693 -testLoggerAdapter verbose: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:35.172-1200 [INFO] server-30b015ac-9bf9-5785-9244-abf831fd55ad: Installed remote snapshot - testing.go:149: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:35.172-1200 [INFO] server-30b015ac-9bf9-5785-9244-abf831fd55ad: Installed remote snapshot -testLoggerAdapter verbose: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:35.172-1200 [WARN] server-30b015ac-9bf9-5785-9244-abf831fd55ad: skipping application of old log: index=102 - testing.go:149: server-30b015ac-9bf9-5785-9244-abf831fd55ad: 2024-06-16T08:26:35.172-1200 [WARN] server-30b015ac-9bf9-5785-9244-abf831fd55ad: skipping application of old log: index=102 -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.172-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: pipelining replication: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.172-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: pipelining replication: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.175-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: aborting pipeline replication: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.175-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: aborting pipeline replication: peer="{Voter server-30b015ac-9bf9-5785-9244-abf831fd55ad 30b015ac-9bf9-5785-9244-abf831fd55ad}" -testLoggerAdapter verbose: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.175-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: aborting pipeline replication: peer="{Voter server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}" - testing.go:149: server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: 2024-06-16T08:26:35.175-1200 [INFO] server-629dc9d9-a9d2-e9be-7e56-8c6a792f52e7: aborting pipeline replication: peer="{Voter server-b357c0c5-4d5a-09d9-0c03-78e414f2b6ee b357c0c5-4d5a-09d9-0c03-78e414f2b6ee}" ---- PASS: TestRaft_SendSnapshotFollower (0.28s) +testLoggerAdapter verbose: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.631+1400 [ERROR] server-03622733-cd5d-ccfb-64e0-3a8c894869be: failed to take snapshot: error="nothing new to snapshot" + testing.go:149: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.631+1400 [ERROR] server-03622733-cd5d-ccfb-64e0-3a8c894869be: failed to take snapshot: error="nothing new to snapshot" +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.631+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: starting snapshot up to: index=102 + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.631+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: starting snapshot up to: index=102 + testing.go:153: 2023-05-16T04:06:15.631+1400 [INFO] creating new snapshot: path=/tmp/raft4236312094/snapshots/2-102-1684159575631.tmp +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.632+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: compacting logs: from=1 to=92 + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.632+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.632+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: snapshot complete up to: index=102 + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.632+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.632+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: starting snapshot up to: index=30 + testing.go:149: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.632+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: starting snapshot up to: index=30 + testing.go:153: 2023-05-16T04:06:15.632+1400 [INFO] creating new snapshot: path=/tmp/raft720526378/snapshots/2-30-1684159575632.tmp +testLoggerAdapter verbose: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.633+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: compacting logs: from=1 to=30 + testing.go:149: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.633+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: compacting logs: from=1 to=30 +testLoggerAdapter verbose: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.633+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: snapshot complete up to: index=30 + testing.go:149: server-e7cba1d4-896d-0553-c5ee-41509e938548: 2023-05-16T04:06:15.633+1400 [INFO] server-e7cba1d4-896d-0553-c5ee-41509e938548: snapshot complete up to: index=30 +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.639+1400 [ERROR] server-1ed42e07-fa6f-872e-16cd-268f263532c0: failed to get log: index=2 error="log not found" + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.639+1400 [ERROR] server-1ed42e07-fa6f-872e-16cd-268f263532c0: failed to get log: index=2 error="log not found" + testing.go:153: 2023-05-16T04:06:15.640+1400 [INFO] creating new snapshot: path=/tmp/raft1387116560/snapshots/2-102-1684159575640.tmp +testLoggerAdapter verbose: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.641+1400 [INFO] server-03622733-cd5d-ccfb-64e0-3a8c894869be: copied to local snapshot: bytes=693 + testing.go:149: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.641+1400 [INFO] server-03622733-cd5d-ccfb-64e0-3a8c894869be: copied to local snapshot: bytes=693 +testLoggerAdapter verbose: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.641+1400 [INFO] server-03622733-cd5d-ccfb-64e0-3a8c894869be: Installed remote snapshot + testing.go:149: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.641+1400 [INFO] server-03622733-cd5d-ccfb-64e0-3a8c894869be: Installed remote snapshot +testLoggerAdapter verbose: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.641+1400 [WARN] server-03622733-cd5d-ccfb-64e0-3a8c894869be: skipping application of old log: index=102 + testing.go:149: server-03622733-cd5d-ccfb-64e0-3a8c894869be: 2023-05-16T04:06:15.641+1400 [WARN] server-03622733-cd5d-ccfb-64e0-3a8c894869be: skipping application of old log: index=102 +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.641+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: pipelining replication: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.641+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: pipelining replication: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.646+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: aborting pipeline replication: peer="{Voter server-e7cba1d4-896d-0553-c5ee-41509e938548 e7cba1d4-896d-0553-c5ee-41509e938548}" + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.646+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: aborting pipeline replication: peer="{Voter server-e7cba1d4-896d-0553-c5ee-41509e938548 e7cba1d4-896d-0553-c5ee-41509e938548}" +testLoggerAdapter verbose: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.646+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: aborting pipeline replication: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" + testing.go:149: server-1ed42e07-fa6f-872e-16cd-268f263532c0: 2023-05-16T04:06:15.646+1400 [INFO] server-1ed42e07-fa6f-872e-16cd-268f263532c0: aborting pipeline replication: peer="{Voter server-03622733-cd5d-ccfb-64e0-3a8c894869be 03622733-cd5d-ccfb-64e0-3a8c894869be}" +--- PASS: TestRaft_SendSnapshotFollower (0.29s) === RUN TestRaft_SendSnapshotAndLogsFollower -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.176-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb Address:6fc5843f-7c3f-e456-7d8e-964684ef9cbb} {Suffrage:Voter ID:server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 Address:6b1b6425-c8ad-18af-bcf5-16ce97c318c5} {Suffrage:Voter ID:server-61080878-7c41-ecce-3952-326e5e3bf68b Address:61080878-7c41-ecce-3952-326e5e3bf68b}]" - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.176-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb Address:6fc5843f-7c3f-e456-7d8e-964684ef9cbb} {Suffrage:Voter ID:server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 Address:6b1b6425-c8ad-18af-bcf5-16ce97c318c5} {Suffrage:Voter ID:server-61080878-7c41-ecce-3952-326e5e3bf68b Address:61080878-7c41-ecce-3952-326e5e3bf68b}]" -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.176-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: entering follower state: follower="Node at 6fc5843f-7c3f-e456-7d8e-964684ef9cbb [Follower]" leader= - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.176-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: entering follower state: follower="Node at 6fc5843f-7c3f-e456-7d8e-964684ef9cbb [Follower]" leader= -testLoggerAdapter verbose: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.177-1200 [INFO] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb Address:6fc5843f-7c3f-e456-7d8e-964684ef9cbb} {Suffrage:Voter ID:server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 Address:6b1b6425-c8ad-18af-bcf5-16ce97c318c5} {Suffrage:Voter ID:server-61080878-7c41-ecce-3952-326e5e3bf68b Address:61080878-7c41-ecce-3952-326e5e3bf68b}]" - testing.go:149: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.177-1200 [INFO] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb Address:6fc5843f-7c3f-e456-7d8e-964684ef9cbb} {Suffrage:Voter ID:server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 Address:6b1b6425-c8ad-18af-bcf5-16ce97c318c5} {Suffrage:Voter ID:server-61080878-7c41-ecce-3952-326e5e3bf68b Address:61080878-7c41-ecce-3952-326e5e3bf68b}]" -testLoggerAdapter verbose: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.177-1200 [INFO] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: entering follower state: follower="Node at 6b1b6425-c8ad-18af-bcf5-16ce97c318c5 [Follower]" leader= - testing.go:149: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.177-1200 [INFO] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: entering follower state: follower="Node at 6b1b6425-c8ad-18af-bcf5-16ce97c318c5 [Follower]" leader= -testLoggerAdapter verbose: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.177-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb Address:6fc5843f-7c3f-e456-7d8e-964684ef9cbb} {Suffrage:Voter ID:server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 Address:6b1b6425-c8ad-18af-bcf5-16ce97c318c5} {Suffrage:Voter ID:server-61080878-7c41-ecce-3952-326e5e3bf68b Address:61080878-7c41-ecce-3952-326e5e3bf68b}]" - testing.go:149: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.177-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb Address:6fc5843f-7c3f-e456-7d8e-964684ef9cbb} {Suffrage:Voter ID:server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 Address:6b1b6425-c8ad-18af-bcf5-16ce97c318c5} {Suffrage:Voter ID:server-61080878-7c41-ecce-3952-326e5e3bf68b Address:61080878-7c41-ecce-3952-326e5e3bf68b}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.177-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:35.177-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.177-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: entering follower state: follower="Node at 61080878-7c41-ecce-3952-326e5e3bf68b [Follower]" leader= - testing.go:149: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.177-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: entering follower state: follower="Node at 61080878-7c41-ecce-3952-326e5e3bf68b [Follower]" leader= -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [WARN] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [WARN] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: entering candidate state: node="Node at 6fc5843f-7c3f-e456-7d8e-964684ef9cbb [Candidate]" term=2 - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: entering candidate state: node="Node at 6fc5843f-7c3f-e456-7d8e-964684ef9cbb [Candidate]" term=2 -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: election won: tally=2 - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: election won: tally=2 -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: entering leader state: leader="Node at 6fc5843f-7c3f-e456-7d8e-964684ef9cbb [Leader]" - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: entering leader state: leader="Node at 6fc5843f-7c3f-e456-7d8e-964684ef9cbb [Leader]" -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: added peer, starting replication: peer=server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: added peer, starting replication: peer=server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: added peer, starting replication: peer=server-61080878-7c41-ecce-3952-326e5e3bf68b - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: added peer, starting replication: peer=server-61080878-7c41-ecce-3952-326e5e3bf68b -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: pipelining replication: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: pipelining replication: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: pipelining replication: peer="{Voter server-61080878-7c41-ecce-3952-326e5e3bf68b 61080878-7c41-ecce-3952-326e5e3bf68b}" - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.243-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: pipelining replication: peer="{Voter server-61080878-7c41-ecce-3952-326e5e3bf68b 61080878-7c41-ecce-3952-326e5e3bf68b}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.349-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:35.24366656 -1200 -12 m=+12.250983041 (2 nodes), 66.24922ms from start of poll, 67.05874ms from cluster start. Timeout at 2024-06-16 08:26:35.34949594 -1200 -12 m=+12.356812421, 105.82938ms after stability - testing.go:149: cluster: 2024-06-16T08:26:35.349-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:35.24366656 -1200 -12 m=+12.250983041 (2 nodes), 66.24922ms from start of poll, 67.05874ms from cluster start. Timeout at 2024-06-16 08:26:35.34949594 -1200 -12 m=+12.356812421, 105.82938ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.349-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:35.349-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.455-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:35.34973602 -1200 -12 m=+12.357052541 (1 nodes), 1.54µs from start of poll, 173.12824ms from cluster start. Timeout at 2024-06-16 08:26:35.45494422 -1200 -12 m=+12.462260701, 105.20816ms after stability - testing.go:149: cluster: 2024-06-16T08:26:35.455-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:35.34973602 -1200 -12 m=+12.357052541 (1 nodes), 1.54µs from start of poll, 173.12824ms from cluster start. Timeout at 2024-06-16 08:26:35.45494422 -1200 -12 m=+12.462260701, 105.20816ms after stability -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.455-1200 [ERROR] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: failed to pipeline appendEntries: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" error="append pipeline closed" - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.455-1200 [ERROR] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: failed to pipeline appendEntries: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" error="append pipeline closed" -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.455-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: aborting pipeline replication: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.455-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: aborting pipeline replication: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.455-1200 [ERROR] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: failed to appendEntries to: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" error="failed to connect to peer: 6b1b6425-c8ad-18af-bcf5-16ce97c318c5" - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.455-1200 [ERROR] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: failed to appendEntries to: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" error="failed to connect to peer: 6b1b6425-c8ad-18af-bcf5-16ce97c318c5" +testLoggerAdapter verbose: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.648+1400 [INFO] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcae9302-e6da-e3b6-97fa-5a309038ac87 Address:bcae9302-e6da-e3b6-97fa-5a309038ac87} {Suffrage:Voter ID:server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 Address:6b3433b5-5490-c2d5-6dad-8df73d1d7285} {Suffrage:Voter ID:server-95aabfe7-fa96-a83a-9d30-011608429910 Address:95aabfe7-fa96-a83a-9d30-011608429910}]" + testing.go:149: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.648+1400 [INFO] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcae9302-e6da-e3b6-97fa-5a309038ac87 Address:bcae9302-e6da-e3b6-97fa-5a309038ac87} {Suffrage:Voter ID:server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 Address:6b3433b5-5490-c2d5-6dad-8df73d1d7285} {Suffrage:Voter ID:server-95aabfe7-fa96-a83a-9d30-011608429910 Address:95aabfe7-fa96-a83a-9d30-011608429910}]" +testLoggerAdapter verbose: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.648+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcae9302-e6da-e3b6-97fa-5a309038ac87 Address:bcae9302-e6da-e3b6-97fa-5a309038ac87} {Suffrage:Voter ID:server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 Address:6b3433b5-5490-c2d5-6dad-8df73d1d7285} {Suffrage:Voter ID:server-95aabfe7-fa96-a83a-9d30-011608429910 Address:95aabfe7-fa96-a83a-9d30-011608429910}]" + testing.go:149: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.648+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcae9302-e6da-e3b6-97fa-5a309038ac87 Address:bcae9302-e6da-e3b6-97fa-5a309038ac87} {Suffrage:Voter ID:server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 Address:6b3433b5-5490-c2d5-6dad-8df73d1d7285} {Suffrage:Voter ID:server-95aabfe7-fa96-a83a-9d30-011608429910 Address:95aabfe7-fa96-a83a-9d30-011608429910}]" +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.649+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcae9302-e6da-e3b6-97fa-5a309038ac87 Address:bcae9302-e6da-e3b6-97fa-5a309038ac87} {Suffrage:Voter ID:server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 Address:6b3433b5-5490-c2d5-6dad-8df73d1d7285} {Suffrage:Voter ID:server-95aabfe7-fa96-a83a-9d30-011608429910 Address:95aabfe7-fa96-a83a-9d30-011608429910}]" + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.649+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcae9302-e6da-e3b6-97fa-5a309038ac87 Address:bcae9302-e6da-e3b6-97fa-5a309038ac87} {Suffrage:Voter ID:server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 Address:6b3433b5-5490-c2d5-6dad-8df73d1d7285} {Suffrage:Voter ID:server-95aabfe7-fa96-a83a-9d30-011608429910 Address:95aabfe7-fa96-a83a-9d30-011608429910}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.649+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:15.649+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.649+1400 [INFO] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: entering follower state: follower="Node at bcae9302-e6da-e3b6-97fa-5a309038ac87 [Follower]" leader= + testing.go:149: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.649+1400 [INFO] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: entering follower state: follower="Node at bcae9302-e6da-e3b6-97fa-5a309038ac87 [Follower]" leader= +testLoggerAdapter verbose: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.649+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: entering follower state: follower="Node at 6b3433b5-5490-c2d5-6dad-8df73d1d7285 [Follower]" leader= + testing.go:149: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.649+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: entering follower state: follower="Node at 6b3433b5-5490-c2d5-6dad-8df73d1d7285 [Follower]" leader= +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.649+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: entering follower state: follower="Node at 95aabfe7-fa96-a83a-9d30-011608429910 [Follower]" leader= + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.649+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: entering follower state: follower="Node at 95aabfe7-fa96-a83a-9d30-011608429910 [Follower]" leader= +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.709+1400 [WARN] server-95aabfe7-fa96-a83a-9d30-011608429910: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.709+1400 [WARN] server-95aabfe7-fa96-a83a-9d30-011608429910: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: entering candidate state: node="Node at 95aabfe7-fa96-a83a-9d30-011608429910 [Candidate]" term=2 + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: entering candidate state: node="Node at 95aabfe7-fa96-a83a-9d30-011608429910 [Candidate]" term=2 +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: election won: tally=2 + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: election won: tally=2 +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: entering leader state: leader="Node at 95aabfe7-fa96-a83a-9d30-011608429910 [Leader]" + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: entering leader state: leader="Node at 95aabfe7-fa96-a83a-9d30-011608429910 [Leader]" +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: added peer, starting replication: peer=server-bcae9302-e6da-e3b6-97fa-5a309038ac87 + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: added peer, starting replication: peer=server-bcae9302-e6da-e3b6-97fa-5a309038ac87 +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: added peer, starting replication: peer=server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: added peer, starting replication: peer=server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: pipelining replication: peer="{Voter server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 6b3433b5-5490-c2d5-6dad-8df73d1d7285}" + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: pipelining replication: peer="{Voter server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 6b3433b5-5490-c2d5-6dad-8df73d1d7285}" +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: pipelining replication: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.710+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: pipelining replication: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.817+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:15.711053027 +1400 +14 m=+12.513634642 (2 nodes), 61.85243ms from start of poll, 62.786761ms from cluster start. Timeout at 2023-05-16 04:06:15.816984217 +1400 +14 m=+12.619565852, 105.93121ms after stability + testing.go:149: cluster: 2023-05-16T04:06:15.817+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:15.711053027 +1400 +14 m=+12.513634642 (2 nodes), 61.85243ms from start of poll, 62.786761ms from cluster start. Timeout at 2023-05-16 04:06:15.816984217 +1400 +14 m=+12.619565852, 105.93121ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.817+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:15.817+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:15.923+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:15.81723556 +1400 +14 m=+12.619817175 (1 nodes), 3.24µs from start of poll, 168.969294ms from cluster start. Timeout at 2023-05-16 04:06:15.923215331 +1400 +14 m=+12.725796966, 105.979791ms after stability + testing.go:149: cluster: 2023-05-16T04:06:15.923+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:15.81723556 +1400 +14 m=+12.619817175 (1 nodes), 3.24µs from start of poll, 168.969294ms from cluster start. Timeout at 2023-05-16 04:06:15.923215331 +1400 +14 m=+12.725796966, 105.979791ms after stability +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.923+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to pipeline appendEntries: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" error="append pipeline closed" + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.923+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to pipeline appendEntries: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" error="append pipeline closed" +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.923+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: aborting pipeline replication: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.923+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: aborting pipeline replication: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.923+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to appendEntries to: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" error="failed to connect to peer: bcae9302-e6da-e3b6-97fa-5a309038ac87" + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.923+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to appendEntries to: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" error="failed to connect to peer: bcae9302-e6da-e3b6-97fa-5a309038ac87" +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.925+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to heartbeat to: peer=bcae9302-e6da-e3b6-97fa-5a309038ac87 error="failed to connect to peer: bcae9302-e6da-e3b6-97fa-5a309038ac87" + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.925+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to heartbeat to: peer=bcae9302-e6da-e3b6-97fa-5a309038ac87 error="failed to connect to peer: bcae9302-e6da-e3b6-97fa-5a309038ac87" raft_test.go:1362: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.457-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: starting snapshot up to: index=102 - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.457-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: starting snapshot up to: index=102 - testing.go:153: 2024-06-16T08:26:35.457-1200 [INFO] creating new snapshot: path=/tmp/raft2529079893/snapshots/2-102-1718569595457.tmp -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.457-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: compacting logs: from=1 to=92 - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.457-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.458-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: snapshot complete up to: index=102 - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.458-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.458-1200 [ERROR] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: failed to take snapshot: error="nothing new to snapshot" - testing.go:149: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.458-1200 [ERROR] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: failed to take snapshot: error="nothing new to snapshot" -testLoggerAdapter verbose: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.458-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: starting snapshot up to: index=99 - testing.go:149: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.458-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: starting snapshot up to: index=99 - testing.go:153: 2024-06-16T08:26:35.458-1200 [INFO] creating new snapshot: path=/tmp/raft3706296007/snapshots/2-99-1718569595458.tmp -testLoggerAdapter verbose: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.458-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: compacting logs: from=1 to=92 - testing.go:149: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.458-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.458-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: snapshot complete up to: index=99 - testing.go:149: server-61080878-7c41-ecce-3952-326e5e3bf68b: 2024-06-16T08:26:35.458-1200 [INFO] server-61080878-7c41-ecce-3952-326e5e3bf68b: snapshot complete up to: index=99 -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.460-1200 [ERROR] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: failed to heartbeat to: peer=6b1b6425-c8ad-18af-bcf5-16ce97c318c5 error="failed to connect to peer: 6b1b6425-c8ad-18af-bcf5-16ce97c318c5" - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.460-1200 [ERROR] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: failed to heartbeat to: peer=6b1b6425-c8ad-18af-bcf5-16ce97c318c5 error="failed to connect to peer: 6b1b6425-c8ad-18af-bcf5-16ce97c318c5" +testLoggerAdapter verbose: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.927+1400 [ERROR] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: failed to take snapshot: error="nothing new to snapshot" + testing.go:149: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.927+1400 [ERROR] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: failed to take snapshot: error="nothing new to snapshot" +testLoggerAdapter verbose: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.927+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: starting snapshot up to: index=101 + testing.go:149: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.927+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: starting snapshot up to: index=101 + testing.go:153: 2023-05-16T04:06:15.927+1400 [INFO] creating new snapshot: path=/tmp/raft303909923/snapshots/2-101-1684159575927.tmp +testLoggerAdapter verbose: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.928+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: compacting logs: from=1 to=92 + testing.go:149: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.928+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.928+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: snapshot complete up to: index=101 + testing.go:149: server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: 2023-05-16T04:06:15.928+1400 [INFO] server-6b3433b5-5490-c2d5-6dad-8df73d1d7285: snapshot complete up to: index=101 +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.929+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: starting snapshot up to: index=102 + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.929+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: starting snapshot up to: index=102 + testing.go:153: 2023-05-16T04:06:15.929+1400 [INFO] creating new snapshot: path=/tmp/raft419147949/snapshots/2-102-1684159575929.tmp +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.929+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: compacting logs: from=1 to=92 + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.929+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.929+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: snapshot complete up to: index=102 + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.929+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: snapshot complete up to: index=102 raft_test.go:1383: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.466-1200 [ERROR] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: failed to get log: index=2 error="log not found" - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.466-1200 [ERROR] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: failed to get log: index=2 error="log not found" - testing.go:153: 2024-06-16T08:26:35.466-1200 [INFO] creating new snapshot: path=/tmp/raft1240183984/snapshots/2-102-1718569595466.tmp -testLoggerAdapter verbose: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.467-1200 [INFO] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: copied to local snapshot: bytes=693 - testing.go:149: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.467-1200 [INFO] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: copied to local snapshot: bytes=693 -testLoggerAdapter verbose: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.467-1200 [INFO] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: Installed remote snapshot - testing.go:149: server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: 2024-06-16T08:26:35.467-1200 [INFO] server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5: Installed remote snapshot -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.468-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: pipelining replication: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.468-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: pipelining replication: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.472-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: aborting pipeline replication: peer="{Voter server-61080878-7c41-ecce-3952-326e5e3bf68b 61080878-7c41-ecce-3952-326e5e3bf68b}" - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.472-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: aborting pipeline replication: peer="{Voter server-61080878-7c41-ecce-3952-326e5e3bf68b 61080878-7c41-ecce-3952-326e5e3bf68b}" -testLoggerAdapter verbose: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.472-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: aborting pipeline replication: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" - testing.go:149: server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: 2024-06-16T08:26:35.472-1200 [INFO] server-6fc5843f-7c3f-e456-7d8e-964684ef9cbb: aborting pipeline replication: peer="{Voter server-6b1b6425-c8ad-18af-bcf5-16ce97c318c5 6b1b6425-c8ad-18af-bcf5-16ce97c318c5}" ---- PASS: TestRaft_SendSnapshotAndLogsFollower (0.30s) +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.934+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to get log: index=2 error="log not found" + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.934+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to get log: index=2 error="log not found" + testing.go:153: 2023-05-16T04:06:15.938+1400 [INFO] creating new snapshot: path=/tmp/raft4039688190/snapshots/2-102-1684159575938.tmp +testLoggerAdapter verbose: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.939+1400 [INFO] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: copied to local snapshot: bytes=693 + testing.go:149: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.939+1400 [INFO] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: copied to local snapshot: bytes=693 +testLoggerAdapter verbose: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.940+1400 [INFO] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: Installed remote snapshot + testing.go:149: server-bcae9302-e6da-e3b6-97fa-5a309038ac87: 2023-05-16T04:06:15.940+1400 [INFO] server-bcae9302-e6da-e3b6-97fa-5a309038ac87: Installed remote snapshot +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.941+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: pipelining replication: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.941+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: pipelining replication: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.950+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: aborting pipeline replication: peer="{Voter server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 6b3433b5-5490-c2d5-6dad-8df73d1d7285}" + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.950+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: aborting pipeline replication: peer="{Voter server-6b3433b5-5490-c2d5-6dad-8df73d1d7285 6b3433b5-5490-c2d5-6dad-8df73d1d7285}" +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.950+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: aborting pipeline replication: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:15.950+1400 [INFO] server-95aabfe7-fa96-a83a-9d30-011608429910: aborting pipeline replication: peer="{Voter server-bcae9302-e6da-e3b6-97fa-5a309038ac87 bcae9302-e6da-e3b6-97fa-5a309038ac87}" +testLoggerAdapter verbose: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:16.000+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to heartbeat to: peer=6b3433b5-5490-c2d5-6dad-8df73d1d7285 error="command timed out" + testing.go:149: server-95aabfe7-fa96-a83a-9d30-011608429910: 2023-05-16T04:06:16.000+1400 [ERROR] server-95aabfe7-fa96-a83a-9d30-011608429910: failed to heartbeat to: peer=6b3433b5-5490-c2d5-6dad-8df73d1d7285 error="command timed out" +--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.35s) === RUN TestRaft_ReJoinFollower -testLoggerAdapter verbose: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.474-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 Address:b2af09c4-4aa9-b4f3-a665-b75c2004eae1} {Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac}]" - testing.go:149: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.474-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 Address:b2af09c4-4aa9-b4f3-a665-b75c2004eae1} {Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac}]" -testLoggerAdapter verbose: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.474-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: entering follower state: follower="Node at b2af09c4-4aa9-b4f3-a665-b75c2004eae1 [Follower]" leader= - testing.go:149: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.474-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: entering follower state: follower="Node at b2af09c4-4aa9-b4f3-a665-b75c2004eae1 [Follower]" leader= -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.474-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 Address:b2af09c4-4aa9-b4f3-a665-b75c2004eae1} {Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac}]" - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.474-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 Address:b2af09c4-4aa9-b4f3-a665-b75c2004eae1} {Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac}]" -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.474-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering follower state: follower="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Follower]" leader= - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.474-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering follower state: follower="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Follower]" leader= -testLoggerAdapter verbose: server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: 2024-06-16T08:26:35.474-1200 [INFO] server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 Address:b2af09c4-4aa9-b4f3-a665-b75c2004eae1} {Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac}]" - testing.go:149: server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: 2024-06-16T08:26:35.474-1200 [INFO] server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 Address:b2af09c4-4aa9-b4f3-a665-b75c2004eae1} {Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.474-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:35.474-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: 2024-06-16T08:26:35.474-1200 [INFO] server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: entering follower state: follower="Node at 59c5c5c4-b16d-c323-cd3f-1756057bd3ac [Follower]" leader= - testing.go:149: server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: 2024-06-16T08:26:35.474-1200 [INFO] server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: entering follower state: follower="Node at 59c5c5c4-b16d-c323-cd3f-1756057bd3ac [Follower]" leader= -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering candidate state: node="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Candidate]" term=2 - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering candidate state: node="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Candidate]" term=2 -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: election won: tally=2 - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: election won: tally=2 -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering leader state: leader="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Leader]" - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering leader state: leader="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Leader]" -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: pipelining replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: pipelining replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: pipelining replication: peer="{Voter server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac 59c5c5c4-b16d-c323-cd3f-1756057bd3ac}" - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.545-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: pipelining replication: peer="{Voter server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac 59c5c5c4-b16d-c323-cd3f-1756057bd3ac}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.650-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:35.54573382 -1200 -12 m=+12.553050301 (1 nodes), 71.03842ms from start of poll, 71.89454ms from cluster start. Timeout at 2024-06-16 08:26:35.65074902 -1200 -12 m=+12.658065501, 105.0152ms after stability - testing.go:149: cluster: 2024-06-16T08:26:35.650-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:35.54573382 -1200 -12 m=+12.553050301 (1 nodes), 71.03842ms from start of poll, 71.89454ms from cluster start. Timeout at 2024-06-16 08:26:35.65074902 -1200 -12 m=+12.658065501, 105.0152ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.656-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:35.656-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.761-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:35.65633362 -1200 -12 m=+12.663650101 (2 nodes), 3.12µs from start of poll, 182.49434ms from cluster start. Timeout at 2024-06-16 08:26:35.76134064 -1200 -12 m=+12.768657121, 105.00702ms after stability - testing.go:149: cluster: 2024-06-16T08:26:35.761-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:35.65633362 -1200 -12 m=+12.663650101 (2 nodes), 3.12µs from start of poll, 182.49434ms from cluster start. Timeout at 2024-06-16 08:26:35.76134064 -1200 -12 m=+12.768657121, 105.00702ms after stability -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.761-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: updating configuration: command=RemoveServer server-id=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 server-addr= servers="[{Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac}]" - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.761-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: updating configuration: command=RemoveServer server-id=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 server-addr= servers="[{Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac}]" -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.761-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: removed peer, stopping replication: peer=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 last-index=3 - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.761-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: removed peer, stopping replication: peer=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 last-index=3 -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.761-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: aborting pipeline replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.761-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: aborting pipeline replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" -testLoggerAdapter verbose: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.823-1200 [WARN] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: heartbeat timeout reached, starting election: last-leader=d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 - testing.go:149: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.823-1200 [WARN] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: heartbeat timeout reached, starting election: last-leader=d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 -testLoggerAdapter verbose: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.823-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: entering candidate state: node="Node at b2af09c4-4aa9-b4f3-a665-b75c2004eae1 [Candidate]" term=3 - testing.go:149: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.823-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: entering candidate state: node="Node at b2af09c4-4aa9-b4f3-a665-b75c2004eae1 [Candidate]" term=3 -testLoggerAdapter verbose: server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: 2024-06-16T08:26:35.823-1200 [WARN] server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: rejecting vote request since we have a leader: from=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 leader=d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 - testing.go:149: server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: 2024-06-16T08:26:35.823-1200 [WARN] server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: rejecting vote request since we have a leader: from=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 leader=d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.823-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: rejecting vote request since we have a leader: from=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 leader=d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.823-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: rejecting vote request since we have a leader: from=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 leader=d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.867-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: updating configuration: command=AddStaging server-id=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 server-addr=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 servers="[{Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac} {Suffrage:Voter ID:server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 Address:b2af09c4-4aa9-b4f3-a665-b75c2004eae1}]" - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.867-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: updating configuration: command=AddStaging server-id=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 server-addr=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 servers="[{Suffrage:Voter ID:server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 Address:d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3} {Suffrage:Voter ID:server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac Address:59c5c5c4-b16d-c323-cd3f-1756057bd3ac} {Suffrage:Voter ID:server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 Address:b2af09c4-4aa9-b4f3-a665-b75c2004eae1}]" -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.867-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.867-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:35.867-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:35.867-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.867-1200 [ERROR] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: peer has newer term, stopping replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.867-1200 [ERROR] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: peer has newer term, stopping replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.867-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering follower state: follower="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Follower]" leader= - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.867-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering follower state: follower="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Follower]" leader= -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.868-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: aborting pipeline replication: peer="{Voter server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac 59c5c5c4-b16d-c323-cd3f-1756057bd3ac}" - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.868-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: aborting pipeline replication: peer="{Voter server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac 59c5c5c4-b16d-c323-cd3f-1756057bd3ac}" -testLoggerAdapter verbose: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.910-1200 [WARN] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: Election timeout reached, restarting election - testing.go:149: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.910-1200 [WARN] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: Election timeout reached, restarting election -testLoggerAdapter verbose: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.911-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: entering candidate state: node="Node at b2af09c4-4aa9-b4f3-a665-b75c2004eae1 [Candidate]" term=4 - testing.go:149: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.911-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: entering candidate state: node="Node at b2af09c4-4aa9-b4f3-a665-b75c2004eae1 [Candidate]" term=4 -testLoggerAdapter verbose: server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: 2024-06-16T08:26:35.911-1200 [WARN] server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: rejecting vote request since we have a leader: from=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 leader=d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 - testing.go:149: server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: 2024-06-16T08:26:35.911-1200 [WARN] server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac: rejecting vote request since we have a leader: from=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 leader=d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.911-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: rejecting vote request since our last index is greater: candidate=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 last-index=4 last-candidate-index=3 - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.911-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: rejecting vote request since our last index is greater: candidate=b2af09c4-4aa9-b4f3-a665-b75c2004eae1 last-index=4 last-candidate-index=3 -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering candidate state: node="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Candidate]" term=5 - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering candidate state: node="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Candidate]" term=5 -testLoggerAdapter verbose: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.923-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: entering follower state: follower="Node at b2af09c4-4aa9-b4f3-a665-b75c2004eae1 [Follower]" leader= - testing.go:149: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.923-1200 [INFO] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: entering follower state: follower="Node at b2af09c4-4aa9-b4f3-a665-b75c2004eae1 [Follower]" leader= -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: election won: tally=2 - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: election won: tally=2 -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering leader state: leader="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Leader]" - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: entering leader state: leader="Node at d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3 [Leader]" -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: added peer, starting replication: peer=server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: pipelining replication: peer="{Voter server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac 59c5c5c4-b16d-c323-cd3f-1756057bd3ac}" - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.923-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: pipelining replication: peer="{Voter server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac 59c5c5c4-b16d-c323-cd3f-1756057bd3ac}" -testLoggerAdapter verbose: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.923-1200 [WARN] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: failed to get previous log: previous-index=4 last-index=3 error="log not found" - testing.go:149: server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: 2024-06-16T08:26:35.923-1200 [WARN] server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1: failed to get previous log: previous-index=4 last-index=3 error="log not found" -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.924-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: appendEntries rejected, sending older logs: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" next=4 - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.924-1200 [WARN] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: appendEntries rejected, sending older logs: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" next=4 -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.924-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: pipelining replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:35.924-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: pipelining replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.029-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:35.92372826 -1200 -12 m=+12.931044741 (1 nodes), 55.83098ms from start of poll, 449.88898ms from cluster start. Timeout at 2024-06-16 08:26:36.02949294 -1200 -12 m=+13.036809441, 105.7647ms after stability - testing.go:149: cluster: 2024-06-16T08:26:36.029-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:35.92372826 -1200 -12 m=+12.931044741 (1 nodes), 55.83098ms from start of poll, 449.88898ms from cluster start. Timeout at 2024-06-16 08:26:36.02949294 -1200 -12 m=+13.036809441, 105.7647ms after stability -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:36.029-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: aborting pipeline replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:36.029-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: aborting pipeline replication: peer="{Voter server-b2af09c4-4aa9-b4f3-a665-b75c2004eae1 b2af09c4-4aa9-b4f3-a665-b75c2004eae1}" -testLoggerAdapter verbose: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:36.029-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: aborting pipeline replication: peer="{Voter server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac 59c5c5c4-b16d-c323-cd3f-1756057bd3ac}" - testing.go:149: server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: 2024-06-16T08:26:36.029-1200 [INFO] server-d9c5321a-af6e-a3ff-ae74-fb08b9e8e3a3: aborting pipeline replication: peer="{Voter server-59c5c5c4-b16d-c323-cd3f-1756057bd3ac 59c5c5c4-b16d-c323-cd3f-1756057bd3ac}" +testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.002+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e074892-d43a-9b83-058c-4c6dc15f2751 Address:7e074892-d43a-9b83-058c-4c6dc15f2751} {Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7}]" + testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.002+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e074892-d43a-9b83-058c-4c6dc15f2751 Address:7e074892-d43a-9b83-058c-4c6dc15f2751} {Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7}]" +testLoggerAdapter verbose: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.002+1400 [INFO] server-3018a7db-32dc-5433-aac3-a047171aacd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e074892-d43a-9b83-058c-4c6dc15f2751 Address:7e074892-d43a-9b83-058c-4c6dc15f2751} {Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7}]" + testing.go:149: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.002+1400 [INFO] server-3018a7db-32dc-5433-aac3-a047171aacd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e074892-d43a-9b83-058c-4c6dc15f2751 Address:7e074892-d43a-9b83-058c-4c6dc15f2751} {Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7}]" +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.002+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e074892-d43a-9b83-058c-4c6dc15f2751 Address:7e074892-d43a-9b83-058c-4c6dc15f2751} {Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7}]" + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.002+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e074892-d43a-9b83-058c-4c6dc15f2751 Address:7e074892-d43a-9b83-058c-4c6dc15f2751} {Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.002+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:16.002+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.002+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering follower state: follower="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Follower]" leader= + testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.002+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering follower state: follower="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Follower]" leader= +testLoggerAdapter verbose: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.003+1400 [INFO] server-3018a7db-32dc-5433-aac3-a047171aacd6: entering follower state: follower="Node at 3018a7db-32dc-5433-aac3-a047171aacd6 [Follower]" leader= + testing.go:149: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.003+1400 [INFO] server-3018a7db-32dc-5433-aac3-a047171aacd6: entering follower state: follower="Node at 3018a7db-32dc-5433-aac3-a047171aacd6 [Follower]" leader= +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.003+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering follower state: follower="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Follower]" leader= + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.003+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering follower state: follower="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Follower]" leader= +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.057+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.057+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering candidate state: node="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Candidate]" term=2 + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering candidate state: node="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Candidate]" term=2 +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: election won: tally=2 + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: election won: tally=2 +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering leader state: leader="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Leader]" + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering leader state: leader="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Leader]" +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-7e074892-d43a-9b83-058c-4c6dc15f2751 + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-7e074892-d43a-9b83-058c-4c6dc15f2751 +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-3018a7db-32dc-5433-aac3-a047171aacd6 + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-3018a7db-32dc-5433-aac3-a047171aacd6 +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: pipelining replication: peer="{Voter server-3018a7db-32dc-5433-aac3-a047171aacd6 3018a7db-32dc-5433-aac3-a047171aacd6}" + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.058+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: pipelining replication: peer="{Voter server-3018a7db-32dc-5433-aac3-a047171aacd6 3018a7db-32dc-5433-aac3-a047171aacd6}" +testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.058+1400 [WARN] server-7e074892-d43a-9b83-058c-4c6dc15f2751: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.058+1400 [WARN] server-7e074892-d43a-9b83-058c-4c6dc15f2751: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.059+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering candidate state: node="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Candidate]" term=2 + testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.059+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering candidate state: node="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Candidate]" term=2 +testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.059+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: duplicate requestVote for same term: term=2 + testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.059+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: duplicate requestVote for same term: term=2 +testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.059+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering follower state: follower="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Follower]" leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 + testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.059+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering follower state: follower="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Follower]" leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.059+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: pipelining replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.059+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: pipelining replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.059+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.059+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 +testLoggerAdapter verbose: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.059+1400 [WARN] server-3018a7db-32dc-5433-aac3-a047171aacd6: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 + testing.go:149: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.059+1400 [WARN] server-3018a7db-32dc-5433-aac3-a047171aacd6: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.165+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:16.059768842 +1400 +14 m=+12.862350457 (1 nodes), 56.825271ms from start of poll, 57.867983ms from cluster start. Timeout at 2023-05-16 04:06:16.165852534 +1400 +14 m=+12.968434189, 106.083732ms after stability + testing.go:149: cluster: 2023-05-16T04:06:16.165+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:16.059768842 +1400 +14 m=+12.862350457 (1 nodes), 56.825271ms from start of poll, 57.867983ms from cluster start. Timeout at 2023-05-16 04:06:16.165852534 +1400 +14 m=+12.968434189, 106.083732ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.171+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:16.171+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.277+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:16.171784324 +1400 +14 m=+12.974365959 (2 nodes), 4.58µs from start of poll, 169.883485ms from cluster start. Timeout at 2023-05-16 04:06:16.277510992 +1400 +14 m=+13.080092627, 105.726668ms after stability + testing.go:149: cluster: 2023-05-16T04:06:16.277+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:16.171784324 +1400 +14 m=+12.974365959 (2 nodes), 4.58µs from start of poll, 169.883485ms from cluster start. Timeout at 2023-05-16 04:06:16.277510992 +1400 +14 m=+13.080092627, 105.726668ms after stability +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.277+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: updating configuration: command=RemoveServer server-id=server-7e074892-d43a-9b83-058c-4c6dc15f2751 server-addr= servers="[{Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7}]" + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.277+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: updating configuration: command=RemoveServer server-id=server-7e074892-d43a-9b83-058c-4c6dc15f2751 server-addr= servers="[{Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7}]" +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.277+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: removed peer, stopping replication: peer=server-7e074892-d43a-9b83-058c-4c6dc15f2751 last-index=3 + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.277+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: removed peer, stopping replication: peer=server-7e074892-d43a-9b83-058c-4c6dc15f2751 last-index=3 +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.278+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: aborting pipeline replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.278+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: aborting pipeline replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" +testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.340+1400 [WARN] server-7e074892-d43a-9b83-058c-4c6dc15f2751: heartbeat timeout reached, starting election: last-leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 + testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.340+1400 [WARN] server-7e074892-d43a-9b83-058c-4c6dc15f2751: heartbeat timeout reached, starting election: last-leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 +testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.341+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering candidate state: node="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Candidate]" term=3 + testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.341+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering candidate state: node="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Candidate]" term=3 +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.341+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.341+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 +testLoggerAdapter verbose: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.341+1400 [WARN] server-3018a7db-32dc-5433-aac3-a047171aacd6: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 + testing.go:149: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.341+1400 [WARN] server-3018a7db-32dc-5433-aac3-a047171aacd6: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.384+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: updating configuration: command=AddStaging server-id=server-7e074892-d43a-9b83-058c-4c6dc15f2751 server-addr=7e074892-d43a-9b83-058c-4c6dc15f2751 servers="[{Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7} {Suffrage:Voter ID:server-7e074892-d43a-9b83-058c-4c6dc15f2751 Address:7e074892-d43a-9b83-058c-4c6dc15f2751}]" + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.384+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: updating configuration: command=AddStaging server-id=server-7e074892-d43a-9b83-058c-4c6dc15f2751 server-addr=7e074892-d43a-9b83-058c-4c6dc15f2751 servers="[{Suffrage:Voter ID:server-3018a7db-32dc-5433-aac3-a047171aacd6 Address:3018a7db-32dc-5433-aac3-a047171aacd6} {Suffrage:Voter ID:server-7f0d116e-6b51-5f11-8a07-db536b6189b7 Address:7f0d116e-6b51-5f11-8a07-db536b6189b7} {Suffrage:Voter ID:server-7e074892-d43a-9b83-058c-4c6dc15f2751 Address:7e074892-d43a-9b83-058c-4c6dc15f2751}]" +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.384+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-7e074892-d43a-9b83-058c-4c6dc15f2751 + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.384+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-7e074892-d43a-9b83-058c-4c6dc15f2751 +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.384+1400 [ERROR] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: peer has newer term, stopping replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.384+1400 [ERROR] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: peer has newer term, stopping replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.385+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering follower state: follower="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Follower]" leader= + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.385+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering follower state: follower="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Follower]" leader= +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.385+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:16.385+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.385+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: aborting pipeline replication: peer="{Voter server-3018a7db-32dc-5433-aac3-a047171aacd6 3018a7db-32dc-5433-aac3-a047171aacd6}" + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.385+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: aborting pipeline replication: peer="{Voter server-3018a7db-32dc-5433-aac3-a047171aacd6 3018a7db-32dc-5433-aac3-a047171aacd6}" +testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.441+1400 [WARN] server-7e074892-d43a-9b83-058c-4c6dc15f2751: Election timeout reached, restarting election + testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.441+1400 [WARN] server-7e074892-d43a-9b83-058c-4c6dc15f2751: Election timeout reached, restarting election +testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.441+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering candidate state: node="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Candidate]" term=4 + testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.441+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering candidate state: node="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Candidate]" term=4 +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.441+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: rejecting vote request since our last index is greater: candidate=7e074892-d43a-9b83-058c-4c6dc15f2751 last-index=4 last-candidate-index=3 + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.441+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: rejecting vote request since our last index is greater: candidate=7e074892-d43a-9b83-058c-4c6dc15f2751 last-index=4 last-candidate-index=3 +testLoggerAdapter verbose: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.441+1400 [WARN] server-3018a7db-32dc-5433-aac3-a047171aacd6: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 + testing.go:149: server-3018a7db-32dc-5433-aac3-a047171aacd6: 2023-05-16T04:06:16.441+1400 [WARN] server-3018a7db-32dc-5433-aac3-a047171aacd6: rejecting vote request since we have a leader: from=7e074892-d43a-9b83-058c-4c6dc15f2751 leader=7f0d116e-6b51-5f11-8a07-db536b6189b7 +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering candidate state: node="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Candidate]" term=5 + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering candidate state: node="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Candidate]" term=5 +testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.455+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering follower state: follower="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Follower]" leader= + testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.455+1400 [INFO] server-7e074892-d43a-9b83-058c-4c6dc15f2751: entering follower state: follower="Node at 7e074892-d43a-9b83-058c-4c6dc15f2751 [Follower]" leader= +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: election won: tally=2 + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: election won: tally=2 +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering leader state: leader="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Leader]" + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: entering leader state: leader="Node at 7f0d116e-6b51-5f11-8a07-db536b6189b7 [Leader]" +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-3018a7db-32dc-5433-aac3-a047171aacd6 + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-3018a7db-32dc-5433-aac3-a047171aacd6 +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-7e074892-d43a-9b83-058c-4c6dc15f2751 + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.455+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: added peer, starting replication: peer=server-7e074892-d43a-9b83-058c-4c6dc15f2751 +testLoggerAdapter verbose: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.456+1400 [WARN] server-7e074892-d43a-9b83-058c-4c6dc15f2751: failed to get previous log: previous-index=4 last-index=3 error="log not found" + testing.go:149: server-7e074892-d43a-9b83-058c-4c6dc15f2751: 2023-05-16T04:06:16.456+1400 [WARN] server-7e074892-d43a-9b83-058c-4c6dc15f2751: failed to get previous log: previous-index=4 last-index=3 error="log not found" +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.456+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: appendEntries rejected, sending older logs: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" next=4 + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.456+1400 [WARN] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: appendEntries rejected, sending older logs: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" next=4 +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.456+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: pipelining replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.456+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: pipelining replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.456+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: pipelining replication: peer="{Voter server-3018a7db-32dc-5433-aac3-a047171aacd6 3018a7db-32dc-5433-aac3-a047171aacd6}" + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.456+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: pipelining replication: peer="{Voter server-3018a7db-32dc-5433-aac3-a047171aacd6 3018a7db-32dc-5433-aac3-a047171aacd6}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.562+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:16.456772007 +1400 +14 m=+13.259353642 (1 nodes), 71.589925ms from start of poll, 454.871168ms from cluster start. Timeout at 2023-05-16 04:06:16.562175091 +1400 +14 m=+13.364756706, 105.403064ms after stability + testing.go:149: cluster: 2023-05-16T04:06:16.562+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:16.456772007 +1400 +14 m=+13.259353642 (1 nodes), 71.589925ms from start of poll, 454.871168ms from cluster start. Timeout at 2023-05-16 04:06:16.562175091 +1400 +14 m=+13.364756706, 105.403064ms after stability +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.562+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: aborting pipeline replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.562+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: aborting pipeline replication: peer="{Voter server-7e074892-d43a-9b83-058c-4c6dc15f2751 7e074892-d43a-9b83-058c-4c6dc15f2751}" +testLoggerAdapter verbose: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.562+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: aborting pipeline replication: peer="{Voter server-3018a7db-32dc-5433-aac3-a047171aacd6 3018a7db-32dc-5433-aac3-a047171aacd6}" + testing.go:149: server-7f0d116e-6b51-5f11-8a07-db536b6189b7: 2023-05-16T04:06:16.562+1400 [INFO] server-7f0d116e-6b51-5f11-8a07-db536b6189b7: aborting pipeline replication: peer="{Voter server-3018a7db-32dc-5433-aac3-a047171aacd6 3018a7db-32dc-5433-aac3-a047171aacd6}" --- PASS: TestRaft_ReJoinFollower (0.56s) === RUN TestRaft_LeaderLeaseExpire -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.031-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b79eafcb-5990-d1d8-594f-e3e6a8304541 Address:b79eafcb-5990-d1d8-594f-e3e6a8304541} {Suffrage:Voter ID:server-7073535a-f1bb-b845-4065-b4fdc653862c Address:7073535a-f1bb-b845-4065-b4fdc653862c}]" - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.031-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b79eafcb-5990-d1d8-594f-e3e6a8304541 Address:b79eafcb-5990-d1d8-594f-e3e6a8304541} {Suffrage:Voter ID:server-7073535a-f1bb-b845-4065-b4fdc653862c Address:7073535a-f1bb-b845-4065-b4fdc653862c}]" -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.031-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering follower state: follower="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Follower]" leader= - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.031-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering follower state: follower="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Follower]" leader= -testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.031-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b79eafcb-5990-d1d8-594f-e3e6a8304541 Address:b79eafcb-5990-d1d8-594f-e3e6a8304541} {Suffrage:Voter ID:server-7073535a-f1bb-b845-4065-b4fdc653862c Address:7073535a-f1bb-b845-4065-b4fdc653862c}]" - testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.031-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b79eafcb-5990-d1d8-594f-e3e6a8304541 Address:b79eafcb-5990-d1d8-594f-e3e6a8304541} {Suffrage:Voter ID:server-7073535a-f1bb-b845-4065-b4fdc653862c Address:7073535a-f1bb-b845-4065-b4fdc653862c}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.031-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:36.031-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.031-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: entering follower state: follower="Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Follower]" leader= - testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.031-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: entering follower state: follower="Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Follower]" leader= -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering candidate state: node="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Candidate]" term=2 - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering candidate state: node="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Candidate]" term=2 -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: election won: tally=2 - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: election won: tally=2 -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering leader state: leader="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Leader]" - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering leader state: leader="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Leader]" -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: added peer, starting replication: peer=server-7073535a-f1bb-b845-4065-b4fdc653862c - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.115-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: added peer, starting replication: peer=server-7073535a-f1bb-b845-4065-b4fdc653862c -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.116-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: pipelining replication: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.116-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: pipelining replication: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.221-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:36.1158521 -1200 -12 m=+13.123168581 (1 nodes), 84.2956ms from start of poll, 84.93222ms from cluster start. Timeout at 2024-06-16 08:26:36.22114212 -1200 -12 m=+13.228458581, 105.29ms after stability - testing.go:149: cluster: 2024-06-16T08:26:36.221-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:36.1158521 -1200 -12 m=+13.123168581 (1 nodes), 84.2956ms from start of poll, 84.93222ms from cluster start. Timeout at 2024-06-16 08:26:36.22114212 -1200 -12 m=+13.228458581, 105.29ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.226-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:36.226-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.331-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:36.2266148 -1200 -12 m=+13.233931261 (1 nodes), 880ns from start of poll, 195.6949ms from cluster start. Timeout at 2024-06-16 08:26:36.33179496 -1200 -12 m=+13.339111441, 105.18018ms after stability - testing.go:149: cluster: 2024-06-16T08:26:36.331-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:36.2266148 -1200 -12 m=+13.233931261 (1 nodes), 880ns from start of poll, 195.6949ms from cluster start. Timeout at 2024-06-16 08:26:36.33179496 -1200 -12 m=+13.339111441, 105.18018ms after stability - raft_test.go:1492: [INFO] Disconnecting Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Follower] -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.332-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to heartbeat to: peer=7073535a-f1bb-b845-4065-b4fdc653862c error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.332-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to heartbeat to: peer=7073535a-f1bb-b845-4065-b4fdc653862c error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.338-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to pipeline appendEntries: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="append pipeline closed" - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.338-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to pipeline appendEntries: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="append pipeline closed" -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.338-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: aborting pipeline replication: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.338-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: aborting pipeline replication: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.343-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to appendEntries to: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.343-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to appendEntries to: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.354-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to heartbeat to: peer=7073535a-f1bb-b845-4065-b4fdc653862c error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.354-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to heartbeat to: peer=7073535a-f1bb-b845-4065-b4fdc653862c error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.360-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to appendEntries to: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.360-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to appendEntries to: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.375-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to heartbeat to: peer=7073535a-f1bb-b845-4065-b4fdc653862c error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.375-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to heartbeat to: peer=7073535a-f1bb-b845-4065-b4fdc653862c error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.380-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to contact: server-id=server-7073535a-f1bb-b845-4065-b4fdc653862c time=50.81704ms - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.380-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to contact: server-id=server-7073535a-f1bb-b845-4065-b4fdc653862c time=50.81704ms -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.380-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to contact quorum of nodes, stepping down - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.380-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to contact quorum of nodes, stepping down -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.380-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering follower state: follower="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Follower]" leader= - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.380-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering follower state: follower="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Follower]" leader= -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.380-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:36.380-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.381-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to appendEntries to: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.381-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to appendEntries to: peer="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" -testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.425-1200 [WARN] server-7073535a-f1bb-b845-4065-b4fdc653862c: heartbeat timeout reached, starting election: last-leader=b79eafcb-5990-d1d8-594f-e3e6a8304541 - testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.425-1200 [WARN] server-7073535a-f1bb-b845-4065-b4fdc653862c: heartbeat timeout reached, starting election: last-leader=b79eafcb-5990-d1d8-594f-e3e6a8304541 -testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.425-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: entering candidate state: node="Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Candidate]" term=3 - testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.425-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: entering candidate state: node="Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Candidate]" term=3 -testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.425-1200 [ERROR] server-7073535a-f1bb-b845-4065-b4fdc653862c: failed to make requestVote RPC: target="{Voter server-b79eafcb-5990-d1d8-594f-e3e6a8304541 b79eafcb-5990-d1d8-594f-e3e6a8304541}" error="failed to connect to peer: b79eafcb-5990-d1d8-594f-e3e6a8304541" - testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.425-1200 [ERROR] server-7073535a-f1bb-b845-4065-b4fdc653862c: failed to make requestVote RPC: target="{Voter server-b79eafcb-5990-d1d8-594f-e3e6a8304541 b79eafcb-5990-d1d8-594f-e3e6a8304541}" error="failed to connect to peer: b79eafcb-5990-d1d8-594f-e3e6a8304541" -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.456-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.456-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.456-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering candidate state: node="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Candidate]" term=3 - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.456-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering candidate state: node="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Candidate]" term=3 -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.457-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to make requestVote RPC: target="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.457-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to make requestVote RPC: target="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" -testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.500-1200 [WARN] server-7073535a-f1bb-b845-4065-b4fdc653862c: Election timeout reached, restarting election - testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.500-1200 [WARN] server-7073535a-f1bb-b845-4065-b4fdc653862c: Election timeout reached, restarting election -testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.500-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: entering candidate state: node="Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Candidate]" term=4 - testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.500-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: entering candidate state: node="Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Candidate]" term=4 -testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.500-1200 [ERROR] server-7073535a-f1bb-b845-4065-b4fdc653862c: failed to make requestVote RPC: target="{Voter server-b79eafcb-5990-d1d8-594f-e3e6a8304541 b79eafcb-5990-d1d8-594f-e3e6a8304541}" error="failed to connect to peer: b79eafcb-5990-d1d8-594f-e3e6a8304541" - testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.500-1200 [ERROR] server-7073535a-f1bb-b845-4065-b4fdc653862c: failed to make requestVote RPC: target="{Voter server-b79eafcb-5990-d1d8-594f-e3e6a8304541 b79eafcb-5990-d1d8-594f-e3e6a8304541}" error="failed to connect to peer: b79eafcb-5990-d1d8-594f-e3e6a8304541" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.530-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:36.42558108 -1200 -12 m=+13.432897561 (0 nodes), 44.74156ms from start of poll, 394.6612ms from cluster start. Timeout at 2024-06-16 08:26:36.53077266 -1200 -12 m=+13.538089141, 105.19158ms after stability - testing.go:149: cluster: 2024-06-16T08:26:36.530-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:36.42558108 -1200 -12 m=+13.432897561 (0 nodes), 44.74156ms from start of poll, 394.6612ms from cluster start. Timeout at 2024-06-16 08:26:36.53077266 -1200 -12 m=+13.538089141, 105.19158ms after stability -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.546-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: Election timeout reached, restarting election - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.546-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: Election timeout reached, restarting election -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.546-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering candidate state: node="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Candidate]" term=4 - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.546-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering candidate state: node="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Candidate]" term=4 -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.546-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to make requestVote RPC: target="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.546-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to make requestVote RPC: target="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" -testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.589-1200 [WARN] server-7073535a-f1bb-b845-4065-b4fdc653862c: Election timeout reached, restarting election - testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.589-1200 [WARN] server-7073535a-f1bb-b845-4065-b4fdc653862c: Election timeout reached, restarting election -testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.589-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: entering candidate state: node="Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Candidate]" term=5 - testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.589-1200 [INFO] server-7073535a-f1bb-b845-4065-b4fdc653862c: entering candidate state: node="Node at 7073535a-f1bb-b845-4065-b4fdc653862c [Candidate]" term=5 -testLoggerAdapter verbose: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.589-1200 [ERROR] server-7073535a-f1bb-b845-4065-b4fdc653862c: failed to make requestVote RPC: target="{Voter server-b79eafcb-5990-d1d8-594f-e3e6a8304541 b79eafcb-5990-d1d8-594f-e3e6a8304541}" error="failed to connect to peer: b79eafcb-5990-d1d8-594f-e3e6a8304541" - testing.go:149: server-7073535a-f1bb-b845-4065-b4fdc653862c: 2024-06-16T08:26:36.589-1200 [ERROR] server-7073535a-f1bb-b845-4065-b4fdc653862c: failed to make requestVote RPC: target="{Voter server-b79eafcb-5990-d1d8-594f-e3e6a8304541 b79eafcb-5990-d1d8-594f-e3e6a8304541}" error="failed to connect to peer: b79eafcb-5990-d1d8-594f-e3e6a8304541" -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.607-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: Election timeout reached, restarting election - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.607-1200 [WARN] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: Election timeout reached, restarting election -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.607-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering candidate state: node="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Candidate]" term=5 - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.607-1200 [INFO] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: entering candidate state: node="Node at b79eafcb-5990-d1d8-594f-e3e6a8304541 [Candidate]" term=5 -testLoggerAdapter verbose: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.607-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to make requestVote RPC: target="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" - testing.go:149: server-b79eafcb-5990-d1d8-594f-e3e6a8304541: 2024-06-16T08:26:36.607-1200 [ERROR] server-b79eafcb-5990-d1d8-594f-e3e6a8304541: failed to make requestVote RPC: target="{Voter server-7073535a-f1bb-b845-4065-b4fdc653862c 7073535a-f1bb-b845-4065-b4fdc653862c}" error="failed to connect to peer: 7073535a-f1bb-b845-4065-b4fdc653862c" ---- PASS: TestRaft_LeaderLeaseExpire (0.61s) +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.563+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d Address:e56a0d25-53a3-dde4-b8d8-e63211ee6e5d} {Suffrage:Voter ID:server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 Address:d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}]" + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.563+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d Address:e56a0d25-53a3-dde4-b8d8-e63211ee6e5d} {Suffrage:Voter ID:server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 Address:d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}]" +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.564+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d Address:e56a0d25-53a3-dde4-b8d8-e63211ee6e5d} {Suffrage:Voter ID:server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 Address:d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}]" + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.564+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d Address:e56a0d25-53a3-dde4-b8d8-e63211ee6e5d} {Suffrage:Voter ID:server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 Address:d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.564+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:16.564+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.564+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering follower state: follower="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Follower]" leader= + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.564+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering follower state: follower="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Follower]" leader= +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.564+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering follower state: follower="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Follower]" leader= + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.564+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering follower state: follower="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Follower]" leader= +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.623+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.623+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.623+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.623+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=2 + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.623+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.624+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=2 + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.624+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=2 + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.623+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=2 +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.624+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: duplicate requestVote for same term: term=2 + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.624+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: duplicate requestVote for same term: term=2 +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.624+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: duplicate requestVote for same term: term=2 + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.624+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: duplicate requestVote for same term: term=2 +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.676+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: Election timeout reached, restarting election + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.676+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: Election timeout reached, restarting election +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=3 + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=3 +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.677+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering follower state: follower="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Follower]" leader= + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.677+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering follower state: follower="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Follower]" leader= +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: election won: tally=2 + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: election won: tally=2 +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering leader state: leader="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Leader]" + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering leader state: leader="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Leader]" +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: added peer, starting replication: peer=server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: added peer, starting replication: peer=server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: pipelining replication: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.677+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: pipelining replication: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.783+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:16.677785455 +1400 +14 m=+13.480367070 (1 nodes), 113.373698ms from start of poll, 114.065046ms from cluster start. Timeout at 2023-05-16 04:06:16.783698645 +1400 +14 m=+13.586280320, 105.91325ms after stability + testing.go:149: cluster: 2023-05-16T04:06:16.783+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:16.677785455 +1400 +14 m=+13.480367070 (1 nodes), 113.373698ms from start of poll, 114.065046ms from cluster start. Timeout at 2023-05-16 04:06:16.783698645 +1400 +14 m=+13.586280320, 105.91325ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.789+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:16.789+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.895+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:16.789630715 +1400 +14 m=+13.592212330 (1 nodes), 1.94µs from start of poll, 225.910306ms from cluster start. Timeout at 2023-05-16 04:06:16.895638966 +1400 +14 m=+13.698220641, 106.008311ms after stability + testing.go:149: cluster: 2023-05-16T04:06:16.895+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:16.789630715 +1400 +14 m=+13.592212330 (1 nodes), 1.94µs from start of poll, 225.910306ms from cluster start. Timeout at 2023-05-16 04:06:16.895638966 +1400 +14 m=+13.698220641, 106.008311ms after stability + raft_test.go:1492: [INFO] Disconnecting Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Follower] +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.899+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to pipeline appendEntries: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="append pipeline closed" + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.899+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to pipeline appendEntries: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="append pipeline closed" +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.899+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: aborting pipeline replication: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.899+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: aborting pipeline replication: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.901+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to heartbeat to: peer=d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.901+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to heartbeat to: peer=d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.906+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to appendEntries to: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.906+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to appendEntries to: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.921+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to heartbeat to: peer=d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.921+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to heartbeat to: peer=d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.926+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to appendEntries to: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.926+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to appendEntries to: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.939+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to heartbeat to: peer=d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.939+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to heartbeat to: peer=d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.944+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to appendEntries to: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.944+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to appendEntries to: peer="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.944+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to contact: server-id=server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 time=51.267705ms + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.944+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to contact: server-id=server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 time=51.267705ms +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.944+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to contact quorum of nodes, stepping down + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.944+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to contact quorum of nodes, stepping down +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.944+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering follower state: follower="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Follower]" leader= + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:16.944+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering follower state: follower="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Follower]" leader= +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:16.944+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:16.944+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.980+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: heartbeat timeout reached, starting election: last-leader=e56a0d25-53a3-dde4-b8d8-e63211ee6e5d + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.980+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: heartbeat timeout reached, starting election: last-leader=e56a0d25-53a3-dde4-b8d8-e63211ee6e5d +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.980+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=4 + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.980+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=4 +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.980+1400 [ERROR] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: failed to make requestVote RPC: target="{Voter server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d e56a0d25-53a3-dde4-b8d8-e63211ee6e5d}" error="failed to connect to peer: e56a0d25-53a3-dde4-b8d8-e63211ee6e5d" + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:16.980+1400 [ERROR] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: failed to make requestVote RPC: target="{Voter server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d e56a0d25-53a3-dde4-b8d8-e63211ee6e5d}" error="failed to connect to peer: e56a0d25-53a3-dde4-b8d8-e63211ee6e5d" +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.004+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.004+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.004+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=4 + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.004+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=4 +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.004+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to make requestVote RPC: target="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.004+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to make requestVote RPC: target="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.031+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: Election timeout reached, restarting election + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.031+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: Election timeout reached, restarting election +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.031+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=5 + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.031+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=5 +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.031+1400 [ERROR] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: failed to make requestVote RPC: target="{Voter server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d e56a0d25-53a3-dde4-b8d8-e63211ee6e5d}" error="failed to connect to peer: e56a0d25-53a3-dde4-b8d8-e63211ee6e5d" + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.031+1400 [ERROR] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: failed to make requestVote RPC: target="{Voter server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d e56a0d25-53a3-dde4-b8d8-e63211ee6e5d}" error="failed to connect to peer: e56a0d25-53a3-dde4-b8d8-e63211ee6e5d" +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.076+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: Election timeout reached, restarting election + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.076+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: Election timeout reached, restarting election +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.076+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=5 + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.076+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=5 +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.077+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to make requestVote RPC: target="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.077+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to make requestVote RPC: target="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.083+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: Election timeout reached, restarting election + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.083+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: Election timeout reached, restarting election +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.083+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=6 + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.083+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=6 +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.083+1400 [ERROR] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: failed to make requestVote RPC: target="{Voter server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d e56a0d25-53a3-dde4-b8d8-e63211ee6e5d}" error="failed to connect to peer: e56a0d25-53a3-dde4-b8d8-e63211ee6e5d" + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.083+1400 [ERROR] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: failed to make requestVote RPC: target="{Voter server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d e56a0d25-53a3-dde4-b8d8-e63211ee6e5d}" error="failed to connect to peer: e56a0d25-53a3-dde4-b8d8-e63211ee6e5d" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:17.110+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.004736014 +1400 +14 m=+13.807317649 (0 nodes), 59.848267ms from start of poll, 441.015625ms from cluster start. Timeout at 2023-05-16 04:06:17.110044056 +1400 +14 m=+13.912625751, 105.308102ms after stability + testing.go:149: cluster: 2023-05-16T04:06:17.110+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.004736014 +1400 +14 m=+13.807317649 (0 nodes), 59.848267ms from start of poll, 441.015625ms from cluster start. Timeout at 2023-05-16 04:06:17.110044056 +1400 +14 m=+13.912625751, 105.308102ms after stability +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.148+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: Election timeout reached, restarting election + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.148+1400 [WARN] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: Election timeout reached, restarting election +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.148+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=6 + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.148+1400 [INFO] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: entering candidate state: node="Node at e56a0d25-53a3-dde4-b8d8-e63211ee6e5d [Candidate]" term=6 +testLoggerAdapter verbose: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.148+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to make requestVote RPC: target="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" + testing.go:149: server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: 2023-05-16T04:06:17.148+1400 [ERROR] server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d: failed to make requestVote RPC: target="{Voter server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 d08425b6-fcbb-c124-8ba2-af7a00bb9ba1}" error="failed to connect to peer: d08425b6-fcbb-c124-8ba2-af7a00bb9ba1" +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.154+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: Election timeout reached, restarting election + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.154+1400 [WARN] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: Election timeout reached, restarting election +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.154+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=7 + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.154+1400 [INFO] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: entering candidate state: node="Node at d08425b6-fcbb-c124-8ba2-af7a00bb9ba1 [Candidate]" term=7 +testLoggerAdapter verbose: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.154+1400 [ERROR] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: failed to make requestVote RPC: target="{Voter server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d e56a0d25-53a3-dde4-b8d8-e63211ee6e5d}" error="failed to connect to peer: e56a0d25-53a3-dde4-b8d8-e63211ee6e5d" + testing.go:149: server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: 2023-05-16T04:06:17.154+1400 [ERROR] server-d08425b6-fcbb-c124-8ba2-af7a00bb9ba1: failed to make requestVote RPC: target="{Voter server-e56a0d25-53a3-dde4-b8d8-e63211ee6e5d e56a0d25-53a3-dde4-b8d8-e63211ee6e5d}" error="failed to connect to peer: e56a0d25-53a3-dde4-b8d8-e63211ee6e5d" +--- PASS: TestRaft_LeaderLeaseExpire (0.65s) === RUN TestRaft_Barrier -testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.638-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5 Address:4fd32242-2c8f-4569-6ffc-b136a20ce6e5} {Suffrage:Voter ID:server-50e866b8-aa0a-b005-cc90-24821d827da2 Address:50e866b8-aa0a-b005-cc90-24821d827da2} {Suffrage:Voter ID:server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c Address:8a3deebf-8cb3-c1f8-c09e-8717b23f599c}]" - testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.638-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5 Address:4fd32242-2c8f-4569-6ffc-b136a20ce6e5} {Suffrage:Voter ID:server-50e866b8-aa0a-b005-cc90-24821d827da2 Address:50e866b8-aa0a-b005-cc90-24821d827da2} {Suffrage:Voter ID:server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c Address:8a3deebf-8cb3-c1f8-c09e-8717b23f599c}]" -testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.639-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: entering follower state: follower="Node at 4fd32242-2c8f-4569-6ffc-b136a20ce6e5 [Follower]" leader= - testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.639-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: entering follower state: follower="Node at 4fd32242-2c8f-4569-6ffc-b136a20ce6e5 [Follower]" leader= -testLoggerAdapter verbose: server-50e866b8-aa0a-b005-cc90-24821d827da2: 2024-06-16T08:26:36.639-1200 [INFO] server-50e866b8-aa0a-b005-cc90-24821d827da2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5 Address:4fd32242-2c8f-4569-6ffc-b136a20ce6e5} {Suffrage:Voter ID:server-50e866b8-aa0a-b005-cc90-24821d827da2 Address:50e866b8-aa0a-b005-cc90-24821d827da2} {Suffrage:Voter ID:server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c Address:8a3deebf-8cb3-c1f8-c09e-8717b23f599c}]" - testing.go:149: server-50e866b8-aa0a-b005-cc90-24821d827da2: 2024-06-16T08:26:36.639-1200 [INFO] server-50e866b8-aa0a-b005-cc90-24821d827da2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5 Address:4fd32242-2c8f-4569-6ffc-b136a20ce6e5} {Suffrage:Voter ID:server-50e866b8-aa0a-b005-cc90-24821d827da2 Address:50e866b8-aa0a-b005-cc90-24821d827da2} {Suffrage:Voter ID:server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c Address:8a3deebf-8cb3-c1f8-c09e-8717b23f599c}]" -testLoggerAdapter verbose: server-50e866b8-aa0a-b005-cc90-24821d827da2: 2024-06-16T08:26:36.639-1200 [INFO] server-50e866b8-aa0a-b005-cc90-24821d827da2: entering follower state: follower="Node at 50e866b8-aa0a-b005-cc90-24821d827da2 [Follower]" leader= - testing.go:149: server-50e866b8-aa0a-b005-cc90-24821d827da2: 2024-06-16T08:26:36.639-1200 [INFO] server-50e866b8-aa0a-b005-cc90-24821d827da2: entering follower state: follower="Node at 50e866b8-aa0a-b005-cc90-24821d827da2 [Follower]" leader= -testLoggerAdapter verbose: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.639-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5 Address:4fd32242-2c8f-4569-6ffc-b136a20ce6e5} {Suffrage:Voter ID:server-50e866b8-aa0a-b005-cc90-24821d827da2 Address:50e866b8-aa0a-b005-cc90-24821d827da2} {Suffrage:Voter ID:server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c Address:8a3deebf-8cb3-c1f8-c09e-8717b23f599c}]" - testing.go:149: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.639-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5 Address:4fd32242-2c8f-4569-6ffc-b136a20ce6e5} {Suffrage:Voter ID:server-50e866b8-aa0a-b005-cc90-24821d827da2 Address:50e866b8-aa0a-b005-cc90-24821d827da2} {Suffrage:Voter ID:server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c Address:8a3deebf-8cb3-c1f8-c09e-8717b23f599c}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.639-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:36.639-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.639-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: entering follower state: follower="Node at 8a3deebf-8cb3-c1f8-c09e-8717b23f599c [Follower]" leader= - testing.go:149: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.639-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: entering follower state: follower="Node at 8a3deebf-8cb3-c1f8-c09e-8717b23f599c [Follower]" leader= -testLoggerAdapter verbose: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.694-1200 [WARN] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.694-1200 [WARN] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.694-1200 [WARN] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.694-1200 [WARN] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.694-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: entering candidate state: node="Node at 4fd32242-2c8f-4569-6ffc-b136a20ce6e5 [Candidate]" term=2 -testLoggerAdapter verbose: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.694-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: entering candidate state: node="Node at 8a3deebf-8cb3-c1f8-c09e-8717b23f599c [Candidate]" term=2 - testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.694-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: entering candidate state: node="Node at 4fd32242-2c8f-4569-6ffc-b136a20ce6e5 [Candidate]" term=2 - testing.go:149: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.694-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: entering candidate state: node="Node at 8a3deebf-8cb3-c1f8-c09e-8717b23f599c [Candidate]" term=2 -testLoggerAdapter verbose: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.695-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: duplicate requestVote for same term: term=2 - testing.go:149: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.695-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: server-50e866b8-aa0a-b005-cc90-24821d827da2: 2024-06-16T08:26:36.695-1200 [INFO] server-50e866b8-aa0a-b005-cc90-24821d827da2: duplicate requestVote for same term: term=2 - testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: election won: tally=2 - testing.go:149: server-50e866b8-aa0a-b005-cc90-24821d827da2: 2024-06-16T08:26:36.695-1200 [INFO] server-50e866b8-aa0a-b005-cc90-24821d827da2: duplicate requestVote for same term: term=2 - testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: election won: tally=2 -testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: entering leader state: leader="Node at 4fd32242-2c8f-4569-6ffc-b136a20ce6e5 [Leader]" - testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: entering leader state: leader="Node at 4fd32242-2c8f-4569-6ffc-b136a20ce6e5 [Leader]" -testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: added peer, starting replication: peer=server-50e866b8-aa0a-b005-cc90-24821d827da2 - testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: added peer, starting replication: peer=server-50e866b8-aa0a-b005-cc90-24821d827da2 -testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: added peer, starting replication: peer=server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c - testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: added peer, starting replication: peer=server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c -testLoggerAdapter verbose: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.695-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: entering follower state: follower="Node at 8a3deebf-8cb3-c1f8-c09e-8717b23f599c [Follower]" leader=4fd32242-2c8f-4569-6ffc-b136a20ce6e5 - testing.go:149: server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: 2024-06-16T08:26:36.695-1200 [INFO] server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c: entering follower state: follower="Node at 8a3deebf-8cb3-c1f8-c09e-8717b23f599c [Follower]" leader=4fd32242-2c8f-4569-6ffc-b136a20ce6e5 -testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: pipelining replication: peer="{Voter server-50e866b8-aa0a-b005-cc90-24821d827da2 50e866b8-aa0a-b005-cc90-24821d827da2}" - testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: pipelining replication: peer="{Voter server-50e866b8-aa0a-b005-cc90-24821d827da2 50e866b8-aa0a-b005-cc90-24821d827da2}" -testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: pipelining replication: peer="{Voter server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c 8a3deebf-8cb3-c1f8-c09e-8717b23f599c}" - testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.695-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: pipelining replication: peer="{Voter server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c 8a3deebf-8cb3-c1f8-c09e-8717b23f599c}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.800-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:36.6956387 -1200 -12 m=+13.702955161 (1 nodes), 55.79202ms from start of poll, 57.86948ms from cluster start. Timeout at 2024-06-16 08:26:36.80080624 -1200 -12 m=+13.808122741, 105.16758ms after stability - testing.go:149: cluster: 2024-06-16T08:26:36.800-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:36.6956387 -1200 -12 m=+13.702955161 (1 nodes), 55.79202ms from start of poll, 57.86948ms from cluster start. Timeout at 2024-06-16 08:26:36.80080624 -1200 -12 m=+13.808122741, 105.16758ms after stability -testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.814-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: aborting pipeline replication: peer="{Voter server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c 8a3deebf-8cb3-c1f8-c09e-8717b23f599c}" - testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.814-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: aborting pipeline replication: peer="{Voter server-8a3deebf-8cb3-c1f8-c09e-8717b23f599c 8a3deebf-8cb3-c1f8-c09e-8717b23f599c}" -testLoggerAdapter verbose: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.814-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: aborting pipeline replication: peer="{Voter server-50e866b8-aa0a-b005-cc90-24821d827da2 50e866b8-aa0a-b005-cc90-24821d827da2}" - testing.go:149: server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: 2024-06-16T08:26:36.814-1200 [INFO] server-4fd32242-2c8f-4569-6ffc-b136a20ce6e5: aborting pipeline replication: peer="{Voter server-50e866b8-aa0a-b005-cc90-24821d827da2 50e866b8-aa0a-b005-cc90-24821d827da2}" ---- PASS: TestRaft_Barrier (0.18s) +testLoggerAdapter verbose: server-2741c739-3eb5-496c-763f-89d501309423: 2023-05-16T04:06:17.217+1400 [INFO] server-2741c739-3eb5-496c-763f-89d501309423: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2741c739-3eb5-496c-763f-89d501309423 Address:2741c739-3eb5-496c-763f-89d501309423} {Suffrage:Voter ID:server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 Address:5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46} {Suffrage:Voter ID:server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 Address:8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42}]" + testing.go:149: server-2741c739-3eb5-496c-763f-89d501309423: 2023-05-16T04:06:17.217+1400 [INFO] server-2741c739-3eb5-496c-763f-89d501309423: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2741c739-3eb5-496c-763f-89d501309423 Address:2741c739-3eb5-496c-763f-89d501309423} {Suffrage:Voter ID:server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 Address:5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46} {Suffrage:Voter ID:server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 Address:8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42}]" +testLoggerAdapter verbose: server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46: 2023-05-16T04:06:17.218+1400 [INFO] server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2741c739-3eb5-496c-763f-89d501309423 Address:2741c739-3eb5-496c-763f-89d501309423} {Suffrage:Voter ID:server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 Address:5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46} {Suffrage:Voter ID:server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 Address:8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42}]" + testing.go:149: server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46: 2023-05-16T04:06:17.218+1400 [INFO] server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2741c739-3eb5-496c-763f-89d501309423 Address:2741c739-3eb5-496c-763f-89d501309423} {Suffrage:Voter ID:server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 Address:5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46} {Suffrage:Voter ID:server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 Address:8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42}]" +testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.218+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2741c739-3eb5-496c-763f-89d501309423 Address:2741c739-3eb5-496c-763f-89d501309423} {Suffrage:Voter ID:server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 Address:5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46} {Suffrage:Voter ID:server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 Address:8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42}]" + testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.218+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2741c739-3eb5-496c-763f-89d501309423 Address:2741c739-3eb5-496c-763f-89d501309423} {Suffrage:Voter ID:server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 Address:5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46} {Suffrage:Voter ID:server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 Address:8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:17.218+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:17.218+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-2741c739-3eb5-496c-763f-89d501309423: 2023-05-16T04:06:17.218+1400 [INFO] server-2741c739-3eb5-496c-763f-89d501309423: entering follower state: follower="Node at 2741c739-3eb5-496c-763f-89d501309423 [Follower]" leader= + testing.go:149: server-2741c739-3eb5-496c-763f-89d501309423: 2023-05-16T04:06:17.218+1400 [INFO] server-2741c739-3eb5-496c-763f-89d501309423: entering follower state: follower="Node at 2741c739-3eb5-496c-763f-89d501309423 [Follower]" leader= +testLoggerAdapter verbose: server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46: 2023-05-16T04:06:17.218+1400 [INFO] server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46: entering follower state: follower="Node at 5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 [Follower]" leader= + testing.go:149: server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46: 2023-05-16T04:06:17.218+1400 [INFO] server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46: entering follower state: follower="Node at 5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 [Follower]" leader= +testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.218+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: entering follower state: follower="Node at 8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 [Follower]" leader= + testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.218+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: entering follower state: follower="Node at 8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 [Follower]" leader= +testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [WARN] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [WARN] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: entering candidate state: node="Node at 8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 [Candidate]" term=2 + testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: entering candidate state: node="Node at 8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 [Candidate]" term=2 +testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: election won: tally=2 + testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: election won: tally=2 +testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: entering leader state: leader="Node at 8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 [Leader]" + testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: entering leader state: leader="Node at 8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42 [Leader]" +testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: added peer, starting replication: peer=server-2741c739-3eb5-496c-763f-89d501309423 + testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: added peer, starting replication: peer=server-2741c739-3eb5-496c-763f-89d501309423 +testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: added peer, starting replication: peer=server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 + testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.293+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: added peer, starting replication: peer=server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 +testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.294+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: pipelining replication: peer="{Voter server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46}" + testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.294+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: pipelining replication: peer="{Voter server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46}" +testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.294+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: pipelining replication: peer="{Voter server-2741c739-3eb5-496c-763f-89d501309423 2741c739-3eb5-496c-763f-89d501309423}" + testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.294+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: pipelining replication: peer="{Voter server-2741c739-3eb5-496c-763f-89d501309423 2741c739-3eb5-496c-763f-89d501309423}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:17.399+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.294569114 +1400 +14 m=+14.097150769 (1 nodes), 76.045137ms from start of poll, 76.993669ms from cluster start. Timeout at 2023-05-16 04:06:17.399828396 +1400 +14 m=+14.202410031, 105.259262ms after stability + testing.go:149: cluster: 2023-05-16T04:06:17.399+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.294569114 +1400 +14 m=+14.097150769 (1 nodes), 76.045137ms from start of poll, 76.993669ms from cluster start. Timeout at 2023-05-16 04:06:17.399828396 +1400 +14 m=+14.202410031, 105.259262ms after stability +testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.417+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: aborting pipeline replication: peer="{Voter server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46}" + testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.417+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: aborting pipeline replication: peer="{Voter server-5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46}" +testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.417+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: aborting pipeline replication: peer="{Voter server-2741c739-3eb5-496c-763f-89d501309423 2741c739-3eb5-496c-763f-89d501309423}" + testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.417+1400 [INFO] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: aborting pipeline replication: peer="{Voter server-2741c739-3eb5-496c-763f-89d501309423 2741c739-3eb5-496c-763f-89d501309423}" +testLoggerAdapter verbose: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.468+1400 [ERROR] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: failed to heartbeat to: peer=5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 error="command timed out" + testing.go:149: server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: 2023-05-16T04:06:17.468+1400 [ERROR] server-8eb9ce99-397d-33f5-4ea1-5f67e1ef0c42: failed to heartbeat to: peer=5cfc66f3-c9d4-521d-2ed2-f64ea9e03c46 error="command timed out" +--- PASS: TestRaft_Barrier (0.25s) === RUN TestRaft_VerifyLeader -testLoggerAdapter verbose: server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a: 2024-06-16T08:26:36.815-1200 [INFO] server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a Address:a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a} {Suffrage:Voter ID:server-483446d2-b505-f048-e20e-d1719c98309c Address:483446d2-b505-f048-e20e-d1719c98309c} {Suffrage:Voter ID:server-339546d9-6142-e78a-8480-6ad5f21e1382 Address:339546d9-6142-e78a-8480-6ad5f21e1382}]" - testing.go:149: server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a: 2024-06-16T08:26:36.815-1200 [INFO] server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a Address:a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a} {Suffrage:Voter ID:server-483446d2-b505-f048-e20e-d1719c98309c Address:483446d2-b505-f048-e20e-d1719c98309c} {Suffrage:Voter ID:server-339546d9-6142-e78a-8480-6ad5f21e1382 Address:339546d9-6142-e78a-8480-6ad5f21e1382}]" -testLoggerAdapter verbose: server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a: 2024-06-16T08:26:36.815-1200 [INFO] server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a: entering follower state: follower="Node at a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a [Follower]" leader= - testing.go:149: server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a: 2024-06-16T08:26:36.815-1200 [INFO] server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a: entering follower state: follower="Node at a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a [Follower]" leader= -testLoggerAdapter verbose: server-483446d2-b505-f048-e20e-d1719c98309c: 2024-06-16T08:26:36.815-1200 [INFO] server-483446d2-b505-f048-e20e-d1719c98309c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a Address:a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a} {Suffrage:Voter ID:server-483446d2-b505-f048-e20e-d1719c98309c Address:483446d2-b505-f048-e20e-d1719c98309c} {Suffrage:Voter ID:server-339546d9-6142-e78a-8480-6ad5f21e1382 Address:339546d9-6142-e78a-8480-6ad5f21e1382}]" - testing.go:149: server-483446d2-b505-f048-e20e-d1719c98309c: 2024-06-16T08:26:36.815-1200 [INFO] server-483446d2-b505-f048-e20e-d1719c98309c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a Address:a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a} {Suffrage:Voter ID:server-483446d2-b505-f048-e20e-d1719c98309c Address:483446d2-b505-f048-e20e-d1719c98309c} {Suffrage:Voter ID:server-339546d9-6142-e78a-8480-6ad5f21e1382 Address:339546d9-6142-e78a-8480-6ad5f21e1382}]" -testLoggerAdapter verbose: server-483446d2-b505-f048-e20e-d1719c98309c: 2024-06-16T08:26:36.816-1200 [INFO] server-483446d2-b505-f048-e20e-d1719c98309c: entering follower state: follower="Node at 483446d2-b505-f048-e20e-d1719c98309c [Follower]" leader= - testing.go:149: server-483446d2-b505-f048-e20e-d1719c98309c: 2024-06-16T08:26:36.816-1200 [INFO] server-483446d2-b505-f048-e20e-d1719c98309c: entering follower state: follower="Node at 483446d2-b505-f048-e20e-d1719c98309c [Follower]" leader= -testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.816-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a Address:a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a} {Suffrage:Voter ID:server-483446d2-b505-f048-e20e-d1719c98309c Address:483446d2-b505-f048-e20e-d1719c98309c} {Suffrage:Voter ID:server-339546d9-6142-e78a-8480-6ad5f21e1382 Address:339546d9-6142-e78a-8480-6ad5f21e1382}]" - testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.816-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a Address:a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a} {Suffrage:Voter ID:server-483446d2-b505-f048-e20e-d1719c98309c Address:483446d2-b505-f048-e20e-d1719c98309c} {Suffrage:Voter ID:server-339546d9-6142-e78a-8480-6ad5f21e1382 Address:339546d9-6142-e78a-8480-6ad5f21e1382}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.816-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:36.816-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.816-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: entering follower state: follower="Node at 339546d9-6142-e78a-8480-6ad5f21e1382 [Follower]" leader= - testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.816-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: entering follower state: follower="Node at 339546d9-6142-e78a-8480-6ad5f21e1382 [Follower]" leader= -testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.869-1200 [WARN] server-339546d9-6142-e78a-8480-6ad5f21e1382: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.869-1200 [WARN] server-339546d9-6142-e78a-8480-6ad5f21e1382: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.869-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: entering candidate state: node="Node at 339546d9-6142-e78a-8480-6ad5f21e1382 [Candidate]" term=2 - testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.869-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: entering candidate state: node="Node at 339546d9-6142-e78a-8480-6ad5f21e1382 [Candidate]" term=2 -testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: election won: tally=2 - testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: election won: tally=2 -testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: entering leader state: leader="Node at 339546d9-6142-e78a-8480-6ad5f21e1382 [Leader]" - testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: entering leader state: leader="Node at 339546d9-6142-e78a-8480-6ad5f21e1382 [Leader]" -testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: added peer, starting replication: peer=server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a - testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: added peer, starting replication: peer=server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a -testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: added peer, starting replication: peer=server-483446d2-b505-f048-e20e-d1719c98309c - testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: added peer, starting replication: peer=server-483446d2-b505-f048-e20e-d1719c98309c -testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: pipelining replication: peer="{Voter server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a}" - testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: pipelining replication: peer="{Voter server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a}" -testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: pipelining replication: peer="{Voter server-483446d2-b505-f048-e20e-d1719c98309c 483446d2-b505-f048-e20e-d1719c98309c}" - testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.870-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: pipelining replication: peer="{Voter server-483446d2-b505-f048-e20e-d1719c98309c 483446d2-b505-f048-e20e-d1719c98309c}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:36.975-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:36.87025556 -1200 -12 m=+13.877572061 (1 nodes), 53.95122ms from start of poll, 54.80834ms from cluster start. Timeout at 2024-06-16 08:26:36.97582156 -1200 -12 m=+13.983138041, 105.56598ms after stability - testing.go:149: cluster: 2024-06-16T08:26:36.975-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:36.87025556 -1200 -12 m=+13.877572061 (1 nodes), 53.95122ms from start of poll, 54.80834ms from cluster start. Timeout at 2024-06-16 08:26:36.97582156 -1200 -12 m=+13.983138041, 105.56598ms after stability -testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.976-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: aborting pipeline replication: peer="{Voter server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a}" - testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.976-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: aborting pipeline replication: peer="{Voter server-a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a}" -testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.976-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: aborting pipeline replication: peer="{Voter server-483446d2-b505-f048-e20e-d1719c98309c 483446d2-b505-f048-e20e-d1719c98309c}" - testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:36.976-1200 [INFO] server-339546d9-6142-e78a-8480-6ad5f21e1382: aborting pipeline replication: peer="{Voter server-483446d2-b505-f048-e20e-d1719c98309c 483446d2-b505-f048-e20e-d1719c98309c}" -testLoggerAdapter verbose: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:37.027-1200 [ERROR] server-339546d9-6142-e78a-8480-6ad5f21e1382: failed to heartbeat to: peer=a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a error="command timed out" - testing.go:149: server-339546d9-6142-e78a-8480-6ad5f21e1382: 2024-06-16T08:26:37.027-1200 [ERROR] server-339546d9-6142-e78a-8480-6ad5f21e1382: failed to heartbeat to: peer=a3582faa-6ddd-a5bd-1e64-8e4f9acb0e9a error="command timed out" ---- PASS: TestRaft_VerifyLeader (0.21s) +testLoggerAdapter verbose: server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d: 2023-05-16T04:06:17.470+1400 [INFO] server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d Address:b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d} {Suffrage:Voter ID:server-2b87f816-3ce9-00ec-811b-1f6d404b6331 Address:2b87f816-3ce9-00ec-811b-1f6d404b6331} {Suffrage:Voter ID:server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 Address:2db3bca4-7544-c836-6f8a-94ac1a7a8c40}]" + testing.go:149: server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d: 2023-05-16T04:06:17.470+1400 [INFO] server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d Address:b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d} {Suffrage:Voter ID:server-2b87f816-3ce9-00ec-811b-1f6d404b6331 Address:2b87f816-3ce9-00ec-811b-1f6d404b6331} {Suffrage:Voter ID:server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 Address:2db3bca4-7544-c836-6f8a-94ac1a7a8c40}]" +testLoggerAdapter verbose: server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d: 2023-05-16T04:06:17.470+1400 [INFO] server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d: entering follower state: follower="Node at b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d [Follower]" leader= + testing.go:149: server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d: 2023-05-16T04:06:17.470+1400 [INFO] server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d: entering follower state: follower="Node at b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d [Follower]" leader= +testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.470+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d Address:b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d} {Suffrage:Voter ID:server-2b87f816-3ce9-00ec-811b-1f6d404b6331 Address:2b87f816-3ce9-00ec-811b-1f6d404b6331} {Suffrage:Voter ID:server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 Address:2db3bca4-7544-c836-6f8a-94ac1a7a8c40}]" + testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.470+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d Address:b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d} {Suffrage:Voter ID:server-2b87f816-3ce9-00ec-811b-1f6d404b6331 Address:2b87f816-3ce9-00ec-811b-1f6d404b6331} {Suffrage:Voter ID:server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 Address:2db3bca4-7544-c836-6f8a-94ac1a7a8c40}]" +testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.470+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: entering follower state: follower="Node at 2b87f816-3ce9-00ec-811b-1f6d404b6331 [Follower]" leader= + testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.470+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: entering follower state: follower="Node at 2b87f816-3ce9-00ec-811b-1f6d404b6331 [Follower]" leader= +testLoggerAdapter verbose: server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40: 2023-05-16T04:06:17.471+1400 [INFO] server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d Address:b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d} {Suffrage:Voter ID:server-2b87f816-3ce9-00ec-811b-1f6d404b6331 Address:2b87f816-3ce9-00ec-811b-1f6d404b6331} {Suffrage:Voter ID:server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 Address:2db3bca4-7544-c836-6f8a-94ac1a7a8c40}]" + testing.go:149: server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40: 2023-05-16T04:06:17.471+1400 [INFO] server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d Address:b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d} {Suffrage:Voter ID:server-2b87f816-3ce9-00ec-811b-1f6d404b6331 Address:2b87f816-3ce9-00ec-811b-1f6d404b6331} {Suffrage:Voter ID:server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 Address:2db3bca4-7544-c836-6f8a-94ac1a7a8c40}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:17.471+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:17.471+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40: 2023-05-16T04:06:17.471+1400 [INFO] server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40: entering follower state: follower="Node at 2db3bca4-7544-c836-6f8a-94ac1a7a8c40 [Follower]" leader= + testing.go:149: server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40: 2023-05-16T04:06:17.471+1400 [INFO] server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40: entering follower state: follower="Node at 2db3bca4-7544-c836-6f8a-94ac1a7a8c40 [Follower]" leader= +testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [WARN] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [WARN] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: entering candidate state: node="Node at 2b87f816-3ce9-00ec-811b-1f6d404b6331 [Candidate]" term=2 + testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: entering candidate state: node="Node at 2b87f816-3ce9-00ec-811b-1f6d404b6331 [Candidate]" term=2 +testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: election won: tally=2 + testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: election won: tally=2 +testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: entering leader state: leader="Node at 2b87f816-3ce9-00ec-811b-1f6d404b6331 [Leader]" + testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: entering leader state: leader="Node at 2b87f816-3ce9-00ec-811b-1f6d404b6331 [Leader]" +testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: added peer, starting replication: peer=server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d + testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: added peer, starting replication: peer=server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d +testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: added peer, starting replication: peer=server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 + testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.549+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: added peer, starting replication: peer=server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 +testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.550+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: pipelining replication: peer="{Voter server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 2db3bca4-7544-c836-6f8a-94ac1a7a8c40}" + testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.550+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: pipelining replication: peer="{Voter server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 2db3bca4-7544-c836-6f8a-94ac1a7a8c40}" +testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.550+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: pipelining replication: peer="{Voter server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d}" + testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.550+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: pipelining replication: peer="{Voter server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:17.655+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.550496914 +1400 +14 m=+14.353078529 (1 nodes), 79.265655ms from start of poll, 80.374028ms from cluster start. Timeout at 2023-05-16 04:06:17.655839017 +1400 +14 m=+14.458420652, 105.342123ms after stability + testing.go:149: cluster: 2023-05-16T04:06:17.655+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.550496914 +1400 +14 m=+14.353078529 (1 nodes), 79.265655ms from start of poll, 80.374028ms from cluster start. Timeout at 2023-05-16 04:06:17.655839017 +1400 +14 m=+14.458420652, 105.342123ms after stability +testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.656+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: aborting pipeline replication: peer="{Voter server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 2db3bca4-7544-c836-6f8a-94ac1a7a8c40}" + testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.656+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: aborting pipeline replication: peer="{Voter server-2db3bca4-7544-c836-6f8a-94ac1a7a8c40 2db3bca4-7544-c836-6f8a-94ac1a7a8c40}" +testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.656+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: aborting pipeline replication: peer="{Voter server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d}" + testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.656+1400 [INFO] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: aborting pipeline replication: peer="{Voter server-b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d}" +testLoggerAdapter verbose: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.707+1400 [ERROR] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: failed to heartbeat to: peer=b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d error="command timed out" + testing.go:149: server-2b87f816-3ce9-00ec-811b-1f6d404b6331: 2023-05-16T04:06:17.707+1400 [ERROR] server-2b87f816-3ce9-00ec-811b-1f6d404b6331: failed to heartbeat to: peer=b9fd0b8a-4f10-f30b-364d-33f8cb3efd7d error="command timed out" +--- PASS: TestRaft_VerifyLeader (0.24s) === RUN TestRaft_VerifyLeader_Single -testLoggerAdapter verbose: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.028-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d22b69d4-d137-d51b-7a09-477cfcb9f193 Address:d22b69d4-d137-d51b-7a09-477cfcb9f193}]" - testing.go:149: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.028-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d22b69d4-d137-d51b-7a09-477cfcb9f193 Address:d22b69d4-d137-d51b-7a09-477cfcb9f193}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.028-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:37.028-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.028-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: entering follower state: follower="Node at d22b69d4-d137-d51b-7a09-477cfcb9f193 [Follower]" leader= - testing.go:149: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.028-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: entering follower state: follower="Node at d22b69d4-d137-d51b-7a09-477cfcb9f193 [Follower]" leader= -testLoggerAdapter verbose: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.092-1200 [WARN] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.092-1200 [WARN] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.093-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: entering candidate state: node="Node at d22b69d4-d137-d51b-7a09-477cfcb9f193 [Candidate]" term=2 - testing.go:149: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.093-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: entering candidate state: node="Node at d22b69d4-d137-d51b-7a09-477cfcb9f193 [Candidate]" term=2 -testLoggerAdapter verbose: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.093-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: election won: tally=1 - testing.go:149: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.093-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: election won: tally=1 -testLoggerAdapter verbose: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.093-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: entering leader state: leader="Node at d22b69d4-d137-d51b-7a09-477cfcb9f193 [Leader]" - testing.go:149: server-d22b69d4-d137-d51b-7a09-477cfcb9f193: 2024-06-16T08:26:37.093-1200 [INFO] server-d22b69d4-d137-d51b-7a09-477cfcb9f193: entering leader state: leader="Node at d22b69d4-d137-d51b-7a09-477cfcb9f193 [Leader]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.199-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:37.09335442 -1200 -12 m=+14.100670921 (1 nodes), 64.97168ms from start of poll, 65.33102ms from cluster start. Timeout at 2024-06-16 08:26:37.19919004 -1200 -12 m=+14.206506521, 105.8356ms after stability - testing.go:149: cluster: 2024-06-16T08:26:37.199-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:37.09335442 -1200 -12 m=+14.100670921 (1 nodes), 64.97168ms from start of poll, 65.33102ms from cluster start. Timeout at 2024-06-16 08:26:37.19919004 -1200 -12 m=+14.206506521, 105.8356ms after stability ---- PASS: TestRaft_VerifyLeader_Single (0.17s) +testLoggerAdapter verbose: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.708+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11a64167-65eb-1084-fa9c-f340e70b44b0 Address:11a64167-65eb-1084-fa9c-f340e70b44b0}]" + testing.go:149: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.708+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-11a64167-65eb-1084-fa9c-f340e70b44b0 Address:11a64167-65eb-1084-fa9c-f340e70b44b0}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:17.708+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:17.708+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.708+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: entering follower state: follower="Node at 11a64167-65eb-1084-fa9c-f340e70b44b0 [Follower]" leader= + testing.go:149: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.708+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: entering follower state: follower="Node at 11a64167-65eb-1084-fa9c-f340e70b44b0 [Follower]" leader= +testLoggerAdapter verbose: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.803+1400 [WARN] server-11a64167-65eb-1084-fa9c-f340e70b44b0: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.803+1400 [WARN] server-11a64167-65eb-1084-fa9c-f340e70b44b0: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.803+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: entering candidate state: node="Node at 11a64167-65eb-1084-fa9c-f340e70b44b0 [Candidate]" term=2 + testing.go:149: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.803+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: entering candidate state: node="Node at 11a64167-65eb-1084-fa9c-f340e70b44b0 [Candidate]" term=2 +testLoggerAdapter verbose: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.803+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: election won: tally=1 + testing.go:149: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.803+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: election won: tally=1 +testLoggerAdapter verbose: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.803+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: entering leader state: leader="Node at 11a64167-65eb-1084-fa9c-f340e70b44b0 [Leader]" + testing.go:149: server-11a64167-65eb-1084-fa9c-f340e70b44b0: 2023-05-16T04:06:17.803+1400 [INFO] server-11a64167-65eb-1084-fa9c-f340e70b44b0: entering leader state: leader="Node at 11a64167-65eb-1084-fa9c-f340e70b44b0 [Leader]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:17.911+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.804093487 +1400 +14 m=+14.606675102 (1 nodes), 95.358865ms from start of poll, 95.79595ms from cluster start. Timeout at 2023-05-16 04:06:17.911871299 +1400 +14 m=+14.714453014, 107.777912ms after stability + testing.go:149: cluster: 2023-05-16T04:06:17.911+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.804093487 +1400 +14 m=+14.606675102 (1 nodes), 95.358865ms from start of poll, 95.79595ms from cluster start. Timeout at 2023-05-16 04:06:17.911871299 +1400 +14 m=+14.714453014, 107.777912ms after stability +--- PASS: TestRaft_VerifyLeader_Single (0.20s) === RUN TestRaft_VerifyLeader_Fail -testLoggerAdapter verbose: server-7bc6efaf-433f-16bd-6dd9-b487baff2f46: 2024-06-16T08:26:37.200-1200 [INFO] server-7bc6efaf-433f-16bd-6dd9-b487baff2f46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 Address:7bc6efaf-433f-16bd-6dd9-b487baff2f46} {Suffrage:Voter ID:server-486bdc5b-8954-f333-21ce-9a0723b174a8 Address:486bdc5b-8954-f333-21ce-9a0723b174a8}]" - testing.go:149: server-7bc6efaf-433f-16bd-6dd9-b487baff2f46: 2024-06-16T08:26:37.200-1200 [INFO] server-7bc6efaf-433f-16bd-6dd9-b487baff2f46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 Address:7bc6efaf-433f-16bd-6dd9-b487baff2f46} {Suffrage:Voter ID:server-486bdc5b-8954-f333-21ce-9a0723b174a8 Address:486bdc5b-8954-f333-21ce-9a0723b174a8}]" -testLoggerAdapter verbose: server-7bc6efaf-433f-16bd-6dd9-b487baff2f46: 2024-06-16T08:26:37.200-1200 [INFO] server-7bc6efaf-433f-16bd-6dd9-b487baff2f46: entering follower state: follower="Node at 7bc6efaf-433f-16bd-6dd9-b487baff2f46 [Follower]" leader= - testing.go:149: server-7bc6efaf-433f-16bd-6dd9-b487baff2f46: 2024-06-16T08:26:37.200-1200 [INFO] server-7bc6efaf-433f-16bd-6dd9-b487baff2f46: entering follower state: follower="Node at 7bc6efaf-433f-16bd-6dd9-b487baff2f46 [Follower]" leader= -testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.200-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 Address:7bc6efaf-433f-16bd-6dd9-b487baff2f46} {Suffrage:Voter ID:server-486bdc5b-8954-f333-21ce-9a0723b174a8 Address:486bdc5b-8954-f333-21ce-9a0723b174a8}]" - testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.200-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 Address:7bc6efaf-433f-16bd-6dd9-b487baff2f46} {Suffrage:Voter ID:server-486bdc5b-8954-f333-21ce-9a0723b174a8 Address:486bdc5b-8954-f333-21ce-9a0723b174a8}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.200-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:37.200-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.200-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: entering follower state: follower="Node at 486bdc5b-8954-f333-21ce-9a0723b174a8 [Follower]" leader= - testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.200-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: entering follower state: follower="Node at 486bdc5b-8954-f333-21ce-9a0723b174a8 [Follower]" leader= -testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.266-1200 [WARN] server-486bdc5b-8954-f333-21ce-9a0723b174a8: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.266-1200 [WARN] server-486bdc5b-8954-f333-21ce-9a0723b174a8: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.266-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: entering candidate state: node="Node at 486bdc5b-8954-f333-21ce-9a0723b174a8 [Candidate]" term=2 - testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.266-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: entering candidate state: node="Node at 486bdc5b-8954-f333-21ce-9a0723b174a8 [Candidate]" term=2 -testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.267-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: election won: tally=2 - testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.267-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: election won: tally=2 -testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.267-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: entering leader state: leader="Node at 486bdc5b-8954-f333-21ce-9a0723b174a8 [Leader]" - testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.267-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: entering leader state: leader="Node at 486bdc5b-8954-f333-21ce-9a0723b174a8 [Leader]" -testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.267-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: added peer, starting replication: peer=server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 - testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.267-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: added peer, starting replication: peer=server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 -testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.267-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: pipelining replication: peer="{Voter server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 7bc6efaf-433f-16bd-6dd9-b487baff2f46}" - testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.267-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: pipelining replication: peer="{Voter server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 7bc6efaf-433f-16bd-6dd9-b487baff2f46}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.372-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:37.2671909 -1200 -12 m=+14.274507421 (1 nodes), 66.4374ms from start of poll, 67.04816ms from cluster start. Timeout at 2024-06-16 08:26:37.37239802 -1200 -12 m=+14.379714481, 105.20706ms after stability - testing.go:149: cluster: 2024-06-16T08:26:37.372-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:37.2671909 -1200 -12 m=+14.274507421 (1 nodes), 66.4374ms from start of poll, 67.04816ms from cluster start. Timeout at 2024-06-16 08:26:37.37239802 -1200 -12 m=+14.379714481, 105.20706ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.372-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:37.372-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.478-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:37.3726227 -1200 -12 m=+14.379939181 (1 nodes), 1.3µs from start of poll, 172.47992ms from cluster start. Timeout at 2024-06-16 08:26:37.47856566 -1200 -12 m=+14.485882121, 105.94294ms after stability - testing.go:149: cluster: 2024-06-16T08:26:37.478-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:37.3726227 -1200 -12 m=+14.379939181 (1 nodes), 1.3µs from start of poll, 172.47992ms from cluster start. Timeout at 2024-06-16 08:26:37.47856566 -1200 -12 m=+14.485882121, 105.94294ms after stability -testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.478-1200 [WARN] server-486bdc5b-8954-f333-21ce-9a0723b174a8: new leader elected, stepping down - testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.478-1200 [WARN] server-486bdc5b-8954-f333-21ce-9a0723b174a8: new leader elected, stepping down -testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.478-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: entering follower state: follower="Node at 486bdc5b-8954-f333-21ce-9a0723b174a8 [Follower]" leader= - testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.478-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: entering follower state: follower="Node at 486bdc5b-8954-f333-21ce-9a0723b174a8 [Follower]" leader= -testLoggerAdapter verbose: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.478-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: aborting pipeline replication: peer="{Voter server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 7bc6efaf-433f-16bd-6dd9-b487baff2f46}" - testing.go:149: server-486bdc5b-8954-f333-21ce-9a0723b174a8: 2024-06-16T08:26:37.478-1200 [INFO] server-486bdc5b-8954-f333-21ce-9a0723b174a8: aborting pipeline replication: peer="{Voter server-7bc6efaf-433f-16bd-6dd9-b487baff2f46 7bc6efaf-433f-16bd-6dd9-b487baff2f46}" ---- PASS: TestRaft_VerifyLeader_Fail (0.28s) +testLoggerAdapter verbose: server-041ad9e3-9828-91b6-adeb-e58df252c453: 2023-05-16T04:06:17.913+1400 [INFO] server-041ad9e3-9828-91b6-adeb-e58df252c453: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-041ad9e3-9828-91b6-adeb-e58df252c453 Address:041ad9e3-9828-91b6-adeb-e58df252c453} {Suffrage:Voter ID:server-6e35990a-9189-393b-58ad-d8792d9e41fa Address:6e35990a-9189-393b-58ad-d8792d9e41fa}]" + testing.go:149: server-041ad9e3-9828-91b6-adeb-e58df252c453: 2023-05-16T04:06:17.913+1400 [INFO] server-041ad9e3-9828-91b6-adeb-e58df252c453: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-041ad9e3-9828-91b6-adeb-e58df252c453 Address:041ad9e3-9828-91b6-adeb-e58df252c453} {Suffrage:Voter ID:server-6e35990a-9189-393b-58ad-d8792d9e41fa Address:6e35990a-9189-393b-58ad-d8792d9e41fa}]" +testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.913+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-041ad9e3-9828-91b6-adeb-e58df252c453 Address:041ad9e3-9828-91b6-adeb-e58df252c453} {Suffrage:Voter ID:server-6e35990a-9189-393b-58ad-d8792d9e41fa Address:6e35990a-9189-393b-58ad-d8792d9e41fa}]" + testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.913+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-041ad9e3-9828-91b6-adeb-e58df252c453 Address:041ad9e3-9828-91b6-adeb-e58df252c453} {Suffrage:Voter ID:server-6e35990a-9189-393b-58ad-d8792d9e41fa Address:6e35990a-9189-393b-58ad-d8792d9e41fa}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:17.913+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:17.913+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-041ad9e3-9828-91b6-adeb-e58df252c453: 2023-05-16T04:06:17.913+1400 [INFO] server-041ad9e3-9828-91b6-adeb-e58df252c453: entering follower state: follower="Node at 041ad9e3-9828-91b6-adeb-e58df252c453 [Follower]" leader= + testing.go:149: server-041ad9e3-9828-91b6-adeb-e58df252c453: 2023-05-16T04:06:17.913+1400 [INFO] server-041ad9e3-9828-91b6-adeb-e58df252c453: entering follower state: follower="Node at 041ad9e3-9828-91b6-adeb-e58df252c453 [Follower]" leader= +testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.913+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: entering follower state: follower="Node at 6e35990a-9189-393b-58ad-d8792d9e41fa [Follower]" leader= + testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.913+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: entering follower state: follower="Node at 6e35990a-9189-393b-58ad-d8792d9e41fa [Follower]" leader= +testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.966+1400 [WARN] server-6e35990a-9189-393b-58ad-d8792d9e41fa: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.966+1400 [WARN] server-6e35990a-9189-393b-58ad-d8792d9e41fa: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: entering candidate state: node="Node at 6e35990a-9189-393b-58ad-d8792d9e41fa [Candidate]" term=2 + testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: entering candidate state: node="Node at 6e35990a-9189-393b-58ad-d8792d9e41fa [Candidate]" term=2 +testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: election won: tally=2 + testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: election won: tally=2 +testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: entering leader state: leader="Node at 6e35990a-9189-393b-58ad-d8792d9e41fa [Leader]" + testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: entering leader state: leader="Node at 6e35990a-9189-393b-58ad-d8792d9e41fa [Leader]" +testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: added peer, starting replication: peer=server-041ad9e3-9828-91b6-adeb-e58df252c453 + testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: added peer, starting replication: peer=server-041ad9e3-9828-91b6-adeb-e58df252c453 +testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: pipelining replication: peer="{Voter server-041ad9e3-9828-91b6-adeb-e58df252c453 041ad9e3-9828-91b6-adeb-e58df252c453}" + testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:17.967+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: pipelining replication: peer="{Voter server-041ad9e3-9828-91b6-adeb-e58df252c453 041ad9e3-9828-91b6-adeb-e58df252c453}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.073+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.967825459 +1400 +14 m=+14.770407094 (1 nodes), 54.048998ms from start of poll, 54.718045ms from cluster start. Timeout at 2023-05-16 04:06:18.073493106 +1400 +14 m=+14.876074761, 105.667667ms after stability + testing.go:149: cluster: 2023-05-16T04:06:18.073+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:17.967825459 +1400 +14 m=+14.770407094 (1 nodes), 54.048998ms from start of poll, 54.718045ms from cluster start. Timeout at 2023-05-16 04:06:18.073493106 +1400 +14 m=+14.876074761, 105.667667ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.073+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:18.073+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.179+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:18.073756829 +1400 +14 m=+14.876338484 (1 nodes), 2.5µs from start of poll, 160.649435ms from cluster start. Timeout at 2023-05-16 04:06:18.179544818 +1400 +14 m=+14.982126433, 105.787949ms after stability + testing.go:149: cluster: 2023-05-16T04:06:18.179+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:18.073756829 +1400 +14 m=+14.876338484 (1 nodes), 2.5µs from start of poll, 160.649435ms from cluster start. Timeout at 2023-05-16 04:06:18.179544818 +1400 +14 m=+14.982126433, 105.787949ms after stability +testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:18.179+1400 [WARN] server-6e35990a-9189-393b-58ad-d8792d9e41fa: new leader elected, stepping down + testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:18.179+1400 [WARN] server-6e35990a-9189-393b-58ad-d8792d9e41fa: new leader elected, stepping down +testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:18.179+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: entering follower state: follower="Node at 6e35990a-9189-393b-58ad-d8792d9e41fa [Follower]" leader= + testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:18.179+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: entering follower state: follower="Node at 6e35990a-9189-393b-58ad-d8792d9e41fa [Follower]" leader= +testLoggerAdapter verbose: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:18.180+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: aborting pipeline replication: peer="{Voter server-041ad9e3-9828-91b6-adeb-e58df252c453 041ad9e3-9828-91b6-adeb-e58df252c453}" + testing.go:149: server-6e35990a-9189-393b-58ad-d8792d9e41fa: 2023-05-16T04:06:18.180+1400 [INFO] server-6e35990a-9189-393b-58ad-d8792d9e41fa: aborting pipeline replication: peer="{Voter server-041ad9e3-9828-91b6-adeb-e58df252c453 041ad9e3-9828-91b6-adeb-e58df252c453}" +--- PASS: TestRaft_VerifyLeader_Fail (0.27s) === RUN TestRaft_VerifyLeader_PartialConnect -testLoggerAdapter verbose: server-9302f777-e9f9-27e6-cce1-8b1b5c384063: 2024-06-16T08:26:37.480-1200 [INFO] server-9302f777-e9f9-27e6-cce1-8b1b5c384063: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9302f777-e9f9-27e6-cce1-8b1b5c384063 Address:9302f777-e9f9-27e6-cce1-8b1b5c384063} {Suffrage:Voter ID:server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 Address:9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62} {Suffrage:Voter ID:server-d4637939-e57b-e172-71e0-4c15fcf51275 Address:d4637939-e57b-e172-71e0-4c15fcf51275}]" - testing.go:149: server-9302f777-e9f9-27e6-cce1-8b1b5c384063: 2024-06-16T08:26:37.480-1200 [INFO] server-9302f777-e9f9-27e6-cce1-8b1b5c384063: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9302f777-e9f9-27e6-cce1-8b1b5c384063 Address:9302f777-e9f9-27e6-cce1-8b1b5c384063} {Suffrage:Voter ID:server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 Address:9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62} {Suffrage:Voter ID:server-d4637939-e57b-e172-71e0-4c15fcf51275 Address:d4637939-e57b-e172-71e0-4c15fcf51275}]" -testLoggerAdapter verbose: server-9302f777-e9f9-27e6-cce1-8b1b5c384063: 2024-06-16T08:26:37.480-1200 [INFO] server-9302f777-e9f9-27e6-cce1-8b1b5c384063: entering follower state: follower="Node at 9302f777-e9f9-27e6-cce1-8b1b5c384063 [Follower]" leader= - testing.go:149: server-9302f777-e9f9-27e6-cce1-8b1b5c384063: 2024-06-16T08:26:37.480-1200 [INFO] server-9302f777-e9f9-27e6-cce1-8b1b5c384063: entering follower state: follower="Node at 9302f777-e9f9-27e6-cce1-8b1b5c384063 [Follower]" leader= -testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.480-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9302f777-e9f9-27e6-cce1-8b1b5c384063 Address:9302f777-e9f9-27e6-cce1-8b1b5c384063} {Suffrage:Voter ID:server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 Address:9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62} {Suffrage:Voter ID:server-d4637939-e57b-e172-71e0-4c15fcf51275 Address:d4637939-e57b-e172-71e0-4c15fcf51275}]" - testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.480-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9302f777-e9f9-27e6-cce1-8b1b5c384063 Address:9302f777-e9f9-27e6-cce1-8b1b5c384063} {Suffrage:Voter ID:server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 Address:9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62} {Suffrage:Voter ID:server-d4637939-e57b-e172-71e0-4c15fcf51275 Address:d4637939-e57b-e172-71e0-4c15fcf51275}]" -testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.480-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: entering follower state: follower="Node at 9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 [Follower]" leader= - testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.480-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: entering follower state: follower="Node at 9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 [Follower]" leader= -testLoggerAdapter verbose: server-d4637939-e57b-e172-71e0-4c15fcf51275: 2024-06-16T08:26:37.480-1200 [INFO] server-d4637939-e57b-e172-71e0-4c15fcf51275: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9302f777-e9f9-27e6-cce1-8b1b5c384063 Address:9302f777-e9f9-27e6-cce1-8b1b5c384063} {Suffrage:Voter ID:server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 Address:9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62} {Suffrage:Voter ID:server-d4637939-e57b-e172-71e0-4c15fcf51275 Address:d4637939-e57b-e172-71e0-4c15fcf51275}]" - testing.go:149: server-d4637939-e57b-e172-71e0-4c15fcf51275: 2024-06-16T08:26:37.480-1200 [INFO] server-d4637939-e57b-e172-71e0-4c15fcf51275: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9302f777-e9f9-27e6-cce1-8b1b5c384063 Address:9302f777-e9f9-27e6-cce1-8b1b5c384063} {Suffrage:Voter ID:server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 Address:9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62} {Suffrage:Voter ID:server-d4637939-e57b-e172-71e0-4c15fcf51275 Address:d4637939-e57b-e172-71e0-4c15fcf51275}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.481-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:37.481-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-d4637939-e57b-e172-71e0-4c15fcf51275: 2024-06-16T08:26:37.481-1200 [INFO] server-d4637939-e57b-e172-71e0-4c15fcf51275: entering follower state: follower="Node at d4637939-e57b-e172-71e0-4c15fcf51275 [Follower]" leader= - testing.go:149: server-d4637939-e57b-e172-71e0-4c15fcf51275: 2024-06-16T08:26:37.481-1200 [INFO] server-d4637939-e57b-e172-71e0-4c15fcf51275: entering follower state: follower="Node at d4637939-e57b-e172-71e0-4c15fcf51275 [Follower]" leader= -testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.558-1200 [WARN] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.558-1200 [WARN] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.558-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: entering candidate state: node="Node at 9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 [Candidate]" term=2 - testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.558-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: entering candidate state: node="Node at 9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 [Candidate]" term=2 -testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: election won: tally=2 - testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: election won: tally=2 -testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: entering leader state: leader="Node at 9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 [Leader]" - testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: entering leader state: leader="Node at 9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62 [Leader]" -testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: added peer, starting replication: peer=server-9302f777-e9f9-27e6-cce1-8b1b5c384063 - testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: added peer, starting replication: peer=server-9302f777-e9f9-27e6-cce1-8b1b5c384063 -testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: added peer, starting replication: peer=server-d4637939-e57b-e172-71e0-4c15fcf51275 - testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: added peer, starting replication: peer=server-d4637939-e57b-e172-71e0-4c15fcf51275 -testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: pipelining replication: peer="{Voter server-9302f777-e9f9-27e6-cce1-8b1b5c384063 9302f777-e9f9-27e6-cce1-8b1b5c384063}" - testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: pipelining replication: peer="{Voter server-9302f777-e9f9-27e6-cce1-8b1b5c384063 9302f777-e9f9-27e6-cce1-8b1b5c384063}" -testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: pipelining replication: peer="{Voter server-d4637939-e57b-e172-71e0-4c15fcf51275 d4637939-e57b-e172-71e0-4c15fcf51275}" - testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.559-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: pipelining replication: peer="{Voter server-d4637939-e57b-e172-71e0-4c15fcf51275 d4637939-e57b-e172-71e0-4c15fcf51275}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.665-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:37.55915026 -1200 -12 m=+14.566466721 (1 nodes), 78.08304ms from start of poll, 79.02318ms from cluster start. Timeout at 2024-06-16 08:26:37.66515364 -1200 -12 m=+14.672470121, 106.0034ms after stability - testing.go:149: cluster: 2024-06-16T08:26:37.665-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:37.55915026 -1200 -12 m=+14.566466721 (1 nodes), 78.08304ms from start of poll, 79.02318ms from cluster start. Timeout at 2024-06-16 08:26:37.66515364 -1200 -12 m=+14.672470121, 106.0034ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.670-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:37.670-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.776-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:37.67066182 -1200 -12 m=+14.677978301 (2 nodes), 2.5µs from start of poll, 190.53476ms from cluster start. Timeout at 2024-06-16 08:26:37.77631462 -1200 -12 m=+14.783631081, 105.65278ms after stability - testing.go:149: cluster: 2024-06-16T08:26:37.776-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:37.67066182 -1200 -12 m=+14.677978301 (2 nodes), 2.5µs from start of poll, 190.53476ms from cluster start. Timeout at 2024-06-16 08:26:37.77631462 -1200 -12 m=+14.783631081, 105.65278ms after stability - raft_test.go:1647: [INFO] Disconnecting Node at 9302f777-e9f9-27e6-cce1-8b1b5c384063 [Follower] -testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.776-1200 [ERROR] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: failed to heartbeat to: peer=9302f777-e9f9-27e6-cce1-8b1b5c384063 error="failed to connect to peer: 9302f777-e9f9-27e6-cce1-8b1b5c384063" - testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.776-1200 [ERROR] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: failed to heartbeat to: peer=9302f777-e9f9-27e6-cce1-8b1b5c384063 error="failed to connect to peer: 9302f777-e9f9-27e6-cce1-8b1b5c384063" -testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.776-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: aborting pipeline replication: peer="{Voter server-d4637939-e57b-e172-71e0-4c15fcf51275 d4637939-e57b-e172-71e0-4c15fcf51275}" - testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.776-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: aborting pipeline replication: peer="{Voter server-d4637939-e57b-e172-71e0-4c15fcf51275 d4637939-e57b-e172-71e0-4c15fcf51275}" -testLoggerAdapter verbose: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.776-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: aborting pipeline replication: peer="{Voter server-9302f777-e9f9-27e6-cce1-8b1b5c384063 9302f777-e9f9-27e6-cce1-8b1b5c384063}" - testing.go:149: server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: 2024-06-16T08:26:37.776-1200 [INFO] server-9e4909bb-4c1b-c9d1-ccbd-0e0b9a3eed62: aborting pipeline replication: peer="{Voter server-9302f777-e9f9-27e6-cce1-8b1b5c384063 9302f777-e9f9-27e6-cce1-8b1b5c384063}" ---- PASS: TestRaft_VerifyLeader_PartialConnect (0.30s) +testLoggerAdapter verbose: server-898c8559-a733-964d-1704-e94acb5b2708: 2023-05-16T04:06:18.181+1400 [INFO] server-898c8559-a733-964d-1704-e94acb5b2708: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-898c8559-a733-964d-1704-e94acb5b2708 Address:898c8559-a733-964d-1704-e94acb5b2708} {Suffrage:Voter ID:server-041eaa1e-a03b-7d12-5024-2cd990b6e860 Address:041eaa1e-a03b-7d12-5024-2cd990b6e860} {Suffrage:Voter ID:server-9f38cc7c-059a-daae-bbea-8063509e276f Address:9f38cc7c-059a-daae-bbea-8063509e276f}]" + testing.go:149: server-898c8559-a733-964d-1704-e94acb5b2708: 2023-05-16T04:06:18.181+1400 [INFO] server-898c8559-a733-964d-1704-e94acb5b2708: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-898c8559-a733-964d-1704-e94acb5b2708 Address:898c8559-a733-964d-1704-e94acb5b2708} {Suffrage:Voter ID:server-041eaa1e-a03b-7d12-5024-2cd990b6e860 Address:041eaa1e-a03b-7d12-5024-2cd990b6e860} {Suffrage:Voter ID:server-9f38cc7c-059a-daae-bbea-8063509e276f Address:9f38cc7c-059a-daae-bbea-8063509e276f}]" +testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.181+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-898c8559-a733-964d-1704-e94acb5b2708 Address:898c8559-a733-964d-1704-e94acb5b2708} {Suffrage:Voter ID:server-041eaa1e-a03b-7d12-5024-2cd990b6e860 Address:041eaa1e-a03b-7d12-5024-2cd990b6e860} {Suffrage:Voter ID:server-9f38cc7c-059a-daae-bbea-8063509e276f Address:9f38cc7c-059a-daae-bbea-8063509e276f}]" + testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.181+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-898c8559-a733-964d-1704-e94acb5b2708 Address:898c8559-a733-964d-1704-e94acb5b2708} {Suffrage:Voter ID:server-041eaa1e-a03b-7d12-5024-2cd990b6e860 Address:041eaa1e-a03b-7d12-5024-2cd990b6e860} {Suffrage:Voter ID:server-9f38cc7c-059a-daae-bbea-8063509e276f Address:9f38cc7c-059a-daae-bbea-8063509e276f}]" +testLoggerAdapter verbose: server-9f38cc7c-059a-daae-bbea-8063509e276f: 2023-05-16T04:06:18.182+1400 [INFO] server-9f38cc7c-059a-daae-bbea-8063509e276f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-898c8559-a733-964d-1704-e94acb5b2708 Address:898c8559-a733-964d-1704-e94acb5b2708} {Suffrage:Voter ID:server-041eaa1e-a03b-7d12-5024-2cd990b6e860 Address:041eaa1e-a03b-7d12-5024-2cd990b6e860} {Suffrage:Voter ID:server-9f38cc7c-059a-daae-bbea-8063509e276f Address:9f38cc7c-059a-daae-bbea-8063509e276f}]" + testing.go:149: server-9f38cc7c-059a-daae-bbea-8063509e276f: 2023-05-16T04:06:18.182+1400 [INFO] server-9f38cc7c-059a-daae-bbea-8063509e276f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-898c8559-a733-964d-1704-e94acb5b2708 Address:898c8559-a733-964d-1704-e94acb5b2708} {Suffrage:Voter ID:server-041eaa1e-a03b-7d12-5024-2cd990b6e860 Address:041eaa1e-a03b-7d12-5024-2cd990b6e860} {Suffrage:Voter ID:server-9f38cc7c-059a-daae-bbea-8063509e276f Address:9f38cc7c-059a-daae-bbea-8063509e276f}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.182+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:18.182+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-898c8559-a733-964d-1704-e94acb5b2708: 2023-05-16T04:06:18.182+1400 [INFO] server-898c8559-a733-964d-1704-e94acb5b2708: entering follower state: follower="Node at 898c8559-a733-964d-1704-e94acb5b2708 [Follower]" leader= + testing.go:149: server-898c8559-a733-964d-1704-e94acb5b2708: 2023-05-16T04:06:18.182+1400 [INFO] server-898c8559-a733-964d-1704-e94acb5b2708: entering follower state: follower="Node at 898c8559-a733-964d-1704-e94acb5b2708 [Follower]" leader= +testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.182+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: entering follower state: follower="Node at 041eaa1e-a03b-7d12-5024-2cd990b6e860 [Follower]" leader= + testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.182+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: entering follower state: follower="Node at 041eaa1e-a03b-7d12-5024-2cd990b6e860 [Follower]" leader= +testLoggerAdapter verbose: server-9f38cc7c-059a-daae-bbea-8063509e276f: 2023-05-16T04:06:18.182+1400 [INFO] server-9f38cc7c-059a-daae-bbea-8063509e276f: entering follower state: follower="Node at 9f38cc7c-059a-daae-bbea-8063509e276f [Follower]" leader= + testing.go:149: server-9f38cc7c-059a-daae-bbea-8063509e276f: 2023-05-16T04:06:18.182+1400 [INFO] server-9f38cc7c-059a-daae-bbea-8063509e276f: entering follower state: follower="Node at 9f38cc7c-059a-daae-bbea-8063509e276f [Follower]" leader= +testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [WARN] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [WARN] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: entering candidate state: node="Node at 041eaa1e-a03b-7d12-5024-2cd990b6e860 [Candidate]" term=2 + testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: entering candidate state: node="Node at 041eaa1e-a03b-7d12-5024-2cd990b6e860 [Candidate]" term=2 +testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: election won: tally=2 + testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: election won: tally=2 +testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: entering leader state: leader="Node at 041eaa1e-a03b-7d12-5024-2cd990b6e860 [Leader]" + testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: entering leader state: leader="Node at 041eaa1e-a03b-7d12-5024-2cd990b6e860 [Leader]" +testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: added peer, starting replication: peer=server-898c8559-a733-964d-1704-e94acb5b2708 + testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: added peer, starting replication: peer=server-898c8559-a733-964d-1704-e94acb5b2708 +testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: added peer, starting replication: peer=server-9f38cc7c-059a-daae-bbea-8063509e276f + testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.240+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: added peer, starting replication: peer=server-9f38cc7c-059a-daae-bbea-8063509e276f +testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.241+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: pipelining replication: peer="{Voter server-9f38cc7c-059a-daae-bbea-8063509e276f 9f38cc7c-059a-daae-bbea-8063509e276f}" + testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.241+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: pipelining replication: peer="{Voter server-9f38cc7c-059a-daae-bbea-8063509e276f 9f38cc7c-059a-daae-bbea-8063509e276f}" +testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.241+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: pipelining replication: peer="{Voter server-898c8559-a733-964d-1704-e94acb5b2708 898c8559-a733-964d-1704-e94acb5b2708}" + testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.241+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: pipelining replication: peer="{Voter server-898c8559-a733-964d-1704-e94acb5b2708 898c8559-a733-964d-1704-e94acb5b2708}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.347+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:18.241539909 +1400 +14 m=+15.044121544 (1 nodes), 59.299559ms from start of poll, 60.246931ms from cluster start. Timeout at 2023-05-16 04:06:18.347027614 +1400 +14 m=+15.149609249, 105.487705ms after stability + testing.go:149: cluster: 2023-05-16T04:06:18.347+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:18.241539909 +1400 +14 m=+15.044121544 (1 nodes), 59.299559ms from start of poll, 60.246931ms from cluster start. Timeout at 2023-05-16 04:06:18.347027614 +1400 +14 m=+15.149609249, 105.487705ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.353+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:18.353+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.459+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:18.353335469 +1400 +14 m=+15.155917104 (2 nodes), 3.26µs from start of poll, 172.042491ms from cluster start. Timeout at 2023-05-16 04:06:18.459060116 +1400 +14 m=+15.261641751, 105.724647ms after stability + testing.go:149: cluster: 2023-05-16T04:06:18.459+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:18.353335469 +1400 +14 m=+15.155917104 (2 nodes), 3.26µs from start of poll, 172.042491ms from cluster start. Timeout at 2023-05-16 04:06:18.459060116 +1400 +14 m=+15.261641751, 105.724647ms after stability + raft_test.go:1647: [INFO] Disconnecting Node at 898c8559-a733-964d-1704-e94acb5b2708 [Follower] +testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.459+1400 [ERROR] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: failed to heartbeat to: peer=898c8559-a733-964d-1704-e94acb5b2708 error="failed to connect to peer: 898c8559-a733-964d-1704-e94acb5b2708" + testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.459+1400 [ERROR] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: failed to heartbeat to: peer=898c8559-a733-964d-1704-e94acb5b2708 error="failed to connect to peer: 898c8559-a733-964d-1704-e94acb5b2708" +testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.459+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: aborting pipeline replication: peer="{Voter server-9f38cc7c-059a-daae-bbea-8063509e276f 9f38cc7c-059a-daae-bbea-8063509e276f}" + testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.459+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: aborting pipeline replication: peer="{Voter server-9f38cc7c-059a-daae-bbea-8063509e276f 9f38cc7c-059a-daae-bbea-8063509e276f}" +testLoggerAdapter verbose: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.459+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: aborting pipeline replication: peer="{Voter server-898c8559-a733-964d-1704-e94acb5b2708 898c8559-a733-964d-1704-e94acb5b2708}" + testing.go:149: server-041eaa1e-a03b-7d12-5024-2cd990b6e860: 2023-05-16T04:06:18.459+1400 [INFO] server-041eaa1e-a03b-7d12-5024-2cd990b6e860: aborting pipeline replication: peer="{Voter server-898c8559-a733-964d-1704-e94acb5b2708 898c8559-a733-964d-1704-e94acb5b2708}" +--- PASS: TestRaft_VerifyLeader_PartialConnect (0.28s) === RUN TestRaft_StartAsLeader -testLoggerAdapter verbose: server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765: 2024-06-16T08:26:37.778-1200 [INFO] server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765 Address:8716a8e2-6f8f-d72c-c28a-bbfc84c20765}]" - testing.go:149: server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765: 2024-06-16T08:26:37.778-1200 [INFO] server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765 Address:8716a8e2-6f8f-d72c-c28a-bbfc84c20765}]" -testLoggerAdapter verbose: server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765: 2024-06-16T08:26:37.778-1200 [INFO] server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765: entering leader state: leader="Node at 8716a8e2-6f8f-d72c-c28a-bbfc84c20765 [Leader]" - testing.go:149: server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765: 2024-06-16T08:26:37.778-1200 [INFO] server-8716a8e2-6f8f-d72c-c28a-bbfc84c20765: entering leader state: leader="Node at 8716a8e2-6f8f-d72c-c28a-bbfc84c20765 [Leader]" +testLoggerAdapter verbose: server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1: 2023-05-16T04:06:18.460+1400 [INFO] server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1 Address:2af8cc22-51eb-7b07-74b5-19ad6d1eefa1}]" + testing.go:149: server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1: 2023-05-16T04:06:18.460+1400 [INFO] server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1 Address:2af8cc22-51eb-7b07-74b5-19ad6d1eefa1}]" +testLoggerAdapter verbose: server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1: 2023-05-16T04:06:18.461+1400 [INFO] server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1: entering leader state: leader="Node at 2af8cc22-51eb-7b07-74b5-19ad6d1eefa1 [Leader]" + testing.go:149: server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1: 2023-05-16T04:06:18.461+1400 [INFO] server-2af8cc22-51eb-7b07-74b5-19ad6d1eefa1: entering leader state: leader="Node at 2af8cc22-51eb-7b07-74b5-19ad6d1eefa1 [Leader]" --- PASS: TestRaft_StartAsLeader (0.00s) === RUN TestRaft_NotifyCh -testLoggerAdapter verbose: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.778-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e23deb9-1301-c7f9-7649-5ffa948073cf Address:3e23deb9-1301-c7f9-7649-5ffa948073cf}]" - testing.go:149: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.778-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e23deb9-1301-c7f9-7649-5ffa948073cf Address:3e23deb9-1301-c7f9-7649-5ffa948073cf}]" -testLoggerAdapter verbose: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.779-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: entering follower state: follower="Node at 3e23deb9-1301-c7f9-7649-5ffa948073cf [Follower]" leader= - testing.go:149: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.779-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: entering follower state: follower="Node at 3e23deb9-1301-c7f9-7649-5ffa948073cf [Follower]" leader= -testLoggerAdapter verbose: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.830-1200 [WARN] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.830-1200 [WARN] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.830-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: entering candidate state: node="Node at 3e23deb9-1301-c7f9-7649-5ffa948073cf [Candidate]" term=2 - testing.go:149: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.830-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: entering candidate state: node="Node at 3e23deb9-1301-c7f9-7649-5ffa948073cf [Candidate]" term=2 -testLoggerAdapter verbose: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.831-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: election won: tally=1 - testing.go:149: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.831-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: election won: tally=1 -testLoggerAdapter verbose: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.831-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: entering leader state: leader="Node at 3e23deb9-1301-c7f9-7649-5ffa948073cf [Leader]" - testing.go:149: server-3e23deb9-1301-c7f9-7649-5ffa948073cf: 2024-06-16T08:26:37.831-1200 [INFO] server-3e23deb9-1301-c7f9-7649-5ffa948073cf: entering leader state: leader="Node at 3e23deb9-1301-c7f9-7649-5ffa948073cf [Leader]" ---- PASS: TestRaft_NotifyCh (0.05s) +testLoggerAdapter verbose: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.461+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ba23a44-267a-81b6-1805-e686655a6c81 Address:3ba23a44-267a-81b6-1805-e686655a6c81}]" + testing.go:149: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.461+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ba23a44-267a-81b6-1805-e686655a6c81 Address:3ba23a44-267a-81b6-1805-e686655a6c81}]" +testLoggerAdapter verbose: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.462+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: entering follower state: follower="Node at 3ba23a44-267a-81b6-1805-e686655a6c81 [Follower]" leader= + testing.go:149: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.462+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: entering follower state: follower="Node at 3ba23a44-267a-81b6-1805-e686655a6c81 [Follower]" leader= +testLoggerAdapter verbose: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.516+1400 [WARN] server-3ba23a44-267a-81b6-1805-e686655a6c81: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.516+1400 [WARN] server-3ba23a44-267a-81b6-1805-e686655a6c81: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.516+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: entering candidate state: node="Node at 3ba23a44-267a-81b6-1805-e686655a6c81 [Candidate]" term=2 + testing.go:149: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.516+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: entering candidate state: node="Node at 3ba23a44-267a-81b6-1805-e686655a6c81 [Candidate]" term=2 +testLoggerAdapter verbose: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.516+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: election won: tally=1 + testing.go:149: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.516+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: election won: tally=1 +testLoggerAdapter verbose: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.516+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: entering leader state: leader="Node at 3ba23a44-267a-81b6-1805-e686655a6c81 [Leader]" + testing.go:149: server-3ba23a44-267a-81b6-1805-e686655a6c81: 2023-05-16T04:06:18.516+1400 [INFO] server-3ba23a44-267a-81b6-1805-e686655a6c81: entering leader state: leader="Node at 3ba23a44-267a-81b6-1805-e686655a6c81 [Leader]" +--- PASS: TestRaft_NotifyCh (0.06s) === RUN TestRaft_Voting -testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.832-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7 Address:fb9848aa-fe9c-019f-2f64-85cc632d6ac7} {Suffrage:Voter ID:server-c0bab25e-7e19-06a0-2904-472e4a26c59e Address:c0bab25e-7e19-06a0-2904-472e4a26c59e} {Suffrage:Voter ID:server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 Address:d19d1332-f7ab-7cbb-7c64-ddf9777ae649}]" - testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.832-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7 Address:fb9848aa-fe9c-019f-2f64-85cc632d6ac7} {Suffrage:Voter ID:server-c0bab25e-7e19-06a0-2904-472e4a26c59e Address:c0bab25e-7e19-06a0-2904-472e4a26c59e} {Suffrage:Voter ID:server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 Address:d19d1332-f7ab-7cbb-7c64-ddf9777ae649}]" -testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.832-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: entering follower state: follower="Node at fb9848aa-fe9c-019f-2f64-85cc632d6ac7 [Follower]" leader= - testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.832-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: entering follower state: follower="Node at fb9848aa-fe9c-019f-2f64-85cc632d6ac7 [Follower]" leader= -testLoggerAdapter verbose: server-c0bab25e-7e19-06a0-2904-472e4a26c59e: 2024-06-16T08:26:37.832-1200 [INFO] server-c0bab25e-7e19-06a0-2904-472e4a26c59e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7 Address:fb9848aa-fe9c-019f-2f64-85cc632d6ac7} {Suffrage:Voter ID:server-c0bab25e-7e19-06a0-2904-472e4a26c59e Address:c0bab25e-7e19-06a0-2904-472e4a26c59e} {Suffrage:Voter ID:server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 Address:d19d1332-f7ab-7cbb-7c64-ddf9777ae649}]" - testing.go:149: server-c0bab25e-7e19-06a0-2904-472e4a26c59e: 2024-06-16T08:26:37.832-1200 [INFO] server-c0bab25e-7e19-06a0-2904-472e4a26c59e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7 Address:fb9848aa-fe9c-019f-2f64-85cc632d6ac7} {Suffrage:Voter ID:server-c0bab25e-7e19-06a0-2904-472e4a26c59e Address:c0bab25e-7e19-06a0-2904-472e4a26c59e} {Suffrage:Voter ID:server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 Address:d19d1332-f7ab-7cbb-7c64-ddf9777ae649}]" -testLoggerAdapter verbose: server-c0bab25e-7e19-06a0-2904-472e4a26c59e: 2024-06-16T08:26:37.832-1200 [INFO] server-c0bab25e-7e19-06a0-2904-472e4a26c59e: entering follower state: follower="Node at c0bab25e-7e19-06a0-2904-472e4a26c59e [Follower]" leader= - testing.go:149: server-c0bab25e-7e19-06a0-2904-472e4a26c59e: 2024-06-16T08:26:37.832-1200 [INFO] server-c0bab25e-7e19-06a0-2904-472e4a26c59e: entering follower state: follower="Node at c0bab25e-7e19-06a0-2904-472e4a26c59e [Follower]" leader= -testLoggerAdapter verbose: server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: 2024-06-16T08:26:37.832-1200 [INFO] server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7 Address:fb9848aa-fe9c-019f-2f64-85cc632d6ac7} {Suffrage:Voter ID:server-c0bab25e-7e19-06a0-2904-472e4a26c59e Address:c0bab25e-7e19-06a0-2904-472e4a26c59e} {Suffrage:Voter ID:server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 Address:d19d1332-f7ab-7cbb-7c64-ddf9777ae649}]" - testing.go:149: server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: 2024-06-16T08:26:37.832-1200 [INFO] server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7 Address:fb9848aa-fe9c-019f-2f64-85cc632d6ac7} {Suffrage:Voter ID:server-c0bab25e-7e19-06a0-2904-472e4a26c59e Address:c0bab25e-7e19-06a0-2904-472e4a26c59e} {Suffrage:Voter ID:server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 Address:d19d1332-f7ab-7cbb-7c64-ddf9777ae649}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:37.832-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:37.832-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: 2024-06-16T08:26:37.832-1200 [INFO] server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: entering follower state: follower="Node at d19d1332-f7ab-7cbb-7c64-ddf9777ae649 [Follower]" leader= - testing.go:149: server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: 2024-06-16T08:26:37.832-1200 [INFO] server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: entering follower state: follower="Node at d19d1332-f7ab-7cbb-7c64-ddf9777ae649 [Follower]" leader= -testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [WARN] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [WARN] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: entering candidate state: node="Node at fb9848aa-fe9c-019f-2f64-85cc632d6ac7 [Candidate]" term=2 - testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: entering candidate state: node="Node at fb9848aa-fe9c-019f-2f64-85cc632d6ac7 [Candidate]" term=2 -testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: election won: tally=2 - testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: election won: tally=2 -testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: entering leader state: leader="Node at fb9848aa-fe9c-019f-2f64-85cc632d6ac7 [Leader]" - testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: entering leader state: leader="Node at fb9848aa-fe9c-019f-2f64-85cc632d6ac7 [Leader]" -testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: added peer, starting replication: peer=server-c0bab25e-7e19-06a0-2904-472e4a26c59e - testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: added peer, starting replication: peer=server-c0bab25e-7e19-06a0-2904-472e4a26c59e -testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: added peer, starting replication: peer=server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 - testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: added peer, starting replication: peer=server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 -testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: pipelining replication: peer="{Voter server-c0bab25e-7e19-06a0-2904-472e4a26c59e c0bab25e-7e19-06a0-2904-472e4a26c59e}" - testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: pipelining replication: peer="{Voter server-c0bab25e-7e19-06a0-2904-472e4a26c59e c0bab25e-7e19-06a0-2904-472e4a26c59e}" -testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: pipelining replication: peer="{Voter server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 d19d1332-f7ab-7cbb-7c64-ddf9777ae649}" - testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:37.913-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: pipelining replication: peer="{Voter server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 d19d1332-f7ab-7cbb-7c64-ddf9777ae649}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.019-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:37.91352908 -1200 -12 m=+14.920845561 (2 nodes), 80.748ms from start of poll, 81.56062ms from cluster start. Timeout at 2024-06-16 08:26:38.01954832 -1200 -12 m=+15.026864781, 106.01922ms after stability - testing.go:149: cluster: 2024-06-16T08:26:38.019-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:37.91352908 -1200 -12 m=+14.920845561 (2 nodes), 80.748ms from start of poll, 81.56062ms from cluster start. Timeout at 2024-06-16 08:26:38.01954832 -1200 -12 m=+15.026864781, 106.01922ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.019-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:38.019-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.124-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.01981172 -1200 -12 m=+15.027128181 (1 nodes), 1.8µs from start of poll, 187.84324ms from cluster start. Timeout at 2024-06-16 08:26:38.12486722 -1200 -12 m=+15.132183721, 105.05554ms after stability - testing.go:149: cluster: 2024-06-16T08:26:38.124-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.01981172 -1200 -12 m=+15.027128181 (1 nodes), 1.8µs from start of poll, 187.84324ms from cluster start. Timeout at 2024-06-16 08:26:38.12486722 -1200 -12 m=+15.132183721, 105.05554ms after stability -testLoggerAdapter verbose: server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: 2024-06-16T08:26:38.125-1200 [WARN] server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: rejecting vote request since we have a leader: from=c0bab25e-7e19-06a0-2904-472e4a26c59e leader=fb9848aa-fe9c-019f-2f64-85cc632d6ac7 - testing.go:149: server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: 2024-06-16T08:26:38.125-1200 [WARN] server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649: rejecting vote request since we have a leader: from=c0bab25e-7e19-06a0-2904-472e4a26c59e leader=fb9848aa-fe9c-019f-2f64-85cc632d6ac7 -testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:38.125-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: aborting pipeline replication: peer="{Voter server-c0bab25e-7e19-06a0-2904-472e4a26c59e c0bab25e-7e19-06a0-2904-472e4a26c59e}" - testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:38.125-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: aborting pipeline replication: peer="{Voter server-c0bab25e-7e19-06a0-2904-472e4a26c59e c0bab25e-7e19-06a0-2904-472e4a26c59e}" -testLoggerAdapter verbose: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:38.125-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: aborting pipeline replication: peer="{Voter server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 d19d1332-f7ab-7cbb-7c64-ddf9777ae649}" - testing.go:149: server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: 2024-06-16T08:26:38.125-1200 [INFO] server-fb9848aa-fe9c-019f-2f64-85cc632d6ac7: aborting pipeline replication: peer="{Voter server-d19d1332-f7ab-7cbb-7c64-ddf9777ae649 d19d1332-f7ab-7cbb-7c64-ddf9777ae649}" ---- PASS: TestRaft_Voting (0.29s) +testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.518+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77f13ea4-1642-0b64-dcd1-eef457780c43 Address:77f13ea4-1642-0b64-dcd1-eef457780c43} {Suffrage:Voter ID:server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a Address:9ebbf41e-b993-2ba7-fd13-b4f512eb226a} {Suffrage:Voter ID:server-97226467-e0a6-5686-538f-707a3f5ac009 Address:97226467-e0a6-5686-538f-707a3f5ac009}]" + testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.518+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77f13ea4-1642-0b64-dcd1-eef457780c43 Address:77f13ea4-1642-0b64-dcd1-eef457780c43} {Suffrage:Voter ID:server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a Address:9ebbf41e-b993-2ba7-fd13-b4f512eb226a} {Suffrage:Voter ID:server-97226467-e0a6-5686-538f-707a3f5ac009 Address:97226467-e0a6-5686-538f-707a3f5ac009}]" +testLoggerAdapter verbose: server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a: 2023-05-16T04:06:18.518+1400 [INFO] server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77f13ea4-1642-0b64-dcd1-eef457780c43 Address:77f13ea4-1642-0b64-dcd1-eef457780c43} {Suffrage:Voter ID:server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a Address:9ebbf41e-b993-2ba7-fd13-b4f512eb226a} {Suffrage:Voter ID:server-97226467-e0a6-5686-538f-707a3f5ac009 Address:97226467-e0a6-5686-538f-707a3f5ac009}]" + testing.go:149: server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a: 2023-05-16T04:06:18.518+1400 [INFO] server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77f13ea4-1642-0b64-dcd1-eef457780c43 Address:77f13ea4-1642-0b64-dcd1-eef457780c43} {Suffrage:Voter ID:server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a Address:9ebbf41e-b993-2ba7-fd13-b4f512eb226a} {Suffrage:Voter ID:server-97226467-e0a6-5686-538f-707a3f5ac009 Address:97226467-e0a6-5686-538f-707a3f5ac009}]" +testLoggerAdapter verbose: server-97226467-e0a6-5686-538f-707a3f5ac009: 2023-05-16T04:06:18.518+1400 [INFO] server-97226467-e0a6-5686-538f-707a3f5ac009: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77f13ea4-1642-0b64-dcd1-eef457780c43 Address:77f13ea4-1642-0b64-dcd1-eef457780c43} {Suffrage:Voter ID:server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a Address:9ebbf41e-b993-2ba7-fd13-b4f512eb226a} {Suffrage:Voter ID:server-97226467-e0a6-5686-538f-707a3f5ac009 Address:97226467-e0a6-5686-538f-707a3f5ac009}]" + testing.go:149: server-97226467-e0a6-5686-538f-707a3f5ac009: 2023-05-16T04:06:18.518+1400 [INFO] server-97226467-e0a6-5686-538f-707a3f5ac009: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77f13ea4-1642-0b64-dcd1-eef457780c43 Address:77f13ea4-1642-0b64-dcd1-eef457780c43} {Suffrage:Voter ID:server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a Address:9ebbf41e-b993-2ba7-fd13-b4f512eb226a} {Suffrage:Voter ID:server-97226467-e0a6-5686-538f-707a3f5ac009 Address:97226467-e0a6-5686-538f-707a3f5ac009}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.518+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:18.518+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.518+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: entering follower state: follower="Node at 77f13ea4-1642-0b64-dcd1-eef457780c43 [Follower]" leader= + testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.518+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: entering follower state: follower="Node at 77f13ea4-1642-0b64-dcd1-eef457780c43 [Follower]" leader= +testLoggerAdapter verbose: server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a: 2023-05-16T04:06:18.518+1400 [INFO] server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a: entering follower state: follower="Node at 9ebbf41e-b993-2ba7-fd13-b4f512eb226a [Follower]" leader= + testing.go:149: server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a: 2023-05-16T04:06:18.518+1400 [INFO] server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a: entering follower state: follower="Node at 9ebbf41e-b993-2ba7-fd13-b4f512eb226a [Follower]" leader= +testLoggerAdapter verbose: server-97226467-e0a6-5686-538f-707a3f5ac009: 2023-05-16T04:06:18.519+1400 [INFO] server-97226467-e0a6-5686-538f-707a3f5ac009: entering follower state: follower="Node at 97226467-e0a6-5686-538f-707a3f5ac009 [Follower]" leader= + testing.go:149: server-97226467-e0a6-5686-538f-707a3f5ac009: 2023-05-16T04:06:18.519+1400 [INFO] server-97226467-e0a6-5686-538f-707a3f5ac009: entering follower state: follower="Node at 97226467-e0a6-5686-538f-707a3f5ac009 [Follower]" leader= +testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [WARN] server-77f13ea4-1642-0b64-dcd1-eef457780c43: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [WARN] server-77f13ea4-1642-0b64-dcd1-eef457780c43: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: entering candidate state: node="Node at 77f13ea4-1642-0b64-dcd1-eef457780c43 [Candidate]" term=2 + testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: entering candidate state: node="Node at 77f13ea4-1642-0b64-dcd1-eef457780c43 [Candidate]" term=2 +testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: election won: tally=2 + testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: election won: tally=2 +testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: entering leader state: leader="Node at 77f13ea4-1642-0b64-dcd1-eef457780c43 [Leader]" + testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: entering leader state: leader="Node at 77f13ea4-1642-0b64-dcd1-eef457780c43 [Leader]" +testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: added peer, starting replication: peer=server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a + testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: added peer, starting replication: peer=server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a +testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: added peer, starting replication: peer=server-97226467-e0a6-5686-538f-707a3f5ac009 + testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: added peer, starting replication: peer=server-97226467-e0a6-5686-538f-707a3f5ac009 +testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: pipelining replication: peer="{Voter server-97226467-e0a6-5686-538f-707a3f5ac009 97226467-e0a6-5686-538f-707a3f5ac009}" + testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.582+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: pipelining replication: peer="{Voter server-97226467-e0a6-5686-538f-707a3f5ac009 97226467-e0a6-5686-538f-707a3f5ac009}" +testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.583+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: pipelining replication: peer="{Voter server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a 9ebbf41e-b993-2ba7-fd13-b4f512eb226a}" + testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.583+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: pipelining replication: peer="{Voter server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a 9ebbf41e-b993-2ba7-fd13-b4f512eb226a}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.688+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:18.583331563 +1400 +14 m=+15.385913178 (2 nodes), 64.546282ms from start of poll, 65.561254ms from cluster start. Timeout at 2023-05-16 04:06:18.688423363 +1400 +14 m=+15.491004998, 105.09182ms after stability + testing.go:149: cluster: 2023-05-16T04:06:18.688+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:18.583331563 +1400 +14 m=+15.385913178 (2 nodes), 64.546282ms from start of poll, 65.561254ms from cluster start. Timeout at 2023-05-16 04:06:18.688423363 +1400 +14 m=+15.491004998, 105.09182ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.688+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:18.688+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.794+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:18.688694086 +1400 +14 m=+15.491275721 (1 nodes), 2.06µs from start of poll, 170.923797ms from cluster start. Timeout at 2023-05-16 04:06:18.79405295 +1400 +14 m=+15.596634605, 105.358884ms after stability + testing.go:149: cluster: 2023-05-16T04:06:18.794+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:18.688694086 +1400 +14 m=+15.491275721 (1 nodes), 2.06µs from start of poll, 170.923797ms from cluster start. Timeout at 2023-05-16 04:06:18.79405295 +1400 +14 m=+15.596634605, 105.358884ms after stability +testLoggerAdapter verbose: server-97226467-e0a6-5686-538f-707a3f5ac009: 2023-05-16T04:06:18.794+1400 [WARN] server-97226467-e0a6-5686-538f-707a3f5ac009: rejecting vote request since we have a leader: from=9ebbf41e-b993-2ba7-fd13-b4f512eb226a leader=77f13ea4-1642-0b64-dcd1-eef457780c43 + testing.go:149: server-97226467-e0a6-5686-538f-707a3f5ac009: 2023-05-16T04:06:18.794+1400 [WARN] server-97226467-e0a6-5686-538f-707a3f5ac009: rejecting vote request since we have a leader: from=9ebbf41e-b993-2ba7-fd13-b4f512eb226a leader=77f13ea4-1642-0b64-dcd1-eef457780c43 +testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.794+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: aborting pipeline replication: peer="{Voter server-97226467-e0a6-5686-538f-707a3f5ac009 97226467-e0a6-5686-538f-707a3f5ac009}" + testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.794+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: aborting pipeline replication: peer="{Voter server-97226467-e0a6-5686-538f-707a3f5ac009 97226467-e0a6-5686-538f-707a3f5ac009}" +testLoggerAdapter verbose: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.794+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: aborting pipeline replication: peer="{Voter server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a 9ebbf41e-b993-2ba7-fd13-b4f512eb226a}" + testing.go:149: server-77f13ea4-1642-0b64-dcd1-eef457780c43: 2023-05-16T04:06:18.794+1400 [INFO] server-77f13ea4-1642-0b64-dcd1-eef457780c43: aborting pipeline replication: peer="{Voter server-9ebbf41e-b993-2ba7-fd13-b4f512eb226a 9ebbf41e-b993-2ba7-fd13-b4f512eb226a}" +--- PASS: TestRaft_Voting (0.28s) === RUN TestRaft_ProtocolVersion_RejectRPC -testLoggerAdapter verbose: server-142a0958-dc90-dcc8-0452-42518ebfa8cc: 2024-06-16T08:26:38.126-1200 [INFO] server-142a0958-dc90-dcc8-0452-42518ebfa8cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-142a0958-dc90-dcc8-0452-42518ebfa8cc Address:142a0958-dc90-dcc8-0452-42518ebfa8cc} {Suffrage:Voter ID:server-f80f0a21-23cf-101c-34bb-29cc543c36df Address:f80f0a21-23cf-101c-34bb-29cc543c36df} {Suffrage:Voter ID:server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 Address:4000368a-7fef-2bd9-a4f7-3cbc7723cec3}]" - testing.go:149: server-142a0958-dc90-dcc8-0452-42518ebfa8cc: 2024-06-16T08:26:38.126-1200 [INFO] server-142a0958-dc90-dcc8-0452-42518ebfa8cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-142a0958-dc90-dcc8-0452-42518ebfa8cc Address:142a0958-dc90-dcc8-0452-42518ebfa8cc} {Suffrage:Voter ID:server-f80f0a21-23cf-101c-34bb-29cc543c36df Address:f80f0a21-23cf-101c-34bb-29cc543c36df} {Suffrage:Voter ID:server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 Address:4000368a-7fef-2bd9-a4f7-3cbc7723cec3}]" -testLoggerAdapter verbose: server-142a0958-dc90-dcc8-0452-42518ebfa8cc: 2024-06-16T08:26:38.127-1200 [INFO] server-142a0958-dc90-dcc8-0452-42518ebfa8cc: entering follower state: follower="Node at 142a0958-dc90-dcc8-0452-42518ebfa8cc [Follower]" leader= -testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.127-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-142a0958-dc90-dcc8-0452-42518ebfa8cc Address:142a0958-dc90-dcc8-0452-42518ebfa8cc} {Suffrage:Voter ID:server-f80f0a21-23cf-101c-34bb-29cc543c36df Address:f80f0a21-23cf-101c-34bb-29cc543c36df} {Suffrage:Voter ID:server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 Address:4000368a-7fef-2bd9-a4f7-3cbc7723cec3}]" - testing.go:149: server-142a0958-dc90-dcc8-0452-42518ebfa8cc: 2024-06-16T08:26:38.127-1200 [INFO] server-142a0958-dc90-dcc8-0452-42518ebfa8cc: entering follower state: follower="Node at 142a0958-dc90-dcc8-0452-42518ebfa8cc [Follower]" leader= - testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.127-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-142a0958-dc90-dcc8-0452-42518ebfa8cc Address:142a0958-dc90-dcc8-0452-42518ebfa8cc} {Suffrage:Voter ID:server-f80f0a21-23cf-101c-34bb-29cc543c36df Address:f80f0a21-23cf-101c-34bb-29cc543c36df} {Suffrage:Voter ID:server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 Address:4000368a-7fef-2bd9-a4f7-3cbc7723cec3}]" -testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.127-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: entering follower state: follower="Node at f80f0a21-23cf-101c-34bb-29cc543c36df [Follower]" leader= - testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.127-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: entering follower state: follower="Node at f80f0a21-23cf-101c-34bb-29cc543c36df [Follower]" leader= -testLoggerAdapter verbose: server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3: 2024-06-16T08:26:38.127-1200 [INFO] server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-142a0958-dc90-dcc8-0452-42518ebfa8cc Address:142a0958-dc90-dcc8-0452-42518ebfa8cc} {Suffrage:Voter ID:server-f80f0a21-23cf-101c-34bb-29cc543c36df Address:f80f0a21-23cf-101c-34bb-29cc543c36df} {Suffrage:Voter ID:server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 Address:4000368a-7fef-2bd9-a4f7-3cbc7723cec3}]" - testing.go:149: server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3: 2024-06-16T08:26:38.127-1200 [INFO] server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-142a0958-dc90-dcc8-0452-42518ebfa8cc Address:142a0958-dc90-dcc8-0452-42518ebfa8cc} {Suffrage:Voter ID:server-f80f0a21-23cf-101c-34bb-29cc543c36df Address:f80f0a21-23cf-101c-34bb-29cc543c36df} {Suffrage:Voter ID:server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 Address:4000368a-7fef-2bd9-a4f7-3cbc7723cec3}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.127-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:38.127-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3: 2024-06-16T08:26:38.127-1200 [INFO] server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3: entering follower state: follower="Node at 4000368a-7fef-2bd9-a4f7-3cbc7723cec3 [Follower]" leader= - testing.go:149: server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3: 2024-06-16T08:26:38.127-1200 [INFO] server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3: entering follower state: follower="Node at 4000368a-7fef-2bd9-a4f7-3cbc7723cec3 [Follower]" leader= -testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.181-1200 [WARN] server-f80f0a21-23cf-101c-34bb-29cc543c36df: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.181-1200 [WARN] server-f80f0a21-23cf-101c-34bb-29cc543c36df: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: entering candidate state: node="Node at f80f0a21-23cf-101c-34bb-29cc543c36df [Candidate]" term=2 - testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: entering candidate state: node="Node at f80f0a21-23cf-101c-34bb-29cc543c36df [Candidate]" term=2 -testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: election won: tally=2 - testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: election won: tally=2 -testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: entering leader state: leader="Node at f80f0a21-23cf-101c-34bb-29cc543c36df [Leader]" - testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: entering leader state: leader="Node at f80f0a21-23cf-101c-34bb-29cc543c36df [Leader]" -testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: added peer, starting replication: peer=server-142a0958-dc90-dcc8-0452-42518ebfa8cc - testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: added peer, starting replication: peer=server-142a0958-dc90-dcc8-0452-42518ebfa8cc -testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: added peer, starting replication: peer=server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 - testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: added peer, starting replication: peer=server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 -testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: pipelining replication: peer="{Voter server-142a0958-dc90-dcc8-0452-42518ebfa8cc 142a0958-dc90-dcc8-0452-42518ebfa8cc}" - testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: pipelining replication: peer="{Voter server-142a0958-dc90-dcc8-0452-42518ebfa8cc 142a0958-dc90-dcc8-0452-42518ebfa8cc}" -testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: pipelining replication: peer="{Voter server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 4000368a-7fef-2bd9-a4f7-3cbc7723cec3}" - testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.182-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: pipelining replication: peer="{Voter server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 4000368a-7fef-2bd9-a4f7-3cbc7723cec3}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.288-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:38.1823694 -1200 -12 m=+15.189685881 (2 nodes), 54.74634ms from start of poll, 55.74502ms from cluster start. Timeout at 2024-06-16 08:26:38.28809394 -1200 -12 m=+15.295410421, 105.72454ms after stability - testing.go:149: cluster: 2024-06-16T08:26:38.288-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:38.1823694 -1200 -12 m=+15.189685881 (2 nodes), 54.74634ms from start of poll, 55.74502ms from cluster start. Timeout at 2024-06-16 08:26:38.28809394 -1200 -12 m=+15.295410421, 105.72454ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.288-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:38.288-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.393-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.2882826 -1200 -12 m=+15.295599061 (1 nodes), 1.68µs from start of poll, 161.6582ms from cluster start. Timeout at 2024-06-16 08:26:38.39366024 -1200 -12 m=+15.400976721, 105.37766ms after stability - testing.go:149: cluster: 2024-06-16T08:26:38.393-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.2882826 -1200 -12 m=+15.295599061 (1 nodes), 1.68µs from start of poll, 161.6582ms from cluster start. Timeout at 2024-06-16 08:26:38.39366024 -1200 -12 m=+15.400976721, 105.37766ms after stability -testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.393-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: aborting pipeline replication: peer="{Voter server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 4000368a-7fef-2bd9-a4f7-3cbc7723cec3}" - testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.393-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: aborting pipeline replication: peer="{Voter server-4000368a-7fef-2bd9-a4f7-3cbc7723cec3 4000368a-7fef-2bd9-a4f7-3cbc7723cec3}" -testLoggerAdapter verbose: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.393-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: aborting pipeline replication: peer="{Voter server-142a0958-dc90-dcc8-0452-42518ebfa8cc 142a0958-dc90-dcc8-0452-42518ebfa8cc}" - testing.go:149: server-f80f0a21-23cf-101c-34bb-29cc543c36df: 2024-06-16T08:26:38.393-1200 [INFO] server-f80f0a21-23cf-101c-34bb-29cc543c36df: aborting pipeline replication: peer="{Voter server-142a0958-dc90-dcc8-0452-42518ebfa8cc 142a0958-dc90-dcc8-0452-42518ebfa8cc}" +testLoggerAdapter verbose: server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d: 2023-05-16T04:06:18.796+1400 [INFO] server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d Address:030f13fa-8e96-6dd9-facf-6ac36a7aac3d} {Suffrage:Voter ID:server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 Address:3a357a48-8a72-f73f-468d-5ba6197fa8a3} {Suffrage:Voter ID:server-be946513-49ca-a889-e985-34e5143e9042 Address:be946513-49ca-a889-e985-34e5143e9042}]" + testing.go:149: server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d: 2023-05-16T04:06:18.796+1400 [INFO] server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d Address:030f13fa-8e96-6dd9-facf-6ac36a7aac3d} {Suffrage:Voter ID:server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 Address:3a357a48-8a72-f73f-468d-5ba6197fa8a3} {Suffrage:Voter ID:server-be946513-49ca-a889-e985-34e5143e9042 Address:be946513-49ca-a889-e985-34e5143e9042}]" +testLoggerAdapter verbose: server-3a357a48-8a72-f73f-468d-5ba6197fa8a3: 2023-05-16T04:06:18.796+1400 [INFO] server-3a357a48-8a72-f73f-468d-5ba6197fa8a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d Address:030f13fa-8e96-6dd9-facf-6ac36a7aac3d} {Suffrage:Voter ID:server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 Address:3a357a48-8a72-f73f-468d-5ba6197fa8a3} {Suffrage:Voter ID:server-be946513-49ca-a889-e985-34e5143e9042 Address:be946513-49ca-a889-e985-34e5143e9042}]" + testing.go:149: server-3a357a48-8a72-f73f-468d-5ba6197fa8a3: 2023-05-16T04:06:18.796+1400 [INFO] server-3a357a48-8a72-f73f-468d-5ba6197fa8a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d Address:030f13fa-8e96-6dd9-facf-6ac36a7aac3d} {Suffrage:Voter ID:server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 Address:3a357a48-8a72-f73f-468d-5ba6197fa8a3} {Suffrage:Voter ID:server-be946513-49ca-a889-e985-34e5143e9042 Address:be946513-49ca-a889-e985-34e5143e9042}]" +testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.796+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d Address:030f13fa-8e96-6dd9-facf-6ac36a7aac3d} {Suffrage:Voter ID:server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 Address:3a357a48-8a72-f73f-468d-5ba6197fa8a3} {Suffrage:Voter ID:server-be946513-49ca-a889-e985-34e5143e9042 Address:be946513-49ca-a889-e985-34e5143e9042}]" + testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.796+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d Address:030f13fa-8e96-6dd9-facf-6ac36a7aac3d} {Suffrage:Voter ID:server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 Address:3a357a48-8a72-f73f-468d-5ba6197fa8a3} {Suffrage:Voter ID:server-be946513-49ca-a889-e985-34e5143e9042 Address:be946513-49ca-a889-e985-34e5143e9042}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.796+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:18.796+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d: 2023-05-16T04:06:18.796+1400 [INFO] server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d: entering follower state: follower="Node at 030f13fa-8e96-6dd9-facf-6ac36a7aac3d [Follower]" leader= + testing.go:149: server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d: 2023-05-16T04:06:18.796+1400 [INFO] server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d: entering follower state: follower="Node at 030f13fa-8e96-6dd9-facf-6ac36a7aac3d [Follower]" leader= +testLoggerAdapter verbose: server-3a357a48-8a72-f73f-468d-5ba6197fa8a3: 2023-05-16T04:06:18.796+1400 [INFO] server-3a357a48-8a72-f73f-468d-5ba6197fa8a3: entering follower state: follower="Node at 3a357a48-8a72-f73f-468d-5ba6197fa8a3 [Follower]" leader= + testing.go:149: server-3a357a48-8a72-f73f-468d-5ba6197fa8a3: 2023-05-16T04:06:18.796+1400 [INFO] server-3a357a48-8a72-f73f-468d-5ba6197fa8a3: entering follower state: follower="Node at 3a357a48-8a72-f73f-468d-5ba6197fa8a3 [Follower]" leader= +testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.796+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: entering follower state: follower="Node at be946513-49ca-a889-e985-34e5143e9042 [Follower]" leader= + testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.796+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: entering follower state: follower="Node at be946513-49ca-a889-e985-34e5143e9042 [Follower]" leader= +testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.850+1400 [WARN] server-be946513-49ca-a889-e985-34e5143e9042: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.850+1400 [WARN] server-be946513-49ca-a889-e985-34e5143e9042: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.850+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: entering candidate state: node="Node at be946513-49ca-a889-e985-34e5143e9042 [Candidate]" term=2 + testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.850+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: entering candidate state: node="Node at be946513-49ca-a889-e985-34e5143e9042 [Candidate]" term=2 +testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.850+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: election won: tally=2 + testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.850+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: election won: tally=2 +testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: entering leader state: leader="Node at be946513-49ca-a889-e985-34e5143e9042 [Leader]" + testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: entering leader state: leader="Node at be946513-49ca-a889-e985-34e5143e9042 [Leader]" +testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: added peer, starting replication: peer=server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d + testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: added peer, starting replication: peer=server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d +testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: added peer, starting replication: peer=server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 + testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: added peer, starting replication: peer=server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 +testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: pipelining replication: peer="{Voter server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 3a357a48-8a72-f73f-468d-5ba6197fa8a3}" + testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: pipelining replication: peer="{Voter server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 3a357a48-8a72-f73f-468d-5ba6197fa8a3}" +testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: pipelining replication: peer="{Voter server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d 030f13fa-8e96-6dd9-facf-6ac36a7aac3d}" + testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:18.851+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: pipelining replication: peer="{Voter server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d 030f13fa-8e96-6dd9-facf-6ac36a7aac3d}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.957+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:18.851592209 +1400 +14 m=+15.654173844 (2 nodes), 54.812587ms from start of poll, 55.718078ms from cluster start. Timeout at 2023-05-16 04:06:18.957174035 +1400 +14 m=+15.759755670, 105.581826ms after stability + testing.go:149: cluster: 2023-05-16T04:06:18.957+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:18.851592209 +1400 +14 m=+15.654173844 (2 nodes), 54.812587ms from start of poll, 55.718078ms from cluster start. Timeout at 2023-05-16 04:06:18.957174035 +1400 +14 m=+15.759755670, 105.581826ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:18.957+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:18.957+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.062+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:18.957402257 +1400 +14 m=+15.759983892 (1 nodes), 1.96µs from start of poll, 161.528126ms from cluster start. Timeout at 2023-05-16 04:06:19.06268426 +1400 +14 m=+15.865265895, 105.282003ms after stability + testing.go:149: cluster: 2023-05-16T04:06:19.062+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:18.957402257 +1400 +14 m=+15.759983892 (1 nodes), 1.96µs from start of poll, 161.528126ms from cluster start. Timeout at 2023-05-16 04:06:19.06268426 +1400 +14 m=+15.865265895, 105.282003ms after stability +testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:19.063+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: aborting pipeline replication: peer="{Voter server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 3a357a48-8a72-f73f-468d-5ba6197fa8a3}" + testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:19.063+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: aborting pipeline replication: peer="{Voter server-3a357a48-8a72-f73f-468d-5ba6197fa8a3 3a357a48-8a72-f73f-468d-5ba6197fa8a3}" +testLoggerAdapter verbose: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:19.063+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: aborting pipeline replication: peer="{Voter server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d 030f13fa-8e96-6dd9-facf-6ac36a7aac3d}" + testing.go:149: server-be946513-49ca-a889-e985-34e5143e9042: 2023-05-16T04:06:19.063+1400 [INFO] server-be946513-49ca-a889-e985-34e5143e9042: aborting pipeline replication: peer="{Voter server-030f13fa-8e96-6dd9-facf-6ac36a7aac3d 030f13fa-8e96-6dd9-facf-6ac36a7aac3d}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 -testLoggerAdapter verbose: d4477fa5-ec15-842a-b091-51c27397c59c: 2024-06-16T08:26:38.395-1200 [INFO] d4477fa5-ec15-842a-b091-51c27397c59c: initial configuration: index=1 servers="[{Suffrage:Voter ID:d4477fa5-ec15-842a-b091-51c27397c59c Address:d4477fa5-ec15-842a-b091-51c27397c59c} {Suffrage:Voter ID:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 Address:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1}]" - testing.go:149: d4477fa5-ec15-842a-b091-51c27397c59c: 2024-06-16T08:26:38.395-1200 [INFO] d4477fa5-ec15-842a-b091-51c27397c59c: initial configuration: index=1 servers="[{Suffrage:Voter ID:d4477fa5-ec15-842a-b091-51c27397c59c Address:d4477fa5-ec15-842a-b091-51c27397c59c} {Suffrage:Voter ID:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 Address:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1}]" -testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.395-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: initial configuration: index=1 servers="[{Suffrage:Voter ID:d4477fa5-ec15-842a-b091-51c27397c59c Address:d4477fa5-ec15-842a-b091-51c27397c59c} {Suffrage:Voter ID:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 Address:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1}]" - testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.395-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: initial configuration: index=1 servers="[{Suffrage:Voter ID:d4477fa5-ec15-842a-b091-51c27397c59c Address:d4477fa5-ec15-842a-b091-51c27397c59c} {Suffrage:Voter ID:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 Address:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1}]" -testLoggerAdapter verbose: d4477fa5-ec15-842a-b091-51c27397c59c: 2024-06-16T08:26:38.395-1200 [INFO] d4477fa5-ec15-842a-b091-51c27397c59c: entering follower state: follower="Node at d4477fa5-ec15-842a-b091-51c27397c59c [Follower]" leader= -testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.395-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: entering follower state: follower="Node at e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 [Follower]" leader= - testing.go:149: d4477fa5-ec15-842a-b091-51c27397c59c: 2024-06-16T08:26:38.395-1200 [INFO] d4477fa5-ec15-842a-b091-51c27397c59c: entering follower state: follower="Node at d4477fa5-ec15-842a-b091-51c27397c59c [Follower]" leader= - testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.395-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: entering follower state: follower="Node at e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 [Follower]" leader= -testLoggerAdapter verbose: b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: 2024-06-16T08:26:38.395-1200 [INFO] b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: initial configuration: index=0 servers=[] - testing.go:149: b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: 2024-06-16T08:26:38.395-1200 [INFO] b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.395-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:38.395-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: 2024-06-16T08:26:38.395-1200 [INFO] b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: entering follower state: follower="Node at b0359b58-f66c-2b3d-f4ac-c9646ef95e9d [Follower]" leader= - testing.go:149: b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: 2024-06-16T08:26:38.395-1200 [INFO] b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: entering follower state: follower="Node at b0359b58-f66c-2b3d-f4ac-c9646ef95e9d [Follower]" leader= -testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [WARN] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: heartbeat timeout reached, starting election: last-leader= - testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [WARN] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: entering candidate state: node="Node at e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 [Candidate]" term=2 - testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: entering candidate state: node="Node at e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 [Candidate]" term=2 -testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: election won: tally=2 - testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: election won: tally=2 -testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: entering leader state: leader="Node at e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 [Leader]" - testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: entering leader state: leader="Node at e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 [Leader]" -testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: added peer, starting replication: peer=d4477fa5-ec15-842a-b091-51c27397c59c - testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: added peer, starting replication: peer=d4477fa5-ec15-842a-b091-51c27397c59c -testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: pipelining replication: peer="{Voter d4477fa5-ec15-842a-b091-51c27397c59c d4477fa5-ec15-842a-b091-51c27397c59c}" - testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.447-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: pipelining replication: peer="{Voter d4477fa5-ec15-842a-b091-51c27397c59c d4477fa5-ec15-842a-b091-51c27397c59c}" -testLoggerAdapter verbose: b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: 2024-06-16T08:26:38.450-1200 [WARN] b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: no known peers, aborting election - testing.go:149: b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: 2024-06-16T08:26:38.450-1200 [WARN] b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: no known peers, aborting election -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.552-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.44743852 -1200 -12 m=+15.454755001 (1 nodes), 51.62112ms from start of poll, 52.45526ms from cluster start. Timeout at 2024-06-16 08:26:38.552614 -1200 -12 m=+15.559930481, 105.17548ms after stability - testing.go:149: cluster: 2024-06-16T08:26:38.552-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.44743852 -1200 -12 m=+15.454755001 (1 nodes), 51.62112ms from start of poll, 52.45526ms from cluster start. Timeout at 2024-06-16 08:26:38.552614 -1200 -12 m=+15.559930481, 105.17548ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.552-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:38.552-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.658-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.55281652 -1200 -12 m=+15.560132981 (1 nodes), 1.72µs from start of poll, 157.83324ms from cluster start. Timeout at 2024-06-16 08:26:38.6579564 -1200 -12 m=+15.665272881, 105.1399ms after stability - testing.go:149: cluster: 2024-06-16T08:26:38.658-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.55281652 -1200 -12 m=+15.560132981 (1 nodes), 1.72µs from start of poll, 157.83324ms from cluster start. Timeout at 2024-06-16 08:26:38.6579564 -1200 -12 m=+15.665272881, 105.1399ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.658-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:38.658-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.763-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.6581582 -1200 -12 m=+15.665474681 (1 nodes), 2.1µs from start of poll, 263.17494ms from cluster start. Timeout at 2024-06-16 08:26:38.76317236 -1200 -12 m=+15.770488841, 105.01416ms after stability - testing.go:149: cluster: 2024-06-16T08:26:38.763-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.6581582 -1200 -12 m=+15.665474681 (1 nodes), 2.1µs from start of poll, 263.17494ms from cluster start. Timeout at 2024-06-16 08:26:38.76317236 -1200 -12 m=+15.770488841, 105.01416ms after stability -testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.763-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: updating configuration: command=AddStaging server-id=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d server-addr=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d servers="[{Suffrage:Voter ID:d4477fa5-ec15-842a-b091-51c27397c59c Address:d4477fa5-ec15-842a-b091-51c27397c59c} {Suffrage:Voter ID:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 Address:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1} {Suffrage:Voter ID:b0359b58-f66c-2b3d-f4ac-c9646ef95e9d Address:b0359b58-f66c-2b3d-f4ac-c9646ef95e9d}]" - testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.763-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: updating configuration: command=AddStaging server-id=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d server-addr=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d servers="[{Suffrage:Voter ID:d4477fa5-ec15-842a-b091-51c27397c59c Address:d4477fa5-ec15-842a-b091-51c27397c59c} {Suffrage:Voter ID:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 Address:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1} {Suffrage:Voter ID:b0359b58-f66c-2b3d-f4ac-c9646ef95e9d Address:b0359b58-f66c-2b3d-f4ac-c9646ef95e9d}]" -testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.763-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: added peer, starting replication: peer=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d - testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.763-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: added peer, starting replication: peer=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d -testLoggerAdapter verbose: b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: 2024-06-16T08:26:38.763-1200 [WARN] b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: 2024-06-16T08:26:38.763-1200 [WARN] b0359b58-f66c-2b3d-f4ac-c9646ef95e9d: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.763-1200 [WARN] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: appendEntries rejected, sending older logs: peer="{Voter b0359b58-f66c-2b3d-f4ac-c9646ef95e9d b0359b58-f66c-2b3d-f4ac-c9646ef95e9d}" next=1 - testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.763-1200 [WARN] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: appendEntries rejected, sending older logs: peer="{Voter b0359b58-f66c-2b3d-f4ac-c9646ef95e9d b0359b58-f66c-2b3d-f4ac-c9646ef95e9d}" next=1 -testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.763-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: pipelining replication: peer="{Voter b0359b58-f66c-2b3d-f4ac-c9646ef95e9d b0359b58-f66c-2b3d-f4ac-c9646ef95e9d}" - testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.763-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: pipelining replication: peer="{Voter b0359b58-f66c-2b3d-f4ac-c9646ef95e9d b0359b58-f66c-2b3d-f4ac-c9646ef95e9d}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.769-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:38.769-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.874-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.76967666 -1200 -12 m=+15.776993121 (1 nodes), 1.48µs from start of poll, 374.69338ms from cluster start. Timeout at 2024-06-16 08:26:38.87471556 -1200 -12 m=+15.882032061, 105.03894ms after stability - testing.go:149: cluster: 2024-06-16T08:26:38.874-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.76967666 -1200 -12 m=+15.776993121 (1 nodes), 1.48µs from start of poll, 374.69338ms from cluster start. Timeout at 2024-06-16 08:26:38.87471556 -1200 -12 m=+15.882032061, 105.03894ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.874-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:38.874-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.980-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.87489558 -1200 -12 m=+15.882212061 (1 nodes), 1.76µs from start of poll, 479.91232ms from cluster start. Timeout at 2024-06-16 08:26:38.9805888 -1200 -12 m=+15.987905261, 105.6932ms after stability - testing.go:149: cluster: 2024-06-16T08:26:38.980-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:38.87489558 -1200 -12 m=+15.882212061 (1 nodes), 1.76µs from start of poll, 479.91232ms from cluster start. Timeout at 2024-06-16 08:26:38.9805888 -1200 -12 m=+15.987905261, 105.6932ms after stability -testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.980-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: updating configuration: command=RemoveServer server-id=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d server-addr= servers="[{Suffrage:Voter ID:d4477fa5-ec15-842a-b091-51c27397c59c Address:d4477fa5-ec15-842a-b091-51c27397c59c} {Suffrage:Voter ID:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 Address:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1}]" - testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.980-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: updating configuration: command=RemoveServer server-id=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d server-addr= servers="[{Suffrage:Voter ID:d4477fa5-ec15-842a-b091-51c27397c59c Address:d4477fa5-ec15-842a-b091-51c27397c59c} {Suffrage:Voter ID:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1 Address:e0879712-4fb3-f2cb-9e6c-774abe4d5cb1}]" -testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.980-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: removed peer, stopping replication: peer=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d last-index=4 - testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.980-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: removed peer, stopping replication: peer=b0359b58-f66c-2b3d-f4ac-c9646ef95e9d last-index=4 -testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.981-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: aborting pipeline replication: peer="{Voter b0359b58-f66c-2b3d-f4ac-c9646ef95e9d b0359b58-f66c-2b3d-f4ac-c9646ef95e9d}" - testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.981-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: aborting pipeline replication: peer="{Voter b0359b58-f66c-2b3d-f4ac-c9646ef95e9d b0359b58-f66c-2b3d-f4ac-c9646ef95e9d}" -testLoggerAdapter verbose: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.981-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: aborting pipeline replication: peer="{Voter d4477fa5-ec15-842a-b091-51c27397c59c d4477fa5-ec15-842a-b091-51c27397c59c}" - testing.go:149: e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: 2024-06-16T08:26:38.981-1200 [INFO] e0879712-4fb3-f2cb-9e6c-774abe4d5cb1: aborting pipeline replication: peer="{Voter d4477fa5-ec15-842a-b091-51c27397c59c d4477fa5-ec15-842a-b091-51c27397c59c}" ---- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.59s) +testLoggerAdapter verbose: 9e1a7b19-2472-077e-8876-67055c18b3a4: 2023-05-16T04:06:19.064+1400 [INFO] 9e1a7b19-2472-077e-8876-67055c18b3a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:9e1a7b19-2472-077e-8876-67055c18b3a4 Address:9e1a7b19-2472-077e-8876-67055c18b3a4} {Suffrage:Voter ID:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 Address:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2}]" + testing.go:149: 9e1a7b19-2472-077e-8876-67055c18b3a4: 2023-05-16T04:06:19.064+1400 [INFO] 9e1a7b19-2472-077e-8876-67055c18b3a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:9e1a7b19-2472-077e-8876-67055c18b3a4 Address:9e1a7b19-2472-077e-8876-67055c18b3a4} {Suffrage:Voter ID:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 Address:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2}]" +testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.065+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:9e1a7b19-2472-077e-8876-67055c18b3a4 Address:9e1a7b19-2472-077e-8876-67055c18b3a4} {Suffrage:Voter ID:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 Address:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2}]" + testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.065+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:9e1a7b19-2472-077e-8876-67055c18b3a4 Address:9e1a7b19-2472-077e-8876-67055c18b3a4} {Suffrage:Voter ID:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 Address:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2}]" +testLoggerAdapter verbose: c6eed6c2-5993-16be-5926-f5d4aba279d8: 2023-05-16T04:06:19.065+1400 [INFO] c6eed6c2-5993-16be-5926-f5d4aba279d8: initial configuration: index=0 servers=[] + testing.go:149: c6eed6c2-5993-16be-5926-f5d4aba279d8: 2023-05-16T04:06:19.065+1400 [INFO] c6eed6c2-5993-16be-5926-f5d4aba279d8: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.065+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:19.065+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: 9e1a7b19-2472-077e-8876-67055c18b3a4: 2023-05-16T04:06:19.065+1400 [INFO] 9e1a7b19-2472-077e-8876-67055c18b3a4: entering follower state: follower="Node at 9e1a7b19-2472-077e-8876-67055c18b3a4 [Follower]" leader= + testing.go:149: 9e1a7b19-2472-077e-8876-67055c18b3a4: 2023-05-16T04:06:19.065+1400 [INFO] 9e1a7b19-2472-077e-8876-67055c18b3a4: entering follower state: follower="Node at 9e1a7b19-2472-077e-8876-67055c18b3a4 [Follower]" leader= +testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.065+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: entering follower state: follower="Node at 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 [Follower]" leader= + testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.065+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: entering follower state: follower="Node at 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 [Follower]" leader= +testLoggerAdapter verbose: c6eed6c2-5993-16be-5926-f5d4aba279d8: 2023-05-16T04:06:19.065+1400 [INFO] c6eed6c2-5993-16be-5926-f5d4aba279d8: entering follower state: follower="Node at c6eed6c2-5993-16be-5926-f5d4aba279d8 [Follower]" leader= + testing.go:149: c6eed6c2-5993-16be-5926-f5d4aba279d8: 2023-05-16T04:06:19.065+1400 [INFO] c6eed6c2-5993-16be-5926-f5d4aba279d8: entering follower state: follower="Node at c6eed6c2-5993-16be-5926-f5d4aba279d8 [Follower]" leader= +testLoggerAdapter verbose: c6eed6c2-5993-16be-5926-f5d4aba279d8: 2023-05-16T04:06:19.119+1400 [WARN] c6eed6c2-5993-16be-5926-f5d4aba279d8: no known peers, aborting election + testing.go:149: c6eed6c2-5993-16be-5926-f5d4aba279d8: 2023-05-16T04:06:19.119+1400 [WARN] c6eed6c2-5993-16be-5926-f5d4aba279d8: no known peers, aborting election +testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.139+1400 [WARN] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: heartbeat timeout reached, starting election: last-leader= + testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.139+1400 [WARN] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.139+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: entering candidate state: node="Node at 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 [Candidate]" term=2 + testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.139+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: entering candidate state: node="Node at 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 [Candidate]" term=2 +testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.140+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: election won: tally=2 + testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.140+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: election won: tally=2 +testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.140+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: entering leader state: leader="Node at 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 [Leader]" + testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.140+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: entering leader state: leader="Node at 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 [Leader]" +testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.140+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: added peer, starting replication: peer=9e1a7b19-2472-077e-8876-67055c18b3a4 + testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.140+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: added peer, starting replication: peer=9e1a7b19-2472-077e-8876-67055c18b3a4 +testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.140+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: pipelining replication: peer="{Voter 9e1a7b19-2472-077e-8876-67055c18b3a4 9e1a7b19-2472-077e-8876-67055c18b3a4}" + testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.140+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: pipelining replication: peer="{Voter 9e1a7b19-2472-077e-8876-67055c18b3a4 9e1a7b19-2472-077e-8876-67055c18b3a4}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.246+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.140613419 +1400 +14 m=+15.943195034 (1 nodes), 75.052085ms from start of poll, 76.090098ms from cluster start. Timeout at 2023-05-16 04:06:19.24658299 +1400 +14 m=+16.049164605, 105.969571ms after stability + testing.go:149: cluster: 2023-05-16T04:06:19.246+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.140613419 +1400 +14 m=+15.943195034 (1 nodes), 75.052085ms from start of poll, 76.090098ms from cluster start. Timeout at 2023-05-16 04:06:19.24658299 +1400 +14 m=+16.049164605, 105.969571ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.246+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:19.246+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.352+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.246853153 +1400 +14 m=+16.049434768 (1 nodes), 2.14µs from start of poll, 182.329832ms from cluster start. Timeout at 2023-05-16 04:06:19.352708602 +1400 +14 m=+16.155290237, 105.855469ms after stability + testing.go:149: cluster: 2023-05-16T04:06:19.352+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.246853153 +1400 +14 m=+16.049434768 (1 nodes), 2.14µs from start of poll, 182.329832ms from cluster start. Timeout at 2023-05-16 04:06:19.352708602 +1400 +14 m=+16.155290237, 105.855469ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.352+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:19.352+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.459+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.352971085 +1400 +14 m=+16.155552720 (1 nodes), 1.84µs from start of poll, 288.447784ms from cluster start. Timeout at 2023-05-16 04:06:19.458985397 +1400 +14 m=+16.261567052, 106.014332ms after stability + testing.go:149: cluster: 2023-05-16T04:06:19.459+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.352971085 +1400 +14 m=+16.155552720 (1 nodes), 1.84µs from start of poll, 288.447784ms from cluster start. Timeout at 2023-05-16 04:06:19.458985397 +1400 +14 m=+16.261567052, 106.014332ms after stability +testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.459+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: updating configuration: command=AddStaging server-id=c6eed6c2-5993-16be-5926-f5d4aba279d8 server-addr=c6eed6c2-5993-16be-5926-f5d4aba279d8 servers="[{Suffrage:Voter ID:9e1a7b19-2472-077e-8876-67055c18b3a4 Address:9e1a7b19-2472-077e-8876-67055c18b3a4} {Suffrage:Voter ID:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 Address:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2} {Suffrage:Voter ID:c6eed6c2-5993-16be-5926-f5d4aba279d8 Address:c6eed6c2-5993-16be-5926-f5d4aba279d8}]" + testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.459+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: updating configuration: command=AddStaging server-id=c6eed6c2-5993-16be-5926-f5d4aba279d8 server-addr=c6eed6c2-5993-16be-5926-f5d4aba279d8 servers="[{Suffrage:Voter ID:9e1a7b19-2472-077e-8876-67055c18b3a4 Address:9e1a7b19-2472-077e-8876-67055c18b3a4} {Suffrage:Voter ID:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 Address:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2} {Suffrage:Voter ID:c6eed6c2-5993-16be-5926-f5d4aba279d8 Address:c6eed6c2-5993-16be-5926-f5d4aba279d8}]" +testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.459+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: added peer, starting replication: peer=c6eed6c2-5993-16be-5926-f5d4aba279d8 + testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.459+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: added peer, starting replication: peer=c6eed6c2-5993-16be-5926-f5d4aba279d8 +testLoggerAdapter verbose: c6eed6c2-5993-16be-5926-f5d4aba279d8: 2023-05-16T04:06:19.459+1400 [WARN] c6eed6c2-5993-16be-5926-f5d4aba279d8: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: c6eed6c2-5993-16be-5926-f5d4aba279d8: 2023-05-16T04:06:19.459+1400 [WARN] c6eed6c2-5993-16be-5926-f5d4aba279d8: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.459+1400 [WARN] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: appendEntries rejected, sending older logs: peer="{Voter c6eed6c2-5993-16be-5926-f5d4aba279d8 c6eed6c2-5993-16be-5926-f5d4aba279d8}" next=1 + testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.459+1400 [WARN] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: appendEntries rejected, sending older logs: peer="{Voter c6eed6c2-5993-16be-5926-f5d4aba279d8 c6eed6c2-5993-16be-5926-f5d4aba279d8}" next=1 +testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.459+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: pipelining replication: peer="{Voter c6eed6c2-5993-16be-5926-f5d4aba279d8 c6eed6c2-5993-16be-5926-f5d4aba279d8}" + testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.459+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: pipelining replication: peer="{Voter c6eed6c2-5993-16be-5926-f5d4aba279d8 c6eed6c2-5993-16be-5926-f5d4aba279d8}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.460+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:19.460+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.565+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.460254932 +1400 +14 m=+16.262836567 (1 nodes), 1.36µs from start of poll, 395.731631ms from cluster start. Timeout at 2023-05-16 04:06:19.565259331 +1400 +14 m=+16.367840946, 105.004379ms after stability + testing.go:149: cluster: 2023-05-16T04:06:19.565+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.460254932 +1400 +14 m=+16.262836567 (1 nodes), 1.36µs from start of poll, 395.731631ms from cluster start. Timeout at 2023-05-16 04:06:19.565259331 +1400 +14 m=+16.367840946, 105.004379ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.565+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:19.565+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.671+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.565492793 +1400 +14 m=+16.368074428 (1 nodes), 1.56µs from start of poll, 500.969492ms from cluster start. Timeout at 2023-05-16 04:06:19.671050599 +1400 +14 m=+16.473632374, 105.557946ms after stability + testing.go:149: cluster: 2023-05-16T04:06:19.671+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.565492793 +1400 +14 m=+16.368074428 (1 nodes), 1.56µs from start of poll, 500.969492ms from cluster start. Timeout at 2023-05-16 04:06:19.671050599 +1400 +14 m=+16.473632374, 105.557946ms after stability +testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.671+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: updating configuration: command=RemoveServer server-id=c6eed6c2-5993-16be-5926-f5d4aba279d8 server-addr= servers="[{Suffrage:Voter ID:9e1a7b19-2472-077e-8876-67055c18b3a4 Address:9e1a7b19-2472-077e-8876-67055c18b3a4} {Suffrage:Voter ID:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 Address:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2}]" + testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.671+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: updating configuration: command=RemoveServer server-id=c6eed6c2-5993-16be-5926-f5d4aba279d8 server-addr= servers="[{Suffrage:Voter ID:9e1a7b19-2472-077e-8876-67055c18b3a4 Address:9e1a7b19-2472-077e-8876-67055c18b3a4} {Suffrage:Voter ID:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2 Address:919d98fe-8306-7d85-0ccd-cbfa23f5e6b2}]" +testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.671+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: removed peer, stopping replication: peer=c6eed6c2-5993-16be-5926-f5d4aba279d8 last-index=4 + testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.671+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: removed peer, stopping replication: peer=c6eed6c2-5993-16be-5926-f5d4aba279d8 last-index=4 +testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.671+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: aborting pipeline replication: peer="{Voter c6eed6c2-5993-16be-5926-f5d4aba279d8 c6eed6c2-5993-16be-5926-f5d4aba279d8}" + testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.671+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: aborting pipeline replication: peer="{Voter c6eed6c2-5993-16be-5926-f5d4aba279d8 c6eed6c2-5993-16be-5926-f5d4aba279d8}" +testLoggerAdapter verbose: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.671+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: aborting pipeline replication: peer="{Voter 9e1a7b19-2472-077e-8876-67055c18b3a4 9e1a7b19-2472-077e-8876-67055c18b3a4}" + testing.go:149: 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: 2023-05-16T04:06:19.671+1400 [INFO] 919d98fe-8306-7d85-0ccd-cbfa23f5e6b2: aborting pipeline replication: peer="{Voter 9e1a7b19-2472-077e-8876-67055c18b3a4 9e1a7b19-2472-077e-8876-67055c18b3a4}" +--- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.61s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 -testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:38.982-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: initial configuration: index=1 servers="[{Suffrage:Voter ID:8be5d050-17e5-cdf6-4e0f-6b741446a319 Address:8be5d050-17e5-cdf6-4e0f-6b741446a319} {Suffrage:Voter ID:6e1ca5ec-74f1-904c-7558-ddcf468882f0 Address:6e1ca5ec-74f1-904c-7558-ddcf468882f0}]" - testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:38.982-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: initial configuration: index=1 servers="[{Suffrage:Voter ID:8be5d050-17e5-cdf6-4e0f-6b741446a319 Address:8be5d050-17e5-cdf6-4e0f-6b741446a319} {Suffrage:Voter ID:6e1ca5ec-74f1-904c-7558-ddcf468882f0 Address:6e1ca5ec-74f1-904c-7558-ddcf468882f0}]" -testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:38.982-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: entering follower state: follower="Node at 8be5d050-17e5-cdf6-4e0f-6b741446a319 [Follower]" leader= -testLoggerAdapter verbose: 6e1ca5ec-74f1-904c-7558-ddcf468882f0: 2024-06-16T08:26:38.982-1200 [INFO] 6e1ca5ec-74f1-904c-7558-ddcf468882f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:8be5d050-17e5-cdf6-4e0f-6b741446a319 Address:8be5d050-17e5-cdf6-4e0f-6b741446a319} {Suffrage:Voter ID:6e1ca5ec-74f1-904c-7558-ddcf468882f0 Address:6e1ca5ec-74f1-904c-7558-ddcf468882f0}]" - testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:38.982-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: entering follower state: follower="Node at 8be5d050-17e5-cdf6-4e0f-6b741446a319 [Follower]" leader= - testing.go:149: 6e1ca5ec-74f1-904c-7558-ddcf468882f0: 2024-06-16T08:26:38.982-1200 [INFO] 6e1ca5ec-74f1-904c-7558-ddcf468882f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:8be5d050-17e5-cdf6-4e0f-6b741446a319 Address:8be5d050-17e5-cdf6-4e0f-6b741446a319} {Suffrage:Voter ID:6e1ca5ec-74f1-904c-7558-ddcf468882f0 Address:6e1ca5ec-74f1-904c-7558-ddcf468882f0}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:38.982-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: 6e1ca5ec-74f1-904c-7558-ddcf468882f0: 2024-06-16T08:26:38.982-1200 [INFO] 6e1ca5ec-74f1-904c-7558-ddcf468882f0: entering follower state: follower="Node at 6e1ca5ec-74f1-904c-7558-ddcf468882f0 [Follower]" leader= - testing.go:149: cluster: 2024-06-16T08:26:38.982-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: 6e1ca5ec-74f1-904c-7558-ddcf468882f0: 2024-06-16T08:26:38.982-1200 [INFO] 6e1ca5ec-74f1-904c-7558-ddcf468882f0: entering follower state: follower="Node at 6e1ca5ec-74f1-904c-7558-ddcf468882f0 [Follower]" leader= -testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.040-1200 [WARN] 8be5d050-17e5-cdf6-4e0f-6b741446a319: heartbeat timeout reached, starting election: last-leader= - testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.040-1200 [WARN] 8be5d050-17e5-cdf6-4e0f-6b741446a319: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.040-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: entering candidate state: node="Node at 8be5d050-17e5-cdf6-4e0f-6b741446a319 [Candidate]" term=2 - testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.040-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: entering candidate state: node="Node at 8be5d050-17e5-cdf6-4e0f-6b741446a319 [Candidate]" term=2 -testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.040-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: election won: tally=2 - testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.040-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: election won: tally=2 -testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.041-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: entering leader state: leader="Node at 8be5d050-17e5-cdf6-4e0f-6b741446a319 [Leader]" - testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.041-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: entering leader state: leader="Node at 8be5d050-17e5-cdf6-4e0f-6b741446a319 [Leader]" -testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.041-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: added peer, starting replication: peer=6e1ca5ec-74f1-904c-7558-ddcf468882f0 - testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.041-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: added peer, starting replication: peer=6e1ca5ec-74f1-904c-7558-ddcf468882f0 -testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.041-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: pipelining replication: peer="{Voter 6e1ca5ec-74f1-904c-7558-ddcf468882f0 6e1ca5ec-74f1-904c-7558-ddcf468882f0}" - testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.041-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: pipelining replication: peer="{Voter 6e1ca5ec-74f1-904c-7558-ddcf468882f0 6e1ca5ec-74f1-904c-7558-ddcf468882f0}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.147-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:39.04115154 -1200 -12 m=+16.048468021 (1 nodes), 58.53756ms from start of poll, 59.1523ms from cluster start. Timeout at 2024-06-16 08:26:39.14717118 -1200 -12 m=+16.154487661, 106.01964ms after stability - testing.go:149: cluster: 2024-06-16T08:26:39.147-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:39.04115154 -1200 -12 m=+16.048468021 (1 nodes), 58.53756ms from start of poll, 59.1523ms from cluster start. Timeout at 2024-06-16 08:26:39.14717118 -1200 -12 m=+16.154487661, 106.01964ms after stability -testLoggerAdapter verbose: server-2d998396-b25c-5a51-4324-f0d7aebaad6b: 2024-06-16T08:26:39.147-1200 [INFO] server-2d998396-b25c-5a51-4324-f0d7aebaad6b: initial configuration: index=0 servers=[] - testing.go:149: server-2d998396-b25c-5a51-4324-f0d7aebaad6b: 2024-06-16T08:26:39.147-1200 [INFO] server-2d998396-b25c-5a51-4324-f0d7aebaad6b: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.147-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:39.147-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-2d998396-b25c-5a51-4324-f0d7aebaad6b: 2024-06-16T08:26:39.147-1200 [INFO] server-2d998396-b25c-5a51-4324-f0d7aebaad6b: entering follower state: follower="Node at 2d998396-b25c-5a51-4324-f0d7aebaad6b [Follower]" leader= - testing.go:149: server-2d998396-b25c-5a51-4324-f0d7aebaad6b: 2024-06-16T08:26:39.147-1200 [INFO] server-2d998396-b25c-5a51-4324-f0d7aebaad6b: entering follower state: follower="Node at 2d998396-b25c-5a51-4324-f0d7aebaad6b [Follower]" leader= -testLoggerAdapter verbose: server-2d998396-b25c-5a51-4324-f0d7aebaad6b: 2024-06-16T08:26:39.201-1200 [WARN] server-2d998396-b25c-5a51-4324-f0d7aebaad6b: no known peers, aborting election - testing.go:149: server-2d998396-b25c-5a51-4324-f0d7aebaad6b: 2024-06-16T08:26:39.201-1200 [WARN] server-2d998396-b25c-5a51-4324-f0d7aebaad6b: no known peers, aborting election -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.252-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.1477657 -1200 -12 m=+16.155082181 (1 nodes), 1.12µs from start of poll, 165.76646ms from cluster start. Timeout at 2024-06-16 08:26:39.25285324 -1200 -12 m=+16.260169701, 105.08752ms after stability - testing.go:149: cluster: 2024-06-16T08:26:39.252-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.1477657 -1200 -12 m=+16.155082181 (1 nodes), 1.12µs from start of poll, 165.76646ms from cluster start. Timeout at 2024-06-16 08:26:39.25285324 -1200 -12 m=+16.260169701, 105.08752ms after stability -testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.253-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: updating configuration: command=AddStaging server-id=server-2d998396-b25c-5a51-4324-f0d7aebaad6b server-addr=2d998396-b25c-5a51-4324-f0d7aebaad6b servers="[{Suffrage:Voter ID:8be5d050-17e5-cdf6-4e0f-6b741446a319 Address:8be5d050-17e5-cdf6-4e0f-6b741446a319} {Suffrage:Voter ID:6e1ca5ec-74f1-904c-7558-ddcf468882f0 Address:6e1ca5ec-74f1-904c-7558-ddcf468882f0} {Suffrage:Voter ID:server-2d998396-b25c-5a51-4324-f0d7aebaad6b Address:2d998396-b25c-5a51-4324-f0d7aebaad6b}]" - testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.253-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: updating configuration: command=AddStaging server-id=server-2d998396-b25c-5a51-4324-f0d7aebaad6b server-addr=2d998396-b25c-5a51-4324-f0d7aebaad6b servers="[{Suffrage:Voter ID:8be5d050-17e5-cdf6-4e0f-6b741446a319 Address:8be5d050-17e5-cdf6-4e0f-6b741446a319} {Suffrage:Voter ID:6e1ca5ec-74f1-904c-7558-ddcf468882f0 Address:6e1ca5ec-74f1-904c-7558-ddcf468882f0} {Suffrage:Voter ID:server-2d998396-b25c-5a51-4324-f0d7aebaad6b Address:2d998396-b25c-5a51-4324-f0d7aebaad6b}]" -testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.253-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: added peer, starting replication: peer=server-2d998396-b25c-5a51-4324-f0d7aebaad6b - testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.253-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: added peer, starting replication: peer=server-2d998396-b25c-5a51-4324-f0d7aebaad6b -testLoggerAdapter verbose: server-2d998396-b25c-5a51-4324-f0d7aebaad6b: 2024-06-16T08:26:39.253-1200 [WARN] server-2d998396-b25c-5a51-4324-f0d7aebaad6b: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-2d998396-b25c-5a51-4324-f0d7aebaad6b: 2024-06-16T08:26:39.253-1200 [WARN] server-2d998396-b25c-5a51-4324-f0d7aebaad6b: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.253-1200 [WARN] 8be5d050-17e5-cdf6-4e0f-6b741446a319: appendEntries rejected, sending older logs: peer="{Voter server-2d998396-b25c-5a51-4324-f0d7aebaad6b 2d998396-b25c-5a51-4324-f0d7aebaad6b}" next=1 - testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.253-1200 [WARN] 8be5d050-17e5-cdf6-4e0f-6b741446a319: appendEntries rejected, sending older logs: peer="{Voter server-2d998396-b25c-5a51-4324-f0d7aebaad6b 2d998396-b25c-5a51-4324-f0d7aebaad6b}" next=1 -testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.253-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: pipelining replication: peer="{Voter server-2d998396-b25c-5a51-4324-f0d7aebaad6b 2d998396-b25c-5a51-4324-f0d7aebaad6b}" - testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.253-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: pipelining replication: peer="{Voter server-2d998396-b25c-5a51-4324-f0d7aebaad6b 2d998396-b25c-5a51-4324-f0d7aebaad6b}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.259-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:39.259-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.365-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.2593279 -1200 -12 m=+16.266644361 (1 nodes), 1.6µs from start of poll, 277.32864ms from cluster start. Timeout at 2024-06-16 08:26:39.36508228 -1200 -12 m=+16.372398761, 105.7544ms after stability - testing.go:149: cluster: 2024-06-16T08:26:39.365-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.2593279 -1200 -12 m=+16.266644361 (1 nodes), 1.6µs from start of poll, 277.32864ms from cluster start. Timeout at 2024-06-16 08:26:39.36508228 -1200 -12 m=+16.372398761, 105.7544ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.365-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:39.365-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.471-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.36526892 -1200 -12 m=+16.372585381 (1 nodes), 1.24µs from start of poll, 383.26966ms from cluster start. Timeout at 2024-06-16 08:26:39.47114692 -1200 -12 m=+16.478463401, 105.87802ms after stability - testing.go:149: cluster: 2024-06-16T08:26:39.471-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.36526892 -1200 -12 m=+16.372585381 (1 nodes), 1.24µs from start of poll, 383.26966ms from cluster start. Timeout at 2024-06-16 08:26:39.47114692 -1200 -12 m=+16.478463401, 105.87802ms after stability -testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.471-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: updating configuration: command=RemoveServer server-id=6e1ca5ec-74f1-904c-7558-ddcf468882f0 server-addr= servers="[{Suffrage:Voter ID:8be5d050-17e5-cdf6-4e0f-6b741446a319 Address:8be5d050-17e5-cdf6-4e0f-6b741446a319} {Suffrage:Voter ID:server-2d998396-b25c-5a51-4324-f0d7aebaad6b Address:2d998396-b25c-5a51-4324-f0d7aebaad6b}]" - testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.471-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: updating configuration: command=RemoveServer server-id=6e1ca5ec-74f1-904c-7558-ddcf468882f0 server-addr= servers="[{Suffrage:Voter ID:8be5d050-17e5-cdf6-4e0f-6b741446a319 Address:8be5d050-17e5-cdf6-4e0f-6b741446a319} {Suffrage:Voter ID:server-2d998396-b25c-5a51-4324-f0d7aebaad6b Address:2d998396-b25c-5a51-4324-f0d7aebaad6b}]" -testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.471-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: removed peer, stopping replication: peer=6e1ca5ec-74f1-904c-7558-ddcf468882f0 last-index=4 - testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.471-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: removed peer, stopping replication: peer=6e1ca5ec-74f1-904c-7558-ddcf468882f0 last-index=4 -testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.471-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: aborting pipeline replication: peer="{Voter 6e1ca5ec-74f1-904c-7558-ddcf468882f0 6e1ca5ec-74f1-904c-7558-ddcf468882f0}" - testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.471-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: aborting pipeline replication: peer="{Voter 6e1ca5ec-74f1-904c-7558-ddcf468882f0 6e1ca5ec-74f1-904c-7558-ddcf468882f0}" -testLoggerAdapter verbose: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.471-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: aborting pipeline replication: peer="{Voter server-2d998396-b25c-5a51-4324-f0d7aebaad6b 2d998396-b25c-5a51-4324-f0d7aebaad6b}" - testing.go:149: 8be5d050-17e5-cdf6-4e0f-6b741446a319: 2024-06-16T08:26:39.471-1200 [INFO] 8be5d050-17e5-cdf6-4e0f-6b741446a319: aborting pipeline replication: peer="{Voter server-2d998396-b25c-5a51-4324-f0d7aebaad6b 2d998396-b25c-5a51-4324-f0d7aebaad6b}" ---- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.49s) +testLoggerAdapter verbose: 75b2fd5b-075b-1749-1248-85b03602aac3: 2023-05-16T04:06:19.673+1400 [INFO] 75b2fd5b-075b-1749-1248-85b03602aac3: initial configuration: index=1 servers="[{Suffrage:Voter ID:75b2fd5b-075b-1749-1248-85b03602aac3 Address:75b2fd5b-075b-1749-1248-85b03602aac3} {Suffrage:Voter ID:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd Address:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd}]" + testing.go:149: 75b2fd5b-075b-1749-1248-85b03602aac3: 2023-05-16T04:06:19.673+1400 [INFO] 75b2fd5b-075b-1749-1248-85b03602aac3: initial configuration: index=1 servers="[{Suffrage:Voter ID:75b2fd5b-075b-1749-1248-85b03602aac3 Address:75b2fd5b-075b-1749-1248-85b03602aac3} {Suffrage:Voter ID:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd Address:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd}]" +testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.673+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:75b2fd5b-075b-1749-1248-85b03602aac3 Address:75b2fd5b-075b-1749-1248-85b03602aac3} {Suffrage:Voter ID:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd Address:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd}]" + testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.673+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:75b2fd5b-075b-1749-1248-85b03602aac3 Address:75b2fd5b-075b-1749-1248-85b03602aac3} {Suffrage:Voter ID:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd Address:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.673+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:19.673+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: 75b2fd5b-075b-1749-1248-85b03602aac3: 2023-05-16T04:06:19.674+1400 [INFO] 75b2fd5b-075b-1749-1248-85b03602aac3: entering follower state: follower="Node at 75b2fd5b-075b-1749-1248-85b03602aac3 [Follower]" leader= + testing.go:149: 75b2fd5b-075b-1749-1248-85b03602aac3: 2023-05-16T04:06:19.674+1400 [INFO] 75b2fd5b-075b-1749-1248-85b03602aac3: entering follower state: follower="Node at 75b2fd5b-075b-1749-1248-85b03602aac3 [Follower]" leader= +testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.674+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: entering follower state: follower="Node at 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd [Follower]" leader= + testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.674+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: entering follower state: follower="Node at 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd [Follower]" leader= +testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [WARN] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: heartbeat timeout reached, starting election: last-leader= + testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [WARN] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: entering candidate state: node="Node at 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd [Candidate]" term=2 + testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: entering candidate state: node="Node at 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd [Candidate]" term=2 +testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: election won: tally=2 + testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: election won: tally=2 +testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: entering leader state: leader="Node at 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd [Leader]" + testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: entering leader state: leader="Node at 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd [Leader]" +testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: added peer, starting replication: peer=75b2fd5b-075b-1749-1248-85b03602aac3 + testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: added peer, starting replication: peer=75b2fd5b-075b-1749-1248-85b03602aac3 +testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: pipelining replication: peer="{Voter 75b2fd5b-075b-1749-1248-85b03602aac3 75b2fd5b-075b-1749-1248-85b03602aac3}" + testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.728+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: pipelining replication: peer="{Voter 75b2fd5b-075b-1749-1248-85b03602aac3 75b2fd5b-075b-1749-1248-85b03602aac3}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.834+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:19.72875002 +1400 +14 m=+16.531331635 (1 nodes), 54.765886ms from start of poll, 55.415354ms from cluster start. Timeout at 2023-05-16 04:06:19.834214265 +1400 +14 m=+16.636796020, 105.464385ms after stability + testing.go:149: cluster: 2023-05-16T04:06:19.834+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:19.72875002 +1400 +14 m=+16.531331635 (1 nodes), 54.765886ms from start of poll, 55.415354ms from cluster start. Timeout at 2023-05-16 04:06:19.834214265 +1400 +14 m=+16.636796020, 105.464385ms after stability +testLoggerAdapter verbose: server-038e407e-9062-3d91-552a-1f5650a48253: 2023-05-16T04:06:19.834+1400 [INFO] server-038e407e-9062-3d91-552a-1f5650a48253: initial configuration: index=0 servers=[] + testing.go:149: server-038e407e-9062-3d91-552a-1f5650a48253: 2023-05-16T04:06:19.834+1400 [INFO] server-038e407e-9062-3d91-552a-1f5650a48253: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.834+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:19.834+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-038e407e-9062-3d91-552a-1f5650a48253: 2023-05-16T04:06:19.835+1400 [INFO] server-038e407e-9062-3d91-552a-1f5650a48253: entering follower state: follower="Node at 038e407e-9062-3d91-552a-1f5650a48253 [Follower]" leader= + testing.go:149: server-038e407e-9062-3d91-552a-1f5650a48253: 2023-05-16T04:06:19.835+1400 [INFO] server-038e407e-9062-3d91-552a-1f5650a48253: entering follower state: follower="Node at 038e407e-9062-3d91-552a-1f5650a48253 [Follower]" leader= +testLoggerAdapter verbose: server-038e407e-9062-3d91-552a-1f5650a48253: 2023-05-16T04:06:19.890+1400 [WARN] server-038e407e-9062-3d91-552a-1f5650a48253: no known peers, aborting election + testing.go:149: server-038e407e-9062-3d91-552a-1f5650a48253: 2023-05-16T04:06:19.890+1400 [WARN] server-038e407e-9062-3d91-552a-1f5650a48253: no known peers, aborting election +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.940+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.835011634 +1400 +14 m=+16.637593249 (1 nodes), 2.06µs from start of poll, 161.676968ms from cluster start. Timeout at 2023-05-16 04:06:19.940104674 +1400 +14 m=+16.742686309, 105.09306ms after stability + testing.go:149: cluster: 2023-05-16T04:06:19.940+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.835011634 +1400 +14 m=+16.637593249 (1 nodes), 2.06µs from start of poll, 161.676968ms from cluster start. Timeout at 2023-05-16 04:06:19.940104674 +1400 +14 m=+16.742686309, 105.09306ms after stability +testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.940+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: updating configuration: command=AddStaging server-id=server-038e407e-9062-3d91-552a-1f5650a48253 server-addr=038e407e-9062-3d91-552a-1f5650a48253 servers="[{Suffrage:Voter ID:75b2fd5b-075b-1749-1248-85b03602aac3 Address:75b2fd5b-075b-1749-1248-85b03602aac3} {Suffrage:Voter ID:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd Address:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd} {Suffrage:Voter ID:server-038e407e-9062-3d91-552a-1f5650a48253 Address:038e407e-9062-3d91-552a-1f5650a48253}]" + testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.940+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: updating configuration: command=AddStaging server-id=server-038e407e-9062-3d91-552a-1f5650a48253 server-addr=038e407e-9062-3d91-552a-1f5650a48253 servers="[{Suffrage:Voter ID:75b2fd5b-075b-1749-1248-85b03602aac3 Address:75b2fd5b-075b-1749-1248-85b03602aac3} {Suffrage:Voter ID:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd Address:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd} {Suffrage:Voter ID:server-038e407e-9062-3d91-552a-1f5650a48253 Address:038e407e-9062-3d91-552a-1f5650a48253}]" +testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.940+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: added peer, starting replication: peer=server-038e407e-9062-3d91-552a-1f5650a48253 + testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.940+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: added peer, starting replication: peer=server-038e407e-9062-3d91-552a-1f5650a48253 +testLoggerAdapter verbose: server-038e407e-9062-3d91-552a-1f5650a48253: 2023-05-16T04:06:19.940+1400 [WARN] server-038e407e-9062-3d91-552a-1f5650a48253: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-038e407e-9062-3d91-552a-1f5650a48253: 2023-05-16T04:06:19.940+1400 [WARN] server-038e407e-9062-3d91-552a-1f5650a48253: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.940+1400 [WARN] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: appendEntries rejected, sending older logs: peer="{Voter server-038e407e-9062-3d91-552a-1f5650a48253 038e407e-9062-3d91-552a-1f5650a48253}" next=1 + testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.940+1400 [WARN] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: appendEntries rejected, sending older logs: peer="{Voter server-038e407e-9062-3d91-552a-1f5650a48253 038e407e-9062-3d91-552a-1f5650a48253}" next=1 +testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.941+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: pipelining replication: peer="{Voter server-038e407e-9062-3d91-552a-1f5650a48253 038e407e-9062-3d91-552a-1f5650a48253}" + testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:19.941+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: pipelining replication: peer="{Voter server-038e407e-9062-3d91-552a-1f5650a48253 038e407e-9062-3d91-552a-1f5650a48253}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:19.941+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:19.941+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.047+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.94140807 +1400 +14 m=+16.743989685 (1 nodes), 1.3µs from start of poll, 268.073404ms from cluster start. Timeout at 2023-05-16 04:06:20.047218838 +1400 +14 m=+16.849800493, 105.810808ms after stability + testing.go:149: cluster: 2023-05-16T04:06:20.047+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:19.94140807 +1400 +14 m=+16.743989685 (1 nodes), 1.3µs from start of poll, 268.073404ms from cluster start. Timeout at 2023-05-16 04:06:20.047218838 +1400 +14 m=+16.849800493, 105.810808ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.047+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:20.047+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.153+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.047557142 +1400 +14 m=+16.850138777 (1 nodes), 1.94µs from start of poll, 374.222496ms from cluster start. Timeout at 2023-05-16 04:06:20.153336711 +1400 +14 m=+16.955918326, 105.779549ms after stability + testing.go:149: cluster: 2023-05-16T04:06:20.153+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.047557142 +1400 +14 m=+16.850138777 (1 nodes), 1.94µs from start of poll, 374.222496ms from cluster start. Timeout at 2023-05-16 04:06:20.153336711 +1400 +14 m=+16.955918326, 105.779549ms after stability +testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:20.153+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: updating configuration: command=RemoveServer server-id=75b2fd5b-075b-1749-1248-85b03602aac3 server-addr= servers="[{Suffrage:Voter ID:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd Address:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd} {Suffrage:Voter ID:server-038e407e-9062-3d91-552a-1f5650a48253 Address:038e407e-9062-3d91-552a-1f5650a48253}]" + testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:20.153+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: updating configuration: command=RemoveServer server-id=75b2fd5b-075b-1749-1248-85b03602aac3 server-addr= servers="[{Suffrage:Voter ID:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd Address:3d469c01-a1b7-74fd-b122-cc4dfeefb5bd} {Suffrage:Voter ID:server-038e407e-9062-3d91-552a-1f5650a48253 Address:038e407e-9062-3d91-552a-1f5650a48253}]" +testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:20.153+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: removed peer, stopping replication: peer=75b2fd5b-075b-1749-1248-85b03602aac3 last-index=4 + testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:20.153+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: removed peer, stopping replication: peer=75b2fd5b-075b-1749-1248-85b03602aac3 last-index=4 +testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:20.153+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: aborting pipeline replication: peer="{Voter 75b2fd5b-075b-1749-1248-85b03602aac3 75b2fd5b-075b-1749-1248-85b03602aac3}" + testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:20.153+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: aborting pipeline replication: peer="{Voter 75b2fd5b-075b-1749-1248-85b03602aac3 75b2fd5b-075b-1749-1248-85b03602aac3}" +testLoggerAdapter verbose: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:20.154+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: aborting pipeline replication: peer="{Voter server-038e407e-9062-3d91-552a-1f5650a48253 038e407e-9062-3d91-552a-1f5650a48253}" + testing.go:149: 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: 2023-05-16T04:06:20.154+1400 [INFO] 3d469c01-a1b7-74fd-b122-cc4dfeefb5bd: aborting pipeline replication: peer="{Voter server-038e407e-9062-3d91-552a-1f5650a48253 038e407e-9062-3d91-552a-1f5650a48253}" +--- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.48s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer -testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.473-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 Address:2dc81aac-7f82-a03d-abb5-f7d064fdf622} {Suffrage:Voter ID:server-9738814d-4416-3c04-4fee-62a746a4a56d Address:9738814d-4416-3c04-4fee-62a746a4a56d} {Suffrage:Voter ID:server-54ba0f95-76fd-ffb5-6f29-5753657abb1a Address:54ba0f95-76fd-ffb5-6f29-5753657abb1a}]" - testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.473-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 Address:2dc81aac-7f82-a03d-abb5-f7d064fdf622} {Suffrage:Voter ID:server-9738814d-4416-3c04-4fee-62a746a4a56d Address:9738814d-4416-3c04-4fee-62a746a4a56d} {Suffrage:Voter ID:server-54ba0f95-76fd-ffb5-6f29-5753657abb1a Address:54ba0f95-76fd-ffb5-6f29-5753657abb1a}]" -testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.473-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: entering follower state: follower="Node at 2dc81aac-7f82-a03d-abb5-f7d064fdf622 [Follower]" leader= - testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.473-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: entering follower state: follower="Node at 2dc81aac-7f82-a03d-abb5-f7d064fdf622 [Follower]" leader= -testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.473-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 Address:2dc81aac-7f82-a03d-abb5-f7d064fdf622} {Suffrage:Voter ID:server-9738814d-4416-3c04-4fee-62a746a4a56d Address:9738814d-4416-3c04-4fee-62a746a4a56d} {Suffrage:Voter ID:server-54ba0f95-76fd-ffb5-6f29-5753657abb1a Address:54ba0f95-76fd-ffb5-6f29-5753657abb1a}]" - testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.473-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 Address:2dc81aac-7f82-a03d-abb5-f7d064fdf622} {Suffrage:Voter ID:server-9738814d-4416-3c04-4fee-62a746a4a56d Address:9738814d-4416-3c04-4fee-62a746a4a56d} {Suffrage:Voter ID:server-54ba0f95-76fd-ffb5-6f29-5753657abb1a Address:54ba0f95-76fd-ffb5-6f29-5753657abb1a}]" -testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.473-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: entering follower state: follower="Node at 9738814d-4416-3c04-4fee-62a746a4a56d [Follower]" leader= - testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.473-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: entering follower state: follower="Node at 9738814d-4416-3c04-4fee-62a746a4a56d [Follower]" leader= -testLoggerAdapter verbose: server-54ba0f95-76fd-ffb5-6f29-5753657abb1a: 2024-06-16T08:26:39.473-1200 [INFO] server-54ba0f95-76fd-ffb5-6f29-5753657abb1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 Address:2dc81aac-7f82-a03d-abb5-f7d064fdf622} {Suffrage:Voter ID:server-9738814d-4416-3c04-4fee-62a746a4a56d Address:9738814d-4416-3c04-4fee-62a746a4a56d} {Suffrage:Voter ID:server-54ba0f95-76fd-ffb5-6f29-5753657abb1a Address:54ba0f95-76fd-ffb5-6f29-5753657abb1a}]" - testing.go:149: server-54ba0f95-76fd-ffb5-6f29-5753657abb1a: 2024-06-16T08:26:39.473-1200 [INFO] server-54ba0f95-76fd-ffb5-6f29-5753657abb1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 Address:2dc81aac-7f82-a03d-abb5-f7d064fdf622} {Suffrage:Voter ID:server-9738814d-4416-3c04-4fee-62a746a4a56d Address:9738814d-4416-3c04-4fee-62a746a4a56d} {Suffrage:Voter ID:server-54ba0f95-76fd-ffb5-6f29-5753657abb1a Address:54ba0f95-76fd-ffb5-6f29-5753657abb1a}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.473-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:39.473-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-54ba0f95-76fd-ffb5-6f29-5753657abb1a: 2024-06-16T08:26:39.473-1200 [INFO] server-54ba0f95-76fd-ffb5-6f29-5753657abb1a: entering follower state: follower="Node at 54ba0f95-76fd-ffb5-6f29-5753657abb1a [Follower]" leader= - testing.go:149: server-54ba0f95-76fd-ffb5-6f29-5753657abb1a: 2024-06-16T08:26:39.473-1200 [INFO] server-54ba0f95-76fd-ffb5-6f29-5753657abb1a: entering follower state: follower="Node at 54ba0f95-76fd-ffb5-6f29-5753657abb1a [Follower]" leader= -testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.532-1200 [WARN] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.532-1200 [WARN] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.532-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: entering candidate state: node="Node at 2dc81aac-7f82-a03d-abb5-f7d064fdf622 [Candidate]" term=2 - testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.532-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: entering candidate state: node="Node at 2dc81aac-7f82-a03d-abb5-f7d064fdf622 [Candidate]" term=2 -testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: election won: tally=2 - testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: election won: tally=2 -testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: entering leader state: leader="Node at 2dc81aac-7f82-a03d-abb5-f7d064fdf622 [Leader]" - testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: entering leader state: leader="Node at 2dc81aac-7f82-a03d-abb5-f7d064fdf622 [Leader]" -testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: added peer, starting replication: peer=server-9738814d-4416-3c04-4fee-62a746a4a56d - testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: added peer, starting replication: peer=server-9738814d-4416-3c04-4fee-62a746a4a56d -testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: added peer, starting replication: peer=server-54ba0f95-76fd-ffb5-6f29-5753657abb1a - testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: added peer, starting replication: peer=server-54ba0f95-76fd-ffb5-6f29-5753657abb1a -testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: pipelining replication: peer="{Voter server-9738814d-4416-3c04-4fee-62a746a4a56d 9738814d-4416-3c04-4fee-62a746a4a56d}" - testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: pipelining replication: peer="{Voter server-9738814d-4416-3c04-4fee-62a746a4a56d 9738814d-4416-3c04-4fee-62a746a4a56d}" -testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: pipelining replication: peer="{Voter server-54ba0f95-76fd-ffb5-6f29-5753657abb1a 54ba0f95-76fd-ffb5-6f29-5753657abb1a}" - testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.533-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: pipelining replication: peer="{Voter server-54ba0f95-76fd-ffb5-6f29-5753657abb1a 54ba0f95-76fd-ffb5-6f29-5753657abb1a}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.638-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.53328876 -1200 -12 m=+16.540605261 (1 nodes), 59.61988ms from start of poll, 60.40556ms from cluster start. Timeout at 2024-06-16 08:26:39.63851342 -1200 -12 m=+16.645829881, 105.22462ms after stability - testing.go:149: cluster: 2024-06-16T08:26:39.638-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.53328876 -1200 -12 m=+16.540605261 (1 nodes), 59.61988ms from start of poll, 60.40556ms from cluster start. Timeout at 2024-06-16 08:26:39.63851342 -1200 -12 m=+16.645829881, 105.22462ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.638-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:39.638-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.744-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.63871358 -1200 -12 m=+16.646030041 (1 nodes), 1.38µs from start of poll, 165.83034ms from cluster start. Timeout at 2024-06-16 08:26:39.74405586 -1200 -12 m=+16.751372361, 105.34232ms after stability - testing.go:149: cluster: 2024-06-16T08:26:39.744-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.63871358 -1200 -12 m=+16.646030041 (1 nodes), 1.38µs from start of poll, 165.83034ms from cluster start. Timeout at 2024-06-16 08:26:39.74405586 -1200 -12 m=+16.751372361, 105.34232ms after stability -testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: entering candidate state: node="Node at 9738814d-4416-3c04-4fee-62a746a4a56d [Candidate]" term=3 - testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: entering candidate state: node="Node at 9738814d-4416-3c04-4fee-62a746a4a56d [Candidate]" term=3 -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.744-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:39.744-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.744-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: entering follower state: follower="Node at 2dc81aac-7f82-a03d-abb5-f7d064fdf622 [Follower]" leader= - testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.744-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: entering follower state: follower="Node at 2dc81aac-7f82-a03d-abb5-f7d064fdf622 [Follower]" leader= -testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: election won: tally=2 -testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.744-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: aborting pipeline replication: peer="{Voter server-9738814d-4416-3c04-4fee-62a746a4a56d 9738814d-4416-3c04-4fee-62a746a4a56d}" - testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: election won: tally=2 -testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: entering leader state: leader="Node at 9738814d-4416-3c04-4fee-62a746a4a56d [Leader]" - testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.744-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: aborting pipeline replication: peer="{Voter server-9738814d-4416-3c04-4fee-62a746a4a56d 9738814d-4416-3c04-4fee-62a746a4a56d}" - testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: entering leader state: leader="Node at 9738814d-4416-3c04-4fee-62a746a4a56d [Leader]" -testLoggerAdapter verbose: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.744-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: aborting pipeline replication: peer="{Voter server-54ba0f95-76fd-ffb5-6f29-5753657abb1a 54ba0f95-76fd-ffb5-6f29-5753657abb1a}" -testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: added peer, starting replication: peer=server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 - testing.go:149: server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: 2024-06-16T08:26:39.744-1200 [INFO] server-2dc81aac-7f82-a03d-abb5-f7d064fdf622: aborting pipeline replication: peer="{Voter server-54ba0f95-76fd-ffb5-6f29-5753657abb1a 54ba0f95-76fd-ffb5-6f29-5753657abb1a}" - testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: added peer, starting replication: peer=server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 -testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: added peer, starting replication: peer=server-54ba0f95-76fd-ffb5-6f29-5753657abb1a - testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: added peer, starting replication: peer=server-54ba0f95-76fd-ffb5-6f29-5753657abb1a -testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: pipelining replication: peer="{Voter server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 2dc81aac-7f82-a03d-abb5-f7d064fdf622}" - testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: pipelining replication: peer="{Voter server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 2dc81aac-7f82-a03d-abb5-f7d064fdf622}" -testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: pipelining replication: peer="{Voter server-54ba0f95-76fd-ffb5-6f29-5753657abb1a 54ba0f95-76fd-ffb5-6f29-5753657abb1a}" - testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.744-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: pipelining replication: peer="{Voter server-54ba0f95-76fd-ffb5-6f29-5753657abb1a 54ba0f95-76fd-ffb5-6f29-5753657abb1a}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.850-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.74465376 -1200 -12 m=+16.751970241 (1 nodes), 170.8µs from start of poll, 271.77054ms from cluster start. Timeout at 2024-06-16 08:26:39.85048896 -1200 -12 m=+16.857805441, 105.8352ms after stability - testing.go:149: cluster: 2024-06-16T08:26:39.850-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.74465376 -1200 -12 m=+16.751970241 (1 nodes), 170.8µs from start of poll, 271.77054ms from cluster start. Timeout at 2024-06-16 08:26:39.85048896 -1200 -12 m=+16.857805441, 105.8352ms after stability -testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.850-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: aborting pipeline replication: peer="{Voter server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 2dc81aac-7f82-a03d-abb5-f7d064fdf622}" - testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.850-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: aborting pipeline replication: peer="{Voter server-2dc81aac-7f82-a03d-abb5-f7d064fdf622 2dc81aac-7f82-a03d-abb5-f7d064fdf622}" -testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.850-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: aborting pipeline replication: peer="{Voter server-54ba0f95-76fd-ffb5-6f29-5753657abb1a 54ba0f95-76fd-ffb5-6f29-5753657abb1a}" - testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.850-1200 [INFO] server-9738814d-4416-3c04-4fee-62a746a4a56d: aborting pipeline replication: peer="{Voter server-54ba0f95-76fd-ffb5-6f29-5753657abb1a 54ba0f95-76fd-ffb5-6f29-5753657abb1a}" -testLoggerAdapter verbose: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.901-1200 [ERROR] server-9738814d-4416-3c04-4fee-62a746a4a56d: failed to heartbeat to: peer=2dc81aac-7f82-a03d-abb5-f7d064fdf622 error="command timed out" - testing.go:149: server-9738814d-4416-3c04-4fee-62a746a4a56d: 2024-06-16T08:26:39.901-1200 [ERROR] server-9738814d-4416-3c04-4fee-62a746a4a56d: failed to heartbeat to: peer=2dc81aac-7f82-a03d-abb5-f7d064fdf622 error="command timed out" ---- PASS: TestRaft_LeadershipTransfer (0.43s) +testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.155+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab Address:3837dac0-485c-2d3b-dd06-8c114f9ac0ab} {Suffrage:Voter ID:server-6249adaf-7fa0-9a24-f9f2-179cb0877210 Address:6249adaf-7fa0-9a24-f9f2-179cb0877210} {Suffrage:Voter ID:server-b6401cfe-e422-54c7-68c4-fa38095f64d8 Address:b6401cfe-e422-54c7-68c4-fa38095f64d8}]" + testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.155+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab Address:3837dac0-485c-2d3b-dd06-8c114f9ac0ab} {Suffrage:Voter ID:server-6249adaf-7fa0-9a24-f9f2-179cb0877210 Address:6249adaf-7fa0-9a24-f9f2-179cb0877210} {Suffrage:Voter ID:server-b6401cfe-e422-54c7-68c4-fa38095f64d8 Address:b6401cfe-e422-54c7-68c4-fa38095f64d8}]" +testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.156+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab Address:3837dac0-485c-2d3b-dd06-8c114f9ac0ab} {Suffrage:Voter ID:server-6249adaf-7fa0-9a24-f9f2-179cb0877210 Address:6249adaf-7fa0-9a24-f9f2-179cb0877210} {Suffrage:Voter ID:server-b6401cfe-e422-54c7-68c4-fa38095f64d8 Address:b6401cfe-e422-54c7-68c4-fa38095f64d8}]" + testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.156+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab Address:3837dac0-485c-2d3b-dd06-8c114f9ac0ab} {Suffrage:Voter ID:server-6249adaf-7fa0-9a24-f9f2-179cb0877210 Address:6249adaf-7fa0-9a24-f9f2-179cb0877210} {Suffrage:Voter ID:server-b6401cfe-e422-54c7-68c4-fa38095f64d8 Address:b6401cfe-e422-54c7-68c4-fa38095f64d8}]" +testLoggerAdapter verbose: server-b6401cfe-e422-54c7-68c4-fa38095f64d8: 2023-05-16T04:06:20.156+1400 [INFO] server-b6401cfe-e422-54c7-68c4-fa38095f64d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab Address:3837dac0-485c-2d3b-dd06-8c114f9ac0ab} {Suffrage:Voter ID:server-6249adaf-7fa0-9a24-f9f2-179cb0877210 Address:6249adaf-7fa0-9a24-f9f2-179cb0877210} {Suffrage:Voter ID:server-b6401cfe-e422-54c7-68c4-fa38095f64d8 Address:b6401cfe-e422-54c7-68c4-fa38095f64d8}]" + testing.go:149: server-b6401cfe-e422-54c7-68c4-fa38095f64d8: 2023-05-16T04:06:20.156+1400 [INFO] server-b6401cfe-e422-54c7-68c4-fa38095f64d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab Address:3837dac0-485c-2d3b-dd06-8c114f9ac0ab} {Suffrage:Voter ID:server-6249adaf-7fa0-9a24-f9f2-179cb0877210 Address:6249adaf-7fa0-9a24-f9f2-179cb0877210} {Suffrage:Voter ID:server-b6401cfe-e422-54c7-68c4-fa38095f64d8 Address:b6401cfe-e422-54c7-68c4-fa38095f64d8}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.156+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:20.156+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.156+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: entering follower state: follower="Node at 3837dac0-485c-2d3b-dd06-8c114f9ac0ab [Follower]" leader= + testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.156+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: entering follower state: follower="Node at 3837dac0-485c-2d3b-dd06-8c114f9ac0ab [Follower]" leader= +testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.156+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: entering follower state: follower="Node at 6249adaf-7fa0-9a24-f9f2-179cb0877210 [Follower]" leader= + testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.156+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: entering follower state: follower="Node at 6249adaf-7fa0-9a24-f9f2-179cb0877210 [Follower]" leader= +testLoggerAdapter verbose: server-b6401cfe-e422-54c7-68c4-fa38095f64d8: 2023-05-16T04:06:20.156+1400 [INFO] server-b6401cfe-e422-54c7-68c4-fa38095f64d8: entering follower state: follower="Node at b6401cfe-e422-54c7-68c4-fa38095f64d8 [Follower]" leader= + testing.go:149: server-b6401cfe-e422-54c7-68c4-fa38095f64d8: 2023-05-16T04:06:20.156+1400 [INFO] server-b6401cfe-e422-54c7-68c4-fa38095f64d8: entering follower state: follower="Node at b6401cfe-e422-54c7-68c4-fa38095f64d8 [Follower]" leader= +testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [WARN] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [WARN] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: entering candidate state: node="Node at 6249adaf-7fa0-9a24-f9f2-179cb0877210 [Candidate]" term=2 + testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: entering candidate state: node="Node at 6249adaf-7fa0-9a24-f9f2-179cb0877210 [Candidate]" term=2 +testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: election won: tally=2 + testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: election won: tally=2 +testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: entering leader state: leader="Node at 6249adaf-7fa0-9a24-f9f2-179cb0877210 [Leader]" + testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: entering leader state: leader="Node at 6249adaf-7fa0-9a24-f9f2-179cb0877210 [Leader]" +testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: added peer, starting replication: peer=server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab + testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: added peer, starting replication: peer=server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab +testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: added peer, starting replication: peer=server-b6401cfe-e422-54c7-68c4-fa38095f64d8 + testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: added peer, starting replication: peer=server-b6401cfe-e422-54c7-68c4-fa38095f64d8 +testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: pipelining replication: peer="{Voter server-b6401cfe-e422-54c7-68c4-fa38095f64d8 b6401cfe-e422-54c7-68c4-fa38095f64d8}" + testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.225+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: pipelining replication: peer="{Voter server-b6401cfe-e422-54c7-68c4-fa38095f64d8 b6401cfe-e422-54c7-68c4-fa38095f64d8}" +testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.226+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: pipelining replication: peer="{Voter server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab 3837dac0-485c-2d3b-dd06-8c114f9ac0ab}" + testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.226+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: pipelining replication: peer="{Voter server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab 3837dac0-485c-2d3b-dd06-8c114f9ac0ab}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.332+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.226303772 +1400 +14 m=+17.028885407 (1 nodes), 69.662622ms from start of poll, 70.586653ms from cluster start. Timeout at 2023-05-16 04:06:20.332152501 +1400 +14 m=+17.134734156, 105.848749ms after stability + testing.go:149: cluster: 2023-05-16T04:06:20.332+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.226303772 +1400 +14 m=+17.028885407 (1 nodes), 69.662622ms from start of poll, 70.586653ms from cluster start. Timeout at 2023-05-16 04:06:20.332152501 +1400 +14 m=+17.134734156, 105.848749ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.332+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:20.332+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.437+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.332473485 +1400 +14 m=+17.135055100 (1 nodes), 2.1µs from start of poll, 176.756346ms from cluster start. Timeout at 2023-05-16 04:06:20.437477144 +1400 +14 m=+17.240058779, 105.003679ms after stability + testing.go:149: cluster: 2023-05-16T04:06:20.437+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.332473485 +1400 +14 m=+17.135055100 (1 nodes), 2.1µs from start of poll, 176.756346ms from cluster start. Timeout at 2023-05-16 04:06:20.437477144 +1400 +14 m=+17.240058779, 105.003679ms after stability +testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.437+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: entering candidate state: node="Node at 3837dac0-485c-2d3b-dd06-8c114f9ac0ab [Candidate]" term=3 + testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.437+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: entering candidate state: node="Node at 3837dac0-485c-2d3b-dd06-8c114f9ac0ab [Candidate]" term=3 +testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: election won: tally=2 + testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: election won: tally=2 +testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: entering leader state: leader="Node at 3837dac0-485c-2d3b-dd06-8c114f9ac0ab [Leader]" + testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: entering leader state: leader="Node at 3837dac0-485c-2d3b-dd06-8c114f9ac0ab [Leader]" +testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: added peer, starting replication: peer=server-6249adaf-7fa0-9a24-f9f2-179cb0877210 + testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: added peer, starting replication: peer=server-6249adaf-7fa0-9a24-f9f2-179cb0877210 +testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: added peer, starting replication: peer=server-b6401cfe-e422-54c7-68c4-fa38095f64d8 + testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: added peer, starting replication: peer=server-b6401cfe-e422-54c7-68c4-fa38095f64d8 +testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: pipelining replication: peer="{Voter server-b6401cfe-e422-54c7-68c4-fa38095f64d8 b6401cfe-e422-54c7-68c4-fa38095f64d8}" + testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: pipelining replication: peer="{Voter server-b6401cfe-e422-54c7-68c4-fa38095f64d8 b6401cfe-e422-54c7-68c4-fa38095f64d8}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.438+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:20.438+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.438+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: entering follower state: follower="Node at 6249adaf-7fa0-9a24-f9f2-179cb0877210 [Follower]" leader= + testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.438+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: entering follower state: follower="Node at 6249adaf-7fa0-9a24-f9f2-179cb0877210 [Follower]" leader= +testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.438+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: aborting pipeline replication: peer="{Voter server-b6401cfe-e422-54c7-68c4-fa38095f64d8 b6401cfe-e422-54c7-68c4-fa38095f64d8}" + testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.438+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: aborting pipeline replication: peer="{Voter server-b6401cfe-e422-54c7-68c4-fa38095f64d8 b6401cfe-e422-54c7-68c4-fa38095f64d8}" +testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: pipelining replication: peer="{Voter server-6249adaf-7fa0-9a24-f9f2-179cb0877210 6249adaf-7fa0-9a24-f9f2-179cb0877210}" + testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.438+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: pipelining replication: peer="{Voter server-6249adaf-7fa0-9a24-f9f2-179cb0877210 6249adaf-7fa0-9a24-f9f2-179cb0877210}" +testLoggerAdapter verbose: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.439+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: aborting pipeline replication: peer="{Voter server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab 3837dac0-485c-2d3b-dd06-8c114f9ac0ab}" + testing.go:149: server-6249adaf-7fa0-9a24-f9f2-179cb0877210: 2023-05-16T04:06:20.439+1400 [INFO] server-6249adaf-7fa0-9a24-f9f2-179cb0877210: aborting pipeline replication: peer="{Voter server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab 3837dac0-485c-2d3b-dd06-8c114f9ac0ab}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.545+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.439133283 +1400 +14 m=+17.241714918 (1 nodes), 467.545µs from start of poll, 283.416164ms from cluster start. Timeout at 2023-05-16 04:06:20.545061653 +1400 +14 m=+17.347643288, 105.92837ms after stability + testing.go:149: cluster: 2023-05-16T04:06:20.545+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.439133283 +1400 +14 m=+17.241714918 (1 nodes), 467.545µs from start of poll, 283.416164ms from cluster start. Timeout at 2023-05-16 04:06:20.545061653 +1400 +14 m=+17.347643288, 105.92837ms after stability +testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.545+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: aborting pipeline replication: peer="{Voter server-b6401cfe-e422-54c7-68c4-fa38095f64d8 b6401cfe-e422-54c7-68c4-fa38095f64d8}" + testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.545+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: aborting pipeline replication: peer="{Voter server-b6401cfe-e422-54c7-68c4-fa38095f64d8 b6401cfe-e422-54c7-68c4-fa38095f64d8}" +testLoggerAdapter verbose: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.545+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: aborting pipeline replication: peer="{Voter server-6249adaf-7fa0-9a24-f9f2-179cb0877210 6249adaf-7fa0-9a24-f9f2-179cb0877210}" + testing.go:149: server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: 2023-05-16T04:06:20.545+1400 [INFO] server-3837dac0-485c-2d3b-dd06-8c114f9ac0ab: aborting pipeline replication: peer="{Voter server-6249adaf-7fa0-9a24-f9f2-179cb0877210 6249adaf-7fa0-9a24-f9f2-179cb0877210}" +--- PASS: TestRaft_LeadershipTransfer (0.39s) === RUN TestRaft_LeadershipTransferWithOneNode -testLoggerAdapter verbose: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.902-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9db2393c-0ada-1268-71ef-bf3e6fa8b556 Address:9db2393c-0ada-1268-71ef-bf3e6fa8b556}]" - testing.go:149: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.902-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9db2393c-0ada-1268-71ef-bf3e6fa8b556 Address:9db2393c-0ada-1268-71ef-bf3e6fa8b556}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:39.902-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:39.902-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.902-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: entering follower state: follower="Node at 9db2393c-0ada-1268-71ef-bf3e6fa8b556 [Follower]" leader= - testing.go:149: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.902-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: entering follower state: follower="Node at 9db2393c-0ada-1268-71ef-bf3e6fa8b556 [Follower]" leader= -testLoggerAdapter verbose: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.997-1200 [WARN] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.997-1200 [WARN] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.997-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: entering candidate state: node="Node at 9db2393c-0ada-1268-71ef-bf3e6fa8b556 [Candidate]" term=2 - testing.go:149: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.997-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: entering candidate state: node="Node at 9db2393c-0ada-1268-71ef-bf3e6fa8b556 [Candidate]" term=2 -testLoggerAdapter verbose: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.997-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: election won: tally=1 - testing.go:149: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.997-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: election won: tally=1 -testLoggerAdapter verbose: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.997-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: entering leader state: leader="Node at 9db2393c-0ada-1268-71ef-bf3e6fa8b556 [Leader]" - testing.go:149: server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: 2024-06-16T08:26:39.997-1200 [INFO] server-9db2393c-0ada-1268-71ef-bf3e6fa8b556: entering leader state: leader="Node at 9db2393c-0ada-1268-71ef-bf3e6fa8b556 [Leader]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.103-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.99765244 -1200 -12 m=+17.004968941 (1 nodes), 95.44664ms from start of poll, 95.81222ms from cluster start. Timeout at 2024-06-16 08:26:40.10362474 -1200 -12 m=+17.110941221, 105.97228ms after stability - testing.go:149: cluster: 2024-06-16T08:26:40.103-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:39.99765244 -1200 -12 m=+17.004968941 (1 nodes), 95.44664ms from start of poll, 95.81222ms from cluster start. Timeout at 2024-06-16 08:26:40.10362474 -1200 -12 m=+17.110941221, 105.97228ms after stability ---- PASS: TestRaft_LeadershipTransferWithOneNode (0.20s) +testLoggerAdapter verbose: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.546+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fe047cd-368f-42d7-8337-4924b08f6fe8 Address:2fe047cd-368f-42d7-8337-4924b08f6fe8}]" + testing.go:149: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.546+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fe047cd-368f-42d7-8337-4924b08f6fe8 Address:2fe047cd-368f-42d7-8337-4924b08f6fe8}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.546+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:20.546+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.546+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: entering follower state: follower="Node at 2fe047cd-368f-42d7-8337-4924b08f6fe8 [Follower]" leader= + testing.go:149: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.546+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: entering follower state: follower="Node at 2fe047cd-368f-42d7-8337-4924b08f6fe8 [Follower]" leader= +testLoggerAdapter verbose: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.621+1400 [WARN] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.621+1400 [WARN] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.621+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: entering candidate state: node="Node at 2fe047cd-368f-42d7-8337-4924b08f6fe8 [Candidate]" term=2 + testing.go:149: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.621+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: entering candidate state: node="Node at 2fe047cd-368f-42d7-8337-4924b08f6fe8 [Candidate]" term=2 +testLoggerAdapter verbose: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.621+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: election won: tally=1 + testing.go:149: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.621+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: election won: tally=1 +testLoggerAdapter verbose: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.621+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: entering leader state: leader="Node at 2fe047cd-368f-42d7-8337-4924b08f6fe8 [Leader]" + testing.go:149: server-2fe047cd-368f-42d7-8337-4924b08f6fe8: 2023-05-16T04:06:20.621+1400 [INFO] server-2fe047cd-368f-42d7-8337-4924b08f6fe8: entering leader state: leader="Node at 2fe047cd-368f-42d7-8337-4924b08f6fe8 [Leader]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.727+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.621952741 +1400 +14 m=+17.424534356 (1 nodes), 75.040446ms from start of poll, 75.458011ms from cluster start. Timeout at 2023-05-16 04:06:20.727542187 +1400 +14 m=+17.530124022, 105.589666ms after stability + testing.go:149: cluster: 2023-05-16T04:06:20.727+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.621952741 +1400 +14 m=+17.424534356 (1 nodes), 75.040446ms from start of poll, 75.458011ms from cluster start. Timeout at 2023-05-16 04:06:20.727542187 +1400 +14 m=+17.530124022, 105.589666ms after stability +--- PASS: TestRaft_LeadershipTransferWithOneNode (0.18s) === RUN TestRaft_LeadershipTransferWithSevenNodes -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.106-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.106-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.106-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: entering follower state: follower="Node at 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 [Follower]" leader= - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.106-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: entering follower state: follower="Node at 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 [Follower]" leader= -testLoggerAdapter verbose: server-1b19f367-68b0-83c5-a5af-93a15af4b137: 2024-06-16T08:26:40.106-1200 [INFO] server-1b19f367-68b0-83c5-a5af-93a15af4b137: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" - testing.go:149: server-1b19f367-68b0-83c5-a5af-93a15af4b137: 2024-06-16T08:26:40.106-1200 [INFO] server-1b19f367-68b0-83c5-a5af-93a15af4b137: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" -testLoggerAdapter verbose: server-1b19f367-68b0-83c5-a5af-93a15af4b137: 2024-06-16T08:26:40.106-1200 [INFO] server-1b19f367-68b0-83c5-a5af-93a15af4b137: entering follower state: follower="Node at 1b19f367-68b0-83c5-a5af-93a15af4b137 [Follower]" leader= - testing.go:149: server-1b19f367-68b0-83c5-a5af-93a15af4b137: 2024-06-16T08:26:40.106-1200 [INFO] server-1b19f367-68b0-83c5-a5af-93a15af4b137: entering follower state: follower="Node at 1b19f367-68b0-83c5-a5af-93a15af4b137 [Follower]" leader= -testLoggerAdapter verbose: server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16: 2024-06-16T08:26:40.107-1200 [INFO] server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" - testing.go:149: server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16: 2024-06-16T08:26:40.107-1200 [INFO] server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" -testLoggerAdapter verbose: server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16: 2024-06-16T08:26:40.107-1200 [INFO] server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16: entering follower state: follower="Node at d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 [Follower]" leader= - testing.go:149: server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16: 2024-06-16T08:26:40.107-1200 [INFO] server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16: entering follower state: follower="Node at d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 [Follower]" leader= -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.107-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.107-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.107-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: entering follower state: follower="Node at 3b81aed2-de30-b2fd-178b-c0547516bc6f [Follower]" leader= - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.107-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: entering follower state: follower="Node at 3b81aed2-de30-b2fd-178b-c0547516bc6f [Follower]" leader= -testLoggerAdapter verbose: server-adef2612-db49-5c2a-48bb-68bddcfa669c: 2024-06-16T08:26:40.107-1200 [INFO] server-adef2612-db49-5c2a-48bb-68bddcfa669c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" - testing.go:149: server-adef2612-db49-5c2a-48bb-68bddcfa669c: 2024-06-16T08:26:40.107-1200 [INFO] server-adef2612-db49-5c2a-48bb-68bddcfa669c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" -testLoggerAdapter verbose: server-adef2612-db49-5c2a-48bb-68bddcfa669c: 2024-06-16T08:26:40.107-1200 [INFO] server-adef2612-db49-5c2a-48bb-68bddcfa669c: entering follower state: follower="Node at adef2612-db49-5c2a-48bb-68bddcfa669c [Follower]" leader= - testing.go:149: server-adef2612-db49-5c2a-48bb-68bddcfa669c: 2024-06-16T08:26:40.107-1200 [INFO] server-adef2612-db49-5c2a-48bb-68bddcfa669c: entering follower state: follower="Node at adef2612-db49-5c2a-48bb-68bddcfa669c [Follower]" leader= -testLoggerAdapter verbose: server-7d2164a7-6111-1ca9-8553-6b9b14836751: 2024-06-16T08:26:40.107-1200 [INFO] server-7d2164a7-6111-1ca9-8553-6b9b14836751: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" - testing.go:149: server-7d2164a7-6111-1ca9-8553-6b9b14836751: 2024-06-16T08:26:40.107-1200 [INFO] server-7d2164a7-6111-1ca9-8553-6b9b14836751: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" -testLoggerAdapter verbose: server-7d2164a7-6111-1ca9-8553-6b9b14836751: 2024-06-16T08:26:40.108-1200 [INFO] server-7d2164a7-6111-1ca9-8553-6b9b14836751: entering follower state: follower="Node at 7d2164a7-6111-1ca9-8553-6b9b14836751 [Follower]" leader= - testing.go:149: server-7d2164a7-6111-1ca9-8553-6b9b14836751: 2024-06-16T08:26:40.108-1200 [INFO] server-7d2164a7-6111-1ca9-8553-6b9b14836751: entering follower state: follower="Node at 7d2164a7-6111-1ca9-8553-6b9b14836751 [Follower]" leader= -testLoggerAdapter verbose: server-7cc58c3c-784b-80b8-d215-af553a5127ea: 2024-06-16T08:26:40.108-1200 [INFO] server-7cc58c3c-784b-80b8-d215-af553a5127ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" - testing.go:149: server-7cc58c3c-784b-80b8-d215-af553a5127ea: 2024-06-16T08:26:40.108-1200 [INFO] server-7cc58c3c-784b-80b8-d215-af553a5127ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 Address:0d85ca6b-1d2e-8914-f33b-4313d3d15dd1} {Suffrage:Voter ID:server-1b19f367-68b0-83c5-a5af-93a15af4b137 Address:1b19f367-68b0-83c5-a5af-93a15af4b137} {Suffrage:Voter ID:server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 Address:d4bb2e10-f7ab-47a0-ac4b-fdc658518f16} {Suffrage:Voter ID:server-3b81aed2-de30-b2fd-178b-c0547516bc6f Address:3b81aed2-de30-b2fd-178b-c0547516bc6f} {Suffrage:Voter ID:server-adef2612-db49-5c2a-48bb-68bddcfa669c Address:adef2612-db49-5c2a-48bb-68bddcfa669c} {Suffrage:Voter ID:server-7d2164a7-6111-1ca9-8553-6b9b14836751 Address:7d2164a7-6111-1ca9-8553-6b9b14836751} {Suffrage:Voter ID:server-7cc58c3c-784b-80b8-d215-af553a5127ea Address:7cc58c3c-784b-80b8-d215-af553a5127ea}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.108-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:40.108-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-7cc58c3c-784b-80b8-d215-af553a5127ea: 2024-06-16T08:26:40.108-1200 [INFO] server-7cc58c3c-784b-80b8-d215-af553a5127ea: entering follower state: follower="Node at 7cc58c3c-784b-80b8-d215-af553a5127ea [Follower]" leader= - testing.go:149: server-7cc58c3c-784b-80b8-d215-af553a5127ea: 2024-06-16T08:26:40.108-1200 [INFO] server-7cc58c3c-784b-80b8-d215-af553a5127ea: entering follower state: follower="Node at 7cc58c3c-784b-80b8-d215-af553a5127ea [Follower]" leader= -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.158-1200 [WARN] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.158-1200 [WARN] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: entering candidate state: node="Node at 3b81aed2-de30-b2fd-178b-c0547516bc6f [Candidate]" term=2 - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: entering candidate state: node="Node at 3b81aed2-de30-b2fd-178b-c0547516bc6f [Candidate]" term=2 -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: election won: tally=4 - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: election won: tally=4 -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: entering leader state: leader="Node at 3b81aed2-de30-b2fd-178b-c0547516bc6f [Leader]" - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: entering leader state: leader="Node at 3b81aed2-de30-b2fd-178b-c0547516bc6f [Leader]" -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-1b19f367-68b0-83c5-a5af-93a15af4b137 - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-1b19f367-68b0-83c5-a5af-93a15af4b137 -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-adef2612-db49-5c2a-48bb-68bddcfa669c - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-adef2612-db49-5c2a-48bb-68bddcfa669c -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-7d2164a7-6111-1ca9-8553-6b9b14836751 - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-7d2164a7-6111-1ca9-8553-6b9b14836751 -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1}" - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1}" -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-1b19f367-68b0-83c5-a5af-93a15af4b137 1b19f367-68b0-83c5-a5af-93a15af4b137}" - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-1b19f367-68b0-83c5-a5af-93a15af4b137 1b19f367-68b0-83c5-a5af-93a15af4b137}" -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-7cc58c3c-784b-80b8-d215-af553a5127ea - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: added peer, starting replication: peer=server-7cc58c3c-784b-80b8-d215-af553a5127ea -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 d4bb2e10-f7ab-47a0-ac4b-fdc658518f16}" - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 d4bb2e10-f7ab-47a0-ac4b-fdc658518f16}" -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.160-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-7cc58c3c-784b-80b8-d215-af553a5127ea 7cc58c3c-784b-80b8-d215-af553a5127ea}" - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.160-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-7cc58c3c-784b-80b8-d215-af553a5127ea 7cc58c3c-784b-80b8-d215-af553a5127ea}" -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-adef2612-db49-5c2a-48bb-68bddcfa669c adef2612-db49-5c2a-48bb-68bddcfa669c}" - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-adef2612-db49-5c2a-48bb-68bddcfa669c adef2612-db49-5c2a-48bb-68bddcfa669c}" -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-7d2164a7-6111-1ca9-8553-6b9b14836751 7d2164a7-6111-1ca9-8553-6b9b14836751}" - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.159-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: pipelining replication: peer="{Voter server-7d2164a7-6111-1ca9-8553-6b9b14836751 7d2164a7-6111-1ca9-8553-6b9b14836751}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.264-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.15942808 -1200 -12 m=+17.166744561 (1 nodes), 50.9432ms from start of poll, 53.26924ms from cluster start. Timeout at 2024-06-16 08:26:40.26467756 -1200 -12 m=+17.271994021, 105.24946ms after stability - testing.go:149: cluster: 2024-06-16T08:26:40.264-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.15942808 -1200 -12 m=+17.166744561 (1 nodes), 50.9432ms from start of poll, 53.26924ms from cluster start. Timeout at 2024-06-16 08:26:40.26467756 -1200 -12 m=+17.271994021, 105.24946ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.264-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:40.264-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.370-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:40.26489708 -1200 -12 m=+17.272213561 (6 nodes), 5.08µs from start of poll, 158.73824ms from cluster start. Timeout at 2024-06-16 08:26:40.37024978 -1200 -12 m=+17.377566241, 105.35268ms after stability - testing.go:149: cluster: 2024-06-16T08:26:40.370-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:40.26489708 -1200 -12 m=+17.272213561 (6 nodes), 5.08µs from start of poll, 158.73824ms from cluster start. Timeout at 2024-06-16 08:26:40.37024978 -1200 -12 m=+17.377566241, 105.35268ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.370-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:40.370-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.476-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.3705117 -1200 -12 m=+17.377828181 (1 nodes), 2.08µs from start of poll, 264.35286ms from cluster start. Timeout at 2024-06-16 08:26:40.47643874 -1200 -12 m=+17.483755221, 105.92704ms after stability - testing.go:149: cluster: 2024-06-16T08:26:40.476-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.3705117 -1200 -12 m=+17.377828181 (1 nodes), 2.08µs from start of poll, 264.35286ms from cluster start. Timeout at 2024-06-16 08:26:40.47643874 -1200 -12 m=+17.483755221, 105.92704ms after stability -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.476-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: entering candidate state: node="Node at 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 [Candidate]" term=3 - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.476-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: entering candidate state: node="Node at 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 [Candidate]" term=3 -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.476-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:40.476-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.476-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: election won: tally=4 -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: entering follower state: follower="Node at 3b81aed2-de30-b2fd-178b-c0547516bc6f [Follower]" leader= - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.476-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: election won: tally=4 -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.476-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: entering leader state: leader="Node at 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 [Leader]" - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: entering follower state: follower="Node at 3b81aed2-de30-b2fd-178b-c0547516bc6f [Follower]" leader= -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-1b19f367-68b0-83c5-a5af-93a15af4b137 1b19f367-68b0-83c5-a5af-93a15af4b137}" - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.476-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: entering leader state: leader="Node at 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 [Leader]" -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-1b19f367-68b0-83c5-a5af-93a15af4b137 - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-1b19f367-68b0-83c5-a5af-93a15af4b137 - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-1b19f367-68b0-83c5-a5af-93a15af4b137 1b19f367-68b0-83c5-a5af-93a15af4b137}" -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-adef2612-db49-5c2a-48bb-68bddcfa669c adef2612-db49-5c2a-48bb-68bddcfa669c}" - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-adef2612-db49-5c2a-48bb-68bddcfa669c adef2612-db49-5c2a-48bb-68bddcfa669c}" -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-3b81aed2-de30-b2fd-178b-c0547516bc6f -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1}" - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-3b81aed2-de30-b2fd-178b-c0547516bc6f -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-adef2612-db49-5c2a-48bb-68bddcfa669c - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1 0d85ca6b-1d2e-8914-f33b-4313d3d15dd1}" - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-adef2612-db49-5c2a-48bb-68bddcfa669c -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 d4bb2e10-f7ab-47a0-ac4b-fdc658518f16}" -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-3b81aed2-de30-b2fd-178b-c0547516bc6f 3b81aed2-de30-b2fd-178b-c0547516bc6f}" - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 d4bb2e10-f7ab-47a0-ac4b-fdc658518f16}" - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-3b81aed2-de30-b2fd-178b-c0547516bc6f 3b81aed2-de30-b2fd-178b-c0547516bc6f}" -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-7d2164a7-6111-1ca9-8553-6b9b14836751 7d2164a7-6111-1ca9-8553-6b9b14836751}" - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-7d2164a7-6111-1ca9-8553-6b9b14836751 7d2164a7-6111-1ca9-8553-6b9b14836751}" -testLoggerAdapter verbose: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-7cc58c3c-784b-80b8-d215-af553a5127ea 7cc58c3c-784b-80b8-d215-af553a5127ea}" - testing.go:149: server-3b81aed2-de30-b2fd-178b-c0547516bc6f: 2024-06-16T08:26:40.476-1200 [INFO] server-3b81aed2-de30-b2fd-178b-c0547516bc6f: aborting pipeline replication: peer="{Voter server-7cc58c3c-784b-80b8-d215-af553a5127ea 7cc58c3c-784b-80b8-d215-af553a5127ea}" -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 d4bb2e10-f7ab-47a0-ac4b-fdc658518f16}" - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 d4bb2e10-f7ab-47a0-ac4b-fdc658518f16}" -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-7d2164a7-6111-1ca9-8553-6b9b14836751 - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-7d2164a7-6111-1ca9-8553-6b9b14836751 -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-7cc58c3c-784b-80b8-d215-af553a5127ea - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: added peer, starting replication: peer=server-7cc58c3c-784b-80b8-d215-af553a5127ea -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-1b19f367-68b0-83c5-a5af-93a15af4b137 1b19f367-68b0-83c5-a5af-93a15af4b137}" - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-1b19f367-68b0-83c5-a5af-93a15af4b137 1b19f367-68b0-83c5-a5af-93a15af4b137}" -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-adef2612-db49-5c2a-48bb-68bddcfa669c adef2612-db49-5c2a-48bb-68bddcfa669c}" - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-adef2612-db49-5c2a-48bb-68bddcfa669c adef2612-db49-5c2a-48bb-68bddcfa669c}" -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-7cc58c3c-784b-80b8-d215-af553a5127ea 7cc58c3c-784b-80b8-d215-af553a5127ea}" - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-7cc58c3c-784b-80b8-d215-af553a5127ea 7cc58c3c-784b-80b8-d215-af553a5127ea}" -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-7d2164a7-6111-1ca9-8553-6b9b14836751 7d2164a7-6111-1ca9-8553-6b9b14836751}" - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.477-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: pipelining replication: peer="{Voter server-7d2164a7-6111-1ca9-8553-6b9b14836751 7d2164a7-6111-1ca9-8553-6b9b14836751}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.582-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.47700508 -1200 -12 m=+17.484321561 (1 nodes), 133.66µs from start of poll, 370.84624ms from cluster start. Timeout at 2024-06-16 08:26:40.58212122 -1200 -12 m=+17.589437721, 105.11616ms after stability - testing.go:149: cluster: 2024-06-16T08:26:40.582-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.47700508 -1200 -12 m=+17.484321561 (1 nodes), 133.66µs from start of poll, 370.84624ms from cluster start. Timeout at 2024-06-16 08:26:40.58212122 -1200 -12 m=+17.589437721, 105.11616ms after stability -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-7cc58c3c-784b-80b8-d215-af553a5127ea 7cc58c3c-784b-80b8-d215-af553a5127ea}" - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-7cc58c3c-784b-80b8-d215-af553a5127ea 7cc58c3c-784b-80b8-d215-af553a5127ea}" -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-1b19f367-68b0-83c5-a5af-93a15af4b137 1b19f367-68b0-83c5-a5af-93a15af4b137}" - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-1b19f367-68b0-83c5-a5af-93a15af4b137 1b19f367-68b0-83c5-a5af-93a15af4b137}" -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-3b81aed2-de30-b2fd-178b-c0547516bc6f 3b81aed2-de30-b2fd-178b-c0547516bc6f}" - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-3b81aed2-de30-b2fd-178b-c0547516bc6f 3b81aed2-de30-b2fd-178b-c0547516bc6f}" -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 d4bb2e10-f7ab-47a0-ac4b-fdc658518f16}" - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-d4bb2e10-f7ab-47a0-ac4b-fdc658518f16 d4bb2e10-f7ab-47a0-ac4b-fdc658518f16}" -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-adef2612-db49-5c2a-48bb-68bddcfa669c adef2612-db49-5c2a-48bb-68bddcfa669c}" - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-adef2612-db49-5c2a-48bb-68bddcfa669c adef2612-db49-5c2a-48bb-68bddcfa669c}" -testLoggerAdapter verbose: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-7d2164a7-6111-1ca9-8553-6b9b14836751 7d2164a7-6111-1ca9-8553-6b9b14836751}" - testing.go:149: server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: 2024-06-16T08:26:40.582-1200 [INFO] server-0d85ca6b-1d2e-8914-f33b-4313d3d15dd1: aborting pipeline replication: peer="{Voter server-7d2164a7-6111-1ca9-8553-6b9b14836751 7d2164a7-6111-1ca9-8553-6b9b14836751}" ---- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.48s) +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:20.730+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:20.730+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" +testLoggerAdapter verbose: server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e: 2023-05-16T04:06:20.730+1400 [INFO] server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" + testing.go:149: server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e: 2023-05-16T04:06:20.730+1400 [INFO] server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" +testLoggerAdapter verbose: server-bc478a63-a6e6-28f5-2d04-37c72331e776: 2023-05-16T04:06:20.730+1400 [INFO] server-bc478a63-a6e6-28f5-2d04-37c72331e776: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" + testing.go:149: server-bc478a63-a6e6-28f5-2d04-37c72331e776: 2023-05-16T04:06:20.730+1400 [INFO] server-bc478a63-a6e6-28f5-2d04-37c72331e776: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" +testLoggerAdapter verbose: server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51: 2023-05-16T04:06:20.731+1400 [INFO] server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" + testing.go:149: server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51: 2023-05-16T04:06:20.731+1400 [INFO] server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" +testLoggerAdapter verbose: server-5b6bca9a-f4a0-7870-ba64-7f33be112495: 2023-05-16T04:06:20.731+1400 [INFO] server-5b6bca9a-f4a0-7870-ba64-7f33be112495: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" + testing.go:149: server-5b6bca9a-f4a0-7870-ba64-7f33be112495: 2023-05-16T04:06:20.731+1400 [INFO] server-5b6bca9a-f4a0-7870-ba64-7f33be112495: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.731+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.731+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" +testLoggerAdapter verbose: server-c9924079-5f27-61ca-26f7-4c35362c26f4: 2023-05-16T04:06:20.732+1400 [INFO] server-c9924079-5f27-61ca-26f7-4c35362c26f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" + testing.go:149: server-c9924079-5f27-61ca-26f7-4c35362c26f4: 2023-05-16T04:06:20.732+1400 [INFO] server-c9924079-5f27-61ca-26f7-4c35362c26f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e Address:a0c76ffd-41d1-7f5c-9488-fb5e2249d30e} {Suffrage:Voter ID:server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e Address:b81d5f4b-ba2e-5649-2138-d348eeb0f60e} {Suffrage:Voter ID:server-bc478a63-a6e6-28f5-2d04-37c72331e776 Address:bc478a63-a6e6-28f5-2d04-37c72331e776} {Suffrage:Voter ID:server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 Address:dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51} {Suffrage:Voter ID:server-5b6bca9a-f4a0-7870-ba64-7f33be112495 Address:5b6bca9a-f4a0-7870-ba64-7f33be112495} {Suffrage:Voter ID:server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e Address:5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e} {Suffrage:Voter ID:server-c9924079-5f27-61ca-26f7-4c35362c26f4 Address:c9924079-5f27-61ca-26f7-4c35362c26f4}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.732+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:20.732+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51: 2023-05-16T04:06:20.732+1400 [INFO] server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51: entering follower state: follower="Node at dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 [Follower]" leader= + testing.go:149: server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51: 2023-05-16T04:06:20.732+1400 [INFO] server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51: entering follower state: follower="Node at dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 [Follower]" leader= +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:20.732+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: entering follower state: follower="Node at a0c76ffd-41d1-7f5c-9488-fb5e2249d30e [Follower]" leader= + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:20.732+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: entering follower state: follower="Node at a0c76ffd-41d1-7f5c-9488-fb5e2249d30e [Follower]" leader= +testLoggerAdapter verbose: server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e: 2023-05-16T04:06:20.732+1400 [INFO] server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e: entering follower state: follower="Node at b81d5f4b-ba2e-5649-2138-d348eeb0f60e [Follower]" leader= + testing.go:149: server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e: 2023-05-16T04:06:20.732+1400 [INFO] server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e: entering follower state: follower="Node at b81d5f4b-ba2e-5649-2138-d348eeb0f60e [Follower]" leader= +testLoggerAdapter verbose: server-bc478a63-a6e6-28f5-2d04-37c72331e776: 2023-05-16T04:06:20.732+1400 [INFO] server-bc478a63-a6e6-28f5-2d04-37c72331e776: entering follower state: follower="Node at bc478a63-a6e6-28f5-2d04-37c72331e776 [Follower]" leader= + testing.go:149: server-bc478a63-a6e6-28f5-2d04-37c72331e776: 2023-05-16T04:06:20.732+1400 [INFO] server-bc478a63-a6e6-28f5-2d04-37c72331e776: entering follower state: follower="Node at bc478a63-a6e6-28f5-2d04-37c72331e776 [Follower]" leader= +testLoggerAdapter verbose: server-5b6bca9a-f4a0-7870-ba64-7f33be112495: 2023-05-16T04:06:20.732+1400 [INFO] server-5b6bca9a-f4a0-7870-ba64-7f33be112495: entering follower state: follower="Node at 5b6bca9a-f4a0-7870-ba64-7f33be112495 [Follower]" leader= + testing.go:149: server-5b6bca9a-f4a0-7870-ba64-7f33be112495: 2023-05-16T04:06:20.732+1400 [INFO] server-5b6bca9a-f4a0-7870-ba64-7f33be112495: entering follower state: follower="Node at 5b6bca9a-f4a0-7870-ba64-7f33be112495 [Follower]" leader= +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.732+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: entering follower state: follower="Node at 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e [Follower]" leader= + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.732+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: entering follower state: follower="Node at 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e [Follower]" leader= +testLoggerAdapter verbose: server-c9924079-5f27-61ca-26f7-4c35362c26f4: 2023-05-16T04:06:20.733+1400 [INFO] server-c9924079-5f27-61ca-26f7-4c35362c26f4: entering follower state: follower="Node at c9924079-5f27-61ca-26f7-4c35362c26f4 [Follower]" leader= + testing.go:149: server-c9924079-5f27-61ca-26f7-4c35362c26f4: 2023-05-16T04:06:20.733+1400 [INFO] server-c9924079-5f27-61ca-26f7-4c35362c26f4: entering follower state: follower="Node at c9924079-5f27-61ca-26f7-4c35362c26f4 [Follower]" leader= +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.784+1400 [WARN] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.784+1400 [WARN] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.784+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: entering candidate state: node="Node at 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e [Candidate]" term=2 + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.784+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: entering candidate state: node="Node at 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e [Candidate]" term=2 +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: election won: tally=4 + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: election won: tally=4 +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: entering leader state: leader="Node at 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e [Leader]" + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: entering leader state: leader="Node at 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e [Leader]" +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-bc478a63-a6e6-28f5-2d04-37c72331e776 + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-bc478a63-a6e6-28f5-2d04-37c72331e776 +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-5b6bca9a-f4a0-7870-ba64-7f33be112495 + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-5b6bca9a-f4a0-7870-ba64-7f33be112495 +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-c9924079-5f27-61ca-26f7-4c35362c26f4 + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: added peer, starting replication: peer=server-c9924079-5f27-61ca-26f7-4c35362c26f4 +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-c9924079-5f27-61ca-26f7-4c35362c26f4 c9924079-5f27-61ca-26f7-4c35362c26f4}" + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.785+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-c9924079-5f27-61ca-26f7-4c35362c26f4 c9924079-5f27-61ca-26f7-4c35362c26f4}" +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e a0c76ffd-41d1-7f5c-9488-fb5e2249d30e}" + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e a0c76ffd-41d1-7f5c-9488-fb5e2249d30e}" +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e b81d5f4b-ba2e-5649-2138-d348eeb0f60e}" + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e b81d5f4b-ba2e-5649-2138-d348eeb0f60e}" +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-bc478a63-a6e6-28f5-2d04-37c72331e776 bc478a63-a6e6-28f5-2d04-37c72331e776}" + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-bc478a63-a6e6-28f5-2d04-37c72331e776 bc478a63-a6e6-28f5-2d04-37c72331e776}" +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51}" + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51}" +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-5b6bca9a-f4a0-7870-ba64-7f33be112495 5b6bca9a-f4a0-7870-ba64-7f33be112495}" + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:20.786+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: pipelining replication: peer="{Voter server-5b6bca9a-f4a0-7870-ba64-7f33be112495 5b6bca9a-f4a0-7870-ba64-7f33be112495}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.891+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.786153778 +1400 +14 m=+17.588735413 (1 nodes), 52.726162ms from start of poll, 56.512586ms from cluster start. Timeout at 2023-05-16 04:06:20.891425021 +1400 +14 m=+17.694006656, 105.271243ms after stability + testing.go:149: cluster: 2023-05-16T04:06:20.891+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.786153778 +1400 +14 m=+17.588735413 (1 nodes), 52.726162ms from start of poll, 56.512586ms from cluster start. Timeout at 2023-05-16 04:06:20.891425021 +1400 +14 m=+17.694006656, 105.271243ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.891+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:20.891+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.997+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:20.891784305 +1400 +14 m=+17.694365940 (6 nodes), 7.2µs from start of poll, 162.143113ms from cluster start. Timeout at 2023-05-16 04:06:20.997572193 +1400 +14 m=+17.800153808, 105.787868ms after stability + testing.go:149: cluster: 2023-05-16T04:06:20.997+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:20.891784305 +1400 +14 m=+17.694365940 (6 nodes), 7.2µs from start of poll, 162.143113ms from cluster start. Timeout at 2023-05-16 04:06:20.997572193 +1400 +14 m=+17.800153808, 105.787868ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:20.997+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:20.997+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.103+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.997866677 +1400 +14 m=+17.800448292 (1 nodes), 3.9µs from start of poll, 268.225465ms from cluster start. Timeout at 2023-05-16 04:06:21.102998178 +1400 +14 m=+17.905579813, 105.131521ms after stability + testing.go:149: cluster: 2023-05-16T04:06:21.103+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:20.997866677 +1400 +14 m=+17.800448292 (1 nodes), 3.9µs from start of poll, 268.225465ms from cluster start. Timeout at 2023-05-16 04:06:21.102998178 +1400 +14 m=+17.905579813, 105.131521ms after stability +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.103+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: entering candidate state: node="Node at a0c76ffd-41d1-7f5c-9488-fb5e2249d30e [Candidate]" term=3 + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.103+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: entering candidate state: node="Node at a0c76ffd-41d1-7f5c-9488-fb5e2249d30e [Candidate]" term=3 +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.103+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:21.103+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.103+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: election won: tally=4 + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.103+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: election won: tally=4 +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.103+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: entering leader state: leader="Node at a0c76ffd-41d1-7f5c-9488-fb5e2249d30e [Leader]" + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.103+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: entering leader state: leader="Node at a0c76ffd-41d1-7f5c-9488-fb5e2249d30e [Leader]" +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.103+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.103+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-bc478a63-a6e6-28f5-2d04-37c72331e776 + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-bc478a63-a6e6-28f5-2d04-37c72331e776 +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-5b6bca9a-f4a0-7870-ba64-7f33be112495 + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-5b6bca9a-f4a0-7870-ba64-7f33be112495 +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-c9924079-5f27-61ca-26f7-4c35362c26f4 + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: added peer, starting replication: peer=server-c9924079-5f27-61ca-26f7-4c35362c26f4 +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-c9924079-5f27-61ca-26f7-4c35362c26f4 c9924079-5f27-61ca-26f7-4c35362c26f4}" + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.104+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-c9924079-5f27-61ca-26f7-4c35362c26f4 c9924079-5f27-61ca-26f7-4c35362c26f4}" +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.104+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: entering follower state: follower="Node at 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e [Follower]" leader= + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.104+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: entering follower state: follower="Node at 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e [Follower]" leader= +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.104+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-c9924079-5f27-61ca-26f7-4c35362c26f4 c9924079-5f27-61ca-26f7-4c35362c26f4}" + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.104+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-c9924079-5f27-61ca-26f7-4c35362c26f4 c9924079-5f27-61ca-26f7-4c35362c26f4}" +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e b81d5f4b-ba2e-5649-2138-d348eeb0f60e}" + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e b81d5f4b-ba2e-5649-2138-d348eeb0f60e}" +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-bc478a63-a6e6-28f5-2d04-37c72331e776 bc478a63-a6e6-28f5-2d04-37c72331e776}" + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-bc478a63-a6e6-28f5-2d04-37c72331e776 bc478a63-a6e6-28f5-2d04-37c72331e776}" +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51}" + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51}" +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-5b6bca9a-f4a0-7870-ba64-7f33be112495 5b6bca9a-f4a0-7870-ba64-7f33be112495}" + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-5b6bca9a-f4a0-7870-ba64-7f33be112495 5b6bca9a-f4a0-7870-ba64-7f33be112495}" +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e}" + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.105+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: pipelining replication: peer="{Voter server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e}" +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e a0c76ffd-41d1-7f5c-9488-fb5e2249d30e}" + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e a0c76ffd-41d1-7f5c-9488-fb5e2249d30e}" +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e b81d5f4b-ba2e-5649-2138-d348eeb0f60e}" + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e b81d5f4b-ba2e-5649-2138-d348eeb0f60e}" +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-bc478a63-a6e6-28f5-2d04-37c72331e776 bc478a63-a6e6-28f5-2d04-37c72331e776}" + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-bc478a63-a6e6-28f5-2d04-37c72331e776 bc478a63-a6e6-28f5-2d04-37c72331e776}" +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51}" + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51}" +testLoggerAdapter verbose: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-5b6bca9a-f4a0-7870-ba64-7f33be112495 5b6bca9a-f4a0-7870-ba64-7f33be112495}" + testing.go:149: server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: 2023-05-16T04:06:21.106+1400 [INFO] server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e: aborting pipeline replication: peer="{Voter server-5b6bca9a-f4a0-7870-ba64-7f33be112495 5b6bca9a-f4a0-7870-ba64-7f33be112495}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.210+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:21.105051342 +1400 +14 m=+17.907632977 (1 nodes), 1.332976ms from start of poll, 375.41015ms from cluster start. Timeout at 2023-05-16 04:06:21.210573427 +1400 +14 m=+18.013155042, 105.522065ms after stability + testing.go:149: cluster: 2023-05-16T04:06:21.210+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:21.105051342 +1400 +14 m=+17.907632977 (1 nodes), 1.332976ms from start of poll, 375.41015ms from cluster start. Timeout at 2023-05-16 04:06:21.210573427 +1400 +14 m=+18.013155042, 105.522065ms after stability +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.210+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-bc478a63-a6e6-28f5-2d04-37c72331e776 bc478a63-a6e6-28f5-2d04-37c72331e776}" + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.210+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-bc478a63-a6e6-28f5-2d04-37c72331e776 bc478a63-a6e6-28f5-2d04-37c72331e776}" +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51}" + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51 dd0db5f5-7552-fa7c-8a96-9e57d0ea8e51}" +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-5b6bca9a-f4a0-7870-ba64-7f33be112495 5b6bca9a-f4a0-7870-ba64-7f33be112495}" + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-5b6bca9a-f4a0-7870-ba64-7f33be112495 5b6bca9a-f4a0-7870-ba64-7f33be112495}" +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e}" + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e 5b6711ac-fc92-3c9a-f8b0-8a086b6f2f1e}" +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-c9924079-5f27-61ca-26f7-4c35362c26f4 c9924079-5f27-61ca-26f7-4c35362c26f4}" + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-c9924079-5f27-61ca-26f7-4c35362c26f4 c9924079-5f27-61ca-26f7-4c35362c26f4}" +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e b81d5f4b-ba2e-5649-2138-d348eeb0f60e}" + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.211+1400 [INFO] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: aborting pipeline replication: peer="{Voter server-b81d5f4b-ba2e-5649-2138-d348eeb0f60e b81d5f4b-ba2e-5649-2138-d348eeb0f60e}" +testLoggerAdapter verbose: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.261+1400 [ERROR] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: failed to heartbeat to: peer=b81d5f4b-ba2e-5649-2138-d348eeb0f60e error="command timed out" + testing.go:149: server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: 2023-05-16T04:06:21.261+1400 [ERROR] server-a0c76ffd-41d1-7f5c-9488-fb5e2249d30e: failed to heartbeat to: peer=b81d5f4b-ba2e-5649-2138-d348eeb0f60e error="command timed out" +--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.53s) === RUN TestRaft_LeadershipTransferToInvalidID -testLoggerAdapter verbose: server-24e88ed1-84f4-62d8-524a-b72536736910: 2024-06-16T08:26:40.584-1200 [INFO] server-24e88ed1-84f4-62d8-524a-b72536736910: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24e88ed1-84f4-62d8-524a-b72536736910 Address:24e88ed1-84f4-62d8-524a-b72536736910} {Suffrage:Voter ID:server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9 Address:30790dfd-c99d-3c14-0c0d-d26bf16a4de9} {Suffrage:Voter ID:server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 Address:92a4c800-1d35-1df7-2f45-bb26af00e0d0}]" - testing.go:149: server-24e88ed1-84f4-62d8-524a-b72536736910: 2024-06-16T08:26:40.584-1200 [INFO] server-24e88ed1-84f4-62d8-524a-b72536736910: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24e88ed1-84f4-62d8-524a-b72536736910 Address:24e88ed1-84f4-62d8-524a-b72536736910} {Suffrage:Voter ID:server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9 Address:30790dfd-c99d-3c14-0c0d-d26bf16a4de9} {Suffrage:Voter ID:server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 Address:92a4c800-1d35-1df7-2f45-bb26af00e0d0}]" -testLoggerAdapter verbose: server-24e88ed1-84f4-62d8-524a-b72536736910: 2024-06-16T08:26:40.584-1200 [INFO] server-24e88ed1-84f4-62d8-524a-b72536736910: entering follower state: follower="Node at 24e88ed1-84f4-62d8-524a-b72536736910 [Follower]" leader= - testing.go:149: server-24e88ed1-84f4-62d8-524a-b72536736910: 2024-06-16T08:26:40.584-1200 [INFO] server-24e88ed1-84f4-62d8-524a-b72536736910: entering follower state: follower="Node at 24e88ed1-84f4-62d8-524a-b72536736910 [Follower]" leader= -testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.584-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24e88ed1-84f4-62d8-524a-b72536736910 Address:24e88ed1-84f4-62d8-524a-b72536736910} {Suffrage:Voter ID:server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9 Address:30790dfd-c99d-3c14-0c0d-d26bf16a4de9} {Suffrage:Voter ID:server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 Address:92a4c800-1d35-1df7-2f45-bb26af00e0d0}]" - testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.584-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24e88ed1-84f4-62d8-524a-b72536736910 Address:24e88ed1-84f4-62d8-524a-b72536736910} {Suffrage:Voter ID:server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9 Address:30790dfd-c99d-3c14-0c0d-d26bf16a4de9} {Suffrage:Voter ID:server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 Address:92a4c800-1d35-1df7-2f45-bb26af00e0d0}]" -testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.584-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: entering follower state: follower="Node at 30790dfd-c99d-3c14-0c0d-d26bf16a4de9 [Follower]" leader= - testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.584-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: entering follower state: follower="Node at 30790dfd-c99d-3c14-0c0d-d26bf16a4de9 [Follower]" leader= -testLoggerAdapter verbose: server-92a4c800-1d35-1df7-2f45-bb26af00e0d0: 2024-06-16T08:26:40.584-1200 [INFO] server-92a4c800-1d35-1df7-2f45-bb26af00e0d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24e88ed1-84f4-62d8-524a-b72536736910 Address:24e88ed1-84f4-62d8-524a-b72536736910} {Suffrage:Voter ID:server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9 Address:30790dfd-c99d-3c14-0c0d-d26bf16a4de9} {Suffrage:Voter ID:server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 Address:92a4c800-1d35-1df7-2f45-bb26af00e0d0}]" - testing.go:149: server-92a4c800-1d35-1df7-2f45-bb26af00e0d0: 2024-06-16T08:26:40.584-1200 [INFO] server-92a4c800-1d35-1df7-2f45-bb26af00e0d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24e88ed1-84f4-62d8-524a-b72536736910 Address:24e88ed1-84f4-62d8-524a-b72536736910} {Suffrage:Voter ID:server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9 Address:30790dfd-c99d-3c14-0c0d-d26bf16a4de9} {Suffrage:Voter ID:server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 Address:92a4c800-1d35-1df7-2f45-bb26af00e0d0}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.584-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:40.584-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-92a4c800-1d35-1df7-2f45-bb26af00e0d0: 2024-06-16T08:26:40.584-1200 [INFO] server-92a4c800-1d35-1df7-2f45-bb26af00e0d0: entering follower state: follower="Node at 92a4c800-1d35-1df7-2f45-bb26af00e0d0 [Follower]" leader= - testing.go:149: server-92a4c800-1d35-1df7-2f45-bb26af00e0d0: 2024-06-16T08:26:40.584-1200 [INFO] server-92a4c800-1d35-1df7-2f45-bb26af00e0d0: entering follower state: follower="Node at 92a4c800-1d35-1df7-2f45-bb26af00e0d0 [Follower]" leader= -testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [WARN] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [WARN] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: entering candidate state: node="Node at 30790dfd-c99d-3c14-0c0d-d26bf16a4de9 [Candidate]" term=2 - testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: entering candidate state: node="Node at 30790dfd-c99d-3c14-0c0d-d26bf16a4de9 [Candidate]" term=2 -testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: election won: tally=2 - testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: election won: tally=2 -testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: entering leader state: leader="Node at 30790dfd-c99d-3c14-0c0d-d26bf16a4de9 [Leader]" - testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: entering leader state: leader="Node at 30790dfd-c99d-3c14-0c0d-d26bf16a4de9 [Leader]" -testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: added peer, starting replication: peer=server-24e88ed1-84f4-62d8-524a-b72536736910 - testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: added peer, starting replication: peer=server-24e88ed1-84f4-62d8-524a-b72536736910 -testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: added peer, starting replication: peer=server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 - testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: added peer, starting replication: peer=server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 -testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: pipelining replication: peer="{Voter server-24e88ed1-84f4-62d8-524a-b72536736910 24e88ed1-84f4-62d8-524a-b72536736910}" - testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: pipelining replication: peer="{Voter server-24e88ed1-84f4-62d8-524a-b72536736910 24e88ed1-84f4-62d8-524a-b72536736910}" -testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: pipelining replication: peer="{Voter server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 92a4c800-1d35-1df7-2f45-bb26af00e0d0}" - testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.637-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: pipelining replication: peer="{Voter server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 92a4c800-1d35-1df7-2f45-bb26af00e0d0}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.743-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.63782152 -1200 -12 m=+17.645138001 (1 nodes), 52.85372ms from start of poll, 53.68132ms from cluster start. Timeout at 2024-06-16 08:26:40.74379874 -1200 -12 m=+17.751115221, 105.97722ms after stability - testing.go:149: cluster: 2024-06-16T08:26:40.743-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.63782152 -1200 -12 m=+17.645138001 (1 nodes), 52.85372ms from start of poll, 53.68132ms from cluster start. Timeout at 2024-06-16 08:26:40.74379874 -1200 -12 m=+17.751115221, 105.97722ms after stability -testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.744-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: aborting pipeline replication: peer="{Voter server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 92a4c800-1d35-1df7-2f45-bb26af00e0d0}" - testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.744-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: aborting pipeline replication: peer="{Voter server-92a4c800-1d35-1df7-2f45-bb26af00e0d0 92a4c800-1d35-1df7-2f45-bb26af00e0d0}" -testLoggerAdapter verbose: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.744-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: aborting pipeline replication: peer="{Voter server-24e88ed1-84f4-62d8-524a-b72536736910 24e88ed1-84f4-62d8-524a-b72536736910}" - testing.go:149: server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: 2024-06-16T08:26:40.744-1200 [INFO] server-30790dfd-c99d-3c14-0c0d-d26bf16a4de9: aborting pipeline replication: peer="{Voter server-24e88ed1-84f4-62d8-524a-b72536736910 24e88ed1-84f4-62d8-524a-b72536736910}" ---- PASS: TestRaft_LeadershipTransferToInvalidID (0.16s) +testLoggerAdapter verbose: server-c3a2c95d-cbc0-7303-8574-ba9850b6768c: 2023-05-16T04:06:21.263+1400 [INFO] server-c3a2c95d-cbc0-7303-8574-ba9850b6768c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3a2c95d-cbc0-7303-8574-ba9850b6768c Address:c3a2c95d-cbc0-7303-8574-ba9850b6768c} {Suffrage:Voter ID:server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c Address:fb817ebb-3ee6-75da-af44-e887bc6dbc8c} {Suffrage:Voter ID:server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 Address:301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}]" + testing.go:149: server-c3a2c95d-cbc0-7303-8574-ba9850b6768c: 2023-05-16T04:06:21.263+1400 [INFO] server-c3a2c95d-cbc0-7303-8574-ba9850b6768c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3a2c95d-cbc0-7303-8574-ba9850b6768c Address:c3a2c95d-cbc0-7303-8574-ba9850b6768c} {Suffrage:Voter ID:server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c Address:fb817ebb-3ee6-75da-af44-e887bc6dbc8c} {Suffrage:Voter ID:server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 Address:301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}]" +testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.263+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3a2c95d-cbc0-7303-8574-ba9850b6768c Address:c3a2c95d-cbc0-7303-8574-ba9850b6768c} {Suffrage:Voter ID:server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c Address:fb817ebb-3ee6-75da-af44-e887bc6dbc8c} {Suffrage:Voter ID:server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 Address:301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}]" + testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.263+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3a2c95d-cbc0-7303-8574-ba9850b6768c Address:c3a2c95d-cbc0-7303-8574-ba9850b6768c} {Suffrage:Voter ID:server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c Address:fb817ebb-3ee6-75da-af44-e887bc6dbc8c} {Suffrage:Voter ID:server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 Address:301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}]" +testLoggerAdapter verbose: server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238: 2023-05-16T04:06:21.264+1400 [INFO] server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3a2c95d-cbc0-7303-8574-ba9850b6768c Address:c3a2c95d-cbc0-7303-8574-ba9850b6768c} {Suffrage:Voter ID:server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c Address:fb817ebb-3ee6-75da-af44-e887bc6dbc8c} {Suffrage:Voter ID:server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 Address:301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}]" + testing.go:149: server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238: 2023-05-16T04:06:21.264+1400 [INFO] server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3a2c95d-cbc0-7303-8574-ba9850b6768c Address:c3a2c95d-cbc0-7303-8574-ba9850b6768c} {Suffrage:Voter ID:server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c Address:fb817ebb-3ee6-75da-af44-e887bc6dbc8c} {Suffrage:Voter ID:server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 Address:301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.264+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:21.264+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-c3a2c95d-cbc0-7303-8574-ba9850b6768c: 2023-05-16T04:06:21.264+1400 [INFO] server-c3a2c95d-cbc0-7303-8574-ba9850b6768c: entering follower state: follower="Node at c3a2c95d-cbc0-7303-8574-ba9850b6768c [Follower]" leader= + testing.go:149: server-c3a2c95d-cbc0-7303-8574-ba9850b6768c: 2023-05-16T04:06:21.264+1400 [INFO] server-c3a2c95d-cbc0-7303-8574-ba9850b6768c: entering follower state: follower="Node at c3a2c95d-cbc0-7303-8574-ba9850b6768c [Follower]" leader= +testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.264+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: entering follower state: follower="Node at fb817ebb-3ee6-75da-af44-e887bc6dbc8c [Follower]" leader= + testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.264+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: entering follower state: follower="Node at fb817ebb-3ee6-75da-af44-e887bc6dbc8c [Follower]" leader= +testLoggerAdapter verbose: server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238: 2023-05-16T04:06:21.264+1400 [INFO] server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238: entering follower state: follower="Node at 301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 [Follower]" leader= + testing.go:149: server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238: 2023-05-16T04:06:21.264+1400 [INFO] server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238: entering follower state: follower="Node at 301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 [Follower]" leader= +testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [WARN] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [WARN] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: entering candidate state: node="Node at fb817ebb-3ee6-75da-af44-e887bc6dbc8c [Candidate]" term=2 + testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: entering candidate state: node="Node at fb817ebb-3ee6-75da-af44-e887bc6dbc8c [Candidate]" term=2 +testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: election won: tally=2 + testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: election won: tally=2 +testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: entering leader state: leader="Node at fb817ebb-3ee6-75da-af44-e887bc6dbc8c [Leader]" + testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: entering leader state: leader="Node at fb817ebb-3ee6-75da-af44-e887bc6dbc8c [Leader]" +testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: added peer, starting replication: peer=server-c3a2c95d-cbc0-7303-8574-ba9850b6768c + testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: added peer, starting replication: peer=server-c3a2c95d-cbc0-7303-8574-ba9850b6768c +testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: added peer, starting replication: peer=server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 + testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.324+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: added peer, starting replication: peer=server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 +testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.325+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: pipelining replication: peer="{Voter server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}" + testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.325+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: pipelining replication: peer="{Voter server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}" +testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.325+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: pipelining replication: peer="{Voter server-c3a2c95d-cbc0-7303-8574-ba9850b6768c c3a2c95d-cbc0-7303-8574-ba9850b6768c}" + testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.325+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: pipelining replication: peer="{Voter server-c3a2c95d-cbc0-7303-8574-ba9850b6768c c3a2c95d-cbc0-7303-8574-ba9850b6768c}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.430+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:21.325458723 +1400 +14 m=+18.128040338 (1 nodes), 61.03042ms from start of poll, 62.077193ms from cluster start. Timeout at 2023-05-16 04:06:21.430471862 +1400 +14 m=+18.233053497, 105.013159ms after stability + testing.go:149: cluster: 2023-05-16T04:06:21.430+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:21.325458723 +1400 +14 m=+18.128040338 (1 nodes), 61.03042ms from start of poll, 62.077193ms from cluster start. Timeout at 2023-05-16 04:06:21.430471862 +1400 +14 m=+18.233053497, 105.013159ms after stability +testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.430+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: aborting pipeline replication: peer="{Voter server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}" + testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.430+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: aborting pipeline replication: peer="{Voter server-301379f5-2a2b-5f4a-cdeb-7a9cac5c9238 301379f5-2a2b-5f4a-cdeb-7a9cac5c9238}" +testLoggerAdapter verbose: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.431+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: aborting pipeline replication: peer="{Voter server-c3a2c95d-cbc0-7303-8574-ba9850b6768c c3a2c95d-cbc0-7303-8574-ba9850b6768c}" + testing.go:149: server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: 2023-05-16T04:06:21.431+1400 [INFO] server-fb817ebb-3ee6-75da-af44-e887bc6dbc8c: aborting pipeline replication: peer="{Voter server-c3a2c95d-cbc0-7303-8574-ba9850b6768c c3a2c95d-cbc0-7303-8574-ba9850b6768c}" +--- PASS: TestRaft_LeadershipTransferToInvalidID (0.17s) === RUN TestRaft_LeadershipTransferToInvalidAddress -testLoggerAdapter verbose: server-cd678b2b-104a-f4db-7c80-817e7a97742e: 2024-06-16T08:26:40.745-1200 [INFO] server-cd678b2b-104a-f4db-7c80-817e7a97742e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd678b2b-104a-f4db-7c80-817e7a97742e Address:cd678b2b-104a-f4db-7c80-817e7a97742e} {Suffrage:Voter ID:server-be344f10-26a5-ee75-bc9d-aa2e88ff747d Address:be344f10-26a5-ee75-bc9d-aa2e88ff747d} {Suffrage:Voter ID:server-08c1dfe3-2202-3a00-1e95-813d64ccb811 Address:08c1dfe3-2202-3a00-1e95-813d64ccb811}]" - testing.go:149: server-cd678b2b-104a-f4db-7c80-817e7a97742e: 2024-06-16T08:26:40.745-1200 [INFO] server-cd678b2b-104a-f4db-7c80-817e7a97742e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd678b2b-104a-f4db-7c80-817e7a97742e Address:cd678b2b-104a-f4db-7c80-817e7a97742e} {Suffrage:Voter ID:server-be344f10-26a5-ee75-bc9d-aa2e88ff747d Address:be344f10-26a5-ee75-bc9d-aa2e88ff747d} {Suffrage:Voter ID:server-08c1dfe3-2202-3a00-1e95-813d64ccb811 Address:08c1dfe3-2202-3a00-1e95-813d64ccb811}]" -testLoggerAdapter verbose: server-cd678b2b-104a-f4db-7c80-817e7a97742e: 2024-06-16T08:26:40.745-1200 [INFO] server-cd678b2b-104a-f4db-7c80-817e7a97742e: entering follower state: follower="Node at cd678b2b-104a-f4db-7c80-817e7a97742e [Follower]" leader= - testing.go:149: server-cd678b2b-104a-f4db-7c80-817e7a97742e: 2024-06-16T08:26:40.745-1200 [INFO] server-cd678b2b-104a-f4db-7c80-817e7a97742e: entering follower state: follower="Node at cd678b2b-104a-f4db-7c80-817e7a97742e [Follower]" leader= -testLoggerAdapter verbose: server-be344f10-26a5-ee75-bc9d-aa2e88ff747d: 2024-06-16T08:26:40.745-1200 [INFO] server-be344f10-26a5-ee75-bc9d-aa2e88ff747d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd678b2b-104a-f4db-7c80-817e7a97742e Address:cd678b2b-104a-f4db-7c80-817e7a97742e} {Suffrage:Voter ID:server-be344f10-26a5-ee75-bc9d-aa2e88ff747d Address:be344f10-26a5-ee75-bc9d-aa2e88ff747d} {Suffrage:Voter ID:server-08c1dfe3-2202-3a00-1e95-813d64ccb811 Address:08c1dfe3-2202-3a00-1e95-813d64ccb811}]" - testing.go:149: server-be344f10-26a5-ee75-bc9d-aa2e88ff747d: 2024-06-16T08:26:40.745-1200 [INFO] server-be344f10-26a5-ee75-bc9d-aa2e88ff747d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd678b2b-104a-f4db-7c80-817e7a97742e Address:cd678b2b-104a-f4db-7c80-817e7a97742e} {Suffrage:Voter ID:server-be344f10-26a5-ee75-bc9d-aa2e88ff747d Address:be344f10-26a5-ee75-bc9d-aa2e88ff747d} {Suffrage:Voter ID:server-08c1dfe3-2202-3a00-1e95-813d64ccb811 Address:08c1dfe3-2202-3a00-1e95-813d64ccb811}]" -testLoggerAdapter verbose: server-be344f10-26a5-ee75-bc9d-aa2e88ff747d: 2024-06-16T08:26:40.745-1200 [INFO] server-be344f10-26a5-ee75-bc9d-aa2e88ff747d: entering follower state: follower="Node at be344f10-26a5-ee75-bc9d-aa2e88ff747d [Follower]" leader= - testing.go:149: server-be344f10-26a5-ee75-bc9d-aa2e88ff747d: 2024-06-16T08:26:40.745-1200 [INFO] server-be344f10-26a5-ee75-bc9d-aa2e88ff747d: entering follower state: follower="Node at be344f10-26a5-ee75-bc9d-aa2e88ff747d [Follower]" leader= -testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.745-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd678b2b-104a-f4db-7c80-817e7a97742e Address:cd678b2b-104a-f4db-7c80-817e7a97742e} {Suffrage:Voter ID:server-be344f10-26a5-ee75-bc9d-aa2e88ff747d Address:be344f10-26a5-ee75-bc9d-aa2e88ff747d} {Suffrage:Voter ID:server-08c1dfe3-2202-3a00-1e95-813d64ccb811 Address:08c1dfe3-2202-3a00-1e95-813d64ccb811}]" - testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.745-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd678b2b-104a-f4db-7c80-817e7a97742e Address:cd678b2b-104a-f4db-7c80-817e7a97742e} {Suffrage:Voter ID:server-be344f10-26a5-ee75-bc9d-aa2e88ff747d Address:be344f10-26a5-ee75-bc9d-aa2e88ff747d} {Suffrage:Voter ID:server-08c1dfe3-2202-3a00-1e95-813d64ccb811 Address:08c1dfe3-2202-3a00-1e95-813d64ccb811}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.745-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:40.745-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.745-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: entering follower state: follower="Node at 08c1dfe3-2202-3a00-1e95-813d64ccb811 [Follower]" leader= - testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.745-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: entering follower state: follower="Node at 08c1dfe3-2202-3a00-1e95-813d64ccb811 [Follower]" leader= -testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [WARN] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [WARN] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: entering candidate state: node="Node at 08c1dfe3-2202-3a00-1e95-813d64ccb811 [Candidate]" term=2 - testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: entering candidate state: node="Node at 08c1dfe3-2202-3a00-1e95-813d64ccb811 [Candidate]" term=2 -testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: election won: tally=2 - testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: election won: tally=2 -testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: entering leader state: leader="Node at 08c1dfe3-2202-3a00-1e95-813d64ccb811 [Leader]" - testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: entering leader state: leader="Node at 08c1dfe3-2202-3a00-1e95-813d64ccb811 [Leader]" -testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: added peer, starting replication: peer=server-cd678b2b-104a-f4db-7c80-817e7a97742e - testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: added peer, starting replication: peer=server-cd678b2b-104a-f4db-7c80-817e7a97742e -testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: added peer, starting replication: peer=server-be344f10-26a5-ee75-bc9d-aa2e88ff747d - testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: added peer, starting replication: peer=server-be344f10-26a5-ee75-bc9d-aa2e88ff747d -testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: pipelining replication: peer="{Voter server-cd678b2b-104a-f4db-7c80-817e7a97742e cd678b2b-104a-f4db-7c80-817e7a97742e}" - testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: pipelining replication: peer="{Voter server-cd678b2b-104a-f4db-7c80-817e7a97742e cd678b2b-104a-f4db-7c80-817e7a97742e}" -testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: pipelining replication: peer="{Voter server-be344f10-26a5-ee75-bc9d-aa2e88ff747d be344f10-26a5-ee75-bc9d-aa2e88ff747d}" - testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:40.830-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: pipelining replication: peer="{Voter server-be344f10-26a5-ee75-bc9d-aa2e88ff747d be344f10-26a5-ee75-bc9d-aa2e88ff747d}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.935-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:40.83079336 -1200 -12 m=+17.838109981 (2 nodes), 84.88246ms from start of poll, 85.67246ms from cluster start. Timeout at 2024-06-16 08:26:40.935897 -1200 -12 m=+17.943213481, 105.1035ms after stability - testing.go:149: cluster: 2024-06-16T08:26:40.935-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:40.83079336 -1200 -12 m=+17.838109981 (2 nodes), 84.88246ms from start of poll, 85.67246ms from cluster start. Timeout at 2024-06-16 08:26:40.935897 -1200 -12 m=+17.943213481, 105.1035ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:40.936-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:40.936-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.041-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.93641662 -1200 -12 m=+17.943733081 (1 nodes), 1.48µs from start of poll, 191.29556ms from cluster start. Timeout at 2024-06-16 08:26:41.04150304 -1200 -12 m=+18.048819521, 105.08644ms after stability - testing.go:149: cluster: 2024-06-16T08:26:41.041-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:40.93641662 -1200 -12 m=+17.943733081 (1 nodes), 1.48µs from start of poll, 191.29556ms from cluster start. Timeout at 2024-06-16 08:26:41.04150304 -1200 -12 m=+18.048819521, 105.08644ms after stability -testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:41.041-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: aborting pipeline replication: peer="{Voter server-be344f10-26a5-ee75-bc9d-aa2e88ff747d be344f10-26a5-ee75-bc9d-aa2e88ff747d}" - testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:41.041-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: aborting pipeline replication: peer="{Voter server-be344f10-26a5-ee75-bc9d-aa2e88ff747d be344f10-26a5-ee75-bc9d-aa2e88ff747d}" -testLoggerAdapter verbose: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:41.041-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: aborting pipeline replication: peer="{Voter server-cd678b2b-104a-f4db-7c80-817e7a97742e cd678b2b-104a-f4db-7c80-817e7a97742e}" - testing.go:149: server-08c1dfe3-2202-3a00-1e95-813d64ccb811: 2024-06-16T08:26:41.041-1200 [INFO] server-08c1dfe3-2202-3a00-1e95-813d64ccb811: aborting pipeline replication: peer="{Voter server-cd678b2b-104a-f4db-7c80-817e7a97742e cd678b2b-104a-f4db-7c80-817e7a97742e}" ---- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.30s) +testLoggerAdapter verbose: server-63d1e415-1f05-330d-1b02-64bd6757dcbf: 2023-05-16T04:06:21.432+1400 [INFO] server-63d1e415-1f05-330d-1b02-64bd6757dcbf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63d1e415-1f05-330d-1b02-64bd6757dcbf Address:63d1e415-1f05-330d-1b02-64bd6757dcbf} {Suffrage:Voter ID:server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d Address:70bd595e-d2b2-dc1f-5d27-b94b2b97731d} {Suffrage:Voter ID:server-3e142577-3fcc-adc5-50c4-612303809486 Address:3e142577-3fcc-adc5-50c4-612303809486}]" + testing.go:149: server-63d1e415-1f05-330d-1b02-64bd6757dcbf: 2023-05-16T04:06:21.432+1400 [INFO] server-63d1e415-1f05-330d-1b02-64bd6757dcbf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63d1e415-1f05-330d-1b02-64bd6757dcbf Address:63d1e415-1f05-330d-1b02-64bd6757dcbf} {Suffrage:Voter ID:server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d Address:70bd595e-d2b2-dc1f-5d27-b94b2b97731d} {Suffrage:Voter ID:server-3e142577-3fcc-adc5-50c4-612303809486 Address:3e142577-3fcc-adc5-50c4-612303809486}]" +testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.433+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63d1e415-1f05-330d-1b02-64bd6757dcbf Address:63d1e415-1f05-330d-1b02-64bd6757dcbf} {Suffrage:Voter ID:server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d Address:70bd595e-d2b2-dc1f-5d27-b94b2b97731d} {Suffrage:Voter ID:server-3e142577-3fcc-adc5-50c4-612303809486 Address:3e142577-3fcc-adc5-50c4-612303809486}]" + testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.433+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63d1e415-1f05-330d-1b02-64bd6757dcbf Address:63d1e415-1f05-330d-1b02-64bd6757dcbf} {Suffrage:Voter ID:server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d Address:70bd595e-d2b2-dc1f-5d27-b94b2b97731d} {Suffrage:Voter ID:server-3e142577-3fcc-adc5-50c4-612303809486 Address:3e142577-3fcc-adc5-50c4-612303809486}]" +testLoggerAdapter verbose: server-3e142577-3fcc-adc5-50c4-612303809486: 2023-05-16T04:06:21.433+1400 [INFO] server-3e142577-3fcc-adc5-50c4-612303809486: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63d1e415-1f05-330d-1b02-64bd6757dcbf Address:63d1e415-1f05-330d-1b02-64bd6757dcbf} {Suffrage:Voter ID:server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d Address:70bd595e-d2b2-dc1f-5d27-b94b2b97731d} {Suffrage:Voter ID:server-3e142577-3fcc-adc5-50c4-612303809486 Address:3e142577-3fcc-adc5-50c4-612303809486}]" + testing.go:149: server-3e142577-3fcc-adc5-50c4-612303809486: 2023-05-16T04:06:21.433+1400 [INFO] server-3e142577-3fcc-adc5-50c4-612303809486: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63d1e415-1f05-330d-1b02-64bd6757dcbf Address:63d1e415-1f05-330d-1b02-64bd6757dcbf} {Suffrage:Voter ID:server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d Address:70bd595e-d2b2-dc1f-5d27-b94b2b97731d} {Suffrage:Voter ID:server-3e142577-3fcc-adc5-50c4-612303809486 Address:3e142577-3fcc-adc5-50c4-612303809486}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.433+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:21.433+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-63d1e415-1f05-330d-1b02-64bd6757dcbf: 2023-05-16T04:06:21.433+1400 [INFO] server-63d1e415-1f05-330d-1b02-64bd6757dcbf: entering follower state: follower="Node at 63d1e415-1f05-330d-1b02-64bd6757dcbf [Follower]" leader= + testing.go:149: server-63d1e415-1f05-330d-1b02-64bd6757dcbf: 2023-05-16T04:06:21.433+1400 [INFO] server-63d1e415-1f05-330d-1b02-64bd6757dcbf: entering follower state: follower="Node at 63d1e415-1f05-330d-1b02-64bd6757dcbf [Follower]" leader= +testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.433+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: entering follower state: follower="Node at 70bd595e-d2b2-dc1f-5d27-b94b2b97731d [Follower]" leader= + testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.433+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: entering follower state: follower="Node at 70bd595e-d2b2-dc1f-5d27-b94b2b97731d [Follower]" leader= +testLoggerAdapter verbose: server-3e142577-3fcc-adc5-50c4-612303809486: 2023-05-16T04:06:21.433+1400 [INFO] server-3e142577-3fcc-adc5-50c4-612303809486: entering follower state: follower="Node at 3e142577-3fcc-adc5-50c4-612303809486 [Follower]" leader= + testing.go:149: server-3e142577-3fcc-adc5-50c4-612303809486: 2023-05-16T04:06:21.433+1400 [INFO] server-3e142577-3fcc-adc5-50c4-612303809486: entering follower state: follower="Node at 3e142577-3fcc-adc5-50c4-612303809486 [Follower]" leader= +testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [WARN] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [WARN] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: entering candidate state: node="Node at 70bd595e-d2b2-dc1f-5d27-b94b2b97731d [Candidate]" term=2 + testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: entering candidate state: node="Node at 70bd595e-d2b2-dc1f-5d27-b94b2b97731d [Candidate]" term=2 +testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: election won: tally=2 + testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: election won: tally=2 +testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: entering leader state: leader="Node at 70bd595e-d2b2-dc1f-5d27-b94b2b97731d [Leader]" + testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: entering leader state: leader="Node at 70bd595e-d2b2-dc1f-5d27-b94b2b97731d [Leader]" +testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: added peer, starting replication: peer=server-63d1e415-1f05-330d-1b02-64bd6757dcbf + testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.492+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: added peer, starting replication: peer=server-63d1e415-1f05-330d-1b02-64bd6757dcbf +testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.493+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: added peer, starting replication: peer=server-3e142577-3fcc-adc5-50c4-612303809486 + testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.493+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: added peer, starting replication: peer=server-3e142577-3fcc-adc5-50c4-612303809486 +testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.493+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: pipelining replication: peer="{Voter server-3e142577-3fcc-adc5-50c4-612303809486 3e142577-3fcc-adc5-50c4-612303809486}" + testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.493+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: pipelining replication: peer="{Voter server-3e142577-3fcc-adc5-50c4-612303809486 3e142577-3fcc-adc5-50c4-612303809486}" +testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.493+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: pipelining replication: peer="{Voter server-63d1e415-1f05-330d-1b02-64bd6757dcbf 63d1e415-1f05-330d-1b02-64bd6757dcbf}" + testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.493+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: pipelining replication: peer="{Voter server-63d1e415-1f05-330d-1b02-64bd6757dcbf 63d1e415-1f05-330d-1b02-64bd6757dcbf}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.599+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:21.493601167 +1400 +14 m=+18.296182782 (2 nodes), 60.17511ms from start of poll, 61.198502ms from cluster start. Timeout at 2023-05-16 04:06:21.599120372 +1400 +14 m=+18.401702007, 105.519225ms after stability + testing.go:149: cluster: 2023-05-16T04:06:21.599+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:21.493601167 +1400 +14 m=+18.296182782 (2 nodes), 60.17511ms from start of poll, 61.198502ms from cluster start. Timeout at 2023-05-16 04:06:21.599120372 +1400 +14 m=+18.401702007, 105.519225ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.599+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:21.599+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.704+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:21.599468656 +1400 +14 m=+18.402050351 (1 nodes), 2.7µs from start of poll, 167.066071ms from cluster start. Timeout at 2023-05-16 04:06:21.704729178 +1400 +14 m=+18.507310933, 105.260582ms after stability + testing.go:149: cluster: 2023-05-16T04:06:21.704+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:21.599468656 +1400 +14 m=+18.402050351 (1 nodes), 2.7µs from start of poll, 167.066071ms from cluster start. Timeout at 2023-05-16 04:06:21.704729178 +1400 +14 m=+18.507310933, 105.260582ms after stability +testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.705+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: aborting pipeline replication: peer="{Voter server-3e142577-3fcc-adc5-50c4-612303809486 3e142577-3fcc-adc5-50c4-612303809486}" + testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.705+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: aborting pipeline replication: peer="{Voter server-3e142577-3fcc-adc5-50c4-612303809486 3e142577-3fcc-adc5-50c4-612303809486}" +testLoggerAdapter verbose: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.705+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: aborting pipeline replication: peer="{Voter server-63d1e415-1f05-330d-1b02-64bd6757dcbf 63d1e415-1f05-330d-1b02-64bd6757dcbf}" + testing.go:149: server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: 2023-05-16T04:06:21.705+1400 [INFO] server-70bd595e-d2b2-dc1f-5d27-b94b2b97731d: aborting pipeline replication: peer="{Voter server-63d1e415-1f05-330d-1b02-64bd6757dcbf 63d1e415-1f05-330d-1b02-64bd6757dcbf}" +--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.27s) === RUN TestRaft_LeadershipTransferToBehindServer -testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.043-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2fc4e69-3c0e-96b6-b518-1f46772f472a Address:e2fc4e69-3c0e-96b6-b518-1f46772f472a} {Suffrage:Voter ID:server-98584a7b-babb-a728-bde5-c6725db92476 Address:98584a7b-babb-a728-bde5-c6725db92476} {Suffrage:Voter ID:server-56c05aba-b15b-4657-5714-48d65e7bf55d Address:56c05aba-b15b-4657-5714-48d65e7bf55d}]" - testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.043-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2fc4e69-3c0e-96b6-b518-1f46772f472a Address:e2fc4e69-3c0e-96b6-b518-1f46772f472a} {Suffrage:Voter ID:server-98584a7b-babb-a728-bde5-c6725db92476 Address:98584a7b-babb-a728-bde5-c6725db92476} {Suffrage:Voter ID:server-56c05aba-b15b-4657-5714-48d65e7bf55d Address:56c05aba-b15b-4657-5714-48d65e7bf55d}]" -testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.043-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: entering follower state: follower="Node at e2fc4e69-3c0e-96b6-b518-1f46772f472a [Follower]" leader= - testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.043-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: entering follower state: follower="Node at e2fc4e69-3c0e-96b6-b518-1f46772f472a [Follower]" leader= -testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.043-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2fc4e69-3c0e-96b6-b518-1f46772f472a Address:e2fc4e69-3c0e-96b6-b518-1f46772f472a} {Suffrage:Voter ID:server-98584a7b-babb-a728-bde5-c6725db92476 Address:98584a7b-babb-a728-bde5-c6725db92476} {Suffrage:Voter ID:server-56c05aba-b15b-4657-5714-48d65e7bf55d Address:56c05aba-b15b-4657-5714-48d65e7bf55d}]" - testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.043-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2fc4e69-3c0e-96b6-b518-1f46772f472a Address:e2fc4e69-3c0e-96b6-b518-1f46772f472a} {Suffrage:Voter ID:server-98584a7b-babb-a728-bde5-c6725db92476 Address:98584a7b-babb-a728-bde5-c6725db92476} {Suffrage:Voter ID:server-56c05aba-b15b-4657-5714-48d65e7bf55d Address:56c05aba-b15b-4657-5714-48d65e7bf55d}]" -testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.043-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: entering follower state: follower="Node at 98584a7b-babb-a728-bde5-c6725db92476 [Follower]" leader= - testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.043-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: entering follower state: follower="Node at 98584a7b-babb-a728-bde5-c6725db92476 [Follower]" leader= -testLoggerAdapter verbose: server-56c05aba-b15b-4657-5714-48d65e7bf55d: 2024-06-16T08:26:41.043-1200 [INFO] server-56c05aba-b15b-4657-5714-48d65e7bf55d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2fc4e69-3c0e-96b6-b518-1f46772f472a Address:e2fc4e69-3c0e-96b6-b518-1f46772f472a} {Suffrage:Voter ID:server-98584a7b-babb-a728-bde5-c6725db92476 Address:98584a7b-babb-a728-bde5-c6725db92476} {Suffrage:Voter ID:server-56c05aba-b15b-4657-5714-48d65e7bf55d Address:56c05aba-b15b-4657-5714-48d65e7bf55d}]" - testing.go:149: server-56c05aba-b15b-4657-5714-48d65e7bf55d: 2024-06-16T08:26:41.043-1200 [INFO] server-56c05aba-b15b-4657-5714-48d65e7bf55d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2fc4e69-3c0e-96b6-b518-1f46772f472a Address:e2fc4e69-3c0e-96b6-b518-1f46772f472a} {Suffrage:Voter ID:server-98584a7b-babb-a728-bde5-c6725db92476 Address:98584a7b-babb-a728-bde5-c6725db92476} {Suffrage:Voter ID:server-56c05aba-b15b-4657-5714-48d65e7bf55d Address:56c05aba-b15b-4657-5714-48d65e7bf55d}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.043-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:41.043-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-56c05aba-b15b-4657-5714-48d65e7bf55d: 2024-06-16T08:26:41.043-1200 [INFO] server-56c05aba-b15b-4657-5714-48d65e7bf55d: entering follower state: follower="Node at 56c05aba-b15b-4657-5714-48d65e7bf55d [Follower]" leader= - testing.go:149: server-56c05aba-b15b-4657-5714-48d65e7bf55d: 2024-06-16T08:26:41.043-1200 [INFO] server-56c05aba-b15b-4657-5714-48d65e7bf55d: entering follower state: follower="Node at 56c05aba-b15b-4657-5714-48d65e7bf55d [Follower]" leader= -testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [WARN] server-98584a7b-babb-a728-bde5-c6725db92476: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [WARN] server-98584a7b-babb-a728-bde5-c6725db92476: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: entering candidate state: node="Node at 98584a7b-babb-a728-bde5-c6725db92476 [Candidate]" term=2 - testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: entering candidate state: node="Node at 98584a7b-babb-a728-bde5-c6725db92476 [Candidate]" term=2 -testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: election won: tally=2 - testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: election won: tally=2 -testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: entering leader state: leader="Node at 98584a7b-babb-a728-bde5-c6725db92476 [Leader]" - testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: entering leader state: leader="Node at 98584a7b-babb-a728-bde5-c6725db92476 [Leader]" -testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: added peer, starting replication: peer=server-e2fc4e69-3c0e-96b6-b518-1f46772f472a - testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: added peer, starting replication: peer=server-e2fc4e69-3c0e-96b6-b518-1f46772f472a -testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: added peer, starting replication: peer=server-56c05aba-b15b-4657-5714-48d65e7bf55d - testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: added peer, starting replication: peer=server-56c05aba-b15b-4657-5714-48d65e7bf55d -testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: pipelining replication: peer="{Voter server-56c05aba-b15b-4657-5714-48d65e7bf55d 56c05aba-b15b-4657-5714-48d65e7bf55d}" - testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: pipelining replication: peer="{Voter server-56c05aba-b15b-4657-5714-48d65e7bf55d 56c05aba-b15b-4657-5714-48d65e7bf55d}" -testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: pipelining replication: peer="{Voter server-e2fc4e69-3c0e-96b6-b518-1f46772f472a e2fc4e69-3c0e-96b6-b518-1f46772f472a}" - testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.109-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: pipelining replication: peer="{Voter server-e2fc4e69-3c0e-96b6-b518-1f46772f472a e2fc4e69-3c0e-96b6-b518-1f46772f472a}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.215-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:41.10971026 -1200 -12 m=+18.117026741 (1 nodes), 65.81598ms from start of poll, 66.67466ms from cluster start. Timeout at 2024-06-16 08:26:41.21543002 -1200 -12 m=+18.222746501, 105.71976ms after stability - testing.go:149: cluster: 2024-06-16T08:26:41.215-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:41.10971026 -1200 -12 m=+18.117026741 (1 nodes), 65.81598ms from start of poll, 66.67466ms from cluster start. Timeout at 2024-06-16 08:26:41.21543002 -1200 -12 m=+18.222746501, 105.71976ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.215-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:41.215-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.321-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:41.21560404 -1200 -12 m=+18.222920501 (2 nodes), 2.26µs from start of poll, 172.56842ms from cluster start. Timeout at 2024-06-16 08:26:41.3211377 -1200 -12 m=+18.328454161, 105.53366ms after stability - testing.go:149: cluster: 2024-06-16T08:26:41.321-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:41.21560404 -1200 -12 m=+18.222920501 (2 nodes), 2.26µs from start of poll, 172.56842ms from cluster start. Timeout at 2024-06-16 08:26:41.3211377 -1200 -12 m=+18.328454161, 105.53366ms after stability -testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.354-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: entering candidate state: node="Node at e2fc4e69-3c0e-96b6-b518-1f46772f472a [Candidate]" term=3 - testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.354-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: entering candidate state: node="Node at e2fc4e69-3c0e-96b6-b518-1f46772f472a [Candidate]" term=3 -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.354-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:41.354-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.354-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: election won: tally=2 - testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.354-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: election won: tally=2 -testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.354-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: entering follower state: follower="Node at 98584a7b-babb-a728-bde5-c6725db92476 [Follower]" leader= - testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.354-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: entering follower state: follower="Node at 98584a7b-babb-a728-bde5-c6725db92476 [Follower]" leader= -testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.354-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: entering leader state: leader="Node at e2fc4e69-3c0e-96b6-b518-1f46772f472a [Leader]" - testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.354-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: entering leader state: leader="Node at e2fc4e69-3c0e-96b6-b518-1f46772f472a [Leader]" -testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.355-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: added peer, starting replication: peer=server-98584a7b-babb-a728-bde5-c6725db92476 -testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.354-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: aborting pipeline replication: peer="{Voter server-e2fc4e69-3c0e-96b6-b518-1f46772f472a e2fc4e69-3c0e-96b6-b518-1f46772f472a}" - testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.355-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: added peer, starting replication: peer=server-98584a7b-babb-a728-bde5-c6725db92476 -testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.355-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: added peer, starting replication: peer=server-56c05aba-b15b-4657-5714-48d65e7bf55d - testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.354-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: aborting pipeline replication: peer="{Voter server-e2fc4e69-3c0e-96b6-b518-1f46772f472a e2fc4e69-3c0e-96b6-b518-1f46772f472a}" - testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.355-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: added peer, starting replication: peer=server-56c05aba-b15b-4657-5714-48d65e7bf55d -testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.355-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: pipelining replication: peer="{Voter server-98584a7b-babb-a728-bde5-c6725db92476 98584a7b-babb-a728-bde5-c6725db92476}" - testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.355-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: pipelining replication: peer="{Voter server-98584a7b-babb-a728-bde5-c6725db92476 98584a7b-babb-a728-bde5-c6725db92476}" -testLoggerAdapter verbose: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.355-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: aborting pipeline replication: peer="{Voter server-56c05aba-b15b-4657-5714-48d65e7bf55d 56c05aba-b15b-4657-5714-48d65e7bf55d}" - testing.go:149: server-98584a7b-babb-a728-bde5-c6725db92476: 2024-06-16T08:26:41.355-1200 [INFO] server-98584a7b-babb-a728-bde5-c6725db92476: aborting pipeline replication: peer="{Voter server-56c05aba-b15b-4657-5714-48d65e7bf55d 56c05aba-b15b-4657-5714-48d65e7bf55d}" -testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.355-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: pipelining replication: peer="{Voter server-56c05aba-b15b-4657-5714-48d65e7bf55d 56c05aba-b15b-4657-5714-48d65e7bf55d}" - testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.355-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: pipelining replication: peer="{Voter server-56c05aba-b15b-4657-5714-48d65e7bf55d 56c05aba-b15b-4657-5714-48d65e7bf55d}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.460-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:41.35497818 -1200 -12 m=+18.362294641 (1 nodes), 110µs from start of poll, 311.94256ms from cluster start. Timeout at 2024-06-16 08:26:41.460514 -1200 -12 m=+18.467830481, 105.53584ms after stability - testing.go:149: cluster: 2024-06-16T08:26:41.460-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:41.35497818 -1200 -12 m=+18.362294641 (1 nodes), 110µs from start of poll, 311.94256ms from cluster start. Timeout at 2024-06-16 08:26:41.460514 -1200 -12 m=+18.467830481, 105.53584ms after stability -testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.460-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: aborting pipeline replication: peer="{Voter server-98584a7b-babb-a728-bde5-c6725db92476 98584a7b-babb-a728-bde5-c6725db92476}" - testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.460-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: aborting pipeline replication: peer="{Voter server-98584a7b-babb-a728-bde5-c6725db92476 98584a7b-babb-a728-bde5-c6725db92476}" -testLoggerAdapter verbose: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.460-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: aborting pipeline replication: peer="{Voter server-56c05aba-b15b-4657-5714-48d65e7bf55d 56c05aba-b15b-4657-5714-48d65e7bf55d}" - testing.go:149: server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: 2024-06-16T08:26:41.460-1200 [INFO] server-e2fc4e69-3c0e-96b6-b518-1f46772f472a: aborting pipeline replication: peer="{Voter server-56c05aba-b15b-4657-5714-48d65e7bf55d 56c05aba-b15b-4657-5714-48d65e7bf55d}" ---- PASS: TestRaft_LeadershipTransferToBehindServer (0.42s) +testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.706+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-313389c2-f60a-9532-2f79-5beec9a54ec8 Address:313389c2-f60a-9532-2f79-5beec9a54ec8} {Suffrage:Voter ID:server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd Address:15d6d569-6d6e-6b61-7d18-6e269cf5c4bd} {Suffrage:Voter ID:server-e6c9f727-3007-3e26-19fe-1764cdaedea9 Address:e6c9f727-3007-3e26-19fe-1764cdaedea9}]" + testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.706+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-313389c2-f60a-9532-2f79-5beec9a54ec8 Address:313389c2-f60a-9532-2f79-5beec9a54ec8} {Suffrage:Voter ID:server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd Address:15d6d569-6d6e-6b61-7d18-6e269cf5c4bd} {Suffrage:Voter ID:server-e6c9f727-3007-3e26-19fe-1764cdaedea9 Address:e6c9f727-3007-3e26-19fe-1764cdaedea9}]" +testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:21.707+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-313389c2-f60a-9532-2f79-5beec9a54ec8 Address:313389c2-f60a-9532-2f79-5beec9a54ec8} {Suffrage:Voter ID:server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd Address:15d6d569-6d6e-6b61-7d18-6e269cf5c4bd} {Suffrage:Voter ID:server-e6c9f727-3007-3e26-19fe-1764cdaedea9 Address:e6c9f727-3007-3e26-19fe-1764cdaedea9}]" + testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:21.707+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-313389c2-f60a-9532-2f79-5beec9a54ec8 Address:313389c2-f60a-9532-2f79-5beec9a54ec8} {Suffrage:Voter ID:server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd Address:15d6d569-6d6e-6b61-7d18-6e269cf5c4bd} {Suffrage:Voter ID:server-e6c9f727-3007-3e26-19fe-1764cdaedea9 Address:e6c9f727-3007-3e26-19fe-1764cdaedea9}]" +testLoggerAdapter verbose: server-e6c9f727-3007-3e26-19fe-1764cdaedea9: 2023-05-16T04:06:21.707+1400 [INFO] server-e6c9f727-3007-3e26-19fe-1764cdaedea9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-313389c2-f60a-9532-2f79-5beec9a54ec8 Address:313389c2-f60a-9532-2f79-5beec9a54ec8} {Suffrage:Voter ID:server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd Address:15d6d569-6d6e-6b61-7d18-6e269cf5c4bd} {Suffrage:Voter ID:server-e6c9f727-3007-3e26-19fe-1764cdaedea9 Address:e6c9f727-3007-3e26-19fe-1764cdaedea9}]" + testing.go:149: server-e6c9f727-3007-3e26-19fe-1764cdaedea9: 2023-05-16T04:06:21.707+1400 [INFO] server-e6c9f727-3007-3e26-19fe-1764cdaedea9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-313389c2-f60a-9532-2f79-5beec9a54ec8 Address:313389c2-f60a-9532-2f79-5beec9a54ec8} {Suffrage:Voter ID:server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd Address:15d6d569-6d6e-6b61-7d18-6e269cf5c4bd} {Suffrage:Voter ID:server-e6c9f727-3007-3e26-19fe-1764cdaedea9 Address:e6c9f727-3007-3e26-19fe-1764cdaedea9}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.707+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:21.707+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.707+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: entering follower state: follower="Node at 313389c2-f60a-9532-2f79-5beec9a54ec8 [Follower]" leader= + testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.707+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: entering follower state: follower="Node at 313389c2-f60a-9532-2f79-5beec9a54ec8 [Follower]" leader= +testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:21.707+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: entering follower state: follower="Node at 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd [Follower]" leader= + testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:21.707+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: entering follower state: follower="Node at 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd [Follower]" leader= +testLoggerAdapter verbose: server-e6c9f727-3007-3e26-19fe-1764cdaedea9: 2023-05-16T04:06:21.707+1400 [INFO] server-e6c9f727-3007-3e26-19fe-1764cdaedea9: entering follower state: follower="Node at e6c9f727-3007-3e26-19fe-1764cdaedea9 [Follower]" leader= + testing.go:149: server-e6c9f727-3007-3e26-19fe-1764cdaedea9: 2023-05-16T04:06:21.707+1400 [INFO] server-e6c9f727-3007-3e26-19fe-1764cdaedea9: entering follower state: follower="Node at e6c9f727-3007-3e26-19fe-1764cdaedea9 [Follower]" leader= +testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.765+1400 [WARN] server-313389c2-f60a-9532-2f79-5beec9a54ec8: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.765+1400 [WARN] server-313389c2-f60a-9532-2f79-5beec9a54ec8: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.765+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: entering candidate state: node="Node at 313389c2-f60a-9532-2f79-5beec9a54ec8 [Candidate]" term=2 + testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.765+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: entering candidate state: node="Node at 313389c2-f60a-9532-2f79-5beec9a54ec8 [Candidate]" term=2 +testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.765+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: election won: tally=2 + testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.765+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: election won: tally=2 +testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.765+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: entering leader state: leader="Node at 313389c2-f60a-9532-2f79-5beec9a54ec8 [Leader]" + testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.765+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: entering leader state: leader="Node at 313389c2-f60a-9532-2f79-5beec9a54ec8 [Leader]" +testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.766+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: added peer, starting replication: peer=server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd + testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.766+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: added peer, starting replication: peer=server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd +testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.766+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: added peer, starting replication: peer=server-e6c9f727-3007-3e26-19fe-1764cdaedea9 + testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.766+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: added peer, starting replication: peer=server-e6c9f727-3007-3e26-19fe-1764cdaedea9 +testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.766+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: pipelining replication: peer="{Voter server-e6c9f727-3007-3e26-19fe-1764cdaedea9 e6c9f727-3007-3e26-19fe-1764cdaedea9}" + testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.766+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: pipelining replication: peer="{Voter server-e6c9f727-3007-3e26-19fe-1764cdaedea9 e6c9f727-3007-3e26-19fe-1764cdaedea9}" +testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.766+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: pipelining replication: peer="{Voter server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd}" + testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:21.766+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: pipelining replication: peer="{Voter server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.872+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:21.76672115 +1400 +14 m=+18.569302785 (1 nodes), 59.121958ms from start of poll, 60.108429ms from cluster start. Timeout at 2023-05-16 04:06:21.872259455 +1400 +14 m=+18.674841110, 105.538325ms after stability + testing.go:149: cluster: 2023-05-16T04:06:21.872+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:21.76672115 +1400 +14 m=+18.569302785 (1 nodes), 59.121958ms from start of poll, 60.108429ms from cluster start. Timeout at 2023-05-16 04:06:21.872259455 +1400 +14 m=+18.674841110, 105.538325ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.872+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:21.872+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:21.978+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:21.872559219 +1400 +14 m=+18.675140854 (2 nodes), 2.58µs from start of poll, 165.946498ms from cluster start. Timeout at 2023-05-16 04:06:21.978340427 +1400 +14 m=+18.780922042, 105.781188ms after stability + testing.go:149: cluster: 2023-05-16T04:06:21.978+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:21.872559219 +1400 +14 m=+18.675140854 (2 nodes), 2.58µs from start of poll, 165.946498ms from cluster start. Timeout at 2023-05-16 04:06:21.978340427 +1400 +14 m=+18.780922042, 105.781188ms after stability +testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.045+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: entering candidate state: node="Node at 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd [Candidate]" term=3 + testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.045+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: entering candidate state: node="Node at 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd [Candidate]" term=3 +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:22.045+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:22.045+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.045+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: election won: tally=2 + testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.045+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: election won: tally=2 +testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:22.045+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: entering follower state: follower="Node at 313389c2-f60a-9532-2f79-5beec9a54ec8 [Follower]" leader= + testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:22.045+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: entering follower state: follower="Node at 313389c2-f60a-9532-2f79-5beec9a54ec8 [Follower]" leader= +testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.045+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: entering leader state: leader="Node at 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd [Leader]" + testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.045+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: entering leader state: leader="Node at 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd [Leader]" +testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.046+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: added peer, starting replication: peer=server-313389c2-f60a-9532-2f79-5beec9a54ec8 +testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:22.046+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: aborting pipeline replication: peer="{Voter server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd}" + testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.046+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: added peer, starting replication: peer=server-313389c2-f60a-9532-2f79-5beec9a54ec8 +testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.046+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: added peer, starting replication: peer=server-e6c9f727-3007-3e26-19fe-1764cdaedea9 + testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.046+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: added peer, starting replication: peer=server-e6c9f727-3007-3e26-19fe-1764cdaedea9 +testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.046+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: pipelining replication: peer="{Voter server-e6c9f727-3007-3e26-19fe-1764cdaedea9 e6c9f727-3007-3e26-19fe-1764cdaedea9}" + testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.046+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: pipelining replication: peer="{Voter server-e6c9f727-3007-3e26-19fe-1764cdaedea9 e6c9f727-3007-3e26-19fe-1764cdaedea9}" + testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:22.046+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: aborting pipeline replication: peer="{Voter server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd 15d6d569-6d6e-6b61-7d18-6e269cf5c4bd}" +testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.046+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: pipelining replication: peer="{Voter server-313389c2-f60a-9532-2f79-5beec9a54ec8 313389c2-f60a-9532-2f79-5beec9a54ec8}" + testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.046+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: pipelining replication: peer="{Voter server-313389c2-f60a-9532-2f79-5beec9a54ec8 313389c2-f60a-9532-2f79-5beec9a54ec8}" +testLoggerAdapter verbose: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:22.046+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: aborting pipeline replication: peer="{Voter server-e6c9f727-3007-3e26-19fe-1764cdaedea9 e6c9f727-3007-3e26-19fe-1764cdaedea9}" + testing.go:149: server-313389c2-f60a-9532-2f79-5beec9a54ec8: 2023-05-16T04:06:22.046+1400 [INFO] server-313389c2-f60a-9532-2f79-5beec9a54ec8: aborting pipeline replication: peer="{Voter server-e6c9f727-3007-3e26-19fe-1764cdaedea9 e6c9f727-3007-3e26-19fe-1764cdaedea9}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:22.151+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:22.046016226 +1400 +14 m=+18.848597861 (1 nodes), 123.961µs from start of poll, 339.403505ms from cluster start. Timeout at 2023-05-16 04:06:22.151655153 +1400 +14 m=+18.954236788, 105.638927ms after stability + testing.go:149: cluster: 2023-05-16T04:06:22.151+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:22.046016226 +1400 +14 m=+18.848597861 (1 nodes), 123.961µs from start of poll, 339.403505ms from cluster start. Timeout at 2023-05-16 04:06:22.151655153 +1400 +14 m=+18.954236788, 105.638927ms after stability +testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.152+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: aborting pipeline replication: peer="{Voter server-e6c9f727-3007-3e26-19fe-1764cdaedea9 e6c9f727-3007-3e26-19fe-1764cdaedea9}" + testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.152+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: aborting pipeline replication: peer="{Voter server-e6c9f727-3007-3e26-19fe-1764cdaedea9 e6c9f727-3007-3e26-19fe-1764cdaedea9}" +testLoggerAdapter verbose: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.152+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: aborting pipeline replication: peer="{Voter server-313389c2-f60a-9532-2f79-5beec9a54ec8 313389c2-f60a-9532-2f79-5beec9a54ec8}" + testing.go:149: server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: 2023-05-16T04:06:22.152+1400 [INFO] server-15d6d569-6d6e-6b61-7d18-6e269cf5c4bd: aborting pipeline replication: peer="{Voter server-313389c2-f60a-9532-2f79-5beec9a54ec8 313389c2-f60a-9532-2f79-5beec9a54ec8}" +--- PASS: TestRaft_LeadershipTransferToBehindServer (0.45s) === RUN TestRaft_LeadershipTransferToItself -testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.462-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 Address:7d6f5226-e7a8-558d-5f8c-04e7be9c1da5} {Suffrage:Voter ID:server-9754a28d-e77b-5efb-331f-a7162c327381 Address:9754a28d-e77b-5efb-331f-a7162c327381} {Suffrage:Voter ID:server-d68c3943-0312-dc4c-b744-b153b7a95ede Address:d68c3943-0312-dc4c-b744-b153b7a95ede}]" - testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.462-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 Address:7d6f5226-e7a8-558d-5f8c-04e7be9c1da5} {Suffrage:Voter ID:server-9754a28d-e77b-5efb-331f-a7162c327381 Address:9754a28d-e77b-5efb-331f-a7162c327381} {Suffrage:Voter ID:server-d68c3943-0312-dc4c-b744-b153b7a95ede Address:d68c3943-0312-dc4c-b744-b153b7a95ede}]" -testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.462-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: entering follower state: follower="Node at 7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 [Follower]" leader= - testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.462-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: entering follower state: follower="Node at 7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 [Follower]" leader= -testLoggerAdapter verbose: server-9754a28d-e77b-5efb-331f-a7162c327381: 2024-06-16T08:26:41.462-1200 [INFO] server-9754a28d-e77b-5efb-331f-a7162c327381: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 Address:7d6f5226-e7a8-558d-5f8c-04e7be9c1da5} {Suffrage:Voter ID:server-9754a28d-e77b-5efb-331f-a7162c327381 Address:9754a28d-e77b-5efb-331f-a7162c327381} {Suffrage:Voter ID:server-d68c3943-0312-dc4c-b744-b153b7a95ede Address:d68c3943-0312-dc4c-b744-b153b7a95ede}]" - testing.go:149: server-9754a28d-e77b-5efb-331f-a7162c327381: 2024-06-16T08:26:41.462-1200 [INFO] server-9754a28d-e77b-5efb-331f-a7162c327381: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 Address:7d6f5226-e7a8-558d-5f8c-04e7be9c1da5} {Suffrage:Voter ID:server-9754a28d-e77b-5efb-331f-a7162c327381 Address:9754a28d-e77b-5efb-331f-a7162c327381} {Suffrage:Voter ID:server-d68c3943-0312-dc4c-b744-b153b7a95ede Address:d68c3943-0312-dc4c-b744-b153b7a95ede}]" -testLoggerAdapter verbose: server-9754a28d-e77b-5efb-331f-a7162c327381: 2024-06-16T08:26:41.462-1200 [INFO] server-9754a28d-e77b-5efb-331f-a7162c327381: entering follower state: follower="Node at 9754a28d-e77b-5efb-331f-a7162c327381 [Follower]" leader= - testing.go:149: server-9754a28d-e77b-5efb-331f-a7162c327381: 2024-06-16T08:26:41.462-1200 [INFO] server-9754a28d-e77b-5efb-331f-a7162c327381: entering follower state: follower="Node at 9754a28d-e77b-5efb-331f-a7162c327381 [Follower]" leader= -testLoggerAdapter verbose: server-d68c3943-0312-dc4c-b744-b153b7a95ede: 2024-06-16T08:26:41.462-1200 [INFO] server-d68c3943-0312-dc4c-b744-b153b7a95ede: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 Address:7d6f5226-e7a8-558d-5f8c-04e7be9c1da5} {Suffrage:Voter ID:server-9754a28d-e77b-5efb-331f-a7162c327381 Address:9754a28d-e77b-5efb-331f-a7162c327381} {Suffrage:Voter ID:server-d68c3943-0312-dc4c-b744-b153b7a95ede Address:d68c3943-0312-dc4c-b744-b153b7a95ede}]" - testing.go:149: server-d68c3943-0312-dc4c-b744-b153b7a95ede: 2024-06-16T08:26:41.462-1200 [INFO] server-d68c3943-0312-dc4c-b744-b153b7a95ede: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 Address:7d6f5226-e7a8-558d-5f8c-04e7be9c1da5} {Suffrage:Voter ID:server-9754a28d-e77b-5efb-331f-a7162c327381 Address:9754a28d-e77b-5efb-331f-a7162c327381} {Suffrage:Voter ID:server-d68c3943-0312-dc4c-b744-b153b7a95ede Address:d68c3943-0312-dc4c-b744-b153b7a95ede}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.462-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:41.462-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-d68c3943-0312-dc4c-b744-b153b7a95ede: 2024-06-16T08:26:41.462-1200 [INFO] server-d68c3943-0312-dc4c-b744-b153b7a95ede: entering follower state: follower="Node at d68c3943-0312-dc4c-b744-b153b7a95ede [Follower]" leader= - testing.go:149: server-d68c3943-0312-dc4c-b744-b153b7a95ede: 2024-06-16T08:26:41.462-1200 [INFO] server-d68c3943-0312-dc4c-b744-b153b7a95ede: entering follower state: follower="Node at d68c3943-0312-dc4c-b744-b153b7a95ede [Follower]" leader= -testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [WARN] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [WARN] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: entering candidate state: node="Node at 7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 [Candidate]" term=2 - testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: entering candidate state: node="Node at 7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 [Candidate]" term=2 -testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: election won: tally=2 - testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: election won: tally=2 -testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: entering leader state: leader="Node at 7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 [Leader]" - testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: entering leader state: leader="Node at 7d6f5226-e7a8-558d-5f8c-04e7be9c1da5 [Leader]" -testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: added peer, starting replication: peer=server-9754a28d-e77b-5efb-331f-a7162c327381 - testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: added peer, starting replication: peer=server-9754a28d-e77b-5efb-331f-a7162c327381 -testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: added peer, starting replication: peer=server-d68c3943-0312-dc4c-b744-b153b7a95ede - testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.530-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: added peer, starting replication: peer=server-d68c3943-0312-dc4c-b744-b153b7a95ede -testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.531-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: pipelining replication: peer="{Voter server-d68c3943-0312-dc4c-b744-b153b7a95ede d68c3943-0312-dc4c-b744-b153b7a95ede}" - testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.531-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: pipelining replication: peer="{Voter server-d68c3943-0312-dc4c-b744-b153b7a95ede d68c3943-0312-dc4c-b744-b153b7a95ede}" -testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.531-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: pipelining replication: peer="{Voter server-9754a28d-e77b-5efb-331f-a7162c327381 9754a28d-e77b-5efb-331f-a7162c327381}" - testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.531-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: pipelining replication: peer="{Voter server-9754a28d-e77b-5efb-331f-a7162c327381 9754a28d-e77b-5efb-331f-a7162c327381}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.637-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:41.53156266 -1200 -12 m=+18.538879141 (1 nodes), 68.77614ms from start of poll, 69.6601ms from cluster start. Timeout at 2024-06-16 08:26:41.63734374 -1200 -12 m=+18.644660221, 105.78108ms after stability - testing.go:149: cluster: 2024-06-16T08:26:41.637-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:41.53156266 -1200 -12 m=+18.538879141 (1 nodes), 68.77614ms from start of poll, 69.6601ms from cluster start. Timeout at 2024-06-16 08:26:41.63734374 -1200 -12 m=+18.644660221, 105.78108ms after stability -testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.637-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: cannot transfer leadership to itself - testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.637-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: cannot transfer leadership to itself -testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.637-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: aborting pipeline replication: peer="{Voter server-d68c3943-0312-dc4c-b744-b153b7a95ede d68c3943-0312-dc4c-b744-b153b7a95ede}" - testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.637-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: aborting pipeline replication: peer="{Voter server-d68c3943-0312-dc4c-b744-b153b7a95ede d68c3943-0312-dc4c-b744-b153b7a95ede}" -testLoggerAdapter verbose: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.637-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: aborting pipeline replication: peer="{Voter server-9754a28d-e77b-5efb-331f-a7162c327381 9754a28d-e77b-5efb-331f-a7162c327381}" - testing.go:149: server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: 2024-06-16T08:26:41.637-1200 [INFO] server-7d6f5226-e7a8-558d-5f8c-04e7be9c1da5: aborting pipeline replication: peer="{Voter server-9754a28d-e77b-5efb-331f-a7162c327381 9754a28d-e77b-5efb-331f-a7162c327381}" +testLoggerAdapter verbose: server-daa8a26e-5b8f-d054-8241-34f612eba224: 2023-05-16T04:06:22.153+1400 [INFO] server-daa8a26e-5b8f-d054-8241-34f612eba224: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daa8a26e-5b8f-d054-8241-34f612eba224 Address:daa8a26e-5b8f-d054-8241-34f612eba224} {Suffrage:Voter ID:server-21827d4e-199d-cef6-9f4c-689713aed83c Address:21827d4e-199d-cef6-9f4c-689713aed83c} {Suffrage:Voter ID:server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 Address:0e755218-6cff-c1f4-24ba-7ad78d6dfdb6}]" + testing.go:149: server-daa8a26e-5b8f-d054-8241-34f612eba224: 2023-05-16T04:06:22.153+1400 [INFO] server-daa8a26e-5b8f-d054-8241-34f612eba224: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daa8a26e-5b8f-d054-8241-34f612eba224 Address:daa8a26e-5b8f-d054-8241-34f612eba224} {Suffrage:Voter ID:server-21827d4e-199d-cef6-9f4c-689713aed83c Address:21827d4e-199d-cef6-9f4c-689713aed83c} {Suffrage:Voter ID:server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 Address:0e755218-6cff-c1f4-24ba-7ad78d6dfdb6}]" +testLoggerAdapter verbose: server-21827d4e-199d-cef6-9f4c-689713aed83c: 2023-05-16T04:06:22.154+1400 [INFO] server-21827d4e-199d-cef6-9f4c-689713aed83c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daa8a26e-5b8f-d054-8241-34f612eba224 Address:daa8a26e-5b8f-d054-8241-34f612eba224} {Suffrage:Voter ID:server-21827d4e-199d-cef6-9f4c-689713aed83c Address:21827d4e-199d-cef6-9f4c-689713aed83c} {Suffrage:Voter ID:server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 Address:0e755218-6cff-c1f4-24ba-7ad78d6dfdb6}]" + testing.go:149: server-21827d4e-199d-cef6-9f4c-689713aed83c: 2023-05-16T04:06:22.154+1400 [INFO] server-21827d4e-199d-cef6-9f4c-689713aed83c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daa8a26e-5b8f-d054-8241-34f612eba224 Address:daa8a26e-5b8f-d054-8241-34f612eba224} {Suffrage:Voter ID:server-21827d4e-199d-cef6-9f4c-689713aed83c Address:21827d4e-199d-cef6-9f4c-689713aed83c} {Suffrage:Voter ID:server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 Address:0e755218-6cff-c1f4-24ba-7ad78d6dfdb6}]" +testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.154+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daa8a26e-5b8f-d054-8241-34f612eba224 Address:daa8a26e-5b8f-d054-8241-34f612eba224} {Suffrage:Voter ID:server-21827d4e-199d-cef6-9f4c-689713aed83c Address:21827d4e-199d-cef6-9f4c-689713aed83c} {Suffrage:Voter ID:server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 Address:0e755218-6cff-c1f4-24ba-7ad78d6dfdb6}]" + testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.154+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daa8a26e-5b8f-d054-8241-34f612eba224 Address:daa8a26e-5b8f-d054-8241-34f612eba224} {Suffrage:Voter ID:server-21827d4e-199d-cef6-9f4c-689713aed83c Address:21827d4e-199d-cef6-9f4c-689713aed83c} {Suffrage:Voter ID:server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 Address:0e755218-6cff-c1f4-24ba-7ad78d6dfdb6}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:22.154+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:22.154+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-daa8a26e-5b8f-d054-8241-34f612eba224: 2023-05-16T04:06:22.154+1400 [INFO] server-daa8a26e-5b8f-d054-8241-34f612eba224: entering follower state: follower="Node at daa8a26e-5b8f-d054-8241-34f612eba224 [Follower]" leader= + testing.go:149: server-daa8a26e-5b8f-d054-8241-34f612eba224: 2023-05-16T04:06:22.154+1400 [INFO] server-daa8a26e-5b8f-d054-8241-34f612eba224: entering follower state: follower="Node at daa8a26e-5b8f-d054-8241-34f612eba224 [Follower]" leader= +testLoggerAdapter verbose: server-21827d4e-199d-cef6-9f4c-689713aed83c: 2023-05-16T04:06:22.154+1400 [INFO] server-21827d4e-199d-cef6-9f4c-689713aed83c: entering follower state: follower="Node at 21827d4e-199d-cef6-9f4c-689713aed83c [Follower]" leader= + testing.go:149: server-21827d4e-199d-cef6-9f4c-689713aed83c: 2023-05-16T04:06:22.154+1400 [INFO] server-21827d4e-199d-cef6-9f4c-689713aed83c: entering follower state: follower="Node at 21827d4e-199d-cef6-9f4c-689713aed83c [Follower]" leader= +testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.155+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: entering follower state: follower="Node at 0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 [Follower]" leader= + testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.155+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: entering follower state: follower="Node at 0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 [Follower]" leader= +testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.221+1400 [WARN] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.221+1400 [WARN] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.221+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: entering candidate state: node="Node at 0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 [Candidate]" term=2 + testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.221+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: entering candidate state: node="Node at 0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 [Candidate]" term=2 +testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.221+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: election won: tally=2 + testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.221+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: election won: tally=2 +testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: entering leader state: leader="Node at 0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 [Leader]" + testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: entering leader state: leader="Node at 0e755218-6cff-c1f4-24ba-7ad78d6dfdb6 [Leader]" +testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: added peer, starting replication: peer=server-daa8a26e-5b8f-d054-8241-34f612eba224 + testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: added peer, starting replication: peer=server-daa8a26e-5b8f-d054-8241-34f612eba224 +testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: added peer, starting replication: peer=server-21827d4e-199d-cef6-9f4c-689713aed83c + testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: added peer, starting replication: peer=server-21827d4e-199d-cef6-9f4c-689713aed83c +testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: pipelining replication: peer="{Voter server-21827d4e-199d-cef6-9f4c-689713aed83c 21827d4e-199d-cef6-9f4c-689713aed83c}" + testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: pipelining replication: peer="{Voter server-21827d4e-199d-cef6-9f4c-689713aed83c 21827d4e-199d-cef6-9f4c-689713aed83c}" +testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: pipelining replication: peer="{Voter server-daa8a26e-5b8f-d054-8241-34f612eba224 daa8a26e-5b8f-d054-8241-34f612eba224}" + testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.222+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: pipelining replication: peer="{Voter server-daa8a26e-5b8f-d054-8241-34f612eba224 daa8a26e-5b8f-d054-8241-34f612eba224}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:22.328+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:22.222709611 +1400 +14 m=+19.025291246 (1 nodes), 67.980582ms from start of poll, 69.033015ms from cluster start. Timeout at 2023-05-16 04:06:22.328822203 +1400 +14 m=+19.131403838, 106.112592ms after stability + testing.go:149: cluster: 2023-05-16T04:06:22.328+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:22.222709611 +1400 +14 m=+19.025291246 (1 nodes), 67.980582ms from start of poll, 69.033015ms from cluster start. Timeout at 2023-05-16 04:06:22.328822203 +1400 +14 m=+19.131403838, 106.112592ms after stability +testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.329+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: cannot transfer leadership to itself + testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.329+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: cannot transfer leadership to itself +testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.329+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: aborting pipeline replication: peer="{Voter server-21827d4e-199d-cef6-9f4c-689713aed83c 21827d4e-199d-cef6-9f4c-689713aed83c}" + testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.329+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: aborting pipeline replication: peer="{Voter server-21827d4e-199d-cef6-9f4c-689713aed83c 21827d4e-199d-cef6-9f4c-689713aed83c}" +testLoggerAdapter verbose: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.329+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: aborting pipeline replication: peer="{Voter server-daa8a26e-5b8f-d054-8241-34f612eba224 daa8a26e-5b8f-d054-8241-34f612eba224}" + testing.go:149: server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: 2023-05-16T04:06:22.329+1400 [INFO] server-0e755218-6cff-c1f4-24ba-7ad78d6dfdb6: aborting pipeline replication: peer="{Voter server-daa8a26e-5b8f-d054-8241-34f612eba224 daa8a26e-5b8f-d054-8241-34f612eba224}" --- PASS: TestRaft_LeadershipTransferToItself (0.18s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests raft_test.go:2064: Disabled unreliable test --- SKIP: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.00s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout -testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.639-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 Address:ac3c9ff1-6d22-876d-0dcf-db2b1be95c65} {Suffrage:Voter ID:server-3738d183-b7d0-fdce-2491-5120c847ada3 Address:3738d183-b7d0-fdce-2491-5120c847ada3} {Suffrage:Voter ID:server-e3ef3ece-af13-5659-8468-bb183482d52c Address:e3ef3ece-af13-5659-8468-bb183482d52c}]" - testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.639-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 Address:ac3c9ff1-6d22-876d-0dcf-db2b1be95c65} {Suffrage:Voter ID:server-3738d183-b7d0-fdce-2491-5120c847ada3 Address:3738d183-b7d0-fdce-2491-5120c847ada3} {Suffrage:Voter ID:server-e3ef3ece-af13-5659-8468-bb183482d52c Address:e3ef3ece-af13-5659-8468-bb183482d52c}]" -testLoggerAdapter verbose: server-3738d183-b7d0-fdce-2491-5120c847ada3: 2024-06-16T08:26:41.639-1200 [INFO] server-3738d183-b7d0-fdce-2491-5120c847ada3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 Address:ac3c9ff1-6d22-876d-0dcf-db2b1be95c65} {Suffrage:Voter ID:server-3738d183-b7d0-fdce-2491-5120c847ada3 Address:3738d183-b7d0-fdce-2491-5120c847ada3} {Suffrage:Voter ID:server-e3ef3ece-af13-5659-8468-bb183482d52c Address:e3ef3ece-af13-5659-8468-bb183482d52c}]" - testing.go:149: server-3738d183-b7d0-fdce-2491-5120c847ada3: 2024-06-16T08:26:41.639-1200 [INFO] server-3738d183-b7d0-fdce-2491-5120c847ada3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 Address:ac3c9ff1-6d22-876d-0dcf-db2b1be95c65} {Suffrage:Voter ID:server-3738d183-b7d0-fdce-2491-5120c847ada3 Address:3738d183-b7d0-fdce-2491-5120c847ada3} {Suffrage:Voter ID:server-e3ef3ece-af13-5659-8468-bb183482d52c Address:e3ef3ece-af13-5659-8468-bb183482d52c}]" -testLoggerAdapter verbose: server-e3ef3ece-af13-5659-8468-bb183482d52c: 2024-06-16T08:26:41.640-1200 [INFO] server-e3ef3ece-af13-5659-8468-bb183482d52c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 Address:ac3c9ff1-6d22-876d-0dcf-db2b1be95c65} {Suffrage:Voter ID:server-3738d183-b7d0-fdce-2491-5120c847ada3 Address:3738d183-b7d0-fdce-2491-5120c847ada3} {Suffrage:Voter ID:server-e3ef3ece-af13-5659-8468-bb183482d52c Address:e3ef3ece-af13-5659-8468-bb183482d52c}]" - testing.go:149: server-e3ef3ece-af13-5659-8468-bb183482d52c: 2024-06-16T08:26:41.640-1200 [INFO] server-e3ef3ece-af13-5659-8468-bb183482d52c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 Address:ac3c9ff1-6d22-876d-0dcf-db2b1be95c65} {Suffrage:Voter ID:server-3738d183-b7d0-fdce-2491-5120c847ada3 Address:3738d183-b7d0-fdce-2491-5120c847ada3} {Suffrage:Voter ID:server-e3ef3ece-af13-5659-8468-bb183482d52c Address:e3ef3ece-af13-5659-8468-bb183482d52c}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.640-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:41.640-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.640-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: entering follower state: follower="Node at ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 [Follower]" leader= - testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.640-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: entering follower state: follower="Node at ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 [Follower]" leader= -testLoggerAdapter verbose: server-3738d183-b7d0-fdce-2491-5120c847ada3: 2024-06-16T08:26:41.640-1200 [INFO] server-3738d183-b7d0-fdce-2491-5120c847ada3: entering follower state: follower="Node at 3738d183-b7d0-fdce-2491-5120c847ada3 [Follower]" leader= - testing.go:149: server-3738d183-b7d0-fdce-2491-5120c847ada3: 2024-06-16T08:26:41.640-1200 [INFO] server-3738d183-b7d0-fdce-2491-5120c847ada3: entering follower state: follower="Node at 3738d183-b7d0-fdce-2491-5120c847ada3 [Follower]" leader= -testLoggerAdapter verbose: server-e3ef3ece-af13-5659-8468-bb183482d52c: 2024-06-16T08:26:41.640-1200 [INFO] server-e3ef3ece-af13-5659-8468-bb183482d52c: entering follower state: follower="Node at e3ef3ece-af13-5659-8468-bb183482d52c [Follower]" leader= - testing.go:149: server-e3ef3ece-af13-5659-8468-bb183482d52c: 2024-06-16T08:26:41.640-1200 [INFO] server-e3ef3ece-af13-5659-8468-bb183482d52c: entering follower state: follower="Node at e3ef3ece-af13-5659-8468-bb183482d52c [Follower]" leader= -testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.696-1200 [WARN] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.696-1200 [WARN] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.696-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: entering candidate state: node="Node at ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 [Candidate]" term=2 - testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.696-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: entering candidate state: node="Node at ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 [Candidate]" term=2 -testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.696-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: election won: tally=2 - testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.696-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: election won: tally=2 -testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: entering leader state: leader="Node at ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 [Leader]" - testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: entering leader state: leader="Node at ac3c9ff1-6d22-876d-0dcf-db2b1be95c65 [Leader]" -testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: added peer, starting replication: peer=server-3738d183-b7d0-fdce-2491-5120c847ada3 - testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: added peer, starting replication: peer=server-3738d183-b7d0-fdce-2491-5120c847ada3 -testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: added peer, starting replication: peer=server-e3ef3ece-af13-5659-8468-bb183482d52c - testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: added peer, starting replication: peer=server-e3ef3ece-af13-5659-8468-bb183482d52c -testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: pipelining replication: peer="{Voter server-e3ef3ece-af13-5659-8468-bb183482d52c e3ef3ece-af13-5659-8468-bb183482d52c}" - testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: pipelining replication: peer="{Voter server-e3ef3ece-af13-5659-8468-bb183482d52c e3ef3ece-af13-5659-8468-bb183482d52c}" -testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: pipelining replication: peer="{Voter server-3738d183-b7d0-fdce-2491-5120c847ada3 3738d183-b7d0-fdce-2491-5120c847ada3}" - testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:41.697-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: pipelining replication: peer="{Voter server-3738d183-b7d0-fdce-2491-5120c847ada3 3738d183-b7d0-fdce-2491-5120c847ada3}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.802-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:41.69726464 -1200 -12 m=+18.704581101 (1 nodes), 56.9132ms from start of poll, 57.92894ms from cluster start. Timeout at 2024-06-16 08:26:41.80247928 -1200 -12 m=+18.809795761, 105.21466ms after stability - testing.go:149: cluster: 2024-06-16T08:26:41.802-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:41.69726464 -1200 -12 m=+18.704581101 (1 nodes), 56.9132ms from start of poll, 57.92894ms from cluster start. Timeout at 2024-06-16 08:26:41.80247928 -1200 -12 m=+18.809795761, 105.21466ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.802-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:41.802-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:41.908-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:41.80277744 -1200 -12 m=+18.810093921 (2 nodes), 7.68µs from start of poll, 163.44176ms from cluster start. Timeout at 2024-06-16 08:26:41.90839514 -1200 -12 m=+18.915711641, 105.61772ms after stability - testing.go:149: cluster: 2024-06-16T08:26:41.908-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:41.80277744 -1200 -12 m=+18.810093921 (2 nodes), 7.68µs from start of poll, 163.44176ms from cluster start. Timeout at 2024-06-16 08:26:41.90839514 -1200 -12 m=+18.915711641, 105.61772ms after stability -testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:42.344-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: aborting pipeline replication: peer="{Voter server-e3ef3ece-af13-5659-8468-bb183482d52c e3ef3ece-af13-5659-8468-bb183482d52c}" - testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:42.344-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: aborting pipeline replication: peer="{Voter server-e3ef3ece-af13-5659-8468-bb183482d52c e3ef3ece-af13-5659-8468-bb183482d52c}" -testLoggerAdapter verbose: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:42.344-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: aborting pipeline replication: peer="{Voter server-3738d183-b7d0-fdce-2491-5120c847ada3 3738d183-b7d0-fdce-2491-5120c847ada3}" - testing.go:149: server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: 2024-06-16T08:26:42.344-1200 [INFO] server-ac3c9ff1-6d22-876d-0dcf-db2b1be95c65: aborting pipeline replication: peer="{Voter server-3738d183-b7d0-fdce-2491-5120c847ada3 3738d183-b7d0-fdce-2491-5120c847ada3}" ---- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.71s) +testLoggerAdapter verbose: server-33a955cb-f8ac-72fc-2126-42378abd615c: 2023-05-16T04:06:22.331+1400 [INFO] server-33a955cb-f8ac-72fc-2126-42378abd615c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33a955cb-f8ac-72fc-2126-42378abd615c Address:33a955cb-f8ac-72fc-2126-42378abd615c} {Suffrage:Voter ID:server-58827ced-5746-ffdb-b21d-b7ff77619d86 Address:58827ced-5746-ffdb-b21d-b7ff77619d86} {Suffrage:Voter ID:server-a9bf173c-fced-bf88-7997-c052452235f4 Address:a9bf173c-fced-bf88-7997-c052452235f4}]" + testing.go:149: server-33a955cb-f8ac-72fc-2126-42378abd615c: 2023-05-16T04:06:22.331+1400 [INFO] server-33a955cb-f8ac-72fc-2126-42378abd615c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33a955cb-f8ac-72fc-2126-42378abd615c Address:33a955cb-f8ac-72fc-2126-42378abd615c} {Suffrage:Voter ID:server-58827ced-5746-ffdb-b21d-b7ff77619d86 Address:58827ced-5746-ffdb-b21d-b7ff77619d86} {Suffrage:Voter ID:server-a9bf173c-fced-bf88-7997-c052452235f4 Address:a9bf173c-fced-bf88-7997-c052452235f4}]" +testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.331+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33a955cb-f8ac-72fc-2126-42378abd615c Address:33a955cb-f8ac-72fc-2126-42378abd615c} {Suffrage:Voter ID:server-58827ced-5746-ffdb-b21d-b7ff77619d86 Address:58827ced-5746-ffdb-b21d-b7ff77619d86} {Suffrage:Voter ID:server-a9bf173c-fced-bf88-7997-c052452235f4 Address:a9bf173c-fced-bf88-7997-c052452235f4}]" + testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.331+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33a955cb-f8ac-72fc-2126-42378abd615c Address:33a955cb-f8ac-72fc-2126-42378abd615c} {Suffrage:Voter ID:server-58827ced-5746-ffdb-b21d-b7ff77619d86 Address:58827ced-5746-ffdb-b21d-b7ff77619d86} {Suffrage:Voter ID:server-a9bf173c-fced-bf88-7997-c052452235f4 Address:a9bf173c-fced-bf88-7997-c052452235f4}]" +testLoggerAdapter verbose: server-a9bf173c-fced-bf88-7997-c052452235f4: 2023-05-16T04:06:22.331+1400 [INFO] server-a9bf173c-fced-bf88-7997-c052452235f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33a955cb-f8ac-72fc-2126-42378abd615c Address:33a955cb-f8ac-72fc-2126-42378abd615c} {Suffrage:Voter ID:server-58827ced-5746-ffdb-b21d-b7ff77619d86 Address:58827ced-5746-ffdb-b21d-b7ff77619d86} {Suffrage:Voter ID:server-a9bf173c-fced-bf88-7997-c052452235f4 Address:a9bf173c-fced-bf88-7997-c052452235f4}]" + testing.go:149: server-a9bf173c-fced-bf88-7997-c052452235f4: 2023-05-16T04:06:22.331+1400 [INFO] server-a9bf173c-fced-bf88-7997-c052452235f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33a955cb-f8ac-72fc-2126-42378abd615c Address:33a955cb-f8ac-72fc-2126-42378abd615c} {Suffrage:Voter ID:server-58827ced-5746-ffdb-b21d-b7ff77619d86 Address:58827ced-5746-ffdb-b21d-b7ff77619d86} {Suffrage:Voter ID:server-a9bf173c-fced-bf88-7997-c052452235f4 Address:a9bf173c-fced-bf88-7997-c052452235f4}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:22.331+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:22.331+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-33a955cb-f8ac-72fc-2126-42378abd615c: 2023-05-16T04:06:22.332+1400 [INFO] server-33a955cb-f8ac-72fc-2126-42378abd615c: entering follower state: follower="Node at 33a955cb-f8ac-72fc-2126-42378abd615c [Follower]" leader= + testing.go:149: server-33a955cb-f8ac-72fc-2126-42378abd615c: 2023-05-16T04:06:22.332+1400 [INFO] server-33a955cb-f8ac-72fc-2126-42378abd615c: entering follower state: follower="Node at 33a955cb-f8ac-72fc-2126-42378abd615c [Follower]" leader= +testLoggerAdapter verbose: server-a9bf173c-fced-bf88-7997-c052452235f4: 2023-05-16T04:06:22.332+1400 [INFO] server-a9bf173c-fced-bf88-7997-c052452235f4: entering follower state: follower="Node at a9bf173c-fced-bf88-7997-c052452235f4 [Follower]" leader= + testing.go:149: server-a9bf173c-fced-bf88-7997-c052452235f4: 2023-05-16T04:06:22.332+1400 [INFO] server-a9bf173c-fced-bf88-7997-c052452235f4: entering follower state: follower="Node at a9bf173c-fced-bf88-7997-c052452235f4 [Follower]" leader= +testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.332+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: entering follower state: follower="Node at 58827ced-5746-ffdb-b21d-b7ff77619d86 [Follower]" leader= + testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.332+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: entering follower state: follower="Node at 58827ced-5746-ffdb-b21d-b7ff77619d86 [Follower]" leader= +testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [WARN] server-58827ced-5746-ffdb-b21d-b7ff77619d86: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [WARN] server-58827ced-5746-ffdb-b21d-b7ff77619d86: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: entering candidate state: node="Node at 58827ced-5746-ffdb-b21d-b7ff77619d86 [Candidate]" term=2 + testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: entering candidate state: node="Node at 58827ced-5746-ffdb-b21d-b7ff77619d86 [Candidate]" term=2 +testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: election won: tally=2 + testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: election won: tally=2 +testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: entering leader state: leader="Node at 58827ced-5746-ffdb-b21d-b7ff77619d86 [Leader]" + testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: entering leader state: leader="Node at 58827ced-5746-ffdb-b21d-b7ff77619d86 [Leader]" +testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: added peer, starting replication: peer=server-33a955cb-f8ac-72fc-2126-42378abd615c + testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: added peer, starting replication: peer=server-33a955cb-f8ac-72fc-2126-42378abd615c +testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: added peer, starting replication: peer=server-a9bf173c-fced-bf88-7997-c052452235f4 + testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.405+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: added peer, starting replication: peer=server-a9bf173c-fced-bf88-7997-c052452235f4 +testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.406+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: pipelining replication: peer="{Voter server-a9bf173c-fced-bf88-7997-c052452235f4 a9bf173c-fced-bf88-7997-c052452235f4}" + testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.406+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: pipelining replication: peer="{Voter server-a9bf173c-fced-bf88-7997-c052452235f4 a9bf173c-fced-bf88-7997-c052452235f4}" +testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.406+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: pipelining replication: peer="{Voter server-33a955cb-f8ac-72fc-2126-42378abd615c 33a955cb-f8ac-72fc-2126-42378abd615c}" + testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:22.406+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: pipelining replication: peer="{Voter server-33a955cb-f8ac-72fc-2126-42378abd615c 33a955cb-f8ac-72fc-2126-42378abd615c}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:22.512+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:22.40651346 +1400 +14 m=+19.209095095 (1 nodes), 74.535519ms from start of poll, 75.593492ms from cluster start. Timeout at 2023-05-16 04:06:22.512216688 +1400 +14 m=+19.314798383, 105.703288ms after stability + testing.go:149: cluster: 2023-05-16T04:06:22.512+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:22.40651346 +1400 +14 m=+19.209095095 (1 nodes), 74.535519ms from start of poll, 75.593492ms from cluster start. Timeout at 2023-05-16 04:06:22.512216688 +1400 +14 m=+19.314798383, 105.703288ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:22.512+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:22.512+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:22.618+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:22.512506651 +1400 +14 m=+19.315088266 (2 nodes), 3.14µs from start of poll, 181.586663ms from cluster start. Timeout at 2023-05-16 04:06:22.61838064 +1400 +14 m=+19.420962295, 105.874029ms after stability + testing.go:149: cluster: 2023-05-16T04:06:22.618+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:22.512506651 +1400 +14 m=+19.315088266 (2 nodes), 3.14µs from start of poll, 181.586663ms from cluster start. Timeout at 2023-05-16 04:06:22.61838064 +1400 +14 m=+19.420962295, 105.874029ms after stability +testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:23.163+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: aborting pipeline replication: peer="{Voter server-a9bf173c-fced-bf88-7997-c052452235f4 a9bf173c-fced-bf88-7997-c052452235f4}" + testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:23.163+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: aborting pipeline replication: peer="{Voter server-a9bf173c-fced-bf88-7997-c052452235f4 a9bf173c-fced-bf88-7997-c052452235f4}" +testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:23.164+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: aborting pipeline replication: peer="{Voter server-33a955cb-f8ac-72fc-2126-42378abd615c 33a955cb-f8ac-72fc-2126-42378abd615c}" + testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:23.164+1400 [INFO] server-58827ced-5746-ffdb-b21d-b7ff77619d86: aborting pipeline replication: peer="{Voter server-33a955cb-f8ac-72fc-2126-42378abd615c 33a955cb-f8ac-72fc-2126-42378abd615c}" +testLoggerAdapter verbose: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:23.215+1400 [ERROR] server-58827ced-5746-ffdb-b21d-b7ff77619d86: failed to heartbeat to: peer=33a955cb-f8ac-72fc-2126-42378abd615c error="command timed out" + testing.go:149: server-58827ced-5746-ffdb-b21d-b7ff77619d86: 2023-05-16T04:06:23.215+1400 [ERROR] server-58827ced-5746-ffdb-b21d-b7ff77619d86: failed to heartbeat to: peer=33a955cb-f8ac-72fc-2126-42378abd615c error="command timed out" +--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.89s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap -testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.346-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-573f1499-81ca-5c4b-96c8-f7824387fd3d Address:573f1499-81ca-5c4b-96c8-f7824387fd3d} {Suffrage:Voter ID:server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a Address:6f2f7c51-f298-28a0-e5d0-3a74a07d729a}]" - testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.346-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-573f1499-81ca-5c4b-96c8-f7824387fd3d Address:573f1499-81ca-5c4b-96c8-f7824387fd3d} {Suffrage:Voter ID:server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a Address:6f2f7c51-f298-28a0-e5d0-3a74a07d729a}]" -testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.346-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: entering follower state: follower="Node at 573f1499-81ca-5c4b-96c8-f7824387fd3d [Follower]" leader= - testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.346-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: entering follower state: follower="Node at 573f1499-81ca-5c4b-96c8-f7824387fd3d [Follower]" leader= -testLoggerAdapter verbose: server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: 2024-06-16T08:26:42.346-1200 [INFO] server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-573f1499-81ca-5c4b-96c8-f7824387fd3d Address:573f1499-81ca-5c4b-96c8-f7824387fd3d} {Suffrage:Voter ID:server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a Address:6f2f7c51-f298-28a0-e5d0-3a74a07d729a}]" - testing.go:149: server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: 2024-06-16T08:26:42.346-1200 [INFO] server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-573f1499-81ca-5c4b-96c8-f7824387fd3d Address:573f1499-81ca-5c4b-96c8-f7824387fd3d} {Suffrage:Voter ID:server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a Address:6f2f7c51-f298-28a0-e5d0-3a74a07d729a}]" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:42.346-1200 [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-06-16T08:26:42.346-1200 [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: 2024-06-16T08:26:42.346-1200 [INFO] server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: entering follower state: follower="Node at 6f2f7c51-f298-28a0-e5d0-3a74a07d729a [Follower]" leader= - testing.go:149: server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: 2024-06-16T08:26:42.346-1200 [INFO] server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: entering follower state: follower="Node at 6f2f7c51-f298-28a0-e5d0-3a74a07d729a [Follower]" leader= -testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [WARN] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [WARN] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: entering candidate state: node="Node at 573f1499-81ca-5c4b-96c8-f7824387fd3d [Candidate]" term=2 - testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: entering candidate state: node="Node at 573f1499-81ca-5c4b-96c8-f7824387fd3d [Candidate]" term=2 -testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: election won: tally=2 - testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: election won: tally=2 -testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: entering leader state: leader="Node at 573f1499-81ca-5c4b-96c8-f7824387fd3d [Leader]" - testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: entering leader state: leader="Node at 573f1499-81ca-5c4b-96c8-f7824387fd3d [Leader]" -testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: added peer, starting replication: peer=server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a - testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.403-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: added peer, starting replication: peer=server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a -testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.404-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: pipelining replication: peer="{Voter server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a 6f2f7c51-f298-28a0-e5d0-3a74a07d729a}" - testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.404-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: pipelining replication: peer="{Voter server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a 6f2f7c51-f298-28a0-e5d0-3a74a07d729a}" -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:42.510-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:42.40390224 -1200 -12 m=+19.411218721 (1 nodes), 56.91232ms from start of poll, 57.61294ms from cluster start. Timeout at 2024-06-16 08:26:42.50997246 -1200 -12 m=+19.517288941, 106.07022ms after stability - testing.go:149: cluster: 2024-06-16T08:26:42.510-1200 [INFO] stable state for Follower reached at 2024-06-16 08:26:42.40390224 -1200 -12 m=+19.411218721 (1 nodes), 56.91232ms from start of poll, 57.61294ms from cluster start. Timeout at 2024-06-16 08:26:42.50997246 -1200 -12 m=+19.517288941, 106.07022ms after stability -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:42.510-1200 [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-06-16T08:26:42.510-1200 [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-06-16T08:26:42.615-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:42.51022702 -1200 -12 m=+19.517543501 (1 nodes), 2µs from start of poll, 163.93772ms from cluster start. Timeout at 2024-06-16 08:26:42.6156204 -1200 -12 m=+19.622936881, 105.39338ms after stability - testing.go:149: cluster: 2024-06-16T08:26:42.615-1200 [INFO] stable state for Leader reached at 2024-06-16 08:26:42.51022702 -1200 -12 m=+19.517543501 (1 nodes), 2µs from start of poll, 163.93772ms from cluster start. Timeout at 2024-06-16 08:26:42.6156204 -1200 -12 m=+19.622936881, 105.39338ms after stability -testLoggerAdapter verbose: server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: 2024-06-16T08:26:42.627-1200 [INFO] server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-573f1499-81ca-5c4b-96c8-f7824387fd3d Address:573f1499-81ca-5c4b-96c8-f7824387fd3d} {Suffrage:Voter ID:server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a Address:6f2f7c51-f298-28a0-e5d0-3a74a07d729a}]" - testing.go:149: server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: 2024-06-16T08:26:42.627-1200 [INFO] server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-573f1499-81ca-5c4b-96c8-f7824387fd3d Address:573f1499-81ca-5c4b-96c8-f7824387fd3d} {Suffrage:Voter ID:server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a Address:6f2f7c51-f298-28a0-e5d0-3a74a07d729a}]" -testLoggerAdapter verbose: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.627-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: aborting pipeline replication: peer="{Voter server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a 6f2f7c51-f298-28a0-e5d0-3a74a07d729a}" - testing.go:149: server-573f1499-81ca-5c4b-96c8-f7824387fd3d: 2024-06-16T08:26:42.627-1200 [INFO] server-573f1499-81ca-5c4b-96c8-f7824387fd3d: aborting pipeline replication: peer="{Voter server-6f2f7c51-f298-28a0-e5d0-3a74a07d729a 6f2f7c51-f298-28a0-e5d0-3a74a07d729a}" ---- PASS: TestRaft_GetConfigurationNoBootstrap (0.28s) +testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.217+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f Address:0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f} {Suffrage:Voter ID:server-ee56d808-044e-5016-7ae3-7840711d323f Address:ee56d808-044e-5016-7ae3-7840711d323f}]" + testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.217+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f Address:0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f} {Suffrage:Voter ID:server-ee56d808-044e-5016-7ae3-7840711d323f Address:ee56d808-044e-5016-7ae3-7840711d323f}]" +testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.217+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: entering follower state: follower="Node at 0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f [Follower]" leader= +testLoggerAdapter verbose: server-ee56d808-044e-5016-7ae3-7840711d323f: 2023-05-16T04:06:23.217+1400 [INFO] server-ee56d808-044e-5016-7ae3-7840711d323f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f Address:0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f} {Suffrage:Voter ID:server-ee56d808-044e-5016-7ae3-7840711d323f Address:ee56d808-044e-5016-7ae3-7840711d323f}]" + testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.217+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: entering follower state: follower="Node at 0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f [Follower]" leader= + testing.go:149: server-ee56d808-044e-5016-7ae3-7840711d323f: 2023-05-16T04:06:23.217+1400 [INFO] server-ee56d808-044e-5016-7ae3-7840711d323f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f Address:0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f} {Suffrage:Voter ID:server-ee56d808-044e-5016-7ae3-7840711d323f Address:ee56d808-044e-5016-7ae3-7840711d323f}]" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:23.217+1400 [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-05-16T04:06:23.217+1400 [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-ee56d808-044e-5016-7ae3-7840711d323f: 2023-05-16T04:06:23.217+1400 [INFO] server-ee56d808-044e-5016-7ae3-7840711d323f: entering follower state: follower="Node at ee56d808-044e-5016-7ae3-7840711d323f [Follower]" leader= + testing.go:149: server-ee56d808-044e-5016-7ae3-7840711d323f: 2023-05-16T04:06:23.217+1400 [INFO] server-ee56d808-044e-5016-7ae3-7840711d323f: entering follower state: follower="Node at ee56d808-044e-5016-7ae3-7840711d323f [Follower]" leader= +testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.277+1400 [WARN] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.277+1400 [WARN] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.277+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: entering candidate state: node="Node at 0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f [Candidate]" term=2 + testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.277+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: entering candidate state: node="Node at 0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f [Candidate]" term=2 +testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.278+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: election won: tally=2 + testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.278+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: election won: tally=2 +testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.278+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: entering leader state: leader="Node at 0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f [Leader]" + testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.278+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: entering leader state: leader="Node at 0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f [Leader]" +testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.278+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: added peer, starting replication: peer=server-ee56d808-044e-5016-7ae3-7840711d323f + testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.278+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: added peer, starting replication: peer=server-ee56d808-044e-5016-7ae3-7840711d323f +testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.278+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: pipelining replication: peer="{Voter server-ee56d808-044e-5016-7ae3-7840711d323f ee56d808-044e-5016-7ae3-7840711d323f}" + testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.278+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: pipelining replication: peer="{Voter server-ee56d808-044e-5016-7ae3-7840711d323f ee56d808-044e-5016-7ae3-7840711d323f}" +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:23.383+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:23.278537431 +1400 +14 m=+20.081119046 (1 nodes), 60.948019ms from start of poll, 61.760229ms from cluster start. Timeout at 2023-05-16 04:06:23.383737832 +1400 +14 m=+20.186319487, 105.200441ms after stability + testing.go:149: cluster: 2023-05-16T04:06:23.383+1400 [INFO] stable state for Follower reached at 2023-05-16 04:06:23.278537431 +1400 +14 m=+20.081119046 (1 nodes), 60.948019ms from start of poll, 61.760229ms from cluster start. Timeout at 2023-05-16 04:06:23.383737832 +1400 +14 m=+20.186319487, 105.200441ms after stability +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:23.383+1400 [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-05-16T04:06:23.383+1400 [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-05-16T04:06:23.489+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:23.384060536 +1400 +14 m=+20.186642171 (1 nodes), 2.74µs from start of poll, 167.283354ms from cluster start. Timeout at 2023-05-16 04:06:23.48944092 +1400 +14 m=+20.292022555, 105.380384ms after stability + testing.go:149: cluster: 2023-05-16T04:06:23.489+1400 [INFO] stable state for Leader reached at 2023-05-16 04:06:23.384060536 +1400 +14 m=+20.186642171 (1 nodes), 2.74µs from start of poll, 167.283354ms from cluster start. Timeout at 2023-05-16 04:06:23.48944092 +1400 +14 m=+20.292022555, 105.380384ms after stability +testLoggerAdapter verbose: server-ee56d808-044e-5016-7ae3-7840711d323f: 2023-05-16T04:06:23.501+1400 [INFO] server-ee56d808-044e-5016-7ae3-7840711d323f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f Address:0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f} {Suffrage:Voter ID:server-ee56d808-044e-5016-7ae3-7840711d323f Address:ee56d808-044e-5016-7ae3-7840711d323f}]" + testing.go:149: server-ee56d808-044e-5016-7ae3-7840711d323f: 2023-05-16T04:06:23.501+1400 [INFO] server-ee56d808-044e-5016-7ae3-7840711d323f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f Address:0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f} {Suffrage:Voter ID:server-ee56d808-044e-5016-7ae3-7840711d323f Address:ee56d808-044e-5016-7ae3-7840711d323f}]" +testLoggerAdapter verbose: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.501+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: aborting pipeline replication: peer="{Voter server-ee56d808-044e-5016-7ae3-7840711d323f ee56d808-044e-5016-7ae3-7840711d323f}" + testing.go:149: server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: 2023-05-16T04:06:23.501+1400 [INFO] server-0bbd210e-c35b-c6f3-3f7f-a8437ef8bd8f: aborting pipeline replication: peer="{Voter server-ee56d808-044e-5016-7ae3-7840711d323f ee56d808-044e-5016-7ae3-7840711d323f}" +--- PASS: TestRaft_GetConfigurationNoBootstrap (0.29s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_WithAdvertise @@ -3807,7 +3884,7 @@ === RUN TestBackoff --- PASS: TestBackoff (0.00s) PASS -ok github.com/hashicorp/raft 19.655s +ok github.com/hashicorp/raft 20.344s ? github.com/hashicorp/raft/bench [no test files] create-stamp debian/debhelper-build-stamp fakeroot debian/rules binary @@ -3839,12 +3916,14 @@ dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration +I: user script /srv/workspace/pbuilder/12871/tmp/hooks/B01_cleanup starting +I: user script /srv/workspace/pbuilder/12871/tmp/hooks/B01_cleanup finished I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env -I: removing directory /srv/workspace/pbuilder/1861 and its subdirectories -I: Current time: Sun Jun 16 08:26:53 -12 2024 -I: pbuilder-time-stamp: 1718569613 +I: removing directory /srv/workspace/pbuilder/12871 and its subdirectories +I: Current time: Tue May 16 04:06:36 +14 2023 +I: pbuilder-time-stamp: 1684159596